issue_comments

For consistency can reuse the icon used on selected facets:

Should have used porter stemming for this.

It looks to me like each of those <item> blocks has a number of guesses in order of confidence: xml <item x="215" y="190" w="187" h="39"> <t w="57">wonders,</t> <t w="55">wanders,</t> <t w="52">wonders ?</t> <t w="45">wonders</t> <t w="42">wonders.</t> </item> So maybe the best approach here is to just take the first t element within each item.

The XML for the OCR stuff is a bit weird. Currently I'm doing this to it:

https://github.com/dogsheep/evernote-to-sqlite/blob/c33d7b043a45eb3e88676e5fa3ce31755199d9f8/evernote_to_sqlite/utils.py#L70-L78

This can produce some odd results, for example:

Sure 'Sure, 'Sure. Sure, Sure. sure sure. sure ? If you If Yau [you live jive In m 1n an area devoid of natural wonders, wanders, wonders ? wonders wonders. your mind will be blown, blown' blown. blown ? -e i ? ,1 IL it ? at ? KY ? fl ft bat at

Which came from this image:

The XML for that is:

xml <recoIndex docType="unknown" objType="image" objID="05ffb72b307bf495f064243c7099d94f" engineVersion="6.5.17.7" recoType="service" lang="en" objWidth="1000" objHeight="1504"> <item x="68" y="75" w="104" h="37"> <t w="60">Sure</t> <t w="52">'Sure,</t> <t w="47">'Sure.</t> <t w="33">Sure,</t> <t w="26">Sure.</t> </item> <item x="182" y="83" w="92" h="26"> <t w="62">sure</t> <t w="58">sure.</t> <t w="46">sure ?</t> </item> <item x="69" y="132" w="107" h="45"> <t w="81">If you</t> <t w="64">If Yau</t> <t w="31">[you</t> </item> <item x="186" y="132" w="67" h="35"> <t w="85">live</t> <t w="51">jive</t> </item> <item x="263" y="140" w="36" h="27"> <t w="82">In</t> <t w="56">m</t> <t w="53">1n</t> </item> <item x="309" y="140" w="53" h="27"> <t w="82">an</t> </item> <item x="372" y="141" w="90" h="26"> <t w="94">area</t> </item> <item x="472" y="132" w="138" h="35"> <t w="85">devoid</t> </item> <item x="620" y="132" w="43" h="35"> <t w="82">of</t> </item> <item x="68" y="190" w="137" h="35"> <t w="87">natural</t> </item> <item x="215" y="190" w="187" h="39"> <t w="57">wonders,</t> <t w="55">wanders,</t> <t w="52">wonders ?</t> <t w="45">wonders</t> <t w="42">wonders.</t> </item> <item x="410" y="198" w="98" h="36"> <t w="88">your</t> </item> <item x="518" y="190" w="102" h="35"> <t w="86">mind</t> </item> <item x="630" y="190" w="69" h="34"> <t w="87">will</t> </item> <item x="709" y="190" w="55" h="35"> <t w="82">be</t> </item> <item x="774" y="190" w="137" h="34"> <t w="56">blown,</t> <t w="55">blown'</t> <t w="48">blown.</t> <t w="48">blown ?</t> </item> <item x="166" y="736" w="8" h="6"> <t w="66">-e</t> </item> <item x="273" y="966" w="29" h="21"> <t w="11">i ?</t> </item> <item x="281" y="1004" w="28" h="11"> <t w="11">,1</t> </item> <item x="512" y="1083" w="10" h="7"> <t w="10">IL</t> </item> <item x="29" y="1447" w="7" h="23"> <t w="17">it ?</t> <t w="15">at ?</t> <t w="13">KY ?</t> </item> <item x="414" y="841" w="8" h="16"> <t w="22">fl</t> <t w="20">ft</t> <t w="20">bat</t> <t w="19">at</t> </item> </recoIndex>

I haven't done the FTS on OCR yet. I'm going to move that to another ticket because it requires more thought.

... but it's still important to be able to get to the rendered note directly from the browse notes /evernote/notes page. Maybe use a simple render_cell() hook that just knows how to generate the link to the rendered note page?

Maybe the best way do this is with a custom route, /-/evernote/note-id - that way I can clean the HTML and resolve the other things in the <en-note> structure without using render_cell() and the like. My concern about using render_cell() is that it could lead to weird security problems when combined with ?sql= queries.

Or... I could do this client-side. JavaScript that looks for <en-media> tags and fetches the data using fetch() wouldn't be too hard to write.

Alternatively, rather than relying on datasette-media this could base64-embed the images. evernote-to-sqlite could register itself as a Datasette plugin that knows how to do this.

Maybe rename the column to evernote_content and register a render cell hook that knows how to rewrite those note bodies so that they are visible?

Might need to feed them through Bleach too, just in case any nasty code can get into them.

We could even do server-side thumbnailing for some of these images, but I'm inclined to serve up the full size ones and set a width on the image element based on the width attribute on <en-media>.

A live demo would be good too.

Since plugins are installed via pip this would require Datasette to be restarted. This StackOverflow thread looks relevant to that: https://stackoverflow.com/questions/11329917/restart-python-script-from-within-itself

This recipe looks promising: ```python import os import sys import psutil import logging

def restart_program(): """Restarts the current program, with file objects and descriptors cleanup """

try:
    p = psutil.Process(os.getpid())
    for handler in p.get_open_files() + p.connections():
        os.close(handler.fd)
except Exception, e:
    logging.error(e)

python = sys.executable
os.execl(python, python, *sys.argv)

`` https://docs.python.org/3/library/os.html#os.execl says aboutos.execl`:

These functions all execute a new program, replacing the current process; they do not return. On Unix, the new executable is loaded into the current process, and will have the same process id as the caller

Here's the datasette-json-preview plugin I'll be using to experiment with different formats: https://github.com/simonw/datasette-json-preview

Building this plugin reminded me of an oddity of the register_output_renderer() plugin hook: one of the arguments that can be passed to it is data, which is the default internal data structure created by Datasette - but I deliberately avoided documenting that on https://docs.datasette.io/en/stable/plugin_hooks.html#register-output-renderer-datasette because it's not a stable interface.

That's not ideal. I'd like custom renderers to be able to access this data to get at things like suggested facets, on an opt-in basis.

So maybe that kind of stuff is re-implemented as "extras" which are awaitable callables - then renderer plugins can call the extras that they need to as part of their execution.

To illustrate the problem (in this case the need to access next_url) here's my first prototype of the plugin: ```python from datasette import hookimpl from datasette.utils.asgi import Response

@hookimpl def register_output_renderer(datasette): return { "extension": "json-preview", "render": json_preview, }

def json_preview(data, columns, rows): next_url = data.get("next_url") headers = {} if next_url: headers["link"] = '<{}>; rel="next"'.format(next_url) return Response.json([dict(zip(columns, row)) for row in rows], headers=headers) ```

I'm going to prototype what it would look like if the default shape was a list of objects and ?_extra= turns that into an object with a rows key, in a plugin. As a separate extension (maybe .json-preview).

Maybe .jsonfull becomes a new renderer that returns ALL of the defined ?_extra= blocks.

Or... ?_extra=all turns on ALL of the available information blocks (some of which can come from plugins).

Since the total count can be expensive to calculate, I'm inclined to make that an opt-in extra - maybe ?_extra=count.

Based on that, the default JSON shape could look something like this:

json { "rows": [{"id": 1}, {"id": 2}], "next": "2", "next_url": "/db/table?_next=2" } And with ?_extra=count: json { "rows": [{"id": 1}, {"id": 2}], "next": "2", "next_url": "/db/table?_next=2", "count": 31 }

The core issue that I keep reconsidering is whether the default .json representation should be an object or a list.

Arguments in favour of a list:

• It's what I always want. Almost all of the code that I've written against the API myself uses ?_shape=array.
• It's really easy to use. You can pipe it to e.g. sqlite-utils insert, you can load it into JavaScript without thinking about it.

Arguments against:

• Nowhere to put pagination or total counts. I added pagination to the link: HTTP header in #1014 (inspired by the WordPress and GitHub APIs) but I haven't solved for total count, and there's other stuff that's useful like "truncated": true to indicate that more than 1000 results were returned and they were truncated.
• An array is inherently non-extensible: if the root item is an object it's easy to add new features to it in a backwards-compatible way in the future. An array is a fixed format.

But maybe that last point is a positive? It ensures the default .json format remains completely predictable forever.

If .json DID default to an array of objects, the ?_shape= argument could still be used to get back alternative formats.

Maybe .json?_extra=total changes the shape of that default to be this instead:

json { "rows": [{"id": 1}, {"id": 2}], "total": 104 }

The thing I care about most though is next_url. That could be provided like so:

.json?_extra=total&_extra=next - alternative syntax .json?_extra=total,next:

json { "rows": [{"id": 1}, {"id": 2}], "total": 104, "next": "2", "next_url": "/db/table.json?_extra=total&_extra=next&_next=2" } This is feeling a bit verbose for a common combination though.

Should that default also include "columns" as a list of strings? That would be duplicate data of the keys in the "rows" list of objects, and I've never found myself wanting it in my own code - so I'm going to say no.

Demo using paginate-json: % paginate-json 'https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array' | jq '. | length' https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Cd%2Cv http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Ch%2Cr http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Cl%2Cn http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Cp%2Cj http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Ct%2Cf http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Cx%2Cb http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=b%2Ca%2Cx http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=b%2Ce%2Ct http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=b%2Ci%2Cp http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=b%2Cm%2Cl 1001 New documentation: https://docs.datasette.io/en/latest/json_api.html#pagination