issues

This would remove the csvs-to-sqlite step which I end up using for almost everything.

I'm hesitant to introduce pandas as a required dependency though since it require compiling numpy. Could build it so this option is only available if you have pandas installed.

Immutable assumes no other process can modify the file. An option to open reqdonly instead would enable other processes to update the file in place.

At least for the first round of visualizations, any time you construct one using the UI the result should include an "embed this" button that returns source code to copy and paste

These examples should use unpkg.com (or similarl) urls with SRI hashes, eg https://www.srihash.org - and should load data from the datasette JSON API.

https://github.com/Leaflet/Leaflet.markercluster would allow us to paginate-load in an enormous set of rows with latitude/longitude points, e.g. https://australian-dunnies.now.sh/

Here's a demo of it loading 50,000 markers: https://leaflet.github.io/Leaflet.markercluster/example/marker-clustering-realworld.50000.html - and it looks like it's easy to support progress bars for if we were iteratively loading 1,000 markers at a time using datasette pagination.

This command introspects the databases and writes out a JSON summary.

I think I'd like to use datasette build for something more interesting, potentially duplicating functionality from https://github.com/simonw/csvs-to-sqlite

Since the internal method that does this is called ds.inspect() that seems like a reasonable replacement name for the command.

e.g. https://sf-trees.now.sh/sf-trees-ebc2ad9/Street_Tree_List_fts?_search=grove should work

Eg on https://sf-trees.now.sh/sf-trees-ebc2ad9/Street_Tree_List?qSpecies=1

Relates to #86. If you are viewing a filtered page e.g. https://fivethirtyeight.datasettes.com/fivethirtyeight-2628db9/bob-ross%2Felements-by-episode?CLOUDS=1 we should show the count of matching rows.

Since this could be an expensive operation, we will run it with a strict time limit (maybe 50ms). If the time limit is exceeded we will display "many" instead, perhaps? Maybe even link to a count(*) query that would get the full 1000ms time limit which the user can click on if they like (that could even Ajax-in the result).

Eg https://sf-trees.now.sh/sf-trees-ebc2ad9/Street_Tree_List/1

SQLite databases that use FTS include a number of automatically generated tables, e.g.:

https://sf-trees-search.now.sh/sf-trees-search-a899b92

Of these, only the Street_Tree_List table is actually relevant to the user.

We can detect which tables are FTS tables by first finding the virtual tables:

sqlite> .headers on
sqlite> select * from sqlite_master where rootpage = 0;
type|name|tbl_name|rootpage|sql
table|Search|Search|0|CREATE VIRTUAL TABLE "Street_Tree_List_fts" USING FTS4 ("qAddress", "qCaretaker", "qSpecies")

Then parsing the above to figure out which ones are USING FTS? - then assume that any table which starts with that Street_Tree_List_fts prefix was created to support search:

sqlite> select * from sqlite_master where type='table' and tbl_name like 'Street_Tree_List_fts%';
type|name|tbl_name|rootpage|sql
table|Search_content|Search_content|10355|CREATE TABLE 'Street_Tree_List_fts_content'(docid INTEGER PRIMARY KEY, 'c0qAddress', 'c1qCaretaker', 'c2qSpecies')
table|Search_segments|Search_segments|10356|CREATE TABLE 'Street_Tree_List_fts_segments'(blockid INTEGER PRIMARY KEY, block BLOB)
table|Search_segdir|Search_segdir|10357|CREATE TABLE 'Street_Tree_List_fts_segdir'(level INTEGER,idx INTEGER,start_block INTEGER,leaves_end_block INTEGER,end_block INTEGER,root BLOB,PRIMARY KEY(level, idx))
table|Search_docsize|Search_docsize|10359|CREATE TABLE 'Street_Tree_List_fts_docsize'(docid INTEGER PRIMARY KEY, size BLOB)
table|Search_stat|Search_stat|10360|CREATE TABLE 'Street_Tree_List_fts_stat'(id INTEGER PRIMARY KEY, value BLOB)

We won't hide these completely - instead, we'll default the database index view to not showing them with a message that says "5 hidden tables" and support ?_hidden=1 to display them.

Let's do the thing where we attempt to show an accurate count if it can be done in less than 50ms

Here's an example query that searches all FTS indexed columns in a table: https://sf-trees-search.now.sh/sf-trees-search-a899b92?sql=select+*+from+Street_Tree_List+where+rowid+in+%28select+rowid+from+Street_Tree_List_fts+where+Street_Tree_List_fts+match+%27grove+london+dpw%27%29%0D%0A

And here's a query that searches a specific column: https://sf-trees-search.now.sh/sf-trees-search-a899b92?sql=select+*+from+Street_Tree_List+where+rowid+in+%28select+rowid+from+Street_Tree_List_fts+where+qSpecies+match+%27london%27%29%0D%0A

If we detect that a table has FTS enabled (which we can do by looking for it as a content table reference in another FTS table's create definition) we should add a search box to the table page which constructs this query - maybe using ?_search=XXX in the query string?

<s>To support search against specified columns, we can do ?_search__ qSpecies=London.</s> - not necessary, see comment below.

• [x] Detect if a table has a FTS index defined against it as a content= parameter
• [x] Decide what to do if there is more than one FTS index (maybe just pick the first one?)
• [x] Add the ?_search= query string argument
• [x] Add the UI