This worked for me:
<td class="col-{{ cell.column|to_css_class }} type-{{ cell.value_type }}">{{ cell.value | replace('", "','; ') | replace('[\"','') | replace('\"]','')}}</td>

I'm sure there is a prettier (and more flexible) way, but for now, this is ever-so-much more pleasant to look at.

BEFORE:
https://user-images.githubusercontent.com/192568/114062871-01402800-9866-11eb-91ce-91efe4ee45cd.png">

AFTER:
https://user-images.githubusercontent.com/192568/114062829-f2f20c00-9865-11eb-891d-b0c348a9b433.png">

(Note: I didn't figure out how to have one item have no semicolon, while multi-items close with a semicolon, but this is good enough for now. I also didn't figure out how to set up a new jinja filter. I don't want to add to /datasette/utils/init.py as I assume that would get overwritten when upgrading datasette. Having a starter guide on creating jinja filters in datasette would be helpful. (The jinja documentation isn't datasette-specific enough for me to quite nail it.)

https://latest.datasette.io/fixtures?sql=explain+select+*+from+paginated_view will be an interesting test query - because paginated_view is defined like this:

CREATE VIEW paginated_view AS
    SELECT
        content,
        '- ' || content || ' -' AS content_extra
    FROM no_primary_key;

So this will help test that the mechanism isn't confused by output columns that are created through a concatenation expression.

Here's some older example code that works with opcodes from Python, in this case to output indexes used by a query: https://github.com/plasticityai/supersqlite/blob/master/supersqlite/idxchk.py

I may need to do something special for rowid columns - there is a RowId opcode that might come into play here.

There are trained ML models used in Photoprism:
- https://dl.photoprism.org/tensorflow/nasnet.zip
- https://dl.photoprism.org/tensorflow/nsfw.zip

This may be obsoleted by #1293 - it looks like I may be able to auto-detect these foreign keys for arbitrary queries after all.

Extracting variables with this trick appears to work OK, but you have to pass the correct variables to the explain select... query. Using defaultdict seems to work there:

>>> rows = conn.execute('explain select * from repos where id = :id', defaultdict(int))
>>> [dict(r) for r in rows if r['opcode'] == 'Variable']
[{'addr': 2,
  'opcode': 'Variable',
  'p1': 1,
  'p2': 1,
  'p3': 0,
  'p4': ':id',
  'p5': 0,
  'comment': None}]

This almost works, but throws errors with some queries (anything with a rowid column for example) - it needs a bunch of test coverage.

def columns_for_query(conn, sql):
    rows = conn.execute('explain ' + sql).fetchall()
    table_rootpage_by_register = {r['p1']: r['p2'] for r in rows if r['opcode'] == 'OpenRead'}
    names_by_rootpage = dict(
        conn.execute(
            'select rootpage, name from sqlite_master where rootpage in ({})'.format(
                ', '.join(map(str, table_rootpage_by_register.values()))
            )
        )
    )
    columns_by_column_register = {}
    for row in rows:
        if row['opcode'] == 'Column':
            addr, opcode, table_id, cid, column_register, p4, p5, comment = row
            table = names_by_rootpage[table_rootpage_by_register[table_id]]
            columns_by_column_register[column_register] = (table, cid)
    result_row = [dict(r) for r in rows if r['opcode'] == 'ResultRow'][0]
    registers = list(range(result_row["p1"], result_row["p1"] + result_row["p2"] - 1))
    all_column_names = {}
    for table in names_by_rootpage.values():
        table_xinfo = conn.execute('pragma table_xinfo({})'.format(table)).fetchall()
        for row in table_xinfo:
            all_column_names[(table, row["cid"])] = row["name"]
    final_output = []
    for r in registers:
        try:
            table, cid = columns_by_column_register[r]
            final_output.append((table, all_column_names[table, cid]))
        except KeyError:
            final_output.append((None, None))
    return final_output

http://www.sqlite.org/draft/lang_explain.html says:

Applications should not use EXPLAIN or EXPLAIN QUERY PLAN since their exact behavior is variable and only partially documented.

I'm going to keep exploring this though.

... that output might also provide a better way to extract variables than the current mechanism using a regular expression, by looking for the Variable opcodes.

Essential documentation for understanding that output: https://www.sqlite.org/opcode.html

Had a fantastic suggestion on the SQLite forum: it might be possible to get what I want by interpreting the opcodes output by explain select ....

Copying the reply I posted to this thread:

That's really useful, thanks! It looks like it might be possible for me to reconstruct where each column came from using the explain select output.

Here's a complex example: https://calands.datasettes.com/calands?sql=explain+select%0D%0A++AsGeoJSON%28geometry%29%2C+*%0D%0Afrom%0D%0A++CPAD_2020a_SuperUnits%0D%0Awhere%0D%0A++PARK_NAME+like+%27%25mini%25%27+and%0D%0A++Intersects%28GeomFromGeoJSON%28%3Afreedraw%29%2C+geometry%29+%3D+1%0D%0A++and+CPAD_2020a_SuperUnits.rowid+in+%28%0D%0A++++select%0D%0A++++++rowid%0D%0A++++from%0D%0A++++++SpatialIndex%0D%0A++++where%0D%0A++++++f_table_name+%3D+%27CPAD_2020a_SuperUnits%27%0D%0A++++++and+search_frame+%3D+GeomFromGeoJSON%28%3Afreedraw%29%0D%0A++%29&freedraw=%7B%22type%22%3A%22MultiPolygon%22%2C%22coordinates%22%3A%5B%5B%5B%5B-122.42202758789064%2C37.82280243352759%5D%2C%5B-122.39868164062501%2C37.823887203271454%5D%2C%5B-122.38220214843751%2C37.81846319511331%5D%2C%5B-122.35061645507814%2C37.77071473849611%5D%2C%5B-122.34924316406251%2C37.74465712069939%5D%2C%5B-122.37258911132814%2C37.703380457832374%5D%2C%5B-122.39044189453125%2C37.690340943717715%5D%2C%5B-122.41241455078126%2C37.680559803205135%5D%2C%5B-122.44262695312501%2C37.67295135774715%5D%2C%5B-122.47283935546876%2C37.67295135774715%5D%2C%5B-122.52502441406251%2C37.68382032669382%5D%2C%5B-122.53463745117189%2C37.6892542140253%5D%2C%5B-122.54699707031251%2C37.690340943717715%5D%2C%5B-122.55798339843751%2C37.72945260537781%5D%2C%5B-122.54287719726564%2C37.77831314799672%5D%2C%5B-122.49893188476564%2C37.81303878836991%5D%2C%5B-122.46185302734376%2C37.82822612280363%5D%2C%5B-122.42889404296876%2C37.82822612280363%5D%2C%5B-122.42202758789064%2C37.82280243352759%5D%5D%5D%5D%7D

It looks like the opcodes I need to inspect are OpenRead, Column and ResultRow.

OpenRead tells me which tables are being opened - the p2 value (in this case 51) corresponds to the rootpage column in sqlite_master here: https://calands.datasettes.com/calands?sql=select+*+from+sqlite_master - it gets assigned to the register in p1.

The Column opcodes tell me which columns are being read - p1 is that table reference, and p2 is the cid of the column within that table.

The ResultRow opcode then tells me which columns are used in the results. 15 16 means start at the 15th and then read the next 16 columns.

I think this might work!

Sadly it doesn't do what I need. This query should only return one column, but instead I get back every column that was consulted by the query:

https://user-images.githubusercontent.com/9599/113524385-216ca000-9563-11eb-8a7e-cbbe5dfc02f6.png">

Oh wow, I just spotted https://github.com/macbre/sql-metadata

Uses tokenized query returned by python-sqlparse and generates query metadata. Extracts column names and tables used by the query. Provides a helper for normalization of SQL queries and tables aliases resolving.

It's for MySQL, PostgreSQL and Hive right now but maybe getting it working with SQLite wouldn't be too hard?

The other approach I considered for this was to have my own SQL query parser running in Python, which could pick apart a complex query and figure out which column was sourced from which table. I dropped this idea because it felt that the moment select * came into play a pure parsing approach wouldn't work - I'd need knowledge of the schema in order to resolve the *.

A Python parser approach might be good enough to handle a subset of queries - those that don't use select * for example - and maybe that would be worth shipping? The feature doesn't have to be perfect for it to be useful.

I asked about this on the SQLite forum: https://sqlite.org/forum/forumpost/0180277fb7

One option I've not fully explored yet: could I write my own custom SQLite C extension which exposes this functionality as a callable function?

Then I could load that extension and run a SQL query something like this:

select database, table, column from analyze_query(:sql_query)

Where analyze_query(...) would be a fancy virtual table function of some sort that uses the underlying sqlite3_column_database_name() C functions to analyze the SQL query and return details of what it would return.

Worth noting that adding limit 0 to the query still causes it to conduct the permission checks, hopefully while avoiding doing any of the actual work of executing the query:

In [20]: db.execute('select * from compound_primary_key join facetable on facetable.rowid = compound_primary_key.rowid limit 0').fetchall()
    ...: 
args (21, None, None, None, None) kwargs {}
args (20, 'compound_primary_key', 'pk1', 'main', None) kwargs {}
args (20, 'compound_primary_key', 'pk2', 'main', None) kwargs {}
args (20, 'compound_primary_key', 'content', 'main', None) kwargs {}
args (20, 'facetable', 'pk', 'main', None) kwargs {}
args (20, 'facetable', 'created', 'main', None) kwargs {}
args (20, 'facetable', 'planet_int', 'main', None) kwargs {}
args (20, 'facetable', 'on_earth', 'main', None) kwargs {}
args (20, 'facetable', 'state', 'main', None) kwargs {}
args (20, 'facetable', 'city_id', 'main', None) kwargs {}
args (20, 'facetable', 'neighborhood', 'main', None) kwargs {}
args (20, 'facetable', 'tags', 'main', None) kwargs {}
args (20, 'facetable', 'complex_array', 'main', None) kwargs {}
args (20, 'facetable', 'distinct_some_null', 'main', None) kwargs {}
args (20, 'facetable', 'pk', 'main', None) kwargs {}
args (20, 'compound_primary_key', 'ROWID', 'main', None) kwargs {}
Out[20]: []

Here are all of the available constants:

In [3]: for k in dir(sqlite3):
   ...:     if k.startswith("SQLITE_"):
   ...:         print(k, getattr(sqlite3, k))
   ...: 
SQLITE_ALTER_TABLE 26
SQLITE_ANALYZE 28
SQLITE_ATTACH 24
SQLITE_CREATE_INDEX 1
SQLITE_CREATE_TABLE 2
SQLITE_CREATE_TEMP_INDEX 3
SQLITE_CREATE_TEMP_TABLE 4
SQLITE_CREATE_TEMP_TRIGGER 5
SQLITE_CREATE_TEMP_VIEW 6
SQLITE_CREATE_TRIGGER 7
SQLITE_CREATE_VIEW 8
SQLITE_CREATE_VTABLE 29
SQLITE_DELETE 9
SQLITE_DENY 1
SQLITE_DETACH 25
SQLITE_DONE 101
SQLITE_DROP_INDEX 10
SQLITE_DROP_TABLE 11
SQLITE_DROP_TEMP_INDEX 12
SQLITE_DROP_TEMP_TABLE 13
SQLITE_DROP_TEMP_TRIGGER 14
SQLITE_DROP_TEMP_VIEW 15
SQLITE_DROP_TRIGGER 16
SQLITE_DROP_VIEW 17
SQLITE_DROP_VTABLE 30
SQLITE_FUNCTION 31
SQLITE_IGNORE 2
SQLITE_INSERT 18
SQLITE_OK 0
SQLITE_PRAGMA 19
SQLITE_READ 20
SQLITE_RECURSIVE 33
SQLITE_REINDEX 27
SQLITE_SAVEPOINT 32
SQLITE_SELECT 21
SQLITE_TRANSACTION 22
SQLITE_UPDATE 23

A more promising route I found involved the db.set_authorizer method. This can be used to log the permission checks that SQLite uses, including checks for permission to access specific columns of specific tables. For a while I thought this could work!

>>> def print_args(*args, **kwargs):
...    print("args", args, "kwargs", kwargs)
...    return sqlite3.SQLITE_OK

>>> db = sqlite3.connect("fixtures.db")
>>> db.execute('select * from compound_primary_key join facetable on rowid').fetchall()
args (21, None, None, None, None) kwargs {}
args (20, 'compound_primary_key', 'pk1', 'main', None) kwargs {}
args (20, 'compound_primary_key', 'pk2', 'main', None) kwargs {}
args (20, 'compound_primary_key', 'content', 'main', None) kwargs {}
args (20, 'facetable', 'pk', 'main', None) kwargs {}
args (20, 'facetable', 'created', 'main', None) kwargs {}
args (20, 'facetable', 'planet_int', 'main', None) kwargs {}
args (20, 'facetable', 'on_earth', 'main', None) kwargs {}
args (20, 'facetable', 'state', 'main', None) kwargs {}
args (20, 'facetable', 'city_id', 'main', None) kwargs {}
args (20, 'facetable', 'neighborhood', 'main', None) kwargs {}
args (20, 'facetable', 'tags', 'main', None) kwargs {}
args (20, 'facetable', 'complex_array', 'main', None) kwargs {}
args (20, 'facetable', 'distinct_some_null', 'main', None) kwargs {}

Those 20 values (where 20 is SQLITE_READ) looked like they were checking permissions for the columns in the order they would be returned!

Then I found a snag:

In [18]: db.execute('select 1 + 1 + (select max(rowid) from facetable)')
args (21, None, None, None, None) kwargs {}
args (31, None, 'max', None, None) kwargs {}
args (20, 'facetable', 'pk', 'main', None) kwargs {}
args (21, None, None, None, None) kwargs {}
args (20, 'facetable', '', None, None) kwargs {}

Once a subselect is involved the order of the 20 checks no longer matches the order in which the columns are returned from the query.

I've done various pieces of research into this over the past few years. Capturing what I've discovered in this ticket.

The SQLite C API has functions that can help with this: https://www.sqlite.org/c3ref/column_database_name.html details those. But they're not exposed in the Python SQLite library.

Maybe it would be possible to use them via ctypes? My hunch is that I would have to re-implement the full sqlite3 module with ctypes, which sounds daunting.

Could maybe replace this code: https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/utils/__init__.py#L1021-L1026

Useful tutorial series from 2012: https://northredoubt.com/n/2012/01/20/spatialite-speed-test/

Relevant code is some of the most complex in all of Datasette.

https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/views/table.py#L530-L594

And

https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/views/table.py#L743-L771

I'll need to think hard about how to refactor this out into something more understandable before implementing previous links.

Same query again with a, d, v returns 0 results, which is also as we would want: it signifies that we are back to the very first page: https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+pk3%2C+content+from+compound_three_primary_keys+where+%28%28pk1+%3C+%3Ap0%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3C+%3Ap1%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3D+%3Ap1+and+pk3+%3C+%3Ap2%29%29+order+by+pk1+desc%2C+pk2+desc%2C+pk3+desc+limit+1+offset+99&p0=a&p1=d&p2=v

I tried flipping the direction of the sort and the comparison operators and got this: https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+pk3%2C+content+from+compound_three_primary_keys+where+%28%28pk1+%3C+%3Ap0%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3C+%3Ap1%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3D+%3Ap1+and+pk3+%3C+%3Ap2%29%29+order+by+pk1+desc%2C+pk2+desc%2C+pk3+desc+limit+1+offset+99&p0=a&p1=h&p2=r

select pk1, pk2, pk3, content from compound_three_primary_keys where ((pk1 < :p0)
  or
(pk1 = :p0 and pk2 < :p1)
  or
(pk1 = :p0 and pk2 = :p1 and pk3 < :p2)) order by pk1 desc, pk2 desc, pk3 desc limit 1 offset 99

Which returned a-d-v as desired. I messed around with it to find the limit 1 offset 99 values.

Let's figure out the SQL for this. The most complex case is probably this one: https://latest.datasette.io/fixtures/compound_three_primary_keys?_next=a%2Ch%2Cr

Here's the SQL for that page: https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+pk3%2C+content+from+compound_three_primary_keys+where+%28%28pk1+%3E+%3Ap0%29%0A++or%0A%28pk1+%3D+%3Ap0+and+pk2+%3E+%3Ap1%29%0A++or%0A%28pk1+%3D+%3Ap0+and+pk2+%3D+%3Ap1+and+pk3+%3E+%3Ap2%29%29+order+by+pk1%2C+pk2%2C+pk3+limit+101&p0=a&p1=h&p2=r

select pk1, pk2, pk3, content from compound_three_primary_keys where ((pk1 > :p0)
  or
(pk1 = :p0 and pk2 > :p1)
  or
(pk1 = :p0 and pk2 = :p1 and pk3 > :p2)) order by pk1, pk2, pk3 limit 101

Where p0 is a, p1 is h and p2 is r.

Given the above, how would I figure out the correct previous link? It should be https://latest.datasette.io/fixtures/compound_three_primary_keys?_next=a%2Cd%2Cv - a, d, v.

I think my ideal implementation for this would be to reverse the order, grab the previous page-size-plus-one items, then return a ?_next=x token that would provide the previous page sorted back in the expected default order.

The alternative would be to have a ?_previous=x token which can be used to paginate backwards in reverse order, but I think this would be confusing as it would result in "hit next page, then hit previous page" returning you to a new state which features rows in the reverse order.

https://twitter.com/llanga/status/1378431719934681094 looks like I should wait for 3.9.4, out in a few days.

Codecov Report

Merging #1291 (73342b6) into main (0a7621f) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1291   +/-   ##
=======================================
  Coverage   91.51%   91.51%           
=======================================
  Files          34       34           
  Lines        4255     4255           
=======================================
  Hits         3894     3894           
  Misses        361      361           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0a7621f...73342b6. Read the comment docs.

For what it's worth, if anyone finds this in the future, I was having the same issue.

After digging through the code, it turned out that the database download is only available if it the db served in immutable mode, so datasette serve -i xyz.db rather than the doc's quickstart recommendation of datasette serve xyz.db.

I found one example of an implementation of reversed keyset pagination here: https://github.com/tvainika/objection-keyset-pagination/blob/cb21a493c96daa6e63c302efae6718d09aa11661/index.js#L74-L79

Codecov Report

Merging #1290 (2fb1e42) into main (87b583a) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1290   +/-   ##
=======================================
  Coverage   91.51%   91.51%           
=======================================
  Files          34       34           
  Lines        4255     4255           
=======================================
  Hits         3894     3894           
  Misses        361      361           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 87b583a...2fb1e42. Read the comment docs.

https://github.com/simonw/datasette/actions/runs/713207828 ran with pytest-xdist in 4m22s:

https://user-images.githubusercontent.com/9599/113466592-f151bf80-93f1-11eb-9d74-b830e96c19c8.png">

Here's the test suite running on regular pytest in 5m13s:

https://user-images.githubusercontent.com/9599/113466606-00387200-93f2-11eb-95a3-24521c628cbe.png">

Not a huge speed-up because there are only 2 available cores in the GitHub Actions environment, but still worthwhile - especially since this lets people run in parallel on their own laptops.

Looks like -n auto only detected two cores on GitHub Actions: https://github.com/simonw/datasette/runs/2257597137?check_suite_focus=true

============================= test session starts ==============================
platform linux -- Python 3.7.10, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
SQLite: 3.31.1
rootdir: /home/runner/work/datasette/datasette, configfile: pytest.ini
plugins: xdist-2.2.1, timeout-1.4.2, forked-1.3.0, asyncio-0.14.0
gw0 I / gw1 I
gw0 [878] / gw1 [878]

On my Mac pytest-xdist ran the test suite (minus two tests) in 59s, as opposed to 2m23s without xdist.

Yes, this would be nice!

I using Datasette v0.56 and don't see a previous page button.

Yes, I agree.

Alternatively, maybe the header could be at the top and bottom, above the next page button.

Maybe even have the header 50 records down?

Hello again,

I was able to get my facets running with this settings.json, which was lifted from one of Simon's datasette's and slightly modified.

{
    "default_page_size": 100,
    "max_returned_rows": 1000,
    "num_sql_threads": 3,
    "sql_time_limit_ms": 9000,
    "default_facet_size": 10,
    "facet_time_limit_ms": 9000,
    "facet_suggest_time_limit_ms": 500,
    "hash_urls": false,
    "allow_facet": true,
    "suggest_facets": false,
    "default_cache_ttl": 5,
    "default_cache_ttl_hashed": 31536000,
    "cache_size_kb": 0,
    "allow_csv_stream": true,
    "max_csv_mb": 100,
    "truncate_cells_html": 2048,
    "template_debug": false,
    "base_url": "/"
}

Hello,

I'm also experiencing a timeout in my environment. I don't know if it's because I need more indexes or a more powerful system.

My data has 1,271,111 and when I try to create a facet, there's a time out. I've tried this on two different rows that should significantly filter down data: CITY and PARTY_REG.

Simon's johns_hopkins_csse_daily_reports has more rows and it setup with two facets on load. He does have four indexes created, though. Do I need more indexes?

I have one simple one so far:

CREATE INDEX [idx_party_reg]
    ON [county_active] ([PARTY_REG]);

I'm running Datasette 0.56 installed via pip with Python 3.7.3.

4.19.0-10-amd64 #1 SMP Debian 4.19.132-1 (2020-07-24) x86_64 GNU/Linux

$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster

This shows the city in a different color (and not the comma), but I get the idea, and I like it. (Ooh, could be nice to have the gear have an option in array fields to show as bullets or commas or semicolons...)

python:3.9.3-slim-buster isn't on Docker Hub yet either.

Idea: default to displaying single-dimension JSON arrays of strings as a comma-separated list but show the comma in a different colour - something like this:

https://user-images.githubusercontent.com/9599/113445005-dce9d480-93a9-11eb-901c-0c2fb079c150.png">

I used this HTML for the prototype (re-using .type-int just to get the colour):

<td class="col-tags type-str">tag1<span class="type-int">, </span>tag2</td>

This makes senses - showing an array as ["blah", "blah2", "blah3"] isn't particularly human-friendly!

Once again having tests for the Dockerfile as seen in #1272 would be useful here.

https://devcenter.heroku.com/articles/python-support#supported-runtimes looks like Heroku still only have 3.9.2 for the moment.

A few places:

https://github.com/simonw/datasette/blob/7b1a9a1999eb9326ce8ec830d75ac200e5279c46/Dockerfile#L1

https://github.com/simonw/datasette/blob/8e18c7943181f228ce5ebcea48deb59ce50bee1f/datasette/utils/__init__.py#L350

https://github.com/simonw/datasette/blob/8e18c7943181f228ce5ebcea48deb59ce50bee1f/tests/test_package.py#L14

https://github.com/simonw/datasette/blob/8e18c7943181f228ce5ebcea48deb59ce50bee1f/datasette/publish/heroku.py#L177-L178

👋 could I help somehow for this to be merged? As Big Sur is going to be more used as the time goes I think it would be nice to merge and publish a new version. Nice work!

Fantastic. Thank you!

+1 on using nested queries to achieve this! Would be great as streaming CSV is an amazing feature.

Some UX/DX details:

I was expecting it to work to simply add &_stream=on to custom SQL queries because the docs say

Any Datasette table, view or custom SQL query can be exported as CSV.

After a bit of testing back and forth I realized streaming only works for full tables.

Would love this feature because I'm using pandas.read_csv to paint graphs from custom queries and the graphs are cut off because of the 1000 row limit.

Maybe the addition of two template files: 'one_database_index.html' and 'one_table_index.html' would be a better idea than the documentation diff idea. (They could include commented instructions to rename the preferred template 'index.html', along with any other necessary guidance.)

You're right, doing this is really hard at the moment - I'm not sure I know how I would tackle this either, and it's something I've wanted in the past!

I'll have a think about this one.

Thanks!

Codecov Report

Merging #1282 (08f7427) into main (0486303) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1282   +/-   ##
=======================================
  Coverage   91.51%   91.51%           
=======================================
  Files          34       34           
  Lines        4255     4255           
=======================================
  Hits         3894     3894           
  Misses        361      361           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0486303...08f7427. Read the comment docs.

I tried this just now against Datasette 0.56 with the new Dockerfile from #1249 (that uses SQLite and SpatiaLite installed with apt-get install) and the tests all passed.

Sorry I didn't get to this PR sooner. I've joint-credited you in the release notes for this fix: https://docs.datasette.io/en/stable/changelog.html#v0-56

https://github.com/simonw/datasette/runs/2214871602?check_suite_focus=true worked:

https://user-images.githubusercontent.com/9599/112776676-270f3680-8ff5-11eb-90d1-533d4e82f677.png">

Here's the 0.56 image on Docker Hub: https://hub.docker.com/layers/datasetteproject/datasette/0.56/images/sha256-701fc0f299a0ea79434a4852c46dab351254b9ac25dbe3c5f36fd5360caf52f9?context=explore

Got this error:

"docker tag" requires exactly 2 arguments.
See 'docker tag --help'.

Usage:  docker tag SOURCE_IMAGE[:TAG] TARGET_IMAGE[:TAG]

Create a tag TARGET_IMAGE that refers to SOURCE_IMAGE

I'm going to build a new GitHub Actions workflow for this that lets me manually specify a tag to build and push as a Docker image.

I'm going to attempt to fix this manually for the 0.56 release, by building and tagging it by hand and then pushing the 0.56 tag to Docker Hub.

https://docs.docker.com/engine/reference/commandline/push/#push-all-tags-of-an-image

Use the -a (or --all-tags) option to push all tags of a local image.

https://github.com/simonw/datasette/runs/1763835467?check_suite_focus=true for Datasette 0.54 worked, and the output included this:

Successfully tagged ***/datasette:0.54
The push refers to repository [docker.io/***/datasette]
aedd33c6b161: Preparing
...
aedd33c6b161: Pushed
0.54: digest: sha256:65c7e579d1c29755dac5c1ca86b1e97fa88c48bd3d724ac3e02988d0da296140 size: 2005
aedd33c6b161: Preparing
...
5dacd731af1b: Layer already exists
latest: digest: sha256:65c7e579d1c29755dac5c1ca86b1e97fa88c48bd3d724ac3e02988d0da296140 size: 2005

Here's that same section of output from the 0.56 release:

Successfully tagged ***/datasette:0.56
Using default tag: latest
The push refers to repository [docker.io/***/datasette]
4d4a9976adcc: Preparing
...
9b2132a0d5cf: Pushed
latest: digest: sha256:2250d0fbe57b1d615a8d6df0c9d43deb9533532e00bac68854773d8ff8dcf00a size: 1793

The difference here is the "Using default tag: latest" bit.

Here's the diff between Dockerfile in 0.54.1 and 0.56: https://github.com/simonw/datasette/compare/0.54.1...0.56#diff-551d1fcf87f78cc3bc18a7b332a4dc5d8773a512062df881c5aba28a6f5c48d7

https://user-images.githubusercontent.com/9599/112774760-79e5ef80-8fef-11eb-886a-5d546d6d2f3d.png">

I just shipped Datasette 0.56 - here's the CI run: https://github.com/simonw/datasette/runs/2214701802?check_suite_focus=true

It pushed a new latest tag to https://hub.docker.com/r/datasetteproject/datasette/tags?page=1&ordering=last_updated

docker pull datasetteproject/datasette:latest

And then:

docker run datasetteproject/datasette:latest datasette \
  --load-extension=spatialite \
  --get /-/versions.json | jq .sqlite

Outputs:

{
  "version": "3.27.2",
  "fts_versions": [
    "FTS5",
    "FTS4",
    "FTS3"
  ],
  "extensions": {
    "json1": null,
    "spatialite": "5.0.1"
  },
  "compile_options": [
    "COMPILER=gcc-8.3.0",
    "ENABLE_COLUMN_METADATA",
    "ENABLE_DBSTAT_VTAB",
    "ENABLE_FTS3",
    "ENABLE_FTS3_PARENTHESIS",
    "ENABLE_FTS3_TOKENIZER",
    "ENABLE_FTS4",
    "ENABLE_FTS5",
    "ENABLE_JSON1",
    "ENABLE_LOAD_EXTENSION",
    "ENABLE_PREUPDATE_HOOK",
    "ENABLE_RTREE",
    "ENABLE_SESSION",
    "ENABLE_STMTVTAB",
    "ENABLE_UNLOCK_NOTIFY",
    "ENABLE_UPDATE_DELETE_LIMIT",
    "HAVE_ISNAN",
    "LIKE_DOESNT_MATCH_BLOBS",
    "MAX_SCHEMA_RETRY=25",
    "MAX_VARIABLE_NUMBER=250000",
    "OMIT_LOOKASIDE",
    "SECURE_DELETE",
    "SOUNDEX",
    "TEMP_STORE=1",
    "THREADSAFE=1",
    "USE_URI"
  ]
}

This bug should have been fixed in #1229 - let me know if that's not the case!

Thanks

This is interesting!

I've decided to apply a subset of these - the if and elif blocks are a deliberate style choice from me, because I find code clearer when it has if/else as opposed to relying on early termination. Likewise the iteration against .keys() on dictionaries.

I like the other fixes though, I'm about to land them in a separate commit that credits you.

Codecov Report

Merging #1229 (a095248) into main (8e18c79) will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main    #1229   +/-   ##
=======================================
  Coverage   91.51%   91.51%           
=======================================
  Files          34       34           
  Lines        4255     4255           
=======================================
  Hits         3894     3894           
  Misses        361      361           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8e18c79...a095248. Read the comment docs.

Thanks for figuring this out!

Neat fix, thank you!

Thanks for the fix.

Might be easier to do this using https://github.com/coleifer/pysqlite3 rather than try to replace the system sqlite3 on the Ubuntu GitHub Actions instances.

These instructions should help: https://github.com/coleifer/pysqlite3#building-a-statically-linked-library

Testing this on Glitch by adding https://github.com/simonw/datasette/archive/48d5e0e6ac8975cfd869d4e8c69c64ca0c65e29e.zip as a dependency. That fixed it.

Aha! https://www.sqlite.org/pragma.html says:

The table-valued functions for PRAGMA feature was added in SQLite version 3.16.0 (2017-01-02). Prior versions of SQLite cannot use this feature.

Here's where I run that: https://github.com/simonw/datasette/blob/6f41c8a2bef309a66588b2875c3e24d26adb4850/datasette/database.py#L249-L253

That's from when I added the --crossdb option in #1232: https://github.com/simonw/datasette/commit/6f41c8a2bef309a66588b2875c3e24d26adb4850#diff-4e20309c969326a0008dc9237f6807f48d55783315fbfc1e7dfa480b550e16f9R249

The Glitch server logs showed:

ERROR: conn=<sqlite3.Connection object at 0x7f88ebadd8f0>, sql = 'select seq, name, file from pragma_database_list() where seq > 0', params = None: no such table: pragma_database_list

I just managed to replicate this bug on Glitch: https://nicar-2020.glitch.me/data

Invalid SQL
no such table: pragma_database_list

https://nicar-2020.glitch.me/-/versions says:

{
    "python": {
        "version": "3.7.10",
        "full": "3.7.10 (default, Feb 20 2021, 21:21:24) \n[GCC 5.4.0 20160609]"
    },
    "datasette": {
        "version": "0.55"
    },
    "asgi": "3.0",
    "uvicorn": "0.13.4",
    "sqlite": {
        "version": "3.11.0",
        "fts_versions": [
            "FTS4",
            "FTS3"
        ],
        "extensions": {
            "json1": null
        },
        "compile_options": [
            "ENABLE_COLUMN_METADATA",
            "ENABLE_DBSTAT_VTAB",
            "ENABLE_FTS3",
            "ENABLE_FTS3_PARENTHESIS",
            "ENABLE_JSON1",
            "ENABLE_LOAD_EXTENSION",
            "ENABLE_RTREE",
            "ENABLE_UNLOCK_NOTIFY",
            "ENABLE_UPDATE_DELETE_LIMIT",
            "HAVE_ISNAN",
            "LIKE_DOESNT_MATCH_BLOBS",
            "MAX_SCHEMA_RETRY=25",
            "OMIT_LOOKASIDE",
            "SECURE_DELETE",
            "SOUNDEX",
            "SYSTEM_MALLOC",
            "TEMP_STORE=1",
            "THREADSAFE=1"
        ]
    }
}

That's SQLite 3.11.0 from 2016-02-15 with no FTS5.

Codecov Report

Merging #1279 (14d8977) into main (3fcfc85) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1279   +/-   ##
=======================================
  Coverage   91.51%   91.51%           
=======================================
  Files          34       34           
  Lines        4255     4255           
=======================================
  Hits         3894     3894           
  Misses        361      361           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3fcfc85...14d8977. Read the comment docs.

That rivers example in the tutorial would work a lot better with a live demo.

I have a better recipe for using spatial indexes now on https://simonwillison.net/2021/Jan/24/drawing-shapes-spatialite/

select
  AsGeoJSON(geometry), *
from
  CPAD_2020a_SuperUnits
where
  PARK_NAME like '%mini%' and
  Intersects(GeomFromGeoJSON(:freedraw), geometry) = 1
  and CPAD_2020a_SuperUnits.rowid in (
    select
      rowid
    from
      SpatialIndex
    where
      f_table_name = 'CPAD_2020a_SuperUnits'
      and search_frame = GeomFromGeoJSON(:freedraw)
  )

This will give you back an additional column of GeoJSON. You can copy and paste GeoJSON from this column into the debugging tool at geojson.io to visualize it on a map.

That should promote datasette-leaflet-geojson instead.

Related feature request: ability to set default values for canned queries: #1258

Idea for these: imitate https://django-sql-dashboard.readthedocs.io/en/latest/widgets.html#custom-widgets and drive them with templates.

So a custom widget type of textarea would look for a template called widgets/textarea.html - which means users could define brand new custom widgets just by creating their own template files.

https://timezones-api.datasette.io/ is now up and running on Cloud Run.

https://github.com/simonw/timezones-api is that project, it's pretty old now. I'll try to get it running on Cloud Run.

Right now they look like this:

databases:
  fixtures:
    queries:
      neighborhood_search:
        params:
        - text

In addition to being able to specify defaults, I'd also like to add other things in the future - most significantly the ability to specify a different input widget (e.g. textarea v.s. single-line input)

So maybe this looks like:

params:
- name: text
  default: ""
- name: age
  widget: number

I like that idea.

I'll close this issue after I ship Datasette 0.56 and confirm that the Dockerfile was correctly built and published to Docker Hub.

One last test of that Dockerfile:

(datasette) datasette % docker build -f Dockerfile -t datasetteproject/datasette:0.55a --build-arg VERSION=0.55 .
(datasette) datasette % docker run datasetteproject/datasette:0.55a datasette --get '/-/versions.json' | jq
{
  "python": {
    "version": "3.9.2",
    "full": "3.9.2 (default, Feb 19 2021, 17:23:45) \n[GCC 8.3.0]"
  },
  "datasette": {
    "version": "0.55"
  },
  "asgi": "3.0",
  "uvicorn": "0.13.4",
  "sqlite": {
    "version": "3.27.2",
    "fts_versions": [
      "FTS5",
      "FTS4",
      "FTS3"
    ],
    "extensions": {
      "json1": null
    },
    "compile_options": [
      "COMPILER=gcc-8.3.0",
      "ENABLE_COLUMN_METADATA",
      "ENABLE_DBSTAT_VTAB",
      "ENABLE_FTS3",
      "ENABLE_FTS3_PARENTHESIS",
      "ENABLE_FTS3_TOKENIZER",
      "ENABLE_FTS4",
      "ENABLE_FTS5",
      "ENABLE_JSON1",
      "ENABLE_LOAD_EXTENSION",
      "ENABLE_PREUPDATE_HOOK",
      "ENABLE_RTREE",
      "ENABLE_SESSION",
      "ENABLE_STMTVTAB",
      "ENABLE_UNLOCK_NOTIFY",
      "ENABLE_UPDATE_DELETE_LIMIT",
      "HAVE_ISNAN",
      "LIKE_DOESNT_MATCH_BLOBS",
      "MAX_SCHEMA_RETRY=25",
      "MAX_VARIABLE_NUMBER=250000",
      "OMIT_LOOKASIDE",
      "SECURE_DELETE",
      "SOUNDEX",
      "TEMP_STORE=1",
      "THREADSAFE=1",
      "USE_URI"
    ]
  }
}
(datasette) datasette % docker run datasetteproject/datasette:0.55a datasette --get '/-/versions.json' --load-extension=spatialite | jq
{
  "python": {
    "version": "3.9.2",
    "full": "3.9.2 (default, Feb 19 2021, 17:23:45) \n[GCC 8.3.0]"
  },
  "datasette": {
    "version": "0.55"
  },
  "asgi": "3.0",
  "uvicorn": "0.13.4",
  "sqlite": {
    "version": "3.27.2",
    "fts_versions": [
      "FTS5",
      "FTS4",
      "FTS3"
    ],
    "extensions": {
      "json1": null,
      "spatialite": "5.0.1"
    },
    "compile_options": [
      "COMPILER=gcc-8.3.0",
      "ENABLE_COLUMN_METADATA",
      "ENABLE_DBSTAT_VTAB",
      "ENABLE_FTS3",
      "ENABLE_FTS3_PARENTHESIS",
      "ENABLE_FTS3_TOKENIZER",
      "ENABLE_FTS4",
      "ENABLE_FTS5",
      "ENABLE_JSON1",
      "ENABLE_LOAD_EXTENSION",
      "ENABLE_PREUPDATE_HOOK",
      "ENABLE_RTREE",
      "ENABLE_SESSION",
      "ENABLE_STMTVTAB",
      "ENABLE_UNLOCK_NOTIFY",
      "ENABLE_UPDATE_DELETE_LIMIT",
      "HAVE_ISNAN",
      "LIKE_DOESNT_MATCH_BLOBS",
      "MAX_SCHEMA_RETRY=25",
      "MAX_VARIABLE_NUMBER=250000",
      "OMIT_LOOKASIDE",
      "SECURE_DELETE",
      "SOUNDEX",
      "TEMP_STORE=1",
      "THREADSAFE=1",
      "USE_URI"
    ]
  }
}

I'm skipping this for the moment because the new Dockerfile shape introduced in https://github.com/simonw/datasette/issues/1249#issuecomment-804404544 isn't compatible with this technique, since it installs Datasette from PyPI rather than directly from the repo.

Will need to change that if I want to do this unit tests thing.

Part of the challenge here is only running if a Docker daemon is available. I think this pattern works, in tests/test_dockerfile.py:

import httpx
import pathlib
import pytest
import subprocess

root = pathlib.Path(__file__).parent.parent

def docker_is_available():
    try:
        client = httpx.Client(
            transport=httpx.HTTPTransport(uds="/var/run/docker.sock")
        )
        client.get("http://docker/info")
        return True
    except httpx.ConnectError:
        return False


@pytest.fixture
def build_container():
    assert (root / "Dockerfile").exists()
    subprocess.check_call([
        "docker", "build", str(root), "-t", "datasette-dockerfile-test"
    ])


@pytest.mark.skipif(not docker_is_available(),
    reason="Docker is not available"
)
def test_dockerfile(build_container):
    output = subprocess.check_output([
        "docker", "run", "datasette-dockerfile-test", "datasette", "--get", "/_memory?sql=select+1&shape=_array"
    ])
    assert False, "Implement better assertion here"

That's really odd. What version of SQLite are you using on the server? You can tell by visiting https://your-site/-/versions

I fixed this already, it's a duplicate of #1239

Already got that! It's the --nl option - works for both importing and exporting data: https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-json-data

Idea: enhance lambda to allow it to return a dictionary of values, which will then be used to populate new columns. Use a --multicolumn option to indicate this:

sqlite-utils convert lambda mydb.db mytable mycolumn \
  --code '{"first_name": value.split()[0], "last_name": value.split()[1]}' \
  --multicolumn --drop

The --drop means "drop the mycolumn column after making this change".

Maybe --multi is a better name than --multicolumn here, since either way it's going to need additional explanation somewhere.

Would this overlap with #239 at all?

Here's the full current implementation of that tool: https://github.com/simonw/sqlite-transform/blob/0.5/sqlite_transform/cli.py

My current plan is to make this functionality available as the following:

sqlite-utils convert jsonsplit mydb.db mytable mycolumn
sqlite-utils convert parsedatetime mydb.db mytable mycolumn
sqlite-utils convert parsedate mydb.db mytable mycolumn
sqlite-utils convert lambda mydb.db mytable mycolumn --code='str(value).upper()'

What about allowing default parameters when defining the query in metadata.yml? Something like:

databases:
  fec:
    queries:
      search_by_name:
        params:
            - q
        default-param-values:
             q: "text to search"
        sql: |-
          SELECT...

For now, I'm using a custom database-<file>.html file that hardcodes a default param in the link, but I'd rather not customize the template just for that.

datasette package is a mostly unmaintained feature at this point - it has a bit of test coverage but I've not made any improvements to it in a few years, and I don't use it for my own projects.

I'll make this change to package at the same time as I land it for publish though.

Ah, okay so --extra-options applies to both datasette publish and datasette package? There wren't any examples of it being used with publish in the docs, so this tripped me up for a bit.

Yes this is one of the main reasons I'm planning to switch to encouraging YAML be default instead of JSON (while still supporting JSON) - YAML supports comments and multi-line strings.

See #1153 for YAML by default in the documentation.

One issue I could see with adding first class support for metadata in hjson format is that this would require adding an additional dependency to handle this, for a feature that would be unused by many users. I wonder if this could fit in as a plugin instead; if a hook existed for loading metadata (maybe as part of https://github.com/simonw/datasette/issues/860) the metadata could then come from any source, as specified by plugins, e.g. hjson, toml, XML, a database table etc.

Until/unless this exists, a few ideas for how you could add comments:
- Using YAML as you suggest.
- A common pattern is adding a "comment" key for comments to any object in JSON - I don't think including an unnecessary key like this would break anything in Datasette, but not certain.
- You could use another tool as a preprocessor for your JSON metadata - e.g. hjson or Jsonnet. You'd write the metadata in that format, and then convert that into JSON to actually use as your final metadata.