issue_comments

Related: Here's an implementation of a get_metadata() plugin hook by @brandonrobertz https://github.com/next-LI/datasette/commit/3fd8ce91f3108c82227bf65ff033929426c60437

I can use td.type-pk instead - here's the existing HTML:

<td class="col-Link type-pk"><a href="/link/link/1">1</a></td>

Demo:

echo '{"link": "https://example.com/"}' | sqlite-utils insert link.db link -

https://user-images.githubusercontent.com/9599/115949159-cfdf7300-a487-11eb-84d1-437d954d1841.png">

Here's why: https://github.com/simonw/datasette/blob/6ed9238178a56da5fb019f37fb1e1e15886be1d1/datasette/static/app.css#L435-L437

I could have a page about error codes in the docs, then have https://datasette.io/E123 style URLs for each error core which are shown when that error occurs and redirect to the corresponding documentation section.

Can enforce these with a documentation unit test.

https://latest.datasette.io/fixtures?sql=explain+select+*+from+paginated_view will be an interesting test query - because paginated_view is defined like this:

CREATE VIEW paginated_view AS
    SELECT
        content,
        '- ' || content || ' -' AS content_extra
    FROM no_primary_key;

So this will help test that the mechanism isn't confused by output columns that are created through a concatenation expression.

Here's some older example code that works with opcodes from Python, in this case to output indexes used by a query: https://github.com/plasticityai/supersqlite/blob/master/supersqlite/idxchk.py

I may need to do something special for rowid columns - there is a RowId opcode that might come into play here.

This may be obsoleted by #1293 - it looks like I may be able to auto-detect these foreign keys for arbitrary queries after all.

Extracting variables with this trick appears to work OK, but you have to pass the correct variables to the explain select... query. Using defaultdict seems to work there:

>>> rows = conn.execute('explain select * from repos where id = :id', defaultdict(int))
>>> [dict(r) for r in rows if r['opcode'] == 'Variable']
[{'addr': 2,
  'opcode': 'Variable',
  'p1': 1,
  'p2': 1,
  'p3': 0,
  'p4': ':id',
  'p5': 0,
  'comment': None}]

This almost works, but throws errors with some queries (anything with a rowid column for example) - it needs a bunch of test coverage.

def columns_for_query(conn, sql):
    rows = conn.execute('explain ' + sql).fetchall()
    table_rootpage_by_register = {r['p1']: r['p2'] for r in rows if r['opcode'] == 'OpenRead'}
    names_by_rootpage = dict(
        conn.execute(
            'select rootpage, name from sqlite_master where rootpage in ({})'.format(
                ', '.join(map(str, table_rootpage_by_register.values()))
            )
        )
    )
    columns_by_column_register = {}
    for row in rows:
        if row['opcode'] == 'Column':
            addr, opcode, table_id, cid, column_register, p4, p5, comment = row
            table = names_by_rootpage[table_rootpage_by_register[table_id]]
            columns_by_column_register[column_register] = (table, cid)
    result_row = [dict(r) for r in rows if r['opcode'] == 'ResultRow'][0]
    registers = list(range(result_row["p1"], result_row["p1"] + result_row["p2"] - 1))
    all_column_names = {}
    for table in names_by_rootpage.values():
        table_xinfo = conn.execute('pragma table_xinfo({})'.format(table)).fetchall()
        for row in table_xinfo:
            all_column_names[(table, row["cid"])] = row["name"]
    final_output = []
    for r in registers:
        try:
            table, cid = columns_by_column_register[r]
            final_output.append((table, all_column_names[table, cid]))
        except KeyError:
            final_output.append((None, None))
    return final_output

http://www.sqlite.org/draft/lang_explain.html says:

Applications should not use EXPLAIN or EXPLAIN QUERY PLAN since their exact behavior is variable and only partially documented.

I'm going to keep exploring this though.

... that output might also provide a better way to extract variables than the current mechanism using a regular expression, by looking for the Variable opcodes.

Essential documentation for understanding that output: https://www.sqlite.org/opcode.html

Had a fantastic suggestion on the SQLite forum: it might be possible to get what I want by interpreting the opcodes output by explain select ....

Copying the reply I posted to this thread:

That's really useful, thanks! It looks like it might be possible for me to reconstruct where each column came from using the explain select output.

Here's a complex example: https://calands.datasettes.com/calands?sql=explain+select%0D%0A++AsGeoJSON%28geometry%29%2C+*%0D%0Afrom%0D%0A++CPAD_2020a_SuperUnits%0D%0Awhere%0D%0A++PARK_NAME+like+%27%25mini%25%27+and%0D%0A++Intersects%28GeomFromGeoJSON%28%3Afreedraw%29%2C+geometry%29+%3D+1%0D%0A++and+CPAD_2020a_SuperUnits.rowid+in+%28%0D%0A++++select%0D%0A++++++rowid%0D%0A++++from%0D%0A++++++SpatialIndex%0D%0A++++where%0D%0A++++++f_table_name+%3D+%27CPAD_2020a_SuperUnits%27%0D%0A++++++and+search_frame+%3D+GeomFromGeoJSON%28%3Afreedraw%29%0D%0A++%29&freedraw=%7B%22type%22%3A%22MultiPolygon%22%2C%22coordinates%22%3A%5B%5B%5B%5B-122.42202758789064%2C37.82280243352759%5D%2C%5B-122.39868164062501%2C37.823887203271454%5D%2C%5B-122.38220214843751%2C37.81846319511331%5D%2C%5B-122.35061645507814%2C37.77071473849611%5D%2C%5B-122.34924316406251%2C37.74465712069939%5D%2C%5B-122.37258911132814%2C37.703380457832374%5D%2C%5B-122.39044189453125%2C37.690340943717715%5D%2C%5B-122.41241455078126%2C37.680559803205135%5D%2C%5B-122.44262695312501%2C37.67295135774715%5D%2C%5B-122.47283935546876%2C37.67295135774715%5D%2C%5B-122.52502441406251%2C37.68382032669382%5D%2C%5B-122.53463745117189%2C37.6892542140253%5D%2C%5B-122.54699707031251%2C37.690340943717715%5D%2C%5B-122.55798339843751%2C37.72945260537781%5D%2C%5B-122.54287719726564%2C37.77831314799672%5D%2C%5B-122.49893188476564%2C37.81303878836991%5D%2C%5B-122.46185302734376%2C37.82822612280363%5D%2C%5B-122.42889404296876%2C37.82822612280363%5D%2C%5B-122.42202758789064%2C37.82280243352759%5D%5D%5D%5D%7D

It looks like the opcodes I need to inspect are OpenRead, Column and ResultRow.

OpenRead tells me which tables are being opened - the p2 value (in this case 51) corresponds to the rootpage column in sqlite_master here: https://calands.datasettes.com/calands?sql=select+*+from+sqlite_master - it gets assigned to the register in p1.

The Column opcodes tell me which columns are being read - p1 is that table reference, and p2 is the cid of the column within that table.

The ResultRow opcode then tells me which columns are used in the results. 15 16 means start at the 15th and then read the next 16 columns.

I think this might work!

Sadly it doesn't do what I need. This query should only return one column, but instead I get back every column that was consulted by the query:

https://user-images.githubusercontent.com/9599/113524385-216ca000-9563-11eb-8a7e-cbbe5dfc02f6.png">

Oh wow, I just spotted https://github.com/macbre/sql-metadata

Uses tokenized query returned by python-sqlparse and generates query metadata. Extracts column names and tables used by the query. Provides a helper for normalization of SQL queries and tables aliases resolving.

It's for MySQL, PostgreSQL and Hive right now but maybe getting it working with SQLite wouldn't be too hard?

The other approach I considered for this was to have my own SQL query parser running in Python, which could pick apart a complex query and figure out which column was sourced from which table. I dropped this idea because it felt that the moment select * came into play a pure parsing approach wouldn't work - I'd need knowledge of the schema in order to resolve the *.

A Python parser approach might be good enough to handle a subset of queries - those that don't use select * for example - and maybe that would be worth shipping? The feature doesn't have to be perfect for it to be useful.

I asked about this on the SQLite forum: https://sqlite.org/forum/forumpost/0180277fb7

One option I've not fully explored yet: could I write my own custom SQLite C extension which exposes this functionality as a callable function?

Then I could load that extension and run a SQL query something like this:

select database, table, column from analyze_query(:sql_query)

Where analyze_query(...) would be a fancy virtual table function of some sort that uses the underlying sqlite3_column_database_name() C functions to analyze the SQL query and return details of what it would return.

Worth noting that adding limit 0 to the query still causes it to conduct the permission checks, hopefully while avoiding doing any of the actual work of executing the query:

In [20]: db.execute('select * from compound_primary_key join facetable on facetable.rowid = compound_primary_key.rowid limit 0').fetchall()
    ...: 
args (21, None, None, None, None) kwargs {}
args (20, 'compound_primary_key', 'pk1', 'main', None) kwargs {}
args (20, 'compound_primary_key', 'pk2', 'main', None) kwargs {}
args (20, 'compound_primary_key', 'content', 'main', None) kwargs {}
args (20, 'facetable', 'pk', 'main', None) kwargs {}
args (20, 'facetable', 'created', 'main', None) kwargs {}
args (20, 'facetable', 'planet_int', 'main', None) kwargs {}
args (20, 'facetable', 'on_earth', 'main', None) kwargs {}
args (20, 'facetable', 'state', 'main', None) kwargs {}
args (20, 'facetable', 'city_id', 'main', None) kwargs {}
args (20, 'facetable', 'neighborhood', 'main', None) kwargs {}
args (20, 'facetable', 'tags', 'main', None) kwargs {}
args (20, 'facetable', 'complex_array', 'main', None) kwargs {}
args (20, 'facetable', 'distinct_some_null', 'main', None) kwargs {}
args (20, 'facetable', 'pk', 'main', None) kwargs {}
args (20, 'compound_primary_key', 'ROWID', 'main', None) kwargs {}
Out[20]: []

Here are all of the available constants:

In [3]: for k in dir(sqlite3):
   ...:     if k.startswith("SQLITE_"):
   ...:         print(k, getattr(sqlite3, k))
   ...: 
SQLITE_ALTER_TABLE 26
SQLITE_ANALYZE 28
SQLITE_ATTACH 24
SQLITE_CREATE_INDEX 1
SQLITE_CREATE_TABLE 2
SQLITE_CREATE_TEMP_INDEX 3
SQLITE_CREATE_TEMP_TABLE 4
SQLITE_CREATE_TEMP_TRIGGER 5
SQLITE_CREATE_TEMP_VIEW 6
SQLITE_CREATE_TRIGGER 7
SQLITE_CREATE_VIEW 8
SQLITE_CREATE_VTABLE 29
SQLITE_DELETE 9
SQLITE_DENY 1
SQLITE_DETACH 25
SQLITE_DONE 101
SQLITE_DROP_INDEX 10
SQLITE_DROP_TABLE 11
SQLITE_DROP_TEMP_INDEX 12
SQLITE_DROP_TEMP_TABLE 13
SQLITE_DROP_TEMP_TRIGGER 14
SQLITE_DROP_TEMP_VIEW 15
SQLITE_DROP_TRIGGER 16
SQLITE_DROP_VIEW 17
SQLITE_DROP_VTABLE 30
SQLITE_FUNCTION 31
SQLITE_IGNORE 2
SQLITE_INSERT 18
SQLITE_OK 0
SQLITE_PRAGMA 19
SQLITE_READ 20
SQLITE_RECURSIVE 33
SQLITE_REINDEX 27
SQLITE_SAVEPOINT 32
SQLITE_SELECT 21
SQLITE_TRANSACTION 22
SQLITE_UPDATE 23

A more promising route I found involved the db.set_authorizer method. This can be used to log the permission checks that SQLite uses, including checks for permission to access specific columns of specific tables. For a while I thought this could work!

>>> def print_args(*args, **kwargs):
...    print("args", args, "kwargs", kwargs)
...    return sqlite3.SQLITE_OK

>>> db = sqlite3.connect("fixtures.db")
>>> db.execute('select * from compound_primary_key join facetable on rowid').fetchall()
args (21, None, None, None, None) kwargs {}
args (20, 'compound_primary_key', 'pk1', 'main', None) kwargs {}
args (20, 'compound_primary_key', 'pk2', 'main', None) kwargs {}
args (20, 'compound_primary_key', 'content', 'main', None) kwargs {}
args (20, 'facetable', 'pk', 'main', None) kwargs {}
args (20, 'facetable', 'created', 'main', None) kwargs {}
args (20, 'facetable', 'planet_int', 'main', None) kwargs {}
args (20, 'facetable', 'on_earth', 'main', None) kwargs {}
args (20, 'facetable', 'state', 'main', None) kwargs {}
args (20, 'facetable', 'city_id', 'main', None) kwargs {}
args (20, 'facetable', 'neighborhood', 'main', None) kwargs {}
args (20, 'facetable', 'tags', 'main', None) kwargs {}
args (20, 'facetable', 'complex_array', 'main', None) kwargs {}
args (20, 'facetable', 'distinct_some_null', 'main', None) kwargs {}

Those 20 values (where 20 is SQLITE_READ) looked like they were checking permissions for the columns in the order they would be returned!

Then I found a snag:

In [18]: db.execute('select 1 + 1 + (select max(rowid) from facetable)')
args (21, None, None, None, None) kwargs {}
args (31, None, 'max', None, None) kwargs {}
args (20, 'facetable', 'pk', 'main', None) kwargs {}
args (21, None, None, None, None) kwargs {}
args (20, 'facetable', '', None, None) kwargs {}

Once a subselect is involved the order of the 20 checks no longer matches the order in which the columns are returned from the query.

I've done various pieces of research into this over the past few years. Capturing what I've discovered in this ticket.

The SQLite C API has functions that can help with this: https://www.sqlite.org/c3ref/column_database_name.html details those. But they're not exposed in the Python SQLite library.

Maybe it would be possible to use them via ctypes? My hunch is that I would have to re-implement the full sqlite3 module with ctypes, which sounds daunting.

Could maybe replace this code: https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/utils/__init__.py#L1021-L1026

Useful tutorial series from 2012: https://northredoubt.com/n/2012/01/20/spatialite-speed-test/

Relevant code is some of the most complex in all of Datasette.

https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/views/table.py#L530-L594

And

https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/views/table.py#L743-L771

I'll need to think hard about how to refactor this out into something more understandable before implementing previous links.

Same query again with a, d, v returns 0 results, which is also as we would want: it signifies that we are back to the very first page: https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+pk3%2C+content+from+compound_three_primary_keys+where+%28%28pk1+%3C+%3Ap0%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3C+%3Ap1%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3D+%3Ap1+and+pk3+%3C+%3Ap2%29%29+order+by+pk1+desc%2C+pk2+desc%2C+pk3+desc+limit+1+offset+99&p0=a&p1=d&p2=v

I tried flipping the direction of the sort and the comparison operators and got this: https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+pk3%2C+content+from+compound_three_primary_keys+where+%28%28pk1+%3C+%3Ap0%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3C+%3Ap1%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3D+%3Ap1+and+pk3+%3C+%3Ap2%29%29+order+by+pk1+desc%2C+pk2+desc%2C+pk3+desc+limit+1+offset+99&p0=a&p1=h&p2=r

select pk1, pk2, pk3, content from compound_three_primary_keys where ((pk1 < :p0)
  or
(pk1 = :p0 and pk2 < :p1)
  or
(pk1 = :p0 and pk2 = :p1 and pk3 < :p2)) order by pk1 desc, pk2 desc, pk3 desc limit 1 offset 99

Which returned a-d-v as desired. I messed around with it to find the limit 1 offset 99 values.

Let's figure out the SQL for this. The most complex case is probably this one: https://latest.datasette.io/fixtures/compound_three_primary_keys?_next=a%2Ch%2Cr

Here's the SQL for that page: https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+pk3%2C+content+from+compound_three_primary_keys+where+%28%28pk1+%3E+%3Ap0%29%0A++or%0A%28pk1+%3D+%3Ap0+and+pk2+%3E+%3Ap1%29%0A++or%0A%28pk1+%3D+%3Ap0+and+pk2+%3D+%3Ap1+and+pk3+%3E+%3Ap2%29%29+order+by+pk1%2C+pk2%2C+pk3+limit+101&p0=a&p1=h&p2=r

select pk1, pk2, pk3, content from compound_three_primary_keys where ((pk1 > :p0)
  or
(pk1 = :p0 and pk2 > :p1)
  or
(pk1 = :p0 and pk2 = :p1 and pk3 > :p2)) order by pk1, pk2, pk3 limit 101

Where p0 is a, p1 is h and p2 is r.

Given the above, how would I figure out the correct previous link? It should be https://latest.datasette.io/fixtures/compound_three_primary_keys?_next=a%2Cd%2Cv - a, d, v.

I think my ideal implementation for this would be to reverse the order, grab the previous page-size-plus-one items, then return a ?_next=x token that would provide the previous page sorted back in the expected default order.

The alternative would be to have a ?_previous=x token which can be used to paginate backwards in reverse order, but I think this would be confusing as it would result in "hit next page, then hit previous page" returning you to a new state which features rows in the reverse order.

https://twitter.com/llanga/status/1378431719934681094 looks like I should wait for 3.9.4, out in a few days.

I found one example of an implementation of reversed keyset pagination here: https://github.com/tvainika/objection-keyset-pagination/blob/cb21a493c96daa6e63c302efae6718d09aa11661/index.js#L74-L79

https://github.com/simonw/datasette/actions/runs/713207828 ran with pytest-xdist in 4m22s:

https://user-images.githubusercontent.com/9599/113466592-f151bf80-93f1-11eb-9d74-b830e96c19c8.png">

Here's the test suite running on regular pytest in 5m13s:

https://user-images.githubusercontent.com/9599/113466606-00387200-93f2-11eb-95a3-24521c628cbe.png">

Not a huge speed-up because there are only 2 available cores in the GitHub Actions environment, but still worthwhile - especially since this lets people run in parallel on their own laptops.

Looks like -n auto only detected two cores on GitHub Actions: https://github.com/simonw/datasette/runs/2257597137?check_suite_focus=true

============================= test session starts ==============================
platform linux -- Python 3.7.10, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
SQLite: 3.31.1
rootdir: /home/runner/work/datasette/datasette, configfile: pytest.ini
plugins: xdist-2.2.1, timeout-1.4.2, forked-1.3.0, asyncio-0.14.0
gw0 I / gw1 I
gw0 [878] / gw1 [878]

On my Mac pytest-xdist ran the test suite (minus two tests) in 59s, as opposed to 2m23s without xdist.

python:3.9.3-slim-buster isn't on Docker Hub yet either.

Idea: default to displaying single-dimension JSON arrays of strings as a comma-separated list but show the comma in a different colour - something like this:

https://user-images.githubusercontent.com/9599/113445005-dce9d480-93a9-11eb-901c-0c2fb079c150.png">

I used this HTML for the prototype (re-using .type-int just to get the colour):

<td class="col-tags type-str">tag1<span class="type-int">, </span>tag2</td>

This makes senses - showing an array as ["blah", "blah2", "blah3"] isn't particularly human-friendly!

Once again having tests for the Dockerfile as seen in #1272 would be useful here.

https://devcenter.heroku.com/articles/python-support#supported-runtimes looks like Heroku still only have 3.9.2 for the moment.

A few places:

https://github.com/simonw/datasette/blob/7b1a9a1999eb9326ce8ec830d75ac200e5279c46/Dockerfile#L1

https://github.com/simonw/datasette/blob/8e18c7943181f228ce5ebcea48deb59ce50bee1f/datasette/utils/__init__.py#L350

https://github.com/simonw/datasette/blob/8e18c7943181f228ce5ebcea48deb59ce50bee1f/tests/test_package.py#L14

https://github.com/simonw/datasette/blob/8e18c7943181f228ce5ebcea48deb59ce50bee1f/datasette/publish/heroku.py#L177-L178

You're right, doing this is really hard at the moment - I'm not sure I know how I would tackle this either, and it's something I've wanted in the past!

I'll have a think about this one.

Thanks!

I tried this just now against Datasette 0.56 with the new Dockerfile from #1249 (that uses SQLite and SpatiaLite installed with apt-get install) and the tests all passed.

Sorry I didn't get to this PR sooner. I've joint-credited you in the release notes for this fix: https://docs.datasette.io/en/stable/changelog.html#v0-56

https://github.com/simonw/datasette/runs/2214871602?check_suite_focus=true worked:

https://user-images.githubusercontent.com/9599/112776676-270f3680-8ff5-11eb-90d1-533d4e82f677.png">

Here's the 0.56 image on Docker Hub: https://hub.docker.com/layers/datasetteproject/datasette/0.56/images/sha256-701fc0f299a0ea79434a4852c46dab351254b9ac25dbe3c5f36fd5360caf52f9?context=explore

Got this error:

"docker tag" requires exactly 2 arguments.
See 'docker tag --help'.

Usage:  docker tag SOURCE_IMAGE[:TAG] TARGET_IMAGE[:TAG]

Create a tag TARGET_IMAGE that refers to SOURCE_IMAGE

I'm going to build a new GitHub Actions workflow for this that lets me manually specify a tag to build and push as a Docker image.

I'm going to attempt to fix this manually for the 0.56 release, by building and tagging it by hand and then pushing the 0.56 tag to Docker Hub.

https://docs.docker.com/engine/reference/commandline/push/#push-all-tags-of-an-image

Use the -a (or --all-tags) option to push all tags of a local image.

https://github.com/simonw/datasette/runs/1763835467?check_suite_focus=true for Datasette 0.54 worked, and the output included this:

Successfully tagged ***/datasette:0.54
The push refers to repository [docker.io/***/datasette]
aedd33c6b161: Preparing
...
aedd33c6b161: Pushed
0.54: digest: sha256:65c7e579d1c29755dac5c1ca86b1e97fa88c48bd3d724ac3e02988d0da296140 size: 2005
aedd33c6b161: Preparing
...
5dacd731af1b: Layer already exists
latest: digest: sha256:65c7e579d1c29755dac5c1ca86b1e97fa88c48bd3d724ac3e02988d0da296140 size: 2005

Here's that same section of output from the 0.56 release:

Successfully tagged ***/datasette:0.56
Using default tag: latest
The push refers to repository [docker.io/***/datasette]
4d4a9976adcc: Preparing
...
9b2132a0d5cf: Pushed
latest: digest: sha256:2250d0fbe57b1d615a8d6df0c9d43deb9533532e00bac68854773d8ff8dcf00a size: 1793

The difference here is the "Using default tag: latest" bit.

Here's the diff between Dockerfile in 0.54.1 and 0.56: https://github.com/simonw/datasette/compare/0.54.1...0.56#diff-551d1fcf87f78cc3bc18a7b332a4dc5d8773a512062df881c5aba28a6f5c48d7

https://user-images.githubusercontent.com/9599/112774760-79e5ef80-8fef-11eb-886a-5d546d6d2f3d.png">

I just shipped Datasette 0.56 - here's the CI run: https://github.com/simonw/datasette/runs/2214701802?check_suite_focus=true

It pushed a new latest tag to https://hub.docker.com/r/datasetteproject/datasette/tags?page=1&ordering=last_updated

docker pull datasetteproject/datasette:latest

And then:

docker run datasetteproject/datasette:latest datasette \
  --load-extension=spatialite \
  --get /-/versions.json | jq .sqlite

Outputs:

{
  "version": "3.27.2",
  "fts_versions": [
    "FTS5",
    "FTS4",
    "FTS3"
  ],
  "extensions": {
    "json1": null,
    "spatialite": "5.0.1"
  },
  "compile_options": [
    "COMPILER=gcc-8.3.0",
    "ENABLE_COLUMN_METADATA",
    "ENABLE_DBSTAT_VTAB",
    "ENABLE_FTS3",
    "ENABLE_FTS3_PARENTHESIS",
    "ENABLE_FTS3_TOKENIZER",
    "ENABLE_FTS4",
    "ENABLE_FTS5",
    "ENABLE_JSON1",
    "ENABLE_LOAD_EXTENSION",
    "ENABLE_PREUPDATE_HOOK",
    "ENABLE_RTREE",
    "ENABLE_SESSION",
    "ENABLE_STMTVTAB",
    "ENABLE_UNLOCK_NOTIFY",
    "ENABLE_UPDATE_DELETE_LIMIT",
    "HAVE_ISNAN",
    "LIKE_DOESNT_MATCH_BLOBS",
    "MAX_SCHEMA_RETRY=25",
    "MAX_VARIABLE_NUMBER=250000",
    "OMIT_LOOKASIDE",
    "SECURE_DELETE",
    "SOUNDEX",
    "TEMP_STORE=1",
    "THREADSAFE=1",
    "USE_URI"
  ]
}

This bug should have been fixed in #1229 - let me know if that's not the case!

Thanks

This is interesting!

I've decided to apply a subset of these - the if and elif blocks are a deliberate style choice from me, because I find code clearer when it has if/else as opposed to relying on early termination. Likewise the iteration against .keys() on dictionaries.

I like the other fixes though, I'm about to land them in a separate commit that credits you.

Thanks for figuring this out!

Neat fix, thank you!

Thanks for the fix.

Might be easier to do this using https://github.com/coleifer/pysqlite3 rather than try to replace the system sqlite3 on the Ubuntu GitHub Actions instances.

These instructions should help: https://github.com/coleifer/pysqlite3#building-a-statically-linked-library

Testing this on Glitch by adding https://github.com/simonw/datasette/archive/48d5e0e6ac8975cfd869d4e8c69c64ca0c65e29e.zip as a dependency. That fixed it.

Aha! https://www.sqlite.org/pragma.html says:

The table-valued functions for PRAGMA feature was added in SQLite version 3.16.0 (2017-01-02). Prior versions of SQLite cannot use this feature.

Here's where I run that: https://github.com/simonw/datasette/blob/6f41c8a2bef309a66588b2875c3e24d26adb4850/datasette/database.py#L249-L253

That's from when I added the --crossdb option in #1232: https://github.com/simonw/datasette/commit/6f41c8a2bef309a66588b2875c3e24d26adb4850#diff-4e20309c969326a0008dc9237f6807f48d55783315fbfc1e7dfa480b550e16f9R249

The Glitch server logs showed:

ERROR: conn=<sqlite3.Connection object at 0x7f88ebadd8f0>, sql = 'select seq, name, file from pragma_database_list() where seq > 0', params = None: no such table: pragma_database_list

I just managed to replicate this bug on Glitch: https://nicar-2020.glitch.me/data

Invalid SQL
no such table: pragma_database_list

https://nicar-2020.glitch.me/-/versions says:

{
    "python": {
        "version": "3.7.10",
        "full": "3.7.10 (default, Feb 20 2021, 21:21:24) \n[GCC 5.4.0 20160609]"
    },
    "datasette": {
        "version": "0.55"
    },
    "asgi": "3.0",
    "uvicorn": "0.13.4",
    "sqlite": {
        "version": "3.11.0",
        "fts_versions": [
            "FTS4",
            "FTS3"
        ],
        "extensions": {
            "json1": null
        },
        "compile_options": [
            "ENABLE_COLUMN_METADATA",
            "ENABLE_DBSTAT_VTAB",
            "ENABLE_FTS3",
            "ENABLE_FTS3_PARENTHESIS",
            "ENABLE_JSON1",
            "ENABLE_LOAD_EXTENSION",
            "ENABLE_RTREE",
            "ENABLE_UNLOCK_NOTIFY",
            "ENABLE_UPDATE_DELETE_LIMIT",
            "HAVE_ISNAN",
            "LIKE_DOESNT_MATCH_BLOBS",
            "MAX_SCHEMA_RETRY=25",
            "OMIT_LOOKASIDE",
            "SECURE_DELETE",
            "SOUNDEX",
            "SYSTEM_MALLOC",
            "TEMP_STORE=1",
            "THREADSAFE=1"
        ]
    }
}

That's SQLite 3.11.0 from 2016-02-15 with no FTS5.

That rivers example in the tutorial would work a lot better with a live demo.

I have a better recipe for using spatial indexes now on https://simonwillison.net/2021/Jan/24/drawing-shapes-spatialite/

select
  AsGeoJSON(geometry), *
from
  CPAD_2020a_SuperUnits
where
  PARK_NAME like '%mini%' and
  Intersects(GeomFromGeoJSON(:freedraw), geometry) = 1
  and CPAD_2020a_SuperUnits.rowid in (
    select
      rowid
    from
      SpatialIndex
    where
      f_table_name = 'CPAD_2020a_SuperUnits'
      and search_frame = GeomFromGeoJSON(:freedraw)
  )

This will give you back an additional column of GeoJSON. You can copy and paste GeoJSON from this column into the debugging tool at geojson.io to visualize it on a map.

That should promote datasette-leaflet-geojson instead.

Related feature request: ability to set default values for canned queries: #1258

Idea for these: imitate https://django-sql-dashboard.readthedocs.io/en/latest/widgets.html#custom-widgets and drive them with templates.

So a custom widget type of textarea would look for a template called widgets/textarea.html - which means users could define brand new custom widgets just by creating their own template files.

https://timezones-api.datasette.io/ is now up and running on Cloud Run.

https://github.com/simonw/timezones-api is that project, it's pretty old now. I'll try to get it running on Cloud Run.

Right now they look like this:

databases:
  fixtures:
    queries:
      neighborhood_search:
        params:
        - text

In addition to being able to specify defaults, I'd also like to add other things in the future - most significantly the ability to specify a different input widget (e.g. textarea v.s. single-line input)

So maybe this looks like:

params:
- name: text
  default: ""
- name: age
  widget: number

I like that idea.

I'll close this issue after I ship Datasette 0.56 and confirm that the Dockerfile was correctly built and published to Docker Hub.

One last test of that Dockerfile:

(datasette) datasette % docker build -f Dockerfile -t datasetteproject/datasette:0.55a --build-arg VERSION=0.55 .
(datasette) datasette % docker run datasetteproject/datasette:0.55a datasette --get '/-/versions.json' | jq
{
  "python": {
    "version": "3.9.2",
    "full": "3.9.2 (default, Feb 19 2021, 17:23:45) \n[GCC 8.3.0]"
  },
  "datasette": {
    "version": "0.55"
  },
  "asgi": "3.0",
  "uvicorn": "0.13.4",
  "sqlite": {
    "version": "3.27.2",
    "fts_versions": [
      "FTS5",
      "FTS4",
      "FTS3"
    ],
    "extensions": {
      "json1": null
    },
    "compile_options": [
      "COMPILER=gcc-8.3.0",
      "ENABLE_COLUMN_METADATA",
      "ENABLE_DBSTAT_VTAB",
      "ENABLE_FTS3",
      "ENABLE_FTS3_PARENTHESIS",
      "ENABLE_FTS3_TOKENIZER",
      "ENABLE_FTS4",
      "ENABLE_FTS5",
      "ENABLE_JSON1",
      "ENABLE_LOAD_EXTENSION",
      "ENABLE_PREUPDATE_HOOK",
      "ENABLE_RTREE",
      "ENABLE_SESSION",
      "ENABLE_STMTVTAB",
      "ENABLE_UNLOCK_NOTIFY",
      "ENABLE_UPDATE_DELETE_LIMIT",
      "HAVE_ISNAN",
      "LIKE_DOESNT_MATCH_BLOBS",
      "MAX_SCHEMA_RETRY=25",
      "MAX_VARIABLE_NUMBER=250000",
      "OMIT_LOOKASIDE",
      "SECURE_DELETE",
      "SOUNDEX",
      "TEMP_STORE=1",
      "THREADSAFE=1",
      "USE_URI"
    ]
  }
}
(datasette) datasette % docker run datasetteproject/datasette:0.55a datasette --get '/-/versions.json' --load-extension=spatialite | jq
{
  "python": {
    "version": "3.9.2",
    "full": "3.9.2 (default, Feb 19 2021, 17:23:45) \n[GCC 8.3.0]"
  },
  "datasette": {
    "version": "0.55"
  },
  "asgi": "3.0",
  "uvicorn": "0.13.4",
  "sqlite": {
    "version": "3.27.2",
    "fts_versions": [
      "FTS5",
      "FTS4",
      "FTS3"
    ],
    "extensions": {
      "json1": null,
      "spatialite": "5.0.1"
    },
    "compile_options": [
      "COMPILER=gcc-8.3.0",
      "ENABLE_COLUMN_METADATA",
      "ENABLE_DBSTAT_VTAB",
      "ENABLE_FTS3",
      "ENABLE_FTS3_PARENTHESIS",
      "ENABLE_FTS3_TOKENIZER",
      "ENABLE_FTS4",
      "ENABLE_FTS5",
      "ENABLE_JSON1",
      "ENABLE_LOAD_EXTENSION",
      "ENABLE_PREUPDATE_HOOK",
      "ENABLE_RTREE",
      "ENABLE_SESSION",
      "ENABLE_STMTVTAB",
      "ENABLE_UNLOCK_NOTIFY",
      "ENABLE_UPDATE_DELETE_LIMIT",
      "HAVE_ISNAN",
      "LIKE_DOESNT_MATCH_BLOBS",
      "MAX_SCHEMA_RETRY=25",
      "MAX_VARIABLE_NUMBER=250000",
      "OMIT_LOOKASIDE",
      "SECURE_DELETE",
      "SOUNDEX",
      "TEMP_STORE=1",
      "THREADSAFE=1",
      "USE_URI"
    ]
  }
}

I'm skipping this for the moment because the new Dockerfile shape introduced in https://github.com/simonw/datasette/issues/1249#issuecomment-804404544 isn't compatible with this technique, since it installs Datasette from PyPI rather than directly from the repo.

Will need to change that if I want to do this unit tests thing.

Part of the challenge here is only running if a Docker daemon is available. I think this pattern works, in tests/test_dockerfile.py:

import httpx
import pathlib
import pytest
import subprocess

root = pathlib.Path(__file__).parent.parent

def docker_is_available():
    try:
        client = httpx.Client(
            transport=httpx.HTTPTransport(uds="/var/run/docker.sock")
        )
        client.get("http://docker/info")
        return True
    except httpx.ConnectError:
        return False


@pytest.fixture
def build_container():
    assert (root / "Dockerfile").exists()
    subprocess.check_call([
        "docker", "build", str(root), "-t", "datasette-dockerfile-test"
    ])


@pytest.mark.skipif(not docker_is_available(),
    reason="Docker is not available"
)
def test_dockerfile(build_container):
    output = subprocess.check_output([
        "docker", "run", "datasette-dockerfile-test", "datasette", "--get", "/_memory?sql=select+1&shape=_array"
    ])
    assert False, "Implement better assertion here"

That's really odd. What version of SQLite are you using on the server? You can tell by visiting https://your-site/-/versions

I fixed this already, it's a duplicate of #1239

Already got that! It's the --nl option - works for both importing and exporting data: https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-json-data

Idea: enhance lambda to allow it to return a dictionary of values, which will then be used to populate new columns. Use a --multicolumn option to indicate this:

sqlite-utils convert lambda mydb.db mytable mycolumn \
  --code '{"first_name": value.split()[0], "last_name": value.split()[1]}' \
  --multicolumn --drop

The --drop means "drop the mycolumn column after making this change".

Maybe --multi is a better name than --multicolumn here, since either way it's going to need additional explanation somewhere.

Would this overlap with #239 at all?

Here's the full current implementation of that tool: https://github.com/simonw/sqlite-transform/blob/0.5/sqlite_transform/cli.py

My current plan is to make this functionality available as the following:

sqlite-utils convert jsonsplit mydb.db mytable mycolumn
sqlite-utils convert parsedatetime mydb.db mytable mycolumn
sqlite-utils convert parsedate mydb.db mytable mycolumn
sqlite-utils convert lambda mydb.db mytable mycolumn --code='str(value).upper()'

datasette package is a mostly unmaintained feature at this point - it has a bit of test coverage but I've not made any improvements to it in a few years, and I don't use it for my own projects.

I'll make this change to package at the same time as I land it for publish though.

Yes this is one of the main reasons I'm planning to switch to encouraging YAML be default instead of JSON (while still supporting JSON) - YAML supports comments and multi-line strings.

See #1153 for YAML by default in the documentation.

Beginnings of a UI element for switching between them:

<div style="border: 1px solid rgb(225, 228, 229);
  background-color: rgb(238, 255, 204);
  padding: 0.3em;
  position: relative;
  top: 3px;
  font-family: courier;">
<a href="#" style="display: inline-block; padding-left: 0px; padding-right: 2em;">JSON</a>
<a href="#" style="display: inline-block;">YAML</a>
</div>

https://user-images.githubusercontent.com/9599/112194637-51f92500-8bc6-11eb-9662-3faa7ef37538.png">

That <pre> has a padding of 12px, so using 12px padding on the tab links should get them to line up better.

https://cdnjs.cloudflare.com/ajax/libs/js-yaml/4.0.0/js-yaml.min.js is only 12.5KB zipped, 38KB total - so that's not a bad option.

https://github.com/nodeca/js-yaml

I managed to build SpatiaLite such that this isn't necessary any more. I'm still interested in pursuing this further though - it feels like it could be a more robust way of implementing timeouts, but I need to prove to myself that it's better (maybe better performance, or handles more edge-cases?). Not sure how to prove that yet.

I used this code to get that:

var jsonVersion = JSON.stringify(window.jsyaml.load(document.querySelector('.highlight-yaml').textContent), null, 4);
div.querySelector('.highlight pre').innerText = jsonVersion;
div.querySelector('.highlight pre').style.whiteSpace = 'pre-wrap'

One downside of doing this conversion in JavaScript: it's much harder to get the same JSON syntax highlighting as that provided by Sphinx:

https://user-images.githubusercontent.com/9599/112183909-ef9b2700-8bbb-11eb-882b-aeb387c29e61.png">

https://docs.datasette.io/en/stable/metadata.html has this example:

title: Demonstrating Metadata from YAML
description_html: |-
  <p>This description includes a long HTML string</p>
  <ul>
    <li>YAML is better for embedding HTML strings than JSON!</li>
  </ul>
license: ODbL
license_url: https://opendatacommons.org/licenses/odbl/
databases:
  fixtures:
    tables:
      no_primary_key:
        hidden: true
    queries:
      neighborhood_search:
        sql: |-
          select neighborhood, facet_cities.name, state
          from facetable join facet_cities on facetable.city_id = facet_cities.id
          where neighborhood like '%' || :text || '%' order by neighborhood;
        title: Search neighborhoods
        description_html: |-
          <p>This demonstrates <em>basic</em> LIKE search

I ran this in the browser dev tools:

var s = document.createElement('script')
s.src = 'https://cdnjs.cloudflare.com/ajax/libs/js-yaml/4.0.0/js-yaml.min.js'
document.head.appendChild(s)
var yamlExample = document.querySelector('.highlight-yaml').textContent);
console.log(JSON.stringify(window.jsyaml.load(yamlExample), null, 4))

And got:

{
    "title": "Demonstrating Metadata from YAML",
    "description_html": "<p>This description includes a long HTML string</p>\n<ul>\n  <li>YAML is better for embedding HTML strings than JSON!</li>\n</ul>",
    "license": "ODbL",
    "license_url": "https://opendatacommons.org/licenses/odbl/",
    "databases": {
        "fixtures": {
            "tables": {
                "no_primary_key": {
                    "hidden": true
                }
            },
            "queries": {
                "neighborhood_search": {
                    "sql": "select neighborhood, facet_cities.name, state\nfrom facetable join facet_cities on facetable.city_id = facet_cities.id\nwhere neighborhood like '%' || :text || '%' order by neighborhood;",
                    "title": "Search neighborhoods",
                    "description_html": "<p>This demonstrates <em>basic</em> LIKE search"
                }
            }
        }
    }
}

... actually I think I would do that conversion in Python. The client-side YAML parsers all look a little bit heavy to me in terms of additional page weight.

That's a good idea. I could do that with JavaScript - loading YAML and converting it to JSON in JavaScript shouldn't be hard, and it's better than JSON-to-YAML because there's only one correct JSON representation of a YAML file whereas you can represent a JSON document in YAML in a bunch of different ways.

Don't forget to update this bit of the docs: https://docs.datasette.io/en/0.55/spatialite.html#building-spatialite-from-source

The packaged versions of SpatiaLite usually provide SpatiaLite 4.3.0a. For an example of how to build the most recent unstable version, 4.4.0-RC0 (which includes the powerful VirtualKNN module), take a look at the Datasette Dockerfile.

See also #1273

Comments welcome!

This isn't actually working - the outer code attempts to send an .interrupt() call to the connection object via the connections thread-local, which doesn't work because it's a thread-local so the connection isn't visible to that code.

Need to figure out how to communicate with that thread properly.

Also a test that fails in this particular case would be a good idea!

(Without the apt-get update ... SpatiaLite line it's 125MB)

Final version of Dockerfile which installs the specified version from GitHub:

docker build . -t datasette-spatialite --build-arg VERSION=0.55

FROM python:3.9.2-slim-buster as build

# Version of Datasette to install, e.g. 0.55
#   docker build . -t datasette --build-arg VERSION=0.55
ARG VERSION

# software-properties-common provides add-apt-repository
# which we need in order to install a more recent release
# of libsqlite3-mod-spatialite from the sid distribution
RUN apt-get update && \
    apt-get -y --no-install-recommends install software-properties-common && \
    add-apt-repository "deb http://httpredir.debian.org/debian sid main" && \
    apt-get update && \
    apt-get -t sid install -y --no-install-recommends libsqlite3-mod-spatialite && \
    apt-get remove -y software-properties-common && \
    apt clean && \
    rm -rf /var/lib/apt && \
    rm -rf /var/lib/dpkg

RUN pip install https://github.com/simonw/datasette/archive/refs/tags/${VERSION}.zip && \
    find /usr/local/lib -name '__pycache__' | xargs rm -r && \
    rm -rf /root/.cache/pip

EXPOSE 8001
CMD ["datasette"]

Run against 0.55 this produces an image of 262MB

I think part of the reason it's smaller is that I ran pip install datasette instead of using COPY . /datasette followed by pip install /datasette.

Considering the image on Docker Hub is 383MB, I'm happy with getting that down to 262MB. I'm going to stop looking for new optimizations here.