html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app https://github.com/simonw/sqlite-utils/issues/368#issuecomment-1008216371,https://api.github.com/repos/simonw/sqlite-utils/issues/368,1008216371,IC_kwDOCGYnMM48GCkz,9599,simonw,2022-01-09T02:36:22Z,2022-01-09T02:36:22Z,OWNER,"In Python 3.6: https://docs.python.org/3.6/library/subprocess.html > This does not capture stdout or stderr by default. To do so, pass [`PIPE`](https://docs.python.org/3.6/library/subprocess.html#subprocess.PIPE ""subprocess.PIPE"") for the *stdout* and/or *stderr* arguments.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097087280,Offer `python -m sqlite_utils` as an alternative to `sqlite-utils`, https://github.com/simonw/sqlite-utils/issues/368#issuecomment-1008216271,https://api.github.com/repos/simonw/sqlite-utils/issues/368,1008216271,IC_kwDOCGYnMM48GCjP,9599,simonw,2022-01-09T02:35:09Z,2022-01-09T02:35:09Z,OWNER,"Test failure on Python 3.6: > `E TypeError: __init__() got an unexpected keyword argument 'capture_output'`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097087280,Offer `python -m sqlite_utils` as an alternative to `sqlite-utils`, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008216201,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008216201,IC_kwDOCGYnMM48GCiJ,9599,simonw,2022-01-09T02:34:12Z,2022-01-09T02:34:12Z,OWNER,"I can now write tests that look like this: https://github.com/simonw/sqlite-utils/blob/539f5ccd90371fa87f946018f8b77d55929e06db/tests/test_cli.py#L2024-L2030 Which means I can write a test that exercises this bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/368#issuecomment-1008215912,https://api.github.com/repos/simonw/sqlite-utils/issues/368,1008215912,IC_kwDOCGYnMM48GCdo,9599,simonw,2022-01-09T02:30:59Z,2022-01-09T02:30:59Z,OWNER,"Even better, inspired by `rich`, support `python -m sqlite_utils`. https://github.com/Textualize/rich/blob/master/rich/__main__.py","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097087280,Offer `python -m sqlite_utils` as an alternative to `sqlite-utils`, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008214998,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008214998,IC_kwDOCGYnMM48GCPW,9599,simonw,2022-01-09T02:23:20Z,2022-01-09T02:23:20Z,OWNER,"Possible way of running the test: add this to `sqlite_utils/cli.py`: ```python if __name__ == ""__main__"": cli() ``` Now the tool can be run using `python -m sqlite_utils.cli --help` Then in the test use `subprocess` to call `sys.executable` (the path to the current Python interpreter) and pass it `-m sqlite_utils.cli` to run the script!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008214406,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008214406,IC_kwDOCGYnMM48GCGG,9599,simonw,2022-01-09T02:18:21Z,2022-01-09T02:18:21Z,OWNER,"I'm having trouble figuring out the best way to write a unit test for this. Filed a relevant feature request for Click here: - https://github.com/pallets/click/issues/2171","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008163050,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1008163050,IC_kwDOCGYnMM48F1jq,9599,simonw,2022-01-08T22:10:51Z,2022-01-08T22:10:51Z,OWNER,"Is there a downside to having a `sqlite_stat1` table if it has wildly incorrect statistics in it? Imagine the following sequence of events: - User imports a few records, creating the table, using `sqlite-utils insert` - User runs `sqlite-utils create-index ...` which also creates and populates the `sqlite_stat1` table - User runs `insert` again to populate several million new records The user now has a database file with several million records and a statistics table that is wildly out of date, having been populated when they only had a few. Will this result in surprisingly bad query performance compared to it that statistics table did not exist at all? If so, I lean much harder towards `ANALYZE` as a strictly opt-in optimization, maybe with the `--analyze` option added to `sqlite-utils insert` top to help users opt in to updating their statistics after running big inserts.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index, https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1008158616,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1008158616,IC_kwDOCGYnMM48F0eY,9599,simonw,2022-01-08T21:35:32Z,2022-01-08T21:35:32Z,OWNER,"Built a prototype in a branch, see #367.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE, https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008158357,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1008158357,IC_kwDOCGYnMM48F0aV,9599,simonw,2022-01-08T21:33:07Z,2022-01-08T21:33:07Z,OWNER,"The one thing that worries me a little bit about doing this by default is that it adds a surprising new table to the database - it may be confusing to users if they run `create-index` and their database suddenly has a new `sqlite_stat1` table, see https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1008157132 Options here are: - Do it anyway. People can tolerate a surprise table appearing when they create an index. - Only run `ANALYZE` if the user says `sqlite-utils create-index ... --analyze` - Use the `--analyze` option, but also automatically run `ANALYZE` if they create an index and the database they are working with already has a `sqlite_stat1` table I'm currently leading towards that third option - @fgregg any thoughts?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index, https://github.com/simonw/datasette/issues/1587#issuecomment-1008157998,https://api.github.com/repos/simonw/datasette/issues/1587,1008157998,IC_kwDOBm6k_c48F0Uu,9599,simonw,2022-01-08T21:29:54Z,2022-01-08T21:29:54Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/00a2895cd2dc42c63846216b36b2dc9f41170129/datasette/database.py#L339-L354,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097040427,Add `sqlite_stat1`(-4) tables to hidden table list, https://github.com/simonw/datasette/issues/1587#issuecomment-1008157908,https://api.github.com/repos/simonw/datasette/issues/1587,1008157908,IC_kwDOBm6k_c48F0TU,9599,simonw,2022-01-08T21:29:06Z,2022-01-08T21:29:06Z,OWNER,"Depending on the SQLite version (and compile options) that ran `ANALYZE` these can be called: - `sqlite_stat1` - `sqlite_stat2` - `sqlite_stat3` - `sqlite_stat4`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097040427,Add `sqlite_stat1`(-4) tables to hidden table list, https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1008157132,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1008157132,IC_kwDOCGYnMM48F0HM,9599,simonw,2022-01-08T21:23:08Z,2022-01-08T21:25:05Z,OWNER,"Running `ANALYZE` creates a new visible table called `sqlite_stat1`: https://www.sqlite.org/fileformat.html#the_sqlite_stat1_table This should be added to the default list of hidden tables in Datasette. It looks something like this: | tbl | idx | stat | |---------------------------------|------------------------------------|-----------| | _counts | sqlite_autoindex__counts_1 | 5 1 | | global-power-plants_fts_config | global-power-plants_fts_config | 1 1 | | global-power-plants_fts_docsize | | 33643 | | global-power-plants_fts_idx | global-power-plants_fts_idx | 199 40 1 | | global-power-plants_fts_data | | 136 | | global-power-plants | ""global-power-plants_owner"" | 33643 4 | | global-power-plants | ""global-power-plants_country_long"" | 33643 202 | > In each such row, the sqlite_stat.stat column will be a string consisting of a list of integers followed by zero or more arguments. The first integer in this list is the approximate number of rows in the index. (The number of rows in the index is the same as the number of rows in the table, except for partial indexes.) The second integer is the approximate number of rows in the index that have the same value in the first column of the index. The third integer is the number number of rows in the index that have the same value for the first two columns. The N-th integer (for N>1) is the estimated average number of rows in the index which have the same value for the first N-1 columns. For a K-column index, there will be K+1 integers in the stat column. If the index is unique, then the last integer will be 1. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008155916,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008155916,IC_kwDOCGYnMM48Fz0M,9599,simonw,2022-01-08T21:16:46Z,2022-01-08T21:16:46Z,OWNER,"No, `chunks()` seems to work OK in the test I just added.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008154873,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008154873,IC_kwDOCGYnMM48Fzj5,9599,simonw,2022-01-08T21:11:55Z,2022-01-08T21:11:55Z,OWNER,"I'm suspicious that the `chunks()` utility function may not be working correctly: ```pycon In [10]: [list(d) for d in list(chunks('abc', 5))] Out[10]: [['a'], ['b'], ['c']] In [11]: [list(d) for d in list(chunks('abcdefghi', 5))] Out[11]: [['a'], ['b'], ['c'], ['d'], ['e'], ['f'], ['g'], ['h'], ['i']] In [12]: [list(d) for d in list(chunks('abcdefghi', 3))] Out[12]: [['a'], ['b'], ['c'], ['d'], ['e'], ['f'], ['g'], ['h'], ['i']] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008153586,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008153586,IC_kwDOCGYnMM48FzPy,9599,simonw,2022-01-08T21:06:15Z,2022-01-08T21:06:15Z,OWNER,"I added a print statement after `for query, params in queries_and_params` and confirmed that something in the code is waiting until 16 records are available to be inserted and then executing the inserts, even with `--batch-size 1`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008151884,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008151884,IC_kwDOCGYnMM48Fy1M,9599,simonw,2022-01-08T20:59:21Z,2022-01-08T20:59:21Z,OWNER,"(That Heroku example doesn't record the timestamp, which limits its usefulness)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008143248,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008143248,IC_kwDOCGYnMM48FwuQ,9599,simonw,2022-01-08T20:34:12Z,2022-01-08T20:34:12Z,OWNER,Built that tool: https://github.com/simonw/stream-delay and https://pypi.org/project/stream-delay/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008129841,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008129841,IC_kwDOCGYnMM48Ftcx,9599,simonw,2022-01-08T20:04:42Z,2022-01-08T20:04:42Z,OWNER,"It would be easier to test this if I had a utility for streaming out a file one line at a time. A few recipes for this in https://superuser.com/questions/526242/cat-file-to-terminal-at-particular-speed-of-lines-per-second - I'm going to build a quick `stream-delay` tool though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1007643254,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1007643254,IC_kwDOCGYnMM48D2p2,9599,simonw,2022-01-07T18:37:56Z,2022-01-07T18:37:56Z,OWNER,Or I could leave off `--no-analyze` and tell people that if they want to add an index without running analyze they can execute the `CREATE INDEX` themselves.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index, https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1007642831,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1007642831,IC_kwDOCGYnMM48D2jP,9599,simonw,2022-01-07T18:37:18Z,2022-01-07T18:37:18Z,OWNER,"After implementing #366 I can make it so `sqlite-utils create-index` automatically runs `db.analyze(index_name)` afterwards, maybe with a `--no-analyze` option in case anyone wants to opt out of that for specific performance reasons.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index, https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1007641634,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1007641634,IC_kwDOCGYnMM48D2Qi,9599,simonw,2022-01-07T18:35:35Z,2022-01-07T18:35:35Z,OWNER,"Since the existing CLI feature is this: $ sqlite-utils analyze-tables github.db tags I can add `sqlite-utils analyze` to reflect the Python library method.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE, https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1007639860,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1007639860,IC_kwDOCGYnMM48D100,9599,simonw,2022-01-07T18:32:59Z,2022-01-07T18:33:07Z,OWNER,"From the SQLite docs: > If no arguments are given, all attached databases are analyzed. If a schema name is given as the argument, then all tables and indices in that one database are analyzed. If the argument is a table name, then only that table and the indices associated with that table are analyzed. If the argument is an index name, then only that one index is analyzed. So I think this becomes two methods: - `db.analyze()` calls analyze on the whole database - `db.analyze(name_of_table_or_index)` for a specific named table or index - `table.analyze()` is a shortcut for `db.analyze(table.name)`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE, https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1007637963,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1007637963,IC_kwDOCGYnMM48D1XL,9599,simonw,2022-01-07T18:30:13Z,2022-01-07T18:30:13Z,OWNER,"Annoyingly I use the word ""analyze"" to mean something else in the CLI - for these features: - #207 - #320 there's only one method with a similar name in the Python library though and that's this one: https://github.com/simonw/sqlite-utils/blob/6e46b9913411682f3a3ec66f4d58886c1db8654b/sqlite_utils/db.py#L2904-L2906","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE, https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1007634999,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1007634999,IC_kwDOCGYnMM48D0o3,9599,simonw,2022-01-07T18:26:22Z,2022-01-07T18:26:22Z,OWNER,"I've not used the `ANALYZE` feature in SQLite at all before. Should probably add Python library methods for it. Annoyingly I use the word ""analyze"" to mean something else in the CLI - for these features: - #207 - #320","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index, https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1007633376,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1007633376,IC_kwDOCGYnMM48D0Pg,9599,simonw,2022-01-07T18:24:07Z,2022-01-07T18:24:07Z,OWNER,Relevant documentation: https://www.sqlite.org/lang_analyze.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index, https://github.com/simonw/sqlite-utils/issues/363#issuecomment-1006344080,https://api.github.com/repos/simonw/sqlite-utils/issues/363,1006344080,IC_kwDOCGYnMM47-5eQ,9599,simonw,2022-01-06T07:32:05Z,2022-01-06T07:32:05Z,OWNER,As part of this work I should add test coverage of this error message too: https://github.com/simonw/sqlite-utils/blob/413f8ed754e38d7b190de888c85fe8438336cb11/sqlite_utils/cli.py#L826,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094981339,Better error message if `--convert` code fails to return a dict, https://github.com/simonw/sqlite-utils/issues/363#issuecomment-1006343303,https://api.github.com/repos/simonw/sqlite-utils/issues/363,1006343303,IC_kwDOCGYnMM47-5SH,9599,simonw,2022-01-06T07:30:20Z,2022-01-06T07:30:20Z,OWNER,This check should run inside the `.insert_all()` method. It should raise a custom exception which the CLI code can then catch and turn into a click error.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094981339,Better error message if `--convert` code fails to return a dict, https://github.com/simonw/sqlite-utils/issues/356#issuecomment-1006318443,https://api.github.com/repos/simonw/sqlite-utils/issues/356,1006318443,IC_kwDOCGYnMM47-zNr,9599,simonw,2022-01-06T06:30:13Z,2022-01-06T06:30:13Z,OWNER,"Documentation: - https://sqlite-utils.datasette.io/en/latest/cli.html#inserting-unstructured-data-with-lines-and-text - https://sqlite-utils.datasette.io/en/latest/cli.html#applying-conversions-while-inserting-data","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option, https://github.com/simonw/sqlite-utils/issues/356#issuecomment-1006318007,https://api.github.com/repos/simonw/sqlite-utils/issues/356,1006318007,IC_kwDOCGYnMM47-zG3,9599,simonw,2022-01-06T06:28:53Z,2022-01-06T06:28:53Z,OWNER,Implemented in #361.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006315145,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006315145,IC_kwDOCGYnMM47-yaJ,9599,simonw,2022-01-06T06:20:51Z,2022-01-06T06:20:51Z,OWNER,This is all documented. I'm going to rebase-merge it to keep the individual commits.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006311742,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006311742,IC_kwDOCGYnMM47-xk-,9599,simonw,2022-01-06T06:12:19Z,2022-01-06T06:12:19Z,OWNER,"Got that working: ``` % echo 'This is cool' | sqlite-utils insert words.db words - --text --convert '({""word"": w} for w in text.split())' % sqlite-utils dump words.db BEGIN TRANSACTION; CREATE TABLE [words] ( [word] TEXT ); INSERT INTO ""words"" VALUES('This'); INSERT INTO ""words"" VALUES('is'); INSERT INTO ""words"" VALUES('cool'); COMMIT; ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006309834,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006309834,IC_kwDOCGYnMM47-xHK,9599,simonw,2022-01-06T06:08:01Z,2022-01-06T06:08:01Z,OWNER,"For `--text` the conversion function should be allowed to return an iterable instead of a dictionary, in which case it will be treated as the full list of records to be inserted.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006301546,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006301546,IC_kwDOCGYnMM47-vFq,9599,simonw,2022-01-06T05:44:47Z,2022-01-06T05:44:47Z,OWNER,Just need documentation for `--convert` now against the various different types of input.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006300280,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006300280,IC_kwDOCGYnMM47-ux4,9599,simonw,2022-01-06T05:40:45Z,2022-01-06T05:40:45Z,OWNER,"I'm going to rename `--all` to `--text`: > - Use `--text` to write the entire input to a column called ""text"" To avoid that clash with Python's `all()` function.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006299778,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006299778,IC_kwDOCGYnMM47-uqC,9599,simonw,2022-01-06T05:39:10Z,2022-01-06T05:39:10Z,OWNER,`all` is a bad variable name because it clashes with the Python `all()` built-in function.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006295276,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006295276,IC_kwDOCGYnMM47-tjs,9599,simonw,2022-01-06T05:26:11Z,2022-01-06T05:26:11Z,OWNER,"Here's the traceback if your `--convert` function doesn't return a dict right now: ``` % sqlite-utils insert /tmp/all.db blah /tmp/log.log --convert 'all.upper()' --all Traceback (most recent call last): File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/bin/sqlite-utils"", line 33, in sys.exit(load_entry_point('sqlite-utils', 'console_scripts', 'sqlite-utils')()) File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 1137, in __call__ return self.main(*args, **kwargs) File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 1062, in main rv = self.invoke(ctx) File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 1668, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 763, in invoke return __callback(*args, **kwargs) File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py"", line 949, in insert insert_upsert_implementation( File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py"", line 834, in insert_upsert_implementation db[table].insert_all( File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 2602, in insert_all first_record = next(records) File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 3044, in fix_square_braces for record in records: File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py"", line 831, in docs = (decode_base64_values(doc) for doc in docs) File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py"", line 86, in decode_base64_values to_fix = [ File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py"", line 89, in if isinstance(doc[k], dict) TypeError: string indices must be integers ``` I can live with that for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006294777,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006294777,IC_kwDOCGYnMM47-tb5,9599,simonw,2022-01-06T05:24:54Z,2022-01-06T05:24:54Z,OWNER,"> I added a custom error message for if the user's `--convert` code doesn't return a dict. That turned out to be a bad idea because it meant exhausting the iterator early for the check - before we got to the `.insert_all()` code that breaks the iterator up into chunks. I tried fixing that with `itertools.tee()` to run the generator twice but that's grossly memory-inefficient for large imports.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006288444,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006288444,IC_kwDOCGYnMM47-r48,9599,simonw,2022-01-06T05:07:10Z,2022-01-06T05:07:10Z,OWNER,"And here's a demo of `--convert` used with `--all` - I added a custom error message for if the user's `--convert` code doesn't return a dict. ``` % sqlite-utils insert /tmp/all.db blah /tmp/log.log --convert 'all.upper()' --all Error: Records returned by your --convert function must be dicts % sqlite-utils insert /tmp/all.db blah /tmp/log.log --convert '{""all"": all.upper()}' --all % sqlite-utils dump /tmp/all.db BEGIN TRANSACTION; CREATE TABLE [blah] ( [all] TEXT ); INSERT INTO ""blah"" VALUES('INFO: 127.0.0.1:60581 - ""GET / HTTP/1.1"" 200 OK INFO: 127.0.0.1:60581 - ""GET /FOO/-/STATIC/APP.CSS?CEAD5A HTTP/1.1"" 200 OK INFO: 127.0.0.1:60581 - ""GET /FAVICON.ICO HTTP/1.1"" 200 OK INFO: 127.0.0.1:60581 - ""GET /FOO/TIDDLYWIKI HTTP/1.1"" 200 OK INFO: 127.0.0.1:60581 - ""GET /FOO/-/STATIC/APP.CSS?CEAD5A HTTP/1.1"" 200 OK INFO: 127.0.0.1:60584 - ""GET /FOO/-/STATIC/SQL-FORMATTER-2.3.3.MIN.JS HTTP/1.1"" 200 OK INFO: 127.0.0.1:60586 - ""GET /FOO/-/STATIC/CODEMIRROR-5.57.0.MIN.JS HTTP/1.1"" 200 OK INFO: 127.0.0.1:60585 - ""GET /FOO/-/STATIC/CODEMIRROR-5.57.0.MIN.CSS HTTP/1.1"" 200 OK INFO: 127.0.0.1:60588 - ""GET /FOO/-/STATIC/CODEMIRROR-5.57.0-SQL.MIN.JS HTTP/1.1"" 200 OK INFO: 127.0.0.1:60587 - ""GET /FOO/-/STATIC/CM-RESIZE-1.0.1.MIN.JS HTTP/1.1"" 200 OK INFO: 127.0.0.1:60586 - ""GET /FOO/TIDDLYWIKI/TIDDLERS HTTP/1.1"" 200 OK INFO: 127.0.0.1:60586 - ""GET /FOO/-/STATIC/APP.CSS?CEAD5A HTTP/1.1"" 200 OK INFO: 127.0.0.1:60584 - ""GET /FOO/-/STATIC/TABLE.JS HTTP/1.1"" 200 OK '); COMMIT; ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006284673,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006284673,IC_kwDOCGYnMM47-q-B,9599,simonw,2022-01-06T04:55:52Z,2022-01-06T04:55:52Z,OWNER,"Test code that just worked for me: ``` sqlite-utils insert /tmp/blah.db blah /tmp/log.log --convert ' bits = line.split() return dict([(""b_{}"".format(i), bit) for i, bit in enumerate(bits)])' --lines ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006232013,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006232013,IC_kwDOCGYnMM47-eHN,9599,simonw,2022-01-06T02:21:35Z,2022-01-06T02:21:35Z,OWNER,"I'm having second thoughts about this bit: > Your Python code will be passed a ""row"" variable representing the imported row, and can return a modified row. > > If you are using `--lines` your code will be passed a ""line"" variable, and for `--all` an ""all"" variable. The code in question is this: https://github.com/simonw/sqlite-utils/blob/500a35ad4d91c8a6232134ce9406efec11bedff8/sqlite_utils/utils.py#L296-L303 Do I really want to add the complexity of supporting different variable names there? I think always using `value` might be better. Except... `value` made sense for the existing `sqlite-utils convert` command where you are running a conversion function against the value for the column in the current row - is it confusing if applied to lines or documents or `all`?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006230411,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006230411,IC_kwDOCGYnMM47-duL,9599,simonw,2022-01-06T02:17:35Z,2022-01-06T02:17:35Z,OWNER,"Documentation: https://github.com/simonw/sqlite-utils/blob/33223856ff7fe746b7b77750fbe5b218531d0545/docs/cli.rst#inserting-unstructured-data-with---lines-and---all - I went with a single section titled ""Inserting unstructured data with --lines and --all""","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006220129,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006220129,IC_kwDOCGYnMM47-bNh,9599,simonw,2022-01-06T01:52:26Z,2022-01-06T01:52:26Z,OWNER,I'm going to refactor all of the tests for `sqlite-utils insert` into a new `test_cli_insert.py` module.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006219848,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006219848,IC_kwDOCGYnMM47-bJI,9599,simonw,2022-01-06T01:51:36Z,2022-01-06T01:51:36Z,OWNER,"So far I've just implemented the new help: ``` % sqlite-utils insert --help Usage: sqlite-utils insert [OPTIONS] PATH TABLE FILE Insert records from FILE into a table, creating the table if it does not already exist. By default the input is expected to be a JSON array of objects. Or: - Use --nl for newline-delimited JSON objects - Use --csv or --tsv for comma-separated or tab-separated input - Use --lines to write each incoming line to a column called ""line"" - Use --all to write the entire input to a column called ""all"" You can also use --convert to pass a fragment of Python code that will be used to convert each input. Your Python code will be passed a ""row"" variable representing the imported row, and can return a modified row. If you are using --lines your code will be passed a ""line"" variable, and for --all an ""all"" variable. Options: --pk TEXT Columns to use as the primary key, e.g. id --flatten Flatten nested JSON objects, so {""a"": {""b"": 1}} becomes {""a_b"": 1} --nl Expect newline-delimited JSON -c, --csv Expect CSV input --tsv Expect TSV input --lines Treat each line as a single value called 'line' --all Treat input as a single value called 'all' --convert TEXT Python code to convert each item --import TEXT Python modules to import --delimiter TEXT Delimiter to use for CSV files --quotechar TEXT Quote character to use for CSV/TSV --sniff Detect delimiter and quote character --no-headers CSV file has no header row --batch-size INTEGER Commit every X records --alter Alter existing table to add any missing columns --not-null TEXT Columns that should be created as NOT NULL --default ... Default value that should be set for a column --encoding TEXT Character encoding for input, defaults to utf-8 -d, --detect-types Detect types for columns in CSV/TSV data --load-extension TEXT SQLite extensions to load --silent Do not show progress bar --ignore Ignore records if pk already exists --replace Replace records if pk already exists --truncate Truncate table before inserting records, if table already exists -h, --help Show this message and exit. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997496626,https://api.github.com/repos/simonw/sqlite-utils/issues/356,997496626,IC_kwDOCGYnMM47dJcy,9599,simonw,2021-12-20T00:38:15Z,2022-01-06T01:29:03Z,OWNER,"The implementation of this gets a tiny bit complicated. Ignoring `--convert`, the `--lines` option can internally produce `{""line"": ...}` records and the `--all` option can produce `{""all"": ...}` records. But... when `--convert` is used, what should the code run against? It could run against those already-converted records but that's a little bit strange, since you'd have to do this: sqlite-utils insert blah.db blah myfile.txt --all --convert '{""item"": s for s in value[""all""].split(""-"")}' Having to use `value[""all""]` there is unintuitive. It would be nicer to have a `all` variable to work against. But then for `--lines` should the local variable be called `line`? And how best to summarize these different names for local variables in the inline help for the feature?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option, https://github.com/simonw/sqlite-utils/issues/360#issuecomment-1006211113,https://api.github.com/repos/simonw/sqlite-utils/issues/360,1006211113,IC_kwDOCGYnMM47-ZAp,9599,simonw,2022-01-06T01:27:53Z,2022-01-06T01:27:53Z,OWNER,"It looks like you were using `sqlite-utils memory` - that works by loading the entire file into an in-memory database, so 170GB is very likely to run out of RAM. The line of code there exhibits another problem: it's reading the entire JSON file into a Python string, so it looks like it's going to run out of RAM even before it gets to the SQLite in-memory database section. To handle a file of this size you'd need to write it to a SQLite database on-disk first. The `sqlite-utils insert` command can do this, and it should be able to ""stream"" records in from a file without loading the entire thing into memory - but only for JSON-NL and CSV/TSV formats, not for JSON arrays. The code in question is here: https://github.com/simonw/sqlite-utils/blob/f3fd8613113d21d44238a6ec54b375f5aa72c4e0/sqlite_utils/cli.py#L738-L773 That's using Python generators for the CSV/TSV/JSON-NL variants... but it's doing this for regular JSON which requires reading the entire thing into memory: https://github.com/simonw/sqlite-utils/blob/f3fd8613113d21d44238a6ec54b375f5aa72c4e0/sqlite_utils/cli.py#L767 If you have the ability to control how your 170GB file is generated you may have more luck converting it to CSV or TSV or newline-delimited JSON, then using `sqlite-utils insert` to insert it into a database file. To be honest though I've never tested this tooling with anything nearly that big, so it's possible you'll still run into problems. If you do I'd love to hear about them! I would be tempted to tackle this size of job by writing a custom Python script, either using the `sqlite_utils` Python library or even calling `sqlite3` directly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1091819089,MemoryError, https://github.com/simonw/datasette/issues/1534#issuecomment-1005975080,https://api.github.com/repos/simonw/datasette/issues/1534,1005975080,IC_kwDOBm6k_c479fYo,9599,simonw,2022-01-05T18:29:06Z,2022-01-05T18:29:06Z,OWNER,"A really big downside to this is that it turns out many CDNs - apparently including Cloudflare - don't support the Vary header at all! More in this thread: https://twitter.com/simonw/status/1478470282931163137","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065432388,Maybe return JSON from HTML pages if `Accept: application/json` is sent, https://github.com/simonw/datasette/issues/1585#issuecomment-1003575286,https://api.github.com/repos/simonw/datasette/issues/1585,1003575286,IC_kwDOBm6k_c470Vf2,9599,simonw,2022-01-01T15:40:38Z,2022-01-01T15:40:38Z,OWNER,API tutorial: https://firebase.google.com/docs/hosting/api-deploy,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1091838742,Fire base caching for `publish cloudrun`, https://github.com/simonw/datasette/issues/1152#issuecomment-1001791592,https://api.github.com/repos/simonw/datasette/issues/1152,1001791592,IC_kwDOBm6k_c47tiBo,9599,simonw,2021-12-27T23:04:31Z,2021-12-27T23:04:31Z,OWNER,Another option: rethink permissions to always work in terms of where clauses users as part of a SQL query that returns the overall allowed set of databases or tables. This would require rethinking existing permissions but it might be worthwhile prior to 1.0.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770598024,Efficiently calculate list of databases/tables a user can view, https://github.com/simonw/datasette/issues/878#issuecomment-1001699559,https://api.github.com/repos/simonw/datasette/issues/878,1001699559,IC_kwDOBm6k_c47tLjn,9599,simonw,2021-12-27T18:53:04Z,2021-12-27T18:53:04Z,OWNER,"I'm going to see if I can come up with the simplest possible version of this pattern for the `/-/metadata` and `/-/metadata.json` page, then try it for the database query page, before tackling the much more complex table page.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/1576#issuecomment-1000935523,https://api.github.com/repos/simonw/datasette/issues/1576,1000935523,IC_kwDOBm6k_c47qRBj,9599,simonw,2021-12-24T21:33:05Z,2021-12-24T21:33:05Z,OWNER,"Another option would be to attempt to import `contextvars` and, if the import fails (for Python 3.6) continue using the current mechanism - then let Python 3.6 users know in the documentation that under Python 3.6 they will miss out on nested traces.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/1577#issuecomment-1000673444,https://api.github.com/repos/simonw/datasette/issues/1577,1000673444,IC_kwDOBm6k_c47pRCk,9599,simonw,2021-12-24T06:08:58Z,2021-12-24T06:08:58Z,OWNER,"https://pypistats.org/packages/datasette shows a breakdown of downloads by Python version: It looks like on a recent day I had 4,071 downloads from Python 3.7... and just 2 downloads from Python 3.6!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087913724,Drop support for Python 3.6, https://github.com/simonw/datasette/issues/1534#issuecomment-1000535904,https://api.github.com/repos/simonw/datasette/issues/1534,1000535904,IC_kwDOBm6k_c47ovdg,9599,simonw,2021-12-23T21:44:31Z,2021-12-23T21:44:31Z,OWNER,A big downside to this is that I would need to use `Vary: Accept` for when Datasette is running behind a cache such as Cloudflare - would that greatly reduce overall cache efficiency due to subtle variations in the accept headers sent by common browsers?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065432388,Maybe return JSON from HTML pages if `Accept: application/json` is sent, https://github.com/simonw/datasette/issues/1579#issuecomment-1000485719,https://api.github.com/repos/simonw/datasette/issues/1579,1000485719,IC_kwDOBm6k_c47ojNX,9599,simonw,2021-12-23T19:19:45Z,2021-12-23T19:19:45Z,OWNER,All of those removed `block=True` lines in 8c401ee0f054de2f568c3a8302c9223555146407 really help confirm to me that this was a good decision.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087931918,`.execute_write(... block=True)` should be the default behaviour, https://github.com/simonw/datasette/issues/1579#issuecomment-1000485505,https://api.github.com/repos/simonw/datasette/issues/1579,1000485505,IC_kwDOBm6k_c47ojKB,9599,simonw,2021-12-23T19:19:13Z,2021-12-23T19:19:13Z,OWNER,Updated docs for `execute_write_fn()`: https://github.com/simonw/datasette/blob/75153ea9b94d09ec3d61f7c6ebdf378e0c0c7a0b/docs/internals.rst#await-dbexecute_write_fnfn-blocktrue,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087931918,`.execute_write(... block=True)` should be the default behaviour, https://github.com/simonw/datasette/issues/1579#issuecomment-1000481686,https://api.github.com/repos/simonw/datasette/issues/1579,1000481686,IC_kwDOBm6k_c47oiOW,9599,simonw,2021-12-23T19:09:23Z,2021-12-23T19:09:23Z,OWNER,"Re-opening this because I missed updating some of the docs, and I also need to update Datasette's own code to not use `block=True` in a bunch of places.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087931918,`.execute_write(... block=True)` should be the default behaviour, https://github.com/simonw/datasette/issues/1579#issuecomment-1000479737,https://api.github.com/repos/simonw/datasette/issues/1579,1000479737,IC_kwDOBm6k_c47ohv5,9599,simonw,2021-12-23T19:04:23Z,2021-12-23T19:04:23Z,OWNER,Updated documentation: https://github.com/simonw/datasette/blob/00a2895cd2dc42c63846216b36b2dc9f41170129/docs/internals.rst#await-dbexecute_writesql-paramsnone-blocktrue,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087931918,`.execute_write(... block=True)` should be the default behaviour, https://github.com/simonw/datasette/issues/1579#issuecomment-1000477813,https://api.github.com/repos/simonw/datasette/issues/1579,1000477813,IC_kwDOBm6k_c47ohR1,9599,simonw,2021-12-23T18:59:41Z,2021-12-23T18:59:41Z,OWNER,"I'm going to go with `execute_write(..., block=False)` as the mechanism for fire-and-forget write queries.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087931918,`.execute_write(... block=True)` should be the default behaviour, https://github.com/simonw/datasette/issues/1579#issuecomment-1000477621,https://api.github.com/repos/simonw/datasette/issues/1579,1000477621,IC_kwDOBm6k_c47ohO1,9599,simonw,2021-12-23T18:59:12Z,2021-12-23T18:59:12Z,OWNER,"The easiest way to change this would be to default to `block=True` such that you need to pass `block=False` to the APIs to have them do fire-and-forget. An alternative would be to add new, separately named methods which do the fire-and-forget thing. If I hadn't recently added `execute_write_script` and `execute_write_many` in #1570 I'd be more into this idea, but I don't want to end up with eight methods - `execute_write`, `execute_write_queue`, `execute_write_many`, `execute_write_many_queue`, `execute_write_script`, `execute_write_scrript_queue`, `execute_write_fn`, `execute_write_fn_queue`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087931918,`.execute_write(... block=True)` should be the default behaviour, https://github.com/simonw/datasette/issues/1579#issuecomment-1000476413,https://api.github.com/repos/simonw/datasette/issues/1579,1000476413,IC_kwDOBm6k_c47og79,9599,simonw,2021-12-23T18:56:06Z,2021-12-23T18:56:06Z,OWNER,"This is technically a breaking change, but a GitHub code search at https://cs.github.com/?scopeName=All+repos&scope=&q=execute_write%20datasette%20-owner%3Asimonw shows only one repo not-owned-by-me using this, and they're using `block=True`: https://github.com/mfa/datasette-webhook-write/blob/e82440f372a2f2e3ed27d1bd34c9fa3a53b49b94/datasette_webhook_write/__init__.py#L88-L89","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087931918,`.execute_write(... block=True)` should be the default behaviour, https://github.com/simonw/datasette/issues/1578#issuecomment-1000471782,https://api.github.com/repos/simonw/datasette/issues/1578,1000471782,IC_kwDOBm6k_c47ofzm,9599,simonw,2021-12-23T18:44:01Z,2021-12-23T18:44:01Z,OWNER,"The example nginx config on https://docs.datasette.io/en/stable/deploying.html#nginx-proxy-configuration is currently: ``` daemon off; events { worker_connections 1024; } http { server { listen 80; location /my-datasette { proxy_pass http://127.0.0.1:8009/my-datasette; proxy_set_header Host $host; } } } ``` This looks to me like it might exhibit the bug. Need to confirm that and figure out an alternative.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087919372,Confirm if documented nginx proxy config works for row pages with escaped characters in their primary key, https://github.com/simonw/datasette/issues/1578#issuecomment-1000471371,https://api.github.com/repos/simonw/datasette/issues/1578,1000471371,IC_kwDOBm6k_c47oftL,9599,simonw,2021-12-23T18:42:50Z,2021-12-23T18:42:50Z,OWNER,"Confirmed, that fixed the bug for me on my server.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087919372,Confirm if documented nginx proxy config works for row pages with escaped characters in their primary key, https://github.com/simonw/datasette/issues/1578#issuecomment-1000470652,https://api.github.com/repos/simonw/datasette/issues/1578,1000470652,IC_kwDOBm6k_c47ofh8,9599,simonw,2021-12-23T18:40:46Z,2021-12-23T18:40:46Z,OWNER,"[This StackOverflow answer](https://serverfault.com/a/463932) suggests that the fix is to change this: proxy_pass http://127.0.0.1:8000/; To this: proxy_pass http://127.0.0.1:8000; Quoting the nginx documentation: http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass > A request URI is passed to the server as follows: > > - If the `proxy_pass` directive is specified with a URI, then when a request is passed to the server, the part of a [normalized](http://nginx.org/en/docs/http/ngx_http_core_module.html#location) request URI matching the location is replaced by a URI specified in the directive: > > location /name/ { > proxy_pass http://127.0.0.1/remote/; > } > > - If `proxy_pass` is specified without a URI, the request URI is passed to the server in the same form as sent by a client when the original request is processed, or the full normalized request URI is passed when processing the changed URI: > > location /some/path/ { > proxy_pass http://127.0.0.1; > }","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087919372,Confirm if documented nginx proxy config works for row pages with escaped characters in their primary key, https://github.com/simonw/datasette/issues/1578#issuecomment-1000469107,https://api.github.com/repos/simonw/datasette/issues/1578,1000469107,IC_kwDOBm6k_c47ofJz,9599,simonw,2021-12-23T18:36:38Z,2021-12-23T18:36:38Z,OWNER,"This problem doesn't occur on my `localhost` running Uvicorn directly - but I'm seeing it in my production environment that runs Datasette behind an nginx proxy: ``` location / { proxy_pass http://127.0.0.1:8000/; proxy_set_header Host $host; } ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087919372,Confirm if documented nginx proxy config works for row pages with escaped characters in their primary key, https://github.com/simonw/datasette/issues/1577#issuecomment-1000462309,https://api.github.com/repos/simonw/datasette/issues/1577,1000462309,IC_kwDOBm6k_c47odfl,9599,simonw,2021-12-23T18:20:46Z,2021-12-23T18:20:46Z,OWNER,There are a lot of improvements to `asyncio` in 3.7: https://docs.python.org/3/whatsnew/3.7.html#whatsnew37-asyncio,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087913724,Drop support for Python 3.6, https://github.com/simonw/datasette/issues/1577#issuecomment-1000461900,https://api.github.com/repos/simonw/datasette/issues/1577,1000461900,IC_kwDOBm6k_c47odZM,9599,simonw,2021-12-23T18:19:44Z,2021-12-23T18:19:44Z,OWNER,"The 3.7 feature I want to use today is [contextvars](https://docs.python.org/3/library/contextvars.html) - but I have a workaround for the moment, see https://github.com/simonw/datasette/issues/1576#issuecomment-999987418 So I'm going to hold off on dropping 3.6 for a little bit longer. I imagine I'll drop it before Datasette 1.0 though. Leaving this issue open to gather thoughts and feedback on this issue from Datasette users and potential users.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087913724,Drop support for Python 3.6, https://github.com/simonw/datasette/issues/1577#issuecomment-1000461275,https://api.github.com/repos/simonw/datasette/issues/1577,1000461275,IC_kwDOBm6k_c47odPb,9599,simonw,2021-12-23T18:18:11Z,2021-12-23T18:18:11Z,OWNER,"From the Twitter thread, there are still a decent amount of LTS Linux releases out there that are stuck on pre-3.7 Python. Though many of those are 3.5 and Datasette dropped support for 3.5 in November 2019: cf7776d36fbacefa874cbd6e5fcdc9fff7661203","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087913724,Drop support for Python 3.6, https://github.com/simonw/datasette/issues/1576#issuecomment-999990414,https://api.github.com/repos/simonw/datasette/issues/1576,999990414,IC_kwDOBm6k_c47mqSO,9599,simonw,2021-12-23T02:08:39Z,2021-12-23T18:16:35Z,OWNER,"It's tiny: I'm tempted to vendor it. https://github.com/Skyscanner/aiotask-context/blob/master/aiotask_context/__init__.py No, I'll add it as a pinned dependency, which I can then drop when I drop 3.6 support.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/1576#issuecomment-999987418,https://api.github.com/repos/simonw/datasette/issues/1576,999987418,IC_kwDOBm6k_c47mpja,9599,simonw,2021-12-23T01:59:58Z,2021-12-23T02:02:12Z,OWNER,"Another option: https://github.com/Skyscanner/aiotask-context - looks like it might be better as it's been updated for Python 3.7 in this commit https://github.com/Skyscanner/aiotask-context/commit/67108c91d2abb445655cc2af446fdb52ca7890c4 The Skyscanner one doesn't attempt to wrap any existing factories, but that's OK for my purposes since I don't need to handle arbitrary `asyncio` code written by other people.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/1576#issuecomment-999876666,https://api.github.com/repos/simonw/datasette/issues/1576,999876666,IC_kwDOBm6k_c47mOg6,9599,simonw,2021-12-22T20:59:22Z,2021-12-22T21:18:09Z,OWNER,"This article is relevant: [Context information storage for asyncio](https://blog.sqreen.com/asyncio/) - in particular the section https://blog.sqreen.com/asyncio/#context-inheritance-between-tasks which describes exactly the problem I have and their solution, which involves this trickery: ```python def request_task_factory(loop, coro): child_task = asyncio.tasks.Task(coro, loop=loop) parent_task = asyncio.Task.current_task(loop=loop) current_request = getattr(parent_task, 'current_request', None) setattr(child_task, 'current_request', current_request) return child_task loop = asyncio.get_event_loop() loop.set_task_factory(request_task_factory) ``` They released their solution as a library: https://pypi.org/project/aiocontext/ and https://github.com/sqreen/AioContext - but that company was acquired by Datadog back in April and doesn't seem to be actively maintaining their open source stuff any more: https://twitter.com/SqreenIO/status/1384906075506364417","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/1576#issuecomment-999878907,https://api.github.com/repos/simonw/datasette/issues/1576,999878907,IC_kwDOBm6k_c47mPD7,9599,simonw,2021-12-22T21:03:49Z,2021-12-22T21:10:46Z,OWNER,"`context_vars` can solve this but they were introduced in Python 3.7: https://www.python.org/dev/peps/pep-0567/ Python 3.6 support ends in a few days time, and it looks like Glitch has updated to 3.7 now - so maybe I can get away with Datasette needing 3.7 these days? Tweeted about that here: https://twitter.com/simonw/status/1473761478155010048","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/1576#issuecomment-999874886,https://api.github.com/repos/simonw/datasette/issues/1576,999874886,IC_kwDOBm6k_c47mOFG,9599,simonw,2021-12-22T20:55:42Z,2021-12-22T20:57:28Z,OWNER,"One way to solve this would be to introduce a `set_task_id()` method, which sets an ID which will be returned by `get_task_id()` instead of using `id(current_task(loop=loop))`. It would be really nice if I could solve this using `with` syntax somehow. Something like: ```python with trace_child_tasks(): ( suggested_facets, (facet_results, facets_timed_out), ) = await asyncio.gather( execute_suggested_facets(), execute_facets(), ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/1576#issuecomment-999874484,https://api.github.com/repos/simonw/datasette/issues/1576,999874484,IC_kwDOBm6k_c47mN-0,9599,simonw,2021-12-22T20:54:52Z,2021-12-22T20:54:52Z,OWNER,"Here's the full current relevant code from `tracer.py`: https://github.com/simonw/datasette/blob/ace86566b28280091b3844cf5fbecd20158e9004/datasette/tracer.py#L8-L64 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/1518#issuecomment-999870993,https://api.github.com/repos/simonw/datasette/issues/1518,999870993,IC_kwDOBm6k_c47mNIR,9599,simonw,2021-12-22T20:47:18Z,2021-12-22T20:50:24Z,OWNER,"The reason they aren't showing up in the traces is that traces are stored just for the currently executing `asyncio` task ID: https://github.com/simonw/datasette/blob/ace86566b28280091b3844cf5fbecd20158e9004/datasette/tracer.py#L13-L25 This is so traces for other incoming requests don't end up mixed together. But there's no current mechanism to track async tasks that are effectively ""child tasks"" of the current request, and hence should be tracked the same. https://stackoverflow.com/a/69349501/6083 suggests that you pass the task ID as an argument to the child tasks that are executed using `asyncio.gather()` to work around this kind of problem.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-999870282,https://api.github.com/repos/simonw/datasette/issues/1518,999870282,IC_kwDOBm6k_c47mM9K,9599,simonw,2021-12-22T20:45:56Z,2021-12-22T20:46:08Z,OWNER,"> New short-term goal: get facets and suggested facets to execute in parallel with the main query. Generate a trace graph that proves that is happening using `datasette-pretty-traces`. I wrote code to execute those in parallel using `asyncio.gather()` - which seems to work but causes the SQL run inside the parallel `async def` functions not to show up in the trace graph at all. ```diff diff --git a/datasette/views/table.py b/datasette/views/table.py index 9808fd2..ec9db64 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -1,3 +1,4 @@ +import asyncio import urllib import itertools import json @@ -615,44 +616,37 @@ class TableView(RowTableShared): if request.args.get(""_timelimit""): extra_args[""custom_time_limit""] = int(request.args.get(""_timelimit"")) - # Execute the main query! - results = await db.execute(sql, params, truncate=True, **extra_args) - - # Calculate the total count for this query - filtered_table_rows_count = None - if ( - not db.is_mutable - and self.ds.inspect_data - and count_sql == f""select count(*) from {table} "" - ): - # We can use a previously cached table row count - try: - filtered_table_rows_count = self.ds.inspect_data[database][""tables""][ - table - ][""count""] - except KeyError: - pass - - # Otherwise run a select count(*) ... - if count_sql and filtered_table_rows_count is None and not nocount: - try: - count_rows = list(await db.execute(count_sql, from_sql_params)) - filtered_table_rows_count = count_rows[0][0] - except QueryInterrupted: - pass - - # Faceting - if not self.ds.setting(""allow_facet"") and any( - arg.startswith(""_facet"") for arg in request.args - ): - raise BadRequest(""_facet= is not allowed"") + async def execute_count(): + # Calculate the total count for this query + filtered_table_rows_count = None + if ( + not db.is_mutable + and self.ds.inspect_data + and count_sql == f""select count(*) from {table} "" + ): + # We can use a previously cached table row count + try: + filtered_table_rows_count = self.ds.inspect_data[database][ + ""tables"" + ][table][""count""] + except KeyError: + pass + + if count_sql and filtered_table_rows_count is None and not nocount: + try: + count_rows = list(await db.execute(count_sql, from_sql_params)) + filtered_table_rows_count = count_rows[0][0] + except QueryInterrupted: + pass + + return filtered_table_rows_count + + filtered_table_rows_count = await execute_count() # pylint: disable=no-member facet_classes = list( itertools.chain.from_iterable(pm.hook.register_facet_classes()) ) - facet_results = {} - facets_timed_out = [] facet_instances = [] for klass in facet_classes: facet_instances.append( @@ -668,33 +662,58 @@ class TableView(RowTableShared): ) ) - if not nofacet: - for facet in facet_instances: - ( - instance_facet_results, - instance_facets_timed_out, - ) = await facet.facet_results() - for facet_info in instance_facet_results: - base_key = facet_info[""name""] - key = base_key - i = 1 - while key in facet_results: - i += 1 - key = f""{base_key}_{i}"" - facet_results[key] = facet_info - facets_timed_out.extend(instance_facets_timed_out) - - # Calculate suggested facets - suggested_facets = [] - if ( - self.ds.setting(""suggest_facets"") - and self.ds.setting(""allow_facet"") - and not _next - and not nofacet - and not nosuggest - ): - for facet in facet_instances: - suggested_facets.extend(await facet.suggest()) + async def execute_suggested_facets(): + # Calculate suggested facets + suggested_facets = [] + if ( + self.ds.setting(""suggest_facets"") + and self.ds.setting(""allow_facet"") + and not _next + and not nofacet + and not nosuggest + ): + for facet in facet_instances: + suggested_facets.extend(await facet.suggest()) + return suggested_facets + + async def execute_facets(): + facet_results = {} + facets_timed_out = [] + if not self.ds.setting(""allow_facet"") and any( + arg.startswith(""_facet"") for arg in request.args + ): + raise BadRequest(""_facet= is not allowed"") + + if not nofacet: + for facet in facet_instances: + ( + instance_facet_results, + instance_facets_timed_out, + ) = await facet.facet_results() + for facet_info in instance_facet_results: + base_key = facet_info[""name""] + key = base_key + i = 1 + while key in facet_results: + i += 1 + key = f""{base_key}_{i}"" + facet_results[key] = facet_info + facets_timed_out.extend(instance_facets_timed_out) + + return facet_results, facets_timed_out + + # Execute the main query, facets and facet suggestions in parallel: + ( + results, + suggested_facets, + (facet_results, facets_timed_out), + ) = await asyncio.gather( + db.execute(sql, params, truncate=True, **extra_args), + execute_suggested_facets(), + execute_facets(), + ) + + results = await db.execute(sql, params, truncate=True, **extra_args) # Figure out columns and rows for the query columns = [r[0] for r in results.description] ``` Here's the trace for `http://127.0.0.1:4422/fixtures/compound_three_primary_keys?_trace=1&_facet=pk1&_facet=pk2` with the missing facet and facet suggestion queries: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-999863269,https://api.github.com/repos/simonw/datasette/issues/1518,999863269,IC_kwDOBm6k_c47mLPl,9599,simonw,2021-12-22T20:35:41Z,2021-12-22T20:37:13Z,OWNER,"It looks like the count has to be executed before facets can be, because the facet_class constructor needs that total count figure: https://github.com/simonw/datasette/blob/6b1384b2f529134998fb507e63307609a5b7f5c0/datasette/views/table.py#L660-L671 It's used in facet suggestion logic here: https://github.com/simonw/datasette/blob/ace86566b28280091b3844cf5fbecd20158e9004/datasette/facets.py#L172-L178","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-999850191,https://api.github.com/repos/simonw/datasette/issues/1518,999850191,IC_kwDOBm6k_c47mIDP,9599,simonw,2021-12-22T20:29:38Z,2021-12-22T20:29:38Z,OWNER,New short-term goal: get facets and suggested facets to execute in parallel with the main query. Generate a trace graph that proves that is happening using `datasette-pretty-traces`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-999837569,https://api.github.com/repos/simonw/datasette/issues/1518,999837569,IC_kwDOBm6k_c47mE-B,9599,simonw,2021-12-22T20:15:45Z,2021-12-22T20:15:45Z,OWNER,"Also the whole `special_args` v.s. `request.args` thing is pretty confusing, I think that might be an older code pattern back from when I was using Sanic.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-999837220,https://api.github.com/repos/simonw/datasette/issues/1518,999837220,IC_kwDOBm6k_c47mE4k,9599,simonw,2021-12-22T20:15:04Z,2021-12-22T20:15:04Z,OWNER,"I think I can move this much higher up in the method, it's a bit confusing having it half way through: https://github.com/simonw/datasette/blob/6b1384b2f529134998fb507e63307609a5b7f5c0/datasette/views/table.py#L414-L436","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-999831967,https://api.github.com/repos/simonw/datasette/issues/1518,999831967,IC_kwDOBm6k_c47mDmf,9599,simonw,2021-12-22T20:04:47Z,2021-12-22T20:10:11Z,OWNER,"I think I might be able to clean up a lot of the stuff in here using the `render_cell` plugin hook: https://github.com/simonw/datasette/blob/6b1384b2f529134998fb507e63307609a5b7f5c0/datasette/views/table.py#L87-L89 The catch with that hook - https://docs.datasette.io/en/stable/plugin_hooks.html#render-cell-value-column-table-database-datasette - is that it gets called for every single cell. I don't want the overhead of looking up the foreign key relationships etc once for every value in a specific column. But maybe I could extend the hook to include a shared cache that gets used for all of the cells in a specific table? Something like this: ```python render_cell(value, column, table, database, datasette, cache) ``` `cache` is a dictionary - and the same dictionary is passed to every call to that hook while rendering a specific page. It's a bit of a gross hack though, and would it ever be useful for plugins outside of the default plugin in Datasette which does the foreign key stuff? If I can think of one other potential application for this `cache` then I might implement it. No, this optimization doesn't make sense: the most complex cell enrichment logic is the stuff that does a `select * from categories where id in (2, 5, 6)` query, using just the distinct set of IDs that are rendered on the current page. That's not going to fit in the `render_cell` hook no matter how hard I try to warp it into the right shape, because it needs full visibility of all of the results that are being rendered in order to collect those unique ID values.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/pull/1554#issuecomment-998354538,https://api.github.com/repos/simonw/datasette/issues/1554,998354538,IC_kwDOBm6k_c47ga5q,9599,simonw,2021-12-20T23:52:04Z,2021-12-20T23:52:04Z,OWNER,Abandoning this since it didn't work how I wanted.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079129258,TableView refactor, https://github.com/simonw/datasette/issues/1547#issuecomment-997514220,https://api.github.com/repos/simonw/datasette/issues/1547,997514220,IC_kwDOBm6k_c47dNvs,9599,simonw,2021-12-20T01:26:25Z,2021-12-20T01:26:25Z,OWNER,"OK, this should hopefully fix that for you: pip install https://github.com/simonw/datasette/archive/f36e010b3b69ada104b79d83c7685caf9359049e.zip","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1076388044,Writable canned queries fail to load custom templates, https://github.com/simonw/datasette/issues/1547#issuecomment-997513369,https://api.github.com/repos/simonw/datasette/issues/1547,997513369,IC_kwDOBm6k_c47dNiZ,9599,simonw,2021-12-20T01:24:43Z,2021-12-20T01:24:43Z,OWNER,"@wragge thanks, that's a bug! Working on that in #1575.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1076388044,Writable canned queries fail to load custom templates, https://github.com/simonw/datasette/issues/1575#issuecomment-997513177,https://api.github.com/repos/simonw/datasette/issues/1575,997513177,IC_kwDOBm6k_c47dNfZ,9599,simonw,2021-12-20T01:24:25Z,2021-12-20T01:24:25Z,OWNER,Looks like `specname` is new in Pluggy 1.0: https://github.com/pytest-dev/pluggy/blob/main/CHANGELOG.rst#pluggy-100-2021-08-25,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1084257842,__call__() got an unexpected keyword argument 'specname', https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997507074,https://api.github.com/repos/simonw/sqlite-utils/issues/356,997507074,IC_kwDOCGYnMM47dMAC,9599,simonw,2021-12-20T01:10:06Z,2021-12-20T01:16:11Z,OWNER,"Work-in-progress improved help: ``` Usage: sqlite-utils insert [OPTIONS] PATH TABLE FILE Insert records from FILE into a table, creating the table if it does not already exist. By default the input is expected to be a JSON array of objects. Or: - Use --nl for newline-delimited JSON objects - Use --csv or --tsv for comma-separated or tab-separated input - Use --lines to write each incoming line to a column called ""line"" - Use --all to write the entire input to a column called ""all"" You can also use --convert to pass a fragment of Python code that will be used to convert each input. Your Python code will be passed a ""row"" variable representing the imported row, and can return a modified row. If you are using --lines your code will be passed a ""line"" variable, and for --all an ""all"" variable. Options: --pk TEXT Columns to use as the primary key, e.g. id --flatten Flatten nested JSON objects, so {""a"": {""b"": 1}} becomes {""a_b"": 1} --nl Expect newline-delimited JSON -c, --csv Expect CSV input --tsv Expect TSV input --lines Treat each line as a single value called 'line' --all Treat input as a single value called 'all' --convert TEXT Python code to convert each item --import TEXT Python modules to import --delimiter TEXT Delimiter to use for CSV files --quotechar TEXT Quote character to use for CSV/TSV --sniff Detect delimiter and quote character --no-headers CSV file has no header row --batch-size INTEGER Commit every X records --alter Alter existing table to add any missing columns --not-null TEXT Columns that should be created as NOT NULL --default ... Default value that should be set for a column --encoding TEXT Character encoding for input, defaults to utf-8 -d, --detect-types Detect types for columns in CSV/TSV data --load-extension TEXT SQLite extensions to load --silent Do not show progress bar --ignore Ignore records if pk already exists --replace Replace records if pk already exists --truncate Truncate table before inserting records, if table already exists -h, --help Show this message and exit. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option, https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997508728,https://api.github.com/repos/simonw/sqlite-utils/issues/356,997508728,IC_kwDOCGYnMM47dMZ4,9599,simonw,2021-12-20T01:14:43Z,2021-12-20T01:14:43Z,OWNER,(This makes me want `--extract` from #352 even more.),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option, https://github.com/simonw/sqlite-utils/issues/163#issuecomment-997502242,https://api.github.com/repos/simonw/sqlite-utils/issues/163,997502242,IC_kwDOCGYnMM47dK0i,9599,simonw,2021-12-20T00:56:45Z,2021-12-20T00:56:52Z,OWNER,"> Maybe `sqlite-utils` should absorb all of the functionality from `sqlite-transform` - having two separate tools doesn't necessarily make sense. I implemented that in: - #251","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706001517,Idea: conversions= could take Python functions, https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997497262,https://api.github.com/repos/simonw/sqlite-utils/issues/356,997497262,IC_kwDOCGYnMM47dJmu,9599,simonw,2021-12-20T00:40:15Z,2021-12-20T00:40:15Z,OWNER,`--flatten` could do with a better description too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option, https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997496931,https://api.github.com/repos/simonw/sqlite-utils/issues/356,997496931,IC_kwDOCGYnMM47dJhj,9599,simonw,2021-12-20T00:39:14Z,2021-12-20T00:39:52Z,OWNER,"``` % sqlite-utils insert --help Usage: sqlite-utils insert [OPTIONS] PATH TABLE JSON_FILE Insert records from JSON file into a table, creating the table if it does not already exist. Input should be a JSON array of objects, unless --nl or --csv is used. Options: --pk TEXT Columns to use as the primary key, e.g. id --nl Expect newline-delimited JSON --flatten Flatten nested JSON objects -c, --csv Expect CSV --tsv Expect TSV --convert TEXT Python code to convert each item --import TEXT Python modules to import --delimiter TEXT Delimiter to use for CSV files --quotechar TEXT Quote character to use for CSV/TSV --sniff Detect delimiter and quote character --no-headers CSV file has no header row --batch-size INTEGER Commit every X records --alter Alter existing table to add any missing columns --not-null TEXT Columns that should be created as NOT NULL --default ... Default value that should be set for a column --encoding TEXT Character encoding for input, defaults to utf-8 -d, --detect-types Detect types for columns in CSV/TSV data --load-extension TEXT SQLite extensions to load --silent Do not show progress bar --ignore Ignore records if pk already exists --replace Replace records if pk already exists --truncate Truncate table before inserting records, if table already exists -h, --help Show this message and exit. ``` I can add a bunch of extra help at the top there to explain all of this stuff. That ""Input should be a JSON array of objects"" bit could be expanded to several paragraphs.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option, https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997492872,https://api.github.com/repos/simonw/sqlite-utils/issues/356,997492872,IC_kwDOCGYnMM47dIiI,9599,simonw,2021-12-20T00:23:31Z,2021-12-20T00:23:31Z,OWNER,"I think this should work on JSON, or CSV, or individual lines, or the entire content at once. So I'll require `--lines --convert ...` to import individual lines, or `--all --convert` to run the conversion against the entire input at once. What would `--lines` or `--all` do without `--convert`? Maybe insert records as `{""line"": ""line of text""}` or `{""all"": ""whole input}`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option, https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997486156,https://api.github.com/repos/simonw/sqlite-utils/issues/356,997486156,IC_kwDOCGYnMM47dG5M,9599,simonw,2021-12-19T23:51:02Z,2021-12-19T23:51:02Z,OWNER,This is going to need a `--import` multi option too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option, https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997485361,https://api.github.com/repos/simonw/sqlite-utils/issues/356,997485361,IC_kwDOCGYnMM47dGsx,9599,simonw,2021-12-19T23:45:30Z,2021-12-19T23:45:30Z,OWNER,Really interesting example input for this: https://blog.timac.org/2021/1219-state-of-swift-and-swiftui-ios15/iOS13.txt - see https://blog.timac.org/2021/1219-state-of-swift-and-swiftui-ios15/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option, https://github.com/simonw/datasette/issues/1565#issuecomment-997474022,https://api.github.com/repos/simonw/datasette/issues/1565,997474022,IC_kwDOBm6k_c47dD7m,9599,simonw,2021-12-19T22:36:49Z,2021-12-19T22:37:29Z,OWNER,"No way with a tagged template literal to pass an extra database name argument, so instead I need a method that returns a callable that can be used for the tagged template literal for a specific database - or the default database. This could work (bit weird looking though): ```javascript var rows = await datasette.query(""fixtures"")`select * from foo`; ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins, https://github.com/simonw/datasette/issues/1565#issuecomment-997473856,https://api.github.com/repos/simonw/datasette/issues/1565,997473856,IC_kwDOBm6k_c47dD5A,9599,simonw,2021-12-19T22:35:20Z,2021-12-19T22:35:20Z,OWNER,"Quick prototype of that tagged template `query` function: ```javascript function query(pieces, ...parameters) { var qs = new URLSearchParams(); var sql = pieces[0]; parameters.forEach((param, i) => { sql += `:p${i}${pieces[i + 1]}`; qs.append(`p${i}`, param); }); qs.append(""sql"", sql); return qs.toString(); } var id = 4; console.log(query`select * from ids where id > ${id}`); ``` Outputs: ``` p0=4&sql=select+*+from+ids+where+id+%3E+%3Ap0 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins, https://github.com/simonw/datasette/issues/1565#issuecomment-997472639,https://api.github.com/repos/simonw/datasette/issues/1565,997472639,IC_kwDOBm6k_c47dDl_,9599,simonw,2021-12-19T22:25:50Z,2021-12-19T22:25:50Z,OWNER,"Or... ```javascript rows = await datasette.query`select * from searchable where id > ${id}`; ``` And it knows how to turn that into a parameterized call using tagged template literals.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins, https://github.com/simonw/datasette/issues/1565#issuecomment-997472509,https://api.github.com/repos/simonw/datasette/issues/1565,997472509,IC_kwDOBm6k_c47dDj9,9599,simonw,2021-12-19T22:24:50Z,2021-12-19T22:24:50Z,OWNER,"... huh, it could even expose a JavaScript function that can be called to execute a SQL query. ```javascript datasette.query(""select * from blah"").then(...) ``` Maybe it takes an optional second argument that specifies the database - defaulting to the one for the current page.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins, https://github.com/simonw/datasette/issues/1565#issuecomment-997472370,https://api.github.com/repos/simonw/datasette/issues/1565,997472370,IC_kwDOBm6k_c47dDhy,9599,simonw,2021-12-19T22:23:36Z,2021-12-19T22:23:36Z,OWNER,This should also expose the JSON API endpoints used to execute SQL against this database.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins, https://github.com/simonw/datasette/issues/1518#issuecomment-997472214,https://api.github.com/repos/simonw/datasette/issues/1518,997472214,IC_kwDOBm6k_c47dDfW,9599,simonw,2021-12-19T22:22:08Z,2021-12-19T22:22:08Z,OWNER,"I sketched out a chained SQL builder pattern that might be useful for further tidying up this code - though with the new plugin hook I'm less excited about it than I was: ```python class TableQuery: def __init__(self, table, columns, pks, is_view=False, prev=None): self.table = table self.columns = columns self.pks = pks self.is_view = is_view self.prev = prev # These can be changed for different instances in the chain: self._where_clauses = None self._order_by = None self._page_size = None self._offset = None self._select_columns = None self.select_all_columns = '*' self.select_specified_columns = '*' @property def where_clauses(self): wheres = [] current = self while current: if current._where_clauses is not None: wheres.extend(current._where_clauses) current = current.prev return list(reversed(wheres)) def where(self, where): new_cls = TableQuery(self.table, self.columns, self.pks, self.is_view, self) new_cls._where_clauses = [where] return new_cls @classmethod async def introspect(cls, db, table): return cls( table, columns = await db.table_columns(table), pks = await db.primary_keys(table), is_view = bool(await db.get_view_definition(table)) ) @property def sql_from(self): return f""from {self.table}{self.sql_where}"" @property def sql_where(self): if not self.where_clauses: return """" else: return f"" where {' and '.join(self.where_clauses)}"" @property def sql_no_order_no_limit(self): return f""select {self.select_all_columns} from {self.table}{self.sql_where}"" @property def sql(self): return f""select {self.select_specified_columns} from {self.table} {self.sql_where}{self._order_by} limit {self._page_size}{self._offset}"" @property def sql_count(self): return f""select count(*) {self.sql_from}"" def __repr__(self): return f"""" ``` Usage: ```python from datasette.app import Datasette ds = Datasette(memory=True, files=[""/Users/simon/Dropbox/Development/datasette/fixtures.db""]) db = ds.get_database(""fixtures"") query = await TableQuery.introspect(db, ""facetable"") print(query.where(""foo = bar"").where(""baz = 1"").sql_count) # 'select count(*) from facetable where foo = bar and baz = 1' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1547#issuecomment-997471672,https://api.github.com/repos/simonw/datasette/issues/1547,997471672,IC_kwDOBm6k_c47dDW4,9599,simonw,2021-12-19T22:18:26Z,2021-12-19T22:18:26Z,OWNER,"I released this [in an alpha](https://github.com/simonw/datasette/releases/tag/0.60a1), so you can try out this fix using: pip install datasette==0.60a1","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1076388044,Writable canned queries fail to load custom templates, https://github.com/simonw/datasette/issues/1566#issuecomment-997470633,https://api.github.com/repos/simonw/datasette/issues/1566,997470633,IC_kwDOBm6k_c47dDGp,9599,simonw,2021-12-19T22:12:00Z,2021-12-19T22:12:00Z,OWNER,"Released another alpha, 0.60a1: https://github.com/simonw/datasette/releases/tag/0.60a1","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083669410,Release Datasette 0.60, https://github.com/simonw/datasette/issues/1545#issuecomment-997462604,https://api.github.com/repos/simonw/datasette/issues/1545,997462604,IC_kwDOBm6k_c47dBJM,9599,simonw,2021-12-19T21:17:08Z,2021-12-19T21:17:08Z,OWNER,"Here's the relevant code: https://github.com/simonw/datasette/blob/4094741c2881c2ada3f3f878b532fdaec7914953/datasette/app.py#L1204-L1219 It's using `route_path.split(""/"")` which should be OK because that's the incoming `request.path` path - which I would expect to use `/` even on Windows. Then it uses `os.path.join` which should do the right thing. I need to get myself a proper Windows development environment setup to investigate this one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1075893249,Custom pages don't work on windows, https://github.com/simonw/datasette/issues/1573#issuecomment-997462117,https://api.github.com/repos/simonw/datasette/issues/1573,997462117,IC_kwDOBm6k_c47dBBl,9599,simonw,2021-12-19T21:13:13Z,2021-12-19T21:13:13Z,OWNER,This might also be the impetus I need to bring the https://datasette.io/plugins/datasette-pretty-traces plugin into Datasette core itself.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1084185188,Make trace() a documented internal API, https://github.com/simonw/datasette/issues/1547#issuecomment-997460731,https://api.github.com/repos/simonw/datasette/issues/1547,997460731,IC_kwDOBm6k_c47dAr7,9599,simonw,2021-12-19T21:02:15Z,2021-12-19T21:02:15Z,OWNER,"Yes, this is a bug. It looks like the problem is with the `if write:` branch in this code here: https://github.com/simonw/datasette/blob/5fac26aa221a111d7633f2dd92014641f7c0ade9/datasette/views/database.py#L252-L327 Is missing this bit of code: https://github.com/simonw/datasette/blob/5fac26aa221a111d7633f2dd92014641f7c0ade9/datasette/views/database.py#L343-L347","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1076388044,Writable canned queries fail to load custom templates, https://github.com/simonw/datasette/issues/1570#issuecomment-997460061,https://api.github.com/repos/simonw/datasette/issues/1570,997460061,IC_kwDOBm6k_c47dAhd,9599,simonw,2021-12-19T20:56:54Z,2021-12-19T20:56:54Z,OWNER,Documentation: https://docs.datasette.io/en/latest/internals.html#await-db-execute-write-sql-params-none-block-false,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083921371,Separate db.execute_write() into three methods, https://github.com/simonw/datasette/issues/1555#issuecomment-997459958,https://api.github.com/repos/simonw/datasette/issues/1555,997459958,IC_kwDOBm6k_c47dAf2,9599,simonw,2021-12-19T20:55:59Z,2021-12-19T20:55:59Z,OWNER,"Closing this issue because I've optimized this a whole bunch, and it's definitely good enough for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997325189,https://api.github.com/repos/simonw/datasette/issues/1555,997325189,IC_kwDOBm6k_c47cfmF,9599,simonw,2021-12-19T03:55:01Z,2021-12-19T20:54:51Z,OWNER,"It's a bit annoying that the queries no longer show up in the trace at all now, thanks to running in `.execute_fn()`. I wonder if there's something smart I can do about that - maybe have `trace()` record that function with a traceback even though it doesn't have the executed SQL string? 5fac26aa221a111d7633f2dd92014641f7c0ade9 has the same problem.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997459637,https://api.github.com/repos/simonw/datasette/issues/1555,997459637,IC_kwDOBm6k_c47dAa1,9599,simonw,2021-12-19T20:53:46Z,2021-12-19T20:53:46Z,OWNER,Using #1571 showed me that the `DELETE FROM columns/foreign_keys/indexes WHERE database_name = ? and table_name = ?` queries were running way more times than I expected. I came up with a new optimization that just does `DELETE FROM columns/foreign_keys/indexes WHERE database_name = ?` instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1566#issuecomment-997457790,https://api.github.com/repos/simonw/datasette/issues/1566,997457790,IC_kwDOBm6k_c47c_9-,9599,simonw,2021-12-19T20:40:50Z,2021-12-19T20:40:57Z,OWNER,"Also release new version of `datasette-pretty-traces` with this feature: - https://github.com/simonw/datasette-pretty-traces/issues/7","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083669410,Release Datasette 0.60, https://github.com/simonw/datasette/issues/1555#issuecomment-997342494,https://api.github.com/repos/simonw/datasette/issues/1555,997342494,IC_kwDOBm6k_c47cj0e,9599,simonw,2021-12-19T07:22:04Z,2021-12-19T07:22:04Z,OWNER,"Another option would be to provide an abstraction that makes it easier to run a group of SQL queries in the same thread at the same time, and have them traced correctly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997324666,https://api.github.com/repos/simonw/datasette/issues/1555,997324666,IC_kwDOBm6k_c47cfd6,9599,simonw,2021-12-19T03:47:51Z,2021-12-19T03:48:09Z,OWNER,"Here's a hacked together prototype of running all of that stuff inside a single function passed to `.execute_fn()`: ```diff diff --git a/datasette/utils/internal_db.py b/datasette/utils/internal_db.py index 95055d8..58f9982 100644 --- a/datasette/utils/internal_db.py +++ b/datasette/utils/internal_db.py @@ -1,4 +1,5 @@ import textwrap +from datasette.utils import table_column_details async def init_internal_db(db): @@ -70,49 +71,70 @@ async def populate_schema_tables(internal_db, db): ""DELETE FROM tables WHERE database_name = ?"", [database_name], block=True ) tables = (await db.execute(""select * from sqlite_master WHERE type = 'table'"")).rows - tables_to_insert = [] - columns_to_delete = [] - columns_to_insert = [] - foreign_keys_to_delete = [] - foreign_keys_to_insert = [] - indexes_to_delete = [] - indexes_to_insert = [] - for table in tables: - table_name = table[""name""] - tables_to_insert.append( - (database_name, table_name, table[""rootpage""], table[""sql""]) - ) - columns_to_delete.append((database_name, table_name)) - columns = await db.table_column_details(table_name) - columns_to_insert.extend( - { - **{""database_name"": database_name, ""table_name"": table_name}, - **column._asdict(), - } - for column in columns - ) - foreign_keys_to_delete.append((database_name, table_name)) - foreign_keys = ( - await db.execute(f""PRAGMA foreign_key_list([{table_name}])"") - ).rows - foreign_keys_to_insert.extend( - { - **{""database_name"": database_name, ""table_name"": table_name}, - **dict(foreign_key), - } - for foreign_key in foreign_keys - ) - indexes_to_delete.append((database_name, table_name)) - indexes = (await db.execute(f""PRAGMA index_list([{table_name}])"")).rows - indexes_to_insert.extend( - { - **{""database_name"": database_name, ""table_name"": table_name}, - **dict(index), - } - for index in indexes + def collect_info(conn): + tables_to_insert = [] + columns_to_delete = [] + columns_to_insert = [] + foreign_keys_to_delete = [] + foreign_keys_to_insert = [] + indexes_to_delete = [] + indexes_to_insert = [] + + for table in tables: + table_name = table[""name""] + tables_to_insert.append( + (database_name, table_name, table[""rootpage""], table[""sql""]) + ) + columns_to_delete.append((database_name, table_name)) + columns = table_column_details(conn, table_name) + columns_to_insert.extend( + { + **{""database_name"": database_name, ""table_name"": table_name}, + **column._asdict(), + } + for column in columns + ) + foreign_keys_to_delete.append((database_name, table_name)) + foreign_keys = conn.execute( + f""PRAGMA foreign_key_list([{table_name}])"" + ).fetchall() + foreign_keys_to_insert.extend( + { + **{""database_name"": database_name, ""table_name"": table_name}, + **dict(foreign_key), + } + for foreign_key in foreign_keys + ) + indexes_to_delete.append((database_name, table_name)) + indexes = conn.execute(f""PRAGMA index_list([{table_name}])"").fetchall() + indexes_to_insert.extend( + { + **{""database_name"": database_name, ""table_name"": table_name}, + **dict(index), + } + for index in indexes + ) + return ( + tables_to_insert, + columns_to_delete, + columns_to_insert, + foreign_keys_to_delete, + foreign_keys_to_insert, + indexes_to_delete, + indexes_to_insert, ) + ( + tables_to_insert, + columns_to_delete, + columns_to_insert, + foreign_keys_to_delete, + foreign_keys_to_insert, + indexes_to_delete, + indexes_to_insert, + ) = await db.execute_fn(collect_info) + await internal_db.execute_write_many( """""" INSERT INTO tables (database_name, table_name, rootpage, sql) ``` First impressions: it looks like this helps **a lot** - as far as I can tell this is now taking around 21ms to get to the point at which all of those internal databases have been populated, where previously it took more than 180ms. ![CleanShot 2021-12-18 at 19 47 22@2x](https://user-images.githubusercontent.com/9599/146663192-bba098d5-e7bd-4e2e-b525-2270867888a0.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997324156,https://api.github.com/repos/simonw/datasette/issues/1555,997324156,IC_kwDOBm6k_c47cfV8,9599,simonw,2021-12-19T03:40:05Z,2021-12-19T03:40:05Z,OWNER,"Using the prototype of this: - https://github.com/simonw/datasette-pretty-traces/issues/5 I'm seeing about 180ms spent running all of these queries on startup! ![CleanShot 2021-12-18 at 19 38 37@2x](https://user-images.githubusercontent.com/9599/146663045-46bda669-90de-474f-8870-345182725dc1.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997321767,https://api.github.com/repos/simonw/datasette/issues/1555,997321767,IC_kwDOBm6k_c47cewn,9599,simonw,2021-12-19T03:10:58Z,2021-12-19T03:10:58Z,OWNER,"I wonder how much overhead there is switching between the `async` event loop main code and the thread that runs the SQL queries. Would there be a performance boost if I gathered all of the column/index information in a single function run on the thread using `db.execute_fn()` I wonder? It would eliminate a bunch of switching between threads. Would be great to understand how much of an impact that would have.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997321653,https://api.github.com/repos/simonw/datasette/issues/1555,997321653,IC_kwDOBm6k_c47ceu1,9599,simonw,2021-12-19T03:09:43Z,2021-12-19T03:09:43Z,OWNER,"On that same documentation page I just spotted this: > This feature is experimental and is subject to change. Further documentation will become available if and when the table-valued functions for PRAGMAs feature becomes officially supported. This makes me nervous to rely on pragma function optimizations in Datasette itself.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997321477,https://api.github.com/repos/simonw/datasette/issues/1555,997321477,IC_kwDOBm6k_c47cesF,9599,simonw,2021-12-19T03:07:33Z,2021-12-19T03:07:33Z,OWNER,"If I want to continue supporting SQLite prior to 3.16.0 (2017-01-02) I'll need this optimization to only kick in with versions that support table-valued PRAGMA functions, while keeping the old `PRAGMA foreign_key_list(table)` stuff working for those older versions. That's feasible, but it's a bit more work - and I need to make sure I have robust testing in place for SQLite 3.15.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997321327,https://api.github.com/repos/simonw/datasette/issues/1555,997321327,IC_kwDOBm6k_c47cepv,9599,simonw,2021-12-19T03:05:39Z,2021-12-19T03:05:44Z,OWNER,"This caught me out once before in: - https://github.com/simonw/datasette/issues/1276 Turns out Glitch was running SQLite 3.11.0 from 2016-02-15.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997321217,https://api.github.com/repos/simonw/datasette/issues/1555,997321217,IC_kwDOBm6k_c47ceoB,9599,simonw,2021-12-19T03:04:16Z,2021-12-19T03:04:16Z,OWNER,"One thing to watch out for though, from https://sqlite.org/pragma.html#pragfunc > The table-valued functions for PRAGMA feature was added in SQLite version 3.16.0 (2017-01-02). Prior versions of SQLite cannot use this feature. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997321115,https://api.github.com/repos/simonw/datasette/issues/1555,997321115,IC_kwDOBm6k_c47cemb,9599,simonw,2021-12-19T03:03:12Z,2021-12-19T03:03:12Z,OWNER,"Table columns is a bit harder, because `table_xinfo` is only in SQLite 3.26.0 or higher: https://github.com/simonw/datasette/blob/d637ed46762fdbbd8e32b86f258cd9a53c1cfdc7/datasette/utils/__init__.py#L565-L581 So if that function is available: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++table_xinfo.*%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_table_xinfo%28sqlite_master.name%29+AS+table_xinfo%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27 ```sql SELECT sqlite_master.name, table_xinfo.* FROM sqlite_master, pragma_table_xinfo(sqlite_master.name) AS table_xinfo WHERE sqlite_master.type = 'table' ``` And otherwise, using `table_info`: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++table_info.*%2C%0D%0A++0+as+hidden%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_table_info%28sqlite_master.name%29+AS+table_info%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27 ```sql SELECT sqlite_master.name, table_info.*, 0 as hidden FROM sqlite_master, pragma_table_info(sqlite_master.name) AS table_info WHERE sqlite_master.type = 'table' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997320824,https://api.github.com/repos/simonw/datasette/issues/1555,997320824,IC_kwDOBm6k_c47ceh4,9599,simonw,2021-12-19T02:59:57Z,2021-12-19T03:00:44Z,OWNER,"To list all indexes: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++index_list.*%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_index_list%28sqlite_master.name%29+AS+index_list%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27 ```sql SELECT sqlite_master.name, index_list.* FROM sqlite_master, pragma_index_list(sqlite_master.name) AS index_list WHERE sqlite_master.type = 'table' ``` Foreign keys: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++foreign_key_list.*%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_foreign_key_list%28sqlite_master.name%29+AS+foreign_key_list%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27 ```sql SELECT sqlite_master.name, foreign_key_list.* FROM sqlite_master, pragma_foreign_key_list(sqlite_master.name) AS foreign_key_list WHERE sqlite_master.type = 'table' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1566#issuecomment-997272328,https://api.github.com/repos/simonw/datasette/issues/1566,997272328,IC_kwDOBm6k_c47cSsI,9599,simonw,2021-12-18T19:18:01Z,2021-12-18T19:18:01Z,OWNER,"Added some useful new documented internal methods in: - #1570","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083669410,Release Datasette 0.60, https://github.com/simonw/datasette/issues/1555#issuecomment-997272223,https://api.github.com/repos/simonw/datasette/issues/1555,997272223,IC_kwDOBm6k_c47cSqf,9599,simonw,2021-12-18T19:17:13Z,2021-12-18T19:17:13Z,OWNER,That's a good optimization. Still need to deal with the huge flurry of `PRAGMA` queries though before I can consider this done.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1570#issuecomment-997267583,https://api.github.com/repos/simonw/datasette/issues/1570,997267583,IC_kwDOBm6k_c47cRh_,9599,simonw,2021-12-18T18:46:05Z,2021-12-18T18:46:12Z,OWNER,This will replace the work done in #1569.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083921371,Separate db.execute_write() into three methods, https://github.com/simonw/datasette/issues/1555#issuecomment-997267416,https://api.github.com/repos/simonw/datasette/issues/1555,997267416,IC_kwDOBm6k_c47cRfY,9599,simonw,2021-12-18T18:44:53Z,2021-12-18T18:45:28Z,OWNER,"Rather than adding a `executemany=True` parameter, I'm now thinking a better design might be to have three methods: - `db.execute_write(sql, params=None, block=False)` - `db.execute_writescript(sql, block=False)` - `db.execute_writemany(sql, params_seq, block=False)`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1569#issuecomment-997266687,https://api.github.com/repos/simonw/datasette/issues/1569,997266687,IC_kwDOBm6k_c47cRT_,9599,simonw,2021-12-18T18:41:40Z,2021-12-18T18:41:40Z,OWNER,Updated documentation: https://docs.datasette.io/en/latest/internals.html#await-db-execute-write-sql-params-none-executescript-false-block-false,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083895395,"db.execute_write(..., executescript=True) parameter", https://github.com/simonw/datasette/issues/1555#issuecomment-997266100,https://api.github.com/repos/simonw/datasette/issues/1555,997266100,IC_kwDOBm6k_c47cRK0,9599,simonw,2021-12-18T18:40:02Z,2021-12-18T18:40:02Z,OWNER,The implementation of `cursor.executemany()` looks very efficient - it turns into a call to this C function with `multiple` set to `1`: https://github.com/python/cpython/blob/e002bbc6cce637171fb2b1391ffeca8643a13843/Modules/_sqlite/cursor.c#L468-L469,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997262475,https://api.github.com/repos/simonw/datasette/issues/1555,997262475,IC_kwDOBm6k_c47cQSL,9599,simonw,2021-12-18T18:34:18Z,2021-12-18T18:34:18Z,OWNER," Using `executescript=True` that call now takes 1.89ms to create all of those tables.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1569#issuecomment-997249563,https://api.github.com/repos/simonw/datasette/issues/1569,997249563,IC_kwDOBm6k_c47cNIb,9599,simonw,2021-12-18T18:21:23Z,2021-12-18T18:21:23Z,OWNER,"Goal here is to gain the ability to use `conn.executescript()` and still have it show up in the tracer. https://docs.python.org/3/library/sqlite3.html#sqlite3.Cursor.executescript","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083895395,"db.execute_write(..., executescript=True) parameter", https://github.com/simonw/datasette/issues/1555#issuecomment-997248364,https://api.github.com/repos/simonw/datasette/issues/1555,997248364,IC_kwDOBm6k_c47cM1s,9599,simonw,2021-12-18T18:20:10Z,2021-12-18T18:20:10Z,OWNER,"Idea: teach `execute_write` to accept an optional `executescript=True` parameter, like this: ```diff diff --git a/datasette/database.py b/datasette/database.py index 468e936..1a424f5 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -94,10 +94,14 @@ class Database: f""file:{self.path}{qs}"", uri=True, check_same_thread=False ) - async def execute_write(self, sql, params=None, block=False): + async def execute_write(self, sql, params=None, executescript=False, block=False): + assert not executescript and params, ""Cannot use params with executescript=True"" def _inner(conn): with conn: - return conn.execute(sql, params or []) + if executescript: + return conn.executescript(sql) + else: + return conn.execute(sql, params or []) with trace(""sql"", database=self.name, sql=sql.strip(), params=params): results = await self.execute_write_fn(_inner, block=block) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997245301,https://api.github.com/repos/simonw/datasette/issues/1555,997245301,IC_kwDOBm6k_c47cMF1,9599,simonw,2021-12-18T18:17:04Z,2021-12-18T18:17:04Z,OWNER,"One downside of `conn.executescript()` is that it won't be picked up by the tracing mechanism - in fact nothing that uses `await db.execute_write_fn(fn, block=True)` or `await db.execute_fn(fn, block=True)` gets picked up by tracing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997241969,https://api.github.com/repos/simonw/datasette/issues/1555,997241969,IC_kwDOBm6k_c47cLRx,9599,simonw,2021-12-18T18:13:04Z,2021-12-18T18:13:04Z,OWNER,Also: running all of those `CREATE TABLE IF NOT EXISTS` in a single call to `conn.executescript()` rather than as separate queries may speed things up too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997241645,https://api.github.com/repos/simonw/datasette/issues/1555,997241645,IC_kwDOBm6k_c47cLMt,9599,simonw,2021-12-18T18:12:26Z,2021-12-18T18:12:26Z,OWNER,"A simpler optimization would be just to turn all of those column and index reads into a single efficient UNION query against each database, then figure out the most efficient pattern to send them all as writes in one go as opposed to calling `.execute_write()` in a loop.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1566#issuecomment-997235388,https://api.github.com/repos/simonw/datasette/issues/1566,997235388,IC_kwDOBm6k_c47cJq8,9599,simonw,2021-12-18T17:32:07Z,2021-12-18T17:32:07Z,OWNER,I can release a new version of `datasette-leaflet-freedraw` as soon as this is out.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083669410,Release Datasette 0.60, https://github.com/simonw/datasette/issues/1555#issuecomment-997235086,https://api.github.com/repos/simonw/datasette/issues/1555,997235086,IC_kwDOBm6k_c47cJmO,9599,simonw,2021-12-18T17:30:13Z,2021-12-18T17:30:13Z,OWNER,"Now that trace sees write queries (#1568) it's clear that there is a whole lot more DB activity then I had realized: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997234858,https://api.github.com/repos/simonw/datasette/issues/1555,997234858,IC_kwDOBm6k_c47cJiq,9599,simonw,2021-12-18T17:28:44Z,2021-12-18T17:28:44Z,OWNER,"Maybe it would be worth exploring attaching each DB in turn to the _internal connection in order to perform these queries faster. I'm a bit worried about leaks though: the internal database isn't meant to be visible, even temporarily attaching another DB to it could cause SQL queries against that DB to be able to access the internal data. So maybe instead the _internal connection gets to connect to the other DBs? There's a maximum of ten there I think, which is good for most but not all cases. But the cases with the most connected databases will see the worst performance!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1568#issuecomment-997153253,https://api.github.com/repos/simonw/datasette/issues/1568,997153253,IC_kwDOBm6k_c47b1nl,9599,simonw,2021-12-18T06:20:23Z,2021-12-18T06:20:23Z,OWNER,Now running at https://latest-with-plugins.datasette.io/github/commits?_trace=1,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083726550,Trace should show queries on the write connection too, https://github.com/simonw/datasette/issues/1568#issuecomment-997128950,https://api.github.com/repos/simonw/datasette/issues/1568,997128950,IC_kwDOBm6k_c47bvr2,9599,simonw,2021-12-18T02:38:01Z,2021-12-18T02:38:01Z,OWNER,"Prototype: ```diff diff --git a/datasette/database.py b/datasette/database.py index 0a0c104..468e936 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -99,7 +99,9 @@ class Database: with conn: return conn.execute(sql, params or []) - return await self.execute_write_fn(_inner, block=block) + with trace(""sql"", database=self.name, sql=sql.strip(), params=params): + results = await self.execute_write_fn(_inner, block=block) + return results async def execute_write_fn(self, fn, block=False): task_id = uuid.uuid5(uuid.NAMESPACE_DNS, ""datasette.io"") ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083726550,Trace should show queries on the write connection too, https://github.com/simonw/datasette/issues/1555#issuecomment-997128508,https://api.github.com/repos/simonw/datasette/issues/1555,997128508,IC_kwDOBm6k_c47bvk8,9599,simonw,2021-12-18T02:33:57Z,2021-12-18T02:33:57Z,OWNER,"Here's why - `trace` only applies to read, not write SQL operations: https://github.com/simonw/datasette/blob/7c8f8aa209e4ba7bf83976f8495d67c28fbfca24/datasette/database.py#L209-L211","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997128368,https://api.github.com/repos/simonw/datasette/issues/1555,997128368,IC_kwDOBm6k_c47bviw,9599,simonw,2021-12-18T02:32:43Z,2021-12-18T02:32:43Z,OWNER,I wonder why the `INSERT INTO` queries don't show up in that `?trace=1` view?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997128251,https://api.github.com/repos/simonw/datasette/issues/1555,997128251,IC_kwDOBm6k_c47bvg7,9599,simonw,2021-12-18T02:31:51Z,2021-12-18T02:31:51Z,OWNER,"I was thinking it might even be possible to convert this into a `insert into tables select from ...` query: https://github.com/simonw/datasette/blob/c00f29affcafce8314366852ba1a0f5a7dd25690/datasette/utils/internal_db.py#L102-L112 But the `SELECT` runs against a separate database from the `INSERT INTO`, so I would have to setup a cross-database connection for this which feels a little too complicated.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997128080,https://api.github.com/repos/simonw/datasette/issues/1555,997128080,IC_kwDOBm6k_c47bveQ,9599,simonw,2021-12-18T02:30:19Z,2021-12-18T02:30:19Z,OWNER,"I think all of these queries happen in one place - in the `populate_schema_tables()` function - so optimizing them might be localized to just that area of the code, which would be nice: https://github.com/simonw/datasette/blob/c00f29affcafce8314366852ba1a0f5a7dd25690/datasette/utils/internal_db.py#L97-L183","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1561#issuecomment-997127784,https://api.github.com/repos/simonw/datasette/issues/1561,997127784,IC_kwDOBm6k_c47bvZo,9599,simonw,2021-12-18T02:27:56Z,2021-12-18T02:27:56Z,OWNER,"Oh that's an interesting solution, combining the hashes of all of the individual databases. I'm actually not a big fan of `hashed_url` mode - I implemented it right at the start of the project because it felt like a clever hack, and then ended up making it not-the-default a few years ago: - #418 - #419 - #421 I've since not found myself wanting to use it at all for any of my projects - which makes me nervous, because it means there's a pretty complex feature that I'm not using at all, so it's only really protected by the existing unit tests for it. What I'd really like to do is figure out how to have hashed URL mode work entirely as a plugin - then I could extract it from Datasette core entirely (which would simplify a bunch of stuff) but people who find the optimization useful would be able to access it. I'm not sure that the existing plugin hooks are robust enough to do that yet though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082765654,"add hash id to ""_memory"" url if hashed url mode is turned on and crossdb is also turned on", https://github.com/simonw/datasette/issues/1563#issuecomment-997127084,https://api.github.com/repos/simonw/datasette/issues/1563,997127084,IC_kwDOBm6k_c47bvOs,9599,simonw,2021-12-18T02:22:30Z,2021-12-18T02:22:30Z,OWNER,Docs here: https://docs.datasette.io/en/latest/internals.html#datasette-class,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083573206,Datasette(... files=) should not be a required argument, https://github.com/simonw/datasette/issues/1563#issuecomment-997125191,https://api.github.com/repos/simonw/datasette/issues/1563,997125191,IC_kwDOBm6k_c47buxH,9599,simonw,2021-12-18T02:10:20Z,2021-12-18T02:10:20Z,OWNER,I should document the usage of this constructor in https://docs.datasette.io/en/stable/internals.html#datasette-class,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083573206,Datasette(... files=) should not be a required argument, https://github.com/simonw/datasette/issues/1546#issuecomment-997124280,https://api.github.com/repos/simonw/datasette/issues/1546,997124280,IC_kwDOBm6k_c47bui4,9599,simonw,2021-12-18T02:05:16Z,2021-12-18T02:05:16Z,OWNER,"Sure - there are actually several levels to this. The code that creates connections to the database is this: https://github.com/simonw/datasette/blob/83bacfa9452babe7bd66e3579e23af988d00f6ac/datasette/database.py#L72-L95 For files on disk, it does this: ```python # For read-only connections conn = sqlite3.connect( ""file:my.db?mode=ro"", uri=True, check_same_thread=False) # For connections that should be treated as immutable: conn = sqlite3.connect( ""file:my.db?immutable=1"", uri=True, check_same_thread=False) ``` For in-memory databases it runs this after the connection has been created: ```python conn.execute(""PRAGMA query_only=1"") ``` SQLite `PRAGMA` queries are treated as dangerous: someone could run `PRAGMA query_only=0` to turn that previous option off for example. So this function runs against any incoming SQL to verify that it looks like a `SELECT ...` and doesn't have anything like that in it. https://github.com/simonw/datasette/blob/83bacfa9452babe7bd66e3579e23af988d00f6ac/datasette/utils/__init__.py#L195-L204 You can see the tests for that here: https://github.com/simonw/datasette/blob/b1fed48a95516ae84c0f020582303ab50ab817e2/tests/test_utils.py#L136-L170","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1076057610,validating the sql, https://github.com/simonw/datasette/issues/1564#issuecomment-997122938,https://api.github.com/repos/simonw/datasette/issues/1564,997122938,IC_kwDOBm6k_c47buN6,9599,simonw,2021-12-18T01:55:25Z,2021-12-18T01:55:46Z,OWNER,"Made this change while working on this issue: - #1567 I'm going to write a test for this that uses that `sleep()` SQL function from c35b84a2aabe2f14aeacf6cda4110ae1e94d6059.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083581011,_prepare_connection not called on write connections, https://github.com/simonw/datasette/issues/1565#issuecomment-997121215,https://api.github.com/repos/simonw/datasette/issues/1565,997121215,IC_kwDOBm6k_c47bty_,9599,simonw,2021-12-18T01:45:44Z,2021-12-18T01:45:44Z,OWNER,I want to get this into Datasette 0.60 - #1566 - it's a small change that can unlock a lot of potential.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins, https://github.com/simonw/datasette/issues/621#issuecomment-997120723,https://api.github.com/repos/simonw/datasette/issues/621,997120723,IC_kwDOBm6k_c47btrT,9599,simonw,2021-12-18T01:42:33Z,2021-12-18T01:42:33Z,OWNER,I refactored this code out into the `filters.py` module in aa7f0037a46eb76ae6fe9bf2a1f616c58738ecdf,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520681725,Syntax for ?_through= that works as a form field, https://github.com/simonw/datasette/issues/617#issuecomment-552253893,https://api.github.com/repos/simonw/datasette/issues/617,552253893,MDEyOklzc3VlQ29tbWVudDU1MjI1Mzg5Mw==,9599,simonw,2019-11-11T00:46:42Z,2021-12-18T01:41:47Z,OWNER,"As noted in https://github.com/simonw/datasette/issues/621#issuecomment-552253208 a common pattern in this method is blocks of code that append new items to the `where_clauses`, `params` and `extra_human_descriptions` arrays. This is a useful refactoring opportunity. Code that fits this pattern: * The code that builds based on the filters: `where_clauses, params = filters.build_where_clauses(table)` and `human_description_en = filters.human_description_en(extra=extra_human_descriptions)` * Code that handles `?_where=`: `where_clauses.extend(request.args[""_where""])` - though note that this also appends to a `extra_wheres_for_ui` array which nothing else uses * The `_through=` code, see #621 for details * The code that deals with `?_search=` FTS The keyset pagination code modifies `where_clauses` and `params` too, but I don't think it's quite going to work with the same abstraction that would cover the above examples. [UPDATE December 2021 - this comment became the basis for a new `filters_from_request` plugin hook, see also #473]","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",519613116,Refactor TableView.data() method, https://github.com/simonw/datasette/issues/1518#issuecomment-981153060,https://api.github.com/repos/simonw/datasette/issues/1518,981153060,IC_kwDOBm6k_c46ezUk,9599,simonw,2021-11-28T21:13:09Z,2021-12-17T23:37:08Z,OWNER,"Two new requirements inspired by work on the `datasette-table` (and `datasette-notebook`) projects: - #1533 - #1534","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-997082845,https://api.github.com/repos/simonw/datasette/issues/1518,997082845,IC_kwDOBm6k_c47bkbd,9599,simonw,2021-12-17T23:10:09Z,2021-12-17T23:10:17Z,OWNER,These changes so far are now in the 0.60a0 alpha: https://github.com/simonw/datasette/releases/tag/0.60a0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/pull/1559#issuecomment-997082676,https://api.github.com/repos/simonw/datasette/issues/1559,997082676,IC_kwDOBm6k_c47bkY0,9599,simonw,2021-12-17T23:09:41Z,2021-12-17T23:09:41Z,OWNER,This is now available to try out in Datasette 0.60a0: https://github.com/simonw/datasette/releases/tag/0.60a0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082743068,"filters_from_request plugin hook, now used in TableView", https://github.com/simonw/datasette/pull/1562#issuecomment-997082189,https://api.github.com/repos/simonw/datasette/issues/1562,997082189,IC_kwDOBm6k_c47bkRN,9599,simonw,2021-12-17T23:08:14Z,2021-12-17T23:08:14Z,OWNER,"Oh that makes sense: In Python 3.6 this happens: ``` Collecting janus<1.1,>=0.6.2 Using cached janus-0.7.0-py3-none-any.whl (6.9 kB) ``` While in Python 3.7 or higher this happens: ``` Collecting janus<1.1,>=0.6.2 Downloading janus-1.0.0-py3-none-any.whl (6.9 kB) ``` So this is safe to apply because `pip` is smart enough to pick the version of Janus that works for that Python version.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083246400,"Update janus requirement from <0.8,>=0.6.2 to >=0.6.2,<1.1", https://github.com/simonw/datasette/pull/1562#issuecomment-997081673,https://api.github.com/repos/simonw/datasette/issues/1562,997081673,IC_kwDOBm6k_c47bkJJ,9599,simonw,2021-12-17T23:06:38Z,2021-12-17T23:06:38Z,OWNER,"From this diff between `0.7.0` and `1.0`: https://github.com/aio-libs/janus/compare/v0.7.0...v1.0.0 It looks like the only change relevant to compatibility is `loop = asyncio.get_running_loop()` directly instead of falling back to `asyncio.get_event_loop()` if `get_running_loop` isn't available.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083246400,"Update janus requirement from <0.8,>=0.6.2 to >=0.6.2,<1.1", https://github.com/simonw/datasette/pull/1562#issuecomment-997080352,https://api.github.com/repos/simonw/datasette/issues/1562,997080352,IC_kwDOBm6k_c47bj0g,9599,simonw,2021-12-17T23:03:08Z,2021-12-17T23:03:08Z,OWNER,"They say they've dropped 3.6 support, but Datasette's tests against 3.6 are still passing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083246400,"Update janus requirement from <0.8,>=0.6.2 to >=0.6.2,<1.1", https://github.com/simonw/datasette/issues/1566#issuecomment-997078812,https://api.github.com/repos/simonw/datasette/issues/1566,997078812,IC_kwDOBm6k_c47bjcc,9599,simonw,2021-12-17T22:58:55Z,2021-12-17T22:58:55Z,OWNER,The release notes for the 0.60a0 alpha will be useful here: https://github.com/simonw/datasette/releases/tag/0.60a0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083669410,Release Datasette 0.60, https://github.com/simonw/datasette/issues/1565#issuecomment-997077410,https://api.github.com/repos/simonw/datasette/issues/1565,997077410,IC_kwDOBm6k_c47bjGi,9599,simonw,2021-12-17T22:54:45Z,2021-12-17T22:54:45Z,OWNER,"The table page should expose the query both with and without the `limit` clause. The above gave me back: ```sql select id, ACCESS_TYP, UNIT_ID, UNIT_NAME, SUID_NMA, AGNCY_ID, AGNCY_NAME, AGNCY_LEV, AGNCY_TYP, AGNCY_WEB, LAYER, MNG_AG_ID, MNG_AGENCY, MNG_AG_LEV, MNG_AG_TYP, PARK_URL, COUNTY, ACRES, LABEL_NAME, YR_EST, DES_TP, GAP_STS, geometry from CPAD_2020a_Units where ""AGNCY_LEV"" = :p0 order by id limit 101 ``` But I actually wanted to run a `fetch()` against a version of that without the `order by id limit 101` bit (I wanted to figure out the `Extent()` of the `geometry` column) - so I need something like `datasette.table_sql_no_order_no_limit`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins, https://github.com/simonw/datasette/issues/1565#issuecomment-997069128,https://api.github.com/repos/simonw/datasette/issues/1565,997069128,IC_kwDOBm6k_c47bhFI,9599,simonw,2021-12-17T22:31:18Z,2021-12-17T22:31:18Z,OWNER,This should aim to be as consistent as possible with the various arguments to hooks on https://docs.datasette.io/en/stable/plugin_hooks.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins, https://github.com/simonw/datasette/pull/1559#issuecomment-996961196,https://api.github.com/repos/simonw/datasette/issues/1559,996961196,IC_kwDOBm6k_c47bGus,9599,simonw,2021-12-17T19:00:53Z,2021-12-17T19:00:53Z,OWNER,"I'm going to merge this to `main` now. I can continue the refactoring there, but having it in `main` means I can put out an alpha release with the new hook which will unblock me from running tests against it in this repo: https://github.com/simonw/datasette-leaflet-freedraw/pull/8","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082743068,"filters_from_request plugin hook, now used in TableView", https://github.com/simonw/datasette/pull/1559#issuecomment-996959325,https://api.github.com/repos/simonw/datasette/issues/1559,996959325,IC_kwDOBm6k_c47bGRd,9599,simonw,2021-12-17T18:59:54Z,2021-12-17T18:59:54Z,OWNER,I've convinced myself that this plugin hook design is good through this `datasette-leaflet-freedraw` prototype: https://github.com/simonw/datasette-leaflet-freedraw/blob/e8a16a0fe90656b8d655c02881d23a2b9833281d/datasette_leaflet_freedraw/__init__.py,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082743068,"filters_from_request plugin hook, now used in TableView", https://github.com/simonw/datasette/issues/473#issuecomment-996958442,https://api.github.com/repos/simonw/datasette/issues/473,996958442,IC_kwDOBm6k_c47bGDq,9599,simonw,2021-12-17T18:59:27Z,2021-12-17T18:59:27Z,OWNER,I'm happy with how the prototype that used this plugin in `datasette-leaflet-freedraw` turned out: https://github.com/simonw/datasette-leaflet-freedraw/blob/e8a16a0fe90656b8d655c02881d23a2b9833281d/datasette_leaflet_freedraw/__init__.py,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",445850934,Plugin hook: filters_from_request, https://github.com/simonw/datasette/issues/473#issuecomment-996345233,https://api.github.com/repos/simonw/datasette/issues/473,996345233,IC_kwDOBm6k_c47YwWR,9599,simonw,2021-12-17T01:20:31Z,2021-12-17T18:13:01Z,OWNER,I could use this hook to add table filtering on a map to the existing `datasette-leaflet-freedraw` plugin.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",445850934,Plugin hook: filters_from_request, https://github.com/simonw/datasette/pull/1559#issuecomment-996895423,https://api.github.com/repos/simonw/datasette/issues/1559,996895423,IC_kwDOBm6k_c47a2q_,9599,simonw,2021-12-17T17:28:44Z,2021-12-17T17:28:44Z,OWNER,"Before I land this I'm going to build one prototype plugin against it to confirm that the new hook is useful in its current shape. I'll add support for filtering a table by drawing on a map to https://datasette.io/plugins/datasette-leaflet-freedraw","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082743068,"filters_from_request plugin hook, now used in TableView", https://github.com/simonw/datasette/pull/1204#issuecomment-996488925,https://api.github.com/repos/simonw/datasette/issues/1204,996488925,IC_kwDOBm6k_c47ZTbd,9599,simonw,2021-12-17T07:10:48Z,2021-12-17T07:10:48Z,OWNER,I think this is missing the `_macro.html` template file but I have that in my Dropbox.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",793002853,WIP: Plugin includes, https://github.com/simonw/datasette/issues/473#issuecomment-996484551,https://api.github.com/repos/simonw/datasette/issues/473,996484551,IC_kwDOBm6k_c47ZSXH,9599,simonw,2021-12-17T07:02:21Z,2021-12-17T07:04:23Z,OWNER,"The one slightly weird thing about this hook is how it adds `extra_context` without an obvious way for plugins to add extra HTML to the templates based on that context. Maybe I need the proposed mechanism from - #1191 Which has an in-progress PR: - #1204","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",445850934,Plugin hook: filters_from_request, https://github.com/simonw/datasette/issues/1191#issuecomment-761104933,https://api.github.com/repos/simonw/datasette/issues/1191,761104933,MDEyOklzc3VlQ29tbWVudDc2MTEwNDkzMw==,9599,simonw,2021-01-15T18:21:26Z,2021-12-17T07:03:02Z,OWNER,"Also related: #857 (comprehensive documentation of variables available to templates) - since then the plugin hook could be fed the full template context and use that to do its thing. Or maybe the plugin hooks gets to return the name of a template that should be `{% include %}` into the page at that point? But the plugin may want to add extra context that is available to that template include.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",787098345,Ability for plugins to collaborate when adding extra HTML to blocks in default templates, https://github.com/simonw/datasette/pull/1559#issuecomment-996286808,https://api.github.com/repos/simonw/datasette/issues/1559,996286808,IC_kwDOBm6k_c47YiFY,9599,simonw,2021-12-17T00:01:43Z,2021-12-17T00:01:43Z,OWNER,"This already has tests and documentation, and I've used it to refactor out the logic for `?_where=` and `?_search=` and `?_through=`. Do I like this enough to land it on `main`? Also, I think I can still use it to refactor out the `Filters` code that implements `?col=x` and `?col__lt=5` and suchlike.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082743068,"filters_from_request plugin hook, now used in TableView", https://github.com/simonw/datasette/issues/473#issuecomment-996286199,https://api.github.com/repos/simonw/datasette/issues/473,996286199,IC_kwDOBm6k_c47Yh73,9599,simonw,2021-12-17T00:00:22Z,2021-12-17T00:00:22Z,OWNER,Documentation for that hook in the PR branch: https://github.com/simonw/datasette/blob/54e9b3972f277431a001e685f78e5dd6403a6d8d/docs/plugin_hooks.rst#filters_from_requestrequest-database-table-datasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",445850934,Plugin hook: filters_from_request, https://github.com/simonw/datasette/issues/1518#issuecomment-996286104,https://api.github.com/repos/simonw/datasette/issues/1518,996286104,IC_kwDOBm6k_c47Yh6Y,9599,simonw,2021-12-17T00:00:07Z,2021-12-17T00:00:07Z,OWNER,Documentation of the new hook in the PR: https://github.com/simonw/datasette/blob/54e9b3972f277431a001e685f78e5dd6403a6d8d/docs/plugin_hooks.rst#filters_from_requestrequest-database-table-datasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/473#issuecomment-996275108,https://api.github.com/repos/simonw/datasette/issues/473,996275108,IC_kwDOBm6k_c47YfOk,9599,simonw,2021-12-16T23:32:22Z,2021-12-16T23:32:30Z,OWNER,This filter design can only influence the `where` component of the SQL clause - it's not able to modify the `SELECT` columns or adjust the `ORDER BY` or `OFFSET LIMIT` parts. I think that's OK.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",445850934,Plugin hook: filters_from_request, https://github.com/simonw/datasette/issues/1518#issuecomment-996272906,https://api.github.com/repos/simonw/datasette/issues/1518,996272906,IC_kwDOBm6k_c47YesK,9599,simonw,2021-12-16T23:27:42Z,2021-12-16T23:27:42Z,OWNER,Got a TIL out of this: https://til.simonwillison.net/pluggy/multiple-hooks-same-file,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/473#issuecomment-996267817,https://api.github.com/repos/simonw/datasette/issues/473,996267817,IC_kwDOBm6k_c47Ydcp,9599,simonw,2021-12-16T23:17:52Z,2021-12-16T23:19:00Z,OWNER,"I revisited this idea in #1518 and came up with a slightly different name and design for the hook: ```python @hookspec def filters_from_request(request, database, table, datasette): """""" Return FilterArguments( where_clauses=[str, str, str], params={}, human_descriptions=[str, str, str], extra_context={} ) based on the request"""""" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",445850934,Plugin hook: filters_from_request, https://github.com/simonw/datasette/issues/1518#issuecomment-996264617,https://api.github.com/repos/simonw/datasette/issues/1518,996264617,IC_kwDOBm6k_c47Ycqp,9599,simonw,2021-12-16T23:11:12Z,2021-12-16T23:11:12Z,OWNER,I managed to extract both `_search=` and `_where=` out using a prototype of that hook. I wonder if it could extract the complex code for `?_next` too?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-996250585,https://api.github.com/repos/simonw/datasette/issues/1518,996250585,IC_kwDOBm6k_c47YZPZ,9599,simonw,2021-12-16T22:43:37Z,2021-12-16T22:45:07Z,OWNER,"Ran into a problem prototyping that hook up for handling `?_where=` - that feature also adds a little bit of extra template context in order to show the interface for removing wheres - the `extra_wheres_for_ui` variable: https://github.com/simonw/datasette/blob/0663d5525cc41e9260ac7d1f6386d3a6eb5ad2a9/datasette/views/table.py#L457-L463 Maybe change to this? ```python class FilterArguments(NamedTuple): where_clauses: List[str] params: Dict[str, Union[str, int, float]] human_descriptions: List[str] extra_context: Dict[str, Any] ``` That might be necessary for `_search` too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-996248713,https://api.github.com/repos/simonw/datasette/issues/1518,996248713,IC_kwDOBm6k_c47YYyJ,9599,simonw,2021-12-16T22:39:47Z,2021-12-16T22:39:47Z,OWNER,"The hook could return a named tuple like this one: ```python from typing import NamedTuple, List, Optional, Union, Dict class FilterArguments(NamedTuple): where_clauses: List[str] params: Dict[str, Union[str, int, float]] human_descriptions: List[str] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-996240802,https://api.github.com/repos/simonw/datasette/issues/1518,996240802,IC_kwDOBm6k_c47YW2i,9599,simonw,2021-12-16T22:25:00Z,2021-12-16T22:36:04Z,OWNER,"I think that plugin hook would get given the `request` object (and `datasette` and the name of the database and table) and returns a list of SQL fragments, a dictionary of lookup arguments and a list of human-description fragments - or an awaitable. `filters_from_request(request, database, table, datasette)` perhaps? (Similar in name to `actor_from_request`). ```python @hookspec def filters_from_request(request, database, table, datasette): """"""Return (where_clauses, params_dict, human_descriptions) based on the request"""""" ``` Turns out that's pretty much exactly what I implemented in 5116c4ec8aed5091e1f75415424b80f613518dc6 for #473: ```python @hookspec def table_filter(): ""Custom filtering of the current table based on the request"" ``` ```python TableFilter = namedtuple(""TableFilter"", ( ""human_description_extras"", ""where_clauses"", ""params"") ) ``` ```python # filter_arguments plugin hook support for awaitable_fn in pm.hook.table_filter(): extras = await awaitable_fn( view=self, name=name, table=table, request=request ) human_description_extras.extend(extras.human_description_extras) where_clauses.extend(extras.where_clauses) params.update(extras.params) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/sqlite-utils/issues/358#issuecomment-996232461,https://api.github.com/repos/simonw/sqlite-utils/issues/358,996232461,IC_kwDOCGYnMM47YU0N,9599,simonw,2021-12-16T22:10:39Z,2021-12-16T22:10:39Z,OWNER,"This goes beyond the `transform()` method - the curious methods that create new SQL tables could benefit from the ability to add `CHECK` constraints too. I haven't used these myself, do you have any `CREATE TABLE` examples that use them that you can share?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082651698,Support for CHECK constraints, https://github.com/simonw/datasette/issues/1518#issuecomment-996227713,https://api.github.com/repos/simonw/datasette/issues/1518,996227713,IC_kwDOBm6k_c47YTqB,9599,simonw,2021-12-16T22:02:35Z,2021-12-16T22:03:55Z,OWNER,"Is there an opportunity to refactor things using a new plugin hook here? Maybe the `register_filters` hook from #473, where the hook becomes responsible for building where clauses (and human descriptions of them) based on the incoming query string. That version dealt with `Filter` classes, but those might be a bit too low-level for this. `?_spatial_within=GEOJSON` was an interesting idea attached to that issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-996225889,https://api.github.com/repos/simonw/datasette/issues/1518,996225889,IC_kwDOBm6k_c47YTNh,9599,simonw,2021-12-16T21:59:32Z,2021-12-16T22:00:42Z,OWNER,I added a ton of comments to the `data()` method which really helps get a better feel for how this all works: https://github.com/simonw/datasette/blob/0663d5525cc41e9260ac7d1f6386d3a6eb5ad2a9/datasette/views/table.py#L322,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-996225235,https://api.github.com/repos/simonw/datasette/issues/1518,996225235,IC_kwDOBm6k_c47YTDT,9599,simonw,2021-12-16T21:58:24Z,2021-12-16T21:58:41Z,OWNER,"A fundamental operation of this view is to construct the SQL query and accompanying human description based on the incoming query string parameters. The human description is the bit at the top of https://latest.datasette.io/fixtures/searchable?_search=dog&_sort=pk&_facet=text2&text2=sara+weasel that says: > 1 row where search matches ""dog"" and text2 = ""sara weasel"" sorted by pk (Also used in the page ``). The code actually gathers three things: - Fragments of the `where` clause, for example ` ""text2"" = :p0` - Parameters, e.g. `{""p0"": ""sara weasel""}` - Human description components, e.g. `text2 = ""sara weasel""` Some operations such as `?_where=` don't currently provide an extra human description component. `_where=` also doesn't populate a parameter, but maybe it could? Would be neat if in the future `?_where=foo+=+:bar` worked and added a `bar` input field to the screen, as seen with custom queries.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-996219117,https://api.github.com/repos/simonw/datasette/issues/1518,996219117,IC_kwDOBm6k_c47YRjt,9599,simonw,2021-12-16T21:47:51Z,2021-12-16T21:49:24Z,OWNER,"Should facets really not be displayed on pages past page one (where `?_next=` is set)? That made sense to me at the time, but I'm now having second thoughts about it. I guess it's a useful performance tweak for when crawlers keep hitting the `?_next=` link. Actually it looks like facets DO display on subsequent pages, e.g. on https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_next=200 - but facet suggestions do not, thanks to this code: https://github.com/simonw/datasette/blob/2c07327d23d9c5cf939ada9ba4091c1b8b2ba42d/datasette/views/table.py#L777-L785 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1558#issuecomment-996204369,https://api.github.com/repos/simonw/datasette/issues/1558,996204369,IC_kwDOBm6k_c47YN9R,9599,simonw,2021-12-16T21:23:25Z,2021-12-16T21:23:25Z,OWNER,"Related: Following the fix for #625 I noticed that `facets_timed_out` gives you just the column name, but doesn't let you know which particular type of facet (`date` or `array` for example) suffered the timeout: https://github.com/simonw/datasette/blob/0d4145d0f4d8b2a7edc1ba4aac1be56cd536a10a/datasette/facets.py#L269-L270 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082584499,Redesign `facet_results` JSON structure prior to Datasette 1.0, https://github.com/simonw/sqlite-utils/issues/357#issuecomment-996179930,https://api.github.com/repos/simonw/sqlite-utils/issues/357,996179930,IC_kwDOCGYnMM47YH_a,9599,simonw,2021-12-16T20:43:19Z,2021-12-16T20:43:19Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079422215,pytest-runner is not required, https://github.com/simonw/datasette/issues/625#issuecomment-996170510,https://api.github.com/repos/simonw/datasette/issues/625,996170510,IC_kwDOBm6k_c47YFsO,9599,simonw,2021-12-16T20:27:41Z,2021-12-16T20:27:41Z,OWNER,"And here's the new JSON: https://latest.datasette.io/fixtures/facetable.json?_facet=created&_facet_date=created&_facet=tags&_facet_array=tags&_nosuggest=1 ``` { ""database"": ""fixtures"", ""table"": ""facetable"", ""is_view"": false, ""human_description_en"": """", ... ""facet_results"": { ""created"": { ""name"": ""created"", ""type"": ""column"", ... }, ""tags"": { ""name"": ""tags"", ""type"": ""column"", ... }, ""created_2"": { ""name"": ""created"", ""type"": ""date"", ... }, ""tags_2"": { ""name"": ""tags"", ""type"": ""array"", ... } } } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/625#issuecomment-996165659,https://api.github.com/repos/simonw/datasette/issues/625,996165659,IC_kwDOBm6k_c47YEgb,9599,simonw,2021-12-16T20:19:53Z,2021-12-16T20:19:53Z,OWNER,Demo of the fix: https://latest.datasette.io/fixtures/facetable?_facet=created&_facet_date=created&_facet=tags&_facet_array=tags#facet-tags,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/625#issuecomment-996161380,https://api.github.com/repos/simonw/datasette/issues/625,996161380,IC_kwDOBm6k_c47YDdk,9599,simonw,2021-12-16T20:13:05Z,2021-12-16T20:13:05Z,OWNER,I updated the example code in the facet plugin hook documentation: https://github.com/simonw/datasette/blob/95d0dd7a1cf6be6b7da41e1404184217eb93f64a/docs/plugin_hooks.rst#register_facet_classes,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/625#issuecomment-996152213,https://api.github.com/repos/simonw/datasette/issues/625,996152213,IC_kwDOBm6k_c47YBOV,9599,simonw,2021-12-16T19:59:46Z,2021-12-16T20:00:05Z,OWNER,"Since no-one is using that plugin hook I'm going to alter its contract slightly. I'll still keep the existing JSON format working though (until 1.0), since it's much more likely that people are using that JSON somewhere.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/830#issuecomment-996151246,https://api.github.com/repos/simonw/datasette/issues/830,996151246,IC_kwDOBm6k_c47YA_O,9599,simonw,2021-12-16T19:58:22Z,2021-12-16T19:58:22Z,OWNER,"As of today, 16 December 2021, I'm still not seeing any evidence that anyone is using this hook (yet) according to GitHub code search: https://cs.github.com/?scopeName=All+repos&scope=&q=register_facet_classes%20-repo%3Asimonw%2Fdatasette","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636511683,Redesign register_facet_classes plugin hook, https://github.com/simonw/datasette/issues/625#issuecomment-996150904,https://api.github.com/repos/simonw/datasette/issues/625,996150904,IC_kwDOBm6k_c47YA54,9599,simonw,2021-12-16T19:57:52Z,2021-12-16T19:57:52Z,OWNER,Good news - GitHub's new code search doesn't show ANYONE using that plugin hook - not surprising since it has that documentation warning plus it's just not a very clearly usable hook: https://cs.github.com/?scopeName=All+repos&scope=&q=register_facet_classes%20-repo%3Asimonw%2Fdatasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/625#issuecomment-996149720,https://api.github.com/repos/simonw/datasette/issues/625,996149720,IC_kwDOBm6k_c47YAnY,9599,simonw,2021-12-16T19:56:14Z,2021-12-16T19:56:14Z,OWNER,"This bad design is even covered in the plugin hooks documentation: https://docs.datasette.io/en/0.59.4/plugin_hooks.html#register-facet-classes It does at least have the following warning: > **Warning** > > The design of this plugin hook is unstable and may change. See [issue 830](https://github.com/simonw/datasette/issues/830).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/625#issuecomment-996146762,https://api.github.com/repos/simonw/datasette/issues/625,996146762,IC_kwDOBm6k_c47X_5K,9599,simonw,2021-12-16T19:51:44Z,2021-12-16T19:51:44Z,OWNER,"Here's where `facet_results` is built up: https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/views/table.py#L752-L758 So the decision to key things based on column name is actually embedded deep in the existing facet classes here: https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/facets.py#L224-L226 https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/facets.py#L395-L397 https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/facets.py#L510-L512","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/1558#issuecomment-996134716,https://api.github.com/repos/simonw/datasette/issues/1558,996134716,IC_kwDOBm6k_c47X888,9599,simonw,2021-12-16T19:46:21Z,2021-12-16T19:46:21Z,OWNER,"The flaw in the current design is illustrated by this example: ``` ""facet_results"": { ""tags"": { ""name"": ""tags"", ""type"": ""array"", ""results"": [...], ""hideable"": false, ""toggle_url"": ""/fixtures/facetable.json?_facet=tags&_trace=1&_nosuggest=1"", ""truncated"": false }, ""created"": { ""name"": ""created"", ""type"": ""date"", ""results"": [...] ``` This was the cause of the bug in #625 - the each of those objects is keyed by the name of the column, which left no room for faceting the same column once by date and once by column value.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082584499,Redesign `facet_results` JSON structure prior to Datasette 1.0, https://github.com/simonw/datasette/issues/625#issuecomment-996130862,https://api.github.com/repos/simonw/datasette/issues/625,996130862,IC_kwDOBm6k_c47X8Au,9599,simonw,2021-12-16T19:44:48Z,2021-12-16T19:44:48Z,OWNER,"Decision: as an initial fix I'm going to de-duplicate those keys by using `tags__array` etc - with a `_2` on the end if that key is already used. I'll open a separate issue to redesign this better for Datasette 1.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/625#issuecomment-996121736,https://api.github.com/repos/simonw/datasette/issues/625,996121736,IC_kwDOBm6k_c47X5yI,9599,simonw,2021-12-16T19:37:08Z,2021-12-16T19:37:08Z,OWNER,"Really `facet_results` here should be an array of objects, not an object that maps poorly designed string keys to those objects.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/625#issuecomment-996119954,https://api.github.com/repos/simonw/datasette/issues/625,996119954,IC_kwDOBm6k_c47X5WS,9599,simonw,2021-12-16T19:36:01Z,2021-12-16T19:36:11Z,OWNER,"Datasette's own HTML rendering code doesn't actually use the keys in `facet_results` - it instead loops through `sorted_facet_results` which is defined like this: https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/views/table.py#L937-L941 And used like this: https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/templates/table.html#L154-L156","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/625#issuecomment-996118401,https://api.github.com/repos/simonw/datasette/issues/625,996118401,IC_kwDOBm6k_c47X4-B,9599,simonw,2021-12-16T19:34:28Z,2021-12-16T19:34:55Z,OWNER,"The big question here is do I break any existing clients of the `""facet_results""` JSON API? It's still pre-1.0 so I could break them, but I've also built my own code against this in the past so it's likely other people have too. If I don't break them, I will instead need to come up with a naming convention for those keys - something like `""tags__array""` for example. As well as a way to ensure that a column called `tags__array` doesn't end up conflicting with the `tags__array` key!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/1557#issuecomment-996115949,https://api.github.com/repos/simonw/datasette/issues/1557,996115949,IC_kwDOBm6k_c47X4Xt,9599,simonw,2021-12-16T19:30:55Z,2021-12-16T19:30:55Z,OWNER,"Demo: compare https://latest.datasette.io/fixtures/facetable?_facet=_city_id&_nosuggest=1 to https://latest.datasette.io/fixtures/facetable?_facet=_city_id Documentation: bottom of https://docs.datasette.io/en/latest/json_api.html#special-table-arguments","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082564912,`?_nosuggest=1` parameter for disabling facet suggestions on table view, https://github.com/simonw/datasette/issues/1556#issuecomment-996104214,https://api.github.com/repos/simonw/datasette/issues/1556,996104214,IC_kwDOBm6k_c47X1gW,9599,simonw,2021-12-16T19:15:00Z,2021-12-16T19:15:28Z,OWNER,"Demo: https://latest.datasette.io/fixtures/facetable?_facet=planet_int&_facet=_city_id&_facet=created#facet-created <img width=""756"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/146434231-a7e1800b-eeef-4e5b-83e2-3b466c84b51b.png""> ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1081318247,"Show count of facet values always, not just for `?_facet_size=max`", https://github.com/simonw/datasette/issues/1553#issuecomment-996103956,https://api.github.com/repos/simonw/datasette/issues/1553,996103956,IC_kwDOBm6k_c47X1cU,9599,simonw,2021-12-16T19:14:38Z,2021-12-16T19:14:38Z,OWNER,This is a really interesting idea - kind of similar to how many APIs include custom HTTP headers informing of rate-limits.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1079111498,if csv export is truncated in non streaming mode set informative response header, https://github.com/simonw/datasette/issues/625#issuecomment-996100774,https://api.github.com/repos/simonw/datasette/issues/625,996100774,IC_kwDOBm6k_c47X0qm,9599,simonw,2021-12-16T19:10:01Z,2021-12-16T19:10:48Z,OWNER,"I think the problem here may be in the design of the JSON returned by facets. It looks like this: ``` ""facet_results"": { ""tags"": { ""name"": ""tags"", ""type"": ""array"", ""results"": [...], ""hideable"": false, ""toggle_url"": ""/fixtures/facetable.json?_facet=tags&_trace=1&_nosuggest=1"", ""truncated"": false }, ""created"": { ""name"": ""created"", ""type"": ""date"", ""results"": [...] ``` The problem then is that the `tags` key is over-ridden by the second facet with a different type against the same column name! https://latest-with-plugins.datasette.io/fixtures/facetable?_trace=1&_facet=created&_facet_date=created&_facet_array=tags&_facet=tags confirms that the SQL queries for those facets are being executed - but the final JSON doesn't show them on https://latest-with-plugins.datasette.io/fixtures/facetable.json?_trace=1&_facet=created&_facet_date=created&_facet_array=tags&_facet=tags They're not available in the template context either: https://latest-with-plugins.datasette.io/fixtures/facetable?_facet=created&_facet_date=created&_facet_array=tags&_facet=tags&_context=1","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/625#issuecomment-996093884,https://api.github.com/repos/simonw/datasette/issues/625,996093884,IC_kwDOBm6k_c47Xy-8,9599,simonw,2021-12-16T19:00:28Z,2021-12-16T19:00:28Z,OWNER,Implementing #1552 has made a fix for this bug even more important.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/1552#issuecomment-996084899,https://api.github.com/repos/simonw/datasette/issues/1552,996084899,IC_kwDOBm6k_c47Xwyj,9599,simonw,2021-12-16T18:48:14Z,2021-12-16T18:48:14Z,OWNER,Updated documentation: https://github.com/simonw/datasette/blob/20a2ed6bec367d2f6759be4a879364a72780b59d/docs/facets.rst#facets-in-metadatajson,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`), https://github.com/simonw/datasette/issues/1552#issuecomment-996077053,https://api.github.com/repos/simonw/datasette/issues/1552,996077053,IC_kwDOBm6k_c47Xu39,9599,simonw,2021-12-16T18:36:41Z,2021-12-16T18:36:41Z,OWNER,"... actually no, I WILL document this, because not documenting this is what got us to this point in the first place!","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 1, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`), https://github.com/simonw/datasette/issues/1552#issuecomment-996076373,https://api.github.com/repos/simonw/datasette/issues/1552,996076373,IC_kwDOBm6k_c47XutV,9599,simonw,2021-12-16T18:35:40Z,2021-12-16T18:35:40Z,OWNER,"I'm going to ship your fix now, but I'm not going to add this to the documentation yet because I hope to improve the design prior to Datasette 1.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`), https://github.com/simonw/datasette/issues/1552#issuecomment-996046304,https://api.github.com/repos/simonw/datasette/issues/1552,996046304,IC_kwDOBm6k_c47XnXg,9599,simonw,2021-12-16T17:53:40Z,2021-12-16T18:16:12Z,OWNER,"I'm also not convinced that this configuration syntax is right. It's a bit weird having a `""facets""` list that can either by column-name-strings or `{""type-of-facet"": ""column-name""}` objects. Maybe there's a better design for this? Part of the problem here is that facets were designed to accept optional extra configuration - partly to support `m2m` facets in #495 - but I haven't actually shipped any facets that use that ability. Facet by delimiter would be a good one to exercise that ability: - #510","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`), https://github.com/simonw/datasette/issues/1552#issuecomment-996045776,https://api.github.com/repos/simonw/datasette/issues/1552,996045776,IC_kwDOBm6k_c47XnPQ,9599,simonw,2021-12-16T17:52:54Z,2021-12-16T17:52:54Z,OWNER,"I tried that fix you suggested and now this `metadata.json` does the right thing: ```json { ""databases"": { ""fixtures"": { ""tables"": { ""facetable"": { ""facets"": [ { ""array"": ""tags"" } ] } } } } } ``` It does further highlight the bug in #625 though - since then if you try to add `?_facet=tags` to facet by tags treating them NOT as an array your request to do so is ignored.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`), https://github.com/simonw/datasette/issues/1552#issuecomment-996034408,https://api.github.com/repos/simonw/datasette/issues/1552,996034408,IC_kwDOBm6k_c47Xkdo,9599,simonw,2021-12-16T17:37:37Z,2021-12-16T17:37:37Z,OWNER,"I think you're right! I had completely forgotten that piece of code. This just turned into a bug fix and a documentation update. Thanks for the research!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`), https://github.com/simonw/datasette/issues/262#issuecomment-995034911,https://api.github.com/repos/simonw/datasette/issues/262,995034911,IC_kwDOBm6k_c47Twcf,9599,simonw,2021-12-15T18:03:46Z,2021-12-15T18:03:56Z,OWNER,"This is relevant to the big refactor in: - #1518","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,Add ?_extra= mechanism for requesting extra properties in JSON, https://github.com/simonw/datasette/issues/1552#issuecomment-995034143,https://api.github.com/repos/simonw/datasette/issues/1552,995034143,IC_kwDOBm6k_c47TwQf,9599,simonw,2021-12-15T18:02:53Z,2021-12-15T18:02:53Z,OWNER,"This is definitely a missing feature. The ""different types of facet"" stuff feels incomplete to me generally - this is one issue, but this one as well: - #625","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`), https://github.com/simonw/datasette/issues/1423#issuecomment-995023410,https://api.github.com/repos/simonw/datasette/issues/1423,995023410,IC_kwDOBm6k_c47Ttoy,9599,simonw,2021-12-15T17:48:40Z,2021-12-15T17:48:40Z,OWNER,You've caused me to rethink this feature - I no longer think there's value in only showing these numbers if `?_facet_size=max` as opposed to all of the time. New issue coming up.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 1, ""eyes"": 0}",962391325,Show count of facet values if ?_facet_size=max, https://github.com/simonw/datasette/issues/1542#issuecomment-995022217,https://api.github.com/repos/simonw/datasette/issues/1542,995022217,IC_kwDOBm6k_c47TtWJ,9599,simonw,2021-12-15T17:47:07Z,2021-12-15T17:47:07Z,OWNER,"This does make sense to me. I've been hoping to significantly improve the way JavaScript plugins work - there are some notes on that here: - #983 Encouraging plugins such as `datasette-cluster-map` to emit events that can then be listened to by other plugins is a really interesting idea that I hadn't considered.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1072106103,feature request: order and dependency of plugins (that use js), https://github.com/simonw/datasette/issues/1518#issuecomment-994085710,https://api.github.com/repos/simonw/datasette/issues/1518,994085710,IC_kwDOBm6k_c47QItO,9599,simonw,2021-12-14T22:03:16Z,2021-12-14T22:04:28Z,OWNER,"There are actually four forms of SQL query used by the table page: - `from_sql` - just the `from table_name where ...` - `sql_no_order_no_limit` - used for faceting, `""select {select_all_columns} from {table_name} {where}""` - `sql` - the above but with order and limit clauses: `""select {select_specified_columns} from {table_name} {where}{order_by} limit {page_size}{offset}""` - `count_sql` used for the count, built out of `from_sql`: `""select count(*) {from_sql}""` I'm tempted to encapsulate those in a `Query` class.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-994042389,https://api.github.com/repos/simonw/datasette/issues/1518,994042389,IC_kwDOBm6k_c47P-IV,9599,simonw,2021-12-14T21:35:53Z,2021-12-14T21:35:53Z,OWNER,"Maybe a better way to approach this would be to focus on the JSON side of things - try to get a basic JSON version with `?_extra=` support working, then eventually build that up to the point where it can power the HTML version.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/621#issuecomment-994005634,https://api.github.com/repos/simonw/datasette/issues/621,994005634,IC_kwDOBm6k_c47P1KC,9599,simonw,2021-12-14T21:02:50Z,2021-12-14T21:02:50Z,OWNER,"This would also mean that an extra text input box could be easily shown on the page. https://latest-with-plugins.datasette.io/fixtures/roadside_attractions?_through={""table"":""roadside_attraction_characteristics"",""column"":""characteristic_id"",""value"":""1""} but with the annotated box added (and made to look good): <img width=""737"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/146078853-d66e34e0-73c9-4c35-9414-5aff3bf825ee.png""> ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520681725,Syntax for ?_through= that works as a form field, https://github.com/simonw/datasette/issues/621#issuecomment-993958242,https://api.github.com/repos/simonw/datasette/issues/621,993958242,IC_kwDOBm6k_c47Ppli,9599,simonw,2021-12-14T20:33:25Z,2021-12-14T20:33:56Z,OWNER,"Alternative idea: since current syntax is: `?_through={""table"":""roadside_attraction_characteristics"",""column"":""characteristic_id"",""value"":""1""}` The form-encoding-friendly syntax could be: `?_through.{""table"":""roadside_attraction_characteristics"",""column"":""characteristic_id""}=1` Which is more consistent than the array proposal: `?_through.[""roadside_attraction_characteristics"",""characteristic_id""]=1`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520681725,Syntax for ?_through= that works as a form field, https://github.com/simonw/datasette/issues/621#issuecomment-993813210,https://api.github.com/repos/simonw/datasette/issues/621,993813210,IC_kwDOBm6k_c47PGLa,9599,simonw,2021-12-14T17:30:13Z,2021-12-14T20:23:57Z,OWNER,"Might be able to create a web form that's unambiguous using: `https://latest.datasette.io/fixtures/roadside_attractions?_through.[""roadside_attraction_characteristics"",""characteristic_id""]=1` So: ```html <input type=""text"" name=""_through.["roadside_attraction_characteristics","characteristic_id"]' value=""1""> ``` I'm pretty confident this is allowed by the HTML specification. This works: ```html <form action=""https://httpbin.org/get""> <input type=""text"" name='_through.["roadside_attraction_characteristics","characteristic_id"]' value=""1""> <input type=""submit""> </form> ``` ASGI parsing seems to work too: https://latest-with-plugins.datasette.io/-/asgi-scope?_through.[%22roadside_attraction_characteristics%22%2C%22characteristic_id%22]=1","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520681725,Syntax for ?_through= that works as a form field, https://github.com/simonw/datasette/issues/1518#issuecomment-993794247,https://api.github.com/repos/simonw/datasette/issues/1518,993794247,IC_kwDOBm6k_c47PBjH,9599,simonw,2021-12-14T17:09:40Z,2021-12-14T17:09:40Z,OWNER,- `table_actions` should be an extra.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-993000787,https://api.github.com/repos/simonw/datasette/issues/1518,993000787,IC_kwDOBm6k_c47L_1T,9599,simonw,2021-12-13T23:19:20Z,2021-12-14T17:06:05Z,OWNER,"Useful old comment here: https://github.com/simonw/datasette/issues/617#issuecomment-552253893 > As noted in [#621 (comment)](https://github.com/simonw/datasette/issues/621#issuecomment-552253208) a common pattern in this method is blocks of code that append new items to the `where_clauses`, `params` and `extra_human_descriptions` arrays. This is a useful refactoring opportunity. > > Code that fits this pattern: > > * The code that builds based on the filters: `where_clauses, params = filters.build_where_clauses(table)` and `human_description_en = filters.human_description_en(extra=extra_human_descriptions)` > * Code that handles `?_where=`: `where_clauses.extend(request.args[""_where""])` - though note that this also appends to a `extra_wheres_for_ui` array which nothing else uses > * The `_through=` code, see [Syntax for ?_through= that works as a form field #621](https://github.com/simonw/datasette/issues/621) for details > * The code that deals with `?_search=` FTS > > The keyset pagination code modifies `where_clauses` and `params` too, but I don't think it's quite going to work with the same abstraction that would cover the above examples. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/pull/1554#issuecomment-993006521,https://api.github.com/repos/simonw/datasette/issues/1554,993006521,IC_kwDOBm6k_c47MBO5,9599,simonw,2021-12-13T23:28:47Z,2021-12-13T23:28:47Z,OWNER,"That's frustrating: you can only attach comments to lines that were changed in the PR or are within about 3-4 lines of them: ![comments](https://user-images.githubusercontent.com/9599/145905357-5d8873f5-99c9-4b46-b4d5-35d38f5cb686.gif) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079129258,TableView refactor, https://github.com/simonw/datasette/issues/1518#issuecomment-992833868,https://api.github.com/repos/simonw/datasette/issues/1518,992833868,IC_kwDOBm6k_c47LXFM,9599,simonw,2021-12-13T19:59:17Z,2021-12-13T19:59:17Z,OWNER,"Built a new plugin to help with this work by improving the display of `?_trace=1` output: https://datasette.io/plugins/datasette-pretty-traces ![image](https://user-images.githubusercontent.com/9599/145879751-36621f43-ba68-4ccd-b14b-379ed8f2111a.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-991978789,https://api.github.com/repos/simonw/datasette/issues/1518,991978789,IC_kwDOBm6k_c47IGUl,9599,simonw,2021-12-12T22:04:19Z,2021-12-12T22:04:19Z,OWNER,Idea: in JSON output include a `warnings` block listing any _ parameters that were not recognized.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1551#issuecomment-991960719,https://api.github.com/repos/simonw/datasette/issues/1551,991960719,IC_kwDOBm6k_c47IB6P,9599,simonw,2021-12-12T19:58:17Z,2021-12-12T19:58:17Z,OWNER,"Here's an example of the difference that causes: ```pycon >>> import urllib.parse >>> urllib.parse.parse_qs(""foo=bar"") {'foo': ['bar']} >>> urllib.parse.parse_qs(""foo=bar&baz="") {'foo': ['bar']} >>> urllib.parse.parse_qs(""foo=bar&baz="", keep_blank_values=True) {'foo': ['bar'], 'baz': ['']} ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077893013,`keep_blank_values=True` when parsing `request.args`, https://github.com/simonw/datasette/issues/1551#issuecomment-991960416,https://api.github.com/repos/simonw/datasette/issues/1551,991960416,IC_kwDOBm6k_c47IB1g,9599,simonw,2021-12-12T19:56:12Z,2021-12-12T19:56:12Z,OWNER,"Python documentation for `parse_qs`: https://docs.python.org/3/library/urllib.parse.html#urllib.parse.parse_qs > The optional argument *keep_blank_values* is a flag indicating whether blank values in percent-encoded queries should be treated as blank strings. A true value indicates that blanks should be retained as blank strings. The default false value indicates that blank values are to be ignored and treated as if they were not included.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077893013,`keep_blank_values=True` when parsing `request.args`, https://github.com/simonw/datasette/issues/1551#issuecomment-991960179,https://api.github.com/repos/simonw/datasette/issues/1551,991960179,IC_kwDOBm6k_c47IBxz,9599,simonw,2021-12-12T19:54:45Z,2021-12-12T19:54:45Z,OWNER,This is technically a backwards-incompatible for any plugins that use `request.args` - but it's unlikely to break anything. At any rate this needs to happen before Datasette 1.0!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077893013,`keep_blank_values=True` when parsing `request.args`, https://github.com/simonw/datasette/issues/1518#issuecomment-991828014,https://api.github.com/repos/simonw/datasette/issues/1518,991828014,IC_kwDOBm6k_c47Hhgu,9599,simonw,2021-12-12T03:21:35Z,2021-12-12T03:21:35Z,OWNER,"No, removing that gave me the following test failure: ``` tests/test_table_api.py::test_table_filter_queries[/fixtures/simple_primary_key.json?content__exact=-expected_rows2] FAILED [100%] =============================================================================== FAILURES ================================================================================ ______________________________________ test_table_filter_queries[/fixtures/simple_primary_key.json?content__exact=-expected_rows2] ______________________________________ app_client = <datasette.utils.testing.TestClient object at 0x10d45d2d0>, path = '/fixtures/simple_primary_key.json?content__exact=', expected_rows = [['3', '']] @pytest.mark.parametrize( ""path,expected_rows"", [ (""/fixtures/simple_primary_key.json?content=hello"", [[""1"", ""hello""]]), ( ""/fixtures/simple_primary_key.json?content__contains=o"", [ [""1"", ""hello""], [""2"", ""world""], [""4"", ""RENDER_CELL_DEMO""], ], ), (""/fixtures/simple_primary_key.json?content__exact="", [[""3"", """"]]), ( ""/fixtures/simple_primary_key.json?content__not=world"", [ [""1"", ""hello""], [""3"", """"], [""4"", ""RENDER_CELL_DEMO""], [""5"", ""RENDER_CELL_ASYNC""], ], ), ], ) def test_table_filter_queries(app_client, path, expected_rows): response = app_client.get(path) > assert expected_rows == response.json[""rows""] E AssertionError: assert [['3', '']] == [['1', 'hello'],\n ['2', 'world'],\n ['3', ''],\n ['4', 'RENDER_CELL_DEMO'],\n ['5', 'RENDER_CELL_ASYNC']] E At index 0 diff: ['3', ''] != ['1', 'hello'] E Right contains 4 more items, first extra item: ['2', 'world'] E Full diff: E [ E - ['1', E - 'hello'], E - ['2', E - 'world'], E ['3', E ''], E - ['4', E - 'RENDER_CELL_DEMO'], E - ['5', E - 'RENDER_CELL_ASYNC'], E ] /Users/simon/Dropbox/Development/datasette/tests/test_table_api.py:511: AssertionError ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-991827468,https://api.github.com/repos/simonw/datasette/issues/1518,991827468,IC_kwDOBm6k_c47HhYM,9599,simonw,2021-12-12T03:15:00Z,2021-12-12T03:15:00Z,OWNER," I don't think this code is necessary any more: https://github.com/simonw/datasette/blob/492f9835aa7e90540dd0c6324282b109f73df71b/datasette/views/table.py#L396-L399 That dates back from when Datasette was built on top of Sanic and Sanic didn't preserve those query parameters the way I needed it to: https://github.com/simonw/datasette/blob/1f69269fe93e4cd42e56890126cc0dbcf719c6cb/datasette/views/table.py#L202-L206","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-991823001,https://api.github.com/repos/simonw/datasette/issues/1518,991823001,IC_kwDOBm6k_c47HgSZ,9599,simonw,2021-12-12T02:25:32Z,2021-12-12T02:25:32Z,OWNER,The tests for `TableView` are currently mixed in with everything else in `tests/test_api.py` and `tests/html.py` - might be good to split those out into `test_table_html.py` and `test_table_api.py` since they're such a key part of how Datasette works.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-991822853,https://api.github.com/repos/simonw/datasette/issues/1518,991822853,IC_kwDOBm6k_c47HgQF,9599,simonw,2021-12-12T02:24:00Z,2021-12-12T02:24:00Z,OWNER,Rebuilding `TableView` from the ground up is proving not to be much fun. I'm going to explore starting the refactor of the existing code by separating out the bit that generates the SQL query from the rest of it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-991819781,https://api.github.com/repos/simonw/datasette/issues/1518,991819781,IC_kwDOBm6k_c47HfgF,9599,simonw,2021-12-12T01:53:10Z,2021-12-12T01:53:10Z,OWNER,"I have a hunch that the conclusion of this experiment may end up being that the `asyncinject` trick is kinda neat but the code will be easier to maintain (while still executing in parallel) if it's written using `asyncio.gather` directly instead. It's possible `asyncinject` will end up being neat enough that I'll want to keep it though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1550#issuecomment-991805516,https://api.github.com/repos/simonw/datasette/issues/1550,991805516,IC_kwDOBm6k_c47HcBM,9599,simonw,2021-12-11T23:43:24Z,2021-12-11T23:43:24Z,OWNER,"I built a tiny Starlette app to experiment with this a bit: ```python import asyncio import janus from starlette.applications import Starlette from starlette.responses import JSONResponse, HTMLResponse, StreamingResponse from starlette.routing import Route import sqlite3 from concurrent import futures executor = futures.ThreadPoolExecutor(max_workers=10) async def homepage(request): return HTMLResponse( """""" <html> <head><title>SQL CSV Server

SQL CSV Server

"""""" ) def run_query_in_thread(sql, sync_q): db = sqlite3.connect(""../datasette/covid.db"") cursor = db.cursor() cursor.arraysize = 100 # Default is 1 apparently? cursor.execute(sql) columns = [d[0] for d in cursor.description] sync_q.put([columns]) # Now start putting batches of rows while True: rows = cursor.fetchmany() if rows: sync_q.put(rows) else: break # Let queue know we are finished\ sync_q.put(None) async def csv_query(request): sql = request.query_params[""sql""] queue = janus.Queue() loop = asyncio.get_running_loop() async def csv_generator(): loop.run_in_executor(None, run_query_in_thread, sql, queue.sync_q) while True: rows = await queue.async_q.get() if rows is not None: for row in rows: yield "","".join(map(str, row)) + ""\n "" queue.async_q.task_done() else: # Cleanup queue.close() await queue.wait_closed() break return StreamingResponse(csv_generator(), media_type='text/plain') app = Starlette( debug=True, routes=[ Route(""/"", homepage), Route(""/csv"", csv_query), ], ) ``` But.. if I run this in a terminal window: ``` /tmp % wget 'http://127.0.0.1:8000/csv?sql=select+*+from+ny_times_us_counties' ``` it takes about 20 seconds to run and returns a 50MB file - but while it is running no other requests can be served by that server - not even the homepage! So something is blocking the event loop. Maybe I should be using `fut = loop.run_in_executor(None, run_query_in_thread, sql, queue.sync_q)` and then awaiting `fut` somewhere, like in the Janus documentation? Don't think that's needed though. Needs more work to figure out why this is blocking.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077628073,Research option for returning all rows from arbitrary query, https://github.com/simonw/datasette/issues/1550#issuecomment-991761635,https://api.github.com/repos/simonw/datasette/issues/1550,991761635,IC_kwDOBm6k_c47HRTj,9599,simonw,2021-12-11T19:39:01Z,2021-12-11T19:39:01Z,OWNER,"I wonder if this could work for public instances too with some kind of queuing mechanism? I really need to use benchmarking to figure out what the right number of maximum SQLite connections is. I'm just guessing at the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077628073,Research option for returning all rows from arbitrary query, https://github.com/simonw/datasette/issues/1549#issuecomment-991755245,https://api.github.com/repos/simonw/datasette/issues/1549,991755245,IC_kwDOBm6k_c47HPvt,9599,simonw,2021-12-11T19:17:54Z,2021-12-11T19:17:54Z,OWNER,"Also relevant: - #1062 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077620955,Redesign CSV export to improve usability, https://github.com/simonw/datasette/issues/617#issuecomment-991755013,https://api.github.com/repos/simonw/datasette/issues/617,991755013,IC_kwDOBm6k_c47HPsF,9599,simonw,2021-12-11T19:17:11Z,2021-12-11T19:17:11Z,OWNER,This work is now happening in #1518 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",519613116,Refactor TableView.data() method, https://github.com/simonw/datasette/issues/1549#issuecomment-991754794,https://api.github.com/repos/simonw/datasette/issues/1549,991754794,IC_kwDOBm6k_c47HPoq,9599,simonw,2021-12-11T19:16:33Z,2021-12-11T19:16:33Z,OWNER,Good call! I'm doing a refactor #1518 right now which will hopefully bring the functionality of those two much closer - I'll make a note to consider this there too.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077620955,Redesign CSV export to improve usability, https://github.com/simonw/datasette/issues/1549#issuecomment-991752486,https://api.github.com/repos/simonw/datasette/issues/1549,991752486,IC_kwDOBm6k_c47HPEm,9599,simonw,2021-12-11T19:09:15Z,2021-12-11T19:09:15Z,OWNER,"That's what this option does: ![EAB1B9E8-38E9-4C6D-8854-BD1935F163D9](https://user-images.githubusercontent.com/9599/145688531-668bafa1-e287-4bbd-84d6-157241fb1f68.jpeg) The usability of this is pretty terrible though (including ""stream all rows"" - how are people meant to understand what that does?) so it can definitely do with some rethinking.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077620955,Redesign CSV export to improve usability, https://github.com/simonw/sqlite-utils/issues/356#issuecomment-991517209,https://api.github.com/repos/simonw/sqlite-utils/issues/356,991517209,IC_kwDOCGYnMM47GVoZ,9599,simonw,2021-12-11T07:46:41Z,2021-12-11T07:46:41Z,OWNER,"By default this will accept single lines, but maybe there could be a `--all` option which instead grabs all of stdin into a single string against which the conversion function runs - like `git-history file`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option, https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991400016,https://api.github.com/repos/simonw/sqlite-utils/issues/353,991400016,IC_kwDOCGYnMM47F5BQ,9599,simonw,2021-12-11T01:10:52Z,2021-12-11T01:11:02Z,OWNER,"This won't be in a release for a little while, but you can install it to try it out using: pip install https://github.com/simonw/sqlite-utils/archive/ee13f98c2c.zip","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077102934,"Allow passing a file of code to ""sqlite-utils convert""", https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991399782,https://api.github.com/repos/simonw/sqlite-utils/issues/353,991399782,IC_kwDOCGYnMM47F49m,9599,simonw,2021-12-11T01:09:37Z,2021-12-11T01:09:37Z,OWNER,"OK, this is implemented. Updated documentation is here: https://sqlite-utils.datasette.io/en/latest/cli.html#converting-data-in-columns","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077102934,"Allow passing a file of code to ""sqlite-utils convert""", https://github.com/simonw/sqlite-utils/issues/354#issuecomment-991399604,https://api.github.com/repos/simonw/sqlite-utils/issues/354,991399604,IC_kwDOCGYnMM47F460,9599,simonw,2021-12-11T01:08:46Z,2021-12-11T01:08:46Z,OWNER,That passed!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077243232,Test failure in test_rebuild_fts, https://github.com/simonw/sqlite-utils/issues/354#issuecomment-991398367,https://api.github.com/repos/simonw/sqlite-utils/issues/354,991398367,IC_kwDOCGYnMM47F4nf,9599,simonw,2021-12-11T01:03:14Z,2021-12-11T01:03:14Z,OWNER,The new test: https://github.com/simonw/sqlite-utils/blob/ee13f98c2c7ca3b819bd0fc55da3108cb6a6434a/tests/test_fts.py#L270-L277,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077243232,Test failure in test_rebuild_fts, https://github.com/simonw/sqlite-utils/pull/347#issuecomment-991397907,https://api.github.com/repos/simonw/sqlite-utils/issues/347,991397907,IC_kwDOCGYnMM47F4gT,9599,simonw,2021-12-11T01:01:40Z,2021-12-11T01:01:40Z,OWNER,The change I made to that test in #354 might help with this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066603133,Test against pysqlite3 running SQLite 3.37, https://github.com/simonw/sqlite-utils/issues/354#issuecomment-991395919,https://api.github.com/repos/simonw/sqlite-utils/issues/354,991395919,IC_kwDOCGYnMM47F4BP,9599,simonw,2021-12-11T00:52:31Z,2021-12-11T00:52:31Z,OWNER,"It turns out `rebuild` does indeed work against content tables, so I can put that in the test instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077243232,Test failure in test_rebuild_fts, https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991395494,https://api.github.com/repos/simonw/sqlite-utils/issues/355,991395494,IC_kwDOCGYnMM47F36m,9599,simonw,2021-12-11T00:50:22Z,2021-12-11T00:51:15Z,OWNER,"Here's an example of the new (slightly confusing) error message: ```bash sqlite-utils convert fixtures.db roadside_attractions name ' def foo(value) bar baz ' Error: Syntax error in code: def foo(value) invalid syntax ``` Another: ``` sqlite-utils convert fixtures.db roadside_attractions name '$' Error: Syntax error in code: return $ invalid syntax ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077322009,Allow users to pass a full convert() function definition, https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991393684,https://api.github.com/repos/simonw/sqlite-utils/issues/355,991393684,IC_kwDOCGYnMM47F3eU,9599,simonw,2021-12-11T00:42:19Z,2021-12-11T00:49:49Z,OWNER,"Ideally I'd like to show the perfect syntax error messages to the user - but I don't know if it's possible to do this cleanly because the error might occur with their originally entered code OR it might occur after I add `def fn(value)` to it. I'm going to punt on that for the moment and tolerate slightly confusing syntax errors.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077322009,Allow users to pass a full convert() function definition, https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991386841,https://api.github.com/repos/simonw/sqlite-utils/issues/355,991386841,IC_kwDOCGYnMM47F1zZ,9599,simonw,2021-12-11T00:14:11Z,2021-12-11T00:15:15Z,OWNER,"Relevant code: https://github.com/simonw/sqlite-utils/blob/7a43af232e4bc00bd227307665163614e225948b/sqlite_utils/cli.py#L2128-L2135 One way to implement this would be to look to see if the code starts with `def ...` - but that's not going to work for proper module that start with a docstring or imports.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077322009,Allow users to pass a full convert() function definition, https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991387044,https://api.github.com/repos/simonw/sqlite-utils/issues/355,991387044,IC_kwDOCGYnMM47F12k,9599,simonw,2021-12-11T00:14:45Z,2021-12-11T00:14:45Z,OWNER,"Maybe attempt to compile their code, and if it fails try again after adding `def fn(value):` to the start?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077322009,Allow users to pass a full convert() function definition, https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991381679,https://api.github.com/repos/simonw/sqlite-utils/issues/353,991381679,IC_kwDOCGYnMM47F0iv,9599,simonw,2021-12-10T23:58:43Z,2021-12-10T23:59:35Z,OWNER,"I think the fix for this is to change the rules about what code is accepted in both the `-` mode and the literal code string mode: you can pass in a Python expression, OR a fragment that gets turned into a function, OR code that implements its own `def convert(value)` function. So this would work too: ```sh sqlite-utils convert my.db mytable col1 ' def convert(value): return value.upper() ' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077102934,"Allow passing a file of code to ""sqlite-utils convert""", https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991381281,https://api.github.com/repos/simonw/sqlite-utils/issues/353,991381281,IC_kwDOCGYnMM47F0ch,9599,simonw,2021-12-10T23:57:26Z,2021-12-10T23:57:26Z,OWNER,"My first attempt at building this looked a little bit strange, because you would end up having a file like this `convert.py`: ``` value = value.upper() return value ``` Which gets used like this: cat convert.py | sqlite-utils convert my.db mytable col1 - But... that `convert.py` code isn't actually valid Python - it's a weird thing where you have a partial snippet of Python code that gets wrapped in a function automatically. It would be better if you could write `convert.py` as a valid Python file with a function in it, something like this: ```python def convert(value): value = value.upper() return value ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077102934,"Allow passing a file of code to ""sqlite-utils convert""", https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991378346,https://api.github.com/repos/simonw/sqlite-utils/issues/353,991378346,IC_kwDOCGYnMM47Fzuq,9599,simonw,2021-12-10T23:48:28Z,2021-12-10T23:48:28Z,OWNER,"One option: allow `CODE` to be a special value of `-` which means ""read from standard input"". It's a tiny bit of a hack but I think it would work here. If you wanted to replace a column entirely with hyphens you would still be able to do this: sqlite-utils convert my.db mytable col1 '""-""'","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077102934,"Allow passing a file of code to ""sqlite-utils convert""", https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991377288,https://api.github.com/repos/simonw/sqlite-utils/issues/353,991377288,IC_kwDOCGYnMM47FzeI,9599,simonw,2021-12-10T23:45:53Z,2021-12-10T23:45:53Z,OWNER,"One challenge here: the current signature looks like this: ``` % sqlite-utils convert --help Usage: sqlite-utils convert [OPTIONS] DB_PATH TABLE COLUMNS... CODE ``` `CODE` is a positional argument which comes last - and since `COLUMNS` can be one or more items, making `CODE` optional isn't easy.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077102934,"Allow passing a file of code to ""sqlite-utils convert""", https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991376639,https://api.github.com/repos/simonw/sqlite-utils/issues/353,991376639,IC_kwDOCGYnMM47FzT_,9599,simonw,2021-12-10T23:43:45Z,2021-12-10T23:43:45Z,OWNER,"There's a very non-obvious workaround for this at the moment. You can save your code in e.g. a file called` transform.py` - my test one looks like this: ```python def upper(value): return value.upper() ``` Then you can run the following to import and use that function: `PYTHONPATH=. sqlite-utils convert fixtures.db roadside_attractions name 'transform.upper(value)' --import transform` That `PYTHONPATH=. bit is necessary because otherwise the script won't look in the current directory for that `transform.py` module. Now that I've written this down, it's obviously bad! I think your suggestion here is a good idea.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077102934,"Allow passing a file of code to ""sqlite-utils convert""", https://github.com/simonw/sqlite-utils/issues/354#issuecomment-991309759,https://api.github.com/repos/simonw/sqlite-utils/issues/354,991309759,IC_kwDOCGYnMM47Fi-_,9599,simonw,2021-12-10T21:33:18Z,2021-12-10T21:33:18Z,OWNER,"https://www.sqlite.org/fts5.html#the_rebuild_command says: > This command first deletes the entire full-text index, then rebuilds it based on the contents of the table or [content table](https://www.sqlite.org/fts5.html#external_content_tables). It is not available with [contentless tables](https://www.sqlite.org/fts5.html#contentless_tables). > > `INSERT INTO ft(ft) VALUES('rebuild');`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077243232,Test failure in test_rebuild_fts, https://github.com/simonw/sqlite-utils/issues/354#issuecomment-991309002,https://api.github.com/repos/simonw/sqlite-utils/issues/354,991309002,IC_kwDOCGYnMM47FizK,9599,simonw,2021-12-10T21:32:14Z,2021-12-10T21:32:14Z,OWNER,"Here's what the method does: https://github.com/simonw/sqlite-utils/blob/e328db8eba1fbf29a69eda95dfec861954f9e771/sqlite_utils/db.py#L1941-L1952 Maybe I don't need a test that deliberately corrupts the database here? Not sure how to test that `rebuild` has been called though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077243232,Test failure in test_rebuild_fts, https://github.com/simonw/sqlite-utils/issues/354#issuecomment-991307422,https://api.github.com/repos/simonw/sqlite-utils/issues/354,991307422,IC_kwDOCGYnMM47Fiae,9599,simonw,2021-12-10T21:29:34Z,2021-12-10T21:29:34Z,OWNER,Here's the test in question. The way it works is a bit weird (deleting everything in the `_fts_data` table in order to force errors that can be fixed with `.rebuild_fts()`): https://github.com/simonw/sqlite-utils/blob/8ae77a6961fed94ef2c9cc81fcfc7c81d222d9a2/tests/test_fts.py#L257-L285,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077243232,Test failure in test_rebuild_fts, https://github.com/simonw/sqlite-utils/issues/354#issuecomment-991306712,https://api.github.com/repos/simonw/sqlite-utils/issues/354,991306712,IC_kwDOCGYnMM47FiPY,9599,simonw,2021-12-10T21:28:27Z,2021-12-10T21:28:27Z,OWNER,"Failures started with this commit, which only touches documentation so is completely unrelated: https://github.com/simonw/sqlite-utils/commit/e328db8eba1fbf29a69eda95dfec861954f9e771","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077243232,Test failure in test_rebuild_fts, https://github.com/simonw/datasette/issues/1518#issuecomment-991285527,https://api.github.com/repos/simonw/datasette/issues/1518,991285527,IC_kwDOBm6k_c47FdEX,9599,simonw,2021-12-10T20:52:00Z,2021-12-10T20:52:00Z,OWNER,"If I break this up into `@inject` methods, what methods could I have and what would they do? - `resolve_path`: Use request path to resolve the database and table. Could handle hash URLs too (if I don't manage to extract those to a plugin) - would be nice if this could raise a redirect, but I think that will instead have to be one of the things it returns - `build_sql`: Builds the SQL query based on the querystring (and some DB introspection) - `execute_count`: Execute the `count(*)` - `execute_rows`: Execute the `limit 101` to fetch the rows - `execute_facets`: Execute all requested facets (could this do its own `asyncio.gather()` to run facets in parallel?) - `suggest_facets`: Execute facet suggestions Are there any plugin hooks that would make sense to execute in parallel? Actually there might be: I don't think `extra_template_vars`, `extra_css_urls`, `extra_js_urls`, `extra_body_script` depend on each other so it might be possible to execute them in a parallel chunk (at least any of them that return awaitables).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1540#issuecomment-984801331,https://api.github.com/repos/simonw/datasette/issues/1540,984801331,IC_kwDOBm6k_c46suAz,9599,simonw,2021-12-02T16:42:02Z,2021-12-09T23:38:39Z,OWNER,"I'm going to wrap this up in a plugin for the moment - I want it in Datasette core but I'd like to improve the implementation first with things like support for `base_url` which will likely depend on #1533 or similar. Here's the plugin: https://github.com/simonw/datasette-hovercards","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1068791148,Idea: hover to reveal details of linked row, https://github.com/simonw/datasette/issues/1544#issuecomment-988226938,https://api.github.com/repos/simonw/datasette/issues/1544,988226938,IC_kwDOBm6k_c465yV6,9599,simonw,2021-12-07T20:02:44Z,2021-12-07T20:02:44Z,OWNER,I'm feeling rushed today so I'm going to fix this without adding a test!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1073712378,Code that detects the label column for a table is case-sensitive, https://github.com/simonw/datasette/issues/1544#issuecomment-988226523,https://api.github.com/repos/simonw/datasette/issues/1544,988226523,IC_kwDOBm6k_c465yPb,9599,simonw,2021-12-07T20:02:00Z,2021-12-07T20:02:00Z,OWNER,Here's the code at fault: https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/database.py#L288-L291,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1073712378,Code that detects the label column for a table is case-sensitive, https://github.com/simonw/datasette/issues/1527#issuecomment-988154238,https://api.github.com/repos/simonw/datasette/issues/1527,988154238,IC_kwDOBm6k_c465gl-,9599,simonw,2021-12-07T18:05:26Z,2021-12-07T18:05:26Z,OWNER,"Found a new case of this bug: click the ""Apply"" button on https://latest.datasette.io/fixtures/facetable?_sort=pk&_city_id__gt=1 ![apply-bug](https://user-images.githubusercontent.com/9599/145082760-6947c769-480f-45c7-9916-b6cc7f5834f8.gif) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059555791,Columns starting with an underscore behave poorly in filters, https://github.com/simonw/sqlite-utils/issues/349#issuecomment-987461427,https://api.github.com/repos/simonw/sqlite-utils/issues/349,987461427,IC_kwDOCGYnMM4623cz,9599,simonw,2021-12-07T01:03:43Z,2021-12-07T01:04:37Z,OWNER,"In terms of types, I think that means it looks like this: ```python IndexesType = Iterable[ Union[str, Iterable[str]] ] def create( self, columns: Dict[str, Any], pk: Optional[Any] = None, ... indexes: Optional[IndexesType] = None, ): ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1071531082,A way of creating indexes on newly created tables, https://github.com/simonw/sqlite-utils/issues/349#issuecomment-987458772,https://api.github.com/repos/simonw/sqlite-utils/issues/349,987458772,IC_kwDOCGYnMM4622zU,9599,simonw,2021-12-07T01:00:41Z,2021-12-07T01:00:41Z,OWNER,"I think the syntax design of this looks like: ```python item_pk = db[item_table].lookup( {""_item_id"": item_id}, item_to_insert, column_order=(""_id"", ""_item_id""), pk=""_id"", indexes=(""_version"",), ) ``` So it's a sequence of column names... or a sequence of tuples for creating compound indexes: ```python db[""dogs""].insert( {""name"": ""Cleo"", ""species"": ""Mutt"", ""hobbies"": ""Raiding picnics""}, indexes=((""name"", ""species""), ""hobbies""), ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1071531082,A way of creating indexes on newly created tables, https://github.com/simonw/sqlite-utils/issues/352#issuecomment-987454872,https://api.github.com/repos/simonw/sqlite-utils/issues/352,987454872,IC_kwDOCGYnMM46212Y,9599,simonw,2021-12-07T00:56:29Z,2021-12-07T00:56:29Z,OWNER,"Thought about this due to this issue, which should stay consistent with how the `sqlite-utils` CLI works: - https://github.com/simonw/git-history/issues/41","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1072792507,`sqlite-utils insert --extract colname`, https://github.com/simonw/sqlite-utils/issues/351#issuecomment-987437043,https://api.github.com/repos/simonw/sqlite-utils/issues/351,987437043,IC_kwDOCGYnMM462xfz,9599,simonw,2021-12-07T00:41:02Z,2021-12-07T00:41:56Z,OWNER,"The fix there was: ```diff @@ -463,7 +473,7 @@ def compile_convert(convert, imports): locals = {} globals = {""json"": json} for import_ in imports: - globals[import_] = __import__(import_) + globals[import_.split(""."")[0]] = __import__(import_) exec(code_o, globals, locals) ``` With this example (needs to be modified here): ``` git-history file items.xml --convert ' tree = xml.etree.ElementTree.fromstring(content) return [el.attrib for el in tree.iter(""item"")] ' --import xml.etree.ElementTree ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1072780607,Support `--import xml.etree.ElementTree` in `sqlite-utils convert`, https://github.com/simonw/sqlite-utils/issues/349#issuecomment-987349633,https://api.github.com/repos/simonw/sqlite-utils/issues/349,987349633,IC_kwDOCGYnMM462cKB,9599,simonw,2021-12-06T23:19:28Z,2021-12-06T23:19:28Z,OWNER,(I ended up not needing this here since `.lookup()` already creates a unique index on `_item_id` for you. Still could be a useful feature though.),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1071531082,A way of creating indexes on newly created tables, https://github.com/simonw/sqlite-utils/issues/350#issuecomment-987016019,https://api.github.com/repos/simonw/sqlite-utils/issues/350,987016019,IC_kwDOCGYnMM461KtT,9599,simonw,2021-12-06T17:56:57Z,2021-12-06T17:56:57Z,OWNER,"Would be interesting to micro-benchmark this to get an idea for how much of a performance boost it is, since the indexed SQLite lookups used by `table.lookup()` should be really fast already.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1072435124,Optional caching mechanism for table.lookup(), https://github.com/simonw/sqlite-utils/issues/350#issuecomment-987015327,https://api.github.com/repos/simonw/sqlite-utils/issues/350,987015327,IC_kwDOCGYnMM461Kif,9599,simonw,2021-12-06T17:56:05Z,2021-12-06T17:56:05Z,OWNER,Should I implement this remember to apply the optimization in `git-history`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1072435124,Optional caching mechanism for table.lookup(), https://github.com/simonw/sqlite-utils/issues/350#issuecomment-987015063,https://api.github.com/repos/simonw/sqlite-utils/issues/350,987015063,IC_kwDOCGYnMM461KeX,9599,simonw,2021-12-06T17:55:42Z,2021-12-06T17:55:42Z,OWNER,"API could be this: ```python id = db[""columns""].lookup( {""namespace"": namespace_id, ""name"": column}, cache=True ) ``` This could default to a 100 item LRU cache. You could perhaps modify that with `cache_size=500` or with `cache_size=None` to disable the size limit on that cache.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1072435124,Optional caching mechanism for table.lookup(), https://github.com/simonw/datasette/issues/1541#issuecomment-984908185,https://api.github.com/repos/simonw/datasette/issues/1541,984908185,IC_kwDOBm6k_c46tIGZ,9599,simonw,2021-12-02T18:56:54Z,2021-12-02T18:56:54Z,OWNER,Also it should link to foreign keys like the table page does.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1069881276,Different default layout for row page, https://github.com/simonw/datasette/issues/1540#issuecomment-984053760,https://api.github.com/repos/simonw/datasette/issues/1540,984053760,IC_kwDOBm6k_c46p3gA,9599,simonw,2021-12-01T21:05:20Z,2021-12-01T21:05:20Z,OWNER,"I realized you couldn't click the links any more because the hovercard overlapped them, so I changed it to this instead. Need to reconsider the when-to-hide logic though. ```javascript hovercard.style.top = (ev.pageY + 15) + 'px'; ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1068791148,Idea: hover to reveal details of linked row, https://github.com/simonw/datasette/issues/1540#issuecomment-984051925,https://api.github.com/repos/simonw/datasette/issues/1540,984051925,IC_kwDOBm6k_c46p3DV,9599,simonw,2021-12-01T21:03:16Z,2021-12-01T21:03:16Z,OWNER,Needs `pageX` not `clientX` because otherwise it doesn't work when you scroll down the page.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1068791148,Idea: hover to reveal details of linked row, https://github.com/simonw/datasette/issues/1540#issuecomment-984048965,https://api.github.com/repos/simonw/datasette/issues/1540,984048965,IC_kwDOBm6k_c46p2VF,9599,simonw,2021-12-01T20:59:26Z,2021-12-01T21:02:58Z,OWNER,"This is a bit of a mess but it does keep the hovercard around for a moment and then fade it away when you mouse out of it: ```html+jinja {% extends ""base.html"" %} {% block content %}

Hovercards demo

Here is a link to a row {% endblock %} ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1068791148,Idea: hover to reveal details of linked row, https://github.com/simonw/datasette/issues/1540#issuecomment-984037711,https://api.github.com/repos/simonw/datasette/issues/1540,984037711,IC_kwDOBm6k_c46pzlP,9599,simonw,2021-12-01T20:42:17Z,2021-12-01T20:43:14Z,OWNER,"A first prototype (saved as `templates/pages/hovercard.html` and run with `datasette fixtures.db --template-dir=templates`): ```html+jinja {% extends ""base.html"" %} {% block content %}

Hovercards demo

Here is a link to a row {% endblock %} ``` ![hovercard](https://user-images.githubusercontent.com/9599/144310888-6db71bad-b6f6-4d8a-a737-81a618022bbe.gif) Lots of decisions to make here. Most importantly, when should it be hidden again?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1068791148,Idea: hover to reveal details of linked row, https://github.com/simonw/datasette/issues/1540#issuecomment-983985330,https://api.github.com/repos/simonw/datasette/issues/1540,983985330,IC_kwDOBm6k_c46pmyy,9599,simonw,2021-12-01T19:29:05Z,2021-12-01T19:29:05Z,OWNER,"The layout of the hover card could be similar to the one used by `datasette-cluster-map`: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1068791148,Idea: hover to reveal details of linked row, https://github.com/simonw/sqlite-utils/issues/348#issuecomment-983122733,https://api.github.com/repos/simonw/sqlite-utils/issues/348,983122733,IC_kwDOCGYnMM46mUMt,9599,simonw,2021-11-30T23:31:17Z,2021-11-30T23:31:17Z,OWNER,"Potential other options could include: - `--page-size` https://www.sqlite.org/pragma.html#pragma_page_size","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1067771698,Command for creating an empty database, https://github.com/simonw/sqlite-utils/issues/348#issuecomment-983120066,https://api.github.com/repos/simonw/sqlite-utils/issues/348,983120066,IC_kwDOCGYnMM46mTjC,9599,simonw,2021-11-30T23:25:52Z,2021-11-30T23:26:11Z,OWNER,"Maybe this: sqlite-utils create-database my.db With options that include `--enable-wal` (actually that's the only option I can think of). This is consistent with the existing `create-table` and `create-view` commands. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1067771698,Command for creating an empty database, https://github.com/simonw/datasette/issues/1525#issuecomment-982331602,https://api.github.com/repos/simonw/datasette/issues/1525,982331602,IC_kwDOBm6k_c46jTDS,9599,simonw,2021-11-30T06:39:00Z,2021-11-30T06:39:00Z,OWNER,"These two pages now help demonstrate the fix: - https://latest.datasette.io/fixtures/facet_cities/1 - https://latest.datasette.io/fixtures/attraction_characteristic/2 I added a new test for these here: https://github.com/simonw/datasette/blob/35b12746ba2bf9f254791bddac03d25b19be9b77/tests/test_html.py#L823-L848 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059509927,"""Links from other tables"" broken for columns starting with underscore", https://github.com/simonw/datasette/issues/1532#issuecomment-982319210,https://api.github.com/repos/simonw/datasette/issues/1532,982319210,IC_kwDOBm6k_c46jQBq,9599,simonw,2021-11-30T06:12:19Z,2021-11-30T06:12:19Z,OWNER,That's really cool to hear - I've not seen many people actively building on top of the JSON API.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065429936,Use datasette-table Web Component to guide the design of the JSON API for 1.0, https://github.com/simonw/datasette/issues/1527#issuecomment-982318745,https://api.github.com/repos/simonw/datasette/issues/1527,982318745,IC_kwDOBm6k_c46jP6Z,9599,simonw,2021-11-30T06:11:21Z,2021-11-30T06:11:21Z,OWNER,"Manually tested this too, looks like that fixed it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059555791,Columns starting with an underscore behave poorly in filters, https://github.com/simonw/datasette/issues/1527#issuecomment-982235541,https://api.github.com/repos/simonw/datasette/issues/1527,982235541,IC_kwDOBm6k_c46i7mV,9599,simonw,2021-11-30T02:57:34Z,2021-11-30T02:58:44Z,OWNER,"I started fiddling with a test for this which extracts the `` fields, but I probably won't use it: ```python def test_exact_parameter_results_in_correct_hidden_fields(app_client): # https://github.com/simonw/datasette/issues/1527 response = app_client.get( ""/fixtures/facetable?_facet=_neighborhood&_neighborhood__exact=Downtown"" ) # In this case we should NOT have a hidden _neighborhood__exact=Downtown field form = Soup(response.body, ""html.parser"").find(""form"") selects = [ { ""name"": select[""name""], ""value"": select.select(""option[selected]"")[0].text if select.select(""option[selected]"") else """", } for select in form.findAll(""select"") ] inputs = [input.attrs for input in form.findAll(""input"")] # Turn those both into a {name: (value, type)} array form_inputs = {} form_inputs.update( {select[""name""]: (select[""value""], ""select"") for select in selects} ) form_inputs.update( { input[""name""]: (input.get(""value""), input[""type""]) for input in inputs if input.get(""name"") } ) assert form_inputs == { ""_filter_column_1"": (""_neighborhood"", ""select""), ""_filter_op_1"": (""="", ""select""), ""_filter_value_1"": (""Downtown"", ""text""), ""_filter_column"": ("""", ""select""), ""_filter_op"": ("""", ""select""), ""_filter_value"": (None, ""text""), ""_sort"": (""Sort by pk"", ""select""), ""_sort_by_desc"": (None, ""checkbox""), ""_facet"": (""_neighborhood"", ""hidden""), ""_neighborhood__exact"": (""Downtown"", ""hidden""), } ``` The problem is that last hidden field, `_neighborhood__exact=Downtown` - which should not be there.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059555791,Columns starting with an underscore behave poorly in filters, https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982137888,https://api.github.com/repos/simonw/sqlite-utils/issues/347,982137888,IC_kwDOCGYnMM46ijwg,9599,simonw,2021-11-29T23:50:54Z,2021-11-29T23:50:54Z,OWNER,If I'm going to `skipIf()` those tests I need a way to check if `pysqlite3` is being used.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066603133,Test against pysqlite3 running SQLite 3.37, https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982137293,https://api.github.com/repos/simonw/sqlite-utils/issues/347,982137293,IC_kwDOCGYnMM46ijnN,9599,simonw,2021-11-29T23:49:29Z,2021-11-29T23:49:29Z,OWNER,"A short term fix would be to skip those tests against `pysqlite3` - but longer term it would be good to address the underlying issue, particularly for the WAL ones (the FTS ones aren't too worrying since if you deliberately try and break the FTS table it's not hugely problematic if you corrupt your database).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066603133,Test against pysqlite3 running SQLite 3.37, https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982136747,https://api.github.com/repos/simonw/sqlite-utils/issues/347,982136747,IC_kwDOCGYnMM46ijer,9599,simonw,2021-11-29T23:48:05Z,2021-11-29T23:48:05Z,OWNER,"Some interesting test failures in the version that runs with `pysqlite3`: ``` =========================== short test summary info ============================ FAILED tests/test_cli.py::test_enable_wal - assert 0 == 1 FAILED tests/test_cli.py::test_disable_wal - pysqlite3.dbapi2.OperationalErro... FAILED tests/test_fts.py::test_rebuild_fts[searchable] - pysqlite3.dbapi2.Dat... FAILED tests/test_fts.py::test_rebuild_fts[searchable_fts] - pysqlite3.dbapi2... FAILED tests/test_wal.py::test_enable_disable_wal - pysqlite3.dbapi2.Operatio... ================== 5 failed, 750 passed, 3 skipped in 15.20s =================== ``` https://github.com/simonw/sqlite-utils/runs/4360759085 The WAL errors look like this: ``` E pysqlite3.dbapi2.OperationalError: cannot change into wal mode from within a transaction ``` Triggered by a call to `db.enable_wal()` The FTS errors are caused by tests that try to deliberately corrupt the FTS index by running `fresh_db[""searchable_fts_data""].delete_where()` - and then rebuilding it using `rebuild_fts()`: ``` @pytest.mark.parametrize(""table_to_fix"", [""searchable"", ""searchable_fts""]) def test_rebuild_fts(fresh_db, table_to_fix): table = fresh_db[""searchable""] table.insert(search_records[0]) table.enable_fts([""text"", ""country""]) # Run a search rows = list(table.search(""tanuki"")) assert len(rows) == 1 assert { ""rowid"": 1, ""text"": ""tanuki are running tricksters"", ""country"": ""Japan"", ""not_searchable"": ""foo"", }.items() <= rows[0].items() # Delete from searchable_fts_data fresh_db[""searchable_fts_data""].delete_where() # This should have broken the index with pytest.raises(sqlite3.DatabaseError): list(table.search(""tanuki"")) # Running rebuild_fts() should fix it > fresh_db[table_to_fix].rebuild_fts() tests/test_fts.py:277: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sqlite_utils/db.py:1947: in rebuild_fts self.db.execute( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = > sql = ""INSERT INTO [searchable_fts]([searchable_fts]) VALUES('rebuild');"" parameters = None def execute( self, sql: str, parameters: Optional[Union[Iterable, dict]] = None ) -> sqlite3.Cursor: ""Execute SQL query and return a ``sqlite3.Cursor``."" if self._tracer: self._tracer(sql, parameters) if parameters is not None: return self.conn.execute(sql, parameters) else: > return self.conn.execute(sql) E pysqlite3.dbapi2.DatabaseError: database disk image is malformed ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066603133,Test against pysqlite3 running SQLite 3.37, https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982133970,https://api.github.com/repos/simonw/sqlite-utils/issues/347,982133970,IC_kwDOCGYnMM46iizS,9599,simonw,2021-11-29T23:41:17Z,2021-11-29T23:41:17Z,OWNER,"Took a bit of experimenting to get both `mypy` AND `flake8` to ignore the same line. The incantation that worked was this one: https://github.com/simonw/sqlite-utils/blob/f990e134aa8219b687ff6c261330f36824b5df36/sqlite_utils/utils.py#L8 Order here matters - this did NOT work for both tools: ```python from sqlite3.dump import _iterdump as iterdump # noqa: F401 # type: ignore ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066603133,Test against pysqlite3 running SQLite 3.37, https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982126665,https://api.github.com/repos/simonw/sqlite-utils/issues/347,982126665,IC_kwDOCGYnMM46ihBJ,9599,simonw,2021-11-29T23:26:01Z,2021-11-29T23:33:48Z,OWNER,"https://github.com/simonw/sqlite-utils/blob/93b059dd230eae9eaae472b7fbabd4a66feeb79d/.github/workflows/test.yml#L11-L20 This configuration means that the numpy=0, Python=3.10, os=Ubuntu build will additionally use `pysqlite3` with the SQLite 3.37.0. It's failing right now: https://github.com/simonw/sqlite-utils/runs/4360593156 - because `pysqlite3` doesn't provide `.iterdump()`. I can use the workaround from this comment: https://github.com/coleifer/pysqlite3/issues/24#issuecomment-982081267","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066603133,Test against pysqlite3 running SQLite 3.37, https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982129727,https://api.github.com/repos/simonw/sqlite-utils/issues/347,982129727,IC_kwDOCGYnMM46ihw_,9599,simonw,2021-11-29T23:31:58Z,2021-11-29T23:31:58Z,OWNER,"It failed on other Python versions with `mypy`: ``` sqlite_utils/utils.py:8: error: Cannot find implementation or library stub for module named ""sqlite3.dump"" sqlite_utils/utils.py:8: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066603133,Test against pysqlite3 running SQLite 3.37, https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982129218,https://api.github.com/repos/simonw/sqlite-utils/issues/347,982129218,IC_kwDOCGYnMM46ihpC,9599,simonw,2021-11-29T23:31:02Z,2021-11-29T23:31:02Z,OWNER,Here's the test run that's installing `pysqlite3` and that version of SQLite: https://github.com/simonw/sqlite-utils/runs/4360663292,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066603133,Test against pysqlite3 running SQLite 3.37, https://github.com/simonw/sqlite-utils/issues/346#issuecomment-982111751,https://api.github.com/repos/simonw/sqlite-utils/issues/346,982111751,IC_kwDOCGYnMM46idYH,9599,simonw,2021-11-29T23:11:17Z,2021-11-29T23:12:49Z,OWNER,"To keep things simple for the moment I'm only going to add one extra thing to the matrix, and it will be a run of the tests against SQLite 3.37.0 using pysqlite3 on Linux only. I can use this mechanism: https://docs.github.com/en/actions/learn-github-actions/workflow-syntax-for-github-actions#example-including-new-combinations","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066563554,Way to test SQLite 3.37 (and potentially other versions) in CI, https://github.com/simonw/sqlite-utils/issues/346#issuecomment-982094370,https://api.github.com/repos/simonw/sqlite-utils/issues/346,982094370,IC_kwDOCGYnMM46iZIi,9599,simonw,2021-11-29T22:50:49Z,2021-11-29T22:50:49Z,OWNER,I have a working recipe for compiling it for macOS here: https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982006544,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066563554,Way to test SQLite 3.37 (and potentially other versions) in CI, https://github.com/simonw/sqlite-utils/issues/346#issuecomment-982094020,https://api.github.com/repos/simonw/sqlite-utils/issues/346,982094020,IC_kwDOCGYnMM46iZDE,9599,simonw,2021-11-29T22:50:11Z,2021-11-29T22:50:11Z,OWNER,"For the moment I think I'll combine two problems into one, and just add a single matrix alternative that uses `pysqlite3` running SQLite 3.37.0 - only on macOS and Linux so I don't have to figure out how to compile it for Windows.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066563554,Way to test SQLite 3.37 (and potentially other versions) in CI, https://github.com/simonw/sqlite-utils/issues/345#issuecomment-982091363,https://api.github.com/repos/simonw/sqlite-utils/issues/345,982091363,IC_kwDOCGYnMM46iYZj,9599,simonw,2021-11-29T22:45:26Z,2021-11-29T22:45:26Z,OWNER,This is the implementation: https://github.com/simonw/sqlite-utils/blob/213a0ff177f23a35f3b235386366ff132eb879f1/sqlite_utils/db.py#L1236-L1241,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066501534,`table.strict` introspection boolean for identifying STRICT mode tables, https://github.com/simonw/sqlite-utils/issues/345#issuecomment-982090895,https://api.github.com/repos/simonw/sqlite-utils/issues/345,982090895,IC_kwDOCGYnMM46iYSP,9599,simonw,2021-11-29T22:44:36Z,2021-11-29T22:44:36Z,OWNER,Documented here: https://sqlite-utils.datasette.io/en/latest/python-api.html#strict,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066501534,`table.strict` introspection boolean for identifying STRICT mode tables, https://github.com/simonw/sqlite-utils/issues/346#issuecomment-982078527,https://api.github.com/repos/simonw/sqlite-utils/issues/346,982078527,IC_kwDOCGYnMM46iVQ_,9599,simonw,2021-11-29T22:23:03Z,2021-11-29T22:23:03Z,OWNER,"Here's a modified version of the `dump` command that works even with `pysqlite3`: ```python @cli.command() @click.argument( ""path"", type=click.Path(exists=True, file_okay=True, dir_okay=False, allow_dash=False), required=True, ) @load_extension_option def dump(path, load_extension): """"""Output a SQL dump of the schema and full contents of the database"""""" db = sqlite_utils.Database(path) _load_extensions(db, load_extension) # pysqlite3 doesn't implement .iterdump() from sqlite3.dump import _iterdump for line in _iterdump(db.conn): click.echo(line) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066563554,Way to test SQLite 3.37 (and potentially other versions) in CI, https://github.com/simonw/sqlite-utils/issues/346#issuecomment-982077873,https://api.github.com/repos/simonw/sqlite-utils/issues/346,982077873,IC_kwDOCGYnMM46iVGx,9599,simonw,2021-11-29T22:22:05Z,2021-11-29T22:22:05Z,OWNER,"Ideally I'd like an extra set of matrix options for different versions of SQLite. I can use `pysqlite3` for this, but it isn't a completely compatible drop-in replacement - turns out it doesn't support the `iterdump()` method for example, see https://github.com/coleifer/pysqlite3/issues/24","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066563554,Way to test SQLite 3.37 (and potentially other versions) in CI, https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982076924,https://api.github.com/repos/simonw/sqlite-utils/issues/344,982076924,IC_kwDOCGYnMM46iU38,9599,simonw,2021-11-29T22:20:44Z,2021-11-29T22:20:44Z,OWNER,Need to figure out a good pattern for testing this in CI too - it will currently skip the new tests if it doesn't have SQLite 3.37 or higher.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066474200,Support STRICT tables, https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982076702,https://api.github.com/repos/simonw/sqlite-utils/issues/344,982076702,IC_kwDOCGYnMM46iU0e,9599,simonw,2021-11-29T22:20:22Z,2021-11-29T22:20:22Z,OWNER,"I haven't documented `db.supports_strict` yet (I documented `table.strict`) because there wasn't an obvious section of the documentation for it. I need to remember to document it once I add documentation for the `strict=True` parameter.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066474200,Support STRICT tables, https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982049148,https://api.github.com/repos/simonw/sqlite-utils/issues/344,982049148,IC_kwDOCGYnMM46iOF8,9599,simonw,2021-11-29T21:40:59Z,2021-11-29T21:40:59Z,OWNER,I'm going to add that as `db.supports_strict`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066474200,Support STRICT tables, https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982048918,https://api.github.com/repos/simonw/sqlite-utils/issues/344,982048918,IC_kwDOCGYnMM46iOCW,9599,simonw,2021-11-29T21:40:42Z,2021-11-29T21:40:42Z,OWNER,"Here's a function that detects if `strict` is supported or not: ```python import secrets import sqlite3 def supports_strict_tables(db = None): if db is None: db = sqlite3.connect("":memory:"") try: table_name = 't{}'.format(secrets.token_hex(16)) with db: db.execute(""create table {} (name text) strict"".format(table_name)) db.execute(""drop table {}"".format(table_name)) return True except: return False ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066474200,Support STRICT tables, https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982026918,https://api.github.com/repos/simonw/sqlite-utils/issues/344,982026918,IC_kwDOCGYnMM46iIqm,9599,simonw,2021-11-29T21:11:42Z,2021-11-29T21:16:31Z,OWNER,"Made myself a test strict table like so: ```pycon >>> import pysqlite3 >>> conn = pysqlite3.connect(""/tmp/strict-table.db"") >>> conn.execute(""create table foo (id integer primary key, name text, weight real) strict"") >>> cursor = conn.cursor() >>> with conn: ... cursor.execute(""insert into foo (name, weight) values ('Lila', '2.31')"") >>> conn.close() ``` Uploaded that to: https://static.simonwillison.net/static/2021/strict-table.db","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066474200,Support STRICT tables, https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982020757,https://api.github.com/repos/simonw/sqlite-utils/issues/344,982020757,IC_kwDOCGYnMM46iHKV,9599,simonw,2021-11-29T21:03:34Z,2021-11-29T21:03:34Z,OWNER,"From the STRICT docs: > The SQLite parser accepts a comma-separated list of table options after the final close parenthesis in a CREATE TABLE statement. As of this writing (2021-08-23) only two options are recognized: > > - STRICT > - [WITHOUT ROWID](https://www.sqlite.org/withoutrowid.html) So I think I need to read the `CREATE TABLE` statement from the `sqlite_master` table, split on the last `)`, split those tokens on `,` and see if `create` is in there (case insensitive).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066474200,Support STRICT tables, https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982018304,https://api.github.com/repos/simonw/sqlite-utils/issues/344,982018304,IC_kwDOCGYnMM46iGkA,9599,simonw,2021-11-29T21:00:02Z,2021-11-29T21:00:02Z,OWNER,Is there a need for an introspection function for telling if a SQLite table is in strict mode or not? How would that work?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066474200,Support STRICT tables, https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982017994,https://api.github.com/repos/simonw/sqlite-utils/issues/344,982017994,IC_kwDOCGYnMM46iGfK,9599,simonw,2021-11-29T20:59:37Z,2021-11-29T20:59:37Z,OWNER,"I'm leaning towards silently ignore if SQLite version doesn't support it. That means that the first time you attempt to use `strict=True` we will need to run a test against the database connection to see what version of SQLite it uses, then cache the result to avoid making the same call again for the same connection.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066474200,Support STRICT tables, https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982016594,https://api.github.com/repos/simonw/sqlite-utils/issues/344,982016594,IC_kwDOCGYnMM46iGJS,9599,simonw,2021-11-29T20:57:42Z,2021-11-29T20:57:42Z,OWNER,"What should happen if you attempt to use `strict=True` against a SQLite version prior to 3.37.0? An obvious error would be best... but how about silently ignoring it on older versions instead? That would match how we handle `deterministic=True` for registering functions: https://github.com/simonw/sqlite-utils/blob/126703706ea153f63e6134ad14e5712e4bbcb8ae/sqlite_utils/db.py#L372-L380 https://github.com/simonw/sqlite-utils/blob/93c7fd9868fed3193a1732b39bfac539e5812b0b/tests/test_register_function.py#L34-L37 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066474200,Support STRICT tables, https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982014776,https://api.github.com/repos/simonw/sqlite-utils/issues/344,982014776,IC_kwDOCGYnMM46iFs4,9599,simonw,2021-11-29T20:55:19Z,2021-11-29T20:55:19Z,OWNER,"There are a few places that the `strict=True` option could go: - `table.create()` and `table.create_table_sql()` - The `Database()` constructor, to turn it on for all created tables - The `.insert()` / `.insert_all()` etc family of methods that can implicitly create tables I'll definitely implement the first one, and likely the second one too. I'm on the fence with regards to the third one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066474200,Support STRICT tables, https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982006544,https://api.github.com/repos/simonw/sqlite-utils/issues/344,982006544,IC_kwDOCGYnMM46iDsQ,9599,simonw,2021-11-29T20:44:37Z,2021-11-29T20:48:43Z,OWNER,"This worked: ``` cd /tmp mkdir sqlite-3.37 cd sqlite-3.37 wget 'https://www.sqlite.org/2021/sqlite-amalgamation-3370000.zip' unzip sqlite-amalgamation-3370000.zip git clone https://github.com/coleifer/pysqlite3/ cp sqlite-amalgamation-3370000/sqlite3.[ch] pysqlite3 cd pysqlite3 python3 setup.py build_static build bdist_wheel ``` This gave me a file here: ``` pysqlite3 % ls -l dist total 1872 -rw-r--r-- 1 simon wheel 956557 Nov 29 12:38 pysqlite3-0.4.6-cp39-cp39-macosx_10_15_x86_64.whl ``` That wheel only works when installed for Python 3.9 (it failed to install in a Python 3.10 virtual environment) - but `pip install /tmp/sqlite-3.37/pysqlite3/dist/pysqlite3-0.4.6-cp39-cp39-macosx_10_15_x86_64.whl` gave me a working `pysqlite3` - and the following worked: ```pycon >>> import pysqlite3 >>> pysqlite3.connect("":memory:"").execute(""select sqlite_version()"").fetchall() [('3.37.0',)] ``` And if I install `sqlite-utils` in the same virtual environment this works: ``` % sqlite-utils memory 'select sqlite_version()' [{""sqlite_version()"": ""3.37.0""}] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066474200,Support STRICT tables, https://github.com/simonw/sqlite-utils/issues/344#issuecomment-981999025,https://api.github.com/repos/simonw/sqlite-utils/issues/344,981999025,IC_kwDOCGYnMM46iB2x,9599,simonw,2021-11-29T20:34:38Z,2021-11-29T20:35:58Z,OWNER,"I'm going to build my own `pysqlite3` wheel with the latest SQLite to try this out, following the instructions on https://github.com/coleifer/pysqlite3","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066474200,Support STRICT tables, https://github.com/simonw/sqlite-utils/issues/344#issuecomment-981997973,https://api.github.com/repos/simonw/sqlite-utils/issues/344,981997973,IC_kwDOCGYnMM46iBmV,9599,simonw,2021-11-29T20:33:52Z,2021-11-29T20:33:52Z,OWNER,"From that page: > If you try to open a database containing the STRICT keyword in an earlier version of SQLite, it will not recognize the keyword and will report an error (except as noted below. > > [...] > > Because of a quirk in the SQL language parser, versions of SQLite prior to 3.37.0 can still read and write STRICT tables if they set ""PRAGMA writable_schema=ON"" immediately after opening the database file, prior to doing anything else that requires knowing the schema.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066474200,Support STRICT tables, https://github.com/simonw/datasette/issues/1538#issuecomment-981856895,https://api.github.com/repos/simonw/datasette/issues/1538,981856895,IC_kwDOBm6k_c46hfJ_,9599,simonw,2021-11-29T17:32:44Z,2021-11-29T17:32:44Z,OWNER,TIL: https://til.simonwillison.net/datasette/reuse-click-for-register-commands,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066288689,Research pattern for re-registering existing Click tools with register_commands, https://github.com/simonw/datasette/issues/1538#issuecomment-981852280,https://api.github.com/repos/simonw/datasette/issues/1538,981852280,IC_kwDOBm6k_c46heB4,9599,simonw,2021-11-29T17:27:12Z,2021-11-29T17:27:12Z,OWNER,"Thanks to https://stackoverflow.com/a/45514541/6083 I found the right pattern: ```python from datasette import hookimpl from git_history.cli import cli as git_history_cli @hookimpl def register_commands(cli): cli.add_command(git_history_cli, name=""git-history"") ``` I think this is a little bit too obscure to add to the Datasette documentation - I'll turn it into a TIL instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066288689,Research pattern for re-registering existing Click tools with register_commands, https://github.com/simonw/datasette/issues/1538#issuecomment-981849494,https://api.github.com/repos/simonw/datasette/issues/1538,981849494,IC_kwDOBm6k_c46hdWW,9599,simonw,2021-11-29T17:23:52Z,2021-11-29T17:23:52Z,OWNER,"Just trying to use `git_history_file` produces this error: > `TypeError: Attempted to convert a callback into a command twice.` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066288689,Research pattern for re-registering existing Click tools with register_commands, https://github.com/simonw/datasette/issues/1518#issuecomment-981172801,https://api.github.com/repos/simonw/datasette/issues/1518,981172801,IC_kwDOBm6k_c46e4JB,9599,simonw,2021-11-28T23:23:51Z,2021-11-28T23:23:51Z,OWNER,"(I could experiment with merging the two tables by adding a temporary undocumented `?_sql=` parameter to the in-progress table view that sets an alternative query instead of `select cols from table` - added bonus, this will force me to use introspection against the returned columns rather than mixing in the known columns for the specified table)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-981172385,https://api.github.com/repos/simonw/datasette/issues/1518,981172385,IC_kwDOBm6k_c46e4Ch,9599,simonw,2021-11-28T23:21:26Z,2021-11-28T23:21:26Z,OWNER,"Aside: is there any reason this work can't complete the long-running goal of merging the TableView and QueryView, such that most of the features available for tables become available for arbitrary queries too? I had already mentally committed to implementing facets for queries, but I just realized that filters could work too - using either a CTE or a nested query. Pagination is the one holdout here, since table pagination uses keyset pagination over a known order. But maybe arbitrary queries can only be paginated off you order them first?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-981153186,https://api.github.com/repos/simonw/datasette/issues/1518,981153186,IC_kwDOBm6k_c46ezWi,9599,simonw,2021-11-28T21:13:50Z,2021-11-28T21:13:50Z,OWNER,"I'm also going to use the new `datasette-table` Web Component to help guide the design of the new API, which relates directly to this issue too: - #1532","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1534#issuecomment-981149531,https://api.github.com/repos/simonw/datasette/issues/1534,981149531,IC_kwDOBm6k_c46eydb,9599,simonw,2021-11-28T20:48:54Z,2021-11-28T20:48:54Z,OWNER,"If I'm going to do this, is there value in also spotting `Accept: text/csv` and returning CSV for that? I'm pretty sure no client has EVER implemented this though, so it feels like it would be showboating.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065432388,Maybe return JSON from HTML pages if `Accept: application/json` is sent, https://github.com/simonw/datasette/issues/1533#issuecomment-981149039,https://api.github.com/repos/simonw/datasette/issues/1533,981149039,IC_kwDOBm6k_c46eyVv,9599,simonw,2021-11-28T20:45:36Z,2021-11-28T20:45:36Z,OWNER,I built an initial prototype of this in a branch: https://github.com/simonw/datasette/commit/e0a84691c2959f2d1d76948574c9c4a910c7556c - which exposed even more flaws in the way `TableView` is structured (adding custom HTTP headers to the response is way harder than it should be) which I should address in the refactor in #617.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065431383,"Add `Link: rel=""alternate""` header pointing to JSON for a table/query", https://github.com/simonw/sqlite-utils/pull/333#issuecomment-979442854,https://api.github.com/repos/simonw/sqlite-utils/issues/333,979442854,IC_kwDOCGYnMM46YRym,9599,simonw,2021-11-25T19:47:26Z,2021-11-25T19:47:26Z,OWNER,"I just remembered that there's one other place that this could fit: as a Datasette ""insert"" plugin. This is vaporware at the moment, but the idea is that Datasette itself could grow a mechanism for importing data, that's driven by plugins. Out of the box Datasette would be able to import CSV and CSV files, similar to `sqlite-utils insert ... --csv` - but plugins would then be able to add support for additional format such as GeoJSON or - in this case - Parquet. The neat thing about having it as a Datasette plugin is that one plugin would enable three different ways of importing data: 1. Via a new `datasette insert ...` CLI option (similar to `sqlite-utils`) 2. Via a web form upload interface, where authenticated Datasette users would be able to upload files 3. Via an API interface, where files could be programatically submitted to a running Datasette server I started fleshing out this idea quite a while ago but didn't make much concrete progress, maybe I should revisit it: - https://github.com/simonw/datasette/issues/1160","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1039037439,Add functionality to read Parquet files., https://github.com/simonw/datasette/issues/1526#issuecomment-975110692,https://api.github.com/repos/simonw/datasette/issues/1526,975110692,IC_kwDOBm6k_c46HwIk,9599,simonw,2021-11-22T04:49:44Z,2021-11-22T04:49:44Z,OWNER,Fixed in the 0.12 release of that plugin: https://github.com/simonw/datasette-publish-vercel/releases/tag/0.12,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059549523,"Add to vercel.json, rather than overwriting it.", https://github.com/simonw/datasette/issues/1526#issuecomment-975073308,https://api.github.com/repos/simonw/datasette/issues/1526,975073308,IC_kwDOBm6k_c46HnAc,9599,simonw,2021-11-22T04:13:46Z,2021-11-22T04:13:46Z,OWNER,"Addressing that over here (hadn't seen that issue yet, thanks for the prod): https://github.com/simonw/datasette-publish-vercel/issues/51#issuecomment-975073026","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059549523,"Add to vercel.json, rather than overwriting it.", https://github.com/simonw/datasette/issues/1527#issuecomment-974979785,https://api.github.com/repos/simonw/datasette/issues/1527,974979785,IC_kwDOBm6k_c46HQLJ,9599,simonw,2021-11-22T01:02:57Z,2021-11-22T01:03:19Z,OWNER,"I think the root cause is this hidden form field on https://latest.datasette.io/fixtures/facetable?_facet=_neighborhood&_neighborhood__exact=Downtown ```html ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059555791,Columns starting with an underscore behave poorly in filters, https://github.com/simonw/datasette/issues/1525#issuecomment-974913180,https://api.github.com/repos/simonw/datasette/issues/1525,974913180,IC_kwDOBm6k_c46G_6c,9599,simonw,2021-11-21T22:57:08Z,2021-11-21T22:57:08Z,OWNER,https://latest.datasette.io/fixtures/facetable can't quite demonstrate the bug because `_neighborhood` isn't a foreign key - I should rename `city_id` to `_city_id`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059509927,"""Links from other tables"" broken for columns starting with underscore", https://github.com/simonw/datasette/issues/1525#issuecomment-974912985,https://api.github.com/repos/simonw/datasette/issues/1525,974912985,IC_kwDOBm6k_c46G_3Z,9599,simonw,2021-11-21T22:55:42Z,2021-11-21T22:55:42Z,OWNER,Here's the template: https://github.com/simonw/datasette/blob/48f11998b73350057b74fe6ab464d4ac3071637c/datasette/templates/row.html#L39-L41,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059509927,"""Links from other tables"" broken for columns starting with underscore", https://github.com/simonw/datasette/issues/93#issuecomment-974765825,https://api.github.com/repos/simonw/datasette/issues/93,974765825,IC_kwDOBm6k_c46Gb8B,9599,simonw,2021-11-21T07:00:21Z,2021-11-21T07:00:21Z,OWNER,Closing this in favour of Datasette Desktop: https://datasette.io/desktop,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",273944952,Package as standalone binary, https://github.com/simonw/sqlite-utils/pull/333#issuecomment-974754412,https://api.github.com/repos/simonw/sqlite-utils/issues/333,974754412,IC_kwDOCGYnMM46GZJs,9599,simonw,2021-11-21T04:35:32Z,2021-11-21T04:35:32Z,OWNER,"Some other recent projects (like trying to get this library to work in JupyterLite) have made me much more cautious about adding new dependencies, especially dependencies like `pyarrow` which require custom C/Rust extensions. There are a few ways this could work though: - Have this as an optional dependency feature - so it only works if the user installs `pyarrow` as well - Implement this as a separate tool, `parquet-to-sqlite` - which could itself depend on `sqlite-utils` - Add a concept of ""plugins"" to `sqlite-utils`, similar to how those work in Datasette: https://docs.datasette.io/en/stable/plugins.html My favourite option is `parquet-to-sqlite` because that can be built without any additional changes to `sqlite-utils` at all! I find the concept of plugins for `sqlite-utils` interesting. I've so far not had quite enough potential use-cases to convince me this is worthwhile (especially since it should be very easy to build out separate tools entirely), but I'm ready to be convinced that a plugin mechanism would be worthwhile.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1039037439,Add functionality to read Parquet files., https://github.com/simonw/datasette/issues/1524#issuecomment-974725814,https://api.github.com/repos/simonw/datasette/issues/1524,974725814,IC_kwDOBm6k_c46GSK2,9599,simonw,2021-11-20T23:24:01Z,2021-11-20T23:24:01Z,OWNER,"I noticed that `http://datasette-apache-proxy-demo.datasette.io/` wasn't redirecting to `https` so I built a new plugin: https://github.com/simonw/datasette-redirect-to-https ``` % curl -i 'http://datasette-apache-proxy-demo.datasette.io/prefix/fixtures/no_primary_key' HTTP/1.1 301 Moved Permanently date: Sat, 20 Nov 2021 23:22:50 GMT server: Fly/51d150d (2021-11-19) location: https://datasette-apache-proxy-demo.datasette.io/fixtures/no_primary_key x-proxied-by: Apache2 Debian transfer-encoding: chunked via: 1.1 fly.io fly-request-id: 01FMZTHTHVPC8BZY0625D7JV4B ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059219106,"Improve Apache proxy documentation, link to demo", https://github.com/simonw/datasette/issues/1524#issuecomment-974721652,https://api.github.com/repos/simonw/datasette/issues/1524,974721652,IC_kwDOBm6k_c46GRJ0,9599,simonw,2021-11-20T22:41:03Z,2021-11-20T22:41:03Z,OWNER,New TIL: https://til.simonwillison.net/fly/custom-subdomain-fly,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059219106,"Improve Apache proxy documentation, link to demo", https://github.com/simonw/datasette/issues/1524#issuecomment-974707878,https://api.github.com/repos/simonw/datasette/issues/1524,974707878,IC_kwDOBm6k_c46GNym,9599,simonw,2021-11-20T20:34:51Z,2021-11-20T20:38:29Z,OWNER,"I pointed `CNAME` of `datasette-apache-proxy-demo.datasette.io` at `datasette-apache-proxy-demo.fly.dev.` using Vercel DNS: Then I asked Fly to issue a LetsEncrypt certificate for that: ``` % flyctl certs create datasette-apache-proxy-demo.datasette.io # About 53 seconds later: % flyctl certs show datasette-apache-proxy-demo.datasette.io The certificate for datasette-apache-proxy-demo.datasette.io has been issued. Hostname = datasette-apache-proxy-demo.datasette.io DNS Provider = constellix Certificate Authority = Let's Encrypt Issued = ecdsa,rsa Added to App = 53 seconds ago Source = fly ``` https://datasette-apache-proxy-demo.datasette.io/ works now - I'll use that in the documentation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059219106,"Improve Apache proxy documentation, link to demo", https://github.com/simonw/datasette/issues/1524#issuecomment-974704254,https://api.github.com/repos/simonw/datasette/issues/1524,974704254,IC_kwDOBm6k_c46GM5-,9599,simonw,2021-11-20T20:03:51Z,2021-11-20T20:22:52Z,OWNER,"I'm also going to extract the Apache config files from https://github.com/simonw/datasette/blob/250db8192cb8aba5eb8cd301ccc2a49525bc3d24/demos/apache-proxy/Dockerfile into a separate file to make it easier to read. (The supervisor config needs to be dynamically constructed to include $DATASETTE_REF so I will leave it where it is.)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059219106,"Improve Apache proxy documentation, link to demo", https://github.com/simonw/datasette/issues/1519#issuecomment-974701788,https://api.github.com/repos/simonw/datasette/issues/1519,974701788,IC_kwDOBm6k_c46GMTc,9599,simonw,2021-11-20T19:42:29Z,2021-11-20T19:42:29Z,OWNER,"> I think what's happening here is Apache is actually making a request to `/fixtures` rather than making a request to `/prefix/fixtures` - and Datasette is replying to requests on both the prefixed and the non-prefixed paths. > > This is pretty confusing! I think Datasette should ONLY reply to `/prefix/fixtures` instead and return a 404 for `/fixtures` - this would make things a whole lot easier to debug. > > But shipping that change could break existing deployments. Maybe that should be a breaking change for 1.0. On further thought I'm not going to do this. Having Datasette work behind a proxy the way it does right now is clearly easy for people to deploy (now that I've fixed the bugs) and I trust my improved tests to catch problems in the future.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974697824,https://api.github.com/repos/simonw/datasette/issues/1519,974697824,IC_kwDOBm6k_c46GLVg,9599,simonw,2021-11-20T19:11:21Z,2021-11-20T19:11:21Z,OWNER,"OK, i think I got all of them this time! The latest demo is now live at https://datasette-apache-proxy-demo.fly.dev/prefix/fixtures/sortable?_facet=pk2 I'm closing this issue, but feel free to re-open it if you spot any that I missed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1522#issuecomment-974695111,https://api.github.com/repos/simonw/datasette/issues/1522,974695111,IC_kwDOBm6k_c46GKrH,9599,simonw,2021-11-20T18:52:11Z,2021-11-20T18:52:11Z,OWNER,The demo is now live on https://datasette-apache-proxy-demo.fly.dev/prefix/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974693350,https://api.github.com/repos/simonw/datasette/issues/1522,974693350,IC_kwDOBm6k_c46GKPm,9599,simonw,2021-11-20T18:39:27Z,2021-11-20T18:39:27Z,OWNER,"I'm going to go with Fly instead for this, especially as I can keep it within their free tier (and iI want to get more familiar with their platform).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974692546,https://api.github.com/repos/simonw/datasette/issues/1522,974692546,IC_kwDOBm6k_c46GKDC,9599,simonw,2021-11-20T18:34:13Z,2021-11-20T18:34:13Z,OWNER,"Here's why it failed - the `fly.toml` file that was generated when I ran `flyctl launch` had this section: ```toml [[services]] http_checks = [] internal_port = 8080 processes = [""app""] protocol = ""tcp"" script_checks = [] ``` But I need `internal_port` to be 80 for Apache, so I changed that and ran `flyctl deploy --build-arg DATASETTE_REF=main` again - and it worked! https://floral-dust-4577.fly.dev/prefix/ - not seeing any 503 errors there.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974685095,https://api.github.com/repos/simonw/datasette/issues/1522,974685095,IC_kwDOBm6k_c46GIOn,9599,simonw,2021-11-20T17:42:25Z,2021-11-20T17:42:25Z,OWNER,"I tried to deploy it to Fly - initially using `flyctl launch` but then switching to `flyctl deploy` so I could use the `--build-arg` option (posted [a feature request](https://community.fly.io/t/feature-request-flyctl-launch-build-arg/3209) here). Almost got it working, but it failed the health check: ``` % cd datasette/demos/apache-proxy apache-proxy % flyctl launch Creating app in /Users/simon/Dropbox/Development/datasette/demos/apache-proxy Scanning source code Detected Dockerfile app Automatically selected personal organization: Simon Willison ? Select region: sjc (Sunnyvale, California (US)) Created app floral-dust-4577 in organization personal Wrote config file fly.toml Your app is ready. Deploy with `flyctl deploy` ? Would you like to deploy now? Yes Deploying floral-dust-4577 ==> Validating app configuration --> Validating app configuration done Services TCP 80/443 ⇢ 8080 ==> Creating build context --> Creating build context done ==> Building image with Docker Sending build context to Docker daemon 8.704kB ... Error error building: executor failed running [/bin/sh -c pip install https://github.com/simonw/datasette/archive/${DATASETTE_REF}.zip]: exit code: 1 # I didn't pass the build argument, trying again with flyctl deploy apache-proxy % flyctl deploy --build-arg DATASETTE_REF=main Update available 0.0.229 -> v0.0.255 Run ""flyctl version update"" to upgrade Deploying floral-dust-4577 ==> Validating app configuration --> Validating app configuration done Services TCP 80/443 ⇢ 8080 ==> Creating build context --> Creating build context done ==> Building image with Docker Sending build context to Docker daemon 8.704kB [+] Building 15.7s (27/27) ... 0.0s ==> Pushing image to fly The push refers to repository [registry.fly.io/floral-dust-4577] 9bf88c92aa2a: Pushed 3d61728b8391: Pushed ... --> Pushing image done Image: registry.fly.io/floral-dust-4577:deployment-1637429501 Image size: 276 MB ==> Creating release Release v2 created You can detach the terminal anytime without stopping the deployment Monitoring Deployment 1 desired, 1 placed, 0 healthy, 0 unhealthy [health checks: 1 total, 1 critical] 1 desired, 1 placed, 0 healthy, 1 unhealthy [health checks: 1 total, 1 critical] v0 failed - Failed due to unhealthy allocations - no stable job version to auto revert to Failed Instances ==> Failure #1 Instance ID = 36adac86 Version = 0 Region = sjc Desired = run Status = running Health Checks = 1 total, 1 critical Restarts = 0 Created = 4m52s ago Recent Events TIMESTAMP TYPE MESSAGE 2021-11-20T17:32:52Z Received Task received by client 2021-11-20T17:32:52Z Task Setup Building Task Directory 2021-11-20T17:33:02Z Started Task started by client Recent Logs 2021-11-20T17:32:56Z [info] Unpacking image 2021-11-20T17:33:01Z [info] Preparing kernel init 2021-11-20T17:33:01Z [info] Configuring firecracker 2021-11-20T17:33:02Z [info] Starting virtual machine 2021-11-20T17:33:02Z [info] Starting init (commit: 7943db6)... 2021-11-20T17:33:02Z [info] Preparing to run: `/usr/bin/supervisord -c /app/supervisord.conf` as root 2021-11-20T17:33:02Z [info] 2021/11/20 17:33:02 listening on [fdaa:0:4ef:a7b:2295:36ad:ac86:2]:22 (DNS: [fdaa::3]:53) 2021-11-20T17:33:02Z [info] 2021-11-20 17:33:02,374 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message. 2021-11-20T17:33:02Z [info] 2021-11-20 17:33:02,376 INFO supervisord started with pid 510 2021-11-20T17:33:03Z [info] 2021-11-20 17:33:03,379 INFO spawned: 'apache2' with pid 515 2021-11-20T17:33:03Z [info] 2021-11-20 17:33:03,381 INFO spawned: 'datasette' with pid 516 2021-11-20T17:33:05Z [info] 2021-11-20 17:33:05,068 INFO success: apache2 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2021-11-20T17:33:05Z [info] 2021-11-20 17:33:05,068 INFO success: datasette entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2021-11-20T17:33:28Z [error] Health check status changed 'warning' => 'critical' ***v0 failed - Failed due to unhealthy allocations - no stable job version to auto revert to and deploying as v1 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974683220,https://api.github.com/repos/simonw/datasette/issues/1522,974683220,IC_kwDOBm6k_c46GHxU,9599,simonw,2021-11-20T17:29:12Z,2021-11-20T17:29:12Z,OWNER,"> As a a sanity check, would it be worth looking at trying to push the multi-process container on another provider of a knative / cloud run / tekton ? I have a somewhat similar use case for a future proejct, so i'm been very grateful to you sharing all the progress in this issue. That's a great idea. I'll try running on a non-Knative host too (probably Fly - though they actually run containers using Firecracker which ends up being completely different). Cloud Run are the only Knative host I've used, know of any others aside from Scaleway? They look like they're worth getting familiar with.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974682507,https://api.github.com/repos/simonw/datasette/issues/1522,974682507,IC_kwDOBm6k_c46GHmL,9599,simonw,2021-11-20T17:24:13Z,2021-11-20T17:24:13Z,OWNER,"I'm going to leave this issue open, tag it as ""help wanted"" and cross my fingers that someone with Cloud Run deep expertise takes an interest in figuring out what's going wrong here!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974605529,https://api.github.com/repos/simonw/datasette/issues/1522,974605529,IC_kwDOBm6k_c46F0zZ,9599,simonw,2021-11-20T06:52:21Z,2021-11-20T06:52:21Z,OWNER,"I've now tried both Debian and Alpine, and I've tried both `tini` and `supervisord`. Each time I get the same result - I get 503 errors for the first dozen or so refreshes of `/prefix/` followed by it intermittently working. Absolutely stumped.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974605128,https://api.github.com/repos/simonw/datasette/issues/1522,974605128,IC_kwDOBm6k_c46F0tI,9599,simonw,2021-11-20T06:47:59Z,2021-11-20T06:47:59Z,OWNER,"I managed to port the whole thing over to Debian - which took a lot of work because their packaged Apache 2 works very differently from the Alpine one. Once again... I got it working fine on my laptop, but the image deployed to Cloud Run throws 503 errors! ```dockerfile FROM python:3.9.7-slim-bullseye RUN apt-get update && \ apt-get install -y apache2 supervisor && \ apt clean && \ rm -rf /var/lib/apt && \ rm -rf /var/lib/dpkg/info/* # Apache environment, copied from # https://github.com/ijklim/laravel-benfords-law-app/blob/e9bf385dcaddb62ea466a7b245ab6e4ef708c313/docker/os/Dockerfile ENV APACHE_DOCUMENT_ROOT=/var/www/html/public ENV APACHE_RUN_USER www-data ENV APACHE_RUN_GROUP www-data ENV APACHE_PID_FILE /var/run/apache2.pid ENV APACHE_RUN_DIR /var/run/apache2 ENV APACHE_LOCK_DIR /var/lock/apache2 ENV APACHE_LOG_DIR /var/log RUN ln -sf /dev/stdout /var/log/apache2-access.log RUN ln -sf /dev/stderr /var/log/apache2-error.log RUN mkdir -p $APACHE_RUN_DIR $APACHE_LOCK_DIR RUN a2enmod proxy RUN a2enmod proxy_http RUN a2enmod headers ARG DATASETTE_REF RUN pip install https://github.com/simonw/datasette/archive/${DATASETTE_REF}.zip # Append this to the end of the default httpd.conf file RUN echo '\n\ \n\ Options Indexes FollowSymLinks\n\ AllowOverride None\n\ Require all granted\n\ \n\ \n\ \n\ ServerName localhost\n\ DocumentRoot /app/html\n\ ProxyPreserveHost On\n\ ProxyPass /prefix/ http://127.0.0.1:8001/\n\ Header add X-Proxied-By ""Apache2""\n\ \n\ ' > /etc/apache2/sites-enabled/000-default.conf WORKDIR /app RUN mkdir -p /app/html RUN echo 'Datasette' > /app/html/index.html ADD https://latest.datasette.io/fixtures.db /app/fixtures.db EXPOSE 80 RUN echo ""[supervisord]"" >> /app/supervisord.conf RUN echo ""nodaemon=true"" >> /app/supervisord.conf RUN echo """" >> /app/supervisord.conf RUN echo ""[program:apache2]"" >> /app/supervisord.conf RUN echo ""command=apache2 -D FOREGROUND"" >> /app/supervisord.conf RUN echo """" >> /app/supervisord.conf RUN echo ""[program:datasette]"" >> /app/supervisord.conf RUN echo ""command=datasette /app/fixtures.db --setting base_url '/prefix/' --version-note '${DATASETTE_REF}' -h 0.0.0.0 -p 8001"" >> /app/supervisord.conf CMD [""/usr/bin/supervisord"", ""-c"", ""/app/supervisord.conf""] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974602459,https://api.github.com/repos/simonw/datasette/issues/1522,974602459,IC_kwDOBm6k_c46F0Db,9599,simonw,2021-11-20T06:15:58Z,2021-11-20T06:15:58Z,OWNER,First I'm going to try using Debian Buster as the base image instead of Alpine.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974585374,https://api.github.com/repos/simonw/datasette/issues/1522,974585374,IC_kwDOBm6k_c46Fv4e,9599,simonw,2021-11-20T03:28:58Z,2021-11-20T03:28:58Z,OWNER,Based on https://medium.com/google-cloud/init-process-for-containers-d03a471fa0cc I might try s6.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974578141,https://api.github.com/repos/simonw/datasette/issues/1522,974578141,IC_kwDOBm6k_c46FuHd,9599,simonw,2021-11-20T02:27:23Z,2021-11-20T02:27:23Z,OWNER,"Aha! This could be the clue I was looking for: https://www.reddit.com/r/googlecloud/comments/fmkx63/comment/fl5csty/?utm_source=reddit&utm_medium=web2x&context=3 > Are you processing on a background thread in your container? If so, it's likely your problem, because cloud run will put your app into a low power state between http requests. For long running tasks in cloud run, you need to keep the http connection open, and not return until you are done. Maybe the `datasette &` process is being affected by that in some way?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974577949,https://api.github.com/repos/simonw/datasette/issues/1522,974577949,IC_kwDOBm6k_c46FuEd,9599,simonw,2021-11-20T02:26:09Z,2021-11-20T02:26:17Z,OWNER,"So frustrating, that's giving me the same problem after being deployed! 503 errors for the first while, then it starts working.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974577565,https://api.github.com/repos/simonw/datasette/issues/1522,974577565,IC_kwDOBm6k_c46Ft-d,9599,simonw,2021-11-20T02:23:07Z,2021-11-20T02:23:07Z,OWNER,"OK, that works on my laptop - and Ctrl+C quits it, which is nice: ``` apache-proxy % docker run -p 5000:80 --rm datasette-build-arg-demo 2021-11-20 02:22:13,925 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message. 2021-11-20 02:22:13,927 INFO supervisord started with pid 1 2021-11-20 02:22:14,931 INFO spawned: 'datasette' with pid 7 2021-11-20 02:22:14,934 INFO spawned: 'httpd' with pid 8 2021-11-20 02:22:16,484 INFO success: datasette entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2021-11-20 02:22:16,484 INFO success: httpd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) ^C 2021-11-20 02:22:26,285 WARN received SIGINT indicating exit request 2021-11-20 02:22:26,286 INFO waiting for datasette, httpd to die 2021-11-20 02:22:26,315 INFO stopped: httpd (exit status 0) 2021-11-20 02:22:26,540 INFO stopped: datasette (exit status 0) ``` Here's my new Dockerfile: ```dockerfile FROM python:3-alpine RUN apk add --no-cache \ apache2 \ apache2-proxy \ supervisor \ bash ARG DATASETTE_REF RUN pip install https://github.com/simonw/datasette/archive/${DATASETTE_REF}.zip # Append this to the end of the default httpd.conf file RUN echo -e 'ServerName localhost\n\ \n\ \n\ Order deny,allow\n\ Allow from all\n\ \n\ \n\ ProxyPreserveHost On\n\ ProxyPass /prefix/ http://127.0.0.1:8001/\n\ Header add X-Proxied-By ""Apache2""' >> /etc/apache2/httpd.conf RUN echo 'Datasette' > /var/www/localhost/htdocs/index.html WORKDIR /app ADD https://latest.datasette.io/fixtures.db /app/fixtures.db EXPOSE 80 RUN echo ""[supervisord]"" >> /app/supervisord.conf RUN echo ""nodaemon=true"" >> /app/supervisord.conf RUN echo """" >> /app/supervisord.conf RUN echo ""[program:httpd]"" >> /app/supervisord.conf RUN echo ""command=httpd -D FOREGROUND"" >> /app/supervisord.conf RUN echo """" >> /app/supervisord.conf RUN echo ""[program:datasette]"" >> /app/supervisord.conf RUN echo ""command=datasette /app/fixtures.db --setting base_url '/prefix/' --version-note '${DATASETTE_REF}' -h 0.0.0.0 -p 8001"" >> /app/supervisord.conf CMD [""/usr/bin/supervisord"", ""-c"", ""/app/supervisord.conf""] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974577082,https://api.github.com/repos/simonw/datasette/issues/1522,974577082,IC_kwDOBm6k_c46Ft26,9599,simonw,2021-11-20T02:19:27Z,2021-11-20T02:19:27Z,OWNER,"https://docs.docker.com/config/containers/multi-service_container/ suggests `supervisord` as a last resort. https://stackoverflow.com/a/49100302/6083 has a neat looking recipe for than in Alpine: > **1.** `Dockerfile` is: > > FROM alpine:latest > RUN apk update && apk add --no-cache supervisor openssh nginx > COPY supervisord.conf /etc/supervisord.conf > CMD [""/usr/bin/supervisord"", ""-c"", ""/etc/supervisord.conf""] > > **2.** `supervisord.conf` is: > > [supervisord] > nodaemon=true > > [program:sshd] > command=/usr/sbin/sshd -D > > [program:nginx] > command=nginx -c /etc/nginx/nginx.conf ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974576624,https://api.github.com/repos/simonw/datasette/issues/1522,974576624,IC_kwDOBm6k_c46Ftvw,9599,simonw,2021-11-20T02:16:12Z,2021-11-20T02:16:12Z,OWNER,"Again, that approach worked on my laptop but when deployed to Cloud Run mostly gave me 503 errors for the `/prefix/` page, with the occasional 200. I did this: ```Dockerfile RUN echo ""#!/bin/bash"" >> start.sh # Start Datasette running in background with & RUN echo ""datasette /app/fixtures.db --setting base_url '/prefix/' --version-note '${DATASETTE_REF}' -h 0.0.0.0 -p 8001 &"" >> /app/start.sh RUN echo ""httpd -D FOREGROUND"" >> /app/start.sh RUN chmod +x /app/start.sh CMD /app/start.sh ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974576436,https://api.github.com/repos/simonw/datasette/issues/1522,974576436,IC_kwDOBm6k_c46Fts0,9599,simonw,2021-11-20T02:14:45Z,2021-11-20T02:14:45Z,OWNER,I'm going to try running Apache with `httpd -D FOREGROUND` while running `datasette &`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974575512,https://api.github.com/repos/simonw/datasette/issues/1522,974575512,IC_kwDOBm6k_c46FteY,9599,simonw,2021-11-20T02:09:20Z,2021-11-20T02:09:20Z,OWNER,"> **Waiting for health check to begin** makes it sound like the container didn't start properly. That eventually failed, but I did get these in the build logs: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974573616,https://api.github.com/repos/simonw/datasette/issues/1522,974573616,IC_kwDOBm6k_c46FtAw,9599,simonw,2021-11-20T01:58:44Z,2021-11-20T01:58:44Z,OWNER,"Deploy to Cloud Run appears to hang here: ``` Deploying container to Cloud Run service [datasette-apache-proxy-demo] in project [datasette-222320] region [us-central1] ⠧ Deploying... Revision deployment finished. Waiting for health check to begin. ⠧ Creating Revision... . Routing traffic... ✓ Setting IAM Policy... ``` **Waiting for health check to begin** makes it sound like the container didn't start properly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974572505,https://api.github.com/repos/simonw/datasette/issues/1522,974572505,IC_kwDOBm6k_c46FsvZ,9599,simonw,2021-11-20T01:50:48Z,2021-11-20T01:51:41Z,OWNER,"I figured out a recipe to run `httpd` as a service inside Alpine - works great on my laptop, here's my new `Dockerfile`: ```dockerfile FROM python:3-alpine # openrc gives us rc-service RUN apk add --no-cache \ openrc \ apache2 \ apache2-proxy \ bash ARG DATASETTE_REF RUN pip install https://github.com/simonw/datasette/archive/${DATASETTE_REF}.zip # Append this to the end of the default httpd.conf file RUN echo -e 'ServerName localhost\n\ \n\ \n\ Order deny,allow\n\ Allow from all\n\ \n\ \n\ ProxyPreserveHost On\n\ ProxyPass /prefix/ http://127.0.0.1:8001/\n\ Header add X-Proxied-By ""Apache2""' >> /etc/apache2/httpd.conf RUN echo 'Datasette' > /var/www/localhost/htdocs/index.html WORKDIR /app ADD https://latest.datasette.io/fixtures.db /app/fixtures.db EXPOSE 80 # RUN echo -e ""#!/bin/bash\nopenrc default\nrc-service apache2 start;\ndatasette /app/fixtures.db --setting base_url '/prefix/' --version-note '${DATASETTE_REF}' -h 0.0.0.0 -p 8001"" > /app/start.sh RUN echo ""#!/bin/bash"" >> start.sh RUN echo ""openrc default"" >> start.sh RUN echo ""rc-service apache2 start"" >> start.sh RUN echo ""datasette /app/fixtures.db --setting base_url '/prefix/' --version-note '${DATASETTE_REF}' -h 0.0.0.0 -p 8001"" >> /app/start.sh RUN chmod +x /app/start.sh CMD /app/start.sh ``` I'm going to try this on Cloud Run and see if it fixes the 503s One annoying thing about this: Ctrl+C on my laptop no longer stops the container, I have to `docker ps` and then `docker kill xxx` instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974565816,https://api.github.com/repos/simonw/datasette/issues/1522,974565816,IC_kwDOBm6k_c46FrG4,9599,simonw,2021-11-20T01:13:39Z,2021-11-20T01:13:39Z,OWNER,"I have a hunch that running `httpd -D FOREGROUND` doesn't show error logs, which would explain why I can't use the Cloud Run logs to figure out the reason for the 503s.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974565392,https://api.github.com/repos/simonw/datasette/issues/1522,974565392,IC_kwDOBm6k_c46FrAQ,9599,simonw,2021-11-20T01:11:20Z,2021-11-20T01:11:20Z,OWNER,"Yup, that fixed it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974564712,https://api.github.com/repos/simonw/datasette/issues/1522,974564712,IC_kwDOBm6k_c46Fq1o,9599,simonw,2021-11-20T01:07:49Z,2021-11-20T01:10:48Z,OWNER,https://apache-proxy-demo.datasette.io/prefix/fixtures/compound_three_primary_keys has broken suggested facet links - they go to `https://localhost:8001/prefix/fixtures/compound_three_primary_keys?_facet=pk1#facet-pk1` - but I think that's because I'm missing the `ProxyPreserveHost On` setting.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1519#issuecomment-974562942,https://api.github.com/repos/simonw/datasette/issues/1519,974562942,IC_kwDOBm6k_c46FqZ-,9599,simonw,2021-11-20T00:59:32Z,2021-11-20T00:59:32Z,OWNER,"Ouch a nasty bug crept through there - https://datasette-apache-proxy-demo-j7hipcg4aq-uc.a.run.app/prefix/fixtures/compound_three_primary_keys says > 500: name 'ds' is not defined","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974561593,https://api.github.com/repos/simonw/datasette/issues/1519,974561593,IC_kwDOBm6k_c46FqE5,9599,simonw,2021-11-20T00:53:19Z,2021-11-20T00:53:19Z,OWNER,Adding that test found (I hope!) all of the remaining `base_url` bugs. There were a bunch! I think I finally get to close #838 too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974559176,https://api.github.com/repos/simonw/datasette/issues/1519,974559176,IC_kwDOBm6k_c46FpfI,9599,simonw,2021-11-20T00:42:08Z,2021-11-20T00:42:08Z,OWNER,"> In the meantime I can catch these errors by changing the test to run each path twice, once with and once without the prefix. This should accurately simulate how Apache is working here. This worked, I managed to get the tests to fail! Here's the change I made: ```diff diff --git a/tests/test_html.py b/tests/test_html.py index f24165b..dbdfe59 100644 --- a/tests/test_html.py +++ b/tests/test_html.py @@ -1614,12 +1614,19 @@ def test_metadata_sort_desc(app_client): ""/fixtures/compound_three_primary_keys/a,a,a"", ""/fixtures/paginated_view"", ""/fixtures/facetable"", + ""/fixtures?sql=select+1"", ], ) -def test_base_url_config(app_client_base_url_prefix, path): +@pytest.mark.parametrize(""use_prefix"", (True, False)) +def test_base_url_config(app_client_base_url_prefix, path, use_prefix): client = app_client_base_url_prefix - response = client.get(""/prefix/"" + path.lstrip(""/"")) + path_to_get = path + if use_prefix: + path_to_get = ""/prefix/"" + path.lstrip(""/"") + response = client.get(path_to_get) soup = Soup(response.body, ""html.parser"") + if path == ""/fixtures?sql=select+1"": + assert False for el in soup.findAll([""a"", ""link"", ""script""]): if ""href"" in el.attrs: href = el[""href""] @@ -1642,11 +1649,12 @@ def test_base_url_config(app_client_base_url_prefix, path): # If this has been made absolute it may start http://localhost/ if href.startswith(""http://localhost/""): href = href[len(""http://localost/"") :] - assert href.startswith(""/prefix/""), { + assert href.startswith(""/prefix/""), json.dumps({ ""path"": path, + ""path_to_get"": path_to_get, ""href_or_src"": href, ""element_parent"": str(el.parent), - } + }, indent=4, default=repr) def test_base_url_affects_metadata_extra_css_urls(app_client_base_url_prefix): ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974558267,https://api.github.com/repos/simonw/datasette/issues/1519,974558267,IC_kwDOBm6k_c46FpQ7,9599,simonw,2021-11-20T00:37:57Z,2021-11-20T00:37:57Z,OWNER,Thanks to #1522 I have a live demo that exhibits this bug now: https://apache-proxy-demo.datasette.io/prefix/fixtures/attraction_characteristic,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1522#issuecomment-974558076,https://api.github.com/repos/simonw/datasette/issues/1522,974558076,IC_kwDOBm6k_c46FpN8,9599,simonw,2021-11-20T00:36:56Z,2021-11-20T00:36:56Z,OWNER,That 503 error is _really_ frustrating: I have a deploy running at https://apache-proxy-demo.datasette.io/prefix/ and after a fresh deploy it serves 503 errors for quite a while - then eventually starts working.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974557766,https://api.github.com/repos/simonw/datasette/issues/1522,974557766,IC_kwDOBm6k_c46FpJG,9599,simonw,2021-11-20T00:35:25Z,2021-11-20T00:35:25Z,OWNER,Wrote a TIL about `--build-arg` and Cloud Run: https://til.simonwillison.net/cloudrun/using-build-args-with-cloud-run,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974542348,https://api.github.com/repos/simonw/datasette/issues/1522,974542348,IC_kwDOBm6k_c46FlYM,9599,simonw,2021-11-19T23:41:47Z,2021-11-19T23:44:07Z,OWNER,Do I have to use `cloudbuild.yml` to specify these? https://stackoverflow.com/a/58327340/6083 and https://stackoverflow.com/a/66232670/6083 suggest I do.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974541971,https://api.github.com/repos/simonw/datasette/issues/1522,974541971,IC_kwDOBm6k_c46FlST,9599,simonw,2021-11-19T23:40:32Z,2021-11-19T23:40:32Z,OWNER,"I want to be able to use build arguments to specify which commit version or branch of Datasette to deploy. This is proving hard to work out. I have this in my Dockerfile now: ``` ARG DATASETTE_REF RUN pip install https://github.com/simonw/datasette/archive/${DATASETTE_REF}.zip ``` Which works locally: docker build -t datasette-apache-proxy-demo . \ --build-arg DATASETTE_REF=c617e1769ea27e045b0f2907ef49a9a1244e577d But I can't figure out the right incantation to pass to `gcloud build submit`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974523569,https://api.github.com/repos/simonw/datasette/issues/1522,974523569,IC_kwDOBm6k_c46Fgyx,9599,simonw,2021-11-19T22:51:10Z,2021-11-19T22:51:10Z,OWNER,I wan a GitHub Action which I can manually activate to deploy a new version of that demo... and I want it to bake in the latest release of Datasette so I can use it to demonstrate bug fixes.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974523297,https://api.github.com/repos/simonw/datasette/issues/1522,974523297,IC_kwDOBm6k_c46Fguh,9599,simonw,2021-11-19T22:50:31Z,2021-11-19T22:50:31Z,OWNER,Demo code is now at: https://github.com/simonw/datasette/tree/main/demos/apache-proxy,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974521687,https://api.github.com/repos/simonw/datasette/issues/1522,974521687,IC_kwDOBm6k_c46FgVX,9599,simonw,2021-11-19T22:46:26Z,2021-11-19T22:46:26Z,OWNER,"Oh weird, it started working: https://datasette-apache-proxy-demo-j7hipcg4aq-uc.a.run.app/prefix/fixtures/sortable","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974506401,https://api.github.com/repos/simonw/datasette/issues/1522,974506401,IC_kwDOBm6k_c46Fcmh,9599,simonw,2021-11-19T22:11:51Z,2021-11-19T22:11:51Z,OWNER,"This is frustrating: I have the following Dockerfile: ```dockerfile FROM python:3-alpine RUN apk add --no-cache \ apache2 \ apache2-proxy \ bash RUN pip install datasette ENV TINI_VERSION v0.18.0 ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-static /tini RUN chmod +x /tini # Append this to the end of the default httpd.conf file RUN echo $'ServerName localhost\n\ \n\ \n\ Order deny,allow\n\ Allow from all\n\ \n\ \n\ ProxyPass /prefix/ http://localhost:8001/\n\ Header add X-Proxied-By ""Apache2""' >> /etc/apache2/httpd.conf RUN echo $'Datasette' > /var/www/localhost/htdocs/index.html WORKDIR /app ADD https://latest.datasette.io/fixtures.db /app/fixtures.db RUN echo $'#!/usr/bin/env bash\n\ set -e\n\ \n\ httpd -D FOREGROUND &\n\ datasette fixtures.db --setting base_url ""/prefix/"" -h 0.0.0.0 -p 8001 &\n\ \n\ wait -n' > /app/start.sh RUN chmod +x /app/start.sh EXPOSE 80 ENTRYPOINT [""/tini"", ""--"", ""/app/start.sh""] ``` It works fine when I run it locally: ``` docker build -t datasette-apache-proxy-demo . docker run -p 5000:80 datasette-apache-proxy-demo ``` But when I deploy it to Cloud Run with the following script: ```bash #!/bin/bash # https://til.simonwillison.net/cloudrun/ship-dockerfile-to-cloud-run NAME=""datasette-apache-proxy-demo"" PROJECT=$(gcloud config get-value project) IMAGE=""gcr.io/$PROJECT/$NAME"" gcloud builds submit --tag $IMAGE gcloud run deploy \ --allow-unauthenticated \ --platform=managed \ --image $IMAGE $NAME \ --port 80 ``` It serves the `/` page successfully, but hits to `/prefix/` return the following 503 error: > Service Unavailable > > The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later. > > Apache/2.4.51 (Unix) Server at datasette-apache-proxy-demo-j7hipcg4aq-uc.a.run.app Port 80 Cloud Run logs: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1519#issuecomment-974478126,https://api.github.com/repos/simonw/datasette/issues/1519,974478126,IC_kwDOBm6k_c46FVsu,9599,simonw,2021-11-19T21:16:36Z,2021-11-19T21:16:36Z,OWNER,"In the meantime I can catch these errors by changing the test to run each path twice, once with and once without the prefix. This should accurately simulate how Apache is working here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974477465,https://api.github.com/repos/simonw/datasette/issues/1519,974477465,IC_kwDOBm6k_c46FViZ,9599,simonw,2021-11-19T21:15:30Z,2021-11-19T21:15:30Z,OWNER,"I think what's happening here is Apache is actually making a request to `/fixtures` rather than making a request to `/prefix/fixtures` - and Datasette is replying to requests on both the prefixed and the non-prefixed paths. This is pretty confusing! I think Datasette should ONLY reply to `/prefix/fixtures` instead and return a 404 for `/fixtures` - this would make things a whole lot easier to debug. But shipping that change could break existing deployments. Maybe that should be a breaking change for 1.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974450232,https://api.github.com/repos/simonw/datasette/issues/1519,974450232,IC_kwDOBm6k_c46FO44,9599,simonw,2021-11-19T20:41:53Z,2021-11-19T20:42:19Z,OWNER,https://docs.datasette.io/en/stable/deploying.html#apache-proxy-configuration says I should use `ProxyPreserveHost on`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974447950,https://api.github.com/repos/simonw/datasette/issues/1519,974447950,IC_kwDOBm6k_c46FOVO,9599,simonw,2021-11-19T20:40:19Z,2021-11-19T20:40:19Z,OWNER,"Figured it out! The test is not an accurate recreation of what is happening, because it doesn't simulate a request with a path of `/fixtures` that has been redirected by the proxy to `/prefix/fixtures`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1522#issuecomment-974435661,https://api.github.com/repos/simonw/datasette/issues/1522,974435661,IC_kwDOBm6k_c46FLVN,9599,simonw,2021-11-19T20:33:42Z,2021-11-19T20:33:42Z,OWNER,"Should just be a case of deploying this `Dockerfile`: ```Dockerfile FROM python:3-alpine RUN apk add --no-cache \ apache2 \ apache2-proxy \ bash RUN pip install datasette ENV TINI_VERSION v0.18.0 ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-static /tini RUN chmod +x /tini # Append this to the end of the default httpd.conf file RUN echo $'ServerName localhost\n\ \n\ \n\ Order deny,allow\n\ Allow from all\n\ \n\ \n\ ProxyPass /foo/bar/ http://localhost:9000/\n\ Header add X-Proxied-By ""Apache2""' >> /etc/apache2/httpd.conf RUN echo $'Datasette' > /var/www/localhost/htdocs/index.html WORKDIR /app ADD https://latest.datasette.io/fixtures.db /app/fixtures.db RUN echo $'#!/usr/bin/env bash\n\ set -e\n\ \n\ httpd -D FOREGROUND &\n\ datasette fixtures.db --setting base_url ""/foo/bar/"" -p 9000 &\n\ \n\ wait -n' > /app/start.sh RUN chmod +x /app/start.sh EXPOSE 80 ENTRYPOINT [""/tini"", ""--"", ""/app/start.sh""] ``` I can follow this TIL: https://til.simonwillison.net/cloudrun/ship-dockerfile-to-cloud-run","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1521#issuecomment-974433520,https://api.github.com/repos/simonw/datasette/issues/1521,974433520,IC_kwDOBm6k_c46FKzw,9599,simonw,2021-11-19T20:32:29Z,2021-11-19T20:32:29Z,OWNER,This configuration works great.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy, https://github.com/simonw/datasette/issues/1519#issuecomment-974433320,https://api.github.com/repos/simonw/datasette/issues/1519,974433320,IC_kwDOBm6k_c46FKwo,9599,simonw,2021-11-19T20:32:04Z,2021-11-19T20:32:04Z,OWNER,Still not clear why the tests pass but the live example fails.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974433206,https://api.github.com/repos/simonw/datasette/issues/1519,974433206,IC_kwDOBm6k_c46FKu2,9599,simonw,2021-11-19T20:31:52Z,2021-11-19T20:31:52Z,OWNER,"Modified my `Dockerfile` to do this: RUN pip install https://github.com/simonw/datasette/archive/ff0dd4da38d48c2fa9250ecf336002c9ed724e36.zip And now the `request` in that debug `?_context=1` looks like this: ``` ""request"": """" ``` That explains the bug - that request doesn't maintain the original path prefix of `http://localhost:5000/foo/bar/fixtures?sql=` (also it's been rewritten to `localhost:9000` instead of `localhost:5000`).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974422829,https://api.github.com/repos/simonw/datasette/issues/1519,974422829,IC_kwDOBm6k_c46FIMt,9599,simonw,2021-11-19T20:26:35Z,2021-11-19T20:26:35Z,OWNER,"In the `?_context=` debug view the request looks like this: ``` ""request"": """", ``` I'm going to add a `repr()` to it such that it's a bit more useful.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974420619,https://api.github.com/repos/simonw/datasette/issues/1519,974420619,IC_kwDOBm6k_c46FHqL,9599,simonw,2021-11-19T20:25:19Z,2021-11-19T20:25:19Z,OWNER,"The implementations of `path_with_removed_args` and `path_with_format`: https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/utils/__init__.py#L228-L254 https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/utils/__init__.py#L710-L729","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974418496,https://api.github.com/repos/simonw/datasette/issues/1519,974418496,IC_kwDOBm6k_c46FHJA,9599,simonw,2021-11-19T20:24:16Z,2021-11-19T20:24:16Z,OWNER,"Here's the code that generates `edit_sql_url` correctly: https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/views/database.py#L416-L420 And here's the code for `show_hide_link`: https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/views/database.py#L432-L433 And for `url_csv`: https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/views/base.py#L600-L602","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974398399,https://api.github.com/repos/simonw/datasette/issues/1519,974398399,IC_kwDOBm6k_c46FCO_,9599,simonw,2021-11-19T20:08:20Z,2021-11-19T20:22:02Z,OWNER,"The relevant test is this one: https://github.com/simonw/datasette/blob/30255055150d7bc0affc8156adc18295495020ff/tests/test_html.py#L1608-L1649 I modified that test to add `""/fixtures/facetable?sql=select+1""` as one of the tested paths, and dropped in an `assert False` to pause it in the debugger: ``` @pytest.mark.parametrize( ""path"", [ ""/"", ""/fixtures"", ""/fixtures/compound_three_primary_keys"", ""/fixtures/compound_three_primary_keys/a,a,a"", ""/fixtures/paginated_view"", ""/fixtures/facetable"", ""/fixtures?sql=select+1"", ], ) def test_base_url_config(app_client_base_url_prefix, path): client = app_client_base_url_prefix response = client.get(""/prefix/"" + path.lstrip(""/"")) soup = Soup(response.body, ""html.parser"") if path == ""/fixtures?sql=select+1"": > assert False E assert False ``` BUT... in the debugger: ``` (Pdb) print(soup) ...

This data as json, testall, testnone, testresponse, CSV

``` Those all have the correct prefix! But that's not what I'm seeing in my `Dockerfile` reproduction of the issue. Something very weird is going on here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974405016,https://api.github.com/repos/simonw/datasette/issues/1519,974405016,IC_kwDOBm6k_c46FD2Y,9599,simonw,2021-11-19T20:14:19Z,2021-11-19T20:15:05Z,OWNER,"I added `template_debug` in the Dockerfile: ``` datasette fixtures.db --setting template_debug 1 --setting base_url ""/foo/bar/"" -p 9000 &\n\ ``` And then hit `http://localhost:5000/foo/bar/fixtures?sql=select+*+from+compound_three_primary_keys+limit+1&_context=1` to view the template context - and it showed the bug, output edited to just show relevant keys: ```json { ""edit_sql_url"": ""/foo/bar/fixtures?sql=select+%2A+from+compound_three_primary_keys+limit+1"", ""settings"": { ""force_https_urls"": false, ""template_debug"": true, ""trace_debug"": false, ""base_url"": ""/foo/bar/"" }, ""show_hide_link"": ""/fixtures?sql=select+%2A+from+compound_three_primary_keys+limit+1&_context=1&_hide_sql=1"", ""show_hide_text"": ""hide"", ""show_hide_hidden"": """", ""renderers"": { ""json"": ""/fixtures.json?sql=select+*+from+compound_three_primary_keys+limit+1&_context=1"" }, ""url_csv"": ""/fixtures.csv?sql=select+*+from+compound_three_primary_keys+limit+1&_context=1&_size=max"", ""url_csv_path"": ""/fixtures.csv"", ""base_url"": ""/foo/bar/"" } ``` This is so strange. `edit_sql_url` and `base_url` are correct, but `show_hide_link` and `url_csv` and `renderers.json` are not. And it's _really strange_ that the bug doesn't show up in the tests.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974391204,https://api.github.com/repos/simonw/datasette/issues/1519,974391204,IC_kwDOBm6k_c46FAek,9599,simonw,2021-11-19T20:02:41Z,2021-11-19T20:02:41Z,OWNER,"Bug confirmed: ![proxy-bug](https://user-images.githubusercontent.com/9599/142684666-112136bf-9243-4b6e-8202-339fcfe91bcc.gif) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974389472,https://api.github.com/repos/simonw/datasette/issues/1519,974389472,IC_kwDOBm6k_c46FADg,9599,simonw,2021-11-19T20:01:02Z,2021-11-19T20:01:02Z,OWNER,I now have a `Dockerfile` in https://github.com/simonw/datasette/issues/1521#issuecomment-974388295 that I can use to run a local Apache 2 with `mod_proxy` to investigate this class of bugs!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1521#issuecomment-974388295,https://api.github.com/repos/simonw/datasette/issues/1521,974388295,IC_kwDOBm6k_c46E_xH,9599,simonw,2021-11-19T20:00:06Z,2021-11-19T20:00:06Z,OWNER,"And this is the version that proxies to a `base_url` of `/foo/bar/`: ```Dockerfile FROM python:3-alpine RUN apk add --no-cache \ apache2 \ apache2-proxy \ bash RUN pip install datasette ENV TINI_VERSION v0.18.0 ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-static /tini RUN chmod +x /tini # Append this to the end of the default httpd.conf file RUN echo $'ServerName localhost\n\ \n\ \n\ Order deny,allow\n\ Allow from all\n\ \n\ \n\ ProxyPass /foo/bar/ http://localhost:9000/\n\ Header add X-Proxied-By ""Apache2""' >> /etc/apache2/httpd.conf RUN echo $'Datasette' > /var/www/localhost/htdocs/index.html WORKDIR /app ADD https://latest.datasette.io/fixtures.db /app/fixtures.db RUN echo $'#!/usr/bin/env bash\n\ set -e\n\ \n\ httpd -D FOREGROUND &\n\ datasette fixtures.db --setting base_url ""/foo/bar/"" -p 9000 &\n\ \n\ wait -n' > /app/start.sh RUN chmod +x /app/start.sh EXPOSE 80 ENTRYPOINT [""/tini"", ""--"", ""/app/start.sh""] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy, https://github.com/simonw/datasette/issues/1521#issuecomment-974380798,https://api.github.com/repos/simonw/datasette/issues/1521,974380798,IC_kwDOBm6k_c46E97-,9599,simonw,2021-11-19T19:54:26Z,2021-11-19T19:54:26Z,OWNER,"Got it working! Here's a `Dockerfile` which runs completely stand-alone (thanks to using the `echo $'` trick to write out the config files it needs) and successfully serves Datasette behind Apache and `mod_proxy`: ```Dockerfile FROM python:3-alpine RUN apk add --no-cache \ apache2 \ apache2-proxy \ bash RUN pip install datasette ENV TINI_VERSION v0.18.0 ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-static /tini RUN chmod +x /tini # Append this to the end of the default httpd.conf file RUN echo $'ServerName localhost\n\ \n\ \n\ Order deny,allow\n\ Allow from all\n\ \n\ \n\ ProxyPass / http://localhost:9000/\n\ ProxyPassReverse / http://localhost:9000/\n\ Header add X-Proxied-By ""Apache2""' >> /etc/apache2/httpd.conf WORKDIR /app RUN echo $'#!/usr/bin/env bash\n\ set -e\n\ \n\ httpd -D FOREGROUND &\n\ datasette -p 9000 &\n\ \n\ wait -n' > /app/start.sh RUN chmod +x /app/start.sh EXPOSE 80 ENTRYPOINT [""/tini"", ""--"", ""/app/start.sh""] ``` Run it like this: ``` docker build -t datasette-apache2-proxy . docker run -p 5000:80 --rm datasette-apache2-proxy ``` Then run this to confirm: ``` ~ % curl -i 'http://localhost:5000/-/versions.json' HTTP/1.1 200 OK Date: Fri, 19 Nov 2021 19:54:05 GMT Server: uvicorn content-type: application/json; charset=utf-8 X-Proxied-By: Apache2 Transfer-Encoding: chunked {""python"": {""version"": ""3.10.0"", ""full"": ""3.10.0 (default, Nov 13 2021, 03:23:03) [GCC 10.3.1 20210424]""}, ""datasette"": {""version"": ""0.59.2""}, ""asgi"": ""3.0"", ""uvicorn"": ""0.15.0"", ""sqlite"": {""version"": ""3.35.5"", ""fts_versions"": [""FTS5"", ""FTS4"", ""FTS3""], ""extensions"": {""json1"": null}, ""compile_options"": [""COMPILER=gcc-10.3.1 20210424"", ""ENABLE_COLUMN_METADATA"", ""ENABLE_DBSTAT_VTAB"", ""ENABLE_FTS3"", ""ENABLE_FTS3_PARENTHESIS"", ""ENABLE_FTS4"", ""ENABLE_FTS5"", ""ENABLE_GEOPOLY"", ""ENABLE_JSON1"", ""ENABLE_MATH_FUNCTIONS"", ""ENABLE_RTREE"", ""ENABLE_UNLOCK_NOTIFY"", ""MAX_VARIABLE_NUMBER=250000"", ""SECURE_DELETE"", ""THREADSAFE=1"", ""USE_URI""]}} ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy, https://github.com/simonw/datasette/issues/1521#issuecomment-974371116,https://api.github.com/repos/simonw/datasette/issues/1521,974371116,IC_kwDOBm6k_c46E7ks,9599,simonw,2021-11-19T19:45:47Z,2021-11-19T19:45:47Z,OWNER,"https://github.com/krallin/tini says: > *NOTE: If you are using Docker 1.13 or greater, Tini is included in Docker itself. This includes all versions of Docker CE. To enable Tini, just [pass the `--init` flag to `docker run`](https://docs.docker.com/engine/reference/commandline/run/).*","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy, https://github.com/simonw/datasette/issues/1521#issuecomment-974336020,https://api.github.com/repos/simonw/datasette/issues/1521,974336020,IC_kwDOBm6k_c46EzAU,9599,simonw,2021-11-19T19:10:48Z,2021-11-19T19:10:48Z,OWNER,"There's a promising looking minimal Apache 2 proxy config here: https://stackoverflow.com/questions/26474476/minimal-configuration-for-apache-reverse-proxy-in-docker-container ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy, https://github.com/simonw/datasette/issues/1521#issuecomment-974334278,https://api.github.com/repos/simonw/datasette/issues/1521,974334278,IC_kwDOBm6k_c46EylG,9599,simonw,2021-11-19T19:08:09Z,2021-11-19T19:08:09Z,OWNER,"Stripping comments using this StackOverflow recipe: https://unix.stackexchange.com/a/157619 docker run -it --entrypoint sh alpine-apache2-sh \ -c ""cat /etc/apache2/httpd.conf"" | sed '/^[[:blank:]]*#/d;s/#.*//' Result is here: https://gist.github.com/simonw/0a05090df5fcff8e8b3334621fa17976","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy, https://github.com/simonw/datasette/issues/1521#issuecomment-974332787,https://api.github.com/repos/simonw/datasette/issues/1521,974332787,IC_kwDOBm6k_c46EyNz,9599,simonw,2021-11-19T19:05:52Z,2021-11-19T19:05:52Z,OWNER,"Made myself this Dockerfile to let me explore a bit: ```Dockerfile FROM python:3-alpine RUN apk add --no-cache \ apache2 CMD [""sh""] ``` Then: ``` % docker run alpine-apache2-sh % docker run -it alpine-apache2-sh / # ls /etc/apache2/httpd.conf /etc/apache2/httpd.conf / # cat /etc/apache2/httpd.conf # # This is the main Apache HTTP server configuration file. It contains the # configuration directives that give the server its instructions. ... ``` Copying that into a GIST like so: ``` docker run -it --entrypoint sh alpine-apache2-sh -c ""cat /etc/apache2/httpd.conf"" | pbcopy ``` Gist here: https://gist.github.com/simonw/5ea0db6049192cb9f761fbd6beb3a84a","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy, https://github.com/simonw/datasette/issues/1521#issuecomment-974327812,https://api.github.com/repos/simonw/datasette/issues/1521,974327812,IC_kwDOBm6k_c46ExAE,9599,simonw,2021-11-19T18:58:49Z,2021-11-19T18:59:55Z,OWNER,"From this example: https://github.com/tigelane/dockerfiles/blob/06cff2ac8cdc920ebd64f50965115eaa3d0afb84/Alpine-Apache2/Dockerfile#L25-L31 it looks like running `apk add apache2` installs a config file at `/etc/apache2/httpd.conf` - so one approach is to then modify that file. ``` # APACHE - Alpine ################# RUN apk --update add apache2 php5-apache2 && \ #apk add openrc --no-cache && \ rm -rf /var/cache/apk/* && \ sed -i 's/#ServerName www.example.com:80/ServerName localhost/' /etc/apache2/httpd.conf && \ mkdir -p /run/apache2/ # Upload our files from folder ""dist"". COPY dist /var/www/localhost/htdocs # Manually set up the apache environment variables ENV APACHE_RUN_USER www-data ENV APACHE_RUN_GROUP www-data ENV APACHE_LOG_DIR /var/log/apache2 ENV APACHE_LOCK_DIR /var/lock/apache2 ENV APACHE_PID_FILE /var/run/apache2.pid # Execute apache2 on run ######################## EXPOSE 80 ENTRYPOINT [""httpd""] CMD [""-D"", ""FOREGROUND""] ``` I think I'll create my own separate copy and modify that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy, https://github.com/simonw/datasette/issues/1521#issuecomment-974321391,https://api.github.com/repos/simonw/datasette/issues/1521,974321391,IC_kwDOBm6k_c46Evbv,9599,simonw,2021-11-19T18:49:15Z,2021-11-19T18:57:18Z,OWNER,"This pattern looks like it can help: https://ahmet.im/blog/cloud-run-multiple-processes-easy-way/ - see example in https://github.com/ahmetb/multi-process-container-lazy-solution I got that demo working locally like this: ```bash cd /tmp git clone https://github.com/ahmetb/multi-process-container-lazy-solution cd multi-process-container-lazy-solution docker build -t multi-process-container-lazy-solution . docker run -p 5000:8080 --rm multi-process-container-lazy-solution ``` I want to use `apache2` rather than `nginx` though. I found a few relevant examples of Apache in Alpine: - https://github.com/Hacking-Lab/alpine-apache2-reverse-proxy/blob/master/Dockerfile - https://www.sentiatechblog.com/running-apache-in-a-docker-container - https://github.com/search?l=Dockerfile&q=alpine+apache2&type=code ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy, https://github.com/simonw/datasette/issues/1521#issuecomment-974322178,https://api.github.com/repos/simonw/datasette/issues/1521,974322178,IC_kwDOBm6k_c46EvoC,9599,simonw,2021-11-19T18:50:22Z,2021-11-19T18:50:22Z,OWNER,"I'll get this working on my laptop first, but then I want to get it up and running on Cloud Run - maybe with a GitHub Actions workflow in this repo that re-deploys it on manual execution.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy, https://github.com/simonw/datasette/issues/1519#issuecomment-974310208,https://api.github.com/repos/simonw/datasette/issues/1519,974310208,IC_kwDOBm6k_c46EstA,9599,simonw,2021-11-19T18:32:31Z,2021-11-19T18:32:31Z,OWNER,Having a live demo running on Cloud Run that proxies through Apache and uses `base_url` would be incredibly useful for replicating and debugging this kind of thing. I wonder how hard it is to run Apache and `mod_proxy` in the same Docker container as Datasette?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974309591,https://api.github.com/repos/simonw/datasette/issues/1519,974309591,IC_kwDOBm6k_c46EsjX,9599,simonw,2021-11-19T18:31:32Z,2021-11-19T18:31:32Z,OWNER,"`base_url` has been a source of so many bugs like this! I often find them quite hard to replicate, likely because I haven't made myself a good Apache `mod_proxy` testing environment yet.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1520#issuecomment-974308215,https://api.github.com/repos/simonw/datasette/issues/1520,974308215,IC_kwDOBm6k_c46EsN3,9599,simonw,2021-11-19T18:29:26Z,2021-11-19T18:29:26Z,OWNER,"The solution that jumps to mind first is that it would be neat if routes could return something that meant ""actually my bad, I can't handle this after all - move to the next one in the list"". A related idea: it might be useful for custom views like my one here to say ""no actually call the default view for this, but give me back the response so I can modify it in some way"". Kind of like Django or ASGI middleware.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058803238,Pattern for avoiding accidental URL over-rides, https://github.com/simonw/datasette/issues/1518#issuecomment-974300823,https://api.github.com/repos/simonw/datasette/issues/1518,974300823,IC_kwDOBm6k_c46EqaX,9599,simonw,2021-11-19T18:18:32Z,2021-11-19T18:18:32Z,OWNER,"> This may be an argument for continuing to allow non-JSON-objects through to the HTML templates. Need to think about that a bit more. I can definitely support this using pure-JSON - I could make two versions of the row available, one that's an array of cell objects and the other that's an object mapping column names to column raw values.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-974285803,https://api.github.com/repos/simonw/datasette/issues/1518,974285803,IC_kwDOBm6k_c46Emvr,9599,simonw,2021-11-19T17:56:48Z,2021-11-19T18:14:30Z,OWNER,"Very confused by this piece of code here: https://github.com/simonw/datasette/blob/1c13e1af0664a4dfb1e69714c56523279cae09e4/datasette/views/table.py#L37-L63 I added it in https://github.com/simonw/datasette/commit/754836eef043676e84626c4fd3cb993eed0d2976 - in the new world that should probably be replaced by pure JSON. Aha - this comment explains it: https://github.com/simonw/datasette/issues/521#issuecomment-505279560 > I think the trick is to redefine what a ""cell_row"" is. Each row is currently a list of cells: > > https://github.com/simonw/datasette/blob/6341f8cbc7833022012804dea120b838ec1f6558/datasette/views/table.py#L159-L163 > > I can redefine the row (the `cells` variable in the above example) as a thing-that-iterates-cells (hence behaving like a list) but that also supports `__getitem__` access for looking up cell values if you know the name of the column. The goal was to support neater custom templates like this: ```html+jinja {% for row in display_rows %}

{{ row[""First_Name""] }} {{ row[""Last_Name""] }}

... ``` This may be an argument for continuing to allow non-JSON-objects through to the HTML templates. Need to think about that a bit more.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-974287570,https://api.github.com/repos/simonw/datasette/issues/1518,974287570,IC_kwDOBm6k_c46EnLS,9599,simonw,2021-11-19T17:59:33Z,2021-11-19T17:59:33Z,OWNER,"I'm going to try leaning into the `asyncinject` mechanism a bit here. One method can execute and return the raw rows. Another can turn that into the default minimal JSON representation. Then a third can take that (or take both) and use it to inflate out the JSON that the HTML template needs, with those extras and with the rendered cells from plugins.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/sqlite-utils/issues/342#issuecomment-973820125,https://api.github.com/repos/simonw/sqlite-utils/issues/342,973820125,IC_kwDOCGYnMM46C1Dd,9599,simonw,2021-11-19T07:25:55Z,2021-11-19T07:25:55Z,OWNER,"`alter=True` doesn't make sense to support here either, because `.lookup()` already adds missing columns: https://github.com/simonw/sqlite-utils/blob/3b8abe608796e99e4ffc5f3f4597a85e605c0e9b/sqlite_utils/db.py#L2743-L2746","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058196641,Extra options to `lookup()` which get passed to `insert()`, https://github.com/simonw/sqlite-utils/issues/342#issuecomment-973802998,https://api.github.com/repos/simonw/sqlite-utils/issues/342,973802998,IC_kwDOCGYnMM46Cw32,9599,simonw,2021-11-19T06:59:22Z,2021-11-19T06:59:32Z,OWNER,"I don't think I need the `DEFAULT` defaults for `.insert()` either, since it just passes through to `.insert()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058196641,Extra options to `lookup()` which get passed to `insert()`, https://github.com/simonw/sqlite-utils/issues/342#issuecomment-973802766,https://api.github.com/repos/simonw/sqlite-utils/issues/342,973802766,IC_kwDOCGYnMM46Cw0O,9599,simonw,2021-11-19T06:58:45Z,2021-11-19T06:58:45Z,OWNER,"And neither does `hash_id`. On that basis I'm going to specifically list the ones that DO make sense, and hope that I remember to add any new ones in the future. I can add a code comment hint to `.insert()` about that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058196641,Extra options to `lookup()` which get passed to `insert()`, https://github.com/simonw/sqlite-utils/issues/342#issuecomment-973802469,https://api.github.com/repos/simonw/sqlite-utils/issues/342,973802469,IC_kwDOCGYnMM46Cwvl,9599,simonw,2021-11-19T06:58:03Z,2021-11-19T06:58:03Z,OWNER,Also: I don't think `ignore=` and `replace=` make sense in the context of `lookup()`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058196641,Extra options to `lookup()` which get passed to `insert()`, https://github.com/simonw/sqlite-utils/issues/342#issuecomment-973802308,https://api.github.com/repos/simonw/sqlite-utils/issues/342,973802308,IC_kwDOCGYnMM46CwtE,9599,simonw,2021-11-19T06:57:37Z,2021-11-19T06:57:37Z,OWNER,"Here's the current full method signature for `.insert()`: https://github.com/simonw/sqlite-utils/blob/3b8abe608796e99e4ffc5f3f4597a85e605c0e9b/sqlite_utils/db.py#L2462-L2477 I could add a test which uses introspection (`inspect.signature(method).parameters`) to confirm that `.lookup()` has a super-set of the arguments accepted by `.insert()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058196641,Extra options to `lookup()` which get passed to `insert()`, https://github.com/simonw/sqlite-utils/issues/342#issuecomment-973801650,https://api.github.com/repos/simonw/sqlite-utils/issues/342,973801650,IC_kwDOCGYnMM46Cwiy,9599,simonw,2021-11-19T06:55:56Z,2021-11-19T06:55:56Z,OWNER,"`pk` needs to be an explicit argument to `.lookup()`. The rest could be `**kwargs` passed through to `.insert()`, like this hacked together version (docstring removed for brevity): ```python def lookup( self, lookup_values: Dict[str, Any], extra_values: Optional[Dict[str, Any]] = None, pk=""id"", **insert_kwargs, ): """""" assert isinstance(lookup_values, dict) if extra_values is not None: assert isinstance(extra_values, dict) combined_values = dict(lookup_values) if extra_values is not None: combined_values.update(extra_values) if self.exists(): self.add_missing_columns([combined_values]) unique_column_sets = [set(i.columns) for i in self.indexes] if set(lookup_values.keys()) not in unique_column_sets: self.create_index(lookup_values.keys(), unique=True) wheres = [""[{}] = ?"".format(column) for column in lookup_values] rows = list( self.rows_where( "" and "".join(wheres), [value for _, value in lookup_values.items()] ) ) try: return rows[0][pk] except IndexError: return self.insert(combined_values, pk=pk, **insert_kwargs).last_pk else: pk = self.insert(combined_values, pk=pk, **insert_kwargs).last_pk self.create_index(lookup_values.keys(), unique=True) return pk ``` I think I'll explicitly list the parameters, mainly so they can be typed and covered by automatic documentation. I do worry that I'll add more keyword arguments to `.insert()` in the future and forget to mirror them to `.lookup()` though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058196641,Extra options to `lookup()` which get passed to `insert()`, https://github.com/simonw/sqlite-utils/issues/342#issuecomment-973800795,https://api.github.com/repos/simonw/sqlite-utils/issues/342,973800795,IC_kwDOCGYnMM46CwVb,9599,simonw,2021-11-19T06:54:08Z,2021-11-19T06:54:08Z,OWNER,"Looking at the code for `lookup()` it currently hard-codes `pk` to `""id""` - but it actually only calls `.insert()` in two places, both of which could be passed extra arguments. https://github.com/simonw/sqlite-utils/blob/3b8abe608796e99e4ffc5f3f4597a85e605c0e9b/sqlite_utils/db.py#L2756-L2763","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058196641,Extra options to `lookup()` which get passed to `insert()`, https://github.com/simonw/datasette/issues/1518#issuecomment-973700549,https://api.github.com/repos/simonw/datasette/issues/1518,973700549,IC_kwDOBm6k_c46CX3F,9599,simonw,2021-11-19T03:31:20Z,2021-11-19T03:31:26Z,OWNER,"... and while I'm doing all of this I can rewrite the templates to not use those cheating magical functions AND document the template context at the same time, refs: - #1510.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-973700322,https://api.github.com/repos/simonw/datasette/issues/1518,973700322,IC_kwDOBm6k_c46CXzi,9599,simonw,2021-11-19T03:30:30Z,2021-11-19T03:30:30Z,OWNER,"Right now the HTML version gets to cheat - it passes through objects that are not JSON serializable, including custom functions that can then be called by Jinja. I'm interested in maybe removing this cheating - if the HTML version could only request JSON-serializable extras those could be exposed in the API as well. It would also help cleanup the kind-of-nasty pattern I use in the current `BaseView` where everything returns both a bunch of JSON-serializable data AND an awaitable function that then gets to add extra things to the HTML context.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-973698917,https://api.github.com/repos/simonw/datasette/issues/1518,973698917,IC_kwDOBm6k_c46CXdl,9599,simonw,2021-11-19T03:26:18Z,2021-11-19T03:29:03Z,OWNER,"A (likely incomplete) list of features on the table page: - [ ] Display table/database/instance metadata - [ ] Show count of all results - [ ] Display table of results - [ ] Special table display treatment for URLs, numbers - [ ] Allow plugins to modify table cells - [ ] Respect `?_col=` and `?_nocol=` - [ ] Show interface for filtering by columns and operations - [ ] Show search box, support executing FTS searches - [ ] Sort table by specified column - [ ] Paginate table - [ ] Show facet results - [ ] Show suggested facets - [ ] Link to available exports - [ ] Display schema for table - [ ] Maybe it should show the SQL for the query too? - [ ] Handle various non-obvious querystring options, like `?_where=` and `?_through=`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-973699424,https://api.github.com/repos/simonw/datasette/issues/1518,973699424,IC_kwDOBm6k_c46CXlg,9599,simonw,2021-11-19T03:27:49Z,2021-11-19T03:27:49Z,OWNER,"My goal is to break up a lot of this functionality into separate methods. These methods can be executed in parallel by `asyncinject`, but more importantly they can be used to build a much better JSON representation, where the default representation is lighter and `?_extra=x` options can be used to execute more expensive portions and add them to the response. So the HTML version itself needs to be re-written to use those JSON extras.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1517#issuecomment-973696604,https://api.github.com/repos/simonw/datasette/issues/1517,973696604,IC_kwDOBm6k_c46CW5c,9599,simonw,2021-11-19T03:20:00Z,2021-11-19T03:20:00Z,OWNER,Confirmed - my test plugin is indeed correctly over-riding the table page.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1057996111,Let `register_routes()` over-ride default routes within Datasette, https://github.com/simonw/datasette/issues/1518#issuecomment-973687978,https://api.github.com/repos/simonw/datasette/issues/1518,973687978,IC_kwDOBm6k_c46CUyq,9599,simonw,2021-11-19T03:07:47Z,2021-11-19T03:07:47Z,OWNER,"I was wrong about that, you CAN over-ride default routes already.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1517#issuecomment-973686874,https://api.github.com/repos/simonw/datasette/issues/1517,973686874,IC_kwDOBm6k_c46CUha,9599,simonw,2021-11-19T03:06:58Z,2021-11-19T03:06:58Z,OWNER,"I made a mistake: I just wrote a test that proves that plugins CAN over-ride default routes, plus if you look at the code here the plugins get to register themselves first: https://github.com/simonw/datasette/blob/0156c6b5e52d541e93f0d68e9245f20ae83bc933/datasette/app.py#L965-L981","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1057996111,Let `register_routes()` over-ride default routes within Datasette, https://github.com/simonw/datasette/issues/1518#issuecomment-973682389,https://api.github.com/repos/simonw/datasette/issues/1518,973682389,IC_kwDOBm6k_c46CTbV,9599,simonw,2021-11-19T02:57:39Z,2021-11-19T02:57:39Z,OWNER,"Ideally I'd like to execute the existing test suite against the new implementation - that would require me to solve this so I can replace the view with the plugin version though: - #1517 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-973681970,https://api.github.com/repos/simonw/datasette/issues/1518,973681970,IC_kwDOBm6k_c46CTUy,9599,simonw,2021-11-19T02:56:31Z,2021-11-19T02:56:53Z,OWNER,"Here's where I got to with my hacked-together initial plugin prototype - it managed to render the table page with some rows on it (and a bunch of missing functionality such as filters): https://gist.github.com/simonw/281eac9c73b062c3469607ad86470eb2 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/878#issuecomment-973678931,https://api.github.com/repos/simonw/datasette/issues/878,973678931,IC_kwDOBm6k_c46CSlT,9599,simonw,2021-11-19T02:51:17Z,2021-11-19T02:51:17Z,OWNER,"OK, I managed to get a table to render! Here's the code I used - I had to copy a LOT of stuff. https://gist.github.com/simonw/281eac9c73b062c3469607ad86470eb2 I'm going to move this work into a new, separate issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/878#issuecomment-973635157,https://api.github.com/repos/simonw/datasette/issues/878,973635157,IC_kwDOBm6k_c46CH5V,9599,simonw,2021-11-19T01:07:08Z,2021-11-19T01:07:08Z,OWNER,"This exercise is proving so useful in getting my head around how the enormous and complex `TableView` class works again. Here's where I've got to now - I'm systematically working through the variables that are returned for HTML and for JSON copying across code to get it to work: ```python from datasette.database import QueryInterrupted from datasette.utils import escape_sqlite from datasette.utils.asgi import Response, NotFound, Forbidden from datasette.views.base import DatasetteError from datasette import hookimpl from asyncinject import AsyncInject, inject from pprint import pformat class Table(AsyncInject): @inject async def database(self, request, datasette): # TODO: all that nasty hash resolving stuff can go here db_name = request.url_vars[""db_name""] try: db = datasette.databases[db_name] except KeyError: raise NotFound(f""Database '{db_name}' does not exist"") return db @inject async def table_and_format(self, request, database, datasette): table_and_format = request.url_vars[""table_and_format""] # TODO: be a lot smarter here if ""."" in table_and_format: return table_and_format.split(""."", 2) else: return table_and_format, ""html"" @inject async def main(self, request, database, table_and_format, datasette): # TODO: if this is actually a canned query, dispatch to it table, format = table_and_format is_view = bool(await database.get_view_definition(table)) table_exists = bool(await database.table_exists(table)) if not is_view and not table_exists: raise NotFound(f""Table not found: {table}"") await check_permissions( datasette, request, [ (""view-table"", (database.name, table)), (""view-database"", database.name), ""view-instance"", ], ) private = not await datasette.permission_allowed( None, ""view-table"", (database.name, table), default=True ) pks = await database.primary_keys(table) table_columns = await database.table_columns(table) specified_columns = await columns_to_select(datasette, database, table, request) select_specified_columns = "", "".join( escape_sqlite(t) for t in specified_columns ) select_all_columns = "", "".join(escape_sqlite(t) for t in table_columns) use_rowid = not pks and not is_view if use_rowid: select_specified_columns = f""rowid, {select_specified_columns}"" select_all_columns = f""rowid, {select_all_columns}"" order_by = ""rowid"" order_by_pks = ""rowid"" else: order_by_pks = "", "".join([escape_sqlite(pk) for pk in pks]) order_by = order_by_pks if is_view: order_by = """" nocount = request.args.get(""_nocount"") nofacet = request.args.get(""_nofacet"") if request.args.get(""_shape"") in (""array"", ""object""): nocount = True nofacet = True # Next, a TON of SQL to build where_params and filters and suchlike # skipping that and jumping straight to... where_clauses = [] where_clause = """" if where_clauses: where_clause = f""where {' and '.join(where_clauses)} "" from_sql = ""from {table_name} {where}"".format( table_name=escape_sqlite(table), where=(""where {} "".format("" and "".join(where_clauses))) if where_clauses else """", ) from_sql_params ={} params = {} count_sql = f""select count(*) {from_sql}"" sql_no_order_no_limit = ( ""select {select_all_columns} from {table_name} {where}"".format( select_all_columns=select_all_columns, table_name=escape_sqlite(table), where=where_clause, ) ) page_size = 100 offset = "" offset 0"" sql = ""select {select_specified_columns} from {table_name} {where}{order_by} limit {page_size}{offset}"".format( select_specified_columns=select_specified_columns, table_name=escape_sqlite(table), where=where_clause, order_by=order_by, page_size=page_size + 1, offset=offset, ) # Fetch rows results = await database.execute(sql, params, truncate=True) columns = [r[0] for r in results.description] rows = list(results.rows) # Fetch count filtered_table_rows_count = None if count_sql: try: count_rows = list(await database.execute(count_sql, from_sql_params)) filtered_table_rows_count = count_rows[0][0] except QueryInterrupted: pass vars = { ""json"": { # THIS STUFF is from the regular JSON ""database"": database.name, ""table"": table, ""is_view"": is_view, # ""human_description_en"": human_description_en, ""rows"": rows[:page_size], ""truncated"": results.truncated, ""filtered_table_rows_count"": filtered_table_rows_count, # ""expanded_columns"": expanded_columns, # ""expandable_columns"": expandable_columns, ""columns"": columns, ""primary_keys"": pks, # ""units"": units, ""query"": {""sql"": sql, ""params"": params}, # ""facet_results"": facet_results, # ""suggested_facets"": suggested_facets, # ""next"": next_value and str(next_value) or None, # ""next_url"": next_url, ""private"": private, ""allow_execute_sql"": await datasette.permission_allowed( request.actor, ""execute-sql"", database, default=True ), }, ""html"": { # ... this is the HTML special stuff # ""table_actions"": table_actions, # ""supports_search"": bool(fts_table), # ""search"": search or """", ""use_rowid"": use_rowid, # ""filters"": filters, # ""display_columns"": display_columns, # ""filter_columns"": filter_columns, # ""display_rows"": display_rows, # ""facets_timed_out"": facets_timed_out, # ""sorted_facet_results"": sorted( # facet_results.values(), # key=lambda f: (len(f[""results""]), f[""name""]), # reverse=True, # ), # ""show_facet_counts"": special_args.get(""_facet_size"") == ""max"", # ""extra_wheres_for_ui"": extra_wheres_for_ui, # ""form_hidden_args"": form_hidden_args, # ""is_sortable"": any(c[""sortable""] for c in display_columns), # ""path_with_replaced_args"": path_with_replaced_args, # ""path_with_removed_args"": path_with_removed_args, # ""append_querystring"": append_querystring, ""request"": request, # ""sort"": sort, # ""sort_desc"": sort_desc, ""disable_sort"": is_view, # ""custom_table_templates"": [ # f""_table-{to_css_class(database)}-{to_css_class(table)}.html"", # f""_table-table-{to_css_class(database)}-{to_css_class(table)}.html"", # ""_table.html"", # ], # ""metadata"": metadata, # ""view_definition"": await db.get_view_definition(table), # ""table_definition"": await db.get_table_definition(table), }, } # I'm just trying to get HTML to work for the moment if format == ""json"": return Response.json(dict(vars, locals=locals()), default=repr) else: return Response.html(repr(vars[""html""])) async def view(self, request, datasette): return await self.main(request=request, datasette=datasette) @hookimpl def register_routes(): return [ (r""/t/(?P[^/]+)/(?P[^/]+?$)"", Table().view), ] async def check_permissions(datasette, request, permissions): """"""permissions is a list of (action, resource) tuples or 'action' strings"""""" for permission in permissions: if isinstance(permission, str): action = permission resource = None elif isinstance(permission, (tuple, list)) and len(permission) == 2: action, resource = permission else: assert ( False ), ""permission should be string or tuple of two items: {}"".format( repr(permission) ) ok = await datasette.permission_allowed( request.actor, action, resource=resource, default=None, ) if ok is not None: if ok: return else: raise Forbidden(action) async def columns_to_select(datasette, database, table, request): table_columns = await database.table_columns(table) pks = await database.primary_keys(table) columns = list(table_columns) if ""_col"" in request.args: columns = list(pks) _cols = request.args.getlist(""_col"") bad_columns = [column for column in _cols if column not in table_columns] if bad_columns: raise DatasetteError( ""_col={} - invalid columns"".format("", "".join(bad_columns)), status=400, ) # De-duplicate maintaining order: columns.extend(dict.fromkeys(_cols)) if ""_nocol"" in request.args: # Return all columns EXCEPT these bad_columns = [ column for column in request.args.getlist(""_nocol"") if (column not in table_columns) or (column in pks) ] if bad_columns: raise DatasetteError( ""_nocol={} - invalid columns"".format("", "".join(bad_columns)), status=400, ) tmp_columns = [ column for column in columns if column not in request.args.getlist(""_nocol"") ] columns = tmp_columns return columns ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/878#issuecomment-973568285,https://api.github.com/repos/simonw/datasette/issues/878,973568285,IC_kwDOBm6k_c46B3kd,9599,simonw,2021-11-19T00:29:20Z,2021-11-19T00:29:20Z,OWNER,"This is working! ```python from datasette.utils.asgi import Response from datasette import hookimpl import html from asyncinject import AsyncInject, inject class Table(AsyncInject): @inject async def database(self, request): return request.url_vars[""db_name""] @inject async def main(self, request, database): return Response.html(""Database: {}"".format( html.escape(database) )) async def view(self, request): return await self.main(request=request) @hookimpl def register_routes(): return [ (r""/t/(?P[^/]+)/(?P[^/]+?$)"", Table().view), ] ``` This project will definitely show me if I actually like the `asyncinject` patterns or not.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/878#issuecomment-973564260,https://api.github.com/repos/simonw/datasette/issues/878,973564260,IC_kwDOBm6k_c46B2lk,9599,simonw,2021-11-19T00:27:06Z,2021-11-19T00:27:06Z,OWNER,"Problem: the fancy `asyncinject` stuff inteferes with the fancy Datasette thing that introspects view functions to look for what parameters they take: ```python class Table(asyncinject.AsyncInjectAll): async def view(self, request): return Response.html(""Hello from {}"".format( html.escape(repr(request.url_vars)) )) @hookimpl def register_routes(): return [ (r""/t/(?P[^/]+)/(?P[^/]+?$)"", Table().view), ] ``` This failed with error: ""Table.view() takes 1 positional argument but 2 were given"" So I'm going to use `AsyncInject` and have the `view` function NOT use the `@inject` decorator.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/878#issuecomment-973554024,https://api.github.com/repos/simonw/datasette/issues/878,973554024,IC_kwDOBm6k_c46B0Fo,9599,simonw,2021-11-19T00:21:20Z,2021-11-19T00:21:20Z,OWNER,"That's annoying: it looks like plugins can't use `register_routes()` to over-ride default routes within Datasette itself. This didn't work: ```python from datasette.utils.asgi import Response from datasette import hookimpl import html async def table(request): return Response.html(""Hello from {}"".format( html.escape(repr(request.url_vars)) )) @hookimpl def register_routes(): return [ (r""/(?P[^/]+)/(?P[^/]+?$)"", table), ] ``` I'll use a `/t/` prefix for the moment, but this is probably something I'll fix in Datasette itself later.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/878#issuecomment-973542284,https://api.github.com/repos/simonw/datasette/issues/878,973542284,IC_kwDOBm6k_c46BxOM,9599,simonw,2021-11-19T00:16:44Z,2021-11-19T00:16:44Z,OWNER,"``` Development % cookiecutter gh:simonw/datasette-plugin You've downloaded /Users/simon/.cookiecutters/datasette-plugin before. Is it okay to delete and re-download it? [yes]: yes plugin_name []: table-new description []: New implementation of TableView, see https://github.com/simonw/datasette/issues/878 hyphenated [table-new]: underscored [table_new]: github_username []: simonw author_name []: Simon Willison include_static_directory []: include_templates_directory []: ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/878#issuecomment-973527870,https://api.github.com/repos/simonw/datasette/issues/878,973527870,IC_kwDOBm6k_c46Bts-,9599,simonw,2021-11-19T00:13:43Z,2021-11-19T00:13:43Z,OWNER,"New plan: I'm going to build a brand new implementation of `TableView` starting out as a plugin, using the `register_routes()` plugin hook. It will reuse the existing HTML template but will be a completely new Python implementation, based on `asyncinject`. I'm going to start by just getting the table to show up on the page - then I'll add faceting, suggested facets, filters and so-on. Bonus: I'm going to see if I can get it to work for arbitrary SQL queries too (stretch goal).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/878#issuecomment-971209475,https://api.github.com/repos/simonw/datasette/issues/878,971209475,IC_kwDOBm6k_c4543sD,9599,simonw,2021-11-17T05:41:42Z,2021-11-17T05:41:42Z,OWNER,"I'm going to build a brand new implementation of the `TableView` class that doesn't subclass `BaseView` at all, instead using `asyncinject`. If I'm lucky that will clean up the grungiest part of the codebase. I can maybe even run the tests against old `TableView` and `TableView2` to check that they behave the same.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/878#issuecomment-971057553,https://api.github.com/repos/simonw/datasette/issues/878,971057553,IC_kwDOBm6k_c454SmR,9599,simonw,2021-11-17T01:40:45Z,2021-11-17T01:40:45Z,OWNER,"I shipped that code as a new library, `asyncinject`: https://pypi.org/project/asyncinject/ - I'll open a new PR to attempt to refactor `TableView` to use it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/pull/1512#issuecomment-971056169,https://api.github.com/repos/simonw/datasette/issues/1512,971056169,IC_kwDOBm6k_c454SQp,9599,simonw,2021-11-17T01:39:44Z,2021-11-17T01:39:44Z,OWNER,Closing this PR because I shipped the code in it as a separate library instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055402144,New pattern for async view classes, https://github.com/simonw/datasette/pull/1512#issuecomment-971055677,https://api.github.com/repos/simonw/datasette/issues/1512,971055677,IC_kwDOBm6k_c454SI9,9599,simonw,2021-11-17T01:39:25Z,2021-11-17T01:39:25Z,OWNER,https://github.com/simonw/asyncinject version 0.1a0 is now live on PyPI: https://pypi.org/project/asyncinject/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055402144,New pattern for async view classes, https://github.com/simonw/datasette/pull/1512#issuecomment-971010724,https://api.github.com/repos/simonw/datasette/issues/1512,971010724,IC_kwDOBm6k_c454HKk,9599,simonw,2021-11-17T01:12:22Z,2021-11-17T01:12:22Z,OWNER,I'm going to extract out the `asyncinject` stuff into a separate library.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055402144,New pattern for async view classes, https://github.com/simonw/datasette/pull/1512#issuecomment-970861628,https://api.github.com/repos/simonw/datasette/issues/1512,970861628,IC_kwDOBm6k_c453iw8,9599,simonw,2021-11-16T23:46:07Z,2021-11-16T23:46:07Z,OWNER,"I made the changes locally and tested them with Python 3.6 like so: ``` cd /tmp mkdir v cd v pipenv shell --python=python3.6 cd ~/Dropbox/Development/datasette pip install -e '.[test]' pytest tests/test_asyncdi.py ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055402144,New pattern for async view classes, https://github.com/simonw/datasette/pull/1512#issuecomment-970857411,https://api.github.com/repos/simonw/datasette/issues/1512,970857411,IC_kwDOBm6k_c453hvD,9599,simonw,2021-11-16T23:43:21Z,2021-11-16T23:43:21Z,OWNER,"``` E File ""/home/runner/work/datasette/datasette/datasette/utils/vendored_graphlib.py"", line 56 E if (result := self._node2info.get(node)) is None: E ^ E SyntaxError: invalid syntax ``` Oh no - the vendored code I use has `:=` so doesn't work on Python 3.6! Will have to backport it more thoroughly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055402144,New pattern for async view classes, https://github.com/simonw/datasette/issues/1513#issuecomment-970855084,https://api.github.com/repos/simonw/datasette/issues/1513,970855084,IC_kwDOBm6k_c453hKs,9599,simonw,2021-11-16T23:41:46Z,2021-11-16T23:41:46Z,OWNER,Conclusion: using a giant convoluted CTE and UNION ALL query to attempt to calculate facets at the same time as retrieving rows is a net LOSS for performance! Very surprised to see that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time, https://github.com/simonw/datasette/issues/1513#issuecomment-970853917,https://api.github.com/repos/simonw/datasette/issues/1513,970853917,IC_kwDOBm6k_c453g4d,9599,simonw,2021-11-16T23:41:01Z,2021-11-16T23:41:01Z,OWNER,"One very interesting difference between the two: on the single giant query page: ```json { ""request_duration_ms"": 376.4317020000476, ""sum_trace_duration_ms"": 370.0828700000329, ""num_traces"": 5 } ``` And on the page that uses separate queries: ```json { ""request_duration_ms"": 819.012272000009, ""sum_trace_duration_ms"": 201.52852100000018, ""num_traces"": 19 } ``` The separate pages page takes 819ms total to render the page, but spends 201ms across 19 SQL queries. The single big query takes 376ms total to render the page, spending 370ms in 5 queries
Those 5 queries, if you're interested ```sql select database_name, schema_version from databases PRAGMA schema_version PRAGMA schema_version explain with cte as (\r\n select rowid, date, county, state, fips, cases, deaths\r\n from ny_times_us_counties\r\n),\r\ntruncated as (\r\n select null as _facet, null as facet_name, null as facet_count, rowid, date, county, state, fips, cases, deaths\r\n from cte order by date desc limit 4\r\n),\r\nstate_facet as (\r\n select 'state' as _facet, state as facet_name, count(*) as facet_count,\r\n null, null, null, null, null, null, null\r\n from cte group by facet_name order by facet_count desc limit 3\r\n),\r\nfips_facet as (\r\n select 'fips' as _facet, fips as facet_name, count(*) as facet_count,\r\n null, null, null, null, null, null, null\r\n from cte group by facet_name order by facet_count desc limit 3\r\n),\r\ncounty_facet as (\r\n select 'county' as _facet, county as facet_name, count(*) as facet_count,\r\n null, null, null, null, null, null, null\r\n from cte group by facet_name order by facet_count desc limit 3\r\n)\r\nselect * from truncated\r\nunion all select * from state_facet\r\nunion all select * from fips_facet\r\nunion all select * from county_facet with cte as (\r\n select rowid, date, county, state, fips, cases, deaths\r\n from ny_times_us_counties\r\n),\r\ntruncated as (\r\n select null as _facet, null as facet_name, null as facet_count, rowid, date, county, state, fips, cases, deaths\r\n from cte order by date desc limit 4\r\n),\r\nstate_facet as (\r\n select 'state' as _facet, state as facet_name, count(*) as facet_count,\r\n null, null, null, null, null, null, null\r\n from cte group by facet_name order by facet_count desc limit 3\r\n),\r\nfips_facet as (\r\n select 'fips' as _facet, fips as facet_name, count(*) as facet_count,\r\n null, null, null, null, null, null, null\r\n from cte group by facet_name order by facet_count desc limit 3\r\n),\r\ncounty_facet as (\r\n select 'county' as _facet, county as facet_name, count(*) as facet_count,\r\n null, null, null, null, null, null, null\r\n from cte group by facet_name order by facet_count desc limit 3\r\n)\r\nselect * from truncated\r\nunion all select * from state_facet\r\nunion all select * from fips_facet\r\nunion all select * from county_facet ```
All of that additional non-SQL overhead must be stuff relating to Python and template rendering code running on the page. I'm really surprised at how much overhead that is! This is worth researching separately.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time, https://github.com/simonw/datasette/issues/1513#issuecomment-970845844,https://api.github.com/repos/simonw/datasette/issues/1513,970845844,IC_kwDOBm6k_c453e6U,9599,simonw,2021-11-16T23:35:38Z,2021-11-16T23:35:38Z,OWNER,"I tried adding `cases > 10000` but the SQL query now takes too long - so moving this to my laptop. ``` cd /tmp wget https://covid-19.datasettes.com/covid.db datasette covid.db \ --setting facet_time_limit_ms 10000 \ --setting sql_time_limit_ms 10000 \ --setting trace_debug 1 ``` `http://127.0.0.1:8006/covid/ny_times_us_counties?_trace=1&_facet_size=3&_size=2&cases__gt=10000` shows in the traces: ```json [ { ""type"": ""sql"", ""start"": 12.693033525, ""end"": 12.694056904, ""duration_ms"": 1.0233789999993803, ""traceback"": [ "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/base.py\"", line 262, in get\n return await self.view_get(\n"", "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/base.py\"", line 477, in view_get\n response_or_template_contexts = await self.data(\n"", "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/table.py\"", line 705, in data\n results = await db.execute(sql, params, truncate=True, **extra_args)\n"" ], ""database"": ""covid"", ""sql"": ""select rowid, date, county, state, fips, cases, deaths from ny_times_us_counties where \""cases\"" > :p0 order by rowid limit 3"", ""params"": { ""p0"": 10000 } }, { ""type"": ""sql"", ""start"": 12.694285093, ""end"": 12.814936275, ""duration_ms"": 120.65118200000136, ""traceback"": [ "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/base.py\"", line 262, in get\n return await self.view_get(\n"", "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/base.py\"", line 477, in view_get\n response_or_template_contexts = await self.data(\n"", "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/table.py\"", line 723, in data\n count_rows = list(await db.execute(count_sql, from_sql_params))\n"" ], ""database"": ""covid"", ""sql"": ""select count(*) from ny_times_us_counties where \""cases\"" > :p0"", ""params"": { ""p0"": 10000 } }, { ""type"": ""sql"", ""start"": 12.818812089, ""end"": 12.851172544, ""duration_ms"": 32.360455000000954, ""traceback"": [ "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/table.py\"", line 856, in data\n suggested_facets.extend(await facet.suggest())\n"", "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/facets.py\"", line 164, in suggest\n distinct_values = await self.ds.execute(\n"", "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/app.py\"", line 634, in execute\n return await self.databases[db_name].execute(\n"" ], ""database"": ""covid"", ""sql"": ""select county, count(*) as n from (\n select rowid, date, county, state, fips, cases, deaths from ny_times_us_counties where \""cases\"" > :p0 \n ) where county is not null\n group by county\n limit 4"", ""params"": { ""p0"": 10000 } }, { ""type"": ""sql"", ""start"": 12.851418868, ""end"": 12.871268359, ""duration_ms"": 19.84949100000044, ""traceback"": [ "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/table.py\"", line 856, in data\n suggested_facets.extend(await facet.suggest())\n"", "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/facets.py\"", line 164, in suggest\n distinct_values = await self.ds.execute(\n"", "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/app.py\"", line 634, in execute\n return await self.databases[db_name].execute(\n"" ], ""database"": ""covid"", ""sql"": ""select state, count(*) as n from (\n select rowid, date, county, state, fips, cases, deaths from ny_times_us_counties where \""cases\"" > :p0 \n ) where state is not null\n group by state\n limit 4"", ""params"": { ""p0"": 10000 } }, { ""type"": ""sql"", ""start"": 12.871497655, ""end"": 12.897715027, ""duration_ms"": 26.217371999999628, ""traceback"": [ "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/table.py\"", line 856, in data\n suggested_facets.extend(await facet.suggest())\n"", "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/facets.py\"", line 164, in suggest\n distinct_values = await self.ds.execute(\n"", "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/app.py\"", line 634, in execute\n return await self.databases[db_name].execute(\n"" ], ""database"": ""covid"", ""sql"": ""select fips, count(*) as n from (\n select rowid, date, county, state, fips, cases, deaths from ny_times_us_counties where \""cases\"" > :p0 \n ) where fips is not null\n group by fips\n limit 4"", ""params"": { ""p0"": 10000 } } ] ``` So that's: ``` fetch rows: 1.0233789999993803 ms count: 120.65118200000136 ms facet county: 32.360455000000954 ms facet state: 19.84949100000044 ms facet fips: 26.217371999999628 ms ``` = 200.1 ms total Compared to: `http://127.0.0.1:8006/covid?sql=with+cte+as+(%0D%0A++select+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+ny_times_us_counties%0D%0A)%2C%0D%0Atruncated+as+(%0D%0A++select+null+as+_facet%2C+null+as+facet_name%2C+null+as+facet_count%2C+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+cte+order+by+date+desc+limit+4%0D%0A)%2C%0D%0Astate_facet+as+(%0D%0A++select+%27state%27+as+_facet%2C+state+as+facet_name%2C+count(*)+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A)%2C%0D%0Afips_facet+as+(%0D%0A++select+%27fips%27+as+_facet%2C+fips+as+facet_name%2C+count(*)+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A)%2C%0D%0Acounty_facet+as+(%0D%0A++select+%27county%27+as+_facet%2C+county+as+facet_name%2C+count(*)+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A)%0D%0Aselect+*+from+truncated%0D%0Aunion+all+select+*+from+state_facet%0D%0Aunion+all+select+*+from+fips_facet%0D%0Aunion+all+select+*+from+county_facet&_trace=1` Which is 353ms total. The separate queries ran faster! Really surprising result there.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time, https://github.com/simonw/datasette/issues/1513#issuecomment-970828568,https://api.github.com/repos/simonw/datasette/issues/1513,970828568,IC_kwDOBm6k_c453asY,9599,simonw,2021-11-16T23:27:11Z,2021-11-16T23:27:11Z,OWNER,One last experiment: I'm going to try running an expensive query in the CTE portion.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time, https://github.com/simonw/datasette/issues/1513#issuecomment-970827674,https://api.github.com/repos/simonw/datasette/issues/1513,970827674,IC_kwDOBm6k_c453aea,9599,simonw,2021-11-16T23:26:58Z,2021-11-16T23:26:58Z,OWNER,"With trace. https://covid-19.datasettes.com/covid/ny_times_us_counties?_trace=1&_facet_size=3&_size=2&_trace=1 shows the following: ``` fetch rows: 0.41762600005768036 ms facet state: 284.30423800000426 ms facet county: 273.2565999999679 ms facet fips: 197.80996999998024 ms ``` = 755.78843400001ms total It didn't run a count because that's the homepage and the count is cached. So I dropped the count from the query and ran it: https://covid-19.datasettes.com/covid?sql=with+cte+as+(%0D%0A++select+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+ny_times_us_counties%0D%0A)%2C%0D%0Atruncated+as+(%0D%0A++select+null+as+_facet%2C+null+as+facet_name%2C+null+as+facet_count%2C+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+cte+order+by+date+desc+limit+4%0D%0A)%2C%0D%0Astate_facet+as+(%0D%0A++select+%27state%27+as+_facet%2C+state+as+facet_name%2C+count(*)+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A)%2C%0D%0Afips_facet+as+(%0D%0A++select+%27fips%27+as+_facet%2C+fips+as+facet_name%2C+count(*)+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A)%2C%0D%0Acounty_facet+as+(%0D%0A++select+%27county%27+as+_facet%2C+county+as+facet_name%2C+count(*)+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A)%0D%0Aselect+*+from+truncated%0D%0Aunion+all+select+*+from+state_facet%0D%0Aunion+all+select+*+from+fips_facet%0D%0Aunion+all+select+*+from+county_facet&_trace=1 Shows 649.4359889999259 ms for the query - compared to 755.78843400001ms for the separate. So it saved about 100ms. Still not a huge difference though! ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time, https://github.com/simonw/datasette/issues/1513#issuecomment-970780866,https://api.github.com/repos/simonw/datasette/issues/1513,970780866,IC_kwDOBm6k_c453PDC,9599,simonw,2021-11-16T23:01:57Z,2021-11-16T23:01:57Z,OWNER,"One disadvantage to this approach: if you have a SQL time limit of 1s and it takes 0.9s to return the rows but then 0.5s to calculate each of the requested facets the entire query will exceed the time limit. Could work around this by catching that error and then re-running the query just for the rows, but that would result in the user having to wait longer for the results. Could try to remember if that has happened using an in-memory Python data structure and skip the faceting optimization if it's caused problems in the past? That seems a bit gross. Maybe this becomes an opt-in optimization you can request in your `metadata.json` setting for that table, which massively increases the time limit? That's a bit weird too - now there are two separate implementations of the faceting logic, which had better have a REALLY big pay-off to be worth maintaining. What if we kept the query that returns the rows to be displayed on the page separate from the facets, but then executed all of the facets together using this method such that the `cte` only (presumably) has to be calculated once? That would still lead to multiple facets potentially exceeding the SQL time limit when single facets would not have. Maybe a better optimization would be to move facets to happening via `fetch()` calls from the client, so the user gets to see their rows instantly and the facets then appear as and when they are available (though it would cause page jank). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time, https://github.com/simonw/datasette/issues/1513#issuecomment-970766486,https://api.github.com/repos/simonw/datasette/issues/1513,970766486,IC_kwDOBm6k_c453LiW,9599,simonw,2021-11-16T22:52:56Z,2021-11-16T22:56:07Z,OWNER,"https://covid-19.datasettes.com/covid is 805.2MB https://covid-19.datasettes.com/covid/ny_times_us_counties?_trace=1&_facet_size=3&_size=2 Equivalent SQL: https://covid-19.datasettes.com/covid?sql=with+cte+as+%28%0D%0A++select+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+ny_times_us_counties%0D%0A%29%2C%0D%0Atruncated+as+%28%0D%0A++select+null+as+_facet%2C+null+as+facet_name%2C+null+as+facet_count%2C+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+cte+order+by+date+desc+limit+4%0D%0A%29%2C%0D%0Astate_facet+as+%28%0D%0A++select+%27state%27+as+_facet%2C+state+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Afips_facet+as+%28%0D%0A++select+%27fips%27+as+_facet%2C+fips+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Acounty_facet+as+%28%0D%0A++select+%27county%27+as+_facet%2C+county+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Atotal_count+as+%28%0D%0A++select+%27COUNT%27+as+_facet%2C+%27%27+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte%0D%0A%29%0D%0Aselect+*+from+truncated%0D%0Aunion+all+select+*+from+state_facet%0D%0Aunion+all+select+*+from+fips_facet%0D%0Aunion+all+select+*+from+county_facet%0D%0Aunion+all+select+*+from+total_count ```sql with cte as ( select rowid, date, county, state, fips, cases, deaths from ny_times_us_counties ), truncated as ( select null as _facet, null as facet_name, null as facet_count, rowid, date, county, state, fips, cases, deaths from cte order by date desc limit 4 ), state_facet as ( select 'state' as _facet, state as facet_name, count(*) as facet_count, null, null, null, null, null, null, null from cte group by facet_name order by facet_count desc limit 3 ), fips_facet as ( select 'fips' as _facet, fips as facet_name, count(*) as facet_count, null, null, null, null, null, null, null from cte group by facet_name order by facet_count desc limit 3 ), county_facet as ( select 'county' as _facet, county as facet_name, count(*) as facet_count, null, null, null, null, null, null, null from cte group by facet_name order by facet_count desc limit 3 ), total_count as ( select 'COUNT' as _facet, '' as facet_name, count(*) as facet_count, null, null, null, null, null, null, null from cte ) select * from truncated union all select * from state_facet union all select * from fips_facet union all select * from county_facet union all select * from total_count ``` _facet | facet_name | facet_count | rowid | date | county | state | fips | cases | deaths -- | -- | -- | -- | -- | -- | -- | -- | -- | --   |   |   | 1917344 | 2021-11-15 | Autauga | Alabama | 1001 | 10407 | 154   |   |   | 1917345 | 2021-11-15 | Baldwin | Alabama | 1003 | 37875 | 581   |   |   | 1917346 | 2021-11-15 | Barbour | Alabama | 1005 | 3648 | 79   |   |   | 1917347 | 2021-11-15 | Bibb | Alabama | 1007 | 4317 | 92 state | Texas | 148028 |   |   |   |   |   |   |   state | Georgia | 96249 |   |   |   |   |   |   |   state | Virginia | 79315 |   |   |   |   |   |   |   fips |   | 17580 |   |   |   |   |   |   |   fips | 53061 | 665 |   |   |   |   |   |   |   fips | 17031 | 662 |   |   |   |   |   |   |   county | Washington | 18666 |   |   |   |   |   |   |   county | Unknown | 15840 |   |   |   |   |   |   |   county | Jefferson | 15637 |   |   |   |   |   |   |   COUNT |   | 1920593 |   |   |   |   |   |   |  ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time, https://github.com/simonw/datasette/issues/1513#issuecomment-970770304,https://api.github.com/repos/simonw/datasette/issues/1513,970770304,IC_kwDOBm6k_c453MeA,9599,simonw,2021-11-16T22:55:19Z,2021-11-16T22:55:19Z,OWNER,(One thing I really like about this pattern is that it should work exactly the same when used to facet the results of arbitrary SQL queries as it does when faceting results from the table page.),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time, https://github.com/simonw/datasette/issues/1513#issuecomment-970767952,https://api.github.com/repos/simonw/datasette/issues/1513,970767952,IC_kwDOBm6k_c453L5Q,9599,simonw,2021-11-16T22:53:52Z,2021-11-16T22:53:52Z,OWNER,It's going to take another 15 minutes for the build to finish and deploy the version with `_trace=1`: https://github.com/simonw/covid-19-datasette/actions/runs/1469150112,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time, https://github.com/simonw/datasette/issues/1513#issuecomment-970758179,https://api.github.com/repos/simonw/datasette/issues/1513,970758179,IC_kwDOBm6k_c453Jgj,9599,simonw,2021-11-16T22:47:38Z,2021-11-16T22:47:38Z,OWNER,"Trace now enabled: https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_facet_size=3&_size=2&_nocount=1&_trace=1 Here are the relevant traces: ```json [ { ""type"": ""sql"", ""start"": 31.214430154, ""end"": 31.214817089, ""duration_ms"": 0.3869350000016425, ""traceback"": [ "" File \""/usr/local/lib/python3.8/site-packages/datasette/views/base.py\"", line 262, in get\n return await self.view_get(\n"", "" File \""/usr/local/lib/python3.8/site-packages/datasette/views/base.py\"", line 477, in view_get\n response_or_template_contexts = await self.data(\n"", "" File \""/usr/local/lib/python3.8/site-packages/datasette/views/table.py\"", line 705, in data\n results = await db.execute(sql, params, truncate=True, **extra_args)\n"" ], ""database"": ""global-power-plants"", ""sql"": ""select rowid, country, country_long, name, gppd_idnr, capacity_mw, latitude, longitude, primary_fuel, other_fuel1, other_fuel2, other_fuel3, commissioning_year, owner, source, url, geolocation_source, wepp_id, year_of_capacity_data, generation_gwh_2013, generation_gwh_2014, generation_gwh_2015, generation_gwh_2016, generation_gwh_2017, generation_data_source, estimated_generation_gwh from [global-power-plants] order by rowid limit 3"", ""params"": {} }, { ""type"": ""sql"", ""start"": 31.215234586, ""end"": 31.220110342, ""duration_ms"": 4.875756000000564, ""traceback"": [ "" File \""/usr/local/lib/python3.8/site-packages/datasette/views/table.py\"", line 760, in data\n ) = await facet.facet_results()\n"", "" File \""/usr/local/lib/python3.8/site-packages/datasette/facets.py\"", line 212, in facet_results\n facet_rows_results = await self.ds.execute(\n"", "" File \""/usr/local/lib/python3.8/site-packages/datasette/app.py\"", line 634, in execute\n return await self.databases[db_name].execute(\n"" ], ""database"": ""global-power-plants"", ""sql"": ""select country_long as value, count(*) as count from (\n select rowid, country, country_long, name, gppd_idnr, capacity_mw, latitude, longitude, primary_fuel, other_fuel1, other_fuel2, other_fuel3, commissioning_year, owner, source, url, geolocation_source, wepp_id, year_of_capacity_data, generation_gwh_2013, generation_gwh_2014, generation_gwh_2015, generation_gwh_2016, generation_gwh_2017, generation_data_source, estimated_generation_gwh from [global-power-plants] \n )\n where country_long is not null\n group by country_long order by count desc, value limit 4"", ""params"": [] }, { ""type"": ""sql"", ""start"": 31.221062485, ""end"": 31.228968364, ""duration_ms"": 7.905878999999061, ""traceback"": [ "" File \""/usr/local/lib/python3.8/site-packages/datasette/views/table.py\"", line 760, in data\n ) = await facet.facet_results()\n"", "" File \""/usr/local/lib/python3.8/site-packages/datasette/facets.py\"", line 212, in facet_results\n facet_rows_results = await self.ds.execute(\n"", "" File \""/usr/local/lib/python3.8/site-packages/datasette/app.py\"", line 634, in execute\n return await self.databases[db_name].execute(\n"" ], ""database"": ""global-power-plants"", ""sql"": ""select owner as value, count(*) as count from (\n select rowid, country, country_long, name, gppd_idnr, capacity_mw, latitude, longitude, primary_fuel, other_fuel1, other_fuel2, other_fuel3, commissioning_year, owner, source, url, geolocation_source, wepp_id, year_of_capacity_data, generation_gwh_2013, generation_gwh_2014, generation_gwh_2015, generation_gwh_2016, generation_gwh_2017, generation_data_source, estimated_generation_gwh from [global-power-plants] \n )\n where owner is not null\n group by owner order by count desc, value limit 4"", ""params"": [] }, { ""type"": ""sql"", ""start"": 31.229809757, ""end"": 31.253902162, ""duration_ms"": 24.09240499999754, ""traceback"": [ "" File \""/usr/local/lib/python3.8/site-packages/datasette/views/table.py\"", line 760, in data\n ) = await facet.facet_results()\n"", "" File \""/usr/local/lib/python3.8/site-packages/datasette/facets.py\"", line 212, in facet_results\n facet_rows_results = await self.ds.execute(\n"", "" File \""/usr/local/lib/python3.8/site-packages/datasette/app.py\"", line 634, in execute\n return await self.databases[db_name].execute(\n"" ], ""database"": ""global-power-plants"", ""sql"": ""select primary_fuel as value, count(*) as count from (\n select rowid, country, country_long, name, gppd_idnr, capacity_mw, latitude, longitude, primary_fuel, other_fuel1, other_fuel2, other_fuel3, commissioning_year, owner, source, url, geolocation_source, wepp_id, year_of_capacity_data, generation_gwh_2013, generation_gwh_2014, generation_gwh_2015, generation_gwh_2016, generation_gwh_2017, generation_data_source, estimated_generation_gwh from [global-power-plants] \n )\n where primary_fuel is not null\n group by primary_fuel order by count desc, value limit 4"", ""params"": [] }, { ""type"": ""sql"", ""start"": 31.255699745, ""end"": 31.256243889, ""duration_ms"": 0.544143999999136, ""traceback"": [ "" File \""/usr/local/lib/python3.8/site-packages/datasette/facets.py\"", line 145, in suggest\n row_count = await self.get_row_count()\n"", "" File \""/usr/local/lib/python3.8/site-packages/datasette/facets.py\"", line 132, in get_row_count\n await self.ds.execute(\n"", "" File \""/usr/local/lib/python3.8/site-packages/datasette/app.py\"", line 634, in execute\n return await self.databases[db_name].execute(\n"" ], ""database"": ""global-power-plants"", ""sql"": ""select count(*) from (select rowid, country, country_long, name, gppd_idnr, capacity_mw, latitude, longitude, primary_fuel, other_fuel1, other_fuel2, other_fuel3, commissioning_year, owner, source, url, geolocation_source, wepp_id, year_of_capacity_data, generation_gwh_2013, generation_gwh_2014, generation_gwh_2015, generation_gwh_2016, generation_gwh_2017, generation_data_source, estimated_generation_gwh from [global-power-plants] )"", ""params"": [] } ] ``` ``` fetch rows: 0.3869350000016425 ms facet country_long: 4.875756000000564 ms facet owner: 7.905878999999061 ms facet primary_fuel: 24.09240499999754 ms count: 0.544143999999136 ms ``` Total = 37.8ms I modified the query to include the total count as well: https://global-power-plants.datasettes.com/global-power-plants?sql=with+cte+as+%28%0D%0A++select+rowid%2C+country%2C+country_long%2C+name%2C+owner%2C+primary_fuel%0D%0A++from+%5Bglobal-power-plants%5D%0D%0A%29%2C%0D%0Atruncated+as+%28%0D%0A++select+null+as+_facet%2C+null+as+facet_name%2C+null+as+facet_count%2C+rowid%2C+country%2C+country_long%2C+name%2C+owner%2C+primary_fuel%0D%0A++from+cte+order+by+rowid+limit+4%0D%0A%29%2C%0D%0Acountry_long_facet+as+%28%0D%0A++select+%27country_long%27+as+_facet%2C+country_long+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Aowner_facet+as+%28%0D%0A++select+%27owner%27+as+_facet%2C+owner+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Aprimary_fuel_facet+as+%28%0D%0A++select+%27primary_fuel%27+as+_facet%2C+primary_fuel+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Atotal_count+as+%28%0D%0A++select+%27COUNT%27+as+_facet%2C+%27%27+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte%0D%0A%29%0D%0Aselect+*+from+truncated%0D%0Aunion+all+select+*+from+country_long_facet%0D%0Aunion+all+select+*+from+owner_facet%0D%0Aunion+all+select+*+from+primary_fuel_facet%0D%0Aunion+all+select+*+from+total_count&_trace=1 ```sql with cte as ( select rowid, country, country_long, name, owner, primary_fuel from [global-power-plants] ), truncated as ( select null as _facet, null as facet_name, null as facet_count, rowid, country, country_long, name, owner, primary_fuel from cte order by rowid limit 4 ), country_long_facet as ( select 'country_long' as _facet, country_long as facet_name, count(*) as facet_count, null, null, null, null, null, null from cte group by facet_name order by facet_count desc limit 3 ), owner_facet as ( select 'owner' as _facet, owner as facet_name, count(*) as facet_count, null, null, null, null, null, null from cte group by facet_name order by facet_count desc limit 3 ), primary_fuel_facet as ( select 'primary_fuel' as _facet, primary_fuel as facet_name, count(*) as facet_count, null, null, null, null, null, null from cte group by facet_name order by facet_count desc limit 3 ), total_count as ( select 'COUNT' as _facet, '' as facet_name, count(*) as facet_count, null, null, null, null, null, null from cte ) select * from truncated union all select * from country_long_facet union all select * from owner_facet union all select * from primary_fuel_facet union all select * from total_count ``` The trace says that query took 34.801436999998714 ms. To my huge surprise, this convoluted optimization only shaves the sum query time down from 37.8ms to 34.8ms! That entire database file is just 11.1 MB though. Maybe it would make a meaningful difference on something larger?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time, https://github.com/simonw/datasette/issues/1513#issuecomment-970742415,https://api.github.com/repos/simonw/datasette/issues/1513,970742415,IC_kwDOBm6k_c453FqP,9599,simonw,2021-11-16T22:37:14Z,2021-11-16T22:37:14Z,OWNER,"The query takes 42.794ms to run. Here's the equivalent page using separate queries: https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_facet_size=3&_size=2&_nocount=1 Annoyingly I can't disable facet suggestions but keep facets. I'm going to turn on tracing so I can see how long the separate queries took.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time, https://github.com/simonw/datasette/issues/1513#issuecomment-970738130,https://api.github.com/repos/simonw/datasette/issues/1513,970738130,IC_kwDOBm6k_c453EnS,9599,simonw,2021-11-16T22:32:19Z,2021-11-16T22:32:19Z,OWNER,"I came up with the following query which seems to work! ```sql with cte as ( select rowid, country, country_long, name, owner, primary_fuel from [global-power-plants] ), truncated as ( select null as _facet, null as facet_name, null as facet_count, rowid, country, country_long, name, owner, primary_fuel from cte order by rowid limit 4 ), country_long_facet as ( select 'country_long' as _facet, country_long as facet_name, count(*) as facet_count, null, null, null, null, null, null from cte group by facet_name order by facet_count desc limit 3 ), owner_facet as ( select 'owner' as _facet, owner as facet_name, count(*) as facet_count, null, null, null, null, null, null from cte group by facet_name order by facet_count desc limit 3 ), primary_fuel_facet as ( select 'primary_fuel' as _facet, primary_fuel as facet_name, count(*) as facet_count, null, null, null, null, null, null from cte group by facet_name order by facet_count desc limit 3 ) select * from truncated union all select * from country_long_facet union all select * from owner_facet union all select * from primary_fuel_facet ``` (Limits should be 101, 31, 31, 31 but I reduced size to get a shorter example table). Results [look like this](https://global-power-plants.datasettes.com/global-power-plants?sql=with+cte+as+%28%0D%0A++select+rowid%2C+country%2C+country_long%2C+name%2C+owner%2C+primary_fuel%0D%0A++from+%5Bglobal-power-plants%5D%0D%0A%29%2C%0D%0Atruncated+as+%28%0D%0A++select+null+as+_facet%2C+null+as+facet_name%2C+null+as+facet_count%2C+rowid%2C+country%2C+country_long%2C+name%2C+owner%2C+primary_fuel%0D%0A++from+cte+order+by+rowid+limit+4%0D%0A%29%2C%0D%0Acountry_long_facet+as+%28%0D%0A++select+%27country_long%27+as+_facet%2C+country_long+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Aowner_facet+as+%28%0D%0A++select+%27owner%27+as+_facet%2C+owner+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Aprimary_fuel_facet+as+%28%0D%0A++select+%27primary_fuel%27+as+_facet%2C+primary_fuel+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%0D%0Aselect+*+from+truncated%0D%0Aunion+all+select+*+from+country_long_facet%0D%0Aunion+all+select+*+from+owner_facet%0D%0Aunion+all+select+*+from+primary_fuel_facet): _facet | facet_name | facet_count | rowid | country | country_long | name | owner | primary_fuel -- | -- | -- | -- | -- | -- | -- | -- | --   |   |   | 1 | AFG | Afghanistan | Kajaki Hydroelectric Power Plant Afghanistan |   | Hydro   |   |   | 2 | AFG | Afghanistan | Kandahar DOG |   | Solar   |   |   | 3 | AFG | Afghanistan | Kandahar JOL |   | Solar   |   |   | 4 | AFG | Afghanistan | Mahipar Hydroelectric Power Plant Afghanistan |   | Hydro country_long | United States of America | 8688 |   |   |   |   |   |   country_long | China | 4235 |   |   |   |   |   |   country_long | United Kingdom | 2603 |   |   |   |   |   |   owner |   | 14112 |   |   |   |   |   |   owner | Lightsource Renewable Energy | 120 |   |   |   |   |   |   owner | Cypress Creek Renewables | 109 |   |   |   |   |   |   primary_fuel | Solar | 9662 |   |   |   |   |   |   primary_fuel | Hydro | 7155 |   |   |   |   |   |   primary_fuel | Wind | 5188 |   |   |   |   |   |   This is a neat proof of concept. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time, https://github.com/simonw/datasette/pull/1512#issuecomment-970718337,https://api.github.com/repos/simonw/datasette/issues/1512,970718337,IC_kwDOBm6k_c452_yB,9599,simonw,2021-11-16T22:02:30Z,2021-11-16T22:02:30Z,OWNER,"I've decided to make the clever `asyncio` dependency injection opt-in - so you can either decorate with `@inject` or you can set `inject_all = True` on the class - for example: ```python import asyncio from datasette.utils.asyncdi import AsyncBase, inject class Simple(AsyncBase): def __init__(self): self.log = [] @inject async def two(self): self.log.append(""two"") @inject async def one(self, two): self.log.append(""one"") return self.log async def not_inject(self, one, two): return one + two class Complex(AsyncBase): inject_all = True def __init__(self): self.log = [] async def b(self): self.log.append(""b"") async def a(self, b): self.log.append(""a"") async def go(self, a): self.log.append(""go"") return self.log ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055402144,New pattern for async view classes, https://github.com/simonw/datasette/issues/878#issuecomment-970712713,https://api.github.com/repos/simonw/datasette/issues/878,970712713,IC_kwDOBm6k_c452-aJ,9599,simonw,2021-11-16T21:54:33Z,2021-11-16T21:54:33Z,OWNER,I'm going to continue working on this in a PR.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/878#issuecomment-970705738,https://api.github.com/repos/simonw/datasette/issues/878,970705738,IC_kwDOBm6k_c4528tK,9599,simonw,2021-11-16T21:44:31Z,2021-11-16T21:44:31Z,OWNER,Wrote a TIL about what I learned using `TopologicalSorter`: https://til.simonwillison.net/python/graphlib-topologicalsorter,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/878#issuecomment-970673085,https://api.github.com/repos/simonw/datasette/issues/878,970673085,IC_kwDOBm6k_c4520u9,9599,simonw,2021-11-16T20:58:24Z,2021-11-16T20:58:24Z,OWNER,"New test: ```python class Complex(AsyncBase): def __init__(self): self.log = [] async def d(self): await asyncio.sleep(random() * 0.1) print(""LOG: d"") self.log.append(""d"") async def c(self): await asyncio.sleep(random() * 0.1) print(""LOG: c"") self.log.append(""c"") async def b(self, c, d): print(""LOG: b"") self.log.append(""b"") async def a(self, b, c): print(""LOG: a"") self.log.append(""a"") async def go(self, a): print(""LOG: go"") self.log.append(""go"") return self.log @pytest.mark.asyncio async def test_complex(): result = await Complex().go() # 'c' should only be called once assert tuple(result) in ( # c and d could happen in either order (""c"", ""d"", ""b"", ""a"", ""go""), (""d"", ""c"", ""b"", ""a"", ""go""), ) ``` And this code passes it: ```python import asyncio from functools import wraps import inspect try: import graphlib except ImportError: from . import vendored_graphlib as graphlib class AsyncMeta(type): def __new__(cls, name, bases, attrs): # Decorate any items that are 'async def' methods _registry = {} new_attrs = {""_registry"": _registry} for key, value in attrs.items(): if inspect.iscoroutinefunction(value) and not value.__name__ == ""resolve"": new_attrs[key] = make_method(value) _registry[key] = new_attrs[key] else: new_attrs[key] = value # Gather graph for later dependency resolution graph = { key: { p for p in inspect.signature(method).parameters.keys() if p != ""self"" and not p.startswith(""_"") } for key, method in _registry.items() } new_attrs[""_graph""] = graph return super().__new__(cls, name, bases, new_attrs) def make_method(method): parameters = inspect.signature(method).parameters.keys() @wraps(method) async def inner(self, _results=None, **kwargs): print(""\n{}.{}({}) _results={}"".format(self, method.__name__, kwargs, _results)) # Any parameters not provided by kwargs are resolved from registry to_resolve = [p for p in parameters if p not in kwargs and p != ""self""] missing = [p for p in to_resolve if p not in self._registry] assert ( not missing ), ""The following DI parameters could not be found in the registry: {}"".format( missing ) results = {} results.update(kwargs) if to_resolve: resolved_parameters = await self.resolve(to_resolve, _results) results.update(resolved_parameters) return_value = await method(self, **results) if _results is not None: _results[method.__name__] = return_value return return_value return inner class AsyncBase(metaclass=AsyncMeta): async def resolve(self, names, results=None): print(""\n resolve: "", names) if results is None: results = {} # Come up with an execution plan, just for these nodes ts = graphlib.TopologicalSorter() to_do = set(names) done = set() while to_do: item = to_do.pop() dependencies = self._graph[item] ts.add(item, *dependencies) done.add(item) # Add any not-done dependencies to the queue to_do.update({k for k in dependencies if k not in done}) ts.prepare() plan = [] while ts.is_active(): node_group = ts.get_ready() plan.append(node_group) ts.done(*node_group) print(""plan:"", plan) results = {} for node_group in plan: awaitables = [ self._registry[name]( self, _results=results, **{k: v for k, v in results.items() if k in self._graph[name]}, ) for name in node_group ] print("" results = "", results) print("" awaitables: "", awaitables) awaitable_results = await asyncio.gather(*awaitables) results.update( {p[0].__name__: p[1] for p in zip(awaitables, awaitable_results)} ) print("" End of resolve(), returning"", results) return {key: value for key, value in results.items() if key in names} ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/878#issuecomment-970660299,https://api.github.com/repos/simonw/datasette/issues/878,970660299,IC_kwDOBm6k_c452xnL,9599,simonw,2021-11-16T20:39:43Z,2021-11-16T20:42:27Z,OWNER,"But that does seem to be the plan that `TopographicalSorter` provides: ```python graph = {""go"": {""a""}, ""a"": {""b"", ""c""}, ""b"": {""c"", ""d""}} ts = TopologicalSorter(graph) ts.prepare() while ts.is_active(): nodes = ts.get_ready() print(nodes) ts.done(*nodes) ``` Outputs: ``` ('c', 'd') ('b',) ('a',) ('go',) ``` Also: ```python graph = {""go"": {""d"", ""e"", ""f""}, ""d"": {""b"", ""c""}, ""b"": {""c""}} ts = TopologicalSorter(graph) ts.prepare() while ts.is_active(): nodes = ts.get_ready() print(nodes) ts.done(*nodes) ``` Gives: ``` ('e', 'f', 'c') ('b',) ('d',) ('go',) ``` I'm confident that `TopologicalSorter` is the way to do this. I think I need to rewrite my code to call it once to get that plan, then `await asyncio.gather(*nodes)` in turn to execute it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/878#issuecomment-970657874,https://api.github.com/repos/simonw/datasette/issues/878,970657874,IC_kwDOBm6k_c452xBS,9599,simonw,2021-11-16T20:36:01Z,2021-11-16T20:36:01Z,OWNER,"My goal here is to calculate the most efficient way to resolve the different nodes, running them in parallel where possible. So for this class: ```python class Complex(AsyncBase): async def d(self): pass async def c(self): pass async def b(self, c, d): pass async def a(self, b, c): pass async def go(self, a): pass ``` A call to `go()` should do this: - `c` and `d` in parallel - `b` - `a` - `go`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/878#issuecomment-970655927,https://api.github.com/repos/simonw/datasette/issues/878,970655927,IC_kwDOBm6k_c452wi3,9599,simonw,2021-11-16T20:33:11Z,2021-11-16T20:33:11Z,OWNER,"What should be happening here instead is it should resolve the full graph and notice that `c` is depended on by both `b` and `a` - so it should run `c` first, then run the next ones in parallel. So maybe the algorithm I'm inheriting from https://docs.python.org/3/library/graphlib.html isn't the correct algorithm?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/878#issuecomment-970655304,https://api.github.com/repos/simonw/datasette/issues/878,970655304,IC_kwDOBm6k_c452wZI,9599,simonw,2021-11-16T20:32:16Z,2021-11-16T20:32:16Z,OWNER,"This code is really fiddly. I just got to this version: ```python import asyncio from functools import wraps import inspect try: import graphlib except ImportError: from . import vendored_graphlib as graphlib class AsyncMeta(type): def __new__(cls, name, bases, attrs): # Decorate any items that are 'async def' methods _registry = {} new_attrs = {""_registry"": _registry} for key, value in attrs.items(): if inspect.iscoroutinefunction(value) and not value.__name__ == ""resolve"": new_attrs[key] = make_method(value) _registry[key] = new_attrs[key] else: new_attrs[key] = value # Gather graph for later dependency resolution graph = { key: { p for p in inspect.signature(method).parameters.keys() if p != ""self"" and not p.startswith(""_"") } for key, method in _registry.items() } new_attrs[""_graph""] = graph return super().__new__(cls, name, bases, new_attrs) def make_method(method): @wraps(method) async def inner(self, _results=None, **kwargs): print(""inner - _results="", _results) parameters = inspect.signature(method).parameters.keys() # Any parameters not provided by kwargs are resolved from registry to_resolve = [p for p in parameters if p not in kwargs and p != ""self""] missing = [p for p in to_resolve if p not in self._registry] assert ( not missing ), ""The following DI parameters could not be found in the registry: {}"".format( missing ) results = {} results.update(kwargs) if to_resolve: resolved_parameters = await self.resolve(to_resolve, _results) results.update(resolved_parameters) return_value = await method(self, **results) if _results is not None: _results[method.__name__] = return_value return return_value return inner class AsyncBase(metaclass=AsyncMeta): async def resolve(self, names, results=None): print(""\n resolve: "", names) if results is None: results = {} # Resolve them in the correct order ts = graphlib.TopologicalSorter() for name in names: ts.add(name, *self._graph[name]) ts.prepare() async def resolve_nodes(nodes): print("" resolve_nodes"", nodes) print("" (current results = {})"".format(repr(results))) awaitables = [ self._registry[name]( self, _results=results, **{k: v for k, v in results.items() if k in self._graph[name]}, ) for name in nodes if name not in results ] print("" awaitables: "", awaitables) awaitable_results = await asyncio.gather(*awaitables) results.update( {p[0].__name__: p[1] for p in zip(awaitables, awaitable_results)} ) if not ts.is_active(): # Nothing has dependencies - just resolve directly print("" no dependencies, resolve directly"") await resolve_nodes(names) else: # Resolve in topological order while ts.is_active(): nodes = ts.get_ready() print("" ts.get_ready() returned nodes:"", nodes) await resolve_nodes(nodes) for node in nodes: ts.done(node) print("" End of resolve(), returning"", results) return {key: value for key, value in results.items() if key in names} ``` With this test: ```python class Complex(AsyncBase): def __init__(self): self.log = [] async def c(self): print(""LOG: c"") self.log.append(""c"") async def b(self, c): print(""LOG: b"") self.log.append(""b"") async def a(self, b, c): print(""LOG: a"") self.log.append(""a"") async def go(self, a): print(""LOG: go"") self.log.append(""go"") return self.log @pytest.mark.asyncio async def test_complex(): result = await Complex().go() # 'c' should only be called once assert result == [""c"", ""b"", ""a"", ""go""] ``` This test sometimes passes, and sometimes fails! Output for a pass: ``` tests/test_asyncdi.py inner - _results= None resolve: ['a'] ts.get_ready() returned nodes: ('c', 'b') resolve_nodes ('c', 'b') (current results = {}) awaitables: [, ] inner - _results= {} LOG: c inner - _results= {'c': None} resolve: ['c'] ts.get_ready() returned nodes: ('c',) resolve_nodes ('c',) (current results = {'c': None}) awaitables: [] End of resolve(), returning {'c': None} LOG: b ts.get_ready() returned nodes: ('a',) resolve_nodes ('a',) (current results = {'c': None, 'b': None}) awaitables: [] inner - _results= {'c': None, 'b': None} LOG: a End of resolve(), returning {'c': None, 'b': None, 'a': None} LOG: go ``` Output for a fail: ``` tests/test_asyncdi.py inner - _results= None resolve: ['a'] ts.get_ready() returned nodes: ('b', 'c') resolve_nodes ('b', 'c') (current results = {}) awaitables: [, ] inner - _results= {} resolve: ['c'] ts.get_ready() returned nodes: ('c',) resolve_nodes ('c',) (current results = {}) awaitables: [] inner - _results= {} LOG: c inner - _results= {'c': None} LOG: c End of resolve(), returning {'c': None} LOG: b ts.get_ready() returned nodes: ('a',) resolve_nodes ('a',) (current results = {'c': None, 'b': None}) awaitables: [] inner - _results= {'c': None, 'b': None} LOG: a End of resolve(), returning {'c': None, 'b': None, 'a': None} LOG: go F =================================================================================================== FAILURES =================================================================================================== _________________________________________________________________________________________________ test_complex _________________________________________________________________________________________________ @pytest.mark.asyncio async def test_complex(): result = await Complex().go() # 'c' should only be called once > assert result == [""c"", ""b"", ""a"", ""go""] E AssertionError: assert ['c', 'c', 'b', 'a', 'go'] == ['c', 'b', 'a', 'go'] E At index 1 diff: 'c' != 'b' E Left contains one more item: 'go' E Use -v to get the full diff tests/test_asyncdi.py:48: AssertionError ================== short test summary info ================================ FAILED tests/test_asyncdi.py::test_complex - AssertionError: assert ['c', 'c', 'b', 'a', 'go'] == ['c', 'b', 'a', 'go'] ``` I figured out why this is happening. `a` requires `b` and `c` `b` also requires `c` The code decides to run `b` and `c` in parallel. If `c` completes first, then when `b` runs it gets to use the already-calculated result for `c` - so it doesn't need to call `c` again. If `b` gets to that point before `c` does it also needs to call `c`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/878#issuecomment-970624197,https://api.github.com/repos/simonw/datasette/issues/878,970624197,IC_kwDOBm6k_c452ozF,9599,simonw,2021-11-16T19:49:05Z,2021-11-16T19:49:05Z,OWNER,"Here's the latest version of my weird dependency injection async class: ```python import inspect class AsyncMeta(type): def __new__(cls, name, bases, attrs): # Decorate any items that are 'async def' methods _registry = {} new_attrs = {""_registry"": _registry} for key, value in attrs.items(): if inspect.iscoroutinefunction(value) and not value.__name__ == ""resolve"": new_attrs[key] = make_method(value) _registry[key] = new_attrs[key] else: new_attrs[key] = value # Topological sort of _registry by parameter dependencies graph = { key: { p for p in inspect.signature(method).parameters.keys() if p != ""self"" and not p.startswith(""_"") } for key, method in _registry.items() } new_attrs[""_graph""] = graph return super().__new__(cls, name, bases, new_attrs) def make_method(method): @wraps(method) async def inner(self, **kwargs): parameters = inspect.signature(method).parameters.keys() # Any parameters not provided by kwargs are resolved from registry to_resolve = [p for p in parameters if p not in kwargs and p != ""self""] missing = [p for p in to_resolve if p not in self._registry] assert ( not missing ), ""The following DI parameters could not be found in the registry: {}"".format( missing ) results = {} results.update(kwargs) results.update(await self.resolve(to_resolve)) return await method(self, **results) return inner bad = [0] class AsyncBase(metaclass=AsyncMeta): async def resolve(self, names): print("" resolve({})"".format(names)) results = {} # Resolve them in the correct order ts = TopologicalSorter() ts2 = TopologicalSorter() print("" names = "", names) print("" self._graph = "", self._graph) for name in names: if self._graph[name]: ts.add(name, *self._graph[name]) ts2.add(name, *self._graph[name]) print("" static_order ="", tuple(ts2.static_order())) ts.prepare() while ts.is_active(): print("" is_active, i = "", bad[0]) bad[0] += 1 if bad[0] > 20: print("" Infinite loop?"") break nodes = ts.get_ready() print("" Do nodes:"", nodes) awaitables = [self._registry[name](self, **{ k: v for k, v in results.items() if k in self._graph[name] }) for name in nodes] print("" awaitables: "", awaitables) awaitable_results = await asyncio.gather(*awaitables) results.update({ p[0].__name__: p[1] for p in zip(awaitables, awaitable_results) }) print(results) for node in nodes: ts.done(node) return results ``` Example usage: ```python class Foo(AsyncBase): async def graa(self, boff): print(""graa"") return 5 async def boff(self): print(""boff"") return 8 async def other(self, boff, graa): print(""other"") return 5 + boff + graa foo = Foo() await foo.other() ``` Output: ``` resolve(['boff', 'graa']) names = ['boff', 'graa'] self._graph = {'graa': {'boff'}, 'boff': set(), 'other': {'graa', 'boff'}} static_order = ('boff', 'graa') is_active, i = 0 Do nodes: ('boff',) awaitables: [] resolve([]) names = [] self._graph = {'graa': {'boff'}, 'boff': set(), 'other': {'graa', 'boff'}} static_order = () boff {'boff': 8} is_active, i = 1 Do nodes: ('graa',) awaitables: [] resolve([]) names = [] self._graph = {'graa': {'boff'}, 'boff': set(), 'other': {'graa', 'boff'}} static_order = () graa {'boff': 8, 'graa': 5} other 18 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/782#issuecomment-970554697,https://api.github.com/repos/simonw/datasette/issues/782,970554697,IC_kwDOBm6k_c452X1J,9599,simonw,2021-11-16T18:32:03Z,2021-11-16T18:32:03Z,OWNER,"I'm going to take another look at this: - https://github.com/simonw/datasette/issues/878","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format, https://github.com/simonw/datasette/issues/782#issuecomment-970553780,https://api.github.com/repos/simonw/datasette/issues/782,970553780,IC_kwDOBm6k_c452Xm0,9599,simonw,2021-11-16T18:30:51Z,2021-11-16T18:30:58Z,OWNER,"OK, I'm ready to start working on this today. I'm going to go with a default representation that looks like this: ```json { ""rows"": [ {""id"": 1, ""name"": ""One""}, {""id"": 2, ""name"": ""Two""} ], ""next_url"": null } ``` Note that there's no `count` - all it provides is the current selection of results and an indication as to how the next can be retrieved (`null` if there are no more results). I'll implement `?_extra=` to provide everything else.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format, https://github.com/simonw/datasette/issues/1509#issuecomment-970544733,https://api.github.com/repos/simonw/datasette/issues/1509,970544733,IC_kwDOBm6k_c452VZd,9599,simonw,2021-11-16T18:22:32Z,2021-11-16T18:22:32Z,OWNER,"This is mainly happening here: - https://github.com/simonw/datasette/issues/782","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1054243511,Datasette 1.0 JSON API (and documentation), https://github.com/simonw/datasette/issues/448#issuecomment-969621662,https://api.github.com/repos/simonw/datasette/issues/448,969621662,IC_kwDOBm6k_c45y0Ce,9599,simonw,2021-11-16T01:32:04Z,2021-11-16T01:32:04Z,OWNER,"Tests are failing and I think it's because the facets come back in different orders, need a tie-breaker. https://github.com/simonw/datasette/runs/4219325197?check_suite_focus=true","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views, https://github.com/simonw/datasette/issues/1176#issuecomment-969616626,https://api.github.com/repos/simonw/datasette/issues/1176,969616626,IC_kwDOBm6k_c45yyzy,9599,simonw,2021-11-16T01:29:13Z,2021-11-16T01:29:13Z,OWNER,"I'm inclined to create a Sphinx reference documentation page for this, as I did for `sqlite-utils` here: https://sqlite-utils.datasette.io/en/stable/reference.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779691739,"Policy on documenting ""public"" datasette.utils functions", https://github.com/simonw/datasette/issues/1012#issuecomment-969613166,https://api.github.com/repos/simonw/datasette/issues/1012,969613166,IC_kwDOBm6k_c45yx9u,9599,simonw,2021-11-16T01:27:25Z,2021-11-16T01:27:25Z,OWNER,"Requested here: - https://github.com/pypa/trove-classifiers/pull/85","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718540751,For 1.0 update trove classifier in setup.py, https://github.com/simonw/datasette/issues/1012#issuecomment-969602825,https://api.github.com/repos/simonw/datasette/issues/1012,969602825,IC_kwDOBm6k_c45yvcJ,9599,simonw,2021-11-16T01:21:14Z,2021-11-16T01:21:14Z,OWNER,"I'd been wondering how to get new classifiers into Trove - thanks, I'll give this a go.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718540751,For 1.0 update trove classifier in setup.py, https://github.com/simonw/datasette/issues/1511#issuecomment-969600859,https://api.github.com/repos/simonw/datasette/issues/1511,969600859,IC_kwDOBm6k_c45yu9b,9599,simonw,2021-11-16T01:20:13Z,2021-11-16T01:20:13Z,OWNER,"See: - #830","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1054246919,Review plugin hooks for Datasette 1.0, https://github.com/simonw/datasette/issues/448#issuecomment-969582098,https://api.github.com/repos/simonw/datasette/issues/448,969582098,IC_kwDOBm6k_c45yqYS,9599,simonw,2021-11-16T01:10:28Z,2021-11-16T01:10:28Z,OWNER,"Also note that this demo data is using a SQL view to create the JSON arrays - the view is defined as such: ```sql CREATE VIEW ads_with_targets as select ads.*, json_group_array(targets.name) as target_names from ads join ad_targets on ad_targets.ad_id = ads.id join targets on ad_targets.target_id = targets.id group by ad_targets.ad_id; ``` So running JSON faceting on top of that view is a pretty big ask!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views, https://github.com/simonw/datasette/issues/448#issuecomment-969578466,https://api.github.com/repos/simonw/datasette/issues/448,969578466,IC_kwDOBm6k_c45ypfi,9599,simonw,2021-11-16T01:08:29Z,2021-11-16T01:08:29Z,OWNER,Actually with the cache warmed up it looks like the facet query is taking 150ms which is good enough.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views, https://github.com/simonw/datasette/issues/448#issuecomment-969572281,https://api.github.com/repos/simonw/datasette/issues/448,969572281,IC_kwDOBm6k_c45yn-5,9599,simonw,2021-11-16T01:05:11Z,2021-11-16T01:05:11Z,OWNER,"I tried this and it seems to work correctly: ```python for source_and_config in self.get_configs(): config = source_and_config[""config""] source = source_and_config[""source""] column = config.get(""column"") or config[""simple""] facet_sql = """""" with inner as ({sql}), deduped_array_items as ( select distinct j.value, inner.* from json_each([inner].{col}) j join inner ) select value as value, count(*) as count from deduped_array_items group by value order by count(*) desc limit {limit} """""".format( col=escape_sqlite(column), sql=self.sql, limit=facet_size + 1 ) ``` The queries are _very_ slow though - I had to bump up to 2s time limit even against only a view returning 3,499 rows.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views, https://github.com/simonw/datasette/issues/448#issuecomment-969557008,https://api.github.com/repos/simonw/datasette/issues/448,969557008,IC_kwDOBm6k_c45ykQQ,9599,simonw,2021-11-16T00:56:09Z,2021-11-16T00:59:59Z,OWNER,"This looks like it might work: ```sql with inner as ( select * from ads_with_targets where :p0 in ( select value from json_each([ads_with_targets].[target_names]) ) ), deduped_array_items as ( select distinct j.value, inner.* from json_each([inner].[target_names]) j join inner ) select value, count(*) from deduped_array_items group by value order by count(*) desc ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views, https://github.com/simonw/datasette/issues/448#issuecomment-969557972,https://api.github.com/repos/simonw/datasette/issues/448,969557972,IC_kwDOBm6k_c45ykfU,9599,simonw,2021-11-16T00:56:58Z,2021-11-16T00:56:58Z,OWNER,It uses a CTE which were introduced in SQLite 3.8 - and AWS Lambda Python 3.9 still provides 3.7 - but I've checked and I can use `pysqlite3-binary` to work around that there so I'm OK relying on CTEs for this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views, https://github.com/simonw/datasette/issues/448#issuecomment-969449772,https://api.github.com/repos/simonw/datasette/issues/448,969449772,IC_kwDOBm6k_c45yKEs,9599,simonw,2021-11-15T23:48:37Z,2021-11-15T23:48:37Z,OWNER,"Given this query: https://json-view-facet-bug-demo-j7hipcg4aq-uc.a.run.app/russian-ads?sql=select%0D%0A++j.value+as+value%2C%0D%0A++count%28*%29+as+count%0D%0Afrom%0D%0A++%28%0D%0A++++select%0D%0A++++++id%2C%0D%0A++++++file%2C%0D%0A++++++clicks%2C%0D%0A++++++impressions%2C%0D%0A++++++text%2C%0D%0A++++++url%2C%0D%0A++++++spend_amount%2C%0D%0A++++++spend_currency%2C%0D%0A++++++created%2C%0D%0A++++++ended%2C%0D%0A++++++target_names%0D%0A++++from%0D%0A++++++ads_with_targets%0D%0A++++where%0D%0A++++++%3Ap0+in+%28%0D%0A++++++++select%0D%0A++++++++++value%0D%0A++++++++from%0D%0A++++++++++json_each%28%5Bads_with_targets%5D.%5Btarget_names%5D%29%0D%0A++++++%29%0D%0A++%29%0D%0A++join+json_each%28target_names%29+j%0D%0Agroup+by%0D%0A++j.value%0D%0Aorder+by%0D%0A++count+desc%2C%0D%0A++value%0D%0Alimit%0D%0A++31&p0=people_who_match%3Ainterests%3AAfrican-American+culture ```sql select j.value as value, count(*) as count from ( select id, file, clicks, impressions, text, url, spend_amount, spend_currency, created, ended, target_names from ads_with_targets where :p0 in ( select value from json_each([ads_with_targets].[target_names]) ) ) join json_each(target_names) j group by j.value order by count desc, value limit 31 ``` How can I return a count of the number of documents containing each tag, but not the number of total tags that match including duplicates?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views, https://github.com/simonw/datasette/issues/448#issuecomment-969446972,https://api.github.com/repos/simonw/datasette/issues/448,969446972,IC_kwDOBm6k_c45yJY8,9599,simonw,2021-11-15T23:46:13Z,2021-11-15T23:46:13Z,OWNER,"It looks like the problem here is that some of the tags occur more than once in the documents: So they get counted more than once, hence the 182 count for something that couldn't possibly return more than 172 documents.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views, https://github.com/simonw/datasette/issues/448#issuecomment-969442215,https://api.github.com/repos/simonw/datasette/issues/448,969442215,IC_kwDOBm6k_c45yIOn,9599,simonw,2021-11-15T23:42:03Z,2021-11-15T23:42:03Z,OWNER,I think this code is wrong in the `ArrayFacet` class: https://github.com/simonw/datasette/blob/502c02fa6dde6a8bb840af6c4c8cf858aa1db687/datasette/facets.py#L357-L364,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views, https://github.com/simonw/datasette/issues/448#issuecomment-969440918,https://api.github.com/repos/simonw/datasette/issues/448,969440918,IC_kwDOBm6k_c45yH6W,9599,simonw,2021-11-15T23:40:17Z,2021-11-15T23:40:35Z,OWNER,"Applied that fix to the `arraycontains` filter but I'm still getting bad results for the faceting: Should never get 182 results on a page that faceting against only 172 items. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views, https://github.com/simonw/datasette/issues/448#issuecomment-969436930,https://api.github.com/repos/simonw/datasette/issues/448,969436930,IC_kwDOBm6k_c45yG8C,9599,simonw,2021-11-15T23:31:58Z,2021-11-15T23:31:58Z,OWNER,"I think this SQL recipe may work instead: ```sql select * from ads_with_targets where 'people_who_match:interests:African-American Civil Rights Movement (1954—68)' in ( select value from json_each(target_names) ) and 'interests:Martin Luther King III' in ( select value from json_each(target_names) ) ``` https://json-view-facet-bug-demo-j7hipcg4aq-uc.a.run.app/russian-ads?sql=select%0D%0A++*%0D%0Afrom%0D%0A++ads_with_targets%0D%0Awhere%0D%0A++%27people_who_match%3Ainterests%3AAfrican-American+Civil+Rights+Movement+%281954%E2%80%9468%29%27+in+%28%0D%0A++++select%0D%0A++++++value%0D%0A++++from%0D%0A++++++json_each%28target_names%29%0D%0A++%29%0D%0A++and+%27interests%3AMartin+Luther+King+III%27+in+%28%0D%0A++++select%0D%0A++++++value%0D%0A++++from%0D%0A++++++json_each%28target_names%29%0D%0A++%29&interests=&African=&Martin=","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views, https://github.com/simonw/datasette/issues/519#issuecomment-969433734,https://api.github.com/repos/simonw/datasette/issues/519,969433734,IC_kwDOBm6k_c45yGKG,9599,simonw,2021-11-15T23:26:11Z,2021-11-15T23:26:11Z,OWNER,"I'm happy with this as the goals for 1.0. I'm going to close this issue and create three tracking tickets for the three key themes: - https://github.com/simonw/datasette/issues/1509 - https://github.com/simonw/datasette/issues/1510 - https://github.com/simonw/datasette/issues/1511","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459590021,Decide what goes into Datasette 1.0, https://github.com/simonw/sqlite-utils/issues/329#issuecomment-968470212,https://api.github.com/repos/simonw/sqlite-utils/issues/329,968470212,IC_kwDOCGYnMM45ua7E,9599,simonw,2021-11-15T02:49:28Z,2021-11-15T02:49:28Z,OWNER,"I was going to replace all of the `validate_column_names()` bits with something that fixed them instead, but I think I have a better idea: I'm only going to apply the fix for the various '.insert()` methods that create the initial tables. I'll keep the `validate_column_names()` where they are at the moment. Once you've inserted the data and created the tables it will be up to you to use the new, correct column names. This avoids the whole issue of needing to rewrite parameters, and solves the immediate problem which is consuming CSV files with bad column names.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1005891028,Rethink approach to [ and ] in column names (currently throws error), https://github.com/simonw/sqlite-utils/issues/329#issuecomment-968458837,https://api.github.com/repos/simonw/sqlite-utils/issues/329,968458837,IC_kwDOCGYnMM45uYJV,9599,simonw,2021-11-15T02:21:15Z,2021-11-15T02:21:15Z,OWNER,"I'm not going to implement a fix that rewrites the `pk` and `column_order` and other parameters - at least not yet. The main thing I'm trying to fix here is what happens when you attempt to import a CSV file with `[ ]` in the column names, which should be unaffected by that second challenge.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1005891028,Rethink approach to [ and ] in column names (currently throws error), https://github.com/simonw/sqlite-utils/issues/329#issuecomment-968453129,https://api.github.com/repos/simonw/sqlite-utils/issues/329,968453129,IC_kwDOCGYnMM45uWwJ,9599,simonw,2021-11-15T02:07:46Z,2021-11-15T02:07:46Z,OWNER,"If I replace `validate_column_names(row.keys())` with `fix_column_names(row)` I need to decide what to do about things like `pk=` and `column_order=`. What should the following do? ```python table.insert({""foo[bar]"": 4}, pk=""foo[bar]"", column_order=[""foo[bar]""]) ``` Should it spot the old column names in the `pk=` and `column_order=` parameters and pretend that `foo_bar_` was passed instead? ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1005891028,Rethink approach to [ and ] in column names (currently throws error), https://github.com/simonw/sqlite-utils/issues/329#issuecomment-968451954,https://api.github.com/repos/simonw/sqlite-utils/issues/329,968451954,IC_kwDOCGYnMM45uWdy,9599,simonw,2021-11-15T02:05:29Z,2021-11-15T02:05:29Z,OWNER,"> I could even have those replacement characters be properties of the `Database` class, so uses can sub-class and change them. I'm not going to do this, it's unnecessary extra complexity and it means the function that fixes the column names needs to have access to the current `Database` instance.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1005891028,Rethink approach to [ and ] in column names (currently throws error), https://github.com/simonw/sqlite-utils/issues/339#issuecomment-968450579,https://api.github.com/repos/simonw/sqlite-utils/issues/339,968450579,IC_kwDOCGYnMM45uWIT,9599,simonw,2021-11-15T02:02:34Z,2021-11-15T02:02:34Z,OWNER,Documentation: https://github.com/simonw/sqlite-utils/blob/54a2269e91ce72b059618662ed133a85f3d42e4a/docs/python-api.rst#working-with-lookup-tables,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1053122092,`table.lookup()` option to populate additional columns when creating a record, https://github.com/simonw/sqlite-utils/issues/339#issuecomment-968435041,https://api.github.com/repos/simonw/sqlite-utils/issues/339,968435041,IC_kwDOCGYnMM45uSVh,9599,simonw,2021-11-15T01:44:42Z,2021-11-15T01:44:42Z,OWNER,"`lookup(column_values, extra_values)` is one option. `column_values` isn't actually a great name for the first parameter any more, since the second parameter also takes column values. The first parameter is now all about the unique lookup values. Maybe this: lookup(lookup_values, extra_values)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1053122092,`table.lookup()` option to populate additional columns when creating a record, https://github.com/simonw/sqlite-utils/issues/339#issuecomment-968434594,https://api.github.com/repos/simonw/sqlite-utils/issues/339,968434594,IC_kwDOCGYnMM45uSOi,9599,simonw,2021-11-15T01:43:10Z,2021-11-15T01:43:10Z,OWNER,What should I call this parameter? Django has a similar feature where it calls them `defaults=` (for `get_or_create()`) but I'm not a huge fan of that name.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1053122092,`table.lookup()` option to populate additional columns when creating a record, https://github.com/simonw/sqlite-utils/issues/339#issuecomment-968434425,https://api.github.com/repos/simonw/sqlite-utils/issues/339,968434425,IC_kwDOCGYnMM45uSL5,9599,simonw,2021-11-15T01:42:36Z,2021-11-15T01:42:36Z,OWNER,"Here's the current signature of `table.lookup()`: https://github.com/simonw/sqlite-utils/blob/9cda5b070f885a7995f0c307bcc4f45f0812994a/sqlite_utils/db.py#L2716-L2729 I'm going to add a second positional argument which can provide a dictionary of column->value to use when creating the original table and populating the initial row. If the row already exists, those columns will be ignored entirely.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1053122092,`table.lookup()` option to populate additional columns when creating a record, https://github.com/simonw/sqlite-utils/pull/322#issuecomment-968401459,https://api.github.com/repos/simonw/sqlite-utils/issues/322,968401459,IC_kwDOCGYnMM45uKIz,9599,simonw,2021-11-15T00:26:42Z,2021-11-15T00:26:42Z,OWNER,"This relates to the fact that dictionaries, lists and tuples get special treatment and are converted to JSON strings, using this code: https://github.com/simonw/sqlite-utils/blob/e8d958109ee290cfa1b44ef7a39629bb50ab673e/sqlite_utils/db.py#L2937-L2947 So the `COLUMN_TYPE_MAPPING` should include those too - right now it looks like this: https://github.com/simonw/sqlite-utils/blob/e8d958109ee290cfa1b44ef7a39629bb50ab673e/sqlite_utils/db.py#L165-L188","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",979612115,Add dict type to be mapped as TEXT in sqllite, https://github.com/simonw/sqlite-utils/pull/324#issuecomment-968384988,https://api.github.com/repos/simonw/sqlite-utils/issues/324,968384988,IC_kwDOCGYnMM45uGHc,9599,simonw,2021-11-14T23:25:16Z,2021-11-14T23:25:16Z,OWNER,"Yes this was absolutely the intention! Thanks, I wonder how often I've made that mistake in other projects?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",988013247,Use python-dateutil package instead of dateutils, https://github.com/simonw/sqlite-utils/issues/331#issuecomment-968384005,https://api.github.com/repos/simonw/sqlite-utils/issues/331,968384005,IC_kwDOCGYnMM45uF4F,9599,simonw,2021-11-14T23:19:29Z,2021-11-14T23:20:32Z,OWNER,"Tested it like this, against a freshly built `.tar.gz` package from my development environment: ``` (w) w % mypy . hello.py:1: error: Skipping analyzing ""sqlite_utils"": found module but no type hints or library stubs hello.py:1: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports Found 1 error in 1 file (checked 1 source file) (w) w % pip install ~/Dropbox/Development/sqlite-utils/dist/sqlite-utils-3.17.1.tar.gz Processing /Users/simon/Dropbox/Development/sqlite-utils/dist/sqlite-utils-3.17.1.tar.gz ... Successfully installed sqlite-utils-3.17.1 (w) w % mypy . Success: no issues found in 1 source file ``` I tested against the `.whl` too. My `hello.py` script contained this: ```python import sqlite_utils from typing import cast if __name__ == ""__main__"": db = sqlite_utils.Database(memory=True) table = cast(sqlite_utils.db.Table, db[""foo""]) table.insert({""id"": 5}) print(list(db.query(""select * from foo""))) ``` That `cast()` is necessary because without it you get this error: ``` (w) w % mypy . hello.py:7: error: Item ""View"" of ""Union[Table, View]"" has no attribute ""insert"" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1026794056,Mypy error: found module but no type hints or library stubs, https://github.com/simonw/sqlite-utils/issues/331#issuecomment-968381939,https://api.github.com/repos/simonw/sqlite-utils/issues/331,968381939,IC_kwDOCGYnMM45uFXz,9599,simonw,2021-11-14T23:06:20Z,2021-11-14T23:06:20Z,OWNER,Thanks - I didn't know this was needed!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1026794056,Mypy error: found module but no type hints or library stubs, https://github.com/simonw/sqlite-utils/issues/332#issuecomment-968380675,https://api.github.com/repos/simonw/sqlite-utils/issues/332,968380675,IC_kwDOCGYnMM45uFED,9599,simonw,2021-11-14T22:57:56Z,2021-11-14T22:57:56Z,OWNER,This is a great idea.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1028056713,`sqlite-utils memory --flatten` option to flatten nested JSON, https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968380387,https://api.github.com/repos/simonw/sqlite-utils/issues/335,968380387,IC_kwDOCGYnMM45uE_j,9599,simonw,2021-11-14T22:55:56Z,2021-11-14T22:55:56Z,OWNER,"OK, this should fix it.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1042569687,sqlite-utils index-foreign-keys fails due to pre-existing index, https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968371112,https://api.github.com/repos/simonw/sqlite-utils/issues/335,968371112,IC_kwDOCGYnMM45uCuo,9599,simonw,2021-11-14T21:57:43Z,2021-11-14T22:21:31Z,OWNER,"`create_index(..., find_unique_name=)` is good. Default to false. `index_foreign_keys` can set it to true.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1042569687,sqlite-utils index-foreign-keys fails due to pre-existing index, https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968361671,https://api.github.com/repos/simonw/sqlite-utils/issues/335,968361671,IC_kwDOCGYnMM45uAbH,9599,simonw,2021-11-14T20:54:53Z,2021-11-14T21:01:14Z,OWNER,"I'm leaning towards `table.create_index(columns, ignore_existing_name=True)` now. Or `resolve_existing_name` - or `skip_existing_name`? ""ignore"" sounds like it might not create the index if the name exists, but we want to still create the index but pick a new name.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1042569687,sqlite-utils index-foreign-keys fails due to pre-existing index, https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968362285,https://api.github.com/repos/simonw/sqlite-utils/issues/335,968362285,IC_kwDOCGYnMM45uAkt,9599,simonw,2021-11-14T20:59:44Z,2021-11-14T20:59:44Z,OWNER,I think I'll attempt to create the index and re-try if it fails with that error.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1042569687,sqlite-utils index-foreign-keys fails due to pre-existing index, https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968362214,https://api.github.com/repos/simonw/sqlite-utils/issues/335,968362214,IC_kwDOCGYnMM45uAjm,9599,simonw,2021-11-14T20:59:15Z,2021-11-14T20:59:15Z,OWNER,"How to figure out if an index name is already in use? `PRAGMA index_list(t)` requires a table name. This does it: ```sql SELECT name FROM sqlite_master WHERE type = 'index'; ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1042569687,sqlite-utils index-foreign-keys fails due to pre-existing index, https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968361409,https://api.github.com/repos/simonw/sqlite-utils/issues/335,968361409,IC_kwDOCGYnMM45uAXB,9599,simonw,2021-11-14T20:52:55Z,2021-11-14T20:52:55Z,OWNER,"Looking at the method signature: https://github.com/simonw/sqlite-utils/blob/92aa5c9c5d26b0889c8c3d97c76a908d5f8af211/sqlite_utils/db.py#L1518-L1524 `if_not_exists` just adds a `IF NOT EXISTS` clause here: https://github.com/simonw/sqlite-utils/blob/92aa5c9c5d26b0889c8c3d97c76a908d5f8af211/sqlite_utils/db.py#L1549-L1561","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1042569687,sqlite-utils index-foreign-keys fails due to pre-existing index, https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968361285,https://api.github.com/repos/simonw/sqlite-utils/issues/335,968361285,IC_kwDOCGYnMM45uAVF,9599,simonw,2021-11-14T20:51:57Z,2021-11-14T20:51:57Z,OWNER,"SQLite will happily create multiple identical indexes on a table, using more disk space each time: ```pycon >>> import sqlite_utils >>> db = sqlite_utils.Database(""dupes.db"") >>> db[""t""].insert_all({""id"": i} for i in range(10000)) # dupes.db is 98304 bytes >>> db[""t""].create_index([""id""])
# dupes.db is 204800 bytes >>> db[""t""].indexes [Index(seq=0, name='idx_t_id', unique=0, origin='c', partial=0, columns=['id'])] >>> db[""t""].create_index([""id""], index_name=""t_idx_t_id_2"")
# 311296 bytes >>> db[""t""].create_index([""id""], index_name=""t_idx_t_id_3"")
# 417792 bytes >>> db.vacuum() # Still 417792 bytes ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1042569687,sqlite-utils index-foreign-keys fails due to pre-existing index, https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968360538,https://api.github.com/repos/simonw/sqlite-utils/issues/335,968360538,IC_kwDOCGYnMM45uAJa,9599,simonw,2021-11-14T20:46:56Z,2021-11-14T20:46:56Z,OWNER,"I'm tempted to not provide an opt-out option either: if you call `table.create_index(...)` without specifying an index name I think the tool should create the index for you, quietly picking an index name that works. But... it feels wasteful to create an index that exactly duplicates an existing index. Would SQLite even let you do that or would it notice and NOT double the amount of disk space used for that index?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1042569687,sqlite-utils index-foreign-keys fails due to pre-existing index, https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968360387,https://api.github.com/repos/simonw/sqlite-utils/issues/335,968360387,IC_kwDOCGYnMM45uAHD,9599,simonw,2021-11-14T20:45:44Z,2021-11-14T20:45:44Z,OWNER,"What would such an option be called? Some options: - `table.create_index([fk.column], force=True)` - not obvious what `force` means here - `table.create_index([fk.column], ignore_existing_name=True)` - not obvious what `ignore` means here - `table.create_index([fk.column], pick_unique_name=True)` - bit verbose If the user doesn't pass in an explicit name it seems like their intent is ""just create me the index, I don't care what name you use"" - so actually perhaps the default behaviour here should be to pick a new unique name if that name is already in use. Then maybe there should be an option to disable that - some options there: - `table.create_index([fk.column], error_on_existing_index_name=True)` - too verbose - `table.create_index([fk.column], force=False)` - not clear what `force` means ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1042569687,sqlite-utils index-foreign-keys fails due to pre-existing index, https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968359868,https://api.github.com/repos/simonw/sqlite-utils/issues/335,968359868,IC_kwDOCGYnMM45t_-8,9599,simonw,2021-11-14T20:41:42Z,2021-11-14T20:41:42Z,OWNER,"The ""index idx_generators_eia860_report_date already exists"" error suggests that the problem here is actually one of an index name collision. ```python table.create_index([fk.column]) ``` This will derive a name for the index automatically from the name of the table and the name of the passed in columns: https://github.com/simonw/sqlite-utils/blob/92aa5c9c5d26b0889c8c3d97c76a908d5f8af211/sqlite_utils/db.py#L1536-L1539 So perhaps `.create_index()` should grow an extra option that creates the index even if the name already exists, by finding a new name.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1042569687,sqlite-utils index-foreign-keys fails due to pre-existing index, https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968359137,https://api.github.com/repos/simonw/sqlite-utils/issues/335,968359137,IC_kwDOCGYnMM45t_zh,9599,simonw,2021-11-14T20:37:00Z,2021-11-14T20:37:00Z,OWNER,This is strange - the code already checks that an index doesn't exist before attempting to create it: https://github.com/simonw/sqlite-utils/blob/92aa5c9c5d26b0889c8c3d97c76a908d5f8af211/sqlite_utils/db.py#L893-L902,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1042569687,sqlite-utils index-foreign-keys fails due to pre-existing index, https://github.com/simonw/datasette/issues/1507#issuecomment-968210842,https://api.github.com/repos/simonw/datasette/issues/1507,968210842,IC_kwDOBm6k_c45tbma,9599,simonw,2021-11-14T05:41:55Z,2021-11-14T05:41:55Z,OWNER,"Here's the build with that fix: https://readthedocs.org/projects/datasette/builds/15268498/ It passed and published the docs: https://docs.datasette.io/en/latest/changelog.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1052851176,ReadTheDocs build failed for 0.59.2 release, https://github.com/simonw/datasette/issues/1507#issuecomment-968210222,https://api.github.com/repos/simonw/datasette/issues/1507,968210222,IC_kwDOBm6k_c45tbcu,9599,simonw,2021-11-14T05:34:14Z,2021-11-14T05:34:14Z,OWNER,"Here's the new build using Python 3: https://readthedocs.org/projects/datasette/builds/15268482/ It's still broken. Here's one of many issue threads about it, this one has a workaround fix: https://github.com/readthedocs/readthedocs.org/issues/8616#issuecomment-952034858 > For future readers, the solution for this problem is to pin `docutils<0.18` in your `requirements.txt` file, and have a `.readthedocs.yaml` file with these contents: > > ``` > version: 2 > > python: > install: > - requirements: docs/requirements.txt > ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1052851176,ReadTheDocs build failed for 0.59.2 release, https://github.com/simonw/datasette/issues/1507#issuecomment-968209957,https://api.github.com/repos/simonw/datasette/issues/1507,968209957,IC_kwDOBm6k_c45tbYl,9599,simonw,2021-11-14T05:31:07Z,2021-11-14T05:31:07Z,OWNER,"Looks like ReadTheDocs builds started failing for `latest` a few weeks ago: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1052851176,ReadTheDocs build failed for 0.59.2 release, https://github.com/simonw/datasette/issues/1507#issuecomment-968209731,https://api.github.com/repos/simonw/datasette/issues/1507,968209731,IC_kwDOBm6k_c45tbVD,9599,simonw,2021-11-14T05:28:41Z,2021-11-14T05:28:41Z,OWNER,"I will try adding a `.readthedocs.yml` file: https://docs.readthedocs.io/en/stable/config-file/v2.html#python-version This might work: ``` version: 2 build: os: ubuntu-20.04 tools: python: ""3.9"" sphinx: configuration: docs/conf.py ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1052851176,ReadTheDocs build failed for 0.59.2 release, https://github.com/simonw/datasette/issues/1507#issuecomment-968209616,https://api.github.com/repos/simonw/datasette/issues/1507,968209616,IC_kwDOBm6k_c45tbTQ,9599,simonw,2021-11-14T05:27:22Z,2021-11-14T05:27:22Z,OWNER,https://blog.readthedocs.com/default-python-3/ they started defaulting new projects to Python 3 back in Feb 2019 but clearly my project was created before then.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1052851176,ReadTheDocs build failed for 0.59.2 release, https://github.com/simonw/datasette/issues/1507#issuecomment-968209560,https://api.github.com/repos/simonw/datasette/issues/1507,968209560,IC_kwDOBm6k_c45tbSY,9599,simonw,2021-11-14T05:26:36Z,2021-11-14T05:26:36Z,OWNER,"It looks like my builds there still run on Python 2! ``` git clone --no-single-branch --depth 50 https://github.com/simonw/datasette . git checkout --force de1e031713f47fbd51eb7239db3e7e6025fbf81a git clean -d -f -f python2.7 -mvirtualenv /home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2 /home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/bin/python -m pip install --upgrade --no-cache-dir pip setuptools /home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/bin/python -m pip install --upgrade --no-cache-dir mock==1.0.1 pillow==5.4.1 alabaster>=0.7,<0.8,!=0.7.5 commonmark==0.8.1 recommonmark==0.5.0 sphinx<2 sphinx-rtd-theme<0.5 readthedocs-sphinx-ext<2.2 cat docs/conf.py ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1052851176,ReadTheDocs build failed for 0.59.2 release, https://github.com/simonw/datasette/issues/1503#issuecomment-968207906,https://api.github.com/repos/simonw/datasette/issues/1503,968207906,IC_kwDOBm6k_c45ta4i,9599,simonw,2021-11-14T05:08:26Z,2021-11-14T05:08:26Z,OWNER,"Error: ``` def test_table_html_filter_form_column_options( path, expected_column_options, app_client ): response = app_client.get(path) assert response.status == 200 form = Soup(response.body, ""html.parser"").find(""form"") column_options = [ o.attrs.get(""value"") or o.string for o in form.select(""select[name=_filter_column] option"") ] > assert expected_column_options == column_options E AssertionError: assert ['- column -'...wid', 'value'] == ['- column -', 'value'] E At index 1 diff: 'rowid' != 'value' E Left contains one more item: 'value' E Use -v to get the full diff ``` This is because `rowid` isn't a table column but IS returned by the query used on that page. My solution: start with the query columns, but then add any table columns that were not already returned by the query to the end of the `filter_columns` list.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1050163432,`?_nocol=` removes that column from the filter interface, https://github.com/simonw/datasette/issues/1506#issuecomment-968192980,https://api.github.com/repos/simonw/datasette/issues/1506,968192980,IC_kwDOBm6k_c45tXPU,9599,simonw,2021-11-14T02:22:40Z,2021-11-14T02:22:40Z,OWNER,"I think the answer is to spot this case and link to `?_item_exact=x` instead of `?_item=x` - it looks like that's already recommended in the documentation here: https://docs.datasette.io/en/stable/json_api.html#column-filter-arguments > **?column__exact=value** or **?_column=value** > Returns rows where the specified column exactly matches the value. So maybe the facet selection rendering logic needs to spot this and link correctly to it?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1052826038,Columns beginning with an underscore do not facet correctly, https://github.com/simonw/sqlite-utils/issues/336#issuecomment-962411119,https://api.github.com/repos/simonw/sqlite-utils/issues/336,962411119,IC_kwDOCGYnMM45XTpv,9599,simonw,2021-11-06T07:21:04Z,2021-11-06T07:21:04Z,OWNER,I've never used `DEFAULT 'CURRENT_TIMESTAMP'` myself so this one should be an interesting bug to explore.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1044267332,"sqlite-util tranform --column-order mangles columns of type ""timestamp""", https://github.com/simonw/sqlite-utils/issues/206#issuecomment-955367409,https://api.github.com/repos/simonw/sqlite-utils/issues/206,955367409,IC_kwDOCGYnMM448b_x,9599,simonw,2021-10-30T15:50:39Z,2021-10-30T15:50:39Z,OWNER,"What's the error message? Sometimes I pipe JSON through `jq` to check if it's valid: cat my.json | jq","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",761915790,sqlite-utils should suggest --csv if JSON parsing fails, https://github.com/simonw/datasette/issues/1497#issuecomment-953508979,https://api.github.com/repos/simonw/datasette/issues/1497,953508979,IC_kwDOBm6k_c441WRz,9599,simonw,2021-10-28T05:13:49Z,2021-10-28T05:13:49Z,OWNER,Wrote about this in my weeknotes: https://simonwillison.net/2021/Oct/28/weeknotes-kubernetes-web-components/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950417375,https://api.github.com/repos/simonw/datasette/issues/1497,950417375,IC_kwDOBm6k_c44pjff,9599,simonw,2021-10-24T23:36:54Z,2021-10-24T23:36:54Z,OWNER,"Tried fixing this by pushing a new `latest` tag from my laptop: ``` (datasette) datasette % docker pull datasetteproject/datasette:0.59.1 0.59.1: Pulling from datasetteproject/datasette 7d63c13d9b9b: Already exists 6ad2a11ca37b: Already exists e9edbe81a001: Already exists 36629b83aba2: Already exists 7338abefe51c: Already exists 6b825daddc6c: Pull complete d7508b065a21: Pull complete Digest: sha256:dc134f65bec40ed4ea7049188fe1e3915b8e6c3fd999b17effe8ec24868b979c Status: Downloaded newer image for datasetteproject/datasette:0.59.1 docker.io/datasetteproject/datasette:0.59.1 (datasette) datasette % docker tag datasetteproject/datasette:0.59.1 datasetteproject/datasette:latest (datasette) datasette % docker push datasetteproject/datasette:latest The push refers to repository [docker.io/datasetteproject/datasette] d668c99b6ff1: Layer already exists aa20c9013575: Layer already exists c97eebf2b227: Layer already exists 284a6c64b82c: Layer already exists 388eedeb736e: Layer already exists 2feece0964b8: Layer already exists e8b689711f21: Layer already exists errors: denied: requested access to the resource is denied unauthorized: authentication required ``` So I logged in with `docker login`: ``` (datasette) datasette % docker login Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one. Username: datasetteproject Password: ``` And ran the push again and it worked: ``` (datasette) datasette % docker push datasetteproject/datasette:latest The push refers to repository [docker.io/datasetteproject/datasette] d668c99b6ff1: Layer already exists aa20c9013575: Layer already exists c97eebf2b227: Layer already exists 284a6c64b82c: Layer already exists 388eedeb736e: Layer already exists 2feece0964b8: Layer already exists e8b689711f21: Layer already exists latest: digest: sha256:dc134f65bec40ed4ea7049188fe1e3915b8e6c3fd999b17effe8ec24868b979c size: 1793 ``` https://hub.docker.com/layers/datasetteproject/datasette/latest/images/sha256-dc134f65bec40ed4ea7049188fe1e3915b8e6c3fd999b17effe8ec24868b979c?context=explore","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950416802,https://api.github.com/repos/simonw/datasette/issues/1497,950416802,IC_kwDOBm6k_c44pjWi,9599,simonw,2021-10-24T23:32:39Z,2021-10-24T23:32:39Z,OWNER,"That's because the `publish.yml` workflow ends with this, which isn't in the `push_docker_tag.yml` workflow: https://github.com/simonw/datasette/blob/2c31d1cd9cd3b63458ccbe391866499fa3f44978/.github/workflows/publish.yml#L117-L119","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950416682,https://api.github.com/repos/simonw/datasette/issues/1497,950416682,IC_kwDOBm6k_c44pjUq,9599,simonw,2021-10-24T23:31:51Z,2021-10-24T23:31:51Z,OWNER,One catch: the `latest` tag on Docker Hub is still three months old.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950416659,https://api.github.com/repos/simonw/datasette/issues/1497,950416659,IC_kwDOBm6k_c44pjUT,9599,simonw,2021-10-24T23:31:41Z,2021-10-24T23:31:41Z,OWNER,"Published `0.59.1` as well: https://github.com/simonw/datasette/runs/3991214225?check_suite_focus=true Result: https://hub.docker.com/layers/datasetteproject/datasette/0.59.1/images/sha256-dc134f65bec40ed4ea7049188fe1e3915b8e6c3fd999b17effe8ec24868b979c?context=explore","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950416460,https://api.github.com/repos/simonw/datasette/issues/1497,950416460,IC_kwDOBm6k_c44pjRM,9599,simonw,2021-10-24T23:30:10Z,2021-10-24T23:30:10Z,OWNER,"Testing that newly published image: ``` % docker run -p 8002:8001 -v `pwd`:/mnt \ datasetteproject/datasette:0.59 datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db Unable to find image 'datasetteproject/datasette:0.59' locally 0.59: Pulling from datasetteproject/datasette 7d63c13d9b9b: Already exists 6ad2a11ca37b: Already exists e9edbe81a001: Already exists 36629b83aba2: Already exists 7338abefe51c: Already exists 6d71b6b88b82: Pull complete 8c4da3c56bdc: Pull complete Digest: sha256:038decc28e0ea84b281ecc0058fe8eba7aa99596e5a2177ff714092ad03294ed Status: Downloaded newer image for datasetteproject/datasette:0.59 INFO: Started server process [1] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit) ``` and `http://localhost:8002/versions.json` returns: ```json { ""python"": { ""version"": ""3.9.7"", ""full"": ""3.9.7 (default, Oct 12 2021, 02:43:43) \n[GCC 10.2.1 20210110]"" }, ""datasette"": { ""version"": ""0.59"" }, ""asgi"": ""3.0"", ""uvicorn"": ""0.15.0"", ""sqlite"": { ""version"": ""3.34.1"" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950416061,https://api.github.com/repos/simonw/datasette/issues/1497,950416061,IC_kwDOBm6k_c44pjK9,9599,simonw,2021-10-24T23:27:18Z,2021-10-24T23:27:18Z,OWNER,That worked: https://hub.docker.com/layers/datasetteproject/datasette/0.59/images/sha256-038decc28e0ea84b281ecc0058fe8eba7aa99596e5a2177ff714092ad03294ed?context=explore,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950415822,https://api.github.com/repos/simonw/datasette/issues/1497,950415822,IC_kwDOBm6k_c44pjHO,9599,simonw,2021-10-24T23:25:45Z,2021-10-24T23:25:45Z,OWNER,I'm going to attempt to publish `0.59` to Docker Hub using https://github.com/simonw/datasette/blob/2c31d1cd9cd3b63458ccbe391866499fa3f44978/.github/workflows/push_docker_tag.yml - if that works I'll push `0.59.1` as well.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950415129,https://api.github.com/repos/simonw/datasette/issues/1497,950415129,IC_kwDOBm6k_c44pi8Z,9599,simonw,2021-10-24T23:21:33Z,2021-10-24T23:21:33Z,OWNER,That fixed it! Resulting image is 249MB which is a very slight size reduction (I think previous was 259MB (uncompressed).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950413185,https://api.github.com/repos/simonw/datasette/issues/1497,950413185,IC_kwDOBm6k_c44pieB,9599,simonw,2021-10-24T23:16:25Z,2021-10-24T23:18:30Z,OWNER,"Debian stable these days is ""bullseye"" - https://www.debian.org/releases/ - which has the version of SpatiaLite that I was previously pulling in from Sid: https://packages.debian.org/bullseye/libsqlite3-mod-spatialite So upgrading to the 3.9.7-slim-bullseye base image may help. https://hub.docker.com/layers/python/library/python/3.9.7-slim-bullseye/images/sha256-67af5f544115124dc6d6da1d9d2815aa9825f6fd4aa6710adb0ec1725280fb89?context=explore","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950412628,https://api.github.com/repos/simonw/datasette/issues/1497,950412628,IC_kwDOBm6k_c44piVU,9599,simonw,2021-10-24T23:13:20Z,2021-10-24T23:13:27Z,OWNER,"I think the root cause here is that I'm using a Debian Buster base image and then installing SpatiaLite from Debian unstable (sid) - as described in this comment: https://github.com/simonw/datasette/issues/1249#issuecomment-804309510 That's has worked fine in the past, but Sid is unstable - and this seems to be one of those instabilities.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950411417,https://api.github.com/repos/simonw/datasette/issues/1497,950411417,IC_kwDOBm6k_c44piCZ,9599,simonw,2021-10-24T23:06:45Z,2021-10-24T23:11:14Z,OWNER,"Same errors with `3.9.7`: ``` #5 41.46 /usr/bin/perl: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory #5 41.46 dpkg: error processing package libc6:amd64 (--configure): #5 41.46 installed libc6:amd64 package post-installation script subprocess returned error exit status 127 #5 41.47 Errors were encountered while processing: #5 41.47 libc6:amd64 #5 41.50 E: Sub-process /usr/bin/dpkg returned an error code (1) ``` I'm suspicious of this part of the `Dockerfile`: https://github.com/simonw/datasette/blob/e6e44372b34414eac2f36a4c1120af4f755aa423/Dockerfile#L1-L18 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950411912,https://api.github.com/repos/simonw/datasette/issues/1497,950411912,IC_kwDOBm6k_c44piKI,9599,simonw,2021-10-24T23:09:41Z,2021-10-24T23:09:41Z,OWNER,Here that is in the Debian bug tracker: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=993755,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950411808,https://api.github.com/repos/simonw/datasette/issues/1497,950411808,IC_kwDOBm6k_c44piIg,9599,simonw,2021-10-24T23:08:59Z,2021-10-24T23:08:59Z,OWNER,"Looks like it's this bug, reported on the Debian mailing list: https://www.mail-archive.com/debian-bugs-dist@lists.debian.org/msg1818037.html No obvious workaround there though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950411320,https://api.github.com/repos/simonw/datasette/issues/1497,950411320,IC_kwDOBm6k_c44piA4,9599,simonw,2021-10-24T23:06:05Z,2021-10-24T23:06:05Z,OWNER,"Right now the base image is: https://github.com/simonw/datasette/blob/e6e44372b34414eac2f36a4c1120af4f755aa423/Dockerfile#L1 I'm going to try `python:3.9.7-slim-buster` instead: https://hub.docker.com/layers/python/library/python/3.9.7-slim-buster/images/sha256-290b95e4b379762a9bd3d72644598e0972f4e2b5442bba60592c018fadcc744d?context=explore","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950410718,https://api.github.com/repos/simonw/datasette/issues/1497,950410718,IC_kwDOBm6k_c44ph3e,9599,simonw,2021-10-24T23:02:30Z,2021-10-24T23:02:30Z,OWNER,I got the same error publishing 0.59: https://github.com/simonw/datasette/actions/runs/1343251945,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950410554,https://api.github.com/repos/simonw/datasette/issues/1497,950410554,IC_kwDOBm6k_c44ph06,9599,simonw,2021-10-24T23:01:20Z,2021-10-24T23:01:28Z,OWNER,"I can replicate locally by running: ``` docker build -f Dockerfile \ -t datasetteproject/datasette:0.59.1 \ --build-arg VERSION=0.59.1 . ``` This gives me the same error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/pull/1495#issuecomment-950403692,https://api.github.com/repos/simonw/datasette/issues/1495,950403692,IC_kwDOBm6k_c44pgJs,9599,simonw,2021-10-24T22:10:43Z,2021-10-24T22:10:43Z,OWNER,"To land this change we'll need a unit test that demonstrates the new capability - I suggest putting that next to this test: https://github.com/simonw/datasette/blob/15a9d4abfff0c45dee2a9f851326e1d61b1c678c/tests/test_plugins.py#L648-L659 It will also need documentation, which should be added here: https://github.com/simonw/datasette/blob/15a9d4abfff0c45dee2a9f851326e1d61b1c678c/docs/plugin_hooks.rst#register-routes-datasette","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1033678984,Allow routes to have extra options, https://github.com/simonw/datasette/pull/1495#issuecomment-950403521,https://api.github.com/repos/simonw/datasette/issues/1495,950403521,IC_kwDOBm6k_c44pgHB,9599,simonw,2021-10-24T22:09:18Z,2021-10-24T22:09:18Z,OWNER,"This is a great idea - I've wanted this myself before, but never spent any time thinking about how to achieve it. I think your design here is exactly right - an optional third item in the tuple consisting of a dictionary of options to pass to the view function.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1033678984,Allow routes to have extra options, https://github.com/simonw/datasette/issues/1482#issuecomment-950402273,https://api.github.com/repos/simonw/datasette/issues/1482,950402273,IC_kwDOBm6k_c44pfzh,9599,simonw,2021-10-24T22:00:29Z,2021-10-24T22:00:29Z,OWNER,Janus 0.6.2 is out now and should have the fix.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1021550542,Support Python 3.10, https://github.com/simonw/datasette/issues/1496#issuecomment-949912718,https://api.github.com/repos/simonw/datasette/issues/1496,949912718,IC_kwDOBm6k_c44noSO,9599,simonw,2021-10-22T19:38:23Z,2021-10-22T19:38:23Z,OWNER,https://docs.datasette.io/en/latest/sql_queries.html#named-parameters,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1033864602,Named parameters docs should include an example of a cast, https://github.com/simonw/datasette/issues/1493#issuecomment-946360891,https://api.github.com/repos/simonw/datasette/issues/1493,946360891,IC_kwDOBm6k_c44aFI7,9599,simonw,2021-10-19T04:37:27Z,2021-10-19T04:37:27Z,OWNER,"I renamed `/:memory:` to `/_memory` in version 0.55 - https://docs.datasette.io/en/stable/changelog.html#v0-55 But... in 0.59 I stopped following HTTP redirects by default, which is why this used to work and no longer does! So the fix is to update the Homebrew regression test to use this instead: datasette --get '/_memory.json?sql=select+3*5' Thanks for catching this!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1028115674,`--get '/:memory:.json?sql=select+3*5'` error with datasette 0.59, https://github.com/simonw/datasette/issues/1470#issuecomment-946097058,https://api.github.com/repos/simonw/datasette/issues/1470,946097058,IC_kwDOBm6k_c44ZEui,9599,simonw,2021-10-18T19:30:15Z,2021-10-18T19:30:15Z,OWNER,https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_next=200&_sort=rowid is fixed now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",995098231,?_sort=rowid with _next= returns error, https://github.com/simonw/datasette/pull/1481#issuecomment-945020210,https://api.github.com/repos/simonw/datasette/issues/1481,945020210,IC_kwDOBm6k_c44U90y,9599,simonw,2021-10-16T23:19:51Z,2021-10-16T23:19:51Z,OWNER,"Since that Janus PR hasn't been merged yet, one temporary option for a fix would be to entirely vendor the fixed Janus - https://github.com/aio-libs/janus/blob/9e13d3fb74e2c93d7501443b370a455d1b302b1f/janus/__init__.py - since it's only a single module.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10, https://github.com/simonw/datasette/pull/1481#issuecomment-944986367,https://api.github.com/repos/simonw/datasette/issues/1481,944986367,IC_kwDOBm6k_c44U1j_,9599,simonw,2021-10-16T19:07:38Z,2021-10-16T19:09:02Z,OWNER,This is blocking an upgrade for the Homebrew Datasette package: https://github.com/Homebrew/homebrew-core/pull/86932,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10, https://github.com/simonw/datasette/pull/1467#issuecomment-943632697,https://api.github.com/repos/simonw/datasette/issues/1467,943632697,IC_kwDOBm6k_c44PrE5,9599,simonw,2021-10-14T18:54:18Z,2021-10-14T18:54:18Z,OWNER,The test there failed because it turns out there's a whole bunch of places that set the `Access-Control-Allow-Origin` header. I'm going to close this PR and ship a fix that refactors those places to use the same code.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",991575770,Add Authorization header when CORS flag is set, https://github.com/simonw/datasette/pull/1467#issuecomment-943623246,https://api.github.com/repos/simonw/datasette/issues/1467,943623246,IC_kwDOBm6k_c44PoxO,9599,simonw,2021-10-14T18:42:19Z,2021-10-14T18:42:19Z,OWNER,This looks like a good fix to me.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",991575770,Add Authorization header when CORS flag is set, https://github.com/simonw/datasette/pull/1458#issuecomment-943620649,https://api.github.com/repos/simonw/datasette/issues/1458,943620649,IC_kwDOBm6k_c44PoIp,9599,simonw,2021-10-14T18:38:58Z,2021-10-14T18:38:58Z,OWNER,"This is a great idea, thanks.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",988555009,Rework the `--static` documentation a bit, https://github.com/simonw/datasette/pull/1489#issuecomment-943594712,https://api.github.com/repos/simonw/datasette/issues/1489,943594712,IC_kwDOBm6k_c44PhzY,9599,simonw,2021-10-14T18:04:11Z,2021-10-14T18:04:11Z,OWNER,@dependabot recreate,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1026379132,"Update pyyaml requirement from ~=5.3 to >=5.3,<7.0", https://github.com/simonw/datasette/issues/1488#issuecomment-942782673,https://api.github.com/repos/simonw/datasette/issues/1488,942782673,IC_kwDOBm6k_c44MbjR,9599,simonw,2021-10-13T23:04:54Z,2021-10-13T23:04:54Z,OWNER,"I think this is the change in `httpx` which is causing the bug for me: https://github.com/encode/httpx/commit/ff9813e84dab56f0f3c4ef3a159a4cce8c644a91#diff-0d0cbe9ebcd03cc8c780b0407762540a082f70cc64257f2fcd588cc30f43c15cR96 Previously it was using `path` from `path, _, query = full_path.partition(b""?"")` to populate the `raw_path` key - but it changed to instead using `request.url.raw_path` which presumably implements the logic that includes the query string.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1025754125,Upgrade to httpx 0.20.0 (request() got an unexpected keyword argument 'allow_redirects'), https://github.com/simonw/datasette/issues/1488#issuecomment-942779926,https://api.github.com/repos/simonw/datasette/issues/1488,942779926,IC_kwDOBm6k_c44Ma4W,9599,simonw,2021-10-13T22:59:05Z,2021-10-13T22:59:05Z,OWNER,This is weird - as far as I can tell `httpx` has included the query string in `raw_path` for well over a year: https://github.com/encode/httpx/commit/8e4a8a1c73f60fe5754f95b308beaa725cb8791d#diff-c9a78eb3b5f5c4fac4e5552165fbdd5320c7e3fadf9eedabcb5461393466c090R235,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1025754125,Upgrade to httpx 0.20.0 (request() got an unexpected keyword argument 'allow_redirects'), https://github.com/simonw/datasette/issues/1488#issuecomment-942778673,https://api.github.com/repos/simonw/datasette/issues/1488,942778673,IC_kwDOBm6k_c44Makx,9599,simonw,2021-10-13T22:55:44Z,2021-10-13T22:55:44Z,OWNER,"``` (Pdb) request.scope['path'] '/_memory.json' (Pdb) request.scope['raw_path'] b'/_memory.json?sql=select+sqlite_version()' ``` So `raw_path` now includes the query string.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1025754125,Upgrade to httpx 0.20.0 (request() got an unexpected keyword argument 'allow_redirects'), https://github.com/simonw/datasette/issues/1488#issuecomment-942778382,https://api.github.com/repos/simonw/datasette/issues/1488,942778382,IC_kwDOBm6k_c44MagO,9599,simonw,2021-10-13T22:55:01Z,2021-10-13T22:55:01Z,OWNER,"I think the issue is in `route_path()`: ``` > /Users/simon/Dropbox/Development/datasette/datasette/app.py(1182)route_path() -> response = await view(request, send) (Pdb) path '/_memory.json?sql=select+sqlite_version()' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1025754125,Upgrade to httpx 0.20.0 (request() got an unexpected keyword argument 'allow_redirects'), https://github.com/simonw/datasette/issues/1488#issuecomment-942777414,https://api.github.com/repos/simonw/datasette/issues/1488,942777414,IC_kwDOBm6k_c44MaRG,9599,simonw,2021-10-13T22:52:40Z,2021-10-13T22:52:40Z,OWNER,"Upgrading to 0.20.0 gives me lots of the following errors: '{""ok"": false, ""error"": ""Database not found: .json?_sort=relationships"", ""status"": 404, ""title"": null}' It looks like the full query string is now being treated as the name of the database. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1025754125,Upgrade to httpx 0.20.0 (request() got an unexpected keyword argument 'allow_redirects'), https://github.com/simonw/datasette/issues/1469#issuecomment-942725632,https://api.github.com/repos/simonw/datasette/issues/1469,942725632,IC_kwDOBm6k_c44MNoA,9599,simonw,2021-10-13T21:13:30Z,2021-10-13T21:13:30Z,OWNER,"The core problem here is treating the `?_facet=` query string parameters as the point of truth for which facets are currently enabled. Instead, I could use a `data-` attribute on the displayed facets.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",994450961,"Column cog shows ""facet by this"" when already default faceted", https://github.com/simonw/datasette/pull/1487#issuecomment-942722595,https://api.github.com/repos/simonw/datasette/issues/1487,942722595,IC_kwDOBm6k_c44MM4j,9599,simonw,2021-10-13T21:08:53Z,2021-10-13T21:08:53Z,OWNER,Thanks for this!,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1023245060,"Added instructions for installing plugins via pipx, #1486", https://github.com/simonw/datasette/issues/1470#issuecomment-939386591,https://api.github.com/repos/simonw/datasette/issues/1470,939386591,IC_kwDOBm6k_c43_ebf,9599,simonw,2021-10-10T01:17:34Z,2021-10-10T01:17:34Z,OWNER,I'll open a separate issue for removing `_next=` when running a search.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",995098231,?_sort=rowid with _next= returns error, https://github.com/simonw/datasette/issues/1482#issuecomment-939191311,https://api.github.com/repos/simonw/datasette/issues/1482,939191311,IC_kwDOBm6k_c43-uwP,9599,simonw,2021-10-09T00:35:04Z,2021-10-09T00:35:04Z,OWNER,I think that SQLite error message difference was caused by https://github.com/python/cpython/commit/a50e28377bcf37121b55c2de70d95a5386c478f8 or related work.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1021550542,Support Python 3.10, https://github.com/simonw/datasette/pull/1481#issuecomment-939185319,https://api.github.com/repos/simonw/datasette/issues/1481,939185319,IC_kwDOBm6k_c43-tSn,9599,simonw,2021-10-09T00:04:54Z,2021-10-09T00:04:54Z,OWNER,"I applied my PR against Janus to my local copy of Datasette like so: pip uninstall janus pip install https://github.com/aio-libs/janus/archive/9e13d3fb74e2c93d7501443b370a455d1b302b1f.zip Then I ran the Datasette tests and got a much happier pass rate. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10, https://github.com/simonw/datasette/pull/1481#issuecomment-939180313,https://api.github.com/repos/simonw/datasette/issues/1481,939180313,IC_kwDOBm6k_c43-sEZ,9599,simonw,2021-10-08T23:41:39Z,2021-10-08T23:41:39Z,OWNER,I submitted a PR to Janus with a workaround for this: https://github.com/aio-libs/janus/pull/359,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10, https://github.com/simonw/datasette/pull/1481#issuecomment-939100803,https://api.github.com/repos/simonw/datasette/issues/1481,939100803,IC_kwDOBm6k_c43-YqD,9599,simonw,2021-10-08T20:33:42Z,2021-10-08T20:33:42Z,OWNER,"There's a tiny chance this could be a bug in Python 3.10 itself - I filed an issue here: https://bugs.python.org/issue45416 - in which I said: > In Python 3.10 it is not possible to instantiate an asyncio.Condition that wraps an asyncio.Lock without raising a ""loop argument must agree with lock"" exception. > > This code raises that exception: > > asyncio.Condition(asyncio.Lock()) > > This worked in previous Python versions. > > Note that the error only occurs if an event loop is running. Here's a simple script that replicates the problem: > > import asyncio > > # This runs without an exception: > print(asyncio.Condition(asyncio.Lock())) > > # This does not work: > async def example(): > print(asyncio.Condition(asyncio.Lock())) > > # This raises ""ValueError: loop argument must agree with lock"": > asyncio.run(example())","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10, https://github.com/simonw/datasette/pull/1481#issuecomment-939079727,https://api.github.com/repos/simonw/datasette/issues/1481,939079727,IC_kwDOBm6k_c43-Tgv,9599,simonw,2021-10-08T19:50:52Z,2021-10-08T19:50:52Z,OWNER,"And here's the relevant Janus code: https://github.com/aio-libs/janus/blob/d7970f8b76bcac2e087067ca4575ac845e481874/janus/__init__.py#L24-L42 ```python class Queue(Generic[T]): def __init__(self, maxsize: int = 0) -> None: self._loop = current_loop() self._maxsize = maxsize self._init(maxsize) self._unfinished_tasks = 0 self._sync_mutex = threading.Lock() self._sync_not_empty = threading.Condition(self._sync_mutex) self._sync_not_full = threading.Condition(self._sync_mutex) self._all_tasks_done = threading.Condition(self._sync_mutex) self._async_mutex = asyncio.Lock() # ""loop argument must agree with lock"" exception is raised here: self._async_not_empty = asyncio.Condition(self._async_mutex) self._async_not_full = asyncio.Condition(self._async_mutex) self._finished = asyncio.Event() self._finished.set() ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10, https://github.com/simonw/datasette/pull/1481#issuecomment-939078872,https://api.github.com/repos/simonw/datasette/issues/1481,939078872,IC_kwDOBm6k_c43-TTY,9599,simonw,2021-10-08T19:49:08Z,2021-10-08T19:49:08Z,OWNER,"Here's the code that raises that error: https://github.com/python/cpython/blob/bb3e0c240bc60fe08d332ff5955d54197f79751c/Lib/asyncio/locks.py#L219-L234 ```python class Condition(_ContextManagerMixin, mixins._LoopBoundMixin): """"""Asynchronous equivalent to threading.Condition. This class implements condition variable objects. A condition variable allows one or more coroutines to wait until they are notified by another coroutine. A new Lock object is created and used as the underlying lock. """""" def __init__(self, lock=None, *, loop=mixins._marker): super().__init__(loop=loop) if lock is None: lock = Lock() elif lock._loop is not self._get_loop(): raise ValueError(""loop argument must agree with lock"") ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10, https://github.com/simonw/datasette/pull/1481#issuecomment-939078095,https://api.github.com/repos/simonw/datasette/issues/1481,939078095,IC_kwDOBm6k_c43-THP,9599,simonw,2021-10-08T19:47:29Z,2021-10-08T19:47:29Z,OWNER,"Only mention I can find of that ""loop argument must agree with lock"" error is here - which doesn't have any tips for a workaround yet: https://giters.com/django/channels_redis/issues/278","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10, https://github.com/simonw/datasette/pull/1481#issuecomment-939076399,https://api.github.com/repos/simonw/datasette/issues/1481,939076399,IC_kwDOBm6k_c43-Ssv,9599,simonw,2021-10-08T19:43:33Z,2021-10-08T19:43:33Z,OWNER,"So maybe this is an issue with Janus? I'm using https://pypi.org/project/janus/ 0.6.1 which is the latest release, from October 2020.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10, https://github.com/simonw/datasette/pull/1481#issuecomment-939075686,https://api.github.com/repos/simonw/datasette/issues/1481,939075686,IC_kwDOBm6k_c43-Shm,9599,simonw,2021-10-08T19:42:00Z,2021-10-08T19:42:00Z,OWNER,"Running `pytest -x --pdb` helped me see this error: ``` File ""/Users/simon/Dropbox/Development/datasette/datasette/views/base.py"", line 122, in dispatch_request await self.ds.refresh_schemas() File ""/Users/simon/Dropbox/Development/datasette/datasette/app.py"", line 344, in refresh_schemas await self._refresh_schemas() File ""/Users/simon/Dropbox/Development/datasette/datasette/app.py"", line 349, in _refresh_schemas await init_internal_db(internal_db) File ""/Users/simon/Dropbox/Development/datasette/datasette/utils/internal_db.py"", line 5, in init_internal_db await db.execute_write( File ""/Users/simon/Dropbox/Development/datasette/datasette/database.py"", line 102, in execute_write return await self.execute_write_fn(_inner, block=block) File ""/Users/simon/Dropbox/Development/datasette/datasette/database.py"", line 113, in execute_write_fn reply_queue = janus.Queue() File ""/Users/simon/.local/share/virtualenvs/py310-Z8fTATkJ/lib/python3.10/site-packages/janus/__init__.py"", line 39, in __init__ self._async_not_empty = asyncio.Condition(self._async_mutex) File ""/Users/simon/.pyenv/versions/3.10.0/lib/python3.10/asyncio/locks.py"", line 234, in __init__ raise ValueError(""loop argument must agree with lock"") ValueError: loop argument must agree with lock ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10, https://github.com/simonw/datasette/pull/1481#issuecomment-939074818,https://api.github.com/repos/simonw/datasette/issues/1481,939074818,IC_kwDOBm6k_c43-SUC,9599,simonw,2021-10-08T19:40:23Z,2021-10-08T19:40:23Z,OWNER,"Then I created myself a temporary 3.10 environment using `pipenv` like so: cd /tmp mkdir py310 cd py310 pipenv shell --python /Users/simon/.pyenv/versions/3.10.0/bin/python And used that with my Datasette checkout like so: cd ~/.../datasette pip install -e '.[test]' pytest ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10, https://github.com/simonw/datasette/pull/1481#issuecomment-938142436,https://api.github.com/repos/simonw/datasette/issues/1481,938142436,IC_kwDOBm6k_c436urk,9599,simonw,2021-10-07T20:44:43Z,2021-10-07T20:44:43Z,OWNER,"The 3.10 tests failed a lot. Trying to run this locally: ``` /tmp % pyenv install 3.10 python-build: definition not found: 3.10 The following versions contain `3.10' in the name: 3.10.0a6 3.10-dev miniconda-3.10.1 miniconda3-3.10.1 See all available versions with `pyenv install --list'. If the version you need is missing, try upgrading pyenv: brew update && brew upgrade pyenv ``` So trying: brew update && brew upgrade pyenv Then did this: ``` /tmp % brew upgrade pyenv ==> Upgrading 1 outdated package: pyenv 1.2.24.1 -> 2.1.0 ``` This decided to upgrade everything by downloaded everything on the internet. Aah, Homebrew. But it looks like I have `3.10.0` available to `pyenv` now. ``` /tmp % pyenv install 3.10.0 python-build: use openssl@1.1 from homebrew python-build: use readline from homebrew Downloading Python-3.10.0.tar.xz... -> https://www.python.org/ftp/python/3.10.0/Python-3.10.0.tar.xz Installing Python-3.10.0... ... ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10, https://github.com/simonw/datasette/issues/1480#issuecomment-938134038,https://api.github.com/repos/simonw/datasette/issues/1480,938134038,IC_kwDOBm6k_c436soW,9599,simonw,2021-10-07T20:31:46Z,2021-10-07T20:31:46Z,OWNER,"I've had this problem too - my solution was to not use Cloud Run for databases larger than about 2GB, but the way you describe it here makes me think that maybe there is a workaround here which could get it to work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1015646369,Exceeding Cloud Run memory limits when deploying a 4.8G database, https://github.com/simonw/datasette/issues/1470#issuecomment-938131806,https://api.github.com/repos/simonw/datasette/issues/1470,938131806,IC_kwDOBm6k_c436sFe,9599,simonw,2021-10-07T20:28:30Z,2021-10-07T20:28:30Z,OWNER,"On further investigation this isn't related to `_search` at all - it happens when you explicitly sort by `_sort=rowid` and apply a `_next` - https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_next=200 works without an error (currently) - https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_next=200&_sort=rowid shows that error","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",995098231,?_sort=rowid with _next= returns error, https://github.com/simonw/datasette/issues/1470#issuecomment-938124652,https://api.github.com/repos/simonw/datasette/issues/1470,938124652,IC_kwDOBm6k_c436qVs,9599,simonw,2021-10-07T20:17:53Z,2021-10-07T20:18:55Z,OWNER,"Here's the exception: ``` -> params[f""p{len(params)}""] = components[0] (Pdb) list 603 604 # Figure out the SQL for next-based-on-primary-key first 605 next_by_pk_clauses = [] 606 if use_rowid: 607 next_by_pk_clauses.append(f""rowid > :p{len(params)}"") 608 -> params[f""p{len(params)}""] = components[0] 609 else: 610 # Apply the tie-breaker based on primary keys 611 if len(components) == len(pks): 612 param_len = len(params) 613 next_by_pk_clauses.append( ``` Debugger shows that `components` is an empty array, so `components[0]` cannot be resolved: ``` -> params[f""p{len(params)}""] = components[0] (Pdb) params {'search': 'hello'} (Pdb) components [] ``` So the bug is in this code: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/datasette/views/table.py#L604-L617 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",995098231,?_sort=rowid with _next= returns error, https://github.com/simonw/datasette/issues/1479#issuecomment-932808216,https://api.github.com/repos/simonw/datasette/issues/1479,932808216,IC_kwDOBm6k_c43mYYY,9599,simonw,2021-10-02T19:25:09Z,2021-10-02T19:25:09Z,OWNER,"Actually no, from that stack trace you provided: ``` File ""c:\users\grott\anaconda3\lib\site-packages\click\core.py"", line 610, in invoke return callback(*args, **kwargs) File ""c:\users\grott\anaconda3\lib\site-packages\datasette\cli.py"", line 283, in package call(args) File ""c:\users\grott\anaconda3\lib\contextlib.py"", line 119, in __exit__ next(self.gen) File ""c:\users\grott\anaconda3\lib\site-packages\datasette\utils\__init__.py"", line 451, in temporary_docker_directory tmp.cleanup() ``` It looks like the problem occurs here: https://github.com/simonw/datasette/blob/b1fed48a95516ae84c0f020582303ab50ab817e2/datasette/utils/__init__.py#L449-L452","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1010112818,"Win32 ""used by another process"" error with datasette publish", https://github.com/simonw/datasette/issues/1479#issuecomment-932808043,https://api.github.com/repos/simonw/datasette/issues/1479,932808043,IC_kwDOBm6k_c43mYVr,9599,simonw,2021-10-02T19:23:52Z,2021-10-02T19:23:52Z,OWNER,I suspect the root cause of this may be in this code: https://github.com/simonw/datasette/blob/63886178a649586b403966a27a45881709d2b868/datasette/utils/__init__.py#L673-L677,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1010112818,"Win32 ""used by another process"" error with datasette publish", https://github.com/simonw/datasette/issues/1479#issuecomment-932807859,https://api.github.com/repos/simonw/datasette/issues/1479,932807859,IC_kwDOBm6k_c43mYSz,9599,simonw,2021-10-02T19:22:35Z,2021-10-02T19:22:35Z,OWNER,"I'm pretty sure this is a Windows issue, not a Fly issue. I imagine it affects other forms of `datasette publish` too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1010112818,"Win32 ""used by another process"" error with datasette publish", https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926208819,https://api.github.com/repos/simonw/sqlite-utils/issues/329,926208819,IC_kwDOCGYnMM43NNMz,9599,simonw,2021-09-23T22:26:54Z,2021-09-23T22:26:54Z,OWNER,"I could even have those replacement characters be properties of the `Database` class, so uses can sub-class and change them. ```python class Database: left_brace_replace = ""_"" right_brace_replace = ""_"" ... ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1005891028,Rethink approach to [ and ] in column names (currently throws error), https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926207719,https://api.github.com/repos/simonw/sqlite-utils/issues/329,926207719,IC_kwDOCGYnMM43NM7n,9599,simonw,2021-09-23T22:24:09Z,2021-09-23T22:24:09Z,OWNER,"I think I like the underscore option best. I don't like the idea of injecting surprise `( )` parenthesis, and having them vanish entirely could result in things like `item[price]` becoming `itemprice` which feels confusing. `item_price_` is a little ugly but I think I can live with it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1005891028,Rethink approach to [ and ] in column names (currently throws error), https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926207246,https://api.github.com/repos/simonw/sqlite-utils/issues/329,926207246,IC_kwDOCGYnMM43NM0O,9599,simonw,2021-09-23T22:23:09Z,2021-09-23T22:23:09Z,OWNER,"What are my options for replacing those characters? - `[` becomes `(` and `]` becomes `)` - `[` and `]` are removed entirely - `[` and `]` both become `_`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1005891028,Rethink approach to [ and ] in column names (currently throws error), https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926206705,https://api.github.com/repos/simonw/sqlite-utils/issues/329,926206705,IC_kwDOCGYnMM43NMrx,9599,simonw,2021-09-23T22:21:58Z,2021-09-23T22:21:58Z,OWNER,"I'm inclined to just fix them and not have an option for opting-out of fixing them, since it adds quite a bit of cruft to the overall API design for an option that maybe no-one will ever use.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1005891028,Rethink approach to [ and ] in column names (currently throws error), https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926206220,https://api.github.com/repos/simonw/sqlite-utils/issues/329,926206220,IC_kwDOCGYnMM43NMkM,9599,simonw,2021-09-23T22:20:55Z,2021-09-23T22:21:19Z,OWNER,"If I add a new parameter for opting in and out of fixing these, what should it be called? A few options: - `fix_columns=False` - `rename_bad_columns=False` - `fix_column_names=False` - `error_on_invalid_column_names=True`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1005891028,Rethink approach to [ and ] in column names (currently throws error), https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926205047,https://api.github.com/repos/simonw/sqlite-utils/issues/329,926205047,IC_kwDOCGYnMM43NMR3,9599,simonw,2021-09-23T22:18:34Z,2021-09-23T22:19:38Z,OWNER,"Here's the code where this happens: https://github.com/simonw/sqlite-utils/blob/54191d4dc114d7dc21e849b48a4d5ae4f9e601ca/sqlite_utils/db.py#L2943-L2948 It's called from three different methods in `db.py`: `create_table_sql()`, `update()` and `insert_all()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1005891028,Rethink approach to [ and ] in column names (currently throws error), https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926204046,https://api.github.com/repos/simonw/sqlite-utils/issues/329,926204046,IC_kwDOCGYnMM43NMCO,9599,simonw,2021-09-23T22:16:18Z,2021-09-23T22:16:18Z,OWNER,"So either I do the automatic replacement, or I let the user request automatic replacement, or a third option: I do automatic replacement but let the user opt to receive an error instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1005891028,Rethink approach to [ and ] in column names (currently throws error), https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926203501,https://api.github.com/repos/simonw/sqlite-utils/issues/329,926203501,IC_kwDOCGYnMM43NL5t,9599,simonw,2021-09-23T22:15:07Z,2021-09-23T22:15:30Z,OWNER,"Quoting https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586676856 in full: > I'm not sure what to do about this one. > > I can't fix it: this bug in Python's `sqlite3` module means that even if I write a database out with column names that include `[]` I won't be able to read them back again. > > So... I could do one of the following: > > * Throw an error if a column name includes those characters. That's my preferred option I think. > * Automatically replace `[` in column names with `(` and `]` with `)` > * Do the automatic replacement but show a user-visible warning when I do it > * Throw an error, but give the user an option to run with e.g. `--fix-column-names` which applies that automatic fix. > > > Since this is likely to be an incredibly rare edge-case I think I'd rather minimize the amount of code that deals with it, so my preferred option is to just throw that error and stop.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1005891028,Rethink approach to [ and ] in column names (currently throws error), https://github.com/simonw/sqlite-utils/issues/325#issuecomment-925321439,https://api.github.com/repos/simonw/sqlite-utils/issues/325,925321439,IC_kwDOCGYnMM43J0jf,9599,simonw,2021-09-22T20:52:56Z,2021-09-22T20:52:56Z,OWNER,"Updated documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#running-queries-directly-against-csv-or-json > If two files have the same name they will be assigned a numeric suffix: > > $ sqlite-utils memory foo/data.csv bar/data.csv ""select * from data_2""","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",990844088,sqlite-utils memory can't deal with multiple files with the same name, https://github.com/simonw/sqlite-utils/issues/325#issuecomment-925303497,https://api.github.com/repos/simonw/sqlite-utils/issues/325,925303497,IC_kwDOCGYnMM43JwLJ,9599,simonw,2021-09-22T20:25:44Z,2021-09-22T20:25:44Z,OWNER,"Here's the relevant code: https://github.com/simonw/sqlite-utils/blob/7427a9137f60de961b6331d0922a3f03da0d1890/sqlite_utils/cli.py#L1289-L1292 I can fix this by checking to see if `csv_table` is already in use and adding a suffix.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",990844088,sqlite-utils memory can't deal with multiple files with the same name, https://github.com/simonw/sqlite-utils/issues/325#issuecomment-925301981,https://api.github.com/repos/simonw/sqlite-utils/issues/325,925301981,IC_kwDOCGYnMM43Jvzd,9599,simonw,2021-09-22T20:23:25Z,2021-09-22T20:23:25Z,OWNER,"Oddly I can't replicate this on macOS: ``` (sqlite-utils) sqlite-utils % ls foo/*.csv foo/bug.csv (sqlite-utils) sqlite-utils % ls bar/*.csv bar/bug.csv (sqlite-utils) sqlite-utils % sqlite-utils memory foo/bug.csv bar/bug.csv --schema CREATE TABLE ""bug"" ( [col1] TEXT, [col2] TEXT ); CREATE VIEW t1 AS select * from [bug]; CREATE VIEW t AS select * from [bug]; ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",990844088,sqlite-utils memory can't deal with multiple files with the same name, https://github.com/simonw/sqlite-utils/issues/325#issuecomment-925300637,https://api.github.com/repos/simonw/sqlite-utils/issues/325,925300637,IC_kwDOCGYnMM43Jved,9599,simonw,2021-09-22T20:21:26Z,2021-09-22T20:21:26Z,OWNER,"The `t1` and `t2` aliases were meant to handle this case, but the are no good if the tool throws an error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",990844088,sqlite-utils memory can't deal with multiple files with the same name, https://github.com/simonw/sqlite-utils/issues/328#issuecomment-925296085,https://api.github.com/repos/simonw/sqlite-utils/issues/328,925296085,IC_kwDOCGYnMM43JuXV,9599,simonw,2021-09-22T20:14:53Z,2021-09-22T20:14:53Z,OWNER,The bug is in this code: https://github.com/simonw/sqlite-utils/blob/77c240df56068341561e95e4a412cbfa24dc5bc7/sqlite_utils/cli.py#L2205-L2227,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1004613267,Invalid JSON output when no rows, https://github.com/simonw/sqlite-utils/issues/328#issuecomment-925292384,https://api.github.com/repos/simonw/sqlite-utils/issues/328,925292384,IC_kwDOCGYnMM43Jtdg,9599,simonw,2021-09-22T20:09:53Z,2021-09-22T20:09:53Z,OWNER,"Good catch, thanks.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1004613267,Invalid JSON output when no rows, https://github.com/simonw/datasette/issues/111#issuecomment-924437942,https://api.github.com/repos/simonw/datasette/issues/111,924437942,IC_kwDOBm6k_c43Gc22,9599,simonw,2021-09-21T22:32:59Z,2021-09-21T22:47:07Z,OWNER,"Side-note: Django 4.0 [will switch](https://docs.djangoproject.com/en/dev/releases/4.0/#zoneinfo-default-timezone-implementation) from using `pytz` to using the standard library `zoneinfo` module introduced in Python 3.9, which has a backport that works as far back as 3.6: https://github.com/pganssle/zoneinfo (https://pypi.org/project/backports.zoneinfo/) If I need to handle timezones I'll use that, but I think I can get away without it? Django does this: https://github.com/django/django/blob/b0ed619303d2fb723330ca9efa3acf23d49f1d19/setup.cfg#L39-L43 ``` install_requires = asgiref >= 3.3.2 backports.zoneinfo; python_version<""3.9"" sqlparse >= 0.2.2 tzdata; sys_platform == 'win32' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274615452,Add “updated” to metadata, https://github.com/simonw/datasette/issues/111#issuecomment-924443089,https://api.github.com/repos/simonw/datasette/issues/111,924443089,IC_kwDOBm6k_c43GeHR,9599,simonw,2021-09-21T22:45:14Z,2021-09-21T22:45:26Z,OWNER,"The audiences I care about here are: - Producers of this timestamp - generally that will be users who are using `datasette publish` to share their data - Human consumers of this timestamp - end users who look at a Datasette site and want to know how recent the data is - Machine consumers of this timestamp - API integrations that might want to check if a Datasette instance has been updated before downloading new data For producers I think there are going to be two categories. The first is users who run ""publish"" and want the site to reflect when they did so (probably using `--updated=now` when they publish). The second are users who are willing to spend more time thinking about this - for example my various git scraping projects where I want to use a date derived from the git history. For humans... I'd like to be able to calculate a relative time (3 hours ago) in addition to showing the display time, because that helps avoid confusion over timezones. For machine consumers, it might be nice to offer the option of a calculated Unix timestamp-since-1970, since those can be easier to work with in some languages than running a full ISO date parser.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274615452,Add “updated” to metadata, https://github.com/simonw/datasette/issues/111#issuecomment-924438481,https://api.github.com/repos/simonw/datasette/issues/111,924438481,IC_kwDOBm6k_c43Gc_R,9599,simonw,2021-09-21T22:34:03Z,2021-09-21T22:34:21Z,OWNER,"The simplest possible version of this: it's always represented as a UTC ISO datetime, like this: ""updated"": ""2020-10-31T12:00:00+00:00"" Later versions of Datasette could extend this to handle other timezones or support just the date (though that's a backwards incompatible change so probably better to decide on the date thing right now).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274615452,Add “updated” to metadata, https://github.com/simonw/datasette/issues/111#issuecomment-924435971,https://api.github.com/repos/simonw/datasette/issues/111,924435971,IC_kwDOBm6k_c43GcYD,9599,simonw,2021-09-21T22:29:15Z,2021-09-21T22:29:49Z,OWNER,"So this is a metadata key called `updated` which can be applied at the table, database or instance level. It is represented as a `.isoformat()` timestamp. Question: should I support just the date - `yyyy-mm-dd` - in addition to the datetime? I think so. I can easily imagine situations where the exact time of day that a change was made hasn't been recorded, but the overall date is known. But in that case, should the `updated` key sometimes be `yyyy-mm-dd` and sometimes be the full isoformat datetime? Or should there be an `updated_date` key that's used for just the date?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274615452,Add “updated” to metadata, https://github.com/simonw/datasette/issues/111#issuecomment-924432643,https://api.github.com/repos/simonw/datasette/issues/111,924432643,IC_kwDOBm6k_c43GbkD,9599,simonw,2021-09-21T22:23:23Z,2021-09-21T22:23:23Z,OWNER,I'm going to use https://github.com/dateutil/dateutil for this - it's been maintained constantly (by an evolving team of contributors) [since 2003](https://github.com/dateutil/dateutil/commit/68ae2757ae15c84bf947d47a82a314b3b975bc9b) and is a very trustworthy dependency.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274615452,Add “updated” to metadata, https://github.com/simonw/datasette/issues/111#issuecomment-923106887,https://api.github.com/repos/simonw/datasette/issues/111,923106887,IC_kwDOBm6k_c43BX5H,9599,simonw,2021-09-20T16:58:39Z,2021-09-20T16:58:39Z,OWNER,Still a good idea today too! Would be great for https://cdc-vaccination-history.datasette.io/ for example.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274615452,Add “updated” to metadata, https://github.com/simonw/datasette/issues/236#issuecomment-922075480,https://api.github.com/repos/simonw/datasette/issues/236,922075480,IC_kwDOBm6k_c429cFY,9599,simonw,2021-09-17T20:54:13Z,2021-09-17T20:54:13Z,OWNER,"That's so useful @sethvincent! Really interesting reading your code there, especially clever how you're using the `base_url` config. I'd be very interested to see what your demo looks like without using serverless - completely agree that the less additional dependencies there are for this the better. I'm also very interested in figuring out a way to run Datasette in Lambda but with the SQLite database on an EFS volume. Do you have a feel for how hard that would be?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",317001500,datasette publish lambda plugin, https://github.com/simonw/datasette/issues/1466#issuecomment-917840012,https://api.github.com/repos/simonw/datasette/issues/1466,917840012,IC_kwDOBm6k_c42tSCM,9599,simonw,2021-09-13T04:54:59Z,2021-09-13T04:54:59Z,OWNER,Especially relevant now that 0.2.0 is out which is a much higher quality release. https://github.com/simonw/datasette-app/releases/tag/0.2.0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",991467558,Add Datasette Desktop to installation documentation, https://github.com/simonw/datasette/issues/1468#issuecomment-917839801,https://api.github.com/repos/simonw/datasette/issues/1468,917839801,IC_kwDOBm6k_c42tR-5,9599,simonw,2021-09-13T04:54:17Z,2021-09-13T04:54:17Z,OWNER,Here's a already open issue for this: #972,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",994390593,Faceting for custom SQL queries, https://github.com/simonw/datasette/issues/1468#issuecomment-917839507,https://api.github.com/repos/simonw/datasette/issues/1468,917839507,IC_kwDOBm6k_c42tR6T,9599,simonw,2021-09-13T04:53:22Z,2021-09-13T04:53:22Z,OWNER,"At the moment this isn't possible - though there's a workaround which is to define a SQL view for the query, at which point facets will be displayed again. I did a lot of the work required to support this when I refactored how facets worked a while back - but to finally implement this I need to refactor the table view and the arbitrary query view to share much more logic than they do at the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",994390593,Faceting for custom SQL queries, https://github.com/simonw/datasette/issues/1469#issuecomment-917839062,https://api.github.com/repos/simonw/datasette/issues/1469,917839062,IC_kwDOBm6k_c42tRzW,9599,simonw,2021-09-13T04:52:01Z,2021-09-13T04:52:01Z,OWNER,Here's the code at fault: https://github.com/simonw/datasette/blob/b28b6cd2fe97f7e193a235877abeec2c8eb0a821/datasette/static/table.js#L137-L146,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",994450961,"Column cog shows ""facet by this"" when already default faceted", https://github.com/simonw/datasette/issues/1462#issuecomment-914644260,https://api.github.com/repos/simonw/datasette/issues/1462,914644260,IC_kwDOBm6k_c42hF0k,9599,simonw,2021-09-07T21:34:32Z,2021-09-07T21:34:32Z,OWNER,"I think this is a setting. There are two relevant settings at the moment: ``` ""template_debug"": false, ""trace_debug"": false, ``` For consistence then this should be called `something_debug` - but do I want a single setting that exposes the `_internal` database and adds those debug options to the menu, or do I want those as two separate settings? - `internal_debug` to enable access to that `_internal` database - `menu_debug` for those menu options?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",990367646,"Separate out ""debug"" options from ""root"" options", https://github.com/simonw/datasette/issues/1461#issuecomment-914441037,https://api.github.com/repos/simonw/datasette/issues/1461,914441037,IC_kwDOBm6k_c42gUNN,9599,simonw,2021-09-07T16:13:59Z,2021-09-07T16:13:59Z,OWNER,"I don't think I'll adopt it for this project. For example, here: ```diff response = Response.redirect(""/"") - response.set_cookie(""ds_actor"", datasette.sign({ - ""a"": { - ""id"": ""cleopaws"" - } - }, ""actor"")) + response.set_cookie(""ds_actor"", datasette.sign({""a"": {""id"": ""cleopaws""}}, ""actor"")) ``` I chose to use the multi-line version to help emphasize the structure - the single-line replacement loses that. I think I'll continue to make my own editorial choices about how the code examples are laid out.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",989986586,Try blacken-docs, https://github.com/simonw/datasette/issues/1461#issuecomment-914440282,https://api.github.com/repos/simonw/datasette/issues/1461,914440282,IC_kwDOBm6k_c42gUBa,9599,simonw,2021-09-07T16:12:57Z,2021-09-07T16:12:57Z,OWNER,"Here's the diff it produced from that first run: ```diff diff --git a/docs/authentication.rst b/docs/authentication.rst index 0d98cf8..8008023 100644 --- a/docs/authentication.rst +++ b/docs/authentication.rst @@ -381,11 +381,7 @@ Authentication plugins can set signed ``ds_actor`` cookies themselves like so: .. code-block:: python response = Response.redirect(""/"") - response.set_cookie(""ds_actor"", datasette.sign({ - ""a"": { - ""id"": ""cleopaws"" - } - }, ""actor"")) + response.set_cookie(""ds_actor"", datasette.sign({""a"": {""id"": ""cleopaws""}}, ""actor"")) Note that you need to pass ``""actor""`` as the namespace to :ref:`datasette_sign`. @@ -412,12 +408,16 @@ To include an expiry, add a ``""e""`` key to the cookie value containing a `base62 expires_at = int(time.time()) + (24 * 60 * 60) response = Response.redirect(""/"") - response.set_cookie(""ds_actor"", datasette.sign({ - ""a"": { - ""id"": ""cleopaws"" - }, - ""e"": baseconv.base62.encode(expires_at), - }, ""actor"")) + response.set_cookie( + ""ds_actor"", + datasette.sign( + { + ""a"": {""id"": ""cleopaws""}, + ""e"": baseconv.base62.encode(expires_at), + }, + ""actor"", + ), + ) The resulting cookie will encode data that looks something like this: diff --git a/docs/spatialite.rst b/docs/spatialite.rst index d1b300b..556bad8 100644 --- a/docs/spatialite.rst +++ b/docs/spatialite.rst @@ -58,19 +58,22 @@ Here's a recipe for taking a table with existing latitude and longitude columns, .. code-block:: python import sqlite3 - conn = sqlite3.connect('museums.db') + + conn = sqlite3.connect(""museums.db"") # Lead the spatialite extension: conn.enable_load_extension(True) - conn.load_extension('/usr/local/lib/mod_spatialite.dylib') + conn.load_extension(""/usr/local/lib/mod_spatialite.dylib"") # Initialize spatial metadata for this database: - conn.execute('select InitSpatialMetadata(1)') + conn.execute(""select InitSpatialMetadata(1)"") # Add a geometry column called point_geom to our museums table: conn.execute(""SELECT AddGeometryColumn('museums', 'point_geom', 4326, 'POINT', 2);"") # Now update that geometry column with the lat/lon points - conn.execute(''' + conn.execute( + """""" UPDATE museums SET point_geom = GeomFromText('POINT('||""longitude""||' '||""latitude""||')',4326); - ''') + """""" + ) # Now add a spatial index to that column conn.execute('select CreateSpatialIndex(""museums"", ""point_geom"");') # If you don't commit your changes will not be persisted: @@ -186,13 +189,14 @@ Here's Python code to create a SQLite database, enable SpatiaLite, create a plac .. code-block:: python import sqlite3 - conn = sqlite3.connect('places.db') + + conn = sqlite3.connect(""places.db"") # Enable SpatialLite extension conn.enable_load_extension(True) - conn.load_extension('/usr/local/lib/mod_spatialite.dylib') + conn.load_extension(""/usr/local/lib/mod_spatialite.dylib"") # Create the masic countries table - conn.execute('select InitSpatialMetadata(1)') - conn.execute('create table places (id integer primary key, name text);') + conn.execute(""select InitSpatialMetadata(1)"") + conn.execute(""create table places (id integer primary key, name text);"") # Add a MULTIPOLYGON Geometry column conn.execute(""SELECT AddGeometryColumn('places', 'geom', 4326, 'MULTIPOLYGON', 2);"") # Add a spatial index against the new column @@ -201,13 +205,17 @@ Here's Python code to create a SQLite database, enable SpatiaLite, create a plac from shapely.geometry.multipolygon import MultiPolygon from shapely.geometry import shape import requests - geojson = requests.get('https://data.whosonfirst.org/404/227/475/404227475.geojson').json() + + geojson = requests.get( + ""https://data.whosonfirst.org/404/227/475/404227475.geojson"" + ).json() # Convert to ""Well Known Text"" format - wkt = shape(geojson['geometry']).wkt + wkt = shape(geojson[""geometry""]).wkt # Insert and commit the record - conn.execute(""INSERT INTO places (id, name, geom) VALUES(null, ?, GeomFromText(?, 4326))"", ( - ""Wales"", wkt - )) + conn.execute( + ""INSERT INTO places (id, name, geom) VALUES(null, ?, GeomFromText(?, 4326))"", + (""Wales"", wkt), + ) conn.commit() Querying polygons using within() diff --git a/docs/writing_plugins.rst b/docs/writing_plugins.rst index bd60a4b..5af01f6 100644 --- a/docs/writing_plugins.rst +++ b/docs/writing_plugins.rst @@ -18,9 +18,10 @@ The quickest way to start writing a plugin is to create a ``my_plugin.py`` file from datasette import hookimpl + @hookimpl def prepare_connection(conn): - conn.create_function('hello_world', 0, lambda: 'Hello world!') + conn.create_function(""hello_world"", 0, lambda: ""Hello world!"") If you save this in ``plugins/my_plugin.py`` you can then start Datasette like this:: @@ -60,22 +61,18 @@ The example consists of two files: a ``setup.py`` file that defines the plugin: from setuptools import setup - VERSION = '0.1' + VERSION = ""0.1"" setup( - name='datasette-plugin-demos', - description='Examples of plugins for Datasette', - author='Simon Willison', - url='https://github.com/simonw/datasette-plugin-demos', - license='Apache License, Version 2.0', + name=""datasette-plugin-demos"", + description=""Examples of plugins for Datasette"", + author=""Simon Willison"", + url=""https://github.com/simonw/datasette-plugin-demos"", + license=""Apache License, Version 2.0"", version=VERSION, - py_modules=['datasette_plugin_demos'], - entry_points={ - 'datasette': [ - 'plugin_demos = datasette_plugin_demos' - ] - }, - install_requires=['datasette'] + py_modules=[""datasette_plugin_demos""], + entry_points={""datasette"": [""plugin_demos = datasette_plugin_demos""]}, + install_requires=[""datasette""], ) And a Python module file, ``datasette_plugin_demos.py``, that implements the plugin: @@ -88,12 +85,12 @@ And a Python module file, ``datasette_plugin_demos.py``, that implements the plu @hookimpl def prepare_jinja2_environment(env): - env.filters['uppercase'] = lambda u: u.upper() + env.filters[""uppercase""] = lambda u: u.upper() @hookimpl def prepare_connection(conn): - conn.create_function('random_integer', 2, random.randint) + conn.create_function(""random_integer"", 2, random.randint) Having built a plugin in this way you can turn it into an installable package using the following command:: @@ -123,11 +120,13 @@ To bundle the static assets for a plugin in the package that you publish to PyPI .. code-block:: python - package_data={ - 'datasette_plugin_name': [ - 'static/plugin.js', - ], - }, + package_data = ( + { + ""datasette_plugin_name"": [ + ""static/plugin.js"", + ], + }, + ) Where ``datasette_plugin_name`` is the name of the plugin package (note that it uses underscores, not hyphens) and ``static/plugin.js`` is the path within that package to the static file. @@ -152,11 +151,13 @@ Templates should be bundled for distribution using the same ``package_data`` mec .. code-block:: python - package_data={ - 'datasette_plugin_name': [ - 'templates/my_template.html', - ], - }, + package_data = ( + { + ""datasette_plugin_name"": [ + ""templates/my_template.html"", + ], + }, + ) You can also use wildcards here such as ``templates/*.html``. See `datasette-edit-schema `__ for an example of this pattern. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",989986586,Try blacken-docs, https://github.com/simonw/datasette/issues/1461#issuecomment-914439356,https://api.github.com/repos/simonw/datasette/issues/1461,914439356,IC_kwDOBm6k_c42gTy8,9599,simonw,2021-09-07T16:11:37Z,2021-09-07T16:11:37Z,OWNER,"``` (datasette) datasette % blacken-docs docs/*.rst docs/authentication.rst: Rewriting... docs/internals.rst:169: code block parse error Cannot parse: 14:0: docs/plugin_hooks.rst:251: code block parse error Cannot parse: 6:4: ] docs/plugin_hooks.rst:312: code block parse error Cannot parse: 38:0: docs/spatialite.rst: Rewriting... docs/testing_plugins.rst:135: code block parse error Cannot parse: 5:0: docs/writing_plugins.rst: Rewriting... ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",989986586,Try blacken-docs, https://github.com/simonw/datasette/issues/1459#issuecomment-913218494,https://api.github.com/repos/simonw/datasette/issues/1459,913218494,IC_kwDOBm6k_c42bpu-,9599,simonw,2021-09-05T19:58:51Z,2021-09-05T19:59:15Z,OWNER,"This idea makes sense to me. There's actually an existing option that takes a path, called `--get` - it returns the HTML or JSON for that oath directly to the console, eg `datasette my.db --get /mydb/mytable.json` So... one option would be to allow combining that with `-o` to open that URL in the browser: datasette my.db -o --get /mydb So some options here are: - `datasette my.db --open-url /mydb` - `datasette my.db --open-path /mydb` - `datasette my.db --open --get /mydb` I quite like that last combination option, mainly to avoid adding even more command options. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",988556488,suggestion: allow `datasette --open` to take a relative URL, https://github.com/simonw/datasette/pull/1455#issuecomment-913001416,https://api.github.com/repos/simonw/datasette/issues/1455,913001416,IC_kwDOBm6k_c42a0vI,9599,simonw,2021-09-04T16:32:21Z,2021-09-04T16:32:21Z,OWNER,I'll add researchers too.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",988325628,Add scientists to target groups, https://github.com/simonw/datasette/pull/1455#issuecomment-913001298,https://api.github.com/repos/simonw/datasette/issues/1455,913001298,IC_kwDOBm6k_c42a0tS,9599,simonw,2021-09-04T16:31:32Z,2021-09-04T16:31:32Z,OWNER,Great idea!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",988325628,Add scientists to target groups, https://github.com/simonw/datasette/issues/1446#issuecomment-908832938,https://api.github.com/repos/simonw/datasette/issues/1446,908832938,IC_kwDOBm6k_c42K7Cq,9599,simonw,2021-08-31T01:54:59Z,2021-08-31T01:54:59Z,OWNER,I used the sticky footer mechanism in `datasette.app`: https://github.com/simonw/datasette.app/issues/3,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978357984,Modify base.html template to support optional sticky footer, https://github.com/simonw/datasette/issues/1449#issuecomment-907547624,https://api.github.com/repos/simonw/datasette/issues/1449,907547624,IC_kwDOBm6k_c42GBPo,9599,simonw,2021-08-28T01:44:57Z,2021-08-28T01:58:35Z,OWNER,Documentation: https://docs.datasette.io/en/latest/plugin_hooks.html#register-commands-cli,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands, https://github.com/simonw/datasette/issues/1449#issuecomment-907547736,https://api.github.com/repos/simonw/datasette/issues/1449,907547736,IC_kwDOBm6k_c42GBRY,9599,simonw,2021-08-28T01:45:36Z,2021-08-28T01:45:36Z,OWNER,I need to push this out as an alpha so I can release a demo plugin that uses it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands, https://github.com/simonw/datasette/issues/1449#issuecomment-907543982,https://api.github.com/repos/simonw/datasette/issues/1449,907543982,IC_kwDOBm6k_c42GAWu,9599,simonw,2021-08-28T01:14:27Z,2021-08-28T01:14:27Z,OWNER,"Writing the test for this is proving difficult, because the `cli` module has already been imported when I attempt to register a new plugin - so it doesn't pick up on the additional command registrations. Trying to work around that with `importlib.reload(cli)`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands, https://github.com/simonw/datasette/issues/1449#issuecomment-907542214,https://api.github.com/repos/simonw/datasette/issues/1449,907542214,IC_kwDOBm6k_c42F_7G,9599,simonw,2021-08-28T01:02:38Z,2021-08-28T01:02:38Z,OWNER,"Partial prototype of `datasette-verify`: ```python from datasette import hookimpl import click @hookimpl def register_commands(cli): from datasette.cli import sqlite_extensions @cli.command() @click.argument(""files"", type=click.Path(exists=True), nargs=-1) @sqlite_extensions def verify(files, sqlite_extensions): ""Verify that files can be opened by Datasette"" for file in files: print(file) ``` I had to move the `from datasette.cli import sqlite_extensions` inside the hook function to avoid a circular import.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands, https://github.com/simonw/datasette/issues/1449#issuecomment-907540928,https://api.github.com/repos/simonw/datasette/issues/1449,907540928,IC_kwDOBm6k_c42F_nA,9599,simonw,2021-08-28T00:53:37Z,2021-08-28T00:53:37Z,OWNER,I'll probably have to use this mechanism for the tests then: https://til.simonwillison.net/pytest/registering-plugins-in-tests,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands, https://github.com/simonw/datasette/issues/1449#issuecomment-907540790,https://api.github.com/repos/simonw/datasette/issues/1449,907540790,IC_kwDOBm6k_c42F_k2,9599,simonw,2021-08-28T00:52:32Z,2021-08-28T00:52:32Z,OWNER,I don't think I can get this new hook to support the handy [--plugins-dir= mechanism](https://docs.datasette.io/en/stable/plugins.html#one-off-plugins-using-plugins-dir) for loading plugins from Python files as opposed to registering them with setuptools - that mechanism is itself implemented inside of code called by `datasette serve` so I don't have a way of taking advantage of it from outside that command.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands, https://github.com/simonw/datasette/issues/1450#issuecomment-907540240,https://api.github.com/repos/simonw/datasette/issues/1450,907540240,IC_kwDOBm6k_c42F_cQ,9599,simonw,2021-08-28T00:48:30Z,2021-08-28T00:48:30Z,OWNER,"I'll go with this: ``` % datasette --help Usage: datasette [OPTIONS] COMMAND [ARGS]... Datasette is an open source multi-tool for exploring and publishing data About Datasette: https://datasette.io/ Full documentation: https://docs.datasette.io/ Options: --version Show the version and exit. --help Show this message and exit. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981681138,"Datasette --help should show something more useful than ""Datasette!""", https://github.com/simonw/datasette/issues/1449#issuecomment-907539668,https://api.github.com/repos/simonw/datasette/issues/1449,907539668,IC_kwDOBm6k_c42F_TU,9599,simonw,2021-08-28T00:44:16Z,2021-08-28T00:44:16Z,OWNER,"Considering this piece of code: https://github.com/simonw/datasette/blob/a1a33bb5822214be1cebd98cd858b2058d91a4aa/datasette/cli.py#L122-L142 I think the hook itself gets called with a single argument, `cli`, which it can then use in the standard Click way to register extra stuff. I can't pass it `datasette` (like I do with `register_routes()`) because the Datasette object itself is instantiated by the `serve` command, which will not have been called.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands, https://github.com/simonw/datasette/issues/1449#issuecomment-907539251,https://api.github.com/repos/simonw/datasette/issues/1449,907539251,IC_kwDOBm6k_c42F_Mz,9599,simonw,2021-08-28T00:41:37Z,2021-08-28T00:41:50Z,OWNER,The first example plugin I'm going to build for this will be `datasette verify file.db file2.db` - it will take one or more paths to SQLite files and verify if they can be opened by Datasette or not.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands, https://github.com/simonw/datasette/issues/1449#issuecomment-907539065,https://api.github.com/repos/simonw/datasette/issues/1449,907539065,IC_kwDOBm6k_c42F_J5,9599,simonw,2021-08-28T00:40:24Z,2021-08-28T00:40:24Z,OWNER,I'm going to call the new hook `register_commands` - since it will allow ambitious plugins to register more than one command if they want to. That's also pleasingly similar to `register_routes`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands, https://github.com/simonw/datasette/issues/1449#issuecomment-907538940,https://api.github.com/repos/simonw/datasette/issues/1449,907538940,IC_kwDOBm6k_c42F_H8,9599,simonw,2021-08-28T00:39:28Z,2021-08-28T00:39:28Z,OWNER,"Here's the answer to that: ``` ~ % datasette --help Usage: datasette [OPTIONS] COMMAND [ARGS]... Datasette! Options: --version Show the version and exit. --help Show this message and exit. Commands: serve* Serve up specified SQLite database files with a web UI inspect install Install Python packages - e.g. package Package specified SQLite files into a new datasette Docker... plugins List currently available plugins publish Publish specified SQLite database files to the internet... uninstall Uninstall Python packages (e.g. ``` Since it's adding extra things that show up in `--help` under the ""Commands:"" heading, I should call them commands.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands, https://github.com/simonw/datasette/issues/1449#issuecomment-907537693,https://api.github.com/repos/simonw/datasette/issues/1449,907537693,IC_kwDOBm6k_c42F-0d,9599,simonw,2021-08-28T00:31:26Z,2021-08-28T00:31:26Z,OWNER,Terminology question: is it correct to call these subcommands or should they be commands? `publish_subcommand()` adds subcommands of the format `datasette publish X` - but are we instead adding commands with this new one?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands, https://github.com/simonw/datasette/issues/1449#issuecomment-907537610,https://api.github.com/repos/simonw/datasette/issues/1449,907537610,IC_kwDOBm6k_c42F-zK,9599,simonw,2021-08-28T00:30:51Z,2021-08-28T00:30:51Z,OWNER,"There's also the option for plugins to muck around with existing registered commands - this could get a bit untidy if multiple plugins try to do it, but being able to replace `serve` with a fresh implementation that adds an additional command-line option before calling back to the original might open up some interesting possibilities.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands, https://github.com/simonw/datasette/issues/1449#issuecomment-907537366,https://api.github.com/repos/simonw/datasette/issues/1449,907537366,IC_kwDOBm6k_c42F-vW,9599,simonw,2021-08-28T00:29:16Z,2021-08-28T00:29:29Z,OWNER,"The closest plugin hook to this right now is [publish_subcommand](https://docs.datasette.io/en/stable/plugin_hooks.html#publish-subcommand-publish) - which looks like this: ```python @hookimpl def publish_subcommand(publish): @publish.command() @add_common_publish_arguments_and_options @click.option( ""-k"", ""--api_key"", help=""API key for talking to my hosting provider"", ) def my_hosting_provider(...): ``` But there are also several plugin hooks with `register_` prefixes, which may be a good naming convention to stick to here: `register_output_renderer(datasette)`, `register_routes(datasette)`, `register_facet_classes()`, `register_magic_parameters(datasette)`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands, https://github.com/simonw/datasette/issues/859#issuecomment-905900807,https://api.github.com/repos/simonw/datasette/issues/859,905900807,IC_kwDOBm6k_c41_vMH,9599,simonw,2021-08-25T21:51:10Z,2021-08-25T21:51:10Z,OWNER,"10-20 minutes to populate `_internal`! How many databases and tables is that for? I may have to rethink the `_internal` mechanism entirely. One possible alternative would be for the Datasette homepage to just show a list of available databases (maybe only if there are more than X connected) and then load in their metadata only the first time they are accessed. I need to get my own stress testing rig setup for this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts), https://github.com/simonw/sqlite-utils/issues/323#issuecomment-905886797,https://api.github.com/repos/simonw/sqlite-utils/issues/323,905886797,IC_kwDOCGYnMM41_rxN,9599,simonw,2021-08-25T21:25:18Z,2021-08-25T21:25:18Z,OWNER,"As far as I can tell the Python `sqlite3` module doesn't actually have a mechanism for de-registering a custom SQL function. This means that if I implement a mechanism whereby each call to `.convert()` registers a new SQL function with a random suffix (`convert_value_23424()` for example) those functions will stay registered - and if `.convert()` is called a large number of times the number of obsolete custom function registrations will grow without bounds. For that reason, I'm going to `wontfix` this issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",979627285,`table.convert()` method should clean up after itself, https://github.com/simonw/datasette/pull/1447#issuecomment-905097468,https://api.github.com/repos/simonw/datasette/issues/1447,905097468,IC_kwDOBm6k_c418rD8,9599,simonw,2021-08-25T01:28:53Z,2021-08-25T01:28:53Z,OWNER,"Good catch, thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978614898,Remove underscore from search mode parameter name, https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905043974,https://api.github.com/repos/simonw/sqlite-utils/issues/319,905043974,IC_kwDOCGYnMM418eAG,9599,simonw,2021-08-24T23:33:44Z,2021-08-24T23:33:44Z,OWNER,Updated documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#inserting-data-from-files,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",976399638,[Enhancement] Please allow 'insert-files' to insert content as text., https://github.com/simonw/sqlite-utils/pull/321#issuecomment-905040902,https://api.github.com/repos/simonw/sqlite-utils/issues/321,905040902,IC_kwDOCGYnMM418dQG,9599,simonw,2021-08-24T23:25:03Z,2021-08-24T23:25:03Z,OWNER,I'm going to skip this test on windows.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978537855,"Ability to insert file contents as text, in addition to blob", https://github.com/simonw/sqlite-utils/pull/321#issuecomment-905040307,https://api.github.com/repos/simonw/sqlite-utils/issues/321,905040307,IC_kwDOCGYnMM418dGz,9599,simonw,2021-08-24T23:23:36Z,2021-08-24T23:23:36Z,OWNER,"https://discuss.python.org/t/pep-597-use-utf-8-for-default-text-file-encoding/1819 says: > Currently, `TextIOWrapper` uses `locale.getpreferredencoding(False)` (hereinafter called “locale encoding”) when encoding is not specified. > ... > Package authors using macOS or Linux may forget that the default encoding is not always UTF-8.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978537855,"Ability to insert file contents as text, in addition to blob", https://github.com/simonw/sqlite-utils/pull/321#issuecomment-905039576,https://api.github.com/repos/simonw/sqlite-utils/issues/321,905039576,IC_kwDOCGYnMM418c7Y,9599,simonw,2021-08-24T23:21:29Z,2021-08-24T23:21:29Z,OWNER,"Hah, the error here is actually: ``` > assert result.exit_code == 1, result.output E AssertionError: E E assert 0 == 1 E + where 0 = .exit_code ``` So I was expecting an error, but instead the command worked. I suspect this is because on Windows the default character set may not be UTF-8?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978537855,"Ability to insert file contents as text, in addition to blob", https://github.com/simonw/sqlite-utils/pull/321#issuecomment-905037323,https://api.github.com/repos/simonw/sqlite-utils/issues/321,905037323,IC_kwDOCGYnMM418cYL,9599,simonw,2021-08-24T23:15:29Z,2021-08-24T23:15:29Z,OWNER,"Huh, tests are failing but only on Windows!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978537855,"Ability to insert file contents as text, in addition to blob", https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905021933,https://api.github.com/repos/simonw/sqlite-utils/issues/319,905021933,IC_kwDOCGYnMM418Ynt,9599,simonw,2021-08-24T22:36:04Z,2021-08-24T22:36:04Z,OWNER,"> Oh, I misread. Yes some files will not be valid UTF-8, I'd throw a warning and continue (not adding that file) but if you want to get more elaborate you could allow to define a policy on what to do. Not adding the file, index binary content or use a conversion policy like the ones available on Python's decode. I thought about supporting those different policies (with something like `--errors ignore`) but I feel like that's getting a little bit too deep into the weeds. Right now if you try to import an invalid file the behaviour is the same as for the `sqlite-utils insert` command (I added the same detailed error message): ``` Error: Could not read file '/Users/simon/Dropbox/Development/sqlite-utils/data.txt' as text 'utf-8' codec can't decode byte 0xe3 in position 83: invalid continuation byte The input you provided uses a character encoding other than utf-8. You can fix this by passing the --encoding= option with the encoding of the file. If you do not know the encoding, running 'file filename.csv' may tell you. It's often worth trying: --encoding=latin-1 ``` If someone has data that can't be translated to valid text using a known encoding, I'm happy leaving them to have to insert it into a `BLOB` column instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",976399638,[Enhancement] Please allow 'insert-files' to insert content as text., https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905021047,https://api.github.com/repos/simonw/sqlite-utils/issues/319,905021047,IC_kwDOCGYnMM418YZ3,9599,simonw,2021-08-24T22:33:48Z,2021-08-24T22:33:48Z,OWNER,"I had a few doubts about the design just now. Since `content_text` is supported as a special argument, an alternative way of handling the above would be: sqlite-utils insert-files /tmp/text.db files *.txt -c path -c content_text -c size This does exactly the same thing as just using `--text` and not specifying any columns, because the actual implementation of `--text` is as follows: https://github.com/simonw/sqlite-utils/blob/0c796cd945b146b7395ff5f553861400be503867/sqlite_utils/cli.py#L1851-L1855 But actually I think that's OK - ``--text`` is a useful shorthand that avoids you having to remember how to manually specify those columns with `-c`. So I'm going to leave the design as is.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",976399638,[Enhancement] Please allow 'insert-files' to insert content as text., https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905013183,https://api.github.com/repos/simonw/sqlite-utils/issues/319,905013183,IC_kwDOCGYnMM418We_,9599,simonw,2021-08-24T22:15:34Z,2021-08-24T22:15:34Z,OWNER,"Here's the error message I have working for invalid unicode: ``` sqlite-utils insert-files /tmp/text.db files *.txt --text [------------------------------------] 0% Error: Could not read file '/Users/simon/Dropbox/Development/sqlite-utils/data.txt' as text 'utf-8' codec can't decode byte 0xe3 in position 83: invalid continuation byte ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",976399638,[Enhancement] Please allow 'insert-files' to insert content as text., https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905013162,https://api.github.com/repos/simonw/sqlite-utils/issues/319,905013162,IC_kwDOCGYnMM418Weq,9599,simonw,2021-08-24T22:15:31Z,2021-08-24T22:15:31Z,OWNER,"I'm going to assume utf-8 but allow `--encoding` to be used to specify something different, since that option is already supported by other commands.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",976399638,[Enhancement] Please allow 'insert-files' to insert content as text., https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905001586,https://api.github.com/repos/simonw/sqlite-utils/issues/319,905001586,IC_kwDOCGYnMM418Tpy,9599,simonw,2021-08-24T21:52:50Z,2021-08-24T21:52:50Z,OWNER,"Will need to re-title this section of the documentation: https://sqlite-utils.datasette.io/en/3.16/cli.html#inserting-binary-data-from-files - ""Inserting binary data from files"" will become ""Inserting data from files"" I'm OK with keeping the default as `BLOB` but I could add a `--text` option which stores the content as text instead. If the text can't be stored as `utf-8` I'll probably raise an error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",976399638,[Enhancement] Please allow 'insert-files' to insert content as text., https://github.com/simonw/sqlite-utils/issues/319#issuecomment-904999850,https://api.github.com/repos/simonw/sqlite-utils/issues/319,904999850,IC_kwDOCGYnMM418TOq,9599,simonw,2021-08-24T21:49:08Z,2021-08-24T21:49:08Z,OWNER,This is a good idea.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",976399638,[Enhancement] Please allow 'insert-files' to insert content as text., https://github.com/simonw/datasette/issues/1446#issuecomment-904954530,https://api.github.com/repos/simonw/datasette/issues/1446,904954530,IC_kwDOBm6k_c418IKi,9599,simonw,2021-08-24T20:32:47Z,2021-08-24T20:32:47Z,OWNER,"Pasting that CSS into the styles editor in the developer tools on https://latest.datasette.io/ has the desired effect: footer at the bottom of the window unless the page is too long, in which case the footer is at the bottom of the scroll.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978357984,Modify base.html template to support optional sticky footer, https://github.com/simonw/datasette/issues/1446#issuecomment-904866495,https://api.github.com/repos/simonw/datasette/issues/1446,904866495,IC_kwDOBm6k_c417yq_,9599,simonw,2021-08-24T18:13:49Z,2021-08-24T18:13:49Z,OWNER,"OK, now the following optional CSS gives us a sticky footer: ```css html, body { height: 100%; } body { display: flex; flex-direction: column; } .not-footer { flex: 1 0 auto; } footer { flex-shrink: 0; } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978357984,Modify base.html template to support optional sticky footer, https://github.com/simonw/datasette/issues/1445#issuecomment-904037087,https://api.github.com/repos/simonw/datasette/issues/1445,904037087,IC_kwDOBm6k_c414oLf,9599,simonw,2021-08-23T19:10:17Z,2021-08-23T19:10:17Z,OWNER,"Rather than trying to run that monstrosity in a single `union all` query, a better approach may be to use `fetch()` requests as seen in https://datasette.io/plugins/datasette-search-all","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",977323133,Ability to search for text across all columns in a table, https://github.com/simonw/datasette/issues/1445#issuecomment-904036200,https://api.github.com/repos/simonw/datasette/issues/1445,904036200,IC_kwDOBm6k_c414n9o,9599,simonw,2021-08-23T19:08:54Z,2021-08-23T19:08:54Z,OWNER,"Figured out a query for searching across every column in every table! https://til.simonwillison.net/datasette/search-all-columns-trick#user-content-same-trick-for-the-entire-database ```sql with tables as ( select name as table_name from sqlite_master where type = 'table' ), queries as ( select 'select ''' || tables.table_name || ''' as _table, rowid from ""' || tables.table_name || '"" where ' || group_concat( '""' || name || '"" like ''%'' || :search || ''%''', ' or ' ) as query from pragma_table_info(tables.table_name), tables group by tables.table_name ) select group_concat(query, ' union all ') from queries ``` The SQL query this generates for larger databases is _extremely_ long - but it does seem to work for smaller databases.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",977323133,Ability to search for text across all columns in a table, https://github.com/simonw/datasette/issues/1445#issuecomment-904027166,https://api.github.com/repos/simonw/datasette/issues/1445,904027166,IC_kwDOBm6k_c414lwe,9599,simonw,2021-08-23T18:56:20Z,2021-08-23T18:56:20Z,OWNER,A related but potentially even more useful ability would be running a search across every column of every table in a whole database. For anything less than a few 100MB this could be incredibly useful.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",977323133,Ability to search for text across all columns in a table, https://github.com/simonw/datasette/issues/1445#issuecomment-904026253,https://api.github.com/repos/simonw/datasette/issues/1445,904026253,IC_kwDOBm6k_c414liN,9599,simonw,2021-08-23T18:54:49Z,2021-08-23T18:54:49Z,OWNER,"The bigger problem here is UI design. This feels like a pretty niche requirement to me, so adding a prominent search box to the table page (which already has the filters interface, plus the full-text search box for tables that have FTS configured) feels untidy. I could tuck it away in the table cog menu, but that's a weird place for something like this to live. Maybe add it as a new type of filter? Filters apply to specific columns though, so this would be the first filter that applied to _all_ columns - which doesn't really fit the existing filter interface very well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",977323133,Ability to search for text across all columns in a table, https://github.com/simonw/datasette/issues/1445#issuecomment-904024939,https://api.github.com/repos/simonw/datasette/issues/1445,904024939,IC_kwDOBm6k_c414lNr,9599,simonw,2021-08-23T18:52:35Z,2021-08-23T18:52:35Z,OWNER,"The downside of the current implementation of this trick is that it only works for exact LIKE partial matches in a specific table - if you search for `dog cat` and `dog` appears in `title` but `cat` appears in `description` you won't get back that result. I think that's fine though. If you want more advanced search there are other mechanisms you can use. This is meant to be a very quick and dirty starting point for exploring a brand new table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",977323133,Ability to search for text across all columns in a table, https://github.com/simonw/sqlite-utils/issues/320#issuecomment-903288691,https://api.github.com/repos/simonw/sqlite-utils/issues/320,903288691,IC_kwDOCGYnMM411xdz,9599,simonw,2021-08-22T15:46:56Z,2021-08-22T15:46:56Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#schema-analyze-dump-and-save,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",976405225,sqlite-utils memory --analyze option, https://github.com/simonw/sqlite-utils/issues/320#issuecomment-903288430,https://api.github.com/repos/simonw/sqlite-utils/issues/320,903288430,IC_kwDOCGYnMM411xZu,9599,simonw,2021-08-22T15:44:55Z,2021-08-22T15:45:52Z,OWNER,"``` curl 'https://api.github.com/users/dogsheep/repos' | sqlite-utils memory - --analyze ``` ``` stdin.id: (1/73) Total rows: 13 Null rows: 0 Blank rows: 0 Distinct values: 13 stdin.node_id: (2/73) Total rows: 13 Null rows: 0 Blank rows: 0 Distinct values: 13 stdin.name: (3/73) Total rows: 13 Null rows: 0 Blank rows: 0 Distinct values: 13 stdin.full_name: (4/73) Total rows: 13 Null rows: 0 Blank rows: 0 Distinct values: 13 stdin.private: (5/73) Total rows: 13 Null rows: 0 Blank rows: 0 Distinct values: 1 Most common: 13: 0 stdin.owner: (6/73) Total rows: 13 Null rows: 0 Blank rows: 0 Distinct values: 1 Most common: 13: {""login"": ""dogsheep"", ""id"": 53015001, ""node_id"": ""MDEyOk9yZ2FuaXphdGlvbjUzMDE1MD... stdin.html_url: (7/73) Total rows: 13 Null rows: 0 Blank rows: 0 Distinct values: 13 stdin.description: (8/73) Total rows: 13 Null rows: 0 Blank rows: 0 Distinct values: 13 stdin.fork: (9/73) Total rows: 13 Null rows: 0 Blank rows: 0 Distinct values: 1 Most common: 13: 0 stdin.url: (10/73) Total rows: 13 Null rows: 0 Blank rows: 0 Distinct values: 13 stdin.forks_url: (11/73) Total rows: 13 Null rows: 0 Blank rows: 0 Distinct values: 13 stdin.keys_url: (12/73) Total rows: 13 ... ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",976405225,sqlite-utils memory --analyze option, https://github.com/simonw/datasette/issues/894#issuecomment-902375388,https://api.github.com/repos/simonw/datasette/issues/894,902375388,IC_kwDOBm6k_c41ySfc,9599,simonw,2021-08-20T02:07:53Z,2021-08-20T02:07:53Z,OWNER,I could add these sorting links to the cog menu for any `TEXT` columns.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by by column cast to real, https://github.com/simonw/datasette/issues/894#issuecomment-902375088,https://api.github.com/repos/simonw/datasette/issues/894,902375088,IC_kwDOBm6k_c41ySaw,9599,simonw,2021-08-20T02:07:13Z,2021-08-20T02:07:26Z,OWNER,Maybe `?_sort_numeric=col` and `?_sort_numeric_desc=col` would be better here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by by column cast to real, https://github.com/simonw/datasette/issues/1426#issuecomment-902263367,https://api.github.com/repos/simonw/datasette/issues/1426,902263367,IC_kwDOBm6k_c41x3JH,9599,simonw,2021-08-19T21:33:51Z,2021-08-19T21:36:28Z,OWNER,"I was worried about if it's possible to allow access to `/fixtures` but deny access to `/fixtures?sql=...` From various answers on Stack Overflow it looks like this should handle that: ``` User-agent: * Disallow: /fixtures? ``` I could use this for tables too - it may well be OK to access table index pages while still avoiding pagination, facets etc. I think this should block both query strings and row pages while allowing the table page itself: ``` User-agent: * Disallow: /fixtures/searchable? Disallow: /fixtures/searchable/* ``` Could even accompany that with a `sitemap.xml` that explicitly lists all of the tables - which would mean adding sitemaps to Datasette core too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",964322136,"Manage /robots.txt in Datasette core, block robots by default", https://github.com/simonw/datasette/issues/1426#issuecomment-902260338,https://api.github.com/repos/simonw/datasette/issues/1426,902260338,IC_kwDOBm6k_c41x2Zy,9599,simonw,2021-08-19T21:28:25Z,2021-08-19T21:29:40Z,OWNER,"Actually it looks like you can send a `sitemap.xml` to Google using an unauthenticated GET request to: https://www.google.com/ping?sitemap=FULL_URL_OF_SITEMAP According to https://developers.google.com/search/docs/advanced/sitemaps/build-sitemap","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",964322136,"Manage /robots.txt in Datasette core, block robots by default", https://github.com/simonw/datasette/issues/1426#issuecomment-902260799,https://api.github.com/repos/simonw/datasette/issues/1426,902260799,IC_kwDOBm6k_c41x2g_,9599,simonw,2021-08-19T21:29:13Z,2021-08-19T21:29:13Z,OWNER,"Bing's equivalent is: https://www.bing.com/webmasters/help/Sitemaps-3b5cf6ed http://www.bing.com/ping?sitemap=FULL_URL_OF_SITEMAP","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",964322136,"Manage /robots.txt in Datasette core, block robots by default", https://github.com/simonw/datasette/issues/1443#issuecomment-902258509,https://api.github.com/repos/simonw/datasette/issues/1443,902258509,IC_kwDOBm6k_c41x19N,9599,simonw,2021-08-19T21:25:07Z,2021-08-19T21:25:07Z,OWNER,https://docs.datasette.io/en/latest/internals.html#databases,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",974995592,datasette.databases should be a documented property, https://github.com/simonw/datasette/pull/1434#issuecomment-902254712,https://api.github.com/repos/simonw/datasette/issues/1434,902254712,IC_kwDOBm6k_c41x1B4,9599,simonw,2021-08-19T21:18:31Z,2021-08-19T21:18:57Z,OWNER,"I deployed a demo to https://datasette-latest-query-info-j7hipcg4aq-uc.a.run.app using the mechanism from #1442. e.g. demo here: https://datasette-latest-query-info-j7hipcg4aq-uc.a.run.app/fixtures?sql=select+*+from+searchable","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",970463436,Enrich arbitrary query results with foreign key links and column descriptions, https://github.com/simonw/datasette/issues/1415#issuecomment-902251316,https://api.github.com/repos/simonw/datasette/issues/1415,902251316,IC_kwDOBm6k_c41x0M0,9599,simonw,2021-08-19T21:14:15Z,2021-08-19T21:14:15Z,OWNER,"https://github.com/ahmetb/cloud-run-faq#how-do-i-continuously-deploy-to-cloud-run suggests the following: > - `roles/run.admin` to deploy applications > - `roles/iam.serviceAccountUser` on the service account that your app will use It also links to https://cloud.google.com/run/docs/reference/iam/roles","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959137143,feature request: document minimum permissions for service account for cloudrun, https://github.com/simonw/datasette/issues/1415#issuecomment-902250361,https://api.github.com/repos/simonw/datasette/issues/1415,902250361,IC_kwDOBm6k_c41xz95,9599,simonw,2021-08-19T21:12:28Z,2021-08-19T21:12:28Z,OWNER,I would love to know this too! I always find figuring out minimal permissions to be really difficult.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959137143,feature request: document minimum permissions for service account for cloudrun, https://github.com/simonw/datasette/issues/1442#issuecomment-902243498,https://api.github.com/repos/simonw/datasette/issues/1442,902243498,IC_kwDOBm6k_c41xySq,9599,simonw,2021-08-19T21:04:01Z,2021-08-19T21:04:01Z,OWNER,That successfully deployed to https://datasette-latest-deploy-this-branch-j7hipcg4aq-uc.a.run.app/ even though the tests failed.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",974987856,Mechanism to cause specific branches to deploy their own demos, https://github.com/simonw/datasette/issues/1442#issuecomment-902239215,https://api.github.com/repos/simonw/datasette/issues/1442,902239215,IC_kwDOBm6k_c41xxPv,9599,simonw,2021-08-19T20:56:46Z,2021-08-19T20:56:46Z,OWNER,"I'm going to only run the tests if it's a push to `main` - that way I can ship demo branches really quickly, even if they don't yet have passing tests.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",974987856,Mechanism to cause specific branches to deploy their own demos, https://github.com/simonw/datasette/issues/1442#issuecomment-902235714,https://api.github.com/repos/simonw/datasette/issues/1442,902235714,IC_kwDOBm6k_c41xwZC,9599,simonw,2021-08-19T20:50:38Z,2021-08-19T20:50:38Z,OWNER,"Would this allow anyone to push a PR to this repo that would result in their code being deployed against my Cloud Run account? I'm reasonably confident that it would not, since the secrets would not be visible to their PR branch.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",974987856,Mechanism to cause specific branches to deploy their own demos, https://github.com/simonw/datasette/issues/1442#issuecomment-902231018,https://api.github.com/repos/simonw/datasette/issues/1442,902231018,IC_kwDOBm6k_c41xvPq,9599,simonw,2021-08-19T20:42:08Z,2021-08-19T20:42:08Z,OWNER,If I get this working I should document it on https://docs.datasette.io/en/stable/contributing.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",974987856,Mechanism to cause specific branches to deploy their own demos, https://github.com/simonw/datasette/issues/1442#issuecomment-902217726,https://api.github.com/repos/simonw/datasette/issues/1442,902217726,IC_kwDOBm6k_c41xr_-,9599,simonw,2021-08-19T20:21:47Z,2021-08-19T20:21:47Z,OWNER,I think the neatest way to implement this would be for the `on -> push -> branches` list to be the list of branches that should be deployed in this way. The rest of the code can react to that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",974987856,Mechanism to cause specific branches to deploy their own demos, https://github.com/simonw/datasette/issues/1442#issuecomment-902191150,https://api.github.com/repos/simonw/datasette/issues/1442,902191150,IC_kwDOBm6k_c41xlgu,9599,simonw,2021-08-19T19:43:05Z,2021-08-19T19:43:59Z,OWNER,"Maybe as simple as teaching https://github.com/simonw/datasette/blob/main/.github/workflows/deploy-latest.yml to run on pushes to ALL branches: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/.github/workflows/deploy-latest.yml#L3-L6 And then quit early if the branch is not in some allow-list. If it IS in the allow-list, use the name of the branch to dynamically construct the name of the Cloud Run service here: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/.github/workflows/deploy-latest.yml#L60 Need to skip the documentation build and deployment stuff for other branches though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",974987856,Mechanism to cause specific branches to deploy their own demos, https://github.com/simonw/datasette/issues/1293#issuecomment-901475812,https://api.github.com/repos/simonw/datasette/issues/1293,901475812,IC_kwDOBm6k_c41u23k,9599,simonw,2021-08-18T22:41:19Z,2021-08-18T22:41:19Z,OWNER,"> Maybe I split this out into a separate Python library that gets tested against _every_ SQLite release I can possibly try it against, and then bakes out the supported release versions into the library code itself? I'm going to do this, and call the Python library `sqlite-explain`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/sqlite-utils/issues/37#issuecomment-901452199,https://api.github.com/repos/simonw/sqlite-utils/issues/37,901452199,IC_kwDOCGYnMM41uxGn,9599,simonw,2021-08-18T21:48:57Z,2021-08-18T21:48:57Z,OWNER,"I did a bunch of work on this in #266. The library is now pretty thoroughly typed, and I even found a couple of bugs using `mypy` along the way: #313 and #315.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",465815372,Experiment with type hints, https://github.com/simonw/sqlite-utils/issues/318#issuecomment-901440752,https://api.github.com/repos/simonw/sqlite-utils/issues/318,901440752,IC_kwDOCGYnMM41uuTw,9599,simonw,2021-08-18T21:25:30Z,2021-08-18T21:25:30Z,OWNER,"Some questions: - Should this support compression formats other than gzip? - Should `memory` learn to auto-detect gzipped data?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",974067156,Research: handle gzipped CSV directly, https://github.com/simonw/sqlite-utils/issues/318#issuecomment-901440207,https://api.github.com/repos/simonw/sqlite-utils/issues/318,901440207,IC_kwDOCGYnMM41uuLP,9599,simonw,2021-08-18T21:24:28Z,2021-08-18T21:24:49Z,OWNER,"Something like this then: sqlite-utils file.db ""select * from t"" --csv --gz > t.csv.gz Maybe add a `-o t.csv.gz` option too so you don't have to use a `>`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",974067156,Research: handle gzipped CSV directly, https://github.com/simonw/sqlite-utils/issues/295#issuecomment-901403298,https://api.github.com/repos/simonw/sqlite-utils/issues/295,901403298,IC_kwDOCGYnMM41ulKi,9599,simonw,2021-08-18T20:19:04Z,2021-08-18T20:19:04Z,OWNER,"Thanks, this was a bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",934123448,Insert with --tsv and --no-headers give error about --nl arguments, https://github.com/simonw/sqlite-utils/issues/296#issuecomment-901399139,https://api.github.com/repos/simonw/sqlite-utils/issues/296,901399139,IC_kwDOCGYnMM41ukJj,9599,simonw,2021-08-18T20:12:34Z,2021-08-18T20:13:12Z,OWNER,"Documentation for `table.search(..., quote=True)`: https://sqlite-utils.datasette.io/en/latest/python-api.html#searching-with-table-search In the API reference: https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Table.search And for the CLI `--quote` option: https://sqlite-utils.datasette.io/en/latest/cli.html#executing-searches","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944326512,"`table.search(..., quote=True)` parameter and `sqlite-utils search --quote` option", https://github.com/simonw/sqlite-utils/issues/296#issuecomment-901398216,https://api.github.com/repos/simonw/sqlite-utils/issues/296,901398216,IC_kwDOCGYnMM41uj7I,9599,simonw,2021-08-18T20:11:01Z,2021-08-18T20:11:01Z,OWNER,"``` % sqlite-utils search fixtures.db searchable 'dog""' Error: malformed MATCH expression: [dog""] Try running this again with the --quote option ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944326512,"`table.search(..., quote=True)` parameter and `sqlite-utils search --quote` option", https://github.com/simonw/sqlite-utils/issues/296#issuecomment-901390635,https://api.github.com/repos/simonw/sqlite-utils/issues/296,901390635,IC_kwDOCGYnMM41uiEr,9599,simonw,2021-08-18T19:58:53Z,2021-08-18T19:58:53Z,OWNER,"``` sqlite-utils search fixtures.db searchable 'dog""' Error: malformed MATCH expression: [dog""] ``` This error message could suggest retrying with `--quote`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944326512,"`table.search(..., quote=True)` parameter and `sqlite-utils search --quote` option", https://github.com/simonw/sqlite-utils/issues/296#issuecomment-901379930,https://api.github.com/repos/simonw/sqlite-utils/issues/296,901379930,IC_kwDOCGYnMM41ufda,9599,simonw,2021-08-18T19:40:38Z,2021-08-18T19:40:38Z,OWNER,Also add `sqlite-utils search ... --quote` option.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944326512,"`table.search(..., quote=True)` parameter and `sqlite-utils search --quote` option", https://github.com/simonw/sqlite-utils/issues/246#issuecomment-901353345,https://api.github.com/repos/simonw/sqlite-utils/issues/246,901353345,IC_kwDOCGYnMM41uY-B,9599,simonw,2021-08-18T18:57:13Z,2021-08-18T18:57:13Z,OWNER,More documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#quoting-characters-for-use-in-search,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",831751367,Escaping FTS search strings, https://github.com/simonw/sqlite-utils/issues/296#issuecomment-901338841,https://api.github.com/repos/simonw/sqlite-utils/issues/296,901338841,IC_kwDOCGYnMM41uVbZ,9599,simonw,2021-08-18T18:33:26Z,2021-08-18T18:45:12Z,OWNER,"I think I'll do this as an optional `table.search(..., escape=True)` parameter. Actually I'll do `quote=True` for consistency with the new `db.quote_fts()` method.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944326512,"`table.search(..., quote=True)` parameter and `sqlite-utils search --quote` option", https://github.com/simonw/sqlite-utils/issues/246#issuecomment-901345800,https://api.github.com/repos/simonw/sqlite-utils/issues/246,901345800,IC_kwDOCGYnMM41uXII,9599,simonw,2021-08-18T18:44:48Z,2021-08-18T18:44:48Z,OWNER,"The `db.quote_fts(value)` method from #247 can now be used for this - documentation here: https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Database.quote_fts I'll be adding further improvements relating to this (a `table.search(q, quote=True)` parameter) in #296.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",831751367,Escaping FTS search strings, https://github.com/simonw/sqlite-utils/pull/247#issuecomment-901338988,https://api.github.com/repos/simonw/sqlite-utils/issues/247,901338988,IC_kwDOCGYnMM41uVds,9599,simonw,2021-08-18T18:33:39Z,2021-08-18T18:33:39Z,OWNER,This was also requested in #296.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",832687563,FTS quote functionality from datasette, https://github.com/simonw/sqlite-utils/issues/296#issuecomment-901338356,https://api.github.com/repos/simonw/sqlite-utils/issues/296,901338356,IC_kwDOCGYnMM41uVT0,9599,simonw,2021-08-18T18:32:39Z,2021-08-18T18:32:39Z,OWNER,This is a good call. I have a fix for this in Datasette but it's not in `sqlite-utils` yet: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/datasette/utils/__init__.py#L824-L835,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944326512,"`table.search(..., quote=True)` parameter and `sqlite-utils search --quote` option", https://github.com/simonw/sqlite-utils/issues/317#issuecomment-901337305,https://api.github.com/repos/simonw/sqlite-utils/issues/317,901337305,IC_kwDOCGYnMM41uVDZ,9599,simonw,2021-08-18T18:30:59Z,2021-08-18T18:30:59Z,OWNER,"I'm just going to remove this - I added it when the library was mostly undocumented, but it has comprehensive documentation now.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",972827346,Link to a better example on docs index, https://github.com/simonw/datasette/issues/1439#issuecomment-900715375,https://api.github.com/repos/simonw/datasette/issues/1439,900715375,IC_kwDOBm6k_c41r9Nv,9599,simonw,2021-08-18T00:15:28Z,2021-08-18T00:15:28Z,OWNER,"Maybe I should use `-/` to encode forward slashes too, to defend against any ASGI servers that might not implement `raw_path` correctly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-900714630,https://api.github.com/repos/simonw/datasette/issues/1439,900714630,IC_kwDOBm6k_c41r9CG,9599,simonw,2021-08-18T00:13:33Z,2021-08-18T00:13:33Z,OWNER,"The documentation should definitely cover how table names become URLs, in case any third party code needs to be able to calculate this themselves.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-900712981,https://api.github.com/repos/simonw/datasette/issues/1439,900712981,IC_kwDOBm6k_c41r8oV,9599,simonw,2021-08-18T00:09:59Z,2021-08-18T00:12:32Z,OWNER,"So given the original examples, a table called `table.csv` would have the following URLs: - `/db/table-.csv` - the HTML version - `/db/table-.csv.csv` - the CSV version - `/db/table-.csv.json` - the JSON version And if for some horific reason you had a table with the name `/db/table-.csv.csv` (so `/db/` was the first part of the actual table name in SQLite) the URLs would look like this: - `/db/%2Fdb%2Ftable---.csv-.csv` - the HTML version - `/db/%2Fdb%2Ftable---.csv-.csv.csv` - the CSV version - `/db/%2Fdb%2Ftable---.csv-.csv.json` - the JSON version","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-900711967,https://api.github.com/repos/simonw/datasette/issues/1439,900711967,IC_kwDOBm6k_c41r8Yf,9599,simonw,2021-08-18T00:08:09Z,2021-08-18T00:08:09Z,OWNER,"Here's an alternative I just made up which I'm calling ""dot dash"" encoding: ```python def dot_dash_encode(s): return s.replace(""-"", ""--"").replace(""."", ""-."") def dot_dash_decode(s): return s.replace(""-."", ""."").replace(""--"", ""-"") ``` And some examples: ```python for example in ( ""hello"", ""hello.csv"", ""hello-and-so-on.csv"", ""hello-.csv"", ""hello--and--so--on-.csv"", ""hello.csv."", ""hello.csv.-"", ""hello.csv.--"", ): print(example) print(dot_dash_encode(example)) print(example == dot_dash_decode(dot_dash_encode(example))) print() ``` Outputs: ``` hello hello True hello.csv hello-.csv True hello-and-so-on.csv hello--and--so--on-.csv True hello-.csv hello---.csv True hello--and--so--on-.csv hello----and----so----on---.csv True hello.csv. hello-.csv-. True hello.csv.- hello-.csv-.-- True hello.csv.-- hello-.csv-.---- True ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-900709703,https://api.github.com/repos/simonw/datasette/issues/1439,900709703,IC_kwDOBm6k_c41r71H,9599,simonw,2021-08-18T00:03:09Z,2021-08-18T00:03:09Z,OWNER,"But... what if I invent my own escaping scheme? I actually did this once before, in https://github.com/simonw/datasette/commit/9fdb47ca952b93b7b60adddb965ea6642b1ff523 - while I was working on porting Datasette to ASGI in https://github.com/simonw/datasette/issues/272#issuecomment-494192779 because ASGI didn't yet have the `raw_path` mechanism. I could bring that back - it looked like this: ``` ""table/and/slashes"" => ""tableU+002FandU+002Fslashes"" ""~table"" => ""U+007Etable"" ""+bobcats!"" => ""U+002Bbobcats!"" ""U+007Etable"" => ""UU+002B007Etable"" ``` But I didn't particularly like it - it was quite verbose.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-900705226,https://api.github.com/repos/simonw/datasette/issues/1439,900705226,IC_kwDOBm6k_c41r6vK,9599,simonw,2021-08-17T23:50:32Z,2021-08-17T23:50:47Z,OWNER,"An alternative solution would be to use some form of escaping for the characters that form the name of the table. The obvious way to do this would be URL-encoding - but it doesn't hold for `.` characters. The hex for that is `%2E` but watch what happens with that in a URL: ``` # Against Cloud Run: curl -s 'https://datasette.io/-/asgi-scope/foo/bar%2Fbaz%2E' | rg path 'path': '/-/asgi-scope/foo/bar/baz.', 'raw_path': b'/-/asgi-scope/foo/bar%2Fbaz.', 'root_path': '', # Against Vercel: curl -s 'https://til.simonwillison.net/-/asgi-scope/foo/bar%2Fbaz%2E' | rg path 'path': '/-/asgi-scope/foo/bar%2Fbaz%2E', 'raw_path': b'/-/asgi-scope/foo/bar%2Fbaz%2E', 'root_path': '', ``` Surprisingly in this case Vercel DOES keep it intact, but Cloud Run does not. It's still no good though: I need a solution that works on Vercel, Cloud Run and every other potential hosting provider too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-900699670,https://api.github.com/repos/simonw/datasette/issues/1439,900699670,IC_kwDOBm6k_c41r5YW,9599,simonw,2021-08-17T23:34:23Z,2021-08-17T23:34:23Z,OWNER,"The challenge comes down to telling the difference between the following: - `/db/table` - an HTML table page - `/db/table.csv` - the CSV version of `/db/table` - `/db/table.csv` - no this one is actually a database table called `table.csv` - `/db/table.csv.csv` - the CSV version of `/db/table.csv` - `/db/table.csv.csv.csv` and so on...","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1438#issuecomment-900690998,https://api.github.com/repos/simonw/datasette/issues/1438,900690998,IC_kwDOBm6k_c41r3Q2,9599,simonw,2021-08-17T23:11:16Z,2021-08-17T23:12:25Z,OWNER,"I have completely failed to replicate this initial bug - but it's still there on the `thesession.vercel.app` deployment (even though my own deployments to Vercel do not exhibit it). Here's a one-liner to replicate it against that deployment: `curl -s 'https://thesession.vercel.app/thesession?sql=select+*+from+tunes+where+name+like+%22%25wise+maid%25%22' | rg '.csv'` Whit outputs this: `

This data as json, CSV

` It looks like, rather than being URL-encoded, the original query string is somehow making it through to Jinja and then being auto-escaped there. The weird thing is that the equivalent query executed against my `til.simonwillison.net` Vercel instance does this: `curl -s 'https://til.simonwillison.net/fixtures?sql=select+*+from+searchable+where+text1+like+%22%25a%25%22' | rg '.csv'` `

This data as json, CSV

`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",972918533,Query page .csv and .json links are not correctly URL-encoded on Vercel under unknown specific conditions, https://github.com/simonw/datasette/issues/1438#issuecomment-900681413,https://api.github.com/repos/simonw/datasette/issues/1438,900681413,IC_kwDOBm6k_c41r07F,9599,simonw,2021-08-17T22:47:44Z,2021-08-17T22:47:44Z,OWNER,I deployed another copy of `fixtures.db` on Vercel at https://til.simonwillison.net/fixtures so I can compare it with `fixtures.db` on Cloud Run at https://latest.datasette.io/fixtures,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",972918533,Query page .csv and .json links are not correctly URL-encoded on Vercel under unknown specific conditions, https://github.com/simonw/datasette/issues/1438#issuecomment-900518343,https://api.github.com/repos/simonw/datasette/issues/1438,900518343,IC_kwDOBm6k_c41rNHH,9599,simonw,2021-08-17T18:04:42Z,2021-08-17T18:04:42Z,OWNER,Here's how `request.query_string` works: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/datasette/utils/asgi.py#L86-L88,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",972918533,Query page .csv and .json links are not correctly URL-encoded on Vercel under unknown specific conditions, https://github.com/simonw/datasette/issues/1438#issuecomment-900516826,https://api.github.com/repos/simonw/datasette/issues/1438,900516826,IC_kwDOBm6k_c41rMva,9599,simonw,2021-08-17T18:02:27Z,2021-08-17T18:02:27Z,OWNER,"The key difference I can spot between Vercel and Cloud Run is that `+` in a query string gets converted to `%20` by Vercel before it gets to my app, but does not for Cloud Run: ``` # Vercel ~ % curl -s 'https://til.simonwillison.net/-/asgi-scope?sql=select+*+from+tunes+where+name+like+%22%25wise+maid%25%22%0D%0A' | rg 'query_string' -C 2 'method': 'GET', 'path': '/-/asgi-scope', 'query_string': b'sql=select%20*%20from%20tunes%20where%20name%20like%20%22%25' b'wise%20maid%25%22%0D%0A', 'raw_path': b'/-/asgi-scope', # Cloud Run ~ % curl -s 'https://latest-with-plugins.datasette.io/-/asgi-scope?sql=select+*+from+tunes+where+name+like+%22%25wise+maid%25%22%0D%0A' | rg 'query_string' -C 2 'method': 'GET', 'path': '/-/asgi-scope', 'query_string': b'sql=select+*+from+tunes+where+name+like+%22%25wise+maid%25%2' b'2%0D%0A', 'raw_path': b'/-/asgi-scope', ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",972918533,Query page .csv and .json links are not correctly URL-encoded on Vercel under unknown specific conditions, https://github.com/simonw/datasette/issues/1438#issuecomment-900513267,https://api.github.com/repos/simonw/datasette/issues/1438,900513267,IC_kwDOBm6k_c41rL3z,9599,simonw,2021-08-17T17:57:05Z,2021-08-17T17:57:05Z,OWNER,"I'm having trouble replicating this bug outside of Vercel. Against Cloud Run: view-source:https://latest.datasette.io/fixtures?sql=select+*+from+searchable+where+text1+like+%22%25cat%25%22 The HTML here is: ```html

This data as json, ... CSV

```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",972918533,Query page .csv and .json links are not correctly URL-encoded on Vercel under unknown specific conditions, https://github.com/simonw/datasette/issues/1438#issuecomment-900502364,https://api.github.com/repos/simonw/datasette/issues/1438,900502364,IC_kwDOBm6k_c41rJNc,9599,simonw,2021-08-17T17:40:41Z,2021-08-17T17:40:41Z,OWNER,Bug is likely in `path_with_format` itself: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/datasette/utils/__init__.py#L710-L729,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",972918533,Query page .csv and .json links are not correctly URL-encoded on Vercel under unknown specific conditions, https://github.com/simonw/datasette/issues/1438#issuecomment-900500824,https://api.github.com/repos/simonw/datasette/issues/1438,900500824,IC_kwDOBm6k_c41rI1Y,9599,simonw,2021-08-17T17:38:16Z,2021-08-17T17:38:16Z,OWNER,"Relevant template code: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/datasette/templates/query.html#L71 `renderers` comes from here: https://github.com/simonw/datasette/blob/2883098770fc66e50183b2b231edbde20848d4d6/datasette/views/base.py#L593-L608","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",972918533,Query page .csv and .json links are not correctly URL-encoded on Vercel under unknown specific conditions, https://github.com/simonw/datasette/issues/1293#issuecomment-899915829,https://api.github.com/repos/simonw/datasette/issues/1293,899915829,IC_kwDOBm6k_c41o6A1,9599,simonw,2021-08-17T01:02:35Z,2021-08-17T01:02:35Z,OWNER,"New approach: this time I'm building a simplified executor for the bytecode operations themselves. ```python def execute_operations(operations, max_iterations = 100, trace=None): trace = trace or (lambda *args: None) registers: Dict[int, Any] = {} cursors: Dict[int, Tuple[str, Dict]] = {} instruction_pointer = 0 iterations = 0 result_row = None while True: iterations += 1 if iterations > max_iterations: break operation = operations[instruction_pointer] trace(instruction_pointer, dict(operation)) opcode = operation[""opcode""] if opcode == ""Init"": if operation[""p2""] != 0: instruction_pointer = operation[""p2""] continue else: instruction_pointer += 1 continue elif opcode == ""Goto"": instruction_pointer = operation[""p2""] continue elif opcode == ""Halt"": break elif opcode == ""OpenRead"": cursors[operation[""p1""]] = (""database_table"", { ""rootpage"": operation[""p2""], ""connection"": operation[""p3""], }) elif opcode == ""OpenEphemeral"": cursors[operation[""p1""]] = (""ephemeral"", { ""num_columns"": operation[""p2""], ""index_keys"": [], }) elif opcode == ""MakeRecord"": registers[operation[""p3""]] = (""MakeRecord"", { ""registers"": list(range(operation[""p1""] + operation[""p2""])) }) elif opcode == ""IdxInsert"": record = registers[operation[""p2""]] cursors[operation[""p1""]][1][""index_keys""].append(record) elif opcode == ""Rowid"": registers[operation[""p2""]] = (""rowid"", { ""table"": operation[""p1""] }) elif opcode == ""Sequence"": registers[operation[""p2""]] = (""sequence"", { ""next_from_cursor"": operation[""p1""] }) elif opcode == ""Column"": registers[operation[""p3""]] = (""column"", { ""cursor"": operation[""p1""], ""column_offset"": operation[""p2""] }) elif opcode == ""ResultRow"": p1 = operation[""p1""] p2 = operation[""p2""] trace(""ResultRow: "", list(range(p1, p1 + p2)), registers) result_row = [registers.get(i) for i in range(p1, p1 + p2)] elif opcode == ""Integer"": registers[operation[""p2""]] = (""Integer"", operation[""p1""]) elif opcode == ""String8"": registers[operation[""p2""]] = (""String"", operation[""p4""]) instruction_pointer += 1 return {""registers"": registers, ""cursors"": cursors, ""result_row"": result_row} ``` Results are promising! ``` execute_operations(db.execute(""explain select 'hello', 55, rowid, * from searchable"").fetchall()) {'registers': {1: ('String', 'hello'), 2: ('Integer', 55), 3: ('rowid', {'table': 0}), 4: ('rowid', {'table': 0}), 5: ('column', {'cursor': 0, 'column_offset': 1}), 6: ('column', {'cursor': 0, 'column_offset': 2}), 7: ('column', {'cursor': 0, 'column_offset': 3})}, 'cursors': {0: ('database_table', {'rootpage': 32, 'connection': 0})}, 'result_row': [('String', 'hello'), ('Integer', 55), ('rowid', {'table': 0}), ('rowid', {'table': 0}), ('column', {'cursor': 0, 'column_offset': 1}), ('column', {'cursor': 0, 'column_offset': 2}), ('column', {'cursor': 0, 'column_offset': 3})]} ``` Here's what happens with a union across three tables: ``` execute_operations(db.execute(f"""""" explain select data as content from binary_data union select pk as content from complex_foreign_keys union select name as content from facet_cities """"""}).fetchall()) {'registers': {1: ('column', {'cursor': 4, 'column_offset': 0}), 2: ('MakeRecord', {'registers': [0, 1, 2, 3]}), 3: ('column', {'cursor': 0, 'column_offset': 1}), 4: ('column', {'cursor': 3, 'column_offset': 0})}, 'cursors': {3: ('ephemeral', {'num_columns': 1, 'index_keys': [('MakeRecord', {'registers': [0, 1]}), ('MakeRecord', {'registers': [0, 1]}), ('MakeRecord', {'registers': [0, 1, 2, 3]})]}), 2: ('database_table', {'rootpage': 44, 'connection': 0}), 4: ('database_table', {'rootpage': 24, 'connection': 0}), 0: ('database_table', {'rootpage': 42, 'connection': 0})}, 'result_row': [('column', {'cursor': 3, 'column_offset': 0})]} ``` Note how the result_row refers to cursor 3, which is an ephemeral table which had three different sets of `MakeRecord` index keys assigned to it - indicating that the output column is NOT from the same underlying table source.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1423#issuecomment-899749881,https://api.github.com/repos/simonw/datasette/issues/1423,899749881,IC_kwDOBm6k_c41oRf5,9599,simonw,2021-08-16T19:07:02Z,2021-08-16T19:07:02Z,OWNER,"Demo: https://latest.datasette.io/fixtures/compound_three_primary_keys?_facet=content&_facet_size=max&_facet=pk1&_facet=pk2 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",962391325,Show count of facet values if ?_facet_size=max, https://github.com/simonw/datasette/issues/1423#issuecomment-899744109,https://api.github.com/repos/simonw/datasette/issues/1423,899744109,IC_kwDOBm6k_c41oQFt,9599,simonw,2021-08-16T18:58:29Z,2021-08-16T18:58:29Z,OWNER,"I didn't bother with the tooltip, just the visible display if `?_facet_size=max`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",962391325,Show count of facet values if ?_facet_size=max, https://github.com/simonw/datasette/issues/1293#issuecomment-898961535,https://api.github.com/repos/simonw/datasette/issues/1293,898961535,IC_kwDOBm6k_c41lRB_,9599,simonw,2021-08-14T21:37:24Z,2021-08-14T21:37:24Z,OWNER,Did some more research into building SQLite custom versions via `pysqlite3` - here's what I figured out for macOS (which should hopefully work for Linux too): https://til.simonwillison.net/sqlite/build-specific-sqlite-pysqlite-macos,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898936068,https://api.github.com/repos/simonw/datasette/issues/1293,898936068,IC_kwDOBm6k_c41lK0E,9599,simonw,2021-08-14T17:44:54Z,2021-08-14T17:44:54Z,OWNER,"Another interesting query to consider: https://latest.datasette.io/fixtures?sql=explain+select+*+from++pragma_table_info%28+%27123_starts_with_digits%27%29 That one shows `VColumn` instead of `Column`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898933865,https://api.github.com/repos/simonw/datasette/issues/1293,898933865,IC_kwDOBm6k_c41lKRp,9599,simonw,2021-08-14T17:27:16Z,2021-08-14T17:28:29Z,OWNER,"Maybe I split this out into a separate Python library that gets tested against *every* SQLite release I can possibly try it against, and then bakes out the supported release versions into the library code itself? Datasette could depend on that library. The library could be released independently of Datasette any time a new SQLite version comes out. I could even run a separate git scraper repo that checks for new SQLite releases and submits PRs against the library when a new release comes out.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898913629,https://api.github.com/repos/simonw/datasette/issues/1293,898913629,IC_kwDOBm6k_c41lFVd,9599,simonw,2021-08-14T16:14:12Z,2021-08-14T16:14:12Z,OWNER,I would feel a lot more comfortable about all of this if I had a robust mechanism for running the Datasette test suite against multiple versions of SQLite itself.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898913554,https://api.github.com/repos/simonw/datasette/issues/1293,898913554,IC_kwDOBm6k_c41lFUS,9599,simonw,2021-08-14T16:13:40Z,2021-08-14T16:13:40Z,OWNER,"I think I need to care about the following: - `ResultRow` and `Column` for the final result - `OpenRead` for opening tables - `OpenEphemeral` then `MakeRecord` and `IdxInsert` for writing records into ephemeral tables `Column` may reference either a table (from `OpenRead`) or an ephemeral table (from `OpenEphemeral`). That *might* be enough.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/sqlite-utils/issues/316#issuecomment-898824020,https://api.github.com/repos/simonw/sqlite-utils/issues/316,898824020,IC_kwDOCGYnMM41kvdU,9599,simonw,2021-08-14T05:12:23Z,2021-08-14T05:12:23Z,OWNER,No visible backticks on https://sqlite-utils.datasette.io/en/latest/reference.html any more.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",970320615,Fix visible backticks on reference page, https://github.com/simonw/datasette/issues/1293#issuecomment-898788262,https://api.github.com/repos/simonw/datasette/issues/1293,898788262,IC_kwDOBm6k_c41kmum,9599,simonw,2021-08-14T01:22:26Z,2021-08-14T01:51:08Z,OWNER,"Tried a more complicated query: ```sql explain select pk, text1, text2, [name with . and spaces] from searchable where rowid in (select rowid from searchable_fts where searchable_fts match escape_fts(:search)) order by text1 desc limit 101 ``` Here's the explain: ``` sqlite> explain select pk, text1, text2, [name with . and spaces] from searchable where rowid in (select rowid from searchable_fts where searchable_fts match escape_fts(:search)) order by text1 desc limit 101 ...> ; addr opcode p1 p2 p3 p4 p5 comment ---- ------------- ---- ---- ---- ------------- -- ------------- 0 Init 0 41 0 00 Start at 41 1 OpenEphemeral 2 6 0 k(1,-B) 00 nColumn=6 2 Integer 101 1 0 00 r[1]=101; LIMIT counter 3 OpenRead 0 32 0 4 00 root=32 iDb=0; searchable 4 Integer 16 3 0 00 r[3]=16; return address 5 Once 0 16 0 00 6 OpenEphemeral 3 1 0 k(1,) 00 nColumn=1; Result of SELECT 1 7 VOpen 1 0 0 vtab:7FCBCA72BE80 00 8 Function0 1 7 6 unknown(-1) 01 r[6]=func(r[7]) 9 Integer 5 4 0 00 r[4]=5 10 Integer 1 5 0 00 r[5]=1 11 VFilter 1 16 4 00 iplan=r[4] zplan='' 12 Rowid 1 8 0 00 r[8]=rowid 13 MakeRecord 8 1 9 C 00 r[9]=mkrec(r[8]) 14 IdxInsert 3 9 8 1 00 key=r[9] 15 VNext 1 12 0 00 16 Return 3 0 0 00 17 Rewind 3 33 0 00 18 Column 3 0 2 00 r[2]= 19 IsNull 2 32 0 00 if r[2]==NULL goto 32 20 SeekRowid 0 32 2 00 intkey=r[2] 21 Column 0 1 10 00 r[10]=searchable.text1 22 Sequence 2 11 0 00 r[11]=cursor[2].ctr++ 23 IfNotZero 1 27 0 00 if r[1]!=0 then r[1]--, goto 27 24 Last 2 0 0 00 25 IdxLE 2 32 10 1 00 key=r[10] 26 Delete 2 0 0 00 27 Rowid 0 12 0 00 r[12]=rowid 28 Column 0 2 13 00 r[13]=searchable.text2 29 Column 0 3 14 00 r[14]=searchable.name with . and spaces 30 MakeRecord 10 5 16 00 r[16]=mkrec(r[10..14]) 31 IdxInsert 2 16 10 5 00 key=r[16] 32 Next 3 18 0 00 33 Sort 2 40 0 00 34 Column 2 4 15 00 r[15]=[name with . and spaces] 35 Column 2 3 14 00 r[14]=text2 36 Column 2 0 13 00 r[13]=text1 37 Column 2 2 12 00 r[12]=pk 38 ResultRow 12 4 0 00 output=r[12..15] 39 Next 2 34 0 00 40 Halt 0 0 0 00 41 Transaction 0 0 35 0 01 usesStmtJournal=0 42 Variable 1 7 0 :search 00 r[7]=parameter(1,:search) 43 Goto 0 1 0 00 ``` Here the `ResultRow` is for registers `12..15` - but those all refer to `Column` records in `2` - where `2` is the first `OpenEphemeral` declared right at the start. I'm having enormous trouble figuring out how that ephemeral table gets populated by the other operations in a way that would let me derive which columns end up in the `ResultRow`. Frustratingly SQLite seems to be able to figure that out just fine, see the column of comments on the right hand side - but I only get those in the `sqlite3` CLI shell, they're not available to me with SQLite when called as a library from Python. Maybe the key to that is this section: ``` 27 Rowid 0 12 0 00 r[12]=rowid 28 Column 0 2 13 00 r[13]=searchable.text2 29 Column 0 3 14 00 r[14]=searchable.name with . and spaces 30 MakeRecord 10 5 16 00 r[16]=mkrec(r[10..14]) 31 IdxInsert 2 16 10 5 00 key=r[16] ``` MakeRecord: > Convert P2 registers beginning with P1 into the record format use as a data record in a database table or as a key in an index. The Column opcode can decode the record later. > > P4 may be a string that is P2 characters long. The N-th character of the string indicates the column affinity that should be used for the N-th field of the index key. > > The mapping from character to affinity is given by the SQLITE_AFF_ macros defined in sqliteInt.h. > > If P4 is NULL then all index fields have the affinity BLOB. > > The meaning of P5 depends on whether or not the SQLITE_ENABLE_NULL_TRIM compile-time option is enabled: > > * If SQLITE_ENABLE_NULL_TRIM is enabled, then the P5 is the index of the right-most table that can be null-trimmed. > > * If SQLITE_ENABLE_NULL_TRIM is omitted, then P5 has the value OPFLAG_NOCHNG_MAGIC if the MakeRecord opcode is allowed to accept no-change records with serial_type 10. This value is only used inside an assert() and does not affect the end result. IdxInsert: > Register P2 holds an SQL index key made using the MakeRecord instructions. This opcode writes that key into the index P1. Data for the entry is nil. > > If P4 is not zero, then it is the number of values in the unpacked key of reg(P2). In that case, P3 is the index of the first register for the unpacked key. The availability of the unpacked key can sometimes be an optimization. > > If P5 has the OPFLAG_APPEND bit set, that is a hint to the b-tree layer that this insert is likely to be an append. > > If P5 has the OPFLAG_NCHANGE bit set, then the change counter is incremented by this instruction. If the OPFLAG_NCHANGE bit is clear, then the change counter is unchanged. > > If the OPFLAG_USESEEKRESULT flag of P5 is set, the implementation might run faster by avoiding an unnecessary seek on cursor P1. However, the OPFLAG_USESEEKRESULT flag must only be set if there have been no prior seeks on the cursor or if the most recent seek used a key equivalent to P2. > > This instruction only works for indices. The equivalent instruction for tables is Insert. IdxLE: > The P4 register values beginning with P3 form an unpacked index key that omits the PRIMARY KEY or ROWID. Compare this key value against the index that P1 is currently pointing to, ignoring the PRIMARY KEY or ROWID on the P1 index. > > If the P1 index entry is less than or equal to the key value then jump to P2. Otherwise fall through to the next instruction.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898760808,https://api.github.com/repos/simonw/datasette/issues/1293,898760808,IC_kwDOBm6k_c41kgBo,9599,simonw,2021-08-13T23:03:01Z,2021-08-13T23:03:01Z,OWNER,Another idea: strip out any `order by` clause to try and keep this simpler. I doubt that's going to cope with complex nested queries though.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898760020,https://api.github.com/repos/simonw/datasette/issues/1293,898760020,IC_kwDOBm6k_c41kf1U,9599,simonw,2021-08-13T23:00:28Z,2021-08-13T23:01:27Z,OWNER,"New theory: this is all about `SorterOpen` and `SorterInsert`. Consider the following with extra annotations at the end of the lines after the `--`: ``` addr opcode p1 p2 p3 p4 p5 comment ---- ------------- ---- ---- ---- ------------- -- ------------- 0 Init 0 25 0 00 Start at 25 1 SorterOpen 2 5 0 k(1,B) 00 -- New SORTER in r2 with 5 slots 2 OpenRead 0 43 0 7 00 root=43 iDb=0; facetable 3 OpenRead 1 42 0 2 00 root=42 iDb=0; facet_cities 4 Rewind 0 16 0 00 5 Column 0 6 3 00 r[3]=facetable.neighborhood 6 Function0 1 2 1 like(2) 02 r[1]=func(r[2..3]) 7 IfNot 1 15 1 00 8 Column 0 5 4 00 r[4]=facetable.city_id 9 SeekRowid 1 15 4 00 intkey=r[4] 10 Column 1 1 6 00 r[6]=facet_cities.name 11 Column 0 4 7 00 r[7]=facetable.state 12 Column 0 6 5 00 r[5]=facetable.neighborhood 13 MakeRecord 5 3 9 00 r[9]=mkrec(r[5..7]) 14 SorterInsert 2 9 5 3 00 key=r[9]-- WRITES record from r9 (line above) into sorter in r2 15 Next 0 5 0 01 16 OpenPseudo 3 10 5 00 5 columns in r[10] 17 SorterSort 2 24 0 00 -- runs the sort, not relevant to my goal 18 SorterData 2 10 3 00 r[10]=data -- ""Write into register P2 (r10) the current sorter data for sorter cursor P1 (sorter 2)"" 19 Column 3 2 8 00 r[8]=state 20 Column 3 1 7 00 r[7]=facet_cities.name 21 Column 3 0 6 00 r[6]=neighborhood 22 ResultRow 6 3 0 00 output=r[6..8] 23 SorterNext 2 18 0 00 24 Halt 0 0 0 00 25 Transaction 0 0 35 0 01 usesStmtJournal=0 26 String8 0 2 0 %bob% 00 r[2]='%bob%' 27 Goto 0 1 0 00 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898576097,https://api.github.com/repos/simonw/datasette/issues/1293,898576097,IC_kwDOBm6k_c41jy7h,9599,simonw,2021-08-13T16:19:57Z,2021-08-13T16:19:57Z,OWNER,"I think I need to look out for `OpenPseudo` and, when that occurs, take a look at the most recent `SorterInsert` and use that to find the `MakeRecord` and then use the `MakeRecord` to figure out the columns that went into it. After all of that I'll be able to resolve that ""table 3"" reference.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898572065,https://api.github.com/repos/simonw/datasette/issues/1293,898572065,IC_kwDOBm6k_c41jx8h,9599,simonw,2021-08-13T16:13:16Z,2021-08-13T16:13:16Z,OWNER,"Aha! That `MakeRecord` line says `r[5..7]` - and r5 = neighborhood, r6 = facet_cities.name, r7 = facetable.state So if the `MakeRecord` defines what goes into that pseudo-table column 2 of that pseudo-table would be `state` - which is what we want. This is really convoluted. I'm no longer confident I can get this to work in a sensible way, especially since I've not started exploring what complex nested tables with CTEs and sub-selects do yet.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898569319,https://api.github.com/repos/simonw/datasette/issues/1293,898569319,IC_kwDOBm6k_c41jxRn,9599,simonw,2021-08-13T16:09:01Z,2021-08-13T16:10:48Z,OWNER,"Need to figure out what column 2 of that pseudo-table is. I think the answer is here: ``` 4 Rewind 0 16 0 00 5 Column 0 6 3 00 r[3]=facetable.neighborhood 6 Function0 1 2 1 like(2) 02 r[1]=func(r[2..3]) 7 IfNot 1 15 1 00 8 Column 0 5 4 00 r[4]=facetable.city_id 9 SeekRowid 1 15 4 00 intkey=r[4] 10 Column 1 1 6 00 r[6]=facet_cities.name 11 Column 0 4 7 00 r[7]=facetable.state 12 Column 0 6 5 00 r[5]=facetable.neighborhood 13 MakeRecord 5 3 9 00 r[9]=mkrec(r[5..7]) 14 SorterInsert 2 9 5 3 00 key=r[9] 15 Next 0 5 0 01 16 OpenPseudo 3 10 5 00 5 columns in r[10] ``` I think the `OpenPseduo` line puts five columns in `r[10]` - and those five columns are the five from the previous block - maybe the five leading up to the `MakeRecord` call on line 13. In which case column 2 would be `facet_cities.name` - assuming we start counting from 0. But the debug code said ""r[8]=state"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898567974,https://api.github.com/repos/simonw/datasette/issues/1293,898567974,IC_kwDOBm6k_c41jw8m,9599,simonw,2021-08-13T16:07:00Z,2021-08-13T16:07:00Z,OWNER,"So this line: ``` 19 Column 3 2 8 00 r[8]=state ``` Means ""Take column 2 of table 3 (the pseudo-table) and store it in register 8""","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898564705,https://api.github.com/repos/simonw/datasette/issues/1293,898564705,IC_kwDOBm6k_c41jwJh,9599,simonw,2021-08-13T16:02:12Z,2021-08-13T16:04:06Z,OWNER,"More debug output: ``` table_rootpage_by_register={0: 43, 1: 42} names_and_types_by_rootpage={42: ('facet_cities', 'table'), 43: ('facetable', 'table')} table_id=0 cid=6 column_register=3 table_id=0 cid=5 column_register=4 table_id=1 cid=1 column_register=6 table_id=0 cid=4 column_register=7 table_id=0 cid=6 column_register=5 table_id=3 cid=2 column_register=8 table_id=3 cid=2 column_register=8 KeyError 3 table = names_and_types_by_rootpage[table_rootpage_by_register[table_id]][0] names_and_types_by_rootpage={42: ('facet_cities', 'table'), 43: ('facetable', 'table')} table_rootpage_by_register={0: 43, 1: 42} table_id=3 columns_by_column_register[column_register] = (table, cid) column_register=8 = (table='facetable', cid=2) table_id=3 cid=1 column_register=7 KeyError 3 table = names_and_types_by_rootpage[table_rootpage_by_register[table_id]][0] names_and_types_by_rootpage={42: ('facet_cities', 'table'), 43: ('facetable', 'table')} table_rootpage_by_register={0: 43, 1: 42} table_id=3 columns_by_column_register[column_register] = (table, cid) column_register=7 = (table='facetable', cid=1) table_id=3 cid=0 column_register=6 KeyError 3 table = names_and_types_by_rootpage[table_rootpage_by_register[table_id]][0] names_and_types_by_rootpage={42: ('facet_cities', 'table'), 43: ('facetable', 'table')} table_rootpage_by_register={0: 43, 1: 42} table_id=3 columns_by_column_register[column_register] = (table, cid) column_register=6 = (table='facetable', cid=0) result_registers=[6, 7, 8] columns_by_column_register={3: ('facetable', 6), 4: ('facetable', 5), 6: ('facet_cities', 1), 7: ('facetable', 4), 5: ('facetable', 6)} all_column_names={('facet_cities', 0): 'id', ('facet_cities', 1): 'name', ('facetable', 0): 'pk', ('facetable', 1): 'created', ('facetable', 2): 'planet_int', ('facetable', 3): 'on_earth', ('facetable', 4): 'state', ('facetable', 5): 'city_id', ('facetable', 6): 'neighborhood', ('facetable', 7): 'tags', ('facetable', 8): 'complex_array', ('facetable', 9): 'distinct_some_null'} ``` Those `KeyError` are happening here because of a lookup in `table_rootpage_by_register` for `table_id=3` - but `table_rootpage_by_register` only has keys 0 and 1. It looks like that `3` actually corresponds to the `OpenPseudo` table from here: ``` 16 OpenPseudo 3 10 5 00 5 columns in r[10] 17 SorterSort 2 24 0 00 18 SorterData 2 10 3 00 r[10]=data 19 Column 3 2 8 00 r[8]=state 20 Column 3 1 7 00 r[7]=facet_cities.name 21 Column 3 0 6 00 r[6]=neighborhood 22 ResultRow 6 3 0 00 output=r[6..8] ``` Python code: ```python def columns_for_query(conn, sql, params=None): """""" Given a SQLite connection ``conn`` and a SQL query ``sql``, returns a list of ``(table_name, column_name)`` pairs corresponding to the columns that would be returned by that SQL query. Each pair indicates the source table and column for the returned column, or ``(None, None)`` if no table and column could be derived (e.g. for ""select 1"") """""" if sql.lower().strip().startswith(""explain""): return [] opcodes = conn.execute(""explain "" + sql, params).fetchall() table_rootpage_by_register = { r[""p1""]: r[""p2""] for r in opcodes if r[""opcode""] == ""OpenRead"" } print(f""{table_rootpage_by_register=}"") names_and_types_by_rootpage = dict( [(r[0], (r[1], r[2])) for r in conn.execute( ""select rootpage, name, type from sqlite_master where rootpage in ({})"".format( "", "".join(map(str, table_rootpage_by_register.values())) ) )] ) print(f""{names_and_types_by_rootpage=}"") columns_by_column_register = {} for opcode_row in opcodes: if opcode_row[""opcode""] in (""Rowid"", ""Column""): addr, opcode, table_id, cid, column_register, p4, p5, comment = opcode_row print(f""{table_id=} {cid=} {column_register=}"") try: table = names_and_types_by_rootpage[table_rootpage_by_register[table_id]][0] columns_by_column_register[column_register] = (table, cid) except KeyError as e: print("" KeyError"") print("" "", e) print("" table = names_and_types_by_rootpage[table_rootpage_by_register[table_id]][0]"") print(f"" {names_and_types_by_rootpage=} {table_rootpage_by_register=} {table_id=}"") print("" columns_by_column_register[column_register] = (table, cid)"") print(f"" {column_register=} = ({table=}, {cid=})"") pass result_row = [dict(r) for r in opcodes if r[""opcode""] == ""ResultRow""][0] result_registers = list(range(result_row[""p1""], result_row[""p1""] + result_row[""p2""])) print(f""{result_registers=}"") print(f""{columns_by_column_register=}"") all_column_names = {} for (table, _) in names_and_types_by_rootpage.values(): table_xinfo = conn.execute(""pragma table_xinfo({})"".format(table)).fetchall() for column_info in table_xinfo: all_column_names[(table, column_info[""cid""])] = column_info[""name""] print(f""{all_column_names=}"") final_output = [] for register in result_registers: try: table, cid = columns_by_column_register[register] final_output.append((table, all_column_names[table, cid])) except KeyError: final_output.append((None, None)) return final_output ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898554859,https://api.github.com/repos/simonw/datasette/issues/1293,898554859,IC_kwDOBm6k_c41jtvr,9599,simonw,2021-08-13T15:46:18Z,2021-08-13T15:46:18Z,OWNER,So it looks like the bug is in the code that populates `columns_by_column_register`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898554427,https://api.github.com/repos/simonw/datasette/issues/1293,898554427,IC_kwDOBm6k_c41jto7,9599,simonw,2021-08-13T15:45:32Z,2021-08-13T15:45:32Z,OWNER,"Some useful debug output: ``` table_rootpage_by_register={0: 43, 1: 42} names_and_types_by_rootpage={42: ('facet_cities', 'table'), 43: ('facetable', 'table')} result_registers=[6, 7, 8] columns_by_column_register={3: ('facetable', 6), 4: ('facetable', 5), 6: ('facet_cities', 1), 7: ('facetable', 4), 5: ('facetable', 6)} all_column_names={('facet_cities', 0): 'id', ('facet_cities', 1): 'name', ('facetable', 0): 'pk', ('facetable', 1): 'created', ('facetable', 2): 'planet_int', ('facetable', 3): 'on_earth', ('facetable', 4): 'state', ('facetable', 5): 'city_id', ('facetable', 6): 'neighborhood', ('facetable', 7): 'tags', ('facetable', 8): 'complex_array', ('facetable', 9): 'distinct_some_null'} ``` The `result_registers` should each correspond to the correct entry in `columns_by_column_register` but they do not. Python code: ```python def columns_for_query(conn, sql, params=None): """""" Given a SQLite connection ``conn`` and a SQL query ``sql``, returns a list of ``(table_name, column_name)`` pairs corresponding to the columns that would be returned by that SQL query. Each pair indicates the source table and column for the returned column, or ``(None, None)`` if no table and column could be derived (e.g. for ""select 1"") """""" if sql.lower().strip().startswith(""explain""): return [] opcodes = conn.execute(""explain "" + sql, params).fetchall() table_rootpage_by_register = { r[""p1""]: r[""p2""] for r in opcodes if r[""opcode""] == ""OpenRead"" } print(f""{table_rootpage_by_register=}"") names_and_types_by_rootpage = dict( [(r[0], (r[1], r[2])) for r in conn.execute( ""select rootpage, name, type from sqlite_master where rootpage in ({})"".format( "", "".join(map(str, table_rootpage_by_register.values())) ) )] ) print(f""{names_and_types_by_rootpage=}"") columns_by_column_register = {} for opcode in opcodes: if opcode[""opcode""] in (""Rowid"", ""Column""): addr, opcode, table_id, cid, column_register, p4, p5, comment = opcode try: table = names_and_types_by_rootpage[table_rootpage_by_register[table_id]][0] columns_by_column_register[column_register] = (table, cid) except KeyError: pass result_row = [dict(r) for r in opcodes if r[""opcode""] == ""ResultRow""][0] result_registers = list(range(result_row[""p1""], result_row[""p1""] + result_row[""p2""])) print(f""{result_registers=}"") print(f""{columns_by_column_register=}"") all_column_names = {} for (table, _) in names_and_types_by_rootpage.values(): table_xinfo = conn.execute(""pragma table_xinfo({})"".format(table)).fetchall() for column_info in table_xinfo: all_column_names[(table, column_info[""cid""])] = column_info[""name""] print(f""{all_column_names=}"") final_output = [] for register in result_registers: try: table, cid = columns_by_column_register[register] final_output.append((table, all_column_names[table, cid])) except KeyError: final_output.append((None, None)) return final_output ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898545815,https://api.github.com/repos/simonw/datasette/issues/1293,898545815,IC_kwDOBm6k_c41jriX,9599,simonw,2021-08-13T15:31:53Z,2021-08-13T15:31:53Z,OWNER,"My hunch here is that registers or columns are being reused in a way that makes my code break - my code is pretty dumb, there are places in it where maybe the first mention of a register wins instead of the last one?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898541972,https://api.github.com/repos/simonw/datasette/issues/1293,898541972,IC_kwDOBm6k_c41jqmU,9599,simonw,2021-08-13T15:26:06Z,2021-08-13T15:29:06Z,OWNER,"ResultRow: > The registers P1 through P1+P2-1 contain a single row of results. This opcode causes the sqlite3_step() call to terminate with an SQLITE_ROW return code and it sets up the sqlite3_stmt structure to provide access to the r(P1)..r(P1+P2-1) values as the result row. Column: > Interpret the data that cursor P1 points to as a structure built using the MakeRecord instruction. (See the MakeRecord opcode for additional information about the format of the data.) Extract the P2-th column from this record. If there are less that (P2+1) values in the record, extract a NULL. > > The value extracted is stored in register P3.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898541543,https://api.github.com/repos/simonw/datasette/issues/1293,898541543,IC_kwDOBm6k_c41jqfn,9599,simonw,2021-08-13T15:25:26Z,2021-08-13T15:25:26Z,OWNER,"But the debug output here seems to be saying what we want it to say: ``` 17 SorterSort 2 24 0 00 18 SorterData 2 10 3 00 r[10]=data 19 Column 3 2 8 00 r[8]=state 20 Column 3 1 7 00 r[7]=facet_cities.name 21 Column 3 0 6 00 r[6]=neighborhood 22 ResultRow 6 3 0 00 output=r[6..8] ``` We want to get back `neighborhood`, `facet_cities.name`, `state`. Why then are we seeing `[('facet_cities', 'name'), ('facetable', 'state'), (None, None)]`?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898540260,https://api.github.com/repos/simonw/datasette/issues/1293,898540260,IC_kwDOBm6k_c41jqLk,9599,simonw,2021-08-13T15:23:28Z,2021-08-13T15:23:28Z,OWNER,"SorterInsert: > Register P2 holds an SQL index key made using the MakeRecord instructions. This opcode writes that key into the sorter P1. Data for the entry is nil. SorterData: > Write into register P2 the current sorter data for sorter cursor P1. Then clear the column header cache on cursor P3. > > This opcode is normally use to move a record out of the sorter and into a register that is the source for a pseudo-table cursor created using OpenPseudo. That pseudo-table cursor is the one that is identified by parameter P3. Clearing the P3 column cache as part of this opcode saves us from having to issue a separate NullRow instruction to clear that cache. OpenPseudo: > Open a new cursor that points to a fake table that contains a single row of data. The content of that one row is the content of memory register P2. In other words, cursor P1 becomes an alias for the MEM_Blob content contained in register P2. > > A pseudo-table created by this opcode is used to hold a single row output from the sorter so that the row can be decomposed into individual columns using the Column opcode. The Column opcode is the only cursor opcode that works with a pseudo-table. > > P3 is the number of fields in the records that will be stored by the pseudo-table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898536181,https://api.github.com/repos/simonw/datasette/issues/1293,898536181,IC_kwDOBm6k_c41jpL1,9599,simonw,2021-08-13T15:17:20Z,2021-08-13T15:20:33Z,OWNER,"Documentation for `MakeRecord`: https://www.sqlite.org/opcode.html#MakeRecord Running `explain` inside `sqlite3` provides extra comments and indentation which make it easier to understand: ``` sqlite> explain select neighborhood, facet_cities.name, state ...> from facetable ...> join facet_cities ...> on facetable.city_id = facet_cities.id ...> where neighborhood like '%bob%'; addr opcode p1 p2 p3 p4 p5 comment ---- ------------- ---- ---- ---- ------------- -- ------------- 0 Init 0 15 0 00 Start at 15 1 OpenRead 0 43 0 7 00 root=43 iDb=0; facetable 2 OpenRead 1 42 0 2 00 root=42 iDb=0; facet_cities 3 Rewind 0 14 0 00 4 Column 0 6 3 00 r[3]=facetable.neighborhood 5 Function0 1 2 1 like(2) 02 r[1]=func(r[2..3]) 6 IfNot 1 13 1 00 7 Column 0 5 4 00 r[4]=facetable.city_id 8 SeekRowid 1 13 4 00 intkey=r[4] 9 Column 0 6 5 00 r[5]=facetable.neighborhood 10 Column 1 1 6 00 r[6]=facet_cities.name 11 Column 0 4 7 00 r[7]=facetable.state 12 ResultRow 5 3 0 00 output=r[5..7] 13 Next 0 4 0 01 14 Halt 0 0 0 00 15 Transaction 0 0 35 0 01 usesStmtJournal=0 16 String8 0 2 0 %bob% 00 r[2]='%bob%' 17 Goto 0 1 0 00 ``` Compared with: ``` sqlite> explain select neighborhood, facet_cities.name, state ...> from facetable ...> join facet_cities ...> on facetable.city_id = facet_cities.id ...> where neighborhood like '%bob%' order by neighborhood ...> ; addr opcode p1 p2 p3 p4 p5 comment ---- ------------- ---- ---- ---- ------------- -- ------------- 0 Init 0 25 0 00 Start at 25 1 SorterOpen 2 5 0 k(1,B) 00 2 OpenRead 0 43 0 7 00 root=43 iDb=0; facetable 3 OpenRead 1 42 0 2 00 root=42 iDb=0; facet_cities 4 Rewind 0 16 0 00 5 Column 0 6 3 00 r[3]=facetable.neighborhood 6 Function0 1 2 1 like(2) 02 r[1]=func(r[2..3]) 7 IfNot 1 15 1 00 8 Column 0 5 4 00 r[4]=facetable.city_id 9 SeekRowid 1 15 4 00 intkey=r[4] 10 Column 1 1 6 00 r[6]=facet_cities.name 11 Column 0 4 7 00 r[7]=facetable.state 12 Column 0 6 5 00 r[5]=facetable.neighborhood 13 MakeRecord 5 3 9 00 r[9]=mkrec(r[5..7]) 14 SorterInsert 2 9 5 3 00 key=r[9] 15 Next 0 5 0 01 16 OpenPseudo 3 10 5 00 5 columns in r[10] 17 SorterSort 2 24 0 00 18 SorterData 2 10 3 00 r[10]=data 19 Column 3 2 8 00 r[8]=state 20 Column 3 1 7 00 r[7]=facet_cities.name 21 Column 3 0 6 00 r[6]=neighborhood 22 ResultRow 6 3 0 00 output=r[6..8] 23 SorterNext 2 18 0 00 24 Halt 0 0 0 00 25 Transaction 0 0 35 0 01 usesStmtJournal=0 26 String8 0 2 0 %bob% 00 r[2]='%bob%' 27 Goto 0 1 0 00 ``` So actually it looks like the `SorterSort` may be key to understanding this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898527525,https://api.github.com/repos/simonw/datasette/issues/1293,898527525,IC_kwDOBm6k_c41jnEl,9599,simonw,2021-08-13T15:08:03Z,2021-08-13T15:08:03Z,OWNER,Am I going to need to look at the `ResultRow` and its columns but then wind back to that earlier `MakeRecord` and its columns?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898524057,https://api.github.com/repos/simonw/datasette/issues/1293,898524057,IC_kwDOBm6k_c41jmOZ,9599,simonw,2021-08-13T15:06:37Z,2021-08-13T15:06:37Z,OWNER,"Comparing the `explain` for the two versions of that query - one with the order by and one without: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898519924,https://api.github.com/repos/simonw/datasette/issues/1293,898519924,IC_kwDOBm6k_c41jlN0,9599,simonw,2021-08-13T15:03:36Z,2021-08-13T15:03:36Z,OWNER,"Weird edge-case: adding an `order by` changes the order of the columns with respect to the information I am deriving about them. Without order by this gets it right: With order by: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898517872,https://api.github.com/repos/simonw/datasette/issues/1293,898517872,IC_kwDOBm6k_c41jktw,9599,simonw,2021-08-13T15:00:50Z,2021-08-13T15:00:50Z,OWNER,"The primary key column (or `rowid`) often resolves to an `index` record in the `sqlite_master` table, e.g. the second row in this: type | name | tbl_name | rootpage | sql -- | -- | -- | -- | -- table | simple_primary_key | simple_primary_key | 2 | CREATE TABLE simple_primary_key ( id varchar(30) primary key, content text ) index | sqlite_autoindex_simple_primary_key_1 | simple_primary_key | 3 |   ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898506647,https://api.github.com/repos/simonw/datasette/issues/1293,898506647,IC_kwDOBm6k_c41jh-X,9599,simonw,2021-08-13T14:43:19Z,2021-08-13T14:43:19Z,OWNER,Work will continue in PR #1434.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1429#issuecomment-898185944,https://api.github.com/repos/simonw/datasette/issues/1429,898185944,IC_kwDOBm6k_c41iTrY,9599,simonw,2021-08-13T04:37:41Z,2021-08-13T04:37:41Z,OWNER,"If a count is available and the count is less than 1,000 it could say ""Show all"" instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969548935,UI for setting `?_size=max` on table page, https://github.com/simonw/datasette/issues/1432#issuecomment-898084675,https://api.github.com/repos/simonw/datasette/issues/1432,898084675,IC_kwDOBm6k_c41h69D,9599,simonw,2021-08-13T01:11:30Z,2021-08-13T01:11:30Z,OWNER,It's only `datasette-publish-vercel` that will break the actual functionality - the others will have broken tests.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969855774,Rename Datasette.__init__(config=) parameter to settings=, https://github.com/simonw/datasette/issues/1432#issuecomment-898079507,https://api.github.com/repos/simonw/datasette/issues/1432,898079507,IC_kwDOBm6k_c41h5sT,9599,simonw,2021-08-13T01:08:42Z,2021-08-13T01:09:41Z,OWNER,"This is going to break some plugins: https://ripgrep.datasette.io/-/ripgrep?pattern=config%3D&literal=on&glob=%21datasette%2F** > ### datasette-cluster-map/tests/test_cluster_map.py > > @pytest.mark.asyncio > > async def test_respects_base_url(): > ds = Datasette([], memory=True, config={""base_url"": ""/foo/""}) > response = await ds.client.get(""/:memory:?sql=select+1+as+latitude,+2+as+longitude"") > assert ( > > ### datasette-export-notebook/tests/test_export_notebook.py > > @pytest.mark.asyncio > > async def test_notebook_no_csv(db_path): > datasette = Datasette([db_path], config={""allow_csv_stream"": False}) > response = await datasette.client.get(""/db/big.Notebook"") > assert "".csv"" not in response.text > > ### datasette-publish-vercel/tests/test_publish_vercel.py > metadata=metadata, > cors=True, > config={""default_page_size"": 10, ""sql_time_limit_ms"": 2000} > ).app() > """""" > > ### datasette-publish-vercel/datasette_publish_vercel/__init__.py > metadata=metadata{extras}, > cors=True, > config={settings} > > ).app() > > """""".strip() > > ### datasette-search-all/tests/test_search_all.py > > async def test_base_url(db_path, path): > sqlite_utils.Database(db_path)[""creatures""].enable_fts([""name"", ""description""]) > datasette = Datasette([db_path], config={""base_url"": ""/foo/""}) > response = await datasette.client.get(path) > assert response.status_code == 200 I should fix those as soon as this goes out in a release. I won't close this issue until then.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969855774,Rename Datasette.__init__(config=) parameter to settings=, https://github.com/simonw/datasette/issues/1432#issuecomment-898074849,https://api.github.com/repos/simonw/datasette/issues/1432,898074849,IC_kwDOBm6k_c41h4jh,9599,simonw,2021-08-13T01:03:40Z,2021-08-13T01:03:40Z,OWNER,"Also this method: https://github.com/simonw/datasette/blob/77f46297a88ac7e49dad2139410b01ee56d5f99c/datasette/app.py#L422-L424 And the places that use it: https://github.com/simonw/datasette/blob/fc4846850fffd54561bc125332dfe97bb41ff42e/datasette/views/base.py#L617 https://github.com/simonw/datasette/blob/fc4846850fffd54561bc125332dfe97bb41ff42e/datasette/views/database.py#L459 Which is used in this template: https://github.com/simonw/datasette/blob/77f46297a88ac7e49dad2139410b01ee56d5f99c/datasette/templates/table.html#L204 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969855774,Rename Datasette.__init__(config=) parameter to settings=, https://github.com/simonw/datasette/issues/1431#issuecomment-898072940,https://api.github.com/repos/simonw/datasette/issues/1431,898072940,IC_kwDOBm6k_c41h4Fs,9599,simonw,2021-08-13T00:58:40Z,2021-08-13T00:58:40Z,OWNER,"While I'm doing this I should rename this internal variable to avoid confusion in the future: https://github.com/simonw/datasette/blob/e837095ef35ae155b4c78cc9a8b7133a48c94f03/datasette/app.py#L203","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969840302,`--help-config` should be called `--help-settings`, https://github.com/simonw/datasette/issues/1293#issuecomment-813134386,https://api.github.com/repos/simonw/datasette/issues/1293,813134386,MDEyOklzc3VlQ29tbWVudDgxMzEzNDM4Ng==,9599,simonw,2021-04-05T01:20:28Z,2021-08-13T00:42:30Z,OWNER,"... that output might also provide a better way to extract variables than the current mechanism using a regular expression, by looking for the `Variable` opcodes. [UPDATE: it did indeed do that, see #1421]","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898066466,https://api.github.com/repos/simonw/datasette/issues/1293,898066466,IC_kwDOBm6k_c41h2gi,9599,simonw,2021-08-13T00:40:24Z,2021-08-13T00:40:24Z,OWNER,"It figures out renamed columns too: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898065948,https://api.github.com/repos/simonw/datasette/issues/1293,898065948,IC_kwDOBm6k_c41h2Yc,9599,simonw,2021-08-13T00:38:58Z,2021-08-13T00:38:58Z,OWNER,"Trying to run `explain select * from facetable` fails with an error in my prototype, because it tries to execute `explain explain select * from facetable` - so I need to spot that error and ignore it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898065011,https://api.github.com/repos/simonw/datasette/issues/1293,898065011,IC_kwDOBm6k_c41h2Jz,9599,simonw,2021-08-13T00:36:30Z,2021-08-13T00:36:30Z,OWNER,"> https://latest.datasette.io/fixtures?sql=explain+select+*+from+paginated_view will be an interesting test query - because `paginated_view` is defined like this: > > ```sql > CREATE VIEW paginated_view AS > SELECT > content, > '- ' || content || ' -' AS content_extra > FROM no_primary_key; > ``` > > So this will help test that the mechanism isn't confused by output columns that are created through a concatenation expression. Here's what it does for that: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898063815,https://api.github.com/repos/simonw/datasette/issues/1293,898063815,IC_kwDOBm6k_c41h13H,9599,simonw,2021-08-13T00:33:17Z,2021-08-13T00:33:17Z,OWNER,"Improved version of that function: ```python def columns_for_query(conn, sql): """""" Given a SQLite connection ``conn`` and a SQL query ``sql``, returns a list of ``(table_name, column_name)`` pairs, one per returned column. ``(None, None)`` if no table and column could be derived. """""" rows = conn.execute('explain ' + sql).fetchall() table_rootpage_by_register = {r['p1']: r['p2'] for r in rows if r['opcode'] == 'OpenRead'} names_by_rootpage = dict( conn.execute( 'select rootpage, name from sqlite_master where rootpage in ({})'.format( ', '.join(map(str, table_rootpage_by_register.values())) ) ) ) columns_by_column_register = {} for row in rows: if row['opcode'] in ('Rowid', 'Column'): addr, opcode, table_id, cid, column_register, p4, p5, comment = row table = names_by_rootpage[table_rootpage_by_register[table_id]] columns_by_column_register[column_register] = (table, cid) result_row = [dict(r) for r in rows if r['opcode'] == 'ResultRow'][0] registers = list(range(result_row[""p1""], result_row[""p1""] + result_row[""p2""])) all_column_names = {} for table in names_by_rootpage.values(): table_xinfo = conn.execute('pragma table_xinfo({})'.format(table)).fetchall() for row in table_xinfo: all_column_names[(table, row[""cid""])] = row[""name""] final_output = [] for r in registers: try: table, cid = columns_by_column_register[r] final_output.append((table, all_column_names[table, cid])) except KeyError: final_output.append((None, None)) return final_output ``` It works! ```diff diff --git a/datasette/templates/query.html b/datasette/templates/query.html index 75f7f1b..9fe1d4f 100644 --- a/datasette/templates/query.html +++ b/datasette/templates/query.html @@ -67,6 +67,8 @@

+extra_column_info: {{ extra_column_info }} + {% if display_rows %}

This data as {% for name, url in renderers.items() %}{{ name }}{{ "", "" if not loop.last }}{% endfor %}, CSV

diff --git a/datasette/views/database.py b/datasette/views/database.py index 7c36034..02f8039 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -10,6 +10,7 @@ import markupsafe from datasette.utils import ( await_me_maybe, check_visibility, + columns_for_query, derive_named_parameters, to_css_class, validate_sql_select, @@ -248,6 +249,8 @@ class QueryView(DataView): query_error = None + extra_column_info = None + # Execute query - as write or as read if write: if request.method == ""POST"": @@ -334,6 +337,10 @@ class QueryView(DataView): database, sql, params_for_query, truncate=True, **extra_args ) columns = [r[0] for r in results.description] + + # Try to figure out extra column information + db = self.ds.get_database(database) + extra_column_info = await db.execute_fn(lambda conn: columns_for_query(conn, sql)) except sqlite3.DatabaseError as e: query_error = e results = None @@ -462,6 +469,7 @@ class QueryView(DataView): ""show_hide_text"": show_hide_text, ""show_hide_hidden"": markupsafe.Markup(show_hide_hidden), ""hide_sql"": hide_sql, + ""extra_column_info"": extra_column_info, } return ( ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898056013,https://api.github.com/repos/simonw/datasette/issues/1293,898056013,IC_kwDOBm6k_c41hz9N,9599,simonw,2021-08-13T00:12:09Z,2021-08-13T00:12:09Z,OWNER,"Having added column metadata in #1430 (ref #942) I could also include a definition list at the top of the query results page exposing the column descriptions for any columns, using the same EXPLAIN mechanism.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/942#issuecomment-898051645,https://api.github.com/repos/simonw/datasette/issues/942,898051645,IC_kwDOBm6k_c41hy49,9599,simonw,2021-08-13T00:02:25Z,2021-08-13T00:02:25Z,OWNER,"And on mobile: ![5FAF8D73-7199-4BB7-A5B8-9E46DCB4A985](https://user-images.githubusercontent.com/9599/129284817-dc13cbf4-144e-4f4c-8fb7-470602e2eea0.jpeg) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json, https://github.com/simonw/datasette/issues/942#issuecomment-898050457,https://api.github.com/repos/simonw/datasette/issues/942,898050457,IC_kwDOBm6k_c41hymZ,9599,simonw,2021-08-12T23:59:53Z,2021-08-12T23:59:53Z,OWNER,"Documentation: https://docs.datasette.io/en/latest/metadata.html#column-descriptions Live demo: https://latest.datasette.io/fixtures/roadside_attractions ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json, https://github.com/simonw/datasette/issues/942#issuecomment-898037650,https://api.github.com/repos/simonw/datasette/issues/942,898037650,IC_kwDOBm6k_c41hveS,9599,simonw,2021-08-12T23:23:54Z,2021-08-12T23:23:54Z,OWNER,I like this enough that I'm going to ship it as an alpha and try it out on a couple of live projects.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json, https://github.com/simonw/datasette/issues/942#issuecomment-898037456,https://api.github.com/repos/simonw/datasette/issues/942,898037456,IC_kwDOBm6k_c41hvbQ,9599,simonw,2021-08-12T23:23:34Z,2021-08-12T23:23:34Z,OWNER,"Prototype with a `
`: ```diff diff --git a/datasette/static/app.css b/datasette/static/app.css index c6be1e9..bf068fd 100644 --- a/datasette/static/app.css +++ b/datasette/static/app.css @@ -836,6 +841,16 @@ svg.dropdown-menu-icon { background-repeat: no-repeat; } +dl.column-descriptions dt { + font-weight: bold; +} +dl.column-descriptions dd { + padding-left: 1.5em; + white-space: pre-wrap; + line-height: 1.1em; + color: #666; +} + .anim-scale-in { animation-name: scale-in; animation-duration: 0.15s; diff --git a/datasette/templates/table.html b/datasette/templates/table.html index 211352b..466e8a4 100644 --- a/datasette/templates/table.html +++ b/datasette/templates/table.html @@ -51,6 +51,14 @@ {% block description_source_license %}{% include ""_description_source_license.html"" %}{% endblock %} +{% if metadata.columns %} +
+ {% for column_name, column_description in metadata.columns.items() %} +
{{ column_name }}
{{ column_description }}
+ {% endfor %} +
+{% endif %} + {% if filtered_table_rows_count or human_description_en %}

{% if filtered_table_rows_count or filtered_table_rows_count == 0 %}{{ ""{:,}"".format(filtered_table_rows_count) }} row{% if filtered_table_rows_count == 1 %}{% else %}s{% endif %}{% endif %} {% if human_description_en %}{{ human_description_en }}{% endif %} ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json, https://github.com/simonw/datasette/issues/942#issuecomment-898022235,https://api.github.com/repos/simonw/datasette/issues/942,898022235,IC_kwDOBm6k_c41hrtb,9599,simonw,2021-08-12T22:52:23Z,2021-08-12T22:52:23Z,OWNER,I like this. Need to solve for mobile though where the cog menu isn't visible - I think I'll do that with a definition list at the top of the page.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json, https://github.com/simonw/datasette/issues/942#issuecomment-898021895,https://api.github.com/repos/simonw/datasette/issues/942,898021895,IC_kwDOBm6k_c41hroH,9599,simonw,2021-08-12T22:51:36Z,2021-08-12T22:51:36Z,OWNER,"Prototype: ```diff diff --git a/datasette/static/app.css b/datasette/static/app.css index c6be1e9..5ca64cb 100644 --- a/datasette/static/app.css +++ b/datasette/static/app.css @@ -784,9 +784,14 @@ svg.dropdown-menu-icon { font-size: 0.7em; color: #666; margin: 0; - padding: 0; padding: 4px 8px 4px 8px; } +.dropdown-menu .dropdown-column-description { + margin: 0; + color: #666; + padding: 4px 8px 4px 8px; + max-width: 20em; +} .dropdown-menu li { border-bottom: 1px solid #ccc; } diff --git a/datasette/static/table.js b/datasette/static/table.js index 991346d..a903112 100644 --- a/datasette/static/table.js +++ b/datasette/static/table.js @@ -9,6 +9,7 @@ var DROPDOWN_HTML = ``; var DROPDOWN_ICON_SVG = ` @@ -166,6 +167,14 @@ var DROPDOWN_ICON_SVG = `
{% for column in display_columns %} - + {% if not column.sortable %} {{ column.name }} {% else %} diff --git a/datasette/views/table.py b/datasette/views/table.py index 456d806..486a613 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -125,6 +125,7 @@ class RowTableShared(DataView): """"""Returns columns, rows for specified table - including fancy foreign key treatment"""""" db = self.ds.databases[database] table_metadata = self.ds.table_metadata(database, table) + column_descriptions = table_metadata.get(""columns"") or {} column_details = {col.name: col for col in await db.table_column_details(table)} sortable_columns = await self.sortable_columns_for_table(database, table, True) pks = await db.primary_keys(table) @@ -147,6 +148,7 @@ class RowTableShared(DataView): ""is_pk"": r[0] in pks_for_display, ""type"": type_, ""notnull"": notnull, + ""description"": column_descriptions.get(r[0]), } ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json, https://github.com/simonw/datasette/issues/942#issuecomment-897996296,https://api.github.com/repos/simonw/datasette/issues/942,897996296,IC_kwDOBm6k_c41hlYI,9599,simonw,2021-08-12T22:01:36Z,2021-08-12T22:01:36Z,OWNER,"I'm going with `""columns"": {""name-of-column"": ""description-of-column""}`. If I decide to make `""col""` and `""nocol""` available in metadata I'll use those as the keys in the metadata, for consistency with the existing query string parameters. I'm OK with having both `""columns"": ...` and `""col"": ...` keys in the metadata, even though they could be a tiny bit confusing without the documentation.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json, https://github.com/simonw/datasette/issues/1429#issuecomment-897960049,https://api.github.com/repos/simonw/datasette/issues/1429,897960049,IC_kwDOBm6k_c41hchx,9599,simonw,2021-08-12T20:53:04Z,2021-08-12T20:53:04Z,OWNER,"Maybe something like this: > [Next page](#) - 100 per page ([show 1,000 per page](#))","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969548935,UI for setting `?_size=max` on table page, https://github.com/simonw/sqlite-utils/issues/311#issuecomment-896381184,https://api.github.com/repos/simonw/sqlite-utils/issues/311,896381184,IC_kwDOCGYnMM41bbEA,9599,simonw,2021-08-10T23:33:33Z,2021-08-10T23:33:33Z,OWNER,"Now live at https://sqlite-utils.datasette.io/en/latest/reference.html TIL from what I learned today here: https://til.simonwillison.net/sphinx/sphinx-autodoc","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965102534,Add reference documentation generated from docstrings, https://github.com/simonw/sqlite-utils/issues/314#issuecomment-896369551,https://api.github.com/repos/simonw/sqlite-utils/issues/314,896369551,IC_kwDOCGYnMM41bYOP,9599,simonw,2021-08-10T23:06:41Z,2021-08-10T23:06:41Z,OWNER,"I took a big bite out of this when I annotated the ``.insert()`` method - but there are a bunch of other places that still need doing: https://github.com/simonw/sqlite-utils/blob/43bc06481783c3cfcee70c0cb541a686e8894adb/sqlite_utils/db.py#L2382-L2397 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965210966,Type signatures for `.create_table()` and `.create_table_sql()` and `.create()` and `Table.__init__`, https://github.com/simonw/sqlite-utils/issues/314#issuecomment-896344833,https://api.github.com/repos/simonw/sqlite-utils/issues/314,896344833,IC_kwDOCGYnMM41bSMB,9599,simonw,2021-08-10T22:07:34Z,2021-08-10T22:07:34Z,OWNER,"Also the `.insert()` family of methods - they look pretty ugly in Sphinx right now:
I should probably define reusable types for things like `pk=`, which have complex type signatures (a string or a list/tuple of strings) and show up in multiple places.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965210966,Type signatures for `.create_table()` and `.create_table_sql()` and `.create()` and `Table.__init__`, https://github.com/simonw/sqlite-utils/issues/315#issuecomment-896339144,https://api.github.com/repos/simonw/sqlite-utils/issues/315,896339144,IC_kwDOCGYnMM41bQzI,9599,simonw,2021-08-10T21:55:41Z,2021-08-10T21:55:41Z,OWNER,Or should we raise an error if you attempt to call `.delete_where()` on a table that doesn't exist?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965440017,`.delete_where()` returns `[]` when it should return self, https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896284722,https://api.github.com/repos/simonw/sqlite-utils/issues/312,896284722,IC_kwDOCGYnMM41bDgy,9599,simonw,2021-08-10T20:08:03Z,2021-08-10T20:08:21Z,OWNER,"Spotted a rogue backtick: ![A0147E27-7506-49B0-BEFB-20D99BBFEBAD](https://user-images.githubusercontent.com/9599/128927930-b3333dee-a385-409b-a945-f108e6ea40df.jpeg) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965143346,Add reference page to documentation using Sphinx autodoc, https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896200682,https://api.github.com/repos/simonw/sqlite-utils/issues/312,896200682,IC_kwDOCGYnMM41au_q,9599,simonw,2021-08-10T18:03:40Z,2021-08-10T18:03:40Z,OWNER,"Adding type signatures to `create_table()` and `.create_table_sql()` is a bit too involved, I'll do that in a separate issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965143346,Add reference page to documentation using Sphinx autodoc, https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896186025,https://api.github.com/repos/simonw/sqlite-utils/issues/312,896186025,IC_kwDOCGYnMM41arap,9599,simonw,2021-08-10T17:42:51Z,2021-08-10T17:42:51Z,OWNER,That worked! https://sqlite-utils.datasette.io/en/autodoc/reference.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965143346,Add reference page to documentation using Sphinx autodoc, https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896182934,https://api.github.com/repos/simonw/sqlite-utils/issues/312,896182934,IC_kwDOCGYnMM41aqqW,9599,simonw,2021-08-10T17:38:44Z,2021-08-10T17:38:44Z,OWNER,"From https://docs.readthedocs.io/en/stable/config-file/v2.html#packages it looks like I can tell Read The Docs to run `pip install -e .` using a `.readthedocs.yaml` configuration: ```yaml version: 2 sphinx: configuration: docs/conf.py python: version: ""3.9"" install: - method: pip path: . extra_requirements: - docs ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965143346,Add reference page to documentation using Sphinx autodoc, https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896180956,https://api.github.com/repos/simonw/sqlite-utils/issues/312,896180956,IC_kwDOCGYnMM41aqLc,9599,simonw,2021-08-10T17:35:51Z,2021-08-10T17:35:51Z,OWNER,Reading the rest of https://sphinx-rtd-tutorial.readthedocs.io/en/latest/sphinx-config.html#autodoc-configuration it suggests using a `requirements.txt` file to install dependencies - but I use `setup.py` for that so I need to figure out a different pattern here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965143346,Add reference page to documentation using Sphinx autodoc, https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896175438,https://api.github.com/repos/simonw/sqlite-utils/issues/312,896175438,IC_kwDOCGYnMM41ao1O,9599,simonw,2021-08-10T17:28:19Z,2021-08-10T17:28:19Z,OWNER,"https://sphinx-rtd-tutorial.readthedocs.io/en/latest/sphinx-config.html#autodoc-configuration says do something like this at the top of `conf.py`: ```python import os import sys sys.path.insert(0, os.path.abspath('../../simpleble/')) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965143346,Add reference page to documentation using Sphinx autodoc, https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896174456,https://api.github.com/repos/simonw/sqlite-utils/issues/312,896174456,IC_kwDOCGYnMM41aol4,9599,simonw,2021-08-10T17:27:01Z,2021-08-10T17:27:01Z,OWNER,"Docs are now building at https://sqlite-utils.datasette.io/en/autodoc/reference.html But there's a problem! The page is semi-blank: I need to teach Read The Docs how to ensure `sqlite_utils` is available for introspection.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965143346,Add reference page to documentation using Sphinx autodoc, https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896156971,https://api.github.com/repos/simonw/sqlite-utils/issues/312,896156971,IC_kwDOCGYnMM41akUr,9599,simonw,2021-08-10T17:04:22Z,2021-08-10T17:05:59Z,OWNER,"I'm going to get Read The Docs to build the docs for this branch too - on https://readthedocs.org/projects/sqlite-utils/versions/ I am clicking this button: I then set it to ""active"" (so pushes to the branch will build it) and ""hidden"" (so it wouldn't show up in search or in the navigation menu). https://docs.readthedocs.io/en/stable/versions.html#version-states ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965143346,Add reference page to documentation using Sphinx autodoc, https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896154028,https://api.github.com/repos/simonw/sqlite-utils/issues/312,896154028,IC_kwDOCGYnMM41ajms,9599,simonw,2021-08-10T17:01:06Z,2021-08-10T17:01:06Z,OWNER,"On Python 3.6: ``` sqlite_utils/db.py:366: in Database def tables(self) -> List[Table]: E NameError: name 'Table' is not defined ``` Python 3.7 can fix this with `from __future__ import annotations` but since we still support 3.6 I'll have to use a string instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965143346,Add reference page to documentation using Sphinx autodoc, https://github.com/simonw/sqlite-utils/issues/311#issuecomment-896152812,https://api.github.com/repos/simonw/sqlite-utils/issues/311,896152812,IC_kwDOCGYnMM41ajTs,9599,simonw,2021-08-10T16:59:34Z,2021-08-10T16:59:34Z,OWNER,Work will continue in PR #312.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965102534,Add reference documentation generated from docstrings, https://github.com/simonw/sqlite-utils/issues/311#issuecomment-896149590,https://api.github.com/repos/simonw/sqlite-utils/issues/311,896149590,IC_kwDOCGYnMM41aihW,9599,simonw,2021-08-10T16:55:36Z,2021-08-10T16:55:36Z,OWNER,"I'm going to use this as an excuse to add a bunch more type signatures too, refs #266.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965102534,Add reference documentation generated from docstrings, https://github.com/simonw/sqlite-utils/issues/311#issuecomment-896131902,https://api.github.com/repos/simonw/sqlite-utils/issues/311,896131902,IC_kwDOCGYnMM41aeM-,9599,simonw,2021-08-10T16:31:51Z,2021-08-10T16:31:51Z,OWNER,"`make livehtml` wasn't picking up changes I made to the docstrings `.py` files. Fix was to change it to this: ``` sphinx-autobuild -a -b html ""$(SOURCEDIR)"" ""$(BUILDDIR)"" $(SPHINXOPTS) $(0) --watch ../sqlite_utils ``` See https://github.com/executablebooks/sphinx-autobuild#relevant-sphinx-bugs - though that suggested `-a` but didn't suggest `--watch`, which is a tip I got from https://github.com/executablebooks/sphinx-autobuild#working-on-a-sphinx-html-theme ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965102534,Add reference documentation generated from docstrings, https://github.com/simonw/sqlite-utils/issues/309#issuecomment-895622908,https://api.github.com/repos/simonw/sqlite-utils/issues/309,895622908,IC_kwDOCGYnMM41Yh78,9599,simonw,2021-08-09T23:40:29Z,2021-08-09T23:40:29Z,OWNER,TIL about how the stack inspection works: https://til.simonwillison.net/python/find-local-variables-in-exception-traceback,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963897111,"sqlite-utils insert errors should show SQL and parameters, if possible", https://github.com/simonw/sqlite-utils/issues/309#issuecomment-895581038,https://api.github.com/repos/simonw/sqlite-utils/issues/309,895581038,IC_kwDOCGYnMM41YXtu,9599,simonw,2021-08-09T22:03:54Z,2021-08-09T23:39:53Z,OWNER,"Steps to reproduce: echo '{""v"": 34223049823094832094802398430298048240}' | sqlite-utils insert /tmp/blah.db row -","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963897111,"sqlite-utils insert errors should show SQL and parameters, if possible", https://github.com/simonw/sqlite-utils/issues/309#issuecomment-895592507,https://api.github.com/repos/simonw/sqlite-utils/issues/309,895592507,IC_kwDOCGYnMM41Yag7,9599,simonw,2021-08-09T22:26:28Z,2021-08-09T22:33:48Z,OWNER,"Demo: ``` $ echo '{""v"": 34223049823094832094802398430298048240}' | sqlite-utils insert /tmp/blah.db row - Error: Python int too large to convert to SQLite INTEGER sql = INSERT INTO [row] ([v]) VALUES (?); parameters = [34223049823094832094802398430298048240] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963897111,"sqlite-utils insert errors should show SQL and parameters, if possible", https://github.com/simonw/sqlite-utils/issues/309#issuecomment-895587441,https://api.github.com/repos/simonw/sqlite-utils/issues/309,895587441,IC_kwDOCGYnMM41YZRx,9599,simonw,2021-08-09T22:15:45Z,2021-08-09T22:15:45Z,OWNER,"``` OverflowError: Python int too large to convert to SQLite INTEGER >>> import sys >>> def find_variables(tb, vars): to_find = list(vars) found = {} for var in to_find: if var in tb.tb_frame.f_locals: vars.remove(var) found[var] = tb.tb_frame.f_locals[var] if vars and tb.tb_next: found.update(find_variables(tb.tb_next, vars)) return found ... >>> find_variables(sys.last_traceback, [""sql"", ""params""]) {'params': [34223049823094832094802398430298048240], 'sql': 'INSERT INTO [row] ([v]) VALUES (?);'} ```","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963897111,"sqlite-utils insert errors should show SQL and parameters, if possible", https://github.com/simonw/sqlite-utils/issues/309#issuecomment-895587282,https://api.github.com/repos/simonw/sqlite-utils/issues/309,895587282,IC_kwDOCGYnMM41YZPS,9599,simonw,2021-08-09T22:15:25Z,2021-08-09T22:15:25Z,OWNER,"I'm going to use a bit of a dirty trick for this one: I'm going to recursively inspect the stack on an error and try to find the `sql` and `params` variables. That way I can handle this all at the CLI layer without changing the exceptions that are being raised by the Python library.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963897111,"sqlite-utils insert errors should show SQL and parameters, if possible", https://github.com/simonw/sqlite-utils/issues/309#issuecomment-895577012,https://api.github.com/repos/simonw/sqlite-utils/issues/309,895577012,IC_kwDOCGYnMM41YWu0,9599,simonw,2021-08-09T21:55:52Z,2021-08-09T21:59:03Z,OWNER,"Yeah this error message could certainly be more helpful. I thought `OverflowError` might be one of the SQLite exceptions: https://docs.python.org/3/library/sqlite3.html#exceptions - but it turns out it's actually reusing the Python built-in `OverflowError` class: ```python import sqlite3 db = sqlite3.connect("":memory:"") caught = [] try: db.execute(""create table foo (number integer)"") db.execute(""insert into foo (number) values (?)"", [34223049823094832094802398430298048240]) except Exception as e: print(e) caught.append(e) isinstance(caught[0], OverflowError) ``` Here's where that happens in the Python `sqlite3` module code: https://github.com/python/cpython/blob/058fb35b57ca8c5063d16ec818e668b3babfea65/Modules/_sqlite/util.c#L123-L124","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963897111,"sqlite-utils insert errors should show SQL and parameters, if possible", https://github.com/simonw/sqlite-utils/issues/310#issuecomment-895572309,https://api.github.com/repos/simonw/sqlite-utils/issues/310,895572309,IC_kwDOCGYnMM41YVlV,9599,simonw,2021-08-09T21:46:15Z,2021-08-09T21:46:15Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#flattening-nested-json-objects,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",964400482,`sqlite-utils insert --flatten` option to flatten nested JSON, https://github.com/simonw/sqlite-utils/issues/310#issuecomment-895571420,https://api.github.com/repos/simonw/sqlite-utils/issues/310,895571420,IC_kwDOCGYnMM41YVXc,9599,simonw,2021-08-09T21:44:38Z,2021-08-09T21:44:38Z,OWNER,When I ship this I should update the TILs at https://til.simonwillison.net/cloudrun/tailing-cloud-run-request-logs and https://til.simonwillison.net/jq/flatten-nested-json-objects-jq to reference it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",964400482,`sqlite-utils insert --flatten` option to flatten nested JSON, https://github.com/simonw/datasette/issues/1426#issuecomment-895522818,https://api.github.com/repos/simonw/datasette/issues/1426,895522818,IC_kwDOBm6k_c41YJgC,9599,simonw,2021-08-09T20:34:10Z,2021-08-09T20:34:10Z,OWNER,At the very least Datasette should serve a blank `/robots.txt` by default - I'm seeing a ton of 404s for it in the logs.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",964322136,"Manage /robots.txt in Datasette core, block robots by default", https://github.com/simonw/datasette/issues/1426#issuecomment-895510773,https://api.github.com/repos/simonw/datasette/issues/1426,895510773,IC_kwDOBm6k_c41YGj1,9599,simonw,2021-08-09T20:14:50Z,2021-08-09T20:19:22Z,OWNER,"https://twitter.com/mal/status/1424825895139876870 > True pinging google should be part of the build process on a static site :) That's another aspect of this: if you DO want your site crawled, teaching the `datasette publish` command how to ping Google when a deploy has gone out could be a nice improvement. Annoyingly it looks like you need to configure an auth token of some sort in order to use their API though, which is likely too much hassle to be worth building into Datasette itself: https://developers.google.com/search/apis/indexing-api/v3/using-api ``` curl -X POST https://indexing.googleapis.com/v3/urlNotifications:publish -d '{ ""url"": ""https://careers.google.com/jobs/google/technical-writer"", ""type"": ""URL_UPDATED"" }' -H ""Content-Type: application/json"" { ""error"": { ""code"": 401, ""message"": ""Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project."", ""status"": ""UNAUTHENTICATED"" } } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",964322136,"Manage /robots.txt in Datasette core, block robots by default", https://github.com/simonw/datasette/issues/1426#issuecomment-895509536,https://api.github.com/repos/simonw/datasette/issues/1426,895509536,IC_kwDOBm6k_c41YGQg,9599,simonw,2021-08-09T20:12:57Z,2021-08-09T20:12:57Z,OWNER,I could try out the `X-Robots` HTTP header too: https://developers.google.com/search/docs/advanced/robots/robots_meta_tag#xrobotstag,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",964322136,"Manage /robots.txt in Datasette core, block robots by default", https://github.com/simonw/datasette/issues/1426#issuecomment-895500565,https://api.github.com/repos/simonw/datasette/issues/1426,895500565,IC_kwDOBm6k_c41YEEV,9599,simonw,2021-08-09T20:00:04Z,2021-08-09T20:00:04Z,OWNER,"A few options for how this would work: - `datasette ... --robots allow` - `datasette ... --setting robots allow` Options could be: - `allow` - allow all crawling - `deny` - deny all crawling - `limited` - allow access to the homepage and the index pages for each database and each table, but disallow crawling any further than that The ""limited"" mode is particularly interesting. Could even make it the default, but I think that may be a bit too confusing. Idea would be to get the key pages indexed but use `nofollow` to discourage crawlers from indexing individual row pages or deep pages like `https://datasette.io/content/repos?_facet=owner&_facet=language&_facet_array=topics&topics__arraycontains=sqlite#facet-owner`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",964322136,"Manage /robots.txt in Datasette core, block robots by default", https://github.com/simonw/datasette/issues/1421#issuecomment-894930013,https://api.github.com/repos/simonw/datasette/issues/1421,894930013,IC_kwDOBm6k_c41V4xd,9599,simonw,2021-08-09T03:38:06Z,2021-08-09T03:38:06Z,OWNER,"Amusing edge-case: if you run this against a `explain ...` query it falls back to using regular expressions, because `explain explain select ...` is invalid SQL. https://latest.datasette.io/fixtures?sql=explain+select+*+from+facetable%0D%0Awhere+state+%3D+%3Astate%0D%0Aand+on_earth+%3D+%3Aon_earth%0D%0Aand+neighborhood+not+like+%2700%3A04%27&state=&on_earth=","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times", https://github.com/simonw/datasette/issues/1421#issuecomment-894929769,https://api.github.com/repos/simonw/datasette/issues/1421,894929769,IC_kwDOBm6k_c41V4tp,9599,simonw,2021-08-09T03:36:49Z,2021-08-09T03:36:49Z,OWNER,"SQLite carries a warning about using `EXPLAIN` like this: https://www.sqlite.org/lang_explain.html > The output from EXPLAIN and EXPLAIN QUERY PLAN is intended for interactive analysis and troubleshooting only. The details of the output format are subject to change from one release of SQLite to the next. Applications should not use EXPLAIN or EXPLAIN QUERY PLAN since their exact behavior is variable and only partially documented. I think that's OK here, because of the regular expression fallback. If the format changes in the future in a way that breaks the query the error should be caught and the regex-captured parameters should be returned instead. Hmmm... actually that's not entirely true: https://github.com/simonw/datasette/blob/b1fed48a95516ae84c0f020582303ab50ab817e2/datasette/utils/__init__.py#L1084-L1091 If the format changes such that the same columns are returned but the `[row[""p4""].lstrip("":"") for row in results if row[""opcode""] == ""Variable""]` list comprehension returns an empty array it will break Datasette! I'm going to take that risk for the moment, but I'll actively watch out for problems in the future. If this does turn out to be bad I can always go back to the pure regular expression mechanism. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times", https://github.com/simonw/datasette/issues/1421#issuecomment-894929080,https://api.github.com/repos/simonw/datasette/issues/1421,894929080,IC_kwDOBm6k_c41V4i4,9599,simonw,2021-08-09T03:33:02Z,2021-08-09T03:33:02Z,OWNER,"Fixed! Fantastic, this one has been bothering me for *years*. https://latest.datasette.io/fixtures?sql=select+*+from+facetable%0D%0Awhere+state+%3D+%3Astate%0D%0Aand+on_earth+%3D+%3Aon_earth%0D%0Aand+neighborhood+not+like+%2700%3A04%27 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times", https://github.com/simonw/datasette/issues/1421#issuecomment-894927185,https://api.github.com/repos/simonw/datasette/issues/1421,894927185,IC_kwDOBm6k_c41V4FR,9599,simonw,2021-08-09T03:25:01Z,2021-08-09T03:25:01Z,OWNER,"One catch with this approach: if the SQL query is invalid, the parameters will not be extracted and shown as form fields. Maybe that's completely fine? Why display a form if it's going to break when the user actually runs the query? But it does bother me. I worry that someone who is still iterating on and editing their query before actually starting to use it might find the behaviour confusing. So maybe if the query raises an exception it could fall back on the regular expression results?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times", https://github.com/simonw/datasette/issues/1421#issuecomment-894925914,https://api.github.com/repos/simonw/datasette/issues/1421,894925914,IC_kwDOBm6k_c41V3xa,9599,simonw,2021-08-09T03:20:42Z,2021-08-09T03:20:42Z,OWNER,"I think this works! ```python _re_named_parameter = re.compile("":([a-zA-Z0-9_]+)"") async def derive_named_parameters(db, sql): explain = 'explain {}'.format(sql.strip().rstrip("";"")) possible_params = _re_named_parameter.findall(sql) try: results = await db.execute(explain, {p: None for p in possible_params}) return [row[""p4""].lstrip("":"") for row in results if row[""opcode""] == ""Variable""] except sqlite3.DatabaseError: return [] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times", https://github.com/simonw/datasette/issues/1421#issuecomment-894925437,https://api.github.com/repos/simonw/datasette/issues/1421,894925437,IC_kwDOBm6k_c41V3p9,9599,simonw,2021-08-09T03:19:00Z,2021-08-09T03:19:00Z,OWNER,"This may not work: > `ERROR: sql = 'explain select 1 + :one + :two', params = None: You did not supply a value for binding 1.` The `explain` queries themselves want me to pass them parameters. I could try using the regex to pull out candidates and passing `None` for each of those, including incorrect ones like `:31`. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times", https://github.com/simonw/datasette/issues/1421#issuecomment-894922703,https://api.github.com/repos/simonw/datasette/issues/1421,894922703,IC_kwDOBm6k_c41V2_P,9599,simonw,2021-08-09T03:09:29Z,2021-08-09T03:09:29Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/ad90a72afa21b737b162e2bbdddc301a97d575cd/datasette/views/database.py#L225-L231,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times", https://github.com/simonw/datasette/issues/1421#issuecomment-894922145,https://api.github.com/repos/simonw/datasette/issues/1421,894922145,IC_kwDOBm6k_c41V22h,9599,simonw,2021-08-09T03:07:38Z,2021-08-09T03:07:38Z,OWNER,"I hoped this would work: ```sql with foo as ( explain select * from facetable where state = :state and on_earth = :on_earth and neighborhood not like '00:04' ) select p4 from foo where opcode = 'Variable' ``` But sadly [it returns an error](https://latest.datasette.io/fixtures?sql=with+foo+as+%28%0D%0A++explain+select+*+from+facetable%0D%0A++where+state+%3D+%3Astate%0D%0A++and+on_earth+%3D+%3Aon_earth%0D%0A++and+neighborhood+not+like+%2700%3A04%27%0D%0A%29%0D%0Aselect+p4+from+foo+where+opcode+%3D+%27Variable%27&state=&on_earth=&04=): > near ""explain"": syntax error","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times", https://github.com/simonw/datasette/issues/1421#issuecomment-894921512,https://api.github.com/repos/simonw/datasette/issues/1421,894921512,IC_kwDOBm6k_c41V2so,9599,simonw,2021-08-09T03:05:26Z,2021-08-09T03:05:26Z,OWNER,"I may have a way to work around this, using `explain`. Consider this query: ```sql select * from facetable where state = :state and on_earth = :on_earth and neighborhood not like '00:04' ``` Datasette currently gets confused and shows three form fields: https://latest.datasette.io/fixtures?sql=select+*+from+facetable%0D%0Awhere+state+%3D+%3Astate%0D%0Aand+on_earth+%3D+%3Aon_earth%0D%0Aand+neighborhood+not+like+%2700%3A04%27&state=&on_earth=&04= But... if I run `explain` [against that](https://latest.datasette.io/fixtures?sql=explain+select+*+from+facetable%0D%0Awhere+state+%3D+%3Astate%0D%0Aand+on_earth+%3D+%3Aon_earth%0D%0Aand+neighborhood+not+like+%2700%3A04%27&state=&on_earth=&04=) I get this (truncated): addr | opcode | p1 | p2 | p3 | p4 | p5 | comment -- | -- | -- | -- | -- | -- | -- | -- 20 | ResultRow | 6 | 10 | 0 |   | 0 |   21 | Next | 0 | 3 | 0 |   | 1 |   22 | Halt | 0 | 0 | 0 |   | 0 |   23 | Transaction | 0 | 0 | 35 | 0 | 1 |   24 | Variable | 1 | 2 | 0 | :state | 0 |   25 | Variable | 2 | 3 | 0 | :on_earth | 0 |   26 | String8 | 0 | 4 | 0 | 00:04 | 0 |   27 | Goto | 0 | 1 | 0 |   | 0 |   Could it be as simple as pulling out those `Variable` rows to figure out the names of the variables in the query?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times", https://github.com/simonw/datasette/issues/1425#issuecomment-894900267,https://api.github.com/repos/simonw/datasette/issues/1425,894900267,IC_kwDOBm6k_c41Vxgr,9599,simonw,2021-08-09T01:31:22Z,2021-08-09T01:31:22Z,OWNER,"I used this to build a new plugin: https://github.com/simonw/datasette-query-links Demo here: https://latest-with-plugins.datasette.io/fixtures?sql=select%0D%0A++%27select+*+from+[facetable]%27+as+query%0D%0Aunion%0D%0Aselect%0D%0A++%27select+sqlite_version()%27%0D%0Aunion%0D%0Aselect%0D%0A++%27select+this+is+invalid+SQL+so+will+not+be+linked%27","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable, https://github.com/simonw/datasette/issues/1425#issuecomment-894893319,https://api.github.com/repos/simonw/datasette/issues/1425,894893319,IC_kwDOBm6k_c41Vv0H,9599,simonw,2021-08-09T01:08:56Z,2021-08-09T01:09:12Z,OWNER,Demo: https://latest.datasette.io/fixtures/simple_primary_key shows `RENDER_CELL_ASYNC_RESULT` where the CSV version shows `RENDER_CELL_ASYNC`: https://latest.datasette.io/fixtures/simple_primary_key.csv - because of this test plugin code: https://github.com/simonw/datasette/blob/a390bdf9cef01d8723d025fc3348e81345ff4856/tests/plugins/my_plugin.py#L98-L122,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable, https://github.com/simonw/datasette/issues/1425#issuecomment-894884874,https://api.github.com/repos/simonw/datasette/issues/1425,894884874,IC_kwDOBm6k_c41VtwK,9599,simonw,2021-08-09T00:38:20Z,2021-08-09T00:38:20Z,OWNER,I'm trying the version where I remove `firstresult=True`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable, https://github.com/simonw/datasette/issues/1425#issuecomment-894883664,https://api.github.com/repos/simonw/datasette/issues/1425,894883664,IC_kwDOBm6k_c41VtdQ,9599,simonw,2021-08-09T00:33:56Z,2021-08-09T00:33:56Z,OWNER,"I could extract that code out and write my own function which implements the equivalent of calling `pm.hook.render_cell(...)` but runs `await_me_maybe()` before checking if `res is not None`. That's pretty nasty. Could I instead call the plugin hook normally, but then have additional logic which says ""if I await it and it returns `None` then try calling the hook again but skip this one"" - not sure if there's a way to do that either. I could remove the `firstresult=True` from the hookspec - which would cause it to call and return ALL hooks - but then in my own code use only the first one. This is slightly less efficient (since it calls all the hooks and then discards all-but-one value) but it's the least unpleasant in terms of the code I would have to write - plus I don't think it's going to be THAT common for someone to have multiple expensive `render_cell()` hooks installed at once (they are usually pretty cheap).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable, https://github.com/simonw/datasette/issues/1425#issuecomment-894882642,https://api.github.com/repos/simonw/datasette/issues/1425,894882642,IC_kwDOBm6k_c41VtNS,9599,simonw,2021-08-09T00:29:57Z,2021-08-09T00:29:57Z,OWNER,"Here's the code in `pluggy` that implements this: https://github.com/pytest-dev/pluggy/blob/0a064fe275060dbdb1fe6e10c888e72bc400fb33/src/pluggy/callers.py#L31-L43 ```python if hook_impl.hookwrapper: try: gen = hook_impl.function(*args) next(gen) # first yield teardowns.append(gen) except StopIteration: _raise_wrapfail(gen, ""did not yield"") else: res = hook_impl.function(*args) if res is not None: results.append(res) if firstresult: # halt further impl calls break ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable, https://github.com/simonw/datasette/issues/1425#issuecomment-894882123,https://api.github.com/repos/simonw/datasette/issues/1425,894882123,IC_kwDOBm6k_c41VtFL,9599,simonw,2021-08-09T00:27:43Z,2021-08-09T00:27:43Z,OWNER,"Good news: `render_cell()` is the only hook to use `firstresult=True`: https://github.com/simonw/datasette/blob/f3c9edb376a13c09b5ecf97c7390f4e49efaadf2/datasette/hookspecs.py#L62-L64 https://pluggy.readthedocs.io/en/latest/#first-result-only","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable, https://github.com/simonw/datasette/issues/1425#issuecomment-894881448,https://api.github.com/repos/simonw/datasette/issues/1425,894881448,IC_kwDOBm6k_c41Vs6o,9599,simonw,2021-08-09T00:24:25Z,2021-08-09T00:24:39Z,OWNER,"My hunch is that the ""skip this `render_cell()` result if it returns `None`"" logic isn't working correctly, ever since I added the `await_me_maybe` line. Could that be because Pluggy handles the ""do the next if `None` is returned"" logic itself, but I'm no-longer returning `None`, I'm returning an awaitable which when awaited returns `None`. This would suggest that all of the `await_me_maybe()` plugin hooks have the same bug. That's definitely possible - it may well be that no-one has yet stumbled across a bug caused by a plugin returning an awaitable and hence not being skipped, because plugin hooks that return awaitable are rare enough that no-one has tried two plugins which both use that trick. Still don't see why it would pass on my laptop but fail in CI though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable, https://github.com/simonw/datasette/issues/1425#issuecomment-894881016,https://api.github.com/repos/simonw/datasette/issues/1425,894881016,IC_kwDOBm6k_c41Vsz4,9599,simonw,2021-08-09T00:21:53Z,2021-08-09T00:21:53Z,OWNER,"Still one test failure: ``` def test_hook_render_cell_link_from_json(app_client): sql = """""" select '{""href"": ""http://example.com/"", ""label"":""Example""}' """""".strip() path = ""/fixtures?"" + urllib.parse.urlencode({""sql"": sql}) response = app_client.get(path) td = Soup(response.body, ""html.parser"").find(""table"").find(""tbody"").find(""td"") a = td.find(""a"") > assert a is not None, str(a) E AssertionError: None E assert None is not None ``` The weird thing about this one is that I can't replicate it on my laptop - but it happens in CI every time, including when I shell in and try to run that single test.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable, https://github.com/simonw/datasette/issues/1425#issuecomment-894869692,https://api.github.com/repos/simonw/datasette/issues/1425,894869692,IC_kwDOBm6k_c41VqC8,9599,simonw,2021-08-08T23:08:29Z,2021-08-08T23:08:29Z,OWNER,Updated documentation: https://docs.datasette.io/en/latest/plugin_hooks.html#render-cell-value-column-table-database-datasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable, https://github.com/simonw/datasette/issues/1425#issuecomment-894865323,https://api.github.com/repos/simonw/datasette/issues/1425,894865323,IC_kwDOBm6k_c41Vo-r,9599,simonw,2021-08-08T22:33:19Z,2021-08-08T22:33:19Z,OWNER,"I can do this with the `await_me_maybe()` function, as seen here: https://github.com/simonw/datasette/blob/a21853c9dade240734abc6b4f750fae09a3e840a/datasette/app.py#L864-L873","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable, https://github.com/simonw/datasette/issues/1424#issuecomment-894864744,https://api.github.com/repos/simonw/datasette/issues/1424,894864744,IC_kwDOBm6k_c41Vo1o,9599,simonw,2021-08-08T22:27:31Z,2021-08-08T22:27:31Z,OWNER,https://docs.python.org/3/library/sqlite3.html#exceptions is useful - it looks like `sqlite3.DatabaseError` is the super-class of all of the other exceptions that we might see.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963527045,Document exceptions that can be raised by db.execute() and friends, https://github.com/simonw/datasette/issues/1424#issuecomment-894864682,https://api.github.com/repos/simonw/datasette/issues/1424,894864682,IC_kwDOBm6k_c41Vo0q,9599,simonw,2021-08-08T22:26:46Z,2021-08-08T22:26:46Z,OWNER,"Note that the `sqlite3` exceptions are in `sqlite3` if using the Python standard library but are in `pysqlite3` if that module is being used instead. So maybe encourage people to use them from `datasette.sqlite.sqlite3` instead, which will point to the correct package.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963527045,Document exceptions that can be raised by db.execute() and friends, https://github.com/simonw/datasette/issues/1424#issuecomment-894864616,https://api.github.com/repos/simonw/datasette/issues/1424,894864616,IC_kwDOBm6k_c41Vozo,9599,simonw,2021-08-08T22:26:08Z,2021-08-08T22:26:08Z,OWNER,"- `datasette.database.QueryInterrupted` for queries that were interrupted - `sqlite3.OperationalError` - `sqlite3.DatabaseError` and more","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963527045,Document exceptions that can be raised by db.execute() and friends, https://github.com/simonw/datasette/issues/1424#issuecomment-894864404,https://api.github.com/repos/simonw/datasette/issues/1424,894864404,IC_kwDOBm6k_c41VowU,9599,simonw,2021-08-08T22:24:06Z,2021-08-08T22:24:06Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/de5ce2e56339ad8966f417a4758f7c210c017dec/datasette/database.py#L176-L200,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963527045,Document exceptions that can be raised by db.execute() and friends, https://github.com/simonw/datasette/issues/1422#issuecomment-894607989,https://api.github.com/repos/simonw/datasette/issues/1422,894607989,IC_kwDOBm6k_c41UqJ1,9599,simonw,2021-08-07T05:31:57Z,2021-08-07T05:31:57Z,OWNER,"Demo: https://latest.datasette.io/fixtures/neighborhood_search Documentation: https://docs.datasette.io/en/latest/sql_queries.html#additional-canned-query-options","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",961367843,Ability to default to hiding the SQL for a canned query, https://github.com/simonw/datasette/issues/1421#issuecomment-894606843,https://api.github.com/repos/simonw/datasette/issues/1421,894606843,IC_kwDOBm6k_c41Up37,9599,simonw,2021-08-07T05:17:12Z,2021-08-07T05:17:12Z,OWNER,Marking this blocked because I don't have a way around the needing-a-SQLite-SQL-parser problem at the moment.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times", https://github.com/simonw/datasette/issues/1421#issuecomment-894606796,https://api.github.com/repos/simonw/datasette/issues/1421,894606796,IC_kwDOBm6k_c41Up3M,9599,simonw,2021-08-07T05:16:39Z,2021-08-07T05:16:39Z,OWNER,"Urgh, yeah I've seen this one before. Fixing it pretty much requires writing a full SQLite SQL syntax parser in Python, which is frustratingly complicated for solving this issue! You can work around this for a canned query by using the optional `params:` argument documented here: https://docs.datasette.io/en/stable/sql_queries.html#canned-query-parameters","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times", https://github.com/simonw/datasette/issues/1422#issuecomment-894589140,https://api.github.com/repos/simonw/datasette/issues/1422,894589140,IC_kwDOBm6k_c41UljU,9599,simonw,2021-08-07T01:58:16Z,2021-08-07T01:58:24Z,OWNER,Also need to consider this hidden field - it should pass the `_hide_sql` or `_show_sql` parameters depending on the same logic: https://github.com/simonw/datasette/blob/acc22436622ff8476c30acf45ed60f54b4aaa5d9/datasette/templates/query.html#L47-L49,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",961367843,Ability to default to hiding the SQL for a canned query, https://github.com/simonw/datasette/issues/1423#issuecomment-894454644,https://api.github.com/repos/simonw/datasette/issues/1423,894454644,IC_kwDOBm6k_c41UEt0,9599,simonw,2021-08-06T18:52:49Z,2021-08-06T18:52:49Z,OWNER,"This means that the counts would be unavailable to users who cannot see tooltips (e.g. mobile users) on pages that did not have any facets that broke the 30 limit and hence displayed that ""..."" link. I think I'm OK with that, for the moment. May revisit in the future.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",962391325,Show count of facet values if ?_facet_size=max, https://github.com/simonw/datasette/issues/1423#issuecomment-894454087,https://api.github.com/repos/simonw/datasette/issues/1423,894454087,IC_kwDOBm6k_c41UElH,9599,simonw,2021-08-06T18:51:42Z,2021-08-06T18:51:42Z,OWNER,"The invisible tooltip could say ""Showing 30 items, more available"" (helping save you from counting up to 20 if you know about the secret feature). The numbers could then be fully displayed on the ""..."" page.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",962391325,Show count of facet values if ?_facet_size=max, https://github.com/simonw/datasette/issues/1423#issuecomment-894453520,https://api.github.com/repos/simonw/datasette/issues/1423,894453520,IC_kwDOBm6k_c41UEcQ,9599,simonw,2021-08-06T18:50:40Z,2021-08-06T18:50:40Z,OWNER,"Point of confusion: if only 30 options are shown, but there's a `...` at the end, what would the number be? It can't be the total number of facets because we haven't counted them all - but if it's just the number of displayed facets that's like to be confusing. So the original idea of showing the counts only if `_facet_size=max` is a good one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",962391325,Show count of facet values if ?_facet_size=max, https://github.com/simonw/datasette/issues/1423#issuecomment-894452990,https://api.github.com/repos/simonw/datasette/issues/1423,894452990,IC_kwDOBm6k_c41UET-,9599,simonw,2021-08-06T18:49:37Z,2021-08-06T18:49:37Z,OWNER,"Could display them always, like this: That's with `23`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",962391325,Show count of facet values if ?_facet_size=max, https://github.com/simonw/datasette/issues/1423#issuecomment-893996604,https://api.github.com/repos/simonw/datasette/issues/1423,893996604,IC_kwDOBm6k_c41SU48,9599,simonw,2021-08-06T04:43:07Z,2021-08-06T04:43:37Z,OWNER,"Problem: on a page which doesn't have quite enough facet values to trigger the display of the ""..."" link that links to `?_facet_size=max` the user would still have to manually count the values - up to 30 by default. So maybe the count should always be shown, perhaps as a non-bold light colored number? I could even hide it in a non-discoverable tooltip.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",962391325,Show count of facet values if ?_facet_size=max, https://github.com/simonw/datasette/issues/1419#issuecomment-893133496,https://api.github.com/repos/simonw/datasette/issues/1419,893133496,IC_kwDOBm6k_c41PCK4,9599,simonw,2021-08-05T03:22:44Z,2021-08-05T03:22:44Z,OWNER,"I ran into this exact same problem today! I only just learned how to use filter on aggregates: https://til.simonwillison.net/sqlite/sqlite-aggregate-filter-clauses A workaround I used is to add this to the deploy command: datasette publish cloudrun ... --install=pysqlite3-binary This will install the https://pypi.org/project/pysqlite3-binary for package which bundles a more recent SQLite version.","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959710008,`publish cloudrun` should deploy a more recent SQLite version, https://github.com/simonw/datasette/issues/1422#issuecomment-893131703,https://api.github.com/repos/simonw/datasette/issues/1422,893131703,IC_kwDOBm6k_c41PBu3,9599,simonw,2021-08-05T03:16:46Z,2021-08-05T03:16:46Z,OWNER,"The logic for this is a little bit fiddly, due to the need to switch to using `?_show_sql=1` on the link depending on the context. - If metadata says hide and there's no query string, hide and link to `?_show_sql=1` - If metadata says hide but query string says `?_show_sql=1`, show and have hide link linking to URL without `?_show_sql=1` - Otherwise, show and link to `?_hide_sql=1` - ... or if that query string is there then hide and link to URL without `?_hide_sql=1`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",961367843,Ability to default to hiding the SQL for a canned query, https://github.com/simonw/datasette/issues/1422#issuecomment-893122356,https://api.github.com/repos/simonw/datasette/issues/1422,893122356,IC_kwDOBm6k_c41O_c0,9599,simonw,2021-08-05T02:52:31Z,2021-08-05T02:52:44Z,OWNER,"If you do this it should still be possible to view the SQL - which means we need a new parameter. I propose `?_show_sql=1` to over-ride the hidden default. I think the configuration should use `hide_sql: true` - looking like this: ```yaml databases: fixtures: queries: neighborhood_search: hide_sql: true sql: |- select neighborhood, facet_cities.name, state from facetable join facet_cities on facetable.city_id = facet_cities.id where neighborhood like '%' || :text || '%' order by neighborhood title: Search neighborhoods ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",961367843,Ability to default to hiding the SQL for a canned query, https://github.com/simonw/datasette/issues/1420#issuecomment-893079520,https://api.github.com/repos/simonw/datasette/issues/1420,893079520,IC_kwDOBm6k_c41O0_g,9599,simonw,2021-08-05T00:54:59Z,2021-08-05T00:54:59Z,OWNER,"Just saw this error: `ERROR: (gcloud.run.deploy) The `--cpu` flag is not supported on the fully managed version of Cloud Run. Specify `--platform gke` or run `gcloud config set run/platform gke` to work with Cloud Run for Anthos deployed on Google Cloud.` Which is weird because I managed to run this successfully the other day. Maybe a region difference thing?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959898166,`datasette publish cloudrun --cpu X` option, https://github.com/simonw/sqlite-utils/issues/308#issuecomment-892988609,https://api.github.com/repos/simonw/sqlite-utils/issues/308,892988609,IC_kwDOCGYnMM41OezB,9599,simonw,2021-08-04T21:30:59Z,2021-08-04T21:30:59Z,OWNER,"OK, this works great as a proof-of-concept (though maybe I should remove the output from the cells so people have to run it themselves in the Binder environment?) Next challenge: how extensive should it be? What features should I cover? Should I do a basic tutorial and then several advanced tutorials?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",961008507,Add an interactive tutorial as a Jupyter notebook, https://github.com/simonw/sqlite-utils/issues/308#issuecomment-892957281,https://api.github.com/repos/simonw/sqlite-utils/issues/308,892957281,IC_kwDOCGYnMM41OXJh,9599,simonw,2021-08-04T20:37:00Z,2021-08-04T20:37:00Z,OWNER,https://mybinder.org/v2/gh/simonw/sqlite-utils/main?filepath=docs%2Ftutorial.ipynb,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",961008507,Add an interactive tutorial as a Jupyter notebook, https://github.com/simonw/datasette/issues/1420#issuecomment-892376353,https://api.github.com/repos/simonw/datasette/issues/1420,892376353,IC_kwDOBm6k_c41MJUh,9599,simonw,2021-08-04T05:33:12Z,2021-08-04T05:33:12Z,OWNER,"In the Cloud Run console (before I deleted these services) when I click ""Edit and deploy new revision"" I see this for the default one: And this for the big one: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959898166,`datasette publish cloudrun --cpu X` option, https://github.com/simonw/datasette/issues/1420#issuecomment-892374253,https://api.github.com/repos/simonw/datasette/issues/1420,892374253,IC_kwDOBm6k_c41MIzt,9599,simonw,2021-08-04T05:27:21Z,2021-08-04T05:29:59Z,OWNER,"I'll delete these services shortly afterwards. Right now: https://fixtures-over-provisioned-issue-1420-j7hipcg4aq-uc.a.run.app/-/psutil (deployed with `--memory 8G --cpu 4`) returns: ``` process.memory_info() pmem(rss=60456960, vms=518930432, shared=0, text=0, lib=0, data=0, dirty=0) ... psutil.cpu_times(True) scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0) scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0) scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0) scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0) psutil.virtual_memory() svmem(total=2147483648, available=2092531712, percent=2.6, used=33103872, free=2092531712, active=44130304, inactive=10792960, buffers=0, cached=21848064, shared=262144, slab=0) ``` https://fixtures-default-issue-1420-j7hipcg4aq-uc.a.run.app/-/psutil returns: ``` process.memory_info() pmem(rss=49324032, vms=140595200, shared=0, text=0, lib=0, data=0, dirty=0) ... psutil.cpu_times(True) scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0) scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0) psutil.virtual_memory() svmem(total=2147483648, available=2091188224, percent=2.6, used=40071168, free=2091188224, active=41586688, inactive=7983104, buffers=0, cached=16224256, shared=262144, slab=0) ``` These numbers are different enough that I assume this works as advertised. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959898166,`datasette publish cloudrun --cpu X` option, https://github.com/simonw/datasette/issues/1420#issuecomment-892372509,https://api.github.com/repos/simonw/datasette/issues/1420,892372509,IC_kwDOBm6k_c41MIYd,9599,simonw,2021-08-04T05:22:29Z,2021-08-04T05:22:29Z,OWNER,"Testing this manually with: datasette publish cloudrun fixtures.db --memory 8G --cpu 4 \ --service fixtures-over-provisioned-issue-1420 --install datasette-psutil And for comparison: datasette publish cloudrun fixtures.db --service fixtures-default-issue-1420 \ --install datasette-psutil ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959898166,`datasette publish cloudrun --cpu X` option, https://github.com/simonw/datasette/issues/1420#issuecomment-892365639,https://api.github.com/repos/simonw/datasette/issues/1420,892365639,IC_kwDOBm6k_c41MGtH,9599,simonw,2021-08-04T05:05:07Z,2021-08-04T05:05:07Z,OWNER,https://github.com/simonw/datasette/blob/cd8b7bee8fb5c1cdce7c8dbfeb0166011abc72c6/datasette/publish/cloudrun.py#L153-L158,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959898166,`datasette publish cloudrun --cpu X` option, https://github.com/simonw/datasette/pull/1418#issuecomment-891984359,https://api.github.com/repos/simonw/datasette/issues/1418,891984359,IC_kwDOBm6k_c41Kpnn,9599,simonw,2021-08-03T16:21:56Z,2021-08-03T16:21:56Z,OWNER,"Failed with: ``` docs/authentication.rst:63: perfom ==> perform docs/authentication.rst:76: perfom ==> perform docs/changelog.rst:429: repsonse ==> response docs/changelog.rst:503: permissons ==> permissions docs/changelog.rst:717: compatibilty ==> compatibility docs/changelog.rst:1172: browseable ==> browsable docs/deploying.rst:191: similiar ==> similar docs/internals.rst:434: Respons ==> Response, respond docs/internals.rst:440: Respons ==> Response, respond docs/internals.rst:717: tha ==> than, that, the docs/performance.rst:42: databse ==> database docs/plugin_hooks.rst:667: utilites ==> utilities docs/publish.rst:168: countainer ==> container docs/settings.rst:352: inalid ==> invalid docs/sql_queries.rst:406: preceeded ==> preceded, proceeded ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959284434,Spelling corrections plus CI job for codespell, https://github.com/simonw/datasette/issues/1417#issuecomment-891979858,https://api.github.com/repos/simonw/datasette/issues/1417,891979858,IC_kwDOBm6k_c41KohS,9599,simonw,2021-08-03T16:15:30Z,2021-08-03T16:15:30Z,OWNER,"Docs: https://pypi.org/project/codespell/ There's a `codespell --ignore-words=FILE` option for ignoring words. I don't have any that need ignoring yet but I'm going to add that file anyway, that way I can have codespell be a failing test but still provide a way to work around it if it incorrectly flags a correct word.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959278472,Use codespell in CI to spot spelling errors, https://github.com/simonw/sqlite-utils/issues/251#issuecomment-891380382,https://api.github.com/repos/simonw/sqlite-utils/issues/251,891380382,IC_kwDOCGYnMM41IWKe,9599,simonw,2021-08-02T22:39:46Z,2021-08-02T22:39:46Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/stable/cli.html#converting-data-in-columns,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841377702,"""sqlite-utils convert"" command to replace the separate ""sqlite-transform"" tool", https://github.com/simonw/sqlite-utils/issues/298#issuecomment-891359751,https://api.github.com/repos/simonw/sqlite-utils/issues/298,891359751,IC_kwDOCGYnMM41IRIH,9599,simonw,2021-08-02T21:55:16Z,2021-08-02T21:55:16Z,OWNER,"This is a feature already! You can do this: sqlite-utils insert nl-demo.db mytable data.ndjson --nl See https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-newline-delimited-json ","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",951581763,Read lines with JSON object, https://github.com/simonw/sqlite-utils/issues/300#issuecomment-891356906,https://api.github.com/repos/simonw/sqlite-utils/issues/300,891356906,IC_kwDOCGYnMM41IQbq,9599,simonw,2021-08-02T21:49:10Z,2021-08-02T21:49:10Z,OWNER,"I'm nervous to set this by default in the Python library because it's a global setting, so I can't isolate it to just code that is being called by a `sqlite_utils` method. The new `sqlite-utils convert` command from #251 DOES turn it on - I think having it on by default for a CLI tool that uses user-defined functions makes sense: https://github.com/simonw/sqlite-utils/blob/e83aef951bd3e8c179511faddb607239a5fa8682/sqlite_utils/cli.py#L2006 I like the suggestion to include this in the documentation. I'll do that!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",956832836,Returning underlying cause for User Defined Functions , https://github.com/simonw/sqlite-utils/issues/306#issuecomment-891352587,https://api.github.com/repos/simonw/sqlite-utils/issues/306,891352587,IC_kwDOCGYnMM41IPYL,9599,simonw,2021-08-02T21:39:34Z,2021-08-02T21:39:34Z,OWNER,This older TIL would have saved me some time! https://til.simonwillison.net/sphinx/sphinx-ext-extlinks,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",958516743,Configure sphinx.ext.extlinks for issues, https://github.com/simonw/datasette/issues/1227#issuecomment-891352132,https://api.github.com/repos/simonw/datasette/issues/1227,891352132,IC_kwDOBm6k_c41IPRE,9599,simonw,2021-08-02T21:38:39Z,2021-08-02T21:38:39Z,OWNER,Relevant TIL: https://til.simonwillison.net/vscode/vs-code-regular-expressions,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810394616,Configure sphinx.ext.extlinks for issues, https://github.com/simonw/sqlite-utils/issues/306#issuecomment-891352073,https://api.github.com/repos/simonw/sqlite-utils/issues/306,891352073,IC_kwDOCGYnMM41IPQJ,9599,simonw,2021-08-02T21:38:32Z,2021-08-02T21:38:32Z,OWNER,TIL about this: https://til.simonwillison.net/vscode/vs-code-regular-expressions,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",958516743,Configure sphinx.ext.extlinks for issues, https://github.com/simonw/sqlite-utils/issues/304#issuecomment-891257730,https://api.github.com/repos/simonw/sqlite-utils/issues/304,891257730,IC_kwDOCGYnMM41H4OC,9599,simonw,2021-08-02T19:00:00Z,2021-08-02T19:00:00Z,OWNER,"Documented here: - https://sqlite-utils.datasette.io/en/latest/python-api.html#converting-data-in-columns - https://sqlite-utils.datasette.io/en/latest/cli.html#converting-data-in-columns","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957731178,"`table.convert(..., where=)` and `sqlite-utils convert ... --where=`", https://github.com/simonw/sqlite-utils/issues/305#issuecomment-890713185,https://api.github.com/repos/simonw/sqlite-utils/issues/305,890713185,IC_kwDOCGYnMM41FzRh,9599,simonw,2021-08-02T04:53:53Z,2021-08-02T04:53:53Z,OWNER,I'm going to rename `.execute_count()` to `.count_where()` - but I'll keep `.execute_count()` in the codebase since it's a documented API at https://sqlite-utils.datasette.io/en/3.13/python-api.html#count and I don't want to do a major version bump.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957741820,Python: need a way to execute a count with an extra where clause, https://github.com/simonw/sqlite-utils/issues/304#issuecomment-890711924,https://api.github.com/repos/simonw/sqlite-utils/issues/304,890711924,IC_kwDOCGYnMM41Fy90,9599,simonw,2021-08-02T04:50:13Z,2021-08-02T04:50:13Z,OWNER,"For the Python function, the args will be `where=None, where_args=None`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957731178,"`table.convert(..., where=)` and `sqlite-utils convert ... --where=`", https://github.com/simonw/sqlite-utils/issues/304#issuecomment-890704624,https://api.github.com/repos/simonw/sqlite-utils/issues/304,890704624,IC_kwDOCGYnMM41FxLw,9599,simonw,2021-08-02T04:28:42Z,2021-08-02T04:28:42Z,OWNER,For the command-line version this can duplicate the `--param` option to allow named parameters in the where clause: https://github.com/simonw/sqlite-utils/blob/c7e8d72be9fe8fe0811f685a18eebc637662d41b/sqlite_utils/cli.py#L1096-L1102,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957731178,"`table.convert(..., where=)` and `sqlite-utils convert ... --where=`", https://github.com/simonw/sqlite-utils/issues/251#issuecomment-890553783,https://api.github.com/repos/simonw/sqlite-utils/issues/251,890553783,IC_kwDOCGYnMM41FMW3,9599,simonw,2021-08-01T16:59:09Z,2021-08-01T16:59:09Z,OWNER,I'm going with `recipes.jsonsplit()` rather than `recipe.jsonsplit()` because the Python module containing the recipes will be called `recipes`. I'll set up a `r.jsonsplit()` shortcut too as a convenience.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841377702,"""sqlite-utils convert"" command to replace the separate ""sqlite-transform"" tool", https://github.com/simonw/sqlite-utils/issues/251#issuecomment-890552827,https://api.github.com/repos/simonw/sqlite-utils/issues/251,890552827,IC_kwDOCGYnMM41FMH7,9599,simonw,2021-08-01T16:52:00Z,2021-08-01T16:52:00Z,OWNER,I'll finish the work on this in a PR.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841377702,"""sqlite-utils convert"" command to replace the separate ""sqlite-transform"" tool", https://github.com/simonw/sqlite-utils/issues/302#issuecomment-890548009,https://api.github.com/repos/simonw/sqlite-utils/issues/302,890548009,IC_kwDOCGYnMM41FK8p,9599,simonw,2021-08-01T16:18:13Z,2021-08-01T16:18:13Z,OWNER,"Basic API design: db[table].convert(""headline"", lambda v: v.upper()) You can pass a list of columns instead of a single game column string to apply it to multiple columns.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957529248,Python library version of `sqlite-utils convert`, https://github.com/simonw/sqlite-utils/issues/251#issuecomment-890448623,https://api.github.com/repos/simonw/sqlite-utils/issues/251,890448623,IC_kwDOCGYnMM41Eyrv,9599,simonw,2021-08-01T04:33:30Z,2021-08-01T04:33:30Z,OWNER,"I've started an implementation in the `convert` branch - no documentation yet, and I've not implemented the recipes.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841377702,"""sqlite-utils convert"" command to replace the separate ""sqlite-transform"" tool", https://github.com/simonw/sqlite-utils/issues/251#issuecomment-890448119,https://api.github.com/repos/simonw/sqlite-utils/issues/251,890448119,IC_kwDOCGYnMM41Eyj3,9599,simonw,2021-08-01T04:28:05Z,2021-08-01T04:30:28Z,OWNER,"In which case I think `--code` should be a positional argument instead: ``` sqlite-utils convert mydb.db mytable col 'recipe.parsedatetime(value, dayfirst=True)' sqlite-utils convert mydb.db mytable col 'recipe.jsonsplit(value, delimiter="":"")' sqlite-utils convert mydb.db mytable col 'recipe.jsonsplit(value, delimiter="":"")' sqlite-utils convert mydb.db mytable col '{""lower"": value.lower(), ""upper"": value.upper()}' --multi ``` One problem with this: we already accept one or more columns. I think that's OK though since the code is now a required argument, so it means we have to treat everything between the table and the final code argument as a column.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841377702,"""sqlite-utils convert"" command to replace the separate ""sqlite-transform"" tool", https://github.com/simonw/sqlite-utils/issues/251#issuecomment-890447102,https://api.github.com/repos/simonw/sqlite-utils/issues/251,890447102,IC_kwDOCGYnMM41EyT-,9599,simonw,2021-08-01T04:20:18Z,2021-08-01T04:29:26Z,OWNER,"I could stick them in a `recipe` namespace so you do this: ``` sqlite-utils convert mydb.db mytable col --code 'recipe.parsedatetime(value, dayfirst=True)' sqlite-utils convert mydb.db mytable col --code 'recipe.jsonsplit(value, delimiter="":"")' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841377702,"""sqlite-utils convert"" command to replace the separate ""sqlite-transform"" tool", https://github.com/simonw/sqlite-utils/issues/251#issuecomment-890448190,https://api.github.com/repos/simonw/sqlite-utils/issues/251,890448190,IC_kwDOCGYnMM41Eyk-,9599,simonw,2021-08-01T04:28:49Z,2021-08-01T04:28:49Z,OWNER,"Would make sense to accept code from standard input too: echo 'value.upper()' | sqlite-utils convert my.db mytable col -","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841377702,"""sqlite-utils convert"" command to replace the separate ""sqlite-transform"" tool", https://github.com/simonw/sqlite-utils/issues/251#issuecomment-890446808,https://api.github.com/repos/simonw/sqlite-utils/issues/251,890446808,IC_kwDOCGYnMM41EyPY,9599,simonw,2021-08-01T04:18:18Z,2021-08-01T04:28:18Z,OWNER,"Or.... how about making the `parsedate()` and `parsedatetime()` and `jsonsplit()` functions available within the namespace that is configured for the `--code` block? Then you could do something like this: ``` sqlite-utils convert mydb.db mytable col --code 'parsedatetime(value, dayfirst=True)' sqlite-utils convert mydb.db mytable col --code 'jsonsplit(value, delimiter="":"")' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841377702,"""sqlite-utils convert"" command to replace the separate ""sqlite-transform"" tool", https://github.com/simonw/sqlite-utils/issues/251#issuecomment-890446943,https://api.github.com/repos/simonw/sqlite-utils/issues/251,890446943,IC_kwDOCGYnMM41EyRf,9599,simonw,2021-08-01T04:19:09Z,2021-08-01T04:19:09Z,OWNER,"That's a pretty neat fix, though it's a bit more challenging on the documentation front - maybe the help text for `sqlite-utils convert --help` gets a fair bit longer?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841377702,"""sqlite-utils convert"" command to replace the separate ""sqlite-transform"" tool", https://github.com/simonw/sqlite-utils/issues/251#issuecomment-890446506,https://api.github.com/repos/simonw/sqlite-utils/issues/251,890446506,IC_kwDOCGYnMM41EyKq,9599,simonw,2021-08-01T04:16:36Z,2021-08-01T04:16:36Z,OWNER,"Back to the design board then. One way to handle this would be the long-form: ``` sqlite-utils convert jsonsplit mydb.db mytable mycolumn sqlite-utils convert parsedatetime mydb.db mytable mycolumn sqlite-utils convert parsedate mydb.db mytable mycolumn sqlite-utils convert lambda mydb.db mytable mycolumn --code='str(value).upper()' ``` I like the idea that `lambda` is the default action, but in this form it's required that the second argument (the word after `convert`) be the name of the recipe that is being applied to avoid any potential confusion with the database filename. An ugly solution would be to make all four of those options available on `sqlite-utils convert` - and return an error if you try and use one of those without specifying the accompanying recipe. That's a bit gross though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841377702,"""sqlite-utils convert"" command to replace the separate ""sqlite-transform"" tool", https://github.com/simonw/sqlite-utils/issues/251#issuecomment-890446166,https://api.github.com/repos/simonw/sqlite-utils/issues/251,890446166,IC_kwDOCGYnMM41EyFW,9599,simonw,2021-08-01T04:14:26Z,2021-08-01T04:14:26Z,OWNER,"Problem with the `-r/--recipe` idea: the `parsedate` and `parsedatetime` and `jsonsplit` recipes in the current `sqlite-transform` tool all take additional options. For `sqlite-transform parsedate` and `parsedatetime`: ```python @click.option( ""--dayfirst"", is_flag=True, help=""Assume day comes first in ambiguous dates, e.g. 03/04/05"", ) @click.option( ""--yearfirst"", is_flag=True, help=""Assume year comes first in ambiguous dates, e.g. 03/04/05"", ) ``` For `jsonsplit`: ```python @click.option(""--delimiter"", default="","", help=""Delimiter to split on"") @click.option( ""--type"", type=click.Choice((""int"", ""float"")), help=""Type to use for values - int or float (defaults to string)"", ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841377702,"""sqlite-utils convert"" command to replace the separate ""sqlite-transform"" tool", https://github.com/simonw/sqlite-utils/issues/251#issuecomment-890443079,https://api.github.com/repos/simonw/sqlite-utils/issues/251,890443079,IC_kwDOCGYnMM41ExVH,9599,simonw,2021-08-01T03:46:43Z,2021-08-01T03:46:43Z,OWNER,"Note that there's already a concept of `conversions` which might be confused with `convert`? https://sqlite-utils.datasette.io/en/stable/python-api.html#converting-column-values-using-sql-functions ```python db[""example""].insert({ ""name"": ""The Bigfoot Discovery Museum"" }, conversions={""name"": ""upper(?)""}) ``` I think that's OK though - that's a Python library feature, `sqlite-utils convert` is a CLI thing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841377702,"""sqlite-utils convert"" command to replace the separate ""sqlite-transform"" tool", https://github.com/simonw/datasette/issues/1411#issuecomment-890441844,https://api.github.com/repos/simonw/datasette/issues/1411,890441844,IC_kwDOBm6k_c41ExB0,9599,simonw,2021-08-01T03:27:30Z,2021-08-01T03:27:30Z,OWNER,Confirmed: https://latest.datasette.io/fixtures/neighborhood_search?text=cork&_hide_sql=1 no longer exhibits the bug.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957345476,Canned query ?sql= is pointlessly echoed in query string starting from hidden mode, https://github.com/simonw/datasette/issues/1409#issuecomment-890400425,https://api.github.com/repos/simonw/datasette/issues/1409,890400425,IC_kwDOBm6k_c41Em6p,9599,simonw,2021-07-31T20:25:16Z,2021-07-31T20:26:25Z,OWNER,"If I was prone to over-thinking (which I am) I'd note that `allow_facet` and `allow_download` and `allow_csv_stream` are all settings that do NOT have an equivalent in the newer permissions system, which is itself a little weird and inconsistent. So maybe there's a future task where I introduce those as both permissions and metadata `""allow_x""` blocks, then rename the settings themselves to be called `default_allow_facet` and `default_allow_download` and `default_allow_csv_stream`. If I was going to do that I should get it in before Datasette 1.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957310278,`default_allow_sql` setting (a re-imagining of the old `allow_sql` setting), https://github.com/simonw/datasette/issues/1409#issuecomment-890400121,https://api.github.com/repos/simonw/datasette/issues/1409,890400121,IC_kwDOBm6k_c41Em15,9599,simonw,2021-07-31T20:22:21Z,2021-07-31T20:23:34Z,OWNER,"I think `default_allow_sql` is more consistent with the current naming conventions, because both `allow` and `default` are used as prefixes at the moment but neither of them are ever used as a suffix. Plus `default_allow_sql off` makes sense to me but `allow_default_sql off` does not - what is ""default SQL""?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957310278,`default_allow_sql` setting (a re-imagining of the old `allow_sql` setting), https://github.com/simonw/datasette/issues/1409#issuecomment-890400059,https://api.github.com/repos/simonw/datasette/issues/1409,890400059,IC_kwDOBm6k_c41Em07,9599,simonw,2021-07-31T20:21:51Z,2021-07-31T20:21:51Z,OWNER,"One of these two options: - `--setting default_allow_sql off` - `--setting allow_sql_default off` Existing settings from https://docs.datasette.io/en/0.58.1/settings.html with similar names that I need to be consistent with: - `default_page_size` - `allow_facet` - `default_facet_size` - `allow_download` - `default_cache_ttl` - `default_cache_ttl_hashed` - `allow_csv_stream` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957310278,`default_allow_sql` setting (a re-imagining of the old `allow_sql` setting), https://github.com/simonw/datasette/issues/1409#issuecomment-890399806,https://api.github.com/repos/simonw/datasette/issues/1409,890399806,IC_kwDOBm6k_c41Emw-,9599,simonw,2021-07-31T20:18:46Z,2021-07-31T20:18:46Z,OWNER,"My rationale for removing it: https://github.com/simonw/datasette/issues/813#issuecomment-640916290 > Naming problem: Datasette already has a config option with this name: > > $ datasette serve data.db --config allow_sql:1 > > https://datasette.readthedocs.io/en/stable/config.html#allow-sql > > It's confusing to have two things called `allow_sql` that do slightly different things. > > I could retire the `--config allow_sql:0` option entirely, since the new `metadata.json` mechanism can be used to achieve the exact same thing. > > I'm going to do that. This is true. The `""allow_sql""` permissions block in `metadata.json` does indeed have a name that is easily confused with `--setting allow_sql off`. So I definitely need to pick a different name from the setting. `--setting default_allow_sql off` is a good option here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957310278,`default_allow_sql` setting (a re-imagining of the old `allow_sql` setting), https://github.com/simonw/datasette/issues/1409#issuecomment-890397753,https://api.github.com/repos/simonw/datasette/issues/1409,890397753,IC_kwDOBm6k_c41EmQ5,9599,simonw,2021-07-31T19:57:56Z,2021-07-31T19:57:56Z,OWNER,"I think the correct solution is for the default permissions logic to take the `allow_sql` setting into account, and to return `False` if that setting is set to `off` AND the current actor fails the `actor_matches_allow` checks.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957310278,`default_allow_sql` setting (a re-imagining of the old `allow_sql` setting), https://github.com/simonw/datasette/issues/1409#issuecomment-890397652,https://api.github.com/repos/simonw/datasette/issues/1409,890397652,IC_kwDOBm6k_c41EmPU,9599,simonw,2021-07-31T19:56:48Z,2021-07-31T19:56:48Z,OWNER,"The other option would be to use the setting to pick the `default=` argument when calling `self.ds.permission_allowed( request.actor, ""execute-sql"", resource=database, default=True)`. The problem with that is that there are actually a few different places which perform that check, so changing all of them raises the risk of missing one in the future: https://github.com/simonw/datasette/blob/a6c8e7fa4cffdeff84e9e755dcff4788fd6154b8/datasette/views/table.py#L436-L444 https://github.com/simonw/datasette/blob/a6c8e7fa4cffdeff84e9e755dcff4788fd6154b8/datasette/views/table.py#L964-L966 https://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/views/database.py#L220-L221 https://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/views/database.py#L343-L345 https://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/views/database.py#L134-L136 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957310278,`default_allow_sql` setting (a re-imagining of the old `allow_sql` setting), https://github.com/simonw/datasette/issues/1409#issuecomment-890397261,https://api.github.com/repos/simonw/datasette/issues/1409,890397261,IC_kwDOBm6k_c41EmJN,9599,simonw,2021-07-31T19:52:25Z,2021-07-31T19:52:25Z,OWNER,I think I can make this modification by teaching the default permissions code here to take the `allow_sql` setting into account: https://github.com/simonw/datasette/blob/ff253f5242e4b0b5d85d29d38b8461feb5ea997a/datasette/default_permissions.py#L38-L45,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957310278,`default_allow_sql` setting (a re-imagining of the old `allow_sql` setting), https://github.com/simonw/datasette/issues/1409#issuecomment-890397169,https://api.github.com/repos/simonw/datasette/issues/1409,890397169,IC_kwDOBm6k_c41EmHx,9599,simonw,2021-07-31T19:51:35Z,2021-07-31T19:51:35Z,OWNER,I'm going to stick with `--setting allow_sql off`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957310278,`default_allow_sql` setting (a re-imagining of the old `allow_sql` setting), https://github.com/simonw/datasette/issues/1409#issuecomment-890397124,https://api.github.com/repos/simonw/datasette/issues/1409,890397124,IC_kwDOBm6k_c41EmHE,9599,simonw,2021-07-31T19:51:10Z,2021-07-31T19:51:10Z,OWNER,"I think I may like `disable_sql` better. Some options: - `--setting allow_sql off` (consistent with `allow_facet` and `allow_download` and `allow_csv_stream` - all which default to `on` already) - `--setting disable_sql on` - `--setting disable_custom_sql on` The existence of three `allow_*` settings does make a strong argument for staying consistent with that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957310278,`default_allow_sql` setting (a re-imagining of the old `allow_sql` setting), https://github.com/simonw/datasette/issues/1408#issuecomment-890390845,https://api.github.com/repos/simonw/datasette/issues/1408,890390845,IC_kwDOBm6k_c41Ekk9,9599,simonw,2021-07-31T19:00:32Z,2021-07-31T19:00:32Z,OWNER,"When I revisit this I can also look at dropping the `@pytest.mark.serial` hack, and maybe the `restore_working_directory()` fixture hack too: https://github.com/simonw/datasette/blob/ff253f5242e4b0b5d85d29d38b8461feb5ea997a/pytest.ini#L9-L10 https://github.com/simonw/datasette/blob/ff253f5242e4b0b5d85d29d38b8461feb5ea997a/tests/conftest.py#L62-L75","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957302085,"Review places in codebase that use os.chdir(), in particularly relating to tests", https://github.com/simonw/datasette/issues/1408#issuecomment-890390495,https://api.github.com/repos/simonw/datasette/issues/1408,890390495,IC_kwDOBm6k_c41Ekff,9599,simonw,2021-07-31T18:57:39Z,2021-07-31T18:57:39Z,OWNER,Opening this issue as an optional follow-up to the work I did in #1406.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957302085,"Review places in codebase that use os.chdir(), in particularly relating to tests", https://github.com/simonw/datasette/issues/1406#issuecomment-890390342,https://api.github.com/repos/simonw/datasette/issues/1406,890390342,IC_kwDOBm6k_c41EkdG,9599,simonw,2021-07-31T18:56:35Z,2021-07-31T18:56:35Z,OWNER,"But... I've lost enough time to this already, and removing `runner.isolated_filesystem()` has the tests passing again. So I'm not going to work on this any more.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",956303470,Tests failing with FileNotFoundError in runner.isolated_filesystem, https://github.com/simonw/datasette/issues/1406#issuecomment-890390198,https://api.github.com/repos/simonw/datasette/issues/1406,890390198,IC_kwDOBm6k_c41Eka2,9599,simonw,2021-07-31T18:55:33Z,2021-07-31T18:55:33Z,OWNER,"To clarify: the core problem here is that an error is thrown any time you call `os.getcwd()` but the directory you are currently in has been deleted. `runner.isolated_filesystem()` assumes that the current directory in has not been deleted. But the various temporary directory utilities in `pytest` work by creating directories and then deleting them. Maybe there's a larger problem here that I play a bit fast and loose with `os.chdir()` in both the test suite and in various lines of code in Datasette itself (in particular in the publish commands)?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",956303470,Tests failing with FileNotFoundError in runner.isolated_filesystem, https://github.com/simonw/datasette/issues/1407#issuecomment-890388656,https://api.github.com/repos/simonw/datasette/issues/1407,890388656,IC_kwDOBm6k_c41EkCw,9599,simonw,2021-07-31T18:42:41Z,2021-07-31T18:42:41Z,OWNER,I'll try `tempfile.gettempdir()` - on macOS it returns something like `'/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T'` which is still long but hopefully not too long.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957298475,OSError: AF_UNIX path too long in ds_unix_domain_socket_server, https://github.com/simonw/datasette/issues/1407#issuecomment-890388200,https://api.github.com/repos/simonw/datasette/issues/1407,890388200,IC_kwDOBm6k_c41Ej7o,9599,simonw,2021-07-31T18:38:41Z,2021-07-31T18:38:41Z,OWNER,"The `path` variable there looked like this: `/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-696/popen-gw0/uds0/datasette.sock` I think what's happening here is that `pytest-xdist` causes `tmp_path_factory.mktemp(""uds"")` to create significantly longer paths, which in this case is breaking some limit. So for this code to work with `pytest-xdist` I need to make sure the random path to `datasette.sock` is shorter.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957298475,OSError: AF_UNIX path too long in ds_unix_domain_socket_server, https://github.com/simonw/datasette/issues/1406#issuecomment-890259755,https://api.github.com/repos/simonw/datasette/issues/1406,890259755,IC_kwDOBm6k_c41EEkr,9599,simonw,2021-07-31T00:04:54Z,2021-07-31T00:04:54Z,OWNER,"STILL failing. I'm going to try removing all instances of `isolated_filesystem()` in favour of a different pattern using pytest temporary files, then see if I can get that to work without the serial hack. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",956303470,Tests failing with FileNotFoundError in runner.isolated_filesystem, https://github.com/simonw/datasette/issues/1406#issuecomment-889555977,https://api.github.com/repos/simonw/datasette/issues/1406,889555977,IC_kwDOBm6k_c41BYwJ,9599,simonw,2021-07-30T01:06:57Z,2021-07-30T01:06:57Z,OWNER,"Looking at the source code in Click for `isolated_filesystem()`: https://github.com/pallets/click/blob/9da166957f5848b641231d485467f6140bca2bc0/src/click/testing.py#L450-L468 ```python @contextlib.contextmanager def isolated_filesystem( self, temp_dir: t.Optional[t.Union[str, os.PathLike]] = None ) -> t.Iterator[str]: """"""A context manager that creates a temporary directory and changes the current working directory to it. This isolates tests that affect the contents of the CWD to prevent them from interfering with each other. :param temp_dir: Create the temporary directory under this directory. If given, the created directory is not removed when exiting. .. versionchanged:: 8.0 Added the ``temp_dir`` parameter. """""" cwd = os.getcwd() t = tempfile.mkdtemp(dir=temp_dir) os.chdir(t) ``` How about if I pass in that optional `temp_dir` as a temp directory created using the `pytest-xdist` aware pytest mechanisms: https://docs.pytest.org/en/6.2.x/tmpdir.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",956303470,Tests failing with FileNotFoundError in runner.isolated_filesystem, https://github.com/simonw/datasette/issues/1406#issuecomment-889553052,https://api.github.com/repos/simonw/datasette/issues/1406,889553052,IC_kwDOBm6k_c41BYCc,9599,simonw,2021-07-30T00:58:43Z,2021-07-30T00:58:43Z,OWNER,Tests are still failing in the job that calculates coverage.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",956303470,Tests failing with FileNotFoundError in runner.isolated_filesystem, https://github.com/simonw/datasette/issues/1406#issuecomment-889550391,https://api.github.com/repos/simonw/datasette/issues/1406,889550391,IC_kwDOBm6k_c41BXY3,9599,simonw,2021-07-30T00:49:31Z,2021-07-30T00:49:31Z,OWNER,That fixed it. My hunch is that Click's `runner.isolated_filesystem()` mechanism doesn't play well with `pytest-xdist`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",956303470,Tests failing with FileNotFoundError in runner.isolated_filesystem, https://github.com/simonw/datasette/issues/1406#issuecomment-889548536,https://api.github.com/repos/simonw/datasette/issues/1406,889548536,IC_kwDOBm6k_c41BW74,9599,simonw,2021-07-30T00:43:47Z,2021-07-30T00:43:47Z,OWNER,"Still couldn't replicate on my laptop. On a hunch, I'm going to add `@pytest.mark.serial` to every test that uses `runner.isolated_filesystem()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",956303470,Tests failing with FileNotFoundError in runner.isolated_filesystem, https://github.com/simonw/datasette/issues/1406#issuecomment-889547142,https://api.github.com/repos/simonw/datasette/issues/1406,889547142,IC_kwDOBm6k_c41BWmG,9599,simonw,2021-07-30T00:39:49Z,2021-07-30T00:39:49Z,OWNER,It happens in CI but not on my laptop. I think I need to run the tests on my laptop like this: https://github.com/simonw/datasette/blob/121e10c29c5b412fddf0326939f1fe46c3ad9d4a/.github/workflows/test.yml#L27-L30,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",956303470,Tests failing with FileNotFoundError in runner.isolated_filesystem, https://github.com/simonw/datasette/issues/1241#issuecomment-889539227,https://api.github.com/repos/simonw/datasette/issues/1241,889539227,IC_kwDOBm6k_c41BUqb,9599,simonw,2021-07-30T00:15:26Z,2021-07-30T00:15:26Z,OWNER,"One possible treatment: ```html

{% if query.sql and allow_execute_sql %} View and edit SQL {% endif %} Copy and share link

```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",814595021,Share button for copying current URL, https://github.com/simonw/datasette/issues/1405#issuecomment-889525741,https://api.github.com/repos/simonw/datasette/issues/1405,889525741,IC_kwDOBm6k_c41BRXt,9599,simonw,2021-07-29T23:33:30Z,2021-07-29T23:33:30Z,OWNER,New documentation section for `datasette.utils` is here: https://docs.datasette.io/en/latest/internals.html#the-datasette-utils-module,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",955316250,utils.parse_metadata() should be a documented internal function, https://github.com/simonw/datasette/issues/1405#issuecomment-888694261,https://api.github.com/repos/simonw/datasette/issues/1405,888694261,IC_kwDOBm6k_c40-GX1,9599,simonw,2021-07-28T23:52:21Z,2021-07-28T23:52:21Z,OWNER,Document that it can raise a `BadMetadataError` exception.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",955316250,utils.parse_metadata() should be a documented internal function, https://github.com/simonw/datasette/issues/1405#issuecomment-888694144,https://api.github.com/repos/simonw/datasette/issues/1405,888694144,IC_kwDOBm6k_c40-GWA,9599,simonw,2021-07-28T23:51:59Z,2021-07-28T23:51:59Z,OWNER,https://github.com/simonw/datasette/blob/eccfeb0871dd4bc27870faf64f80ac68e5b6bc0d/datasette/utils/__init__.py#L918-L926,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",955316250,utils.parse_metadata() should be a documented internal function, https://github.com/simonw/datasette/issues/1404#issuecomment-887095569,https://api.github.com/repos/simonw/datasette/issues/1404,887095569,IC_kwDOBm6k_c404AER,9599,simonw,2021-07-26T23:27:07Z,2021-07-26T23:27:07Z,OWNER,Updated documentation: https://github.com/simonw/datasette/blob/eccfeb0871dd4bc27870faf64f80ac68e5b6bc0d/docs/plugin_hooks.rst#register-routes-datasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",953352015,`register_routes()` hook should take `datasette` argument, https://github.com/simonw/datasette/issues/1402#issuecomment-886969541,https://api.github.com/repos/simonw/datasette/issues/1402,886969541,IC_kwDOBm6k_c403hTF,9599,simonw,2021-07-26T19:31:40Z,2021-07-26T19:31:40Z,OWNER,"Datasette could do a pretty good job of this by default, using `twitter:card` and `og:url` tags - like on https://til.simonwillison.net/jq/extracting-objects-recursively I could also provide a mechanism to customize these - in particular to add images of some sort. It feels like something that should tie in to the metadata mechanism.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",951185411,feature request: social meta tags, https://github.com/simonw/datasette/issues/1402#issuecomment-886968648,https://api.github.com/repos/simonw/datasette/issues/1402,886968648,IC_kwDOBm6k_c403hFI,9599,simonw,2021-07-26T19:30:14Z,2021-07-26T19:30:14Z,OWNER,"I really like this idea. I was thinking it might make a good plugin, but there's not a great mechanism for plugins to inject extra `` content at the moment - plus this actually feels like a reasonable feature for Datasette core itself.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",951185411,feature request: social meta tags, https://github.com/simonw/sqlite-utils/issues/251#issuecomment-886122696,https://api.github.com/repos/simonw/sqlite-utils/issues/251,886122696,IC_kwDOCGYnMM400SjI,9599,simonw,2021-07-24T23:21:32Z,2021-07-24T23:21:32Z,OWNER,"> ``` > sqlite-utils convert jsonsplit mydb.db mytable mycolumn > sqlite-utils convert parsedatetime mydb.db mytable mycolumn > sqlite-utils convert parsedate mydb.db mytable mycolumn > sqlite-utils convert lambda mydb.db mytable mycolumn --code='str(value).upper()' > ``` This is a bit verbose - and having added `--multi` and `--output` the `lambda` command keeps getting more and more flexible compared to the others. New idea: ditch the sub-sub-commands and move the `jsonsplit` and `parsedate` recipes to be options of `convert` - maybe like this: sqlite-utils convert my.db mytable col1 --jsonsplit or: sqlite-utils convert my.db mytable col1 --recipe jsonsplit or `-r jsonsplit` for short. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841377702,"""sqlite-utils convert"" command to replace the separate ""sqlite-transform"" tool", https://github.com/simonw/sqlite-utils/issues/299#issuecomment-886117120,https://api.github.com/repos/simonw/sqlite-utils/issues/299,886117120,IC_kwDOCGYnMM400RMA,9599,simonw,2021-07-24T22:12:01Z,2021-07-24T22:12:01Z,OWNER,Documentation here: https://sqlite-utils.datasette.io/en/latest/cli.html#showing-the-schema,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",952154468,Ability to see just specific table schemas with `sqlite-utils schema`, https://github.com/simonw/datasette/issues/123#issuecomment-882138084,https://api.github.com/repos/simonw/datasette/issues/123,882138084,IC_kwDOBm6k_c40lFvk,9599,simonw,2021-07-19T00:04:31Z,2021-07-19T00:04:31Z,OWNER,"I've been thinking more about this one today too. An extension of this (touched on in #417, Datasette Library) would be to support pointing Datasette at a directory and having it automatically load any CSV files it finds anywhere in that folder or its descendants - either loading them fully, or providing a UI that allows users to select a file to open it in Datasette. For larger files I think the right thing to do is import them into an on-disk SQLite database, which is limited only by available disk space. For smaller files loading them into an in-memory database should work fine.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",275125561,Datasette serve should accept paths/URLs to CSVs and other file formats, https://github.com/simonw/sqlite-utils/issues/297#issuecomment-882052852,https://api.github.com/repos/simonw/sqlite-utils/issues/297,882052852,IC_kwDOCGYnMM40kw70,9599,simonw,2021-07-18T12:59:20Z,2021-07-18T12:59:20Z,OWNER,I'm not too worried about `sqlite-utils memory` because if your data is large enough that you can benefit from this optimization you probably should use a real file as opposed to a disposable memory database when analyzing it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944846776,Option for importing CSV data using the SQLite .import mechanism, https://github.com/simonw/datasette/issues/1199#issuecomment-881932880,https://api.github.com/repos/simonw/datasette/issues/1199,881932880,IC_kwDOBm6k_c40kTpQ,9599,simonw,2021-07-17T17:39:17Z,2021-07-17T17:39:17Z,OWNER,"I asked about optimizing performance on the SQLite forum and this came up as a suggestion: https://sqlite.org/forum/forumpost/9a6b9ae8e2048c8b?t=c I can start by trying this: PRAGMA mmap_size=268435456;","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",792652391,Experiment with PRAGMA mmap_size=N, https://github.com/simonw/datasette/issues/1396#issuecomment-881686662,https://api.github.com/repos/simonw/datasette/issues/1396,881686662,IC_kwDOBm6k_c40jXiG,9599,simonw,2021-07-16T20:02:44Z,2021-07-16T20:02:44Z,OWNER,Confirmed fixed: 0.58.1 was successfully published to Docker Hub in https://github.com/simonw/datasette/runs/3089447346 and the `latest` tag on https://hub.docker.com/r/datasetteproject/datasette/tags was updated.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944903881,"""invalid reference format"" publishing Docker image", https://github.com/simonw/datasette/issues/1231#issuecomment-881677620,https://api.github.com/repos/simonw/datasette/issues/1231,881677620,IC_kwDOBm6k_c40jVU0,9599,simonw,2021-07-16T19:44:12Z,2021-07-16T19:44:12Z,OWNER,"That fixed the race condition in the `datasette-graphql` tests, which is the only place that I've been able to successfully replicate this. I'm going to land this change.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,Race condition errors in new refresh_schemas() mechanism, https://github.com/simonw/datasette/issues/1231#issuecomment-881674857,https://api.github.com/repos/simonw/datasette/issues/1231,881674857,IC_kwDOBm6k_c40jUpp,9599,simonw,2021-07-16T19:38:39Z,2021-07-16T19:38:39Z,OWNER,I can't replicate the race condition locally with or without this patch. I'm going to push the commit and then test the CI run from `datasette-graphql` that was failing against it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,Race condition errors in new refresh_schemas() mechanism, https://github.com/simonw/datasette/issues/1231#issuecomment-881671706,https://api.github.com/repos/simonw/datasette/issues/1231,881671706,IC_kwDOBm6k_c40jT4a,9599,simonw,2021-07-16T19:32:05Z,2021-07-16T19:32:05Z,OWNER,The test suite passes with that change.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,Race condition errors in new refresh_schemas() mechanism, https://github.com/simonw/datasette/issues/1231#issuecomment-881668759,https://api.github.com/repos/simonw/datasette/issues/1231,881668759,IC_kwDOBm6k_c40jTKX,9599,simonw,2021-07-16T19:27:46Z,2021-07-16T19:27:46Z,OWNER,"Second attempt at this: ```diff diff --git a/datasette/app.py b/datasette/app.py index 5976d8b..5f348cb 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -224,6 +224,7 @@ class Datasette: self.inspect_data = inspect_data self.immutables = set(immutables or []) self.databases = collections.OrderedDict() + self._refresh_schemas_lock = asyncio.Lock() self.crossdb = crossdb if memory or crossdb or not self.files: self.add_database(Database(self, is_memory=True), name=""_memory"") @@ -332,6 +333,12 @@ class Datasette: self.client = DatasetteClient(self) async def refresh_schemas(self): + if self._refresh_schemas_lock.locked(): + return + async with self._refresh_schemas_lock: + await self._refresh_schemas() + + async def _refresh_schemas(self): internal_db = self.databases[""_internal""] if not self.internal_db_created: await init_internal_db(internal_db) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,Race condition errors in new refresh_schemas() mechanism, https://github.com/simonw/datasette/issues/1231#issuecomment-881665383,https://api.github.com/repos/simonw/datasette/issues/1231,881665383,IC_kwDOBm6k_c40jSVn,9599,simonw,2021-07-16T19:21:35Z,2021-07-16T19:21:35Z,OWNER,"https://stackoverflow.com/a/25799871/6083 has a good example of using `asyncio.Lock()`: ```python stuff_lock = asyncio.Lock() async def get_stuff(url): async with stuff_lock: if url in cache: return cache[url] stuff = await aiohttp.request('GET', url) cache[url] = stuff return stuff ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,Race condition errors in new refresh_schemas() mechanism, https://github.com/simonw/datasette/issues/1231#issuecomment-881664408,https://api.github.com/repos/simonw/datasette/issues/1231,881664408,IC_kwDOBm6k_c40jSGY,9599,simonw,2021-07-16T19:19:35Z,2021-07-16T19:19:35Z,OWNER,"The only place that calls `refresh_schemas()` is here: https://github.com/simonw/datasette/blob/dd5ee8e66882c94343cd3f71920878c6cfd0da41/datasette/views/base.py#L120-L124 Ideally only one call to `refresh_schemas()` would be running at any one time.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,Race condition errors in new refresh_schemas() mechanism, https://github.com/simonw/datasette/issues/1231#issuecomment-881663968,https://api.github.com/repos/simonw/datasette/issues/1231,881663968,IC_kwDOBm6k_c40jR_g,9599,simonw,2021-07-16T19:18:42Z,2021-07-16T19:18:42Z,OWNER,The race condition happens inside this method - initially with the call to `await init_internal_db()`: https://github.com/simonw/datasette/blob/dd5ee8e66882c94343cd3f71920878c6cfd0da41/datasette/app.py#L334-L359,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,Race condition errors in new refresh_schemas() mechanism, https://github.com/simonw/datasette/issues/1231#issuecomment-881204782,https://api.github.com/repos/simonw/datasette/issues/1231,881204782,IC_kwDOBm6k_c40hh4u,9599,simonw,2021-07-16T06:14:12Z,2021-07-16T06:14:12Z,OWNER,"Here's the traceback I got from `datasette-graphql` (annoyingly only running the tests in GitHub Actions CI - I've not been able to replicate on my laptop yet): ``` tests/test_utils.py . [100%] =================================== FAILURES =================================== _________________________ test_graphql_examples[path0] _________________________ ds = path = PosixPath('/home/runner/work/datasette-graphql/datasette-graphql/examples/filters.md') @pytest.mark.asyncio @pytest.mark.parametrize( ""path"", (pathlib.Path(__file__).parent.parent / ""examples"").glob(""*.md"") ) async def test_graphql_examples(ds, path): content = path.read_text() query = graphql_re.search(content)[1] try: variables = variables_re.search(content)[1] except TypeError: variables = ""{}"" expected = json.loads(json_re.search(content)[1]) response = await ds.client.post( ""/graphql"", json={ ""query"": query, ""variables"": json.loads(variables), }, ) > assert response.status_code == 200, response.json() E AssertionError: {'data': {'repos_arraycontains': None, 'users_contains': None, 'users_date': None, 'users_endswith': None, ...}, 'erro..."", 'path': ['users_gt']}, {'locations': [{'column': 5, 'line': 34}], 'message': ""'rows'"", 'path': ['users_gte']}, ...]} E assert 500 == 200 E + where 500 = .status_code tests/test_graphql.py:142: AssertionError ----------------------------- Captured stderr call ----------------------------- table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists Traceback (most recent call last): File ""/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/app.py"", line 1171, in route_path response = await view(request, send) File ""/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/views/base.py"", line 151, in view request, **request.scope[""url_route""][""kwargs""] File ""/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/views/base.py"", line 123, in dispatch_request await self.ds.refresh_schemas() File ""/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/app.py"", line 338, in refresh_schemas await init_internal_db(internal_db) File ""/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/utils/internal_db.py"", line 16, in init_internal_db block=True, File ""/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/database.py"", line 102, in execute_write return await self.execute_write_fn(_inner, block=block) File ""/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/database.py"", line 118, in execute_write_fn raise result File ""/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/database.py"", line 139, in _execute_writes result = task.fn(conn) File ""/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/database.py"", line 100, in _inner return conn.execute(sql, params or []) sqlite3.OperationalError: table databases already exists ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,Race condition errors in new refresh_schemas() mechanism, https://github.com/simonw/datasette/issues/1231#issuecomment-881204343,https://api.github.com/repos/simonw/datasette/issues/1231,881204343,IC_kwDOBm6k_c40hhx3,9599,simonw,2021-07-16T06:13:11Z,2021-07-16T06:13:11Z,OWNER,This just broke the `datasette-graphql` test suite: https://github.com/simonw/datasette-graphql/issues/77 - I need to figure out a solution here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,Race condition errors in new refresh_schemas() mechanism, https://github.com/simonw/datasette/issues/1394#issuecomment-881129149,https://api.github.com/repos/simonw/datasette/issues/1394,881129149,IC_kwDOBm6k_c40hPa9,9599,simonw,2021-07-16T02:23:32Z,2021-07-16T02:23:32Z,OWNER,Wrote about this in the annotated release notes for 0.58: https://simonwillison.net/2021/Jul/16/datasette-058/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944870799,Big performance boost on faceting: skip the inner order by, https://github.com/simonw/datasette/issues/759#issuecomment-881125124,https://api.github.com/repos/simonw/datasette/issues/759,881125124,IC_kwDOBm6k_c40hOcE,9599,simonw,2021-07-16T02:11:48Z,2021-07-16T02:11:54Z,OWNER,"I added `""searchmode"": ""raw""` as a supported option for table metadata in #1389 and released that in Datasette 0.58.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612673948,fts search on a column doesn't work anymore due to escape_fts, https://github.com/simonw/datasette/issues/1396#issuecomment-880967052,https://api.github.com/repos/simonw/datasette/issues/1396,880967052,MDEyOklzc3VlQ29tbWVudDg4MDk2NzA1Mg==,9599,simonw,2021-07-15T19:47:25Z,2021-07-15T19:47:25Z,OWNER,Actually I'm going to close this now and re-open it if the problem occurs again in the future.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944903881,"""invalid reference format"" publishing Docker image", https://github.com/simonw/datasette/issues/1394#issuecomment-880900534,https://api.github.com/repos/simonw/datasette/issues/1394,880900534,MDEyOklzc3VlQ29tbWVudDg4MDkwMDUzNA==,9599,simonw,2021-07-15T17:58:03Z,2021-07-15T17:58:03Z,OWNER,Started a conversation about this on the SQLite forum: https://sqlite.org/forum/forumpost/2d76f2bcf65d256a?t=h,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944870799,Big performance boost on faceting: skip the inner order by, https://github.com/simonw/datasette/issues/1396#issuecomment-880374156,https://api.github.com/repos/simonw/datasette/issues/1396,880374156,MDEyOklzc3VlQ29tbWVudDg4MDM3NDE1Ng==,9599,simonw,2021-07-15T04:03:18Z,2021-07-15T04:03:18Z,OWNER,"I fixed `datasette:latest` by running the following on my laptop: ``` docker pull datasetteproject/datasette:0.58 docker tag datasetteproject/datasette:0.58 datasetteproject/datasette:latest docker login -u datasetteproject -p ... docker push datasetteproject/datasette:latest ``` Confirmed on https://hub.docker.com/r/datasetteproject/datasette/tags?page=1&ordering=last_updated that `datasette:latest` and `datasette:0.58` both now have the same digest of `3b5ba478040e`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944903881,"""invalid reference format"" publishing Docker image", https://github.com/simonw/datasette/issues/1396#issuecomment-880372149,https://api.github.com/repos/simonw/datasette/issues/1396,880372149,MDEyOklzc3VlQ29tbWVudDg4MDM3MjE0OQ==,9599,simonw,2021-07-15T03:56:49Z,2021-07-15T03:56:49Z,OWNER,I'm going to leave this open until I next successfully publish a new version.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944903881,"""invalid reference format"" publishing Docker image", https://github.com/simonw/datasette/issues/1396#issuecomment-880326049,https://api.github.com/repos/simonw/datasette/issues/1396,880326049,MDEyOklzc3VlQ29tbWVudDg4MDMyNjA0OQ==,9599,simonw,2021-07-15T01:50:05Z,2021-07-15T01:50:05Z,OWNER,"I think I made a mistake in this commit: https://github.com/simonw/datasette/commit/0486303b60ce2784fd2e2ecdbecf304b7d6e6659 It looks like I copied `$VERSION_TAG` from here - but it's not available in the `publish.yml` flow: https://github.com/simonw/datasette/blob/0486303b60ce2784fd2e2ecdbecf304b7d6e6659/.github/workflows/push_docker_tag.yml#L18-L25","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944903881,"""invalid reference format"" publishing Docker image", https://github.com/simonw/datasette/issues/1396#issuecomment-880325362,https://api.github.com/repos/simonw/datasette/issues/1396,880325362,MDEyOklzc3VlQ29tbWVudDg4MDMyNTM2Mg==,9599,simonw,2021-07-15T01:48:11Z,2021-07-15T01:48:11Z,OWNER,In particular these three lines: https://github.com/simonw/datasette/blob/084cfe1e00e1a4c0515390a513aca286eeea20c2/.github/workflows/publish.yml#L117-L119,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944903881,"""invalid reference format"" publishing Docker image", https://github.com/simonw/datasette/issues/1396#issuecomment-880325004,https://api.github.com/repos/simonw/datasette/issues/1396,880325004,MDEyOklzc3VlQ29tbWVudDg4MDMyNTAwNA==,9599,simonw,2021-07-15T01:47:17Z,2021-07-15T01:47:17Z,OWNER,"This is the part of the publish workflow that failed and threw the ""invalid reference format"" error: https://github.com/simonw/datasette/blob/084cfe1e00e1a4c0515390a513aca286eeea20c2/.github/workflows/publish.yml#L100-L119","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944903881,"""invalid reference format"" publishing Docker image", https://github.com/simonw/datasette/issues/1396#issuecomment-880324637,https://api.github.com/repos/simonw/datasette/issues/1396,880324637,MDEyOklzc3VlQ29tbWVudDg4MDMyNDYzNw==,9599,simonw,2021-07-15T01:46:26Z,2021-07-15T01:46:26Z,OWNER,"I manually published the Docker image using https://github.com/simonw/datasette/actions/workflows/push_docker_tag.yml https://github.com/simonw/datasette/runs/3072505126 The 0.58 release shows up on https://hub.docker.com/r/datasetteproject/datasette/tags?page=1&ordering=last_updated now - BUT the `latest` tag still points to a version from a month ago.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944903881,"""invalid reference format"" publishing Docker image", https://github.com/simonw/datasette/issues/1394#issuecomment-880287483,https://api.github.com/repos/simonw/datasette/issues/1394,880287483,MDEyOklzc3VlQ29tbWVudDg4MDI4NzQ4Mw==,9599,simonw,2021-07-15T00:01:47Z,2021-07-15T00:01:47Z,OWNER,"I wrote this code: ```python _order_by_re = re.compile(r""(^.*) order by [a-zA-Z_][a-zA-Z0-9_]+( desc)?$"", re.DOTALL) _order_by_braces_re = re.compile(r""(^.*) order by \[[^\]]+\]( desc)?$"", re.DOTALL) def strip_order_by(sql): for regex in (_order_by_re, _order_by_braces_re): match = regex.match(sql) if match is not None: return match.group(1) return sql @pytest.mark.parametrize( ""sql,expected"", [ (""blah"", ""blah""), (""select * from foo"", ""select * from foo""), (""select * from foo order by bah"", ""select * from foo""), (""select * from foo order by bah desc"", ""select * from foo""), (""select * from foo order by [select]"", ""select * from foo""), (""select * from foo order by [select] desc"", ""select * from foo""), ], ) def test_strip_order_by(sql, expected): assert strip_order_by(sql) == expected ``` But it turns out I don't need it! The SQL that is passed to the facet class is created by this code: https://github.com/simonw/datasette/blob/ba11ef27edd6981eeb26d7ecf5aa236707f5f8ce/datasette/views/table.py#L677-L684 And the only place that uses that `sql_no_limit` variable is here: https://github.com/simonw/datasette/blob/ba11ef27edd6981eeb26d7ecf5aa236707f5f8ce/datasette/views/table.py#L733-L745 So I can change that to `sql_no_limit_no_order` and fix the bug that way instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944870799,Big performance boost on faceting: skip the inner order by, https://github.com/simonw/datasette/issues/1394#issuecomment-880278256,https://api.github.com/repos/simonw/datasette/issues/1394,880278256,MDEyOklzc3VlQ29tbWVudDg4MDI3ODI1Ng==,9599,simonw,2021-07-14T23:35:18Z,2021-07-14T23:35:18Z,OWNER,"The challenge here is that faceting doesn't currently modify the inner SQL at all - it wraps it so that it can work against any SQL statement (though Datasette itself does not yet take advantage of that ability, only offering faceting on table pages). So just removing the order by wouldn't be appropriate if the inner query looked something like this: ```sql select * from items order by created desc limit 100 ``` Since the intent there would be to return facet counts against only the most recent 100 items. In SQLite the `limit` has to come after the `order by` though, so the fix here could be as easy as using a regular expression to identify queries that end with `order by COLUMN (desc)?` and stripping off that clause.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944870799,Big performance boost on faceting: skip the inner order by, https://github.com/simonw/sqlite-utils/issues/297#issuecomment-880259255,https://api.github.com/repos/simonw/sqlite-utils/issues/297,880259255,MDEyOklzc3VlQ29tbWVudDg4MDI1OTI1NQ==,9599,simonw,2021-07-14T22:48:41Z,2021-07-14T22:48:41Z,OWNER,Should also take advantage of `.mode tabs` to support `sqlite-utils insert blah.db blah blah.csv --tsv --fast`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944846776,Option for importing CSV data using the SQLite .import mechanism, https://github.com/simonw/sqlite-utils/issues/297#issuecomment-880257587,https://api.github.com/repos/simonw/sqlite-utils/issues/297,880257587,MDEyOklzc3VlQ29tbWVudDg4MDI1NzU4Nw==,9599,simonw,2021-07-14T22:44:05Z,2021-07-14T22:44:05Z,OWNER,"https://unix.stackexchange.com/a/642364 suggests you can also use this to import from stdin, like so: sqlite3 -csv $database_file_name "".import '|cat -' $table_name"" Here the `sqlite3 -csv` is an alternative to using `.mode csv`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944846776,Option for importing CSV data using the SQLite .import mechanism, https://github.com/simonw/sqlite-utils/issues/297#issuecomment-880256865,https://api.github.com/repos/simonw/sqlite-utils/issues/297,880256865,MDEyOklzc3VlQ29tbWVudDg4MDI1Njg2NQ==,9599,simonw,2021-07-14T22:42:11Z,2021-07-14T22:42:11Z,OWNER,"Potential workaround for missing `--skip` implementation is that the filename can be a command instead, so maybe it could shell out to `tail -n +1 filename`: > The source argument is the name of a file to be read or, if it begins with a ""|"" character, specifies a command which will be run to produce the input CSV data.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944846776,Option for importing CSV data using the SQLite .import mechanism, https://github.com/simonw/sqlite-utils/issues/297#issuecomment-880256058,https://api.github.com/repos/simonw/sqlite-utils/issues/297,880256058,MDEyOklzc3VlQ29tbWVudDg4MDI1NjA1OA==,9599,simonw,2021-07-14T22:40:01Z,2021-07-14T22:40:47Z,OWNER,"Full docs here: https://www.sqlite.org/draft/cli.html#csv One catch: how this works has changed in recent SQLite versions: https://www.sqlite.org/changes.html - 2020-12-01 (3.34.0) - ""Table name quoting works correctly for the .import dot-command"" - 2020-05-22 (3.32.0) - ""Add options to the .import command: --csv, --ascii, --skip"" - 2017-08-01 (3.20.0) - "" The "".import"" command ignores an initial UTF-8 BOM."" The ""skip"" feature is particularly important to understand. https://www.sqlite.org/draft/cli.html#csv says: > There are two cases to consider: (1) Table ""tab1"" does not previously exist and (2) table ""tab1"" does already exist. > > In the first case, when the table does not previously exist, the table is automatically created and the content of the first row of the input CSV file is used to determine the name of all the columns in the table. In other words, if the table does not previously exist, the first row of the CSV file is interpreted to be column names and the actual data starts on the second row of the CSV file. > > For the second case, when the table already exists, every row of the CSV file, including the first row, is assumed to be actual content. If the CSV file contains an initial row of column labels, you can cause the .import command to skip that initial row using the ""--skip 1"" option. But the `--skip 1` option is only available in 3.32.0 and higher.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944846776,Option for importing CSV data using the SQLite .import mechanism, https://github.com/simonw/datasette/issues/268#issuecomment-880153069,https://api.github.com/repos/simonw/datasette/issues/268,880153069,MDEyOklzc3VlQ29tbWVudDg4MDE1MzA2OQ==,9599,simonw,2021-07-14T19:31:00Z,2021-07-14T19:31:00Z,OWNER,"... though interestingly I can't replicate that error on `latest.datasette.io` - https://latest.datasette.io/fixtures/searchable?_search=park.&_searchmode=raw That's running https://latest.datasette.io/-/versions SQLite 3.35.4 whereas https://www.niche-museums.com/-/versions is running 3.27.2 (the most recent version available with Vercel) - but there's nothing in the SQLite changelog between those two versions that suggests changes to how the FTS5 parser works. https://www.sqlite.org/changes.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323718842,Mechanism for ranking results from SQLite full-text search, https://github.com/simonw/datasette/issues/268#issuecomment-880150755,https://api.github.com/repos/simonw/datasette/issues/268,880150755,MDEyOklzc3VlQ29tbWVudDg4MDE1MDc1NQ==,9599,simonw,2021-07-14T19:26:47Z,2021-07-14T19:29:08Z,OWNER,"> What are the side-effects of turning that on in the query string, or even by default as you suggested? I see that you stated in the docs... ""to ensure they do not cause any confusion for users who are not aware of them"", but I'm not sure what those could be. Mainly that it's possible to generate SQL queries that crash with an error. This was the example that convinced me to default to escaping: - https://www.niche-museums.com/browse/museums?_search=park.&_searchmode=raw (returns `fts5: syntax error near "".""`) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323718842,Mechanism for ranking results from SQLite full-text search, https://github.com/simonw/datasette/pull/1393#issuecomment-879309636,https://api.github.com/repos/simonw/datasette/issues/1393,879309636,MDEyOklzc3VlQ29tbWVudDg3OTMwOTYzNg==,9599,simonw,2021-07-13T18:32:25Z,2021-07-13T18:32:25Z,OWNER,Thanks,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",941412189,Update deploying.rst, https://github.com/simonw/datasette/pull/1392#issuecomment-879277953,https://api.github.com/repos/simonw/datasette/issues/1392,879277953,MDEyOklzc3VlQ29tbWVudDg3OTI3Nzk1Mw==,9599,simonw,2021-07-13T17:42:31Z,2021-07-13T17:42:31Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",941403676,Update deploying.rst, https://github.com/simonw/datasette/issues/511#issuecomment-877835171,https://api.github.com/repos/simonw/datasette/issues/511,877835171,MDEyOklzc3VlQ29tbWVudDg3NzgzNTE3MQ==,9599,simonw,2021-07-11T17:23:05Z,2021-07-11T17:23:05Z,OWNER," == 87 failed, 819 passed, 7 skipped, 29 errors in 2584.85s (0:43:04) == https://github.com/simonw/datasette/runs/3038188870?check_suite_focus=true Full copy of log here: https://gist.github.com/simonw/4b1fdd24496b989fca56bc757be345ad","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",456578474,Get Datasette tests passing on Windows in GitHub Actions, https://github.com/simonw/datasette/issues/511#issuecomment-877726495,https://api.github.com/repos/simonw/datasette/issues/511,877726495,MDEyOklzc3VlQ29tbWVudDg3NzcyNjQ5NQ==,9599,simonw,2021-07-11T01:32:27Z,2021-07-11T01:32:27Z,OWNER,"I'm using `pytest-xdist` and this: pytest -n auto -m ""not serial"" I'll try not using the `-n auto` bit on Windows and see if that helps.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",456578474,Get Datasette tests passing on Windows in GitHub Actions, https://github.com/simonw/datasette/issues/511#issuecomment-877726288,https://api.github.com/repos/simonw/datasette/issues/511,877726288,MDEyOklzc3VlQ29tbWVudDg3NzcyNjI4OA==,9599,simonw,2021-07-11T01:29:41Z,2021-07-11T01:29:41Z,OWNER,"Lots of errors that look like this: ``` 2021-07-11T00:40:32.1189321Z E NotADirectoryError: [WinError 267] The directory name is invalid: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpdr41pgwg\\data.db' 2021-07-11T00:40:32.1190083Z 2021-07-11T00:40:32.1191128Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\shutil.py:596: NotADirectoryError 2021-07-11T00:40:32.1191999Z ___________________ ERROR at teardown of test_insert_error ____________________ 2021-07-11T00:40:32.1192842Z [gw1] win32 -- Python 3.8.10 c:\hostedtoolcache\windows\python\3.8.10\x64\python.exe 2021-07-11T00:40:32.1193387Z 2021-07-11T00:40:32.1193930Z path = 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpry729pq_' 2021-07-11T00:40:32.1194876Z onerror = .onerror at 0x00000291FCEA93A0> 2021-07-11T00:40:32.1195480Z 2021-07-11T00:40:32.1195927Z def _rmtree_unsafe(path, onerror): 2021-07-11T00:40:32.1196435Z try: 2021-07-11T00:40:32.1196910Z with os.scandir(path) as scandir_it: 2021-07-11T00:40:32.1197504Z entries = list(scandir_it) 2021-07-11T00:40:32.1198002Z except OSError: 2021-07-11T00:40:32.1198607Z onerror(os.scandir, path, sys.exc_info()) 2021-07-11T00:40:32.1199137Z entries = [] 2021-07-11T00:40:32.1199637Z for entry in entries: 2021-07-11T00:40:32.1200184Z fullname = entry.path 2021-07-11T00:40:32.1200692Z if _rmtree_isdir(entry): 2021-07-11T00:40:32.1201198Z try: 2021-07-11T00:40:32.1201643Z if entry.is_symlink(): 2021-07-11T00:40:32.1202280Z # This can only happen if someone replaces 2021-07-11T00:40:32.1202944Z # a directory with a symlink after the call to 2021-07-11T00:40:32.1203623Z # os.scandir or entry.is_dir above. 2021-07-11T00:40:32.1204303Z raise OSError(""Cannot call rmtree on a symbolic link"") 2021-07-11T00:40:32.1204942Z except OSError: 2021-07-11T00:40:32.1206416Z onerror(os.path.islink, fullname, sys.exc_info()) 2021-07-11T00:40:32.1207022Z continue 2021-07-11T00:40:32.1207584Z _rmtree_unsafe(fullname, onerror) 2021-07-11T00:40:32.1208074Z else: 2021-07-11T00:40:32.1208496Z try: 2021-07-11T00:40:32.1208926Z > os.unlink(fullname) 2021-07-11T00:40:32.1210053Z E PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpry729pq_\\data.db' 2021-07-11T00:40:32.1210974Z 2021-07-11T00:40:32.1211638Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\shutil.py:616: PermissionError 2021-07-11T00:40:32.1212211Z 2021-07-11T00:40:32.1212846Z During handling of the above exception, another exception occurred: 2021-07-11T00:40:32.1213320Z 2021-07-11T00:40:32.1213797Z func = 2021-07-11T00:40:32.1214529Z path = 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpry729pq_\\data.db' 2021-07-11T00:40:32.1215763Z exc_info = (, PermissionError(13, 'The process cannot access the file because it is being used by another process'), ) 2021-07-11T00:40:32.1217263Z 2021-07-11T00:40:32.1217777Z def onerror(func, path, exc_info): 2021-07-11T00:40:32.1218421Z if issubclass(exc_info[0], PermissionError): 2021-07-11T00:40:32.1219079Z def resetperms(path): 2021-07-11T00:40:32.1219518Z try: 2021-07-11T00:40:32.1219992Z _os.chflags(path, 0) 2021-07-11T00:40:32.1220535Z except AttributeError: 2021-07-11T00:40:32.1221110Z pass 2021-07-11T00:40:32.1221545Z _os.chmod(path, 0o700) 2021-07-11T00:40:32.1221984Z 2021-07-11T00:40:32.1222330Z try: 2021-07-11T00:40:32.1222768Z if path != name: 2021-07-11T00:40:32.1223332Z resetperms(_os.path.dirname(path)) 2021-07-11T00:40:32.1223963Z resetperms(path) 2021-07-11T00:40:32.1224408Z 2021-07-11T00:40:32.1224749Z try: 2021-07-11T00:40:32.1225954Z > _os.unlink(path) 2021-07-11T00:40:32.1227032Z E PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpry729pq_\\data.db' 2021-07-11T00:40:32.1227927Z 2021-07-11T00:40:32.1228646Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\tempfile.py:802: PermissionError 2021-07-11T00:40:32.1229200Z 2021-07-11T00:40:32.1229842Z During handling of the above exception, another exception occurred: 2021-07-11T00:40:32.1230355Z 2021-07-11T00:40:32.1230783Z @pytest.fixture 2021-07-11T00:40:32.1231322Z def canned_write_client(): 2021-07-11T00:40:32.1231805Z with make_app_client( 2021-07-11T00:40:32.1232467Z extra_databases={""data.db"": ""create table names (name text)""}, 2021-07-11T00:40:32.1233104Z metadata={ 2021-07-11T00:40:32.1233535Z ""databases"": { 2021-07-11T00:40:32.1233989Z ""data"": { 2021-07-11T00:40:32.1234416Z ""queries"": { 2021-07-11T00:40:32.1235001Z ""canned_read"": {""sql"": ""select * from names""}, 2021-07-11T00:40:32.1235527Z ""add_name"": { 2021-07-11T00:40:32.1236117Z ""sql"": ""insert into names (name) values (:name)"", 2021-07-11T00:40:32.1236686Z ""write"": True, 2021-07-11T00:40:32.1237317Z ""on_success_redirect"": ""/data/add_name?success"", 2021-07-11T00:40:32.1237882Z }, 2021-07-11T00:40:32.1238331Z ""add_name_specify_id"": { 2021-07-11T00:40:32.1239009Z ""sql"": ""insert into names (rowid, name) values (:rowid, :name)"", 2021-07-11T00:40:32.1239610Z ""write"": True, 2021-07-11T00:40:32.1240259Z ""on_error_redirect"": ""/data/add_name_specify_id?error"", 2021-07-11T00:40:32.1240839Z }, 2021-07-11T00:40:32.1241320Z ""delete_name"": { 2021-07-11T00:40:32.1242504Z ""sql"": ""delete from names where rowid = :rowid"", 2021-07-11T00:40:32.1243127Z ""write"": True, 2021-07-11T00:40:32.1243721Z ""on_success_message"": ""Name deleted"", 2021-07-11T00:40:32.1244282Z ""allow"": {""id"": ""root""}, 2021-07-11T00:40:32.1244749Z }, 2021-07-11T00:40:32.1245959Z ""update_name"": { 2021-07-11T00:40:32.1246614Z ""sql"": ""update names set name = :name where rowid = :rowid"", 2021-07-11T00:40:32.1247267Z ""params"": [""rowid"", ""name"", ""extra""], 2021-07-11T00:40:32.1247828Z ""write"": True, 2021-07-11T00:40:32.1248247Z }, 2021-07-11T00:40:32.1248653Z } 2021-07-11T00:40:32.1249166Z } 2021-07-11T00:40:32.1249577Z } 2021-07-11T00:40:32.1249962Z }, 2021-07-11T00:40:32.1250333Z ) as client: 2021-07-11T00:40:32.1250822Z > yield client 2021-07-11T00:40:32.1251078Z 2021-07-11T00:40:32.1251678Z D:\a\datasette\datasette\tests\test_canned_queries.py:43: 2021-07-11T00:40:32.1252347Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2021-07-11T00:40:32.1253040Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\contextlib.py:120: in __exit__ 2021-07-11T00:40:32.1253759Z next(self.gen) 2021-07-11T00:40:32.1254398Z D:\a\datasette\datasette\tests\fixtures.py:156: in make_app_client 2021-07-11T00:40:32.1255098Z yield TestClient(ds) 2021-07-11T00:40:32.1255796Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\tempfile.py:827: in __exit__ 2021-07-11T00:40:32.1256510Z self.cleanup() 2021-07-11T00:40:32.1257200Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\tempfile.py:831: in cleanup 2021-07-11T00:40:32.1257961Z self._rmtree(self.name) 2021-07-11T00:40:32.1258712Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\tempfile.py:813: in _rmtree 2021-07-11T00:40:32.1259487Z _shutil.rmtree(name, onerror=onerror) 2021-07-11T00:40:32.1260280Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\shutil.py:740: in rmtree 2021-07-11T00:40:32.1261039Z return _rmtree_unsafe(path, onerror) 2021-07-11T00:40:32.1261843Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\shutil.py:618: in _rmtree_unsafe 2021-07-11T00:40:32.1262633Z onerror(os.unlink, fullname, sys.exc_info()) 2021-07-11T00:40:32.1263456Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\tempfile.py:805: in onerror 2021-07-11T00:40:32.1264175Z cls._rmtree(path) 2021-07-11T00:40:32.1264848Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\tempfile.py:813: in _rmtree 2021-07-11T00:40:32.1266329Z _shutil.rmtree(name, onerror=onerror) 2021-07-11T00:40:32.1267082Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\shutil.py:740: in rmtree 2021-07-11T00:40:32.1267858Z return _rmtree_unsafe(path, onerror) 2021-07-11T00:40:32.1268615Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\shutil.py:599: in _rmtree_unsafe 2021-07-11T00:40:32.1269440Z onerror(os.scandir, path, sys.exc_info()) 2021-07-11T00:40:32.1269979Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2021-07-11T00:40:32.1270287Z 2021-07-11T00:40:32.1270947Z path = 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpry729pq_\\data.db' 2021-07-11T00:40:32.1273356Z onerror = .onerror at 0x00000291FCF40E50> 2021-07-11T00:40:32.1273999Z 2021-07-11T00:40:32.1274493Z def _rmtree_unsafe(path, onerror): 2021-07-11T00:40:32.1274953Z try: 2021-07-11T00:40:32.1275461Z > with os.scandir(path) as scandir_it: 2021-07-11T00:40:32.1276459Z E NotADirectoryError: [WinError 267] The directory name is invalid: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpry729pq_\\data.db' 2021-07-11T00:40:32.1277220Z ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",456578474,Get Datasette tests passing on Windows in GitHub Actions, https://github.com/simonw/datasette/issues/511#issuecomment-877725742,https://api.github.com/repos/simonw/datasette/issues/511,877725742,MDEyOklzc3VlQ29tbWVudDg3NzcyNTc0Mg==,9599,simonw,2021-07-11T01:25:01Z,2021-07-11T01:26:38Z,OWNER,"That's weird. https://github.com/simonw/datasette/runs/3037862798 finished running and came up green - but actually a TON of the tests failed on Windows. Not sure why that didn't fail the whole test suite: Also the test suite took 50 minutes on Windows! Here's a copy of the full log file for the tests on Python 3.8 on Windows: https://gist.github.com/simonw/2900ef33693c1bbda09188eb31c8212d","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",456578474,Get Datasette tests passing on Windows in GitHub Actions, https://github.com/simonw/datasette/issues/1388#issuecomment-877725193,https://api.github.com/repos/simonw/datasette/issues/1388,877725193,MDEyOklzc3VlQ29tbWVudDg3NzcyNTE5Mw==,9599,simonw,2021-07-11T01:18:38Z,2021-07-11T01:18:38Z,OWNER,Wrote up a TIL: https://til.simonwillison.net/nginx/proxy-domain-sockets,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket, https://github.com/simonw/datasette/issues/1388#issuecomment-877721003,https://api.github.com/repos/simonw/datasette/issues/1388,877721003,MDEyOklzc3VlQ29tbWVudDg3NzcyMTAwMw==,9599,simonw,2021-07-11T00:21:19Z,2021-07-11T00:21:19Z,OWNER,Documentation: https://docs.datasette.io/en/latest/deploying.html#nginx-proxy-configuration,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket, https://github.com/simonw/datasette/issues/511#issuecomment-877718364,https://api.github.com/repos/simonw/datasette/issues/511,877718364,MDEyOklzc3VlQ29tbWVudDg3NzcxODM2NA==,9599,simonw,2021-07-10T23:54:37Z,2021-07-10T23:54:37Z,OWNER,"Looks like it's not even 10% of the way through, and already a bunch of errors: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",456578474,Get Datasette tests passing on Windows in GitHub Actions, https://github.com/simonw/datasette/issues/511#issuecomment-877718286,https://api.github.com/repos/simonw/datasette/issues/511,877718286,MDEyOklzc3VlQ29tbWVudDg3NzcxODI4Ng==,9599,simonw,2021-07-10T23:53:29Z,2021-07-10T23:53:29Z,OWNER,"Test suite on Windows seems to run a lot slower: From https://github.com/simonw/datasette/actions/runs/1018938850 which is still going. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",456578474,Get Datasette tests passing on Windows in GitHub Actions, https://github.com/simonw/datasette/issues/511#issuecomment-877717791,https://api.github.com/repos/simonw/datasette/issues/511,877717791,MDEyOklzc3VlQ29tbWVudDg3NzcxNzc5MQ==,9599,simonw,2021-07-10T23:45:35Z,2021-07-10T23:45:35Z,OWNER,"> Trying to run on Windows today, I get an error from the utils/asgi.py module. > > It's trying `from os import EX_CANTCREAT` which is Unix-only. I commented this line out, and (so far) it's working. Good news: that line was removed in #1094.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",456578474,Get Datasette tests passing on Windows in GitHub Actions, https://github.com/simonw/datasette/pull/557#issuecomment-877717392,https://api.github.com/repos/simonw/datasette/issues/557,877717392,MDEyOklzc3VlQ29tbWVudDg3NzcxNzM5Mg==,9599,simonw,2021-07-10T23:39:48Z,2021-07-10T23:39:48Z,OWNER,Abandoning this - need to switch to using GitHub Actions for this instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",466996584,Get tests running on Windows using Travis CI, https://github.com/simonw/datasette/issues/1388#issuecomment-877717262,https://api.github.com/repos/simonw/datasette/issues/1388,877717262,MDEyOklzc3VlQ29tbWVudDg3NzcxNzI2Mg==,9599,simonw,2021-07-10T23:37:54Z,2021-07-10T23:37:54Z,OWNER,"> I wonder if `--fd` is worth supporting too? I'm going to hold off on implementing this until someone asks for it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket, https://github.com/simonw/datasette/issues/1388#issuecomment-877716993,https://api.github.com/repos/simonw/datasette/issues/1388,877716993,MDEyOklzc3VlQ29tbWVudDg3NzcxNjk5Mw==,9599,simonw,2021-07-10T23:34:02Z,2021-07-10T23:34:02Z,OWNER,"Figured out an example nginx configuration. This in `nginx.conf`: daemon off; events { worker_connections 1024; } http { server { listen 8092; location / { proxy_pass http://datasette; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } } upstream datasette { server unix:/tmp/datasette.sock; } } Then run `datasette --uds /tmp/datasette.sock` Then run nginx like this: nginx -c ./nginx.conf Then hits to `http://localhost:8092/` will be proxied to Datasette.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket, https://github.com/simonw/datasette/issues/1388#issuecomment-877716359,https://api.github.com/repos/simonw/datasette/issues/1388,877716359,MDEyOklzc3VlQ29tbWVudDg3NzcxNjM1OQ==,9599,simonw,2021-07-10T23:24:58Z,2021-07-10T23:24:58Z,OWNER,"Apparently Windows 10 has Unix domain socket support: https://bugs.python.org/issue33408 > Unix socket (AF_UNIX) is now avalible in Windows 10 (April 2018 Update). Please add Python support for it. > More details about it on https://blogs.msdn.microsoft.com/commandline/2017/12/19/af_unix-comes-to-windows/ But it's not clear if this is going to work. That same issue thread (the issue is still open) suggests using `hasattr(socket, 'AF_UNIX'))` to detect support in tests.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket, https://github.com/simonw/datasette/issues/1388#issuecomment-877716156,https://api.github.com/repos/simonw/datasette/issues/1388,877716156,MDEyOklzc3VlQ29tbWVudDg3NzcxNjE1Ng==,9599,simonw,2021-07-10T23:22:21Z,2021-07-10T23:22:21Z,OWNER,"I don't have the Datasette test suite running on Windows yet, but I'd like it to run there some day - so ideally this test would be skipped if Unix domain sockets are not supported by the underlying operating system.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket, https://github.com/simonw/datasette/issues/1388#issuecomment-877715654,https://api.github.com/repos/simonw/datasette/issues/1388,877715654,MDEyOklzc3VlQ29tbWVudDg3NzcxNTY1NA==,9599,simonw,2021-07-10T23:15:06Z,2021-07-10T23:15:06Z,OWNER,"I can run tests against it using `httpx`: https://www.python-httpx.org/advanced/#usage_1 > ```pycon > >>> import httpx > >>> # Connect to the Docker API via a Unix Socket. > >>> transport = httpx.HTTPTransport(uds=""/var/run/docker.sock"") > >>> client = httpx.Client(transport=transport) > >>> response = client.get(""http://docker/info"") > ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket, https://github.com/simonw/datasette/issues/1388#issuecomment-877714698,https://api.github.com/repos/simonw/datasette/issues/1388,877714698,MDEyOklzc3VlQ29tbWVudDg3NzcxNDY5OA==,9599,simonw,2021-07-10T23:01:37Z,2021-07-10T23:01:37Z,OWNER,"Can test this with: ``` curl --unix-socket ${socket} -i ""http://localhost/"" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket, https://github.com/simonw/datasette/issues/1391#issuecomment-877691558,https://api.github.com/repos/simonw/datasette/issues/1391,877691558,MDEyOklzc3VlQ29tbWVudDg3NzY5MTU1OA==,9599,simonw,2021-07-10T19:26:57Z,2021-07-10T19:26:57Z,OWNER,"The `https://latest.datasette.io/fixtures.db` file no longer includes generated columns, which will help avoid confusion such as seen in #1376.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",941300946,Stop using generated columns in fixtures.db, https://github.com/simonw/datasette/issues/1391#issuecomment-877691427,https://api.github.com/repos/simonw/datasette/issues/1391,877691427,MDEyOklzc3VlQ29tbWVudDg3NzY5MTQyNw==,9599,simonw,2021-07-10T19:26:00Z,2021-07-10T19:26:00Z,OWNER,I had to run the tests locally on my macOS laptop using `pysqlite3` to get a version that supported generated columns - wrote up a TIL about that here: https://til.simonwillison.net/sqlite/pysqlite3-on-macos,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",941300946,Stop using generated columns in fixtures.db, https://github.com/simonw/datasette/issues/1391#issuecomment-877687196,https://api.github.com/repos/simonw/datasette/issues/1391,877687196,MDEyOklzc3VlQ29tbWVudDg3NzY4NzE5Ng==,9599,simonw,2021-07-10T18:58:40Z,2021-07-10T18:58:40Z,OWNER,I can use the `extra_databases` mechanism as demonstrated here: https://github.com/simonw/datasette/blob/9552414e1f968c6fc704031cec349c05e6bc2371/tests/test_canned_queries.py#L8-L12,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",941300946,Stop using generated columns in fixtures.db, https://github.com/simonw/datasette/issues/1391#issuecomment-877686784,https://api.github.com/repos/simonw/datasette/issues/1391,877686784,MDEyOklzc3VlQ29tbWVudDg3NzY4Njc4NA==,9599,simonw,2021-07-10T18:56:03Z,2021-07-10T18:56:03Z,OWNER,Here's the SQL used to generate the table for the test: https://github.com/simonw/datasette/blob/02b19c7a9afd328f22040ab33b5c1911cd904c7c/tests/fixtures.py#L723-L733,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",941300946,Stop using generated columns in fixtures.db, https://github.com/simonw/datasette/issues/1391#issuecomment-877682533,https://api.github.com/repos/simonw/datasette/issues/1391,877682533,MDEyOklzc3VlQ29tbWVudDg3NzY4MjUzMw==,9599,simonw,2021-07-10T18:28:05Z,2021-07-10T18:28:05Z,OWNER,"Here's the test in question: https://github.com/simonw/datasette/blob/a6c55afe8c82ead8deb32f90c9324022fd422324/tests/test_api.py#L2033-L2046 Various other places in the test code also need changing - anything that calls `supports_generated_columns()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",941300946,Stop using generated columns in fixtures.db, https://github.com/simonw/datasette/issues/1389#issuecomment-877681031,https://api.github.com/repos/simonw/datasette/issues/1389,877681031,MDEyOklzc3VlQ29tbWVudDg3NzY4MTAzMQ==,9599,simonw,2021-07-10T18:17:29Z,2021-07-10T18:17:29Z,OWNER,"I don't like `?_searchmode=default` because it suggests ""use the default"" - but it actually over-rides the default that was specified by `""searchmode"": ""raw""` in `metadata.json`. I'm going with `?_searchmode=escaped` instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",940077168,"""searchmode"": ""raw"" in table metadata", https://github.com/simonw/datasette/issues/1390#issuecomment-877310125,https://api.github.com/repos/simonw/datasette/issues/1390,877310125,MDEyOklzc3VlQ29tbWVudDg3NzMxMDEyNQ==,9599,simonw,2021-07-09T16:32:57Z,2021-07-09T16:32:57Z,OWNER,https://docs.datasette.io/en/latest/deploying.html#running-datasette-using-systemd,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",940891698,Mention restarting systemd in documentation, https://github.com/simonw/datasette/issues/1390#issuecomment-877308310,https://api.github.com/repos/simonw/datasette/issues/1390,877308310,MDEyOklzc3VlQ29tbWVudDg3NzMwODMxMA==,9599,simonw,2021-07-09T16:29:48Z,2021-07-09T16:29:48Z,OWNER, sudo systemctl restart datasette.service,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",940891698,Mention restarting systemd in documentation, https://github.com/simonw/datasette/issues/1389#issuecomment-876620095,https://api.github.com/repos/simonw/datasette/issues/1389,876620095,MDEyOklzc3VlQ29tbWVudDg3NjYyMDA5NQ==,9599,simonw,2021-07-08T17:35:09Z,2021-07-08T17:35:09Z,OWNER,Also came up here: https://github.com/simonw/datasette/issues/268#issuecomment-876616414,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",940077168,"""searchmode"": ""raw"" in table metadata", https://github.com/simonw/datasette/issues/1389#issuecomment-876619531,https://api.github.com/repos/simonw/datasette/issues/1389,876619531,MDEyOklzc3VlQ29tbWVudDg3NjYxOTUzMQ==,9599,simonw,2021-07-08T17:34:16Z,2021-07-08T17:34:16Z,OWNER,If I implement this I'll also set it up so `?_searchmode=default` can be used to over-ride that and go back to the default behaviour.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",940077168,"""searchmode"": ""raw"" in table metadata", https://github.com/simonw/datasette/issues/1389#issuecomment-876619271,https://api.github.com/repos/simonw/datasette/issues/1389,876619271,MDEyOklzc3VlQ29tbWVudDg3NjYxOTI3MQ==,9599,simonw,2021-07-08T17:33:49Z,2021-07-08T17:33:49Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/views/table.py#L491-L498,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",940077168,"""searchmode"": ""raw"" in table metadata", https://github.com/simonw/datasette/issues/1389#issuecomment-876618847,https://api.github.com/repos/simonw/datasette/issues/1389,876618847,MDEyOklzc3VlQ29tbWVudDg3NjYxODg0Nw==,9599,simonw,2021-07-08T17:33:08Z,2021-07-08T17:33:08Z,OWNER,Relevant documentation: https://docs.datasette.io/en/0.57.1/full_text_search.html#advanced-sqlite-search-queries,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",940077168,"""searchmode"": ""raw"" in table metadata", https://github.com/simonw/datasette/issues/1389#issuecomment-876618582,https://api.github.com/repos/simonw/datasette/issues/1389,876618582,MDEyOklzc3VlQ29tbWVudDg3NjYxODU4Mg==,9599,simonw,2021-07-08T17:32:40Z,2021-07-08T17:32:40Z,OWNER,This makes sense to me since other useful querystring arguments like this can be set as defaults in the `metadata.json` configuration.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",940077168,"""searchmode"": ""raw"" in table metadata", https://github.com/simonw/datasette/issues/268#issuecomment-876616414,https://api.github.com/repos/simonw/datasette/issues/268,876616414,MDEyOklzc3VlQ29tbWVudDg3NjYxNjQxNA==,9599,simonw,2021-07-08T17:29:04Z,2021-07-08T17:29:04Z,OWNER,"> I had setup a full text search on my instance of Datasette for title data for our public library, and was noticing that some of the features of the SQLite FTS weren't working as expected ... and maybe the issue is in the `escape_fts()` function That's a deliberate feature (albeit controversial, see #759) - part of the main problem here is that it's easy to construct a SQLite full-text search string which results in a database error. This is a bad user-experience! You can opt-in to raw SQL queries by appending `?_searchmode=raw` to the page, see https://docs.datasette.io/en/stable/full_text_search.html#advanced-sqlite-search-queries But maybe there should be an option for turning that on by default without needing the query string? ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323718842,Mechanism for ranking results from SQLite full-text search, https://github.com/simonw/datasette/issues/1388#issuecomment-875742910,https://api.github.com/repos/simonw/datasette/issues/1388,875742910,MDEyOklzc3VlQ29tbWVudDg3NTc0MjkxMA==,9599,simonw,2021-07-07T16:20:50Z,2021-07-07T16:23:02Z,OWNER,"I wonder if `--fd` is worth supporting too? Uvicorn documentation says that's useful for running under process managers, I'd want to understand exactly how to use that (and test it) before adding the feature though. https://www.uvicorn.org/settings/ Docs on how to use a process manager: https://www.uvicorn.org/deployment/#supervisor","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket, https://github.com/simonw/datasette/issues/1388#issuecomment-875741410,https://api.github.com/repos/simonw/datasette/issues/1388,875741410,MDEyOklzc3VlQ29tbWVudDg3NTc0MTQxMA==,9599,simonw,2021-07-07T16:18:50Z,2021-07-07T16:18:50Z,OWNER,"You could actually run Datasette like this today without modifications by running a thin Python script that imports from `datasette.app`, instantiates the ASGI app and passes that to `uvicorn.run` - but I like this as a supported feature too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket, https://github.com/simonw/datasette/issues/1388#issuecomment-875740085,https://api.github.com/repos/simonw/datasette/issues/1388,875740085,MDEyOklzc3VlQ29tbWVudDg3NTc0MDA4NQ==,9599,simonw,2021-07-07T16:17:08Z,2021-07-07T16:17:08Z,OWNER,"Looks pretty easy to implement - here's a hint from Uvicorn source code: https://github.com/encode/uvicorn/blob/b5af1049e63c059dc750a450c807b9768f91e906/uvicorn/main.py#L368 Need to work out a simple pattern for testing this too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket, https://github.com/simonw/datasette/issues/1388#issuecomment-875738149,https://api.github.com/repos/simonw/datasette/issues/1388,875738149,MDEyOklzc3VlQ29tbWVudDg3NTczODE0OQ==,9599,simonw,2021-07-07T16:14:29Z,2021-07-07T16:14:29Z,OWNER,This sounds like a valuable feature for people running Datasette behind a proxy.,"{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket, https://github.com/simonw/datasette/issues/1387#issuecomment-873156408,https://api.github.com/repos/simonw/datasette/issues/1387,873156408,MDEyOklzc3VlQ29tbWVudDg3MzE1NjQwOA==,9599,simonw,2021-07-02T17:37:30Z,2021-07-02T17:37:30Z,OWNER,Updated documentation is here: https://docs.datasette.io/en/latest/deploying.html#apache-proxy-configuration,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",935930820,absolute_url() behind a proxy assembles incorrect http://127.0.0.1:8001/ URLs, https://github.com/simonw/datasette/issues/1387#issuecomment-873141222,https://api.github.com/repos/simonw/datasette/issues/1387,873141222,MDEyOklzc3VlQ29tbWVudDg3MzE0MTIyMg==,9599,simonw,2021-07-02T17:09:32Z,2021-07-02T17:09:32Z,OWNER,I'm going to add this to the suggested Apache configuration at https://docs.datasette.io/en/stable/deploying.html#apache-proxy-configuration,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",935930820,absolute_url() behind a proxy assembles incorrect http://127.0.0.1:8001/ URLs, https://github.com/simonw/datasette/issues/1387#issuecomment-873140742,https://api.github.com/repos/simonw/datasette/issues/1387,873140742,MDEyOklzc3VlQ29tbWVudDg3MzE0MDc0Mg==,9599,simonw,2021-07-02T17:08:40Z,2021-07-02T17:08:40Z,OWNER,`ProxyPreserveHost On` is the Apache setting - it defaults to `Off`: https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypreservehost ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",935930820,absolute_url() behind a proxy assembles incorrect http://127.0.0.1:8001/ URLs, https://github.com/simonw/datasette/issues/1387#issuecomment-873139138,https://api.github.com/repos/simonw/datasette/issues/1387,873139138,MDEyOklzc3VlQ29tbWVudDg3MzEzOTEzOA==,9599,simonw,2021-07-02T17:05:47Z,2021-07-02T17:05:47Z,OWNER,"In this case the proxy is Apache. So there are a couple of potential fixes: - Configure Apache to pass the original HTTP request `Host:` header through to the proxied application. This should then be documented. - Add a new optional feature to Datasette called something like `base_host` which, if set, is always used in place of the host in `request.url` when constructing new URLs.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",935930820,absolute_url() behind a proxy assembles incorrect http://127.0.0.1:8001/ URLs, https://github.com/simonw/datasette/issues/1387#issuecomment-873137935,https://api.github.com/repos/simonw/datasette/issues/1387,873137935,MDEyOklzc3VlQ29tbWVudDg3MzEzNzkzNQ==,9599,simonw,2021-07-02T17:03:36Z,2021-07-02T17:03:36Z,OWNER,"And the links to apply a facet value are broken too! https://ilsweb.cincinnatilibrary.org/collection-analysis/current_collection-3d4a4b7/bib?_facet=bib_level_callnumber ```json { ""value"": ""g l fiction"", ""label"": ""g l fiction"", ""count"": 212, ""toggle_url"": ""https://127.0.0.1:8010/collection-analysis/current_collection-3d4a4b7/bib.json?_facet=bib_level_callnumber&bib_level_callnumber=g+l+fiction"", ""selected"": false } ``` Same problem: https://github.com/simonw/datasette/blob/ea627baccf980d7d8ebc9e1ffff1fe34d556e56f/datasette/facets.py#L251-L261","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",935930820,absolute_url() behind a proxy assembles incorrect http://127.0.0.1:8001/ URLs, https://github.com/simonw/datasette/issues/1387#issuecomment-873136440,https://api.github.com/repos/simonw/datasette/issues/1387,873136440,MDEyOklzc3VlQ29tbWVudDg3MzEzNjQ0MA==,9599,simonw,2021-07-02T17:01:48Z,2021-07-02T17:01:48Z,OWNER,"Here's what's happening: https://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/views/table.py#L827-L829 This is being run through `absolute_url()` - defined here: https://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/app.py#L633-L637 That's because the `next_url` in the JSON needs to be a full URL that a client can retrieve - as opposed to the other links on that page which are all relative links that start with `/`: https://github.com/simonw/datasette/blob/ea627baccf980d7d8ebc9e1ffff1fe34d556e56f/datasette/templates/_table.html#L11-L15 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",935930820,absolute_url() behind a proxy assembles incorrect http://127.0.0.1:8001/ URLs, https://github.com/simonw/datasette/issues/1387#issuecomment-873134866,https://api.github.com/repos/simonw/datasette/issues/1387,873134866,MDEyOklzc3VlQ29tbWVudDg3MzEzNDg2Ng==,9599,simonw,2021-07-02T16:58:52Z,2021-07-02T16:58:52Z,OWNER,What's weird here is that the URL itself is correct - it starts with `/collection-analysis/` as expected - but the hostname is wrong.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",935930820,absolute_url() behind a proxy assembles incorrect http://127.0.0.1:8001/ URLs, https://github.com/simonw/datasette/issues/1101#issuecomment-869812567,https://api.github.com/repos/simonw/datasette/issues/1101,869812567,MDEyOklzc3VlQ29tbWVudDg2OTgxMjU2Nw==,9599,simonw,2021-06-28T16:06:57Z,2021-06-28T16:07:24Z,OWNER,"Relevant blog post: https://simonwillison.net/2021/Jun/25/streaming-large-api-responses/ - including notes on efficiently streaming formats with some kind of separator in between the records (regular JSON). > Some export formats are friendlier for streaming than others. CSV and TSV are pretty easy to stream, as is newline-delimited JSON. > > Regular JSON requires a bit more thought: you can output a `[` character, then output each row in a stream with a comma suffix, then skip the comma for the last row and output a `]`. Doing that requires peeking ahead (looping two at a time) to verify that you haven't yet reached the end. > > Or... Martin De Wulf [pointed out](https://twitter.com/madewulf/status/1405559088994467844) that you can output the first row, then output every other row with a preceeding comma---which avoids the whole ""iterate two at a time"" problem entirely.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749283032,register_output_renderer() should support streaming data, https://github.com/simonw/datasette/issues/1384#issuecomment-869075395,https://api.github.com/repos/simonw/datasette/issues/1384,869075395,MDEyOklzc3VlQ29tbWVudDg2OTA3NTM5NQ==,9599,simonw,2021-06-26T23:54:21Z,2021-06-26T23:59:21Z,OWNER,(It may well be that implementing #1168 involves a switch to async metadata),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata, https://github.com/simonw/datasette/issues/1384#issuecomment-869075368,https://api.github.com/repos/simonw/datasette/issues/1384,869075368,MDEyOklzc3VlQ29tbWVudDg2OTA3NTM2OA==,9599,simonw,2021-06-26T23:53:55Z,2021-06-26T23:53:55Z,OWNER,"Great, let's drop fallback then. My instinct at the moment is to ship this plugin hook as-is but with a warning that it may change before Datasette 1.0 - then before 1.0 either figure out an async variant or finish the database-backed metadata concept from #1168 and recommend that as an alternative.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata, https://github.com/simonw/datasette/issues/1384#issuecomment-869071790,https://api.github.com/repos/simonw/datasette/issues/1384,869071790,MDEyOklzc3VlQ29tbWVudDg2OTA3MTc5MA==,9599,simonw,2021-06-26T23:04:12Z,2021-06-26T23:04:12Z,OWNER,"> Hmmm... that's tricky, since one of the most obvious ways to use this hook is to load metadata from database tables using SQL queries. > > @brandonrobertz do you have a working example of using this hook to populate metadata from database tables I can try? Answering my own question: here's how Brandon implements it in his `datasette-live-config` plugin: https://github.com/next-LI/datasette-live-config/blob/72e335e887f1c69c54c6c2441e07148955b0fc9f/datasette_live_config/__init__.py#L50-L160 That's using a completely separate SQLite connection (actually wrapped in `sqlite-utils`) and making blocking synchronous calls to it. This is a pragmatic solution, which works - and likely performs just fine, because SQL queries like this against a small database are so fast that not running them asynchronously isn't actually a problem. But... it's weird. Everywhere else in Datasette land uses `await db.execute(...)` - but here's an example where users are encouraged to use blocking calls instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata, https://github.com/simonw/datasette/issues/1384#issuecomment-869071435,https://api.github.com/repos/simonw/datasette/issues/1384,869071435,MDEyOklzc3VlQ29tbWVudDg2OTA3MTQzNQ==,9599,simonw,2021-06-26T22:59:26Z,2021-06-26T22:59:26Z,OWNER,"The other alternative is to finish the work to build a `_metadata` internal table, see #1168. The idea there was that if we want to support efficient pagination and search across the metadata for thousands of attached tables powering it with a plugin hook doesn't work well - we don't want to call the hook once for every one of 1,000+ tables just to implement the homepage. So instead, all metadata for all attached databases would be loaded into an in-memory database called `_metadata`. Plugins that want to modify stored metadata could then do so by directly writing to that table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata, https://github.com/simonw/datasette/issues/860#issuecomment-869071236,https://api.github.com/repos/simonw/datasette/issues/860,869071236,MDEyOklzc3VlQ29tbWVudDg2OTA3MTIzNg==,9599,simonw,2021-06-26T22:56:28Z,2021-06-26T22:56:28Z,OWNER,This work is continuing in #1384.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642651572,Plugin hook for instance/database/table metadata, https://github.com/simonw/datasette/issues/1384#issuecomment-869071167,https://api.github.com/repos/simonw/datasette/issues/1384,869071167,MDEyOklzc3VlQ29tbWVudDg2OTA3MTE2Nw==,9599,simonw,2021-06-26T22:55:36Z,2021-06-26T22:55:36Z,OWNER,"Just realized I already have an issue open for this, at #860. I'm going to close that and continue work on this in this issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata, https://github.com/simonw/datasette/issues/1384#issuecomment-869070941,https://api.github.com/repos/simonw/datasette/issues/1384,869070941,MDEyOklzc3VlQ29tbWVudDg2OTA3MDk0MQ==,9599,simonw,2021-06-26T22:53:34Z,2021-06-26T22:53:34Z,OWNER,"The `await` thing is worrying me a lot - it feels like this plugin hook is massively less useful if it can't make it's own DB queries and generally do asynchronous stuff - but I'd also like not to break every existing plugin that calls `datasette.metadata(...)`. One solution that could work: introduce a new method, maybe `await datasette.get_metadata(...)`, which uses this plugin hook - and keep the existing `datasette.metadata()` method (which doesn't call the hook) around. This would ensure existing plugins keep on working. Then, upgrade those plugins separately - with the goal of deprecating and removing `.metadata()` entirely in Datasette 1.0 - having upgraded the plugins in the meantime.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata, https://github.com/simonw/datasette/issues/1384#issuecomment-869070348,https://api.github.com/repos/simonw/datasette/issues/1384,869070348,MDEyOklzc3VlQ29tbWVudDg2OTA3MDM0OA==,9599,simonw,2021-06-26T22:46:18Z,2021-06-26T22:46:18Z,OWNER,"Here's where the plugin hook is called, demonstrating the `fallback=` argument: https://github.com/simonw/datasette/blob/05a312caf3debb51aa1069939923a49e21cd2bd1/datasette/app.py#L426-L472 I'm not convinced of the use-case for passing `fallback=` to the hook here - is there a reason a plugin might care whether fallback is `True` or `False`, seeing as the `metadata()` method already respects that fallback logic on line 459?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata, https://github.com/simonw/datasette/issues/1384#issuecomment-869070076,https://api.github.com/repos/simonw/datasette/issues/1384,869070076,MDEyOklzc3VlQ29tbWVudDg2OTA3MDA3Ng==,9599,simonw,2021-06-26T22:42:21Z,2021-06-26T22:42:21Z,OWNER,"Hmmm... that's tricky, since one of the most obvious ways to use this hook is to load metadata from database tables using SQL queries. @brandonrobertz do you have a working example of using this hook to populate metadata from database tables I can try?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata, https://github.com/simonw/datasette/issues/1384#issuecomment-869069926,https://api.github.com/repos/simonw/datasette/issues/1384,869069926,MDEyOklzc3VlQ29tbWVudDg2OTA2OTkyNg==,9599,simonw,2021-06-26T22:40:15Z,2021-06-26T22:40:53Z,OWNER,"The documentation says: > **datasette**: You can use this to access plugin configuration options via `datasette.plugin_config(your_plugin_name)`, or to execute SQL queries. That's not accurate: since the plugin hook is a regular function, not an awaitable, you can't use it to run `await db.execute(...)` so you can't execute SQL queries. I can fix this with the await-me-maybe pattern, used for other plugin hooks: https://simonwillison.net/2020/Sep/2/await-me-maybe/ BUT... that requires changing the `ds.metadata()` function to be awaitable, which will affect every existing plugn that uses that documented internal method!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata, https://github.com/simonw/datasette/issues/1384#issuecomment-869069768,https://api.github.com/repos/simonw/datasette/issues/1384,869069768,MDEyOklzc3VlQ29tbWVudDg2OTA2OTc2OA==,9599,simonw,2021-06-26T22:37:53Z,2021-06-26T22:37:53Z,OWNER,The documentation doesn't describe the ``fallback`` argument at the moment.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata, https://github.com/simonw/datasette/issues/1384#issuecomment-869069655,https://api.github.com/repos/simonw/datasette/issues/1384,869069655,MDEyOklzc3VlQ29tbWVudDg2OTA2OTY1NQ==,9599,simonw,2021-06-26T22:36:14Z,2021-06-26T22:37:37Z,OWNER,"Documentation for the new hook is now live at https://docs.datasette.io/en/latest/plugin_hooks.html#get-metadata-datasette-key-database-table-fallback Link to the current snapshot of that documentation: https://github.com/simonw/datasette/blob/05a312caf3debb51aa1069939923a49e21cd2bd1/docs/plugin_hooks.rst#get-metadata-datasette-key-database-table-fallback","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata, https://github.com/simonw/datasette/pull/1368#issuecomment-869068554,https://api.github.com/repos/simonw/datasette/issues/1368,869068554,MDEyOklzc3VlQ29tbWVudDg2OTA2ODU1NA==,9599,simonw,2021-06-26T22:23:57Z,2021-06-26T22:23:57Z,OWNER,The only test failure is Black. I'm going to merge this and then reformat.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913865304,DRAFT: A new plugin hook for dynamic metadata, https://github.com/simonw/sqlite-utils/issues/37#issuecomment-868881190,https://api.github.com/repos/simonw/sqlite-utils/issues/37,868881190,MDEyOklzc3VlQ29tbWVudDg2ODg4MTE5MA==,9599,simonw,2021-06-25T23:24:28Z,2021-06-25T23:24:28Z,OWNER,Maybe I could release a separate Python package `types-sqlite-utils-numpy` which adds an over-ridden type definition that includes the `numpy` types?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",465815372,Experiment with type hints, https://github.com/simonw/sqlite-utils/issues/37#issuecomment-868881033,https://api.github.com/repos/simonw/sqlite-utils/issues/37,868881033,MDEyOklzc3VlQ29tbWVudDg2ODg4MTAzMw==,9599,simonw,2021-06-25T23:23:49Z,2021-06-25T23:23:49Z,OWNER,"Twitter conversation about how to add types to the `.create_table(columns=)` parameter: https://twitter.com/simonw/status/1408532867592818693 > Anyone know how to write a mypy type definition for this? > > {""id"": int, ""name"": str, ""image"": bytes, ""weight"": float} > > It's a dict where keys are strings and values are one of int/str/bytes/float (weird API design I know, but I designed this long before I was thinking about mypy) Looks like this could work: ```python def create_table( self, name, columns: Dict[str, Union[Type[int], Type[bytes], Type[str], Type[float]]], pk=None, foreign_keys=None, column_order=None, not_null=None, defaults=None, hash_id=None, extracts=None, ): ``` Except... that method can optionally also accept `numpy` types if `numpy` is installed. I don't know if it's possible to dynamically change a signature based on an import, since `mypy` is a static type analyzer and doesn't ever execute the code.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",465815372,Experiment with type hints, https://github.com/simonw/sqlite-utils/pull/293#issuecomment-868728092,https://api.github.com/repos/simonw/sqlite-utils/issues/293,868728092,MDEyOklzc3VlQ29tbWVudDg2ODcyODA5Mg==,9599,simonw,2021-06-25T17:39:35Z,2021-06-25T17:39:35Z,OWNER,Here's more about this problem: https://github.com/numpy/numpy/issues/15947,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",929748885,Test against Python 3.10-dev, https://github.com/simonw/sqlite-utils/pull/293#issuecomment-868134040,https://api.github.com/repos/simonw/sqlite-utils/issues/293,868134040,MDEyOklzc3VlQ29tbWVudDg2ODEzNDA0MA==,9599,simonw,2021-06-25T01:49:44Z,2021-06-25T01:50:33Z,OWNER,"Test failed on 3.10 with `numpy` on macOS: ``` sqlite_utils/__init__.py:1: in 11 from .db import Database 12 sqlite_utils/db.py:48: in 13 import numpy as np # type: ignore 14 ../../../hostedtoolcache/Python/3.10.0-beta.3/x64/lib/python3.10/site-packages/numpy/__init__.py:391: in 15 raise RuntimeError(msg) 16 E RuntimeError: Polyfit sanity test emitted a warning, most likely due to using a buggy Accelerate backend. If you compiled yourself, more information is available at https://numpy.org/doc/stable/user/building.html#accelerated-blas-lapack-libraries Otherwise report this to the vendor that provided NumPy. 17 E RankWarning: Polyfit may be poorly conditioned 18 Error: Process completed with exit code 4. ```","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 1}",929748885,Test against Python 3.10-dev, https://github.com/simonw/sqlite-utils/issues/290#issuecomment-868021624,https://api.github.com/repos/simonw/sqlite-utils/issues/290,868021624,MDEyOklzc3VlQ29tbWVudDg2ODAyMTYyNA==,9599,simonw,2021-06-24T23:17:38Z,2021-06-24T23:17:38Z,OWNER,The new documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#executing-queries,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",926777310,`db.query()` method (renamed `db.execute_returning_dicts()`), https://github.com/simonw/datasette/issues/1377#issuecomment-867209791,https://api.github.com/repos/simonw/datasette/issues/1377,867209791,MDEyOklzc3VlQ29tbWVudDg2NzIwOTc5MQ==,9599,simonw,2021-06-23T22:51:32Z,2021-06-23T22:51:32Z,OWNER,Documentation: https://docs.datasette.io/en/latest/plugin_hooks.html#skip-csrf-datasette-scope,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",920884085,Mechanism for plugins to exclude certain paths from CSRF checks, https://github.com/simonw/datasette/pull/1368#issuecomment-867102944,https://api.github.com/repos/simonw/datasette/issues/1368,867102944,MDEyOklzc3VlQ29tbWVudDg2NzEwMjk0NA==,9599,simonw,2021-06-23T19:32:01Z,2021-06-23T19:32:01Z,OWNER,"Yes, let's move ahead with getting this into an alpha.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913865304,DRAFT: A new plugin hook for dynamic metadata, https://github.com/simonw/sqlite-utils/issues/291#issuecomment-866466388,https://api.github.com/repos/simonw/sqlite-utils/issues/291,866466388,MDEyOklzc3VlQ29tbWVudDg2NjQ2NjM4OA==,9599,simonw,2021-06-23T02:10:24Z,2021-06-23T02:10:24Z,OWNER,This already helped me spot a bug in a test: https://github.com/simonw/sqlite-utils/commit/90e211e3e2f36d2ff911ecf1afe4470ff45c7c0d#diff-4e8715c7a425ee52e74b7df4d34efd32e8c92f3e60bd51bc2e1ad5943b82032eL1160-L1161,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",927766296,Adopt flake8, https://github.com/simonw/sqlite-utils/issues/291#issuecomment-866461926,https://api.github.com/repos/simonw/sqlite-utils/issues/291,866461926,MDEyOklzc3VlQ29tbWVudDg2NjQ2MTkyNg==,9599,simonw,2021-06-23T01:59:57Z,2021-06-23T01:59:57Z,OWNER,"That shouldn't be failing: it's complaining about these: https://github.com/simonw/sqlite-utils/blob/02898bf7af4a4e484ecc8ec852d5fee98463277b/tests/test_register_function.py#L56-L67 But I added `# noqa: F811` to them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",927766296,Adopt flake8, https://github.com/simonw/sqlite-utils/issues/290#issuecomment-865511810,https://api.github.com/repos/simonw/sqlite-utils/issues/290,865511810,MDEyOklzc3VlQ29tbWVudDg2NTUxMTgxMA==,9599,simonw,2021-06-22T04:07:34Z,2021-06-22T18:26:21Z,OWNER,"That documentation section is pretty weak at the moment - here's the whole thing: > ### Executing queries > > The `db.execute()` and `db.executescript()` methods provide wrappers around `.execute()` and `.executescript()` on the underlying SQLite connection. These wrappers log to the tracer function if one has been registered. > ```python > db = Database(memory=True) > db[""dogs""].insert({""name"": ""Cleo""}) > db.execute(""update dogs set name = 'Cleopaws'"") > ``` > You can pass parameters as an optional second argument, using either a list or a dictionary. These will be correctly quoted and escaped. > ```python > # Using ? and a list: > db.execute(""update dogs set name = ?"", [""Cleopaws""]) > # Or using :name and a dictionary: > db.execute(""update dogs set name = :name"", {""name"": ""Cleopaws""}) > ``` - Talks about `.execute()` - I want to talk about `.query()` instead - Doesn't clarify that `.execute()` returns a `Cursor` - and assumes you know what to do with one - Doesn't show an example of a `select` query at all - The ""tracer function"" bit is confusing (should at least link to docs further down) - For `UPDATE` should show how to access the number of rows modified (probably using `.execute()` there) It does at least cover the two types of parameters, though that could be bulked out.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",926777310,`db.query()` method (renamed `db.execute_returning_dicts()`), https://github.com/simonw/sqlite-utils/issues/37#issuecomment-509681590,https://api.github.com/repos/simonw/sqlite-utils/issues/37,509681590,MDEyOklzc3VlQ29tbWVudDUwOTY4MTU5MA==,9599,simonw,2019-07-09T15:07:12Z,2021-06-22T18:17:53Z,OWNER,"Here's a magic incantation for generating types detected through running the tests with https://github.com/Instagram/MonkeyType ``` pip install pytest-monkeytype pytest --monkeytype-output=./monkeytype.sqlite3 monkeytype list-modules monkeytype apply sqlite_utils.utils monkeytype apply sqlite_utils.cli monkeytype apply sqlite_utils.db ``` Here's the result: https://github.com/simonw/sqlite-utils/commit/d18c694fc25b7dd3d76e250c77ddf56d10ddf935","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",465815372,Experiment with type hints, https://github.com/simonw/sqlite-utils/issues/289#issuecomment-866219755,https://api.github.com/repos/simonw/sqlite-utils/issues/289,866219755,MDEyOklzc3VlQ29tbWVudDg2NjIxOTc1NQ==,9599,simonw,2021-06-22T18:13:26Z,2021-06-22T18:13:26Z,OWNER,Thanks @adamchainz - `mypy` now has a foothold on this project (and runs in CI).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925677191,Mypy fixes for rows_from_file(), https://github.com/simonw/sqlite-utils/issues/267#issuecomment-866184260,https://api.github.com/repos/simonw/sqlite-utils/issues/267,866184260,MDEyOklzc3VlQ29tbWVudDg2NjE4NDI2MA==,9599,simonw,2021-06-22T17:26:18Z,2021-06-22T17:27:27Z,OWNER,"If an`.update()` method doesn't work because it collides with an existing dictionary method a `.pk` property could still be nice: ```python for row in db[""sometable""].rows: db[""sometable""].update(row.pk, {""modified"": 1}) ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",915421499,row.update() or row.pk, https://github.com/simonw/sqlite-utils/issues/267#issuecomment-866182655,https://api.github.com/repos/simonw/sqlite-utils/issues/267,866182655,MDEyOklzc3VlQ29tbWVudDg2NjE4MjY1NQ==,9599,simonw,2021-06-22T17:24:03Z,2021-06-22T17:24:03Z,OWNER,"I'm re-opening this as a research task because it may be possible to cleanly implement this using a `dict` subclass - some notes on that here: https://treyhunner.com/2019/04/why-you-shouldnt-inherit-from-list-and-dict-in-python/ Since this would just be for adding methods (and maybe a property for returning the primary keys for a row) the usual disadvantages of subclassing `dict` described in that article shouldn't apply. One catch: dictionaries already have a `.update()` method! So would have to pick another name.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",915421499,row.update() or row.pk, https://github.com/simonw/sqlite-utils/issues/290#issuecomment-865510796,https://api.github.com/repos/simonw/sqlite-utils/issues/290,865510796,MDEyOklzc3VlQ29tbWVudDg2NTUxMDc5Ng==,9599,simonw,2021-06-22T04:04:40Z,2021-06-22T04:04:48Z,OWNER,"Still needs documentation, which will involve rewriting the whole [Executing queries](https://sqlite-utils.datasette.io/en/3.11/python-api.html#executing-queries) section.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",926777310,`db.query()` method (renamed `db.execute_returning_dicts()`), https://github.com/simonw/sqlite-utils/issues/290#issuecomment-865497846,https://api.github.com/repos/simonw/sqlite-utils/issues/290,865497846,MDEyOklzc3VlQ29tbWVudDg2NTQ5Nzg0Ng==,9599,simonw,2021-06-22T03:21:38Z,2021-06-22T03:21:38Z,OWNER,"The Python docs say: https://docs.python.org/3/library/sqlite3.html > To retrieve data after executing a SELECT statement, you can either treat the cursor as an iterator, call the cursor’s `fetchone()` method to retrieve a single matching row, or call `fetchall()` to get a list of the matching rows. Looking at the C source code, both `fetchmany()` and `fetchall()` work under the hood by assembling a Python list: https://github.com/python/cpython/blob/be1cb3214d09d4bf0288bc45f3c1f167f67e4514/Modules/_sqlite/cursor.c#L907-L972 - see calls to `PyList_Append()` So it looks like the most efficient way to iterate over a cursor may well be `for row in cursor:` - which I think calls this C function: https://github.com/python/cpython/blob/be1cb3214d09d4bf0288bc45f3c1f167f67e4514/Modules/_sqlite/cursor.c#L813-L876","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",926777310,`db.query()` method (renamed `db.execute_returning_dicts()`), https://github.com/simonw/sqlite-utils/issues/290#issuecomment-865495370,https://api.github.com/repos/simonw/sqlite-utils/issues/290,865495370,MDEyOklzc3VlQ29tbWVudDg2NTQ5NTM3MA==,9599,simonw,2021-06-22T03:14:30Z,2021-06-22T03:14:30Z,OWNER,"One small problem with the existing method: https://github.com/simonw/sqlite-utils/blob/8cedc6a8b29180e68326f6b76f249d5e39e4b591/sqlite_utils/db.py#L362-L365 It returns a full list, but what if the user would rather have a generator they can iterate over without loading the results into memory in one go?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",926777310,`db.query()` method (renamed `db.execute_returning_dicts()`), https://github.com/simonw/sqlite-utils/issues/290#issuecomment-865491922,https://api.github.com/repos/simonw/sqlite-utils/issues/290,865491922,MDEyOklzc3VlQ29tbWVudDg2NTQ5MTkyMg==,9599,simonw,2021-06-22T03:05:35Z,2021-06-22T03:05:35Z,OWNER,"Potential names: - `db.query(sql)` - it's weird to have both this and `db.execute()` but it is at least short and memorable - `db.sql(sql)` - `db.execute_d(sql)` - ugly - `db.execute_dicts(sql)` - confusing - `db.execute_sql(sql)` - easily confused with `db.execute(sql)` I think `db.query(sql)` may be the best option here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",926777310,`db.query()` method (renamed `db.execute_returning_dicts()`), https://github.com/simonw/datasette/pull/1368#issuecomment-865160132,https://api.github.com/repos/simonw/datasette/issues/1368,865160132,MDEyOklzc3VlQ29tbWVudDg2NTE2MDEzMg==,9599,simonw,2021-06-21T16:07:06Z,2021-06-21T16:08:48Z,OWNER,"A few tests failed - Black, the test that checks the docs mention the new hook - the most interesting failing test looks like this one: ``` updated_metadata[""databases""][""fixtures""][""queries""][""magic_parameters""][ ""allow"" ] = (allow if ""query"" in permissions else deny) > cascade_app_client.ds._metadata = updated_metadata E AttributeError: can't set attribute ``` From https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/tests/test_permissions.py#L439-L467 This test is directly manipulating `_metadata` purely for the purposes of simulating different permissions - I think updating it to manipulate `_local_metadata` instead would fix that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913865304,DRAFT: A new plugin hook for dynamic metadata, https://github.com/simonw/sqlite-utils/issues/289#issuecomment-864609271,https://api.github.com/repos/simonw/sqlite-utils/issues/289,864609271,MDEyOklzc3VlQ29tbWVudDg2NDYwOTI3MQ==,9599,simonw,2021-06-20T20:42:07Z,2021-06-20T20:42:07Z,OWNER,"Wow, thank you! I didn't know about `typing.cast()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925677191,Mypy fixes for rows_from_file(), https://github.com/simonw/sqlite-utils/issues/286#issuecomment-864594956,https://api.github.com/repos/simonw/sqlite-utils/issues/286,864594956,MDEyOklzc3VlQ29tbWVudDg2NDU5NDk1Ng==,9599,simonw,2021-06-20T18:38:05Z,2021-06-20T18:38:05Z,OWNER,3.10 is out in Homebrew now (they turn that around so fast): https://formulae.brew.sh/formula/sqlite-utils,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925487946,Add installation instructions, https://github.com/simonw/datasette/issues/1382#issuecomment-864480051,https://api.github.com/repos/simonw/datasette/issues/1382,864480051,MDEyOklzc3VlQ29tbWVudDg2NDQ4MDA1MQ==,9599,simonw,2021-06-20T00:20:06Z,2021-06-20T00:21:02Z,OWNER,"Yes you can - thanks for pointing this out, I've added a comment to the `install.sh` script in the `datasette-csvs` Glitch project: ```bash pip3 install -U --no-cache-dir -r requirements.txt --user && \ mkdir -p .data && \ rm .data/data.db || true && \ for f in *.csv do # Add --encoding=latin-1 to the following if your CSVs use a different encoding: sqlite-utils insert .data/data.db ${f%.*} $f --csv done ``` So if you edit that file in your own project and change the line to this: sqlite-utils insert .data/data.db ${f%.*} $f --csv --encoding=iso-8859-1 It should fix this for you.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925406964,Datasette with Glitch - is it possible to use CSV with ISO-8859-1 encoding?, https://github.com/simonw/sqlite-utils/issues/272#issuecomment-864476167,https://api.github.com/repos/simonw/sqlite-utils/issues/272,864476167,MDEyOklzc3VlQ29tbWVudDg2NDQ3NjE2Nw==,9599,simonw,2021-06-19T23:36:48Z,2021-06-19T23:36:48Z,OWNER,Wrote this up on my blog here: https://simonwillison.net/2021/Jun/19/sqlite-utils-memory/ - with a video demo here: https://www.youtube.com/watch?v=OUjd0rkc678,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command", https://github.com/simonw/sqlite-utils/issues/284#issuecomment-864419283,https://api.github.com/repos/simonw/sqlite-utils/issues/284,864419283,MDEyOklzc3VlQ29tbWVudDg2NDQxOTI4Mw==,9599,simonw,2021-06-19T15:15:34Z,2021-06-19T15:15:34Z,OWNER,"I think this code is at fault: https://github.com/simonw/sqlite-utils/blob/5b257949d996fe43dc5d218d4308b88796a90740/sqlite_utils/db.py#L1017-L1023 It's using `.pks` which adds `rowid` if it's missing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925320167,.transform(types=) turns rowid into a concrete column, https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864418795,https://api.github.com/repos/simonw/sqlite-utils/issues/285,864418795,MDEyOklzc3VlQ29tbWVudDg2NDQxODc5NQ==,9599,simonw,2021-06-19T15:11:05Z,2021-06-19T15:11:14Z,OWNER,"Actually I'm going to go with `use_rowid` instead - because the table doesn't inherently use a rowid itself, but you should use one if you want to query it in a way that gives you back a primary key.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925410305,Introspection property for telling if a table is a rowid table, https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864418188,https://api.github.com/repos/simonw/sqlite-utils/issues/285,864418188,MDEyOklzc3VlQ29tbWVudDg2NDQxODE4OA==,9599,simonw,2021-06-19T15:05:53Z,2021-06-19T15:05:53Z,OWNER,"```python @property def uses_rowid(self): return not any(column for column in self.columns if column.is_pk) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925410305,Introspection property for telling if a table is a rowid table, https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864417808,https://api.github.com/repos/simonw/sqlite-utils/issues/285,864417808,MDEyOklzc3VlQ29tbWVudDg2NDQxNzgwOA==,9599,simonw,2021-06-19T15:03:00Z,2021-06-19T15:03:00Z,OWNER,I think I like `table.uses_rowid` best - it reads well.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925410305,Introspection property for telling if a table is a rowid table, https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864417765,https://api.github.com/repos/simonw/sqlite-utils/issues/285,864417765,MDEyOklzc3VlQ29tbWVudDg2NDQxNzc2NQ==,9599,simonw,2021-06-19T15:02:42Z,2021-06-19T15:02:42Z,OWNER,"Some options: - `table.rowid_only` - `table.rowid_as_pk` - `table.no_pks` - `table.no_pk` - `table.uses_rowid` - `table.use_rowid`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925410305,Introspection property for telling if a table is a rowid table, https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864417493,https://api.github.com/repos/simonw/sqlite-utils/issues/285,864417493,MDEyOklzc3VlQ29tbWVudDg2NDQxNzQ5Mw==,9599,simonw,2021-06-19T15:00:43Z,2021-06-19T15:00:43Z,OWNER,"I have to be careful about the language I use here. Here's the official definition: https://www.sqlite.org/rowidtable.html > A ""rowid table"" is any table in an SQLite schema that > > - is *not* a [virtual table](https://www.sqlite.org/vtab.html), and > - is *not* a [WITHOUT ROWID](https://www.sqlite.org/withoutrowid.html) table. > > Most tables in a typical SQLite database schema are rowid tables. > > Rowid tables are distinguished by the fact that they all have a unique, non-NULL, signed 64-bit integer [rowid](https://www.sqlite.org/lang_createtable.html#rowid) that is used as the access key for the data in the underlying [B-tree](https://www.sqlite.org/fileformat2.html#btree) storage engine. So it's not correct to call a table a ""rowid table"" only if it is missing its own primary keys. Maybe `table.has_rowid` is the right language to use here? No, that's no good - because tables with their own primary keys usually also have a rowid.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925410305,Introspection property for telling if a table is a rowid table, https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864417133,https://api.github.com/repos/simonw/sqlite-utils/issues/285,864417133,MDEyOklzc3VlQ29tbWVudDg2NDQxNzEzMw==,9599,simonw,2021-06-19T14:57:36Z,2021-06-19T14:57:36Z,OWNER,"So the logic is: ```python [column.name for column in self.columns if column.is_pk] ``` I need to decide on a property name. Existing names are documented here: https://sqlite-utils.datasette.io/en/stable/python-api.html#introspection","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925410305,Introspection property for telling if a table is a rowid table, https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864417031,https://api.github.com/repos/simonw/sqlite-utils/issues/285,864417031,MDEyOklzc3VlQ29tbWVudDg2NDQxNzAzMQ==,9599,simonw,2021-06-19T14:56:45Z,2021-06-19T14:56:45Z,OWNER,"```pycon >>> db = sqlite_utils.Database(memory=True) >>> db[""rowid_table""].insert({""name"": ""Cleo""})

>>> db[""regular_table""].insert({""id"": 1, ""name"": ""Cleo""}, pk=""id"")
>>> db[""rowid_table""].pks ['rowid'] >>> db[""regular_table""].pks ['id'] ``` But that's because the `.pks` property hides the difference: https://github.com/simonw/sqlite-utils/blob/dc94f4bb8cfe922bb2f9c89f8f0f29092ea63133/sqlite_utils/db.py#L805-L810 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925410305,Introspection property for telling if a table is a rowid table, https://github.com/simonw/sqlite-utils/issues/284#issuecomment-864416911,https://api.github.com/repos/simonw/sqlite-utils/issues/284,864416911,MDEyOklzc3VlQ29tbWVudDg2NDQxNjkxMQ==,9599,simonw,2021-06-19T14:55:45Z,2021-06-19T14:55:45Z,OWNER,"https://github.com/simonw/sqlite-utils/blob/dc94f4bb8cfe922bb2f9c89f8f0f29092ea63133/sqlite_utils/db.py#L805-L810 So I can indeed detect a `rowid` table by looking for no `is_pk` columns.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925320167,.transform(types=) turns rowid into a concrete column, https://github.com/simonw/sqlite-utils/issues/284#issuecomment-864416785,https://api.github.com/repos/simonw/sqlite-utils/issues/284,864416785,MDEyOklzc3VlQ29tbWVudDg2NDQxNjc4NQ==,9599,simonw,2021-06-19T14:54:41Z,2021-06-19T14:54:41Z,OWNER,"```pycon >>> db = sqlite_utils.Database(memory=True) >>> db[""rowid_table""].insert({""name"": ""Cleo""})
>>> db[""regular_table""].insert({""id"": 1, ""name"": ""Cleo""}, pk=""id"")
>>> db[""rowid_table""].pks ['rowid'] >>> db[""regular_table""].pks ['id'] ``` I think I need an introspection property for working out if a table is a `rowid` table or not.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925320167,.transform(types=) turns rowid into a concrete column, https://github.com/simonw/sqlite-utils/issues/283#issuecomment-864416086,https://api.github.com/repos/simonw/sqlite-utils/issues/283,864416086,MDEyOklzc3VlQ29tbWVudDg2NDQxNjA4Ng==,9599,simonw,2021-06-19T14:49:06Z,2021-06-19T14:49:13Z,OWNER,"Once again, this is difficult because of the use of a generator here - `rows_from_file()` only yields rows, so there is no obvious mechanism for it to communicate back to the wrapping code that the detected format was CSV or TSV as opposed to JSON. I'm going to change `rows_from_file()` to return a `(generator, detected_format)` tuple.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925319214,memory: Shouldn't detect types for JSON, https://github.com/simonw/sqlite-utils/issues/284#issuecomment-864358951,https://api.github.com/repos/simonw/sqlite-utils/issues/284,864358951,MDEyOklzc3VlQ29tbWVudDg2NDM1ODk1MQ==,9599,simonw,2021-06-19T05:30:00Z,2021-06-19T05:30:00Z,OWNER,If this can be fixed it will be in the `transform_sql()` method.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925320167,.transform(types=) turns rowid into a concrete column, https://github.com/simonw/sqlite-utils/issues/284#issuecomment-864358680,https://api.github.com/repos/simonw/sqlite-utils/issues/284,864358680,MDEyOklzc3VlQ29tbWVudDg2NDM1ODY4MA==,9599,simonw,2021-06-19T05:27:13Z,2021-06-19T05:27:13Z,OWNER,How easy is it to detect a `rowid` table? Is it as simple as `.pks` returning `None`? If so the documentation should mention that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925320167,.transform(types=) turns rowid into a concrete column, https://github.com/simonw/sqlite-utils/issues/282#issuecomment-864354627,https://api.github.com/repos/simonw/sqlite-utils/issues/282,864354627,MDEyOklzc3VlQ29tbWVudDg2NDM1NDYyNw==,9599,simonw,2021-06-19T04:42:03Z,2021-06-19T04:42:03Z,OWNER,"Demo: curl -s 'https://api.github.com/users/simonw/repos?per_page=100' | \ sqlite-utils memory - 'select sum(size), sum(stargazers_count) from stdin limit 1' [{""sum(size)"": 2042547, ""sum(stargazers_count)"": 6769}] ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925305186,Automatic type detection for CSV data, https://github.com/simonw/sqlite-utils/issues/282#issuecomment-864350407,https://api.github.com/repos/simonw/sqlite-utils/issues/282,864350407,MDEyOklzc3VlQ29tbWVudDg2NDM1MDQwNw==,9599,simonw,2021-06-19T03:52:20Z,2021-06-19T03:52:20Z,OWNER,I'll have an environment variable for `--detect-types` so users who really want that as the default option can turn it on.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925305186,Automatic type detection for CSV data, https://github.com/simonw/sqlite-utils/issues/282#issuecomment-864349123,https://api.github.com/repos/simonw/sqlite-utils/issues/282,864349123,MDEyOklzc3VlQ29tbWVudDg2NDM0OTEyMw==,9599,simonw,2021-06-19T03:36:54Z,2021-06-19T03:36:54Z,OWNER,"I may change the default for `sqlite-utils insert` to detect types if I release `sqlite-utils` 4.0, as a backwards-incompatible change.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925305186,Automatic type detection for CSV data, https://github.com/simonw/sqlite-utils/issues/179#issuecomment-864349066,https://api.github.com/repos/simonw/sqlite-utils/issues/179,864349066,MDEyOklzc3VlQ29tbWVudDg2NDM0OTA2Ng==,9599,simonw,2021-06-19T03:36:04Z,2021-06-19T03:36:04Z,OWNER,This work is going to happen in #282.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",709577625,sqlite-utils transform/insert --detect-types, https://github.com/simonw/sqlite-utils/issues/282#issuecomment-864348954,https://api.github.com/repos/simonw/sqlite-utils/issues/282,864348954,MDEyOklzc3VlQ29tbWVudDg2NDM0ODk1NA==,9599,simonw,2021-06-19T03:34:42Z,2021-06-19T03:35:46Z,OWNER,"I built some prototype code here for something which looks at every row in a CSV import and records the likely types: https://gist.github.com/simonw/465f9356f175d1cf86957947dff501d4 This could be used by the command-line tools to figure out what `table.transform(types=...)` method to use at the end. This is a different approach to the pure SQL version I tried building in https://github.com/simonw/sqlite-utils/issues/179 - I think this is a better approach though, it's less prone to weird idiosyncrasies of SQLite types, and it's also easy for us to add on to the existing CSV import code in a way that won't require scanning the data twice.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925305186,Automatic type detection for CSV data, https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864330508,https://api.github.com/repos/simonw/sqlite-utils/issues/279,864330508,MDEyOklzc3VlQ29tbWVudDg2NDMzMDUwOA==,9599,simonw,2021-06-19T00:34:24Z,2021-06-19T00:34:24Z,OWNER,"Got this working: % curl 'https://api.github.com/repos/simonw/datasette/issues' | sqlite-utils memory - 'select id from stdin' ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924990677,sqlite-utils memory should handle TSV and JSON in addition to CSV, https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864328927,https://api.github.com/repos/simonw/sqlite-utils/issues/279,864328927,MDEyOklzc3VlQ29tbWVudDg2NDMyODkyNw==,9599,simonw,2021-06-19T00:25:08Z,2021-06-19T00:25:17Z,OWNER,"I tried writing this function with type hints, but eventually gave up: ```python def rows_from_file( fp: BinaryIO, format: Optional[Format] = None, dialect: Optional[Type[csv.Dialect]] = None, encoding: Optional[str] = None, ) -> Generator[dict, None, None]: if format == Format.JSON: decoded = json.load(fp) if isinstance(decoded, dict): decoded = [decoded] if not isinstance(decoded, list): raise RowsFromFileBadJSON(""JSON must be a list or a dictionary"") yield from decoded elif format == Format.CSV: decoded_fp = io.TextIOWrapper(fp, encoding=encoding or ""utf-8-sig"") yield from csv.DictReader(decoded_fp) elif format == Format.TSV: yield from rows_from_file( fp, format=Format.CSV, dialect=csv.excel_tab, encoding=encoding ) elif format is None: # Detect the format, then call this recursively buffered = io.BufferedReader(fp, buffer_size=4096) first_bytes = buffered.peek(2048).strip() if first_bytes[0] in (b""["", b""{""): # TODO: Detect newline-JSON yield from rows_from_file(fp, format=Format.JSON) else: dialect = csv.Sniffer().sniff(first_bytes.decode(encoding, ""ignore"")) yield from rows_from_file( fp, format=Format.CSV, dialect=dialect, encoding=encoding ) else: raise RowsFromFileError(""Bad format"") ``` mypy said: ``` sqlite_utils/utils.py:157: error: Argument 1 to ""BufferedReader"" has incompatible type ""BinaryIO""; expected ""RawIOBase"" sqlite_utils/utils.py:163: error: Argument 1 to ""decode"" of ""bytes"" has incompatible type ""Optional[str]""; expected ""str"" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924990677,sqlite-utils memory should handle TSV and JSON in addition to CSV, https://github.com/simonw/sqlite-utils/issues/281#issuecomment-864323438,https://api.github.com/repos/simonw/sqlite-utils/issues/281,864323438,MDEyOklzc3VlQ29tbWVudDg2NDMyMzQzOA==,9599,simonw,2021-06-18T23:55:06Z,2021-06-18T23:55:06Z,OWNER,"The `-:json` idea is flawed: Click thinks that's the syntax for an option called `:json`. I'm going to do `stdin:json` - which means you can't open a file called `stdin` - but you could use `cat stdin | sqlite-utils memory stdin:json ...` instead which is an OK workaround.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924992318,Mechanism for explicitly stating CSV or JSON or TSV for sqlite-utils memory, https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864208476,https://api.github.com/repos/simonw/sqlite-utils/issues/279,864208476,MDEyOklzc3VlQ29tbWVudDg2NDIwODQ3Ng==,9599,simonw,2021-06-18T18:30:08Z,2021-06-18T23:30:19Z,OWNER,"So maybe this is a function which can either be told the format or, if none is provided, it detects one for itself. ```python def rows_from_file(fp, format=None): # ... yield from rows ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924990677,sqlite-utils memory should handle TSV and JSON in addition to CSV, https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864207841,https://api.github.com/repos/simonw/sqlite-utils/issues/279,864207841,MDEyOklzc3VlQ29tbWVudDg2NDIwNzg0MQ==,9599,simonw,2021-06-18T18:28:40Z,2021-06-18T18:28:46Z,OWNER,"```python def detect_format(fp): # ... return ""csv"", fp, dialect # or return ""json"", fp, parsed_data # or return ""json-nl"", fp, docs ``` The mixed return types here are ugly. In all of these cases what we really want is to return a generator of `{...}` objects. So maybe it returns that instead. ```python def filepointer_to_documents(fp): # ... yield from documents ``` I can refactor `sqlite-utils insert` to use this new code too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924990677,sqlite-utils memory should handle TSV and JSON in addition to CSV, https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864206308,https://api.github.com/repos/simonw/sqlite-utils/issues/279,864206308,MDEyOklzc3VlQ29tbWVudDg2NDIwNjMwOA==,9599,simonw,2021-06-18T18:25:04Z,2021-06-18T18:25:04Z,OWNER,"Or... since I'm not using a streaming JSON parser at the moment, if I think something is JSON I can load the entire thing into memory to validate it. I still need to detect newline-delimited JSON. For that I can consume the first line of the input to see if it's a valid JSON object, then maybe sniff the second line too? This does mean that if the input is a single line of GIANT JSON it will all be consumed into memory at once, but that's going to happen anyway. So I need a function which, given a file pointer, consumes from it, detects the type, then returns that type AND a file pointer to the beginning of the file again. I can use `io.BufferedReader` for this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924990677,sqlite-utils memory should handle TSV and JSON in addition to CSV, https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864129273,https://api.github.com/repos/simonw/sqlite-utils/issues/279,864129273,MDEyOklzc3VlQ29tbWVudDg2NDEyOTI3Mw==,9599,simonw,2021-06-18T15:47:47Z,2021-06-18T15:47:47Z,OWNER,"Detecting valid JSON is tricky - just because a stream starts with `[` or `{` doesn't mean the entire stream is valid JSON. You need to parse the entire stream to determine that for sure. One way to solve this would be with a custom state machine. Another would be to use the `ijson` streaming parser - annoyingly it throws the same exception class for invalid JSON for different reasons, but the `e.args[0]` for that exception includes human-readable text about the error - if it's anything other than `parse error: premature EOF` then it probably means the JSON was invalid.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924990677,sqlite-utils memory should handle TSV and JSON in addition to CSV, https://github.com/simonw/sqlite-utils/issues/278#issuecomment-864128489,https://api.github.com/repos/simonw/sqlite-utils/issues/278,864128489,MDEyOklzc3VlQ29tbWVudDg2NDEyODQ4OQ==,9599,simonw,2021-06-18T15:46:24Z,2021-06-18T15:46:24Z,OWNER,A workaround could be to define a bash or zsh alias of some sort.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",923697888,"Support db as first parameter before subcommand, or as environment variable", https://github.com/simonw/sqlite-utils/issues/278#issuecomment-864126781,https://api.github.com/repos/simonw/sqlite-utils/issues/278,864126781,MDEyOklzc3VlQ29tbWVudDg2NDEyNjc4MQ==,9599,simonw,2021-06-18T15:43:19Z,2021-06-18T15:43:19Z,OWNER,"I don't think it's possible to do this without breaking backwards compatibility, unfortunately.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",923697888,"Support db as first parameter before subcommand, or as environment variable", https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864103005,https://api.github.com/repos/simonw/sqlite-utils/issues/279,864103005,MDEyOklzc3VlQ29tbWVudDg2NDEwMzAwNQ==,9599,simonw,2021-06-18T15:04:15Z,2021-06-18T15:04:15Z,OWNER,"To detect JSON, check to see if the stream starts with `[` or `{` - maybe do something more sophisticated than that. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924990677,sqlite-utils memory should handle TSV and JSON in addition to CSV, https://github.com/simonw/sqlite-utils/issues/272#issuecomment-864101267,https://api.github.com/repos/simonw/sqlite-utils/issues/272,864101267,MDEyOklzc3VlQ29tbWVudDg2NDEwMTI2Nw==,9599,simonw,2021-06-18T15:01:41Z,2021-06-18T15:01:41Z,OWNER,I'll split the remaining work out into separate issues.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command", https://github.com/simonw/sqlite-utils/pull/273#issuecomment-864099764,https://api.github.com/repos/simonw/sqlite-utils/issues/273,864099764,MDEyOklzc3VlQ29tbWVudDg2NDA5OTc2NA==,9599,simonw,2021-06-18T14:59:27Z,2021-06-18T14:59:27Z,OWNER,I'm going to merge this as-is and work on the JSON/TSV support in a separate issue.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922099793,sqlite-utils memory command for directly querying CSV/JSON data, https://github.com/simonw/sqlite-utils/pull/277#issuecomment-864092515,https://api.github.com/repos/simonw/sqlite-utils/issues/277,864092515,MDEyOklzc3VlQ29tbWVudDg2NDA5MjUxNQ==,9599,simonw,2021-06-18T14:47:57Z,2021-06-18T14:47:57Z,OWNER,This is a neat improvement.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",923612361,add -h support closes #276, https://github.com/simonw/sqlite-utils/issues/275#issuecomment-862617165,https://api.github.com/repos/simonw/sqlite-utils/issues/275,862617165,MDEyOklzc3VlQ29tbWVudDg2MjYxNzE2NQ==,9599,simonw,2021-06-16T18:34:51Z,2021-06-16T18:34:51Z,OWNER,"Also use this: https://github.com/simonw/datasette/blob/83e9c8bc7585dcc62f200e37c2daefcd669ee05e/codecov.yml And add a badge, as seen on https://github.com/simonw/asgi-csrf","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922955697,Enable code coverage, https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862605436,https://api.github.com/repos/simonw/sqlite-utils/issues/273,862605436,MDEyOklzc3VlQ29tbWVudDg2MjYwNTQzNg==,9599,simonw,2021-06-16T18:19:05Z,2021-06-16T18:19:05Z,OWNER,`--attach` documentation: https://github.com/simonw/sqlite-utils/blob/192dc2c5b73bd836ab8e2e5fed4b36c6ea02f250/docs/cli.rst#joining-in-memory-data-against-existing-databases-using-attach,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922099793,sqlite-utils memory command for directly querying CSV/JSON data, https://github.com/simonw/sqlite-utils/issues/267#issuecomment-862494864,https://api.github.com/repos/simonw/sqlite-utils/issues/267,862494864,MDEyOklzc3VlQ29tbWVudDg2MjQ5NDg2NA==,9599,simonw,2021-06-16T15:51:28Z,2021-06-16T16:26:15Z,OWNER,"I did add a slightly clumsy mechanism recently to help a bit here though: the `pks_and_rows_where()` method: https://sqlite-utils.datasette.io/en/stable/python-api.html#listing-rows-with-their-primary-keys More details in the issue for that feature: #240 The idea here is that if you want to call update you need the primary key for the row - so you can do this: ```python for pk, row in db[""sometable""].pks_and_rows_where(): db[""sometable""].update(pk, {""modified"": 1}"") ``` The `pk` may end up as a single value or a tuple depending on if the table has a compound primary key - but you don't need to worry about that if you use this method as it will return the correct primary key value for you.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",915421499,row.update() or row.pk, https://github.com/simonw/sqlite-utils/issues/131#issuecomment-862495803,https://api.github.com/repos/simonw/sqlite-utils/issues/131,862495803,MDEyOklzc3VlQ29tbWVudDg2MjQ5NTgwMw==,9599,simonw,2021-06-16T15:52:33Z,2021-06-16T15:52:33Z,OWNER,I like `-t` or `--type` for this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675753042,sqlite-utils insert: options for column types, https://github.com/simonw/sqlite-utils/issues/267#issuecomment-862493179,https://api.github.com/repos/simonw/sqlite-utils/issues/267,862493179,MDEyOklzc3VlQ29tbWVudDg2MjQ5MzE3OQ==,9599,simonw,2021-06-16T15:49:13Z,2021-06-16T15:49:13Z,OWNER,"The big challenge here is that the rows returned by this library aren't objects, they are Python dictionaries - so adding methods to them isn't possible without changing the type that is returned by these methods. Part of the philosophy of the library is that it should make it as easy as possible to round-trip between Python dictionaries and SQLite table data, so I don't think adding methods like this is going to fit.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",915421499,row.update() or row.pk, https://github.com/simonw/sqlite-utils/issues/270#issuecomment-862491721,https://api.github.com/repos/simonw/sqlite-utils/issues/270,862491721,MDEyOklzc3VlQ29tbWVudDg2MjQ5MTcyMQ==,9599,simonw,2021-06-16T15:47:06Z,2021-06-16T15:47:06Z,OWNER,"SQLite doesn't have a JSON column type - it has JSON processing functions, but they operate against TEXT columns - so there's nothing I can do here unfortunately.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919314806,Cannot set type JSON, https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862491016,https://api.github.com/repos/simonw/sqlite-utils/issues/272,862491016,MDEyOklzc3VlQ29tbWVudDg2MjQ5MTAxNg==,9599,simonw,2021-06-16T15:46:13Z,2021-06-16T15:46:13Z,OWNER,"Columns from data imported from CSV in this way is currently treated as `TEXT`, which means numeric sorts and suchlike won't work as people might expect. It would be good to do automatic type detection here, see #179.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command", https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862485408,https://api.github.com/repos/simonw/sqlite-utils/issues/272,862485408,MDEyOklzc3VlQ29tbWVudDg2MjQ4NTQwOA==,9599,simonw,2021-06-16T15:38:58Z,2021-06-16T15:39:28Z,OWNER,"Also `sqlite-utils memory` reflects the existing `sqlite-utils :memory:` mechanism, which is a point in its favour. And it helps emphasize that the file you are querying will be loaded into memory, so probably don't try this against a 1GB CSV file.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command", https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862484557,https://api.github.com/repos/simonw/sqlite-utils/issues/272,862484557,MDEyOklzc3VlQ29tbWVudDg2MjQ4NDU1Nw==,9599,simonw,2021-06-16T15:37:51Z,2021-06-16T15:38:34Z,OWNER,"I wonder if there's a better name for this than `sqlite-utils memory`? - `sqlite-utils memory hello.csv ""select * from hello""` - `sqlite-utils mem hello.csv ""select * from hello""` - `sqlite-utils temp hello.csv ""select * from hello""` - `sqlite-utils adhoc hello.csv ""select * from hello""` - `sqlite-utils scratch hello.csv ""select * from hello""` I think `memory` is best. I don't like the others, except for `scratch` which is OK.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command", https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862479704,https://api.github.com/repos/simonw/sqlite-utils/issues/272,862479704,MDEyOklzc3VlQ29tbWVudDg2MjQ3OTcwNA==,9599,simonw,2021-06-16T15:31:31Z,2021-06-16T15:31:31Z,OWNER,"Plus, could I make this change to `sqlite-utils query` without breaking backwards compatibility? Adding a new `sqlite-utils memory` command is completely safe from that perspective.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command", https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862478881,https://api.github.com/repos/simonw/sqlite-utils/issues/272,862478881,MDEyOklzc3VlQ29tbWVudDg2MjQ3ODg4MQ==,9599,simonw,2021-06-16T15:30:24Z,2021-06-16T15:30:24Z,OWNER,"But... `sqlite-utils my.csv ""select * from my""` is a much more compelling initial experience than `sqlite-utils memory my.csv ""select * from my""`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command", https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862475685,https://api.github.com/repos/simonw/sqlite-utils/issues/272,862475685,MDEyOklzc3VlQ29tbWVudDg2MjQ3NTY4NQ==,9599,simonw,2021-06-16T15:26:19Z,2021-06-16T15:29:38Z,OWNER,"Here's a radical idea: what if I combined `sqlite-utils memory` into `sqlite-utils query`? The trick here would be to detect if the arguments passed on the command-line refer to SQLite databases or if they refer to CSV/JSON data that should be imported into temporary tables. Detecting a SQLite database file is actually really easy - they all start with the same binary string: ```pycon >>> open(""my.db"", ""rb"").read(100) b'SQLite format 3\x00... ``` (Need to carefully check that a CSV file with`SQLite format 3` as the first column name doesn't accidentally get interpreted as a SQLite DB though). So then what would the semantics of `sqlite-utils query` (which is also the default command) be? - `sqlite-utils mydb.db ""select * from x""` - `sqlite-utils my.csv ""select * from my""` - `sqlite-utils mydb.db my.csv ""select * from mydb.x join my on ...""` - this is where it gets weird. We can't import the CSV data directly into `mpdb.db` - it's suppose to go into the in-memory database - so now we need to start using database aliases like `mydb.x` because we passed at least one other file? The complexity here is definitely in the handling of a combination of SQLite database files and CSV filenames. Also, `sqlite-utils query` doesn't accept multiple filenames at the moment, so that will change. I'm not 100% sold on this as being better than having a separate `sqlite-utils memory` command, as seen in #273.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command", https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862046009,https://api.github.com/repos/simonw/sqlite-utils/issues/273,862046009,MDEyOklzc3VlQ29tbWVudDg2MjA0NjAwOQ==,9599,simonw,2021-06-16T05:15:38Z,2021-06-16T05:15:38Z,OWNER,"I'm going to add a `--encoding` option - it will affect ALL CSV input files, so if you have CSV files with different encodings you'll need to sort that mess out yourself (likely by importing each CSV file separately into a database using `sqlite-utils insert` with different `--encoding` values).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922099793,sqlite-utils memory command for directly querying CSV/JSON data, https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862045639,https://api.github.com/repos/simonw/sqlite-utils/issues/273,862045639,MDEyOklzc3VlQ29tbWVudDg2MjA0NTYzOQ==,9599,simonw,2021-06-16T05:14:38Z,2021-06-16T05:14:38Z,OWNER,"Can't share much code though since a bunch of that `insert` stuff is specific to that command - showing progress bars, returning errors on illegal option combinations etc.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922099793,sqlite-utils memory command for directly querying CSV/JSON data, https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862045438,https://api.github.com/repos/simonw/sqlite-utils/issues/273,862045438,MDEyOklzc3VlQ29tbWVudDg2MjA0NTQzOA==,9599,simonw,2021-06-16T05:14:00Z,2021-06-16T05:14:00Z,OWNER,I should probably refactor the CSV/JSON/loading stuff into a function in `utils.py` in order to share some of the implementation with the existing `sqlite-utils insert` code: https://github.com/simonw/sqlite-utils/blob/287cdcae8908916687f2ecccc87c38549d004ac6/sqlite_utils/cli.py#L691-L734,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922099793,sqlite-utils memory command for directly querying CSV/JSON data, https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862043974,https://api.github.com/repos/simonw/sqlite-utils/issues/273,862043974,MDEyOklzc3VlQ29tbWVudDg2MjA0Mzk3NA==,9599,simonw,2021-06-16T05:10:12Z,2021-06-16T05:10:12Z,OWNER,"I can stop promoting `:memory:` here and promote `memory` instead: https://github.com/simonw/sqlite-utils/blob/c7234cae8336b8525034e8f917d82dd0699abd42/docs/cli.rst#L83-L86","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922099793,sqlite-utils memory command for directly querying CSV/JSON data, https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862042110,https://api.github.com/repos/simonw/sqlite-utils/issues/273,862042110,MDEyOklzc3VlQ29tbWVudDg2MjA0MjExMA==,9599,simonw,2021-06-16T05:05:51Z,2021-06-16T05:06:11Z,OWNER,"Initial documentation is here: https://github.com/simonw/sqlite-utils/blob/c7234cae8336b8525034e8f917d82dd0699abd42/docs/cli.rst#running-queries-directly-against-csv-data It only talks about CSV at the moment - needs to be updated to mention JSON too once that is implemented.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922099793,sqlite-utils memory command for directly querying CSV/JSON data, https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862040971,https://api.github.com/repos/simonw/sqlite-utils/issues/272,862040971,MDEyOklzc3VlQ29tbWVudDg2MjA0MDk3MQ==,9599,simonw,2021-06-16T05:02:56Z,2021-06-16T05:02:56Z,OWNER,Moving this to a PR.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command", https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862040906,https://api.github.com/repos/simonw/sqlite-utils/issues/272,862040906,MDEyOklzc3VlQ29tbWVudDg2MjA0MDkwNg==,9599,simonw,2021-06-16T05:02:47Z,2021-06-16T05:02:47Z,OWNER,"Got a prototype working! ``` % curl -s 'https://fivethirtyeight.datasettes.com/polls/president_approval_polls.csv?_size=max&_stream=1' | sqlite-utils memory - 'select * from t limit 5' --nl {""rowid"": ""1"", ""question_id"": ""139304"", ""poll_id"": ""74225"", ""state"": """", ""politician_id"": ""11"", ""politician"": ""Donald Trump"", ""pollster_id"": ""568"", ""pollster"": ""YouGov"", ""sponsor_ids"": ""352"", ""sponsors"": ""Economist"", ""display_name"": ""YouGov"", ""pollster_rating_id"": ""391"", ""pollster_rating_name"": ""YouGov"", ""fte_grade"": ""B"", ""sample_size"": ""1500"", ""population"": ""a"", ""population_full"": ""a"", ""methodology"": ""Online"", ""start_date"": ""1/16/21"", ""end_date"": ""1/19/21"", ""sponsor_candidate"": """", ""tracking"": """", ""created_at"": ""1/20/21 10:18"", ""notes"": """", ""url"": ""https://docs.cdn.yougov.com/y9zsit5bzd/weeklytrackingreport.pdf"", ""source"": ""538"", ""yes"": ""42.0"", ""no"": ""53.0""} {""rowid"": ""2"", ""question_id"": ""139305"", ""poll_id"": ""74225"", ""state"": """", ""politician_id"": ""11"", ""politician"": ""Donald Trump"", ""pollster_id"": ""568"", ""pollster"": ""YouGov"", ""sponsor_ids"": ""352"", ""sponsors"": ""Economist"", ""display_name"": ""YouGov"", ""pollster_rating_id"": ""391"", ""pollster_rating_name"": ""YouGov"", ""fte_grade"": ""B"", ""sample_size"": ""1155"", ""population"": ""rv"", ""population_full"": ""rv"", ""methodology"": ""Online"", ""start_date"": ""1/16/21"", ""end_date"": ""1/19/21"", ""sponsor_candidate"": """", ""tracking"": """", ""created_at"": ""1/20/21 10:18"", ""notes"": """", ""url"": ""https://docs.cdn.yougov.com/y9zsit5bzd/weeklytrackingreport.pdf"", ""source"": ""538"", ""yes"": ""44.0"", ""no"": ""55.0""} {""rowid"": ""3"", ""question_id"": ""139306"", ""poll_id"": ""74226"", ""state"": """", ""politician_id"": ""11"", ""politician"": ""Donald Trump"", ""pollster_id"": ""23"", ""pollster"": ""American Research Group"", ""sponsor_ids"": """", ""sponsors"": """", ""display_name"": ""American Research Group"", ""pollster_rating_id"": ""9"", ""pollster_rating_name"": ""American Research Group"", ""fte_grade"": ""B"", ""sample_size"": ""1100"", ""population"": ""a"", ""population_full"": ""a"", ""methodology"": ""Live Phone"", ""start_date"": ""1/16/21"", ""end_date"": ""1/19/21"", ""sponsor_candidate"": """", ""tracking"": """", ""created_at"": ""1/20/21 10:18"", ""notes"": """", ""url"": ""https://americanresearchgroup.com/economy/"", ""source"": ""538"", ""yes"": ""30.0"", ""no"": ""66.0""} {""rowid"": ""4"", ""question_id"": ""139307"", ""poll_id"": ""74226"", ""state"": """", ""politician_id"": ""11"", ""politician"": ""Donald Trump"", ""pollster_id"": ""23"", ""pollster"": ""American Research Group"", ""sponsor_ids"": """", ""sponsors"": """", ""display_name"": ""American Research Group"", ""pollster_rating_id"": ""9"", ""pollster_rating_name"": ""American Research Group"", ""fte_grade"": ""B"", ""sample_size"": ""990"", ""population"": ""rv"", ""population_full"": ""rv"", ""methodology"": ""Live Phone"", ""start_date"": ""1/16/21"", ""end_date"": ""1/19/21"", ""sponsor_candidate"": """", ""tracking"": """", ""created_at"": ""1/20/21 10:18"", ""notes"": """", ""url"": ""https://americanresearchgroup.com/economy/"", ""source"": ""538"", ""yes"": ""29.0"", ""no"": ""67.0""} {""rowid"": ""5"", ""question_id"": ""139298"", ""poll_id"": ""74224"", ""state"": """", ""politician_id"": ""11"", ""politician"": ""Donald Trump"", ""pollster_id"": ""1528"", ""pollster"": ""AtlasIntel"", ""sponsor_ids"": """", ""sponsors"": """", ""display_name"": ""AtlasIntel"", ""pollster_rating_id"": ""546"", ""pollster_rating_name"": ""AtlasIntel"", ""fte_grade"": ""B/C"", ""sample_size"": ""5188"", ""population"": ""a"", ""population_full"": ""a"", ""methodology"": ""Online"", ""start_date"": ""1/15/21"", ""end_date"": ""1/19/21"", ""sponsor_candidate"": """", ""tracking"": """", ""created_at"": ""1/19/21 21:52"", ""notes"": """", ""url"": ""https://projects.fivethirtyeight.com/polls/20210119_US_Atlas2.pdf"", ""source"": ""538"", ""yes"": ""44.6"", ""no"": ""53.9""} ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command", https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862018937,https://api.github.com/repos/simonw/sqlite-utils/issues/272,862018937,MDEyOklzc3VlQ29tbWVudDg2MjAxODkzNw==,9599,simonw,2021-06-16T03:59:28Z,2021-06-16T04:00:05Z,OWNER,"Mainly for debugging purposes it would be useful to be able to save the created in-memory database back to a file again later. This could be done with: sqlite-utils memory blah.csv --save saved.db Can use `.iterdump()` to implement this: https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.iterdump Maybe instead (or as-well-as) offer `--dump` which dumps out the SQL from that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command", https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861989987,https://api.github.com/repos/simonw/sqlite-utils/issues/272,861989987,MDEyOklzc3VlQ29tbWVudDg2MTk4OTk4Nw==,9599,simonw,2021-06-16T02:34:21Z,2021-06-16T02:34:21Z,OWNER,"The documentation already covers this ``` $ sqlite-utils :memory: ""select sqlite_version()"" [{""sqlite_version()"": ""3.29.0""}] ``` https://sqlite-utils.datasette.io/en/latest/cli.html#running-queries-and-returning-json `sqlite-utils memory ""select sqlite_version()""` is a little bit more intuitive than that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command", https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861987651,https://api.github.com/repos/simonw/sqlite-utils/issues/272,861987651,MDEyOklzc3VlQ29tbWVudDg2MTk4NzY1MQ==,9599,simonw,2021-06-16T02:27:20Z,2021-06-16T02:27:20Z,OWNER,Solution: `sqlite-utils memory -` attempts to detect the input based on if it starts with a `{` or `[` (likely JSON) or if it doesn't use the `csv.Sniffer()` mechanism. Or you can use `sqlite-utils memory -:csv` to specifically indicate the type of input.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command", https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861985944,https://api.github.com/repos/simonw/sqlite-utils/issues/272,861985944,MDEyOklzc3VlQ29tbWVudDg2MTk4NTk0NA==,9599,simonw,2021-06-16T02:22:52Z,2021-06-16T02:22:52Z,OWNER,"Another option: allow an optional `:suffix` specifying the type of the file. If this is missing we detect based on the filename. sqlite-utils memory somefile:csv ""select * from somefile"" One catch: how to treat `-` for standard input? cat blah.csv | sqlite-utils memory - ""select * from stdin"" That's fine for CSV, but what about TSV or JSON or nl-JSON? Maybe this: cat blah.csv | sqlite-utils memory -:json ""select * from stdin"" Bit weird though. The alternative would be to support this: cat blah.csv | sqlite-utils memory --load-csv - But that's verbose compared to the version without the long `--load-x` option.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",