html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app
https://github.com/simonw/sqlite-utils/issues/381#issuecomment-1010462035,https://api.github.com/repos/simonw/sqlite-utils/issues/381,1010462035,IC_kwDOCGYnMM48Om1T,9599,simonw,2022-01-11T23:33:37Z,2022-01-11T23:33:37Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#returning-all-rows-in-a-table,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099584685,`sqlite-utils rows` options `--limit` and `--offset`,
https://github.com/simonw/sqlite-utils/issues/382#issuecomment-1010461844,https://api.github.com/repos/simonw/sqlite-utils/issues/382,1010461844,IC_kwDOCGYnMM48OmyU,9599,simonw,2022-01-11T23:33:14Z,2022-01-11T23:33:14Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#returning-all-rows-in-a-table,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099585611,`--where` option for `sqlite-rows`,
https://github.com/simonw/sqlite-utils/issues/381#issuecomment-1010441118,https://api.github.com/repos/simonw/sqlite-utils/issues/381,1010441118,IC_kwDOCGYnMM48Ohue,9599,simonw,2022-01-11T22:56:53Z,2022-01-11T22:57:09Z,OWNER,"`sqlite-utils search` has `--limit` already: https://sqlite-utils.datasette.io/en/latest/cli-reference.html#search
```
--limit INTEGER Number of rows to return - defaults to everything
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099584685,`sqlite-utils rows` options `--limit` and `--offset`,
https://github.com/simonw/sqlite-utils/issues/383#issuecomment-1010440166,https://api.github.com/repos/simonw/sqlite-utils/issues/383,1010440166,IC_kwDOCGYnMM48Ohfm,9599,simonw,2022-01-11T22:55:05Z,2022-01-11T22:55:05Z,OWNER,Twitter thread about this: https://twitter.com/simonw/status/1481020195074293761,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099586786,Add documentation page with the output of `--help`,
https://github.com/simonw/sqlite-utils/issues/383#issuecomment-1010387223,https://api.github.com/repos/simonw/sqlite-utils/issues/383,1010387223,IC_kwDOCGYnMM48OUkX,9599,simonw,2022-01-11T21:45:32Z,2022-01-11T21:45:32Z,OWNER,The new page of documentation: https://sqlite-utils.datasette.io/en/latest/cli-reference.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099586786,Add documentation page with the output of `--help`,
https://github.com/simonw/sqlite-utils/issues/383#issuecomment-1010386802,https://api.github.com/repos/simonw/sqlite-utils/issues/383,1010386802,IC_kwDOCGYnMM48OUdy,9599,simonw,2022-01-11T21:44:53Z,2022-01-11T21:44:53Z,OWNER,Here's the `cog` code I used: https://github.com/simonw/sqlite-utils/blob/1d44b0cc2784c94aed1bcf350225cd86ee1aa7e5/docs/cli-reference.rst#L11-L76,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099586786,Add documentation page with the output of `--help`,
https://github.com/simonw/sqlite-utils/issues/383#issuecomment-1010333511,https://api.github.com/repos/simonw/sqlite-utils/issues/383,1010333511,IC_kwDOCGYnMM48OHdH,9599,simonw,2022-01-11T20:27:08Z,2022-01-11T20:27:08Z,OWNER,"I'll call the new page ""CLI reference"", for consistency with the API reference page here: https://sqlite-utils.datasette.io/en/stable/reference.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099586786,Add documentation page with the output of `--help`,
https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1009548580,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1009548580,IC_kwDOCGYnMM48LH0k,536941,fgregg,2022-01-11T02:43:34Z,2022-01-11T02:43:34Z,CONTRIBUTOR,thanks so much! always a pleasure to see how you work through these things,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index,
https://github.com/simonw/sqlite-utils/issues/380#issuecomment-1009544785,https://api.github.com/repos/simonw/sqlite-utils/issues/380,1009544785,IC_kwDOCGYnMM48LG5R,9599,simonw,2022-01-11T02:32:56Z,2022-01-11T02:32:56Z,OWNER,"CLI and Python library improvements to help run [ANALYZE](https://www.sqlite.org/lang_analyze.html) after creating indexes or inserting rows, to gain better performance from the SQLite query planner when it runs against indexes.
Three new CLI commands: `create-database`, `analyze` and `bulk`.
- New `sqlite-utils create-database` command for creating new empty database files. ([#348](https://github.com/simonw/sqlite-utils/issues/348))
- New Python methods for running `ANALYZE` against a database, table or index: `db.analyze()` and `table.analyze()`, see [Optimizing index usage with ANALYZE](https://sqlite-utils.datasette.io/en/stable/python-api.html#python-api-analyze). ([#366](https://github.com/simonw/sqlite-utils/issues/366))
- New [sqlite-utils analyze command](https://sqlite-utils.datasette.io/en/stable/cli.html#cli-analyze) for running `ANALYZE` using the CLI. ([#379](https://github.com/simonw/sqlite-utils/issues/379))
- The `create-index`, `insert` and `update` commands now have a new `--analyze` option for running `ANALYZE` after the command has completed. ([#379](https://github.com/simonw/sqlite-utils/issues/379))
- New [sqlite-utils bulk command](https://sqlite-utils.datasette.io/en/stable/cli.html#cli-bulk) which can import records in the same way as `sqlite-utils insert` (from JSON, CSV or TSV) and use them to bulk execute a parametrized SQL query. ([#375](https://github.com/simonw/sqlite-utils/issues/375))
- The CLI tool can now also be run using `python -m sqlite_utils`. ([#368](https://github.com/simonw/sqlite-utils/issues/368))
- Using `--fmt` now implies `--table`, so you don't need to pass both options. ([#374](https://github.com/simonw/sqlite-utils/issues/374))
- The `--convert` function applied to rows can now modify the row in place. ([#371](https://github.com/simonw/sqlite-utils/issues/371))
- The [insert-files command](https://sqlite-utils.datasette.io/en/stable/cli.html#cli-insert-files) supports two new columns: `stem` and `suffix`. ([#372](https://github.com/simonw/sqlite-utils/issues/372))
- The `--nl` import option now ignores blank lines in the input. ([#376](https://github.com/simonw/sqlite-utils/issues/376))
- Fixed bug where streaming input to the `insert` command with `--batch-size 1` would appear to only commit after several rows had been ingested, due to unnecessary input buffering. ([#364](https://github.com/simonw/sqlite-utils/issues/364))","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1098574572,Release notes for 3.21,
https://github.com/simonw/sqlite-utils/issues/375#issuecomment-1009536276,https://api.github.com/repos/simonw/sqlite-utils/issues/375,1009536276,IC_kwDOCGYnMM48LE0U,9599,simonw,2022-01-11T02:12:58Z,2022-01-11T02:12:58Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#executing-sql-in-bulk,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097251014,`sqlite-utils bulk` command,
https://github.com/simonw/sqlite-utils/pull/377#issuecomment-1009534817,https://api.github.com/repos/simonw/sqlite-utils/issues/377,1009534817,IC_kwDOCGYnMM48LEdh,9599,simonw,2022-01-11T02:09:38Z,2022-01-11T02:09:38Z,OWNER,"I tested this like so:
```
% wget 'https://raw.githubusercontent.com/wri/global-power-plant-database/master/output_database/global_power_plant_database.csv'
% sqlite-utils create-database test.db
% sqlite-utils create-table test.db power_plants url text owner text
% sqlite-utils schema test.db
CREATE TABLE [power_plants] (
[url] TEXT,
[owner] TEXT
);
% sqlite-utils bulk test.db 'insert into power_plants (url, owner) values (:url, :owner)' global_power_plant_database.csv --csv
[------------------------------------] 0%
[###################################-] 99%
% sqlite-utils tables --counts test.db -t
table count
------------ -------
power_plants 33643
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097477582,`sqlite-utils bulk` command,
https://github.com/simonw/sqlite-utils/pull/377#issuecomment-1009532125,https://api.github.com/repos/simonw/sqlite-utils/issues/377,1009532125,IC_kwDOCGYnMM48LDzd,9599,simonw,2022-01-11T02:03:35Z,2022-01-11T02:03:35Z,OWNER,Documentation: https://github.com/simonw/sqlite-utils/blob/f4ea0d32c0543373eefaa9b9f3911eb07549eecb/docs/cli.rst#executing-sql-in-bulk,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097477582,`sqlite-utils bulk` command,
https://github.com/simonw/sqlite-utils/pull/377#issuecomment-1009531863,https://api.github.com/repos/simonw/sqlite-utils/issues/377,1009531863,IC_kwDOCGYnMM48LDvX,22429695,codecov[bot],2022-01-11T02:03:00Z,2022-01-11T02:03:00Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/377?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#377](https://codecov.io/gh/simonw/sqlite-utils/pull/377?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (f4ea0d3) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/b6dad08a8389736b7e960cfe9bc719cfc21a98f5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (b6dad08) will **decrease** coverage by `0.01%`.
> The diff coverage is `96.00%`.
[![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/377/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/377?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
```diff
@@ Coverage Diff @@
## main #377 +/- ##
==========================================
- Coverage 96.52% 96.50% -0.02%
==========================================
Files 6 6
Lines 2330 2378 +48
==========================================
+ Hits 2249 2295 +46
- Misses 81 83 +2
```
| [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/377?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [sqlite\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/377/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.49% <94.28%> (-0.06%)` | :arrow_down: |
| [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/377/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.68% <100.00%> (+0.03%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/377?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/377?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [b6dad08...f4ea0d3](https://codecov.io/gh/simonw/sqlite-utils/pull/377?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097477582,`sqlite-utils bulk` command,
https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1009521921,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1009521921,IC_kwDOCGYnMM48LBUB,9599,simonw,2022-01-11T01:37:53Z,2022-01-11T01:37:53Z,OWNER,"I decided to go with making this opt-in, mainly for consistency with the other places where I added this feature - see:
- #379
- #366
You can now run the following:
sqlite-utils create-index mydb.db mytable mycolumn --analyze
And ``ANALYZE`` will be run on the index once it has been created.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index,
https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009508865,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1009508865,IC_kwDOCGYnMM48K-IB,9599,simonw,2022-01-11T01:08:51Z,2022-01-11T01:08:51Z,OWNER,"The Python methods are all done now, next step is the CLI options. I'll do those in a separate issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE,
https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009288898,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1009288898,IC_kwDOCGYnMM48KIbC,9599,simonw,2022-01-10T19:54:04Z,2022-01-10T19:54:04Z,OWNER,"Having browsed the API reference I think the methods that would benefit from an `analyze=True` parameter are:
- `db.create_index`
- `table.insert_all`
- `table.upsert_all`
- `table.delete_where`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE,
https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009285627,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1009285627,IC_kwDOCGYnMM48KHn7,9599,simonw,2022-01-10T19:49:19Z,2022-01-10T19:51:25Z,OWNER,Documentation for those two new methods: https://sqlite-utils.datasette.io/en/latest/python-api.html#optimizing-index-usage-with-analyze,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE,
https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009286373,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1009286373,IC_kwDOCGYnMM48KHzl,9599,simonw,2022-01-10T19:50:22Z,2022-01-10T19:50:22Z,OWNER,"With respect to #365, I'm now thinking that having the ability to say ""... and then run ANALYZE"" could be useful for a bunch of Python methods. For example:
```python
db[""dogs""].insert_all(list_of_dogs, analyze=True)
db[""dogs""].create_index([""name""], analyze=True)
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE,
https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009273525,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1009273525,IC_kwDOCGYnMM48KEq1,9599,simonw,2022-01-10T19:32:39Z,2022-01-10T19:32:39Z,OWNER,"I'm going to implement the Python library methods based on the prototype:
```diff
commit 650f97a08f29a688c530e5f6c9eedc9269ed7bdc
Author: Simon Willison
Date: Sat Jan 8 13:34:01 2022 -0800
Initial prototype of .analyze(), refs #366
diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py
index dfc4723..1348b4a 100644
--- a/sqlite_utils/db.py
+++ b/sqlite_utils/db.py
@@ -923,6 +923,13 @@ class Database:
""Run a SQLite ``VACUUM`` against the database.""
self.execute(""VACUUM;"")
+ def analyze(self, name=None):
+ ""Run ``ANALYZE`` against the entire database or a named table or index.""
+ sql = ""ANALYZE""
+ if name is not None:
+ sql += "" [{}]"".format(name)
+ self.execute(sql)
+
class Queryable:
def exists(self) -> bool:
@@ -2902,6 +2909,10 @@ class Table(Queryable):
)
return self
+ def analyze(self):
+ ""Run ANALYZE against this table""
+ self.db.analyze(self.name)
+
def analyze_column(
self, column: str, common_limit: int = 10, value_truncate=None, total_rows=None
) -> ""ColumnDetails"":
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE,
https://github.com/simonw/sqlite-utils/pull/367#issuecomment-1009272446,https://api.github.com/repos/simonw/sqlite-utils/issues/367,1009272446,IC_kwDOCGYnMM48KEZ-,9599,simonw,2022-01-10T19:31:08Z,2022-01-10T19:31:08Z,OWNER,I'm going to implement this in a separate commit from this PR.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097041471,Initial prototype of .analyze() methods,
https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008557414,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008557414,IC_kwDOCGYnMM48HV1m,9599,simonw,2022-01-10T05:36:19Z,2022-01-10T05:36:19Z,OWNER,That did the trick.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item,
https://github.com/simonw/sqlite-utils/issues/375#issuecomment-1008556706,https://api.github.com/repos/simonw/sqlite-utils/issues/375,1008556706,IC_kwDOCGYnMM48HVqi,9599,simonw,2022-01-10T05:33:41Z,2022-01-10T05:33:41Z,OWNER,"I tested the prototype like this:
sqlite-utils blah.db 'create table blah (id integer primary key, name text)'
echo 'id,name
1,Cleo
2,Chicken' > blah.csv
sqlite-utils bulk blah.db 'insert into blah (id, name) values (:id, :name)' blah.csv --csv
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097251014,`sqlite-utils bulk` command,
https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008546573,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008546573,IC_kwDOCGYnMM48HTMN,9599,simonw,2022-01-10T05:05:15Z,2022-01-10T05:05:15Z,OWNER,"Bit nasty but it might work:
```python
def try_until(expected):
tries = 0
while True:
rows = list(Database(db_path)[""rows""].rows)
if rows == expected:
return
tries += 1
if tries > 10:
assert False, ""Expected {}, got {}"".format(expected, rows)
time.sleep(tries * 0.1)
try_until([{""name"": ""Azi""}])
proc.stdin.write(b'{""name"": ""Suna""}\n')
proc.stdin.flush()
try_until([{""name"": ""Azi""}, {""name"": ""Suna""}])
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item,
https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008545140,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008545140,IC_kwDOCGYnMM48HS10,9599,simonw,2022-01-10T05:01:34Z,2022-01-10T05:01:34Z,OWNER,"Urgh, tests are still failing intermittently - for example:
```
time.sleep(0.4)
> assert list(Database(db_path)[""rows""].rows) == [{""name"": ""Azi""}]
E AssertionError: assert [] == [{'name': 'Azi'}]
E Right contains one more item: {'name': 'Azi'}
E Full diff:
E - [{'name': 'Azi'}]
E + []
```
I'm going to change this code to keep on trying up to 10 seconds - that should get the tests to pass faster on most machines.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item,
https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008537194,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008537194,IC_kwDOCGYnMM48HQ5q,9599,simonw,2022-01-10T04:29:53Z,2022-01-10T04:31:29Z,OWNER,"After a bunch of debugging with `print()` statements it's clear that the problem isn't with when things are committed or the size of the batches - it's that the data sent to standard input is all being processed in one go, not a line at a time.
I think that's because it is being buffered by this: https://github.com/simonw/sqlite-utils/blob/d2a79d200f9071a86027365fa2a576865b71064f/sqlite_utils/cli.py#L759-L770
The buffering is there so that we can sniff the first few bytes to detect if it's a CSV file - added in 99ff0a288c08ec2071139c6031eb880fa9c95310 for #230. So maybe for non-CSV inputs we should disable buffering?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item,
https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008526736,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008526736,IC_kwDOCGYnMM48HOWQ,9599,simonw,2022-01-10T04:07:29Z,2022-01-10T04:07:29Z,OWNER,"I think this test is right:
```python
def test_insert_streaming_batch_size_1(db_path):
# https://github.com/simonw/sqlite-utils/issues/364
# Streaming with --batch-size 1 should commit on each record
# Can't use CliRunner().invoke() here bacuse we need to
# run assertions in between writing to process stdin
proc = subprocess.Popen(
[
sys.executable,
""-m"",
""sqlite_utils"",
""insert"",
db_path,
""rows"",
""-"",
""--nl"",
""--batch-size"",
""1"",
],
stdin=subprocess.PIPE,
)
proc.stdin.write(b'{""name"": ""Azi""}')
proc.stdin.flush()
assert list(Database(db_path)[""rows""].rows) == [{""name"": ""Azi""}]
proc.stdin.write(b'{""name"": ""Suna""}')
proc.stdin.flush()
assert list(Database(db_path)[""rows""].rows) == [{""name"": ""Azi""}, {""name"": ""Suna""}]
proc.stdin.close()
proc.wait()
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item,
https://github.com/simonw/sqlite-utils/issues/348#issuecomment-1008383293,https://api.github.com/repos/simonw/sqlite-utils/issues/348,1008383293,IC_kwDOCGYnMM48GrU9,9599,simonw,2022-01-09T20:38:17Z,2022-01-09T20:38:17Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#creating-an-empty-database,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1067771698,Command for creating an empty database,
https://github.com/simonw/sqlite-utils/issues/348#issuecomment-1008367607,https://api.github.com/repos/simonw/sqlite-utils/issues/348,1008367607,IC_kwDOCGYnMM48Gnf3,9599,simonw,2022-01-09T20:22:43Z,2022-01-09T20:22:43Z,OWNER,I'm not going to implement `--page-size` unless someone specifically requests it - I don't like having features that I've never needed to use myself.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1067771698,Command for creating an empty database,
https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008364701,https://api.github.com/repos/simonw/sqlite-utils/issues/371,1008364701,IC_kwDOCGYnMM48Gmyd,9599,simonw,2022-01-09T20:04:35Z,2022-01-09T20:04:35Z,OWNER,"The previous code for highlighting errors in syntax (which was already a bit confused thanks to the added `return`, see https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991393684 - isn't compatible with this approach at all. I'm going to ditch it and just show a generic `Error: Could not compile code` message.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097128334,Support mutating row in `--convert` without returning it,
https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008354207,https://api.github.com/repos/simonw/sqlite-utils/issues/371,1008354207,IC_kwDOCGYnMM48GkOf,9599,simonw,2022-01-09T18:54:54Z,2022-01-09T18:54:54Z,OWNER,"This seems to work:
```python
def _compile_code(code, imports, variable=""value""):
locals = {}
globals = {""r"": recipes, ""recipes"": recipes}
# If user defined a convert() function, return that
try:
exec(code, globals, locals)
return locals[""convert""]
except (AttributeError, SyntaxError, NameError, KeyError, TypeError):
pass
# Try compiling their code as a function instead
body_variants = [code]
# If single line and no 'return', try adding the return
if ""\n"" not in code and not code.strip().startswith(""return ""):
body_variants.insert(0, ""return {}"".format(code))
for variant in body_variants:
new_code = [""def fn({}):"".format(variable)]
for line in variant.split(""\n""):
new_code.append("" {}"".format(line))
try:
code_o = compile(""\n"".join(new_code), """", ""exec"")
break
except SyntaxError:
# Try another variant, e.g. for 'return row[""column""] = 1'
continue
for import_ in imports:
globals[import_.split(""."")[0]] = __import__(import_)
exec(code_o, globals, locals)
return locals[""fn""]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097128334,Support mutating row in `--convert` without returning it,
https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008348032,https://api.github.com/repos/simonw/sqlite-utils/issues/371,1008348032,IC_kwDOCGYnMM48GiuA,9599,simonw,2022-01-09T18:14:02Z,2022-01-09T18:14:02Z,OWNER,Here's the code in question: https://github.com/simonw/sqlite-utils/blob/b8c134059e89f0fa040b84fb7d0bda25b9a52759/sqlite_utils/utils.py#L288-L299,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097128334,Support mutating row in `--convert` without returning it,
https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008347768,https://api.github.com/repos/simonw/sqlite-utils/issues/371,1008347768,IC_kwDOCGYnMM48Gip4,9599,simonw,2022-01-09T18:12:30Z,2022-01-09T18:12:30Z,OWNER,"Tried this test:
```python
result = CliRunner().invoke(
cli.cli,
[
""insert"",
db_path,
""rows"",
""-"",
""--convert"",
'row[""is_chicken""] = True',
],
input='{""name"": ""Azi""}',
)
```
And got this error:
> `E + where 1 = ', 2, 30, ' return row[""is_chicken""] = True\n'))>.exit_code`
The code snippet compilation isn't currently compatible with this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097128334,Support mutating row in `--convert` without returning it,
https://github.com/simonw/sqlite-utils/issues/374#issuecomment-1008346841,https://api.github.com/repos/simonw/sqlite-utils/issues/374,1008346841,IC_kwDOCGYnMM48GibZ,9599,simonw,2022-01-09T18:06:50Z,2022-01-09T18:06:50Z,OWNER,"In addition to a unit test I manually tested all of the above, e.g.
```
% sqlite-utils indexes global-power-plants.db sqlite_master --fmt rst
======= ============ ======= ===== ====== ====== ====== =====
table index_name seqno cid name desc coll key
======= ============ ======= ===== ====== ====== ====== =====
======= ============ ======= ===== ====== ====== ====== =====
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097135860,`--fmt` should imply `-t`,
https://github.com/simonw/sqlite-utils/issues/374#issuecomment-1008346338,https://api.github.com/repos/simonw/sqlite-utils/issues/374,1008346338,IC_kwDOCGYnMM48GiTi,9599,simonw,2022-01-09T18:03:22Z,2022-01-09T18:03:22Z,OWNER,"Commands that support `--fmt` (via the `@output_options` decorator) are:
- `tables`
- `views`
- `query`
- `memory`
- `search`
- `rows`
- `triggers`
- `indexes`
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097135860,`--fmt` should imply `-t`,
https://github.com/simonw/sqlite-utils/issues/374#issuecomment-1008345267,https://api.github.com/repos/simonw/sqlite-utils/issues/374,1008345267,IC_kwDOCGYnMM48GiCz,9599,simonw,2022-01-09T17:56:37Z,2022-01-09T17:56:37Z,OWNER,"Better:
```python
if fmt:
table = True
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097135860,`--fmt` should imply `-t`,
https://github.com/simonw/sqlite-utils/issues/373#issuecomment-1008344980,https://api.github.com/repos/simonw/sqlite-utils/issues/373,1008344980,IC_kwDOCGYnMM48Gh-U,9599,simonw,2022-01-09T17:54:53Z,2022-01-09T17:54:53Z,OWNER,Updated TIL: https://til.simonwillison.net/python/cog-to-update-help-in-readme#user-content-cog-for-restructuredtext,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097135732,List `--fmt` options in the docs ,
https://github.com/simonw/sqlite-utils/issues/373#issuecomment-1008344525,https://api.github.com/repos/simonw/sqlite-utils/issues/373,1008344525,IC_kwDOCGYnMM48Gh3N,9599,simonw,2022-01-09T17:52:22Z,2022-01-09T17:52:22Z,OWNER,Updated docs: https://sqlite-utils.datasette.io/en/latest/cli.html#table-formatted-output,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097135732,List `--fmt` options in the docs ,
https://github.com/simonw/sqlite-utils/issues/373#issuecomment-1008341078,https://api.github.com/repos/simonw/sqlite-utils/issues/373,1008341078,IC_kwDOCGYnMM48GhBW,9599,simonw,2022-01-09T17:31:12Z,2022-01-09T17:31:12Z,OWNER,"Found an example of using `cog` in a rST file here: https://github.com/nedbat/coveragepy/blob/f3238eea7e403d13a217b30579b1a1c2cbff62e3/doc/dbschema.rst#L21
```
.. [[[cog
from coverage.sqldata import SCHEMA_VERSION
print("".. code::"")
print()
print(f"" SCHEMA_VERSION = {SCHEMA_VERSION}"")
print()
.. ]]]
.. code::
SCHEMA_VERSION = 7
.. [[[end]]]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097135732,List `--fmt` options in the docs ,
https://github.com/simonw/sqlite-utils/issues/375#issuecomment-1008338186,https://api.github.com/repos/simonw/sqlite-utils/issues/375,1008338186,IC_kwDOCGYnMM48GgUK,9599,simonw,2022-01-09T17:13:33Z,2022-01-09T17:13:54Z,OWNER," cat blah.csv | sqlite-utils bulk blah.db - \
""insert into blah (:foo, :bar)"" --csv
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097251014,`sqlite-utils bulk` command,
https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008275546,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1008275546,IC_kwDOCGYnMM48GRBa,536941,fgregg,2022-01-09T11:01:15Z,2022-01-09T13:37:51Z,CONTRIBUTOR,"i don’t want to be such a partisan for analyze, but the query planner deciding *not* to use an index based on information collected by analyze is not necessarily a bug, but could be the correct choice.
the original poster in that stack overflow doesn’t say there’s a performance regression ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index,
https://github.com/simonw/datasette/pull/1574#issuecomment-1008279307,https://api.github.com/repos/simonw/datasette/issues/1574,1008279307,IC_kwDOBm6k_c48GR8L,33631,fs111,2022-01-09T11:26:06Z,2022-01-09T11:26:06Z,NONE,"@fgregg my thinking was backwards compatibility. I don't know what people do to their builds, I just wanted a smaller image for my use case.
@simonw any chance to take a look at this? If there is no interest, feel free to close the PR","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1084193403,introduce new option for datasette package to use a slim base image,
https://github.com/simonw/sqlite-utils/issues/374#issuecomment-1008252732,https://api.github.com/repos/simonw/sqlite-utils/issues/374,1008252732,IC_kwDOCGYnMM48GLc8,9599,simonw,2022-01-09T08:25:30Z,2022-01-09T08:25:30Z,OWNER,Need to change `if table:` to `if table or fmt:` in a few places.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097135860,`--fmt` should imply `-t`,
https://github.com/simonw/sqlite-utils/issues/372#issuecomment-1008247370,https://api.github.com/repos/simonw/sqlite-utils/issues/372,1008247370,IC_kwDOCGYnMM48GKJK,9599,simonw,2022-01-09T07:51:18Z,2022-01-09T07:51:18Z,OWNER,"Pathlib says the stem of that would be `dogs.and.cats.jpg` - best stick with that for consistency. https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.suffix
It calls the last bit `suffix` - maybe I should use that instead of `ext`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097129710,Idea: `suffix` and `stem` file columns,
https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008246366,https://api.github.com/repos/simonw/sqlite-utils/issues/371,1008246366,IC_kwDOCGYnMM48GJ5e,9599,simonw,2022-01-09T07:42:14Z,2022-01-09T07:42:14Z,OWNER,Also need to update relevant docs for that example.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097128334,Support mutating row in `--convert` without returning it,
https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008246239,https://api.github.com/repos/simonw/sqlite-utils/issues/371,1008246239,IC_kwDOCGYnMM48GJ3f,9599,simonw,2022-01-09T07:41:24Z,2022-01-09T07:41:24Z,OWNER,"Might be a case of modifying this line: https://github.com/simonw/sqlite-utils/blob/e0c476bc380744680c8b7675c24fb0e9f5ec6dcd/sqlite_utils/cli.py#L828
To:
```python
docs = (fn(doc) or doc for doc in docs)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097128334,Support mutating row in `--convert` without returning it,
https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008234293,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008234293,IC_kwDOCGYnMM48GG81,9599,simonw,2022-01-09T05:37:02Z,2022-01-09T05:37:02Z,OWNER,Calling `p.stdin.close()` and then `p.wait()` terminates the subprocess.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item,
https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008233910,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008233910,IC_kwDOCGYnMM48GG22,9599,simonw,2022-01-09T05:32:53Z,2022-01-09T05:35:45Z,OWNER,"This is strange. The following:
```pycon
>>> import subprocess
>>> p = subprocess.Popen([""sqlite-utils"", ""insert"", ""/tmp/stream.db"", ""stream"", ""-"", ""--nl""], stdin=subprocess.PIPE)
>>> p.stdin.write(b'\n'.join(b'{""id"": %s}' % str(i).encode(""utf-8"") for i in range(1000)))
11889
>>> # At this point /tmp/stream.db is still 0 bytes - but if I then run this:
>>> p.stdin.close()
>>> # /tmp/stream.db is now 20K and contains the written data
```
No wait, mystery solved - I can add `p.stdin.flush()` instead of `p.stdin.close()` and the file suddenly jumps up in size.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item,
https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008232075,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008232075,IC_kwDOCGYnMM48GGaL,9599,simonw,2022-01-09T05:13:15Z,2022-01-09T05:13:56Z,OWNER,"I think the query that will help solve this is:
`explain query plan select * from ny_times_us_counties where state = 1 and county = 2`
In this case, the query planner needs to decide if it should use the index for the `state` column or the index for the `county` column. That's where the statistics come into play. In particular:
| tbl | idx | stat |
|----------------------|---------------------------------|---------------|
| ny_times_us_counties | idx_ny_times_us_counties_date | 2092871 2915 |
| ny_times_us_counties | idx_ny_times_us_counties_fips | 2092871 651 |
| ny_times_us_counties | idx_ny_times_us_counties_county | 2092871 1085 |
| ny_times_us_counties | idx_ny_times_us_counties_state | 2092871 37373 |
Those numbers are explained by this comment in the SQLite C code: https://github.com/sqlite/sqlite/blob/5622c7f97106314719740098cf0854e7eaa81802/src/analyze.c#L41-L55
```
** There is normally one row per index, with the index identified by the
** name in the idx column. The tbl column is the name of the table to
** which the index belongs. In each such row, the stat column will be
** a string consisting of a list of integers. The first integer in this
** list is the number of rows in the index. (This is the same as the
** number of rows in the table, except for partial indices.) The second
** integer is the average number of rows in the index that have the same
** value in the first column of the index.
```
So that table is telling us that using a value in the `county` column will filter down to an average of 1,085 rows, whereas filtering on the `state` column will filter down to an average of 37,373 - so clearly the `county` index is the better index to use here!
Just one catch: against both my` covid.db` and my `covid-analyzed.db` databases the `county` index is picked for both of them - so SQLite is somehow guessing that `county` is a better index even though it doesn't have statistics for that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes,
https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008229839,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1008229839,IC_kwDOCGYnMM48GF3P,9599,simonw,2022-01-09T04:51:44Z,2022-01-09T04:51:44Z,OWNER,"Found one report on Stack Overflow from 9 years ago of someone seeing broken performance after running `ANALYZE`, hard to say that's a trend and not a single weird edge-case though! https://stackoverflow.com/q/12947214/6083","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index,
https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008229341,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008229341,IC_kwDOCGYnMM48GFvd,9599,simonw,2022-01-09T04:45:38Z,2022-01-09T04:47:11Z,OWNER,"This is probably too fancy. I think maybe the way to do this is with `select * from [global-power-plants] where ""country_long"" = 'United Kingdom'` - then mess around with stats to see if I can get it to use the index or not based on them.
Here's the explain for that: https://global-power-plants.datasettes.com/global-power-plants?sql=EXPLAIN+QUERY+PLAN+select+*+from+[global-power-plants]+where+%22country_long%22+%3D+%27United+Kingdom%27","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes,
https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008227625,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008227625,IC_kwDOCGYnMM48GFUp,9599,simonw,2022-01-09T04:25:38Z,2022-01-09T04:25:38Z,OWNER,"```sql
EXPLAIN QUERY PLAN select country_long, count(*) from [global-power-plants] group by country_long
```
https://global-power-plants.datasettes.com/global-power-plants?sql=EXPLAIN+QUERY+PLAN+select+country_long%2C+count%28*%29+from+%5Bglobal-power-plants%5D+group+by+country_long
> SCAN TABLE global-power-plants USING COVERING INDEX ""global-power-plants_country_long""","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes,
https://github.com/simonw/datasette/issues/1588#issuecomment-1008227436,https://api.github.com/repos/simonw/datasette/issues/1588,1008227436,IC_kwDOBm6k_c48GFRs,9599,simonw,2022-01-09T04:23:37Z,2022-01-09T04:25:04Z,OWNER,"Relevant code: https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/utils/__init__.py#L163-L170
https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/utils/__init__.py#L195-L204","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097101917,`explain query plan select` is too strict about whitespace,
https://github.com/simonw/datasette/issues/1588#issuecomment-1008227491,https://api.github.com/repos/simonw/datasette/issues/1588,1008227491,IC_kwDOBm6k_c48GFSj,9599,simonw,2022-01-09T04:24:09Z,2022-01-09T04:24:09Z,OWNER,"I think this is the fix:
```python
re.compile(r""^explain\s+query\s+plan\s+select\b""),
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097101917,`explain query plan select` is too strict about whitespace,
https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008226862,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008226862,IC_kwDOCGYnMM48GFIu,9599,simonw,2022-01-09T04:17:55Z,2022-01-09T04:17:55Z,OWNER,"There are some clues as to what effect ANALYZE has in https://www.sqlite.org/optoverview.html
Some quotes:
> SQLite might use a skip-scan on an index if it knows that the first one or more columns contain many duplication values. If there are too few duplicates in the left-most columns of the index, then it would be faster to simply step ahead to the next value, and thus do a full table scan, than to do a binary search on an index to locate the next left-column value.
>
> The only way that SQLite can know that there are many duplicates in the left-most columns of an index is if the ANALYZE command has been run on the database. Without the results of ANALYZE, SQLite has to guess at the ""shape"" of the data in the table, and the default guess is that there are an average of 10 duplicates for every value in the left-most column of the index. Skip-scan only becomes profitable (it only gets to be faster than a full table scan) when the number of duplicates is about 18 or more. Hence, a skip-scan is never used on a database that has not been analyzed.
And
> Join reordering is automatic and usually works well enough that programmers do not have to think about it, especially if ANALYZE has been used to gather statistics about the available indexes, though occasionally some hints from the programmer are needed.
And
> The various sqlite_statN tables contain information on how selective the various indexes are. For example, the sqlite_stat1 table might indicate that an equality constraint on column x reduces the search space to 10 rows on average, whereas an equality constraint on column y reduces the search space to 3 rows on average. In that case, SQLite would prefer to use index ex2i2 since that index is more selective. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes,
https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008226487,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008226487,IC_kwDOCGYnMM48GFC3,9599,simonw,2022-01-09T04:14:05Z,2022-01-09T04:14:05Z,OWNER,"Didn't manage to spot a meaningful difference with that database either:
```
analyze % python3 -m timeit '__import__(""sqlite3"").connect(""covid.db"").execute(""select fips, count(*) from [ny_times_us_counties] group by fips"").fetchall()'
2 loops, best of 5: 101 msec per loop
analyze % python3 -m timeit '__import__(""sqlite3"").connect(""covid-analyzed.db"").execute(""select fips, count(*) from [ny_times_us_counties] group by fips"").fetchall()'
2 loops, best of 5: 103 msec per loop
```
Maybe `select fips, count(*) from [ny_times_us_counties] group by fips` isn't a good query for testing this?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes,
https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008220270,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008220270,IC_kwDOCGYnMM48GDhu,9599,simonw,2022-01-09T03:12:38Z,2022-01-09T03:13:15Z,OWNER,"Basically no difference using this very basic benchmark:
```
analyze % python3 -m timeit '__import__(""sqlite3"").connect(""global-power-plants.db"").execute(""select country_long, count(*) from [global-power-plants] group by country_long"").fetchall()'
100 loops, best of 5: 2.39 msec per loop
analyze % python3 -m timeit '__import__(""sqlite3"").connect(""global-power-plants-analyzed.db"").execute(""select country_long, count(*) from [global-power-plants] group by country_long"").fetchall()'
100 loops, best of 5: 2.38 msec per loop
```
I should try this against a much larger database.
https://covid-19.datasettes.com/covid.db is 879MB.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes,
https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008219844,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008219844,IC_kwDOCGYnMM48GDbE,9599,simonw,2022-01-09T03:08:09Z,2022-01-09T03:08:09Z,OWNER,"```
analyze % sqlite-utils global-power-plants-analyzed.db 'analyze'
[{""rows_affected"": -1}]
analyze % sqlite-utils tables global-power-plants-analyzed.db
[{""table"": ""global-power-plants""},
{""table"": ""global-power-plants_fts""},
{""table"": ""global-power-plants_fts_data""},
{""table"": ""global-power-plants_fts_idx""},
{""table"": ""global-power-plants_fts_docsize""},
{""table"": ""global-power-plants_fts_config""},
{""table"": ""sqlite_stat1""}]
analyze % sqlite-utils rows global-power-plants-analyzed.db sqlite_stat1 -t
tbl idx stat
------------------------------- ---------------------------------- ---------
global-power-plants_fts_config global-power-plants_fts_config 1 1
global-power-plants_fts_docsize 33643
global-power-plants_fts_idx global-power-plants_fts_idx 199 40 1
global-power-plants_fts_data 136
global-power-plants ""global-power-plants_owner"" 33643 4
global-power-plants ""global-power-plants_country_long"" 33643 202
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes,
https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008219588,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008219588,IC_kwDOCGYnMM48GDXE,9599,simonw,2022-01-09T03:06:42Z,2022-01-09T03:06:42Z,OWNER,"```
analyze % sqlite-utils indexes global-power-plants.db -t
table index_name seqno cid name desc coll key
------------------------------ ------------------------------------------------- ------- ----- ------------ ------ ------ -----
global-power-plants ""global-power-plants_owner"" 0 12 owner 0 BINARY 1
global-power-plants ""global-power-plants_country_long"" 0 1 country_long 0 BINARY 1
global-power-plants_fts_idx sqlite_autoindex_global-power-plants_fts_idx_1 0 0 segid 0 BINARY 1
global-power-plants_fts_idx sqlite_autoindex_global-power-plants_fts_idx_1 1 1 term 0 BINARY 1
global-power-plants_fts_config sqlite_autoindex_global-power-plants_fts_config_1 0 0 k 0 BINARY 1
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes,
https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008219484,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008219484,IC_kwDOCGYnMM48GDVc,9599,simonw,2022-01-09T03:05:44Z,2022-01-09T03:05:44Z,OWNER,I'll start by running some experiments against the 11MB database file from https://global-power-plants.datasettes.com/global-power-plants.db,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes,
https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008219191,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008219191,IC_kwDOCGYnMM48GDQ3,9599,simonw,2022-01-09T03:03:53Z,2022-01-09T03:03:53Z,OWNER,"Refs:
- #366
- #365","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes,
https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008163585,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1008163585,IC_kwDOCGYnMM48F1sB,9599,simonw,2022-01-08T22:14:39Z,2022-01-09T03:03:07Z,OWNER,"The reason I'm hesitating on this is that I've not actually used ANALYZE at all in nearly five years of messing around with SQLite! So I'm nervous that there are surprise downsides I haven't thought of.
My hunch is that ANALYZE is only worth worrying about on much larger databases, in which case I'm OK supporting it as a thoroughly documented power-user feature rather than a default.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index,
https://github.com/simonw/sqlite-utils/issues/368#issuecomment-1008216371,https://api.github.com/repos/simonw/sqlite-utils/issues/368,1008216371,IC_kwDOCGYnMM48GCkz,9599,simonw,2022-01-09T02:36:22Z,2022-01-09T02:36:22Z,OWNER,"In Python 3.6: https://docs.python.org/3.6/library/subprocess.html
> This does not capture stdout or stderr by default. To do so, pass [`PIPE`](https://docs.python.org/3.6/library/subprocess.html#subprocess.PIPE ""subprocess.PIPE"") for the *stdout* and/or *stderr* arguments.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097087280,Offer `python -m sqlite_utils` as an alternative to `sqlite-utils`,
https://github.com/simonw/sqlite-utils/issues/368#issuecomment-1008216271,https://api.github.com/repos/simonw/sqlite-utils/issues/368,1008216271,IC_kwDOCGYnMM48GCjP,9599,simonw,2022-01-09T02:35:09Z,2022-01-09T02:35:09Z,OWNER,"Test failure on Python 3.6:
> `E TypeError: __init__() got an unexpected keyword argument 'capture_output'`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097087280,Offer `python -m sqlite_utils` as an alternative to `sqlite-utils`,
https://github.com/simonw/sqlite-utils/pull/367#issuecomment-1008158799,https://api.github.com/repos/simonw/sqlite-utils/issues/367,1008158799,IC_kwDOCGYnMM48F0hP,22429695,codecov[bot],2022-01-08T21:36:55Z,2022-01-09T02:34:44Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/367?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#367](https://codecov.io/gh/simonw/sqlite-utils/pull/367?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (9848eaa) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/a8f9cc6f64f299830834428509940d448b82b4ed?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (a8f9cc6) will **decrease** coverage by `0.20%`.
> The diff coverage is `50.00%`.
[![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/367/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/367?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
```diff
@@ Coverage Diff @@
## main #367 +/- ##
==========================================
- Coverage 96.44% 96.24% -0.21%
==========================================
Files 5 6 +1
Lines 2307 2317 +10
==========================================
+ Hits 2225 2230 +5
- Misses 82 87 +5
```
| [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/367?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/367/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.15% <28.57%> (-0.42%)` | :arrow_down: |
| [sqlite\_utils/\_\_main\_\_.py](https://codecov.io/gh/simonw/sqlite-utils/pull/367/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL19fbWFpbl9fLnB5) | `100.00% <100.00%> (ø)` | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/367?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/367?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [a8f9cc6...9848eaa](https://codecov.io/gh/simonw/sqlite-utils/pull/367?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097041471,Initial prototype of .analyze() methods,
https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008216201,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008216201,IC_kwDOCGYnMM48GCiJ,9599,simonw,2022-01-09T02:34:12Z,2022-01-09T02:34:12Z,OWNER,"I can now write tests that look like this: https://github.com/simonw/sqlite-utils/blob/539f5ccd90371fa87f946018f8b77d55929e06db/tests/test_cli.py#L2024-L2030
Which means I can write a test that exercises this bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item,
https://github.com/simonw/sqlite-utils/issues/368#issuecomment-1008215912,https://api.github.com/repos/simonw/sqlite-utils/issues/368,1008215912,IC_kwDOCGYnMM48GCdo,9599,simonw,2022-01-09T02:30:59Z,2022-01-09T02:30:59Z,OWNER,"Even better, inspired by `rich`, support `python -m sqlite_utils`. https://github.com/Textualize/rich/blob/master/rich/__main__.py","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097087280,Offer `python -m sqlite_utils` as an alternative to `sqlite-utils`,
https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008214998,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008214998,IC_kwDOCGYnMM48GCPW,9599,simonw,2022-01-09T02:23:20Z,2022-01-09T02:23:20Z,OWNER,"Possible way of running the test: add this to `sqlite_utils/cli.py`:
```python
if __name__ == ""__main__"":
cli()
```
Now the tool can be run using `python -m sqlite_utils.cli --help`
Then in the test use `subprocess` to call `sys.executable` (the path to the current Python interpreter) and pass it `-m sqlite_utils.cli` to run the script!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item,
https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008214406,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008214406,IC_kwDOCGYnMM48GCGG,9599,simonw,2022-01-09T02:18:21Z,2022-01-09T02:18:21Z,OWNER,"I'm having trouble figuring out the best way to write a unit test for this. Filed a relevant feature request for Click here:
- https://github.com/pallets/click/issues/2171","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item,
https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008166084,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1008166084,IC_kwDOCGYnMM48F2TE,536941,fgregg,2022-01-08T22:32:47Z,2022-01-08T22:32:47Z,CONTRIBUTOR,or using “ pragma optimize”,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index,
https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008164786,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1008164786,IC_kwDOCGYnMM48F1-y,536941,fgregg,2022-01-08T22:24:19Z,2022-01-08T22:24:19Z,CONTRIBUTOR,the out-of-date scenario you describe could be addressed by automatically adding an analyze to the insert or convert commands if they implicate an index,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index,
https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008164116,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1008164116,IC_kwDOCGYnMM48F10U,536941,fgregg,2022-01-08T22:18:57Z,2022-01-08T22:18:57Z,CONTRIBUTOR,"the table with the query ran so bad was about 50k.
i think the scenario should not be worse than no stats.
i also did not know that sqlite was so different from postgres and needed an explicit analyze call.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index,
https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008163050,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1008163050,IC_kwDOCGYnMM48F1jq,9599,simonw,2022-01-08T22:10:51Z,2022-01-08T22:10:51Z,OWNER,"Is there a downside to having a `sqlite_stat1` table if it has wildly incorrect statistics in it?
Imagine the following sequence of events:
- User imports a few records, creating the table, using `sqlite-utils insert`
- User runs `sqlite-utils create-index ...` which also creates and populates the `sqlite_stat1` table
- User runs `insert` again to populate several million new records
The user now has a database file with several million records and a statistics table that is wildly out of date, having been populated when they only had a few.
Will this result in surprisingly bad query performance compared to it that statistics table did not exist at all?
If so, I lean much harder towards `ANALYZE` as a strictly opt-in optimization, maybe with the `--analyze` option added to `sqlite-utils insert` top to help users opt in to updating their statistics after running big inserts.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index,
https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008161965,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1008161965,IC_kwDOCGYnMM48F1St,536941,fgregg,2022-01-08T22:02:56Z,2022-01-08T22:02:56Z,CONTRIBUTOR,"for options 2 and 3, i would worry about discoverablity.
in other db’s it is not necessary to explicitly call analyze for most indices. ie for postgres
> The system regularly collects statistics on all of a table's columns. Newly-created non-expression indexes can immediately use these statistics to determine an index's usefulness.
i suppose i would propose raising a warning if the stats table is created that explains what is going on and informs users about a —no-analyze argument.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index,
https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1008158616,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1008158616,IC_kwDOCGYnMM48F0eY,9599,simonw,2022-01-08T21:35:32Z,2022-01-08T21:35:32Z,OWNER,"Built a prototype in a branch, see #367.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE,
https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008158357,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1008158357,IC_kwDOCGYnMM48F0aV,9599,simonw,2022-01-08T21:33:07Z,2022-01-08T21:33:07Z,OWNER,"The one thing that worries me a little bit about doing this by default is that it adds a surprising new table to the database - it may be confusing to users if they run `create-index` and their database suddenly has a new `sqlite_stat1` table, see https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1008157132
Options here are:
- Do it anyway. People can tolerate a surprise table appearing when they create an index.
- Only run `ANALYZE` if the user says `sqlite-utils create-index ... --analyze`
- Use the `--analyze` option, but also automatically run `ANALYZE` if they create an index and the database they are working with already has a `sqlite_stat1` table
I'm currently leading towards that third option - @fgregg any thoughts?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index,
https://github.com/simonw/datasette/issues/1587#issuecomment-1008157998,https://api.github.com/repos/simonw/datasette/issues/1587,1008157998,IC_kwDOBm6k_c48F0Uu,9599,simonw,2022-01-08T21:29:54Z,2022-01-08T21:29:54Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/00a2895cd2dc42c63846216b36b2dc9f41170129/datasette/database.py#L339-L354,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097040427,Add `sqlite_stat1`(-4) tables to hidden table list,
https://github.com/simonw/datasette/issues/1587#issuecomment-1008157908,https://api.github.com/repos/simonw/datasette/issues/1587,1008157908,IC_kwDOBm6k_c48F0TU,9599,simonw,2022-01-08T21:29:06Z,2022-01-08T21:29:06Z,OWNER,"Depending on the SQLite version (and compile options) that ran `ANALYZE` these can be called:
- `sqlite_stat1`
- `sqlite_stat2`
- `sqlite_stat3`
- `sqlite_stat4`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097040427,Add `sqlite_stat1`(-4) tables to hidden table list,
https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1008157132,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1008157132,IC_kwDOCGYnMM48F0HM,9599,simonw,2022-01-08T21:23:08Z,2022-01-08T21:25:05Z,OWNER,"Running `ANALYZE` creates a new visible table called `sqlite_stat1`: https://www.sqlite.org/fileformat.html#the_sqlite_stat1_table
This should be added to the default list of hidden tables in Datasette.
It looks something like this:
| tbl | idx | stat |
|---------------------------------|------------------------------------|-----------|
| _counts | sqlite_autoindex__counts_1 | 5 1 |
| global-power-plants_fts_config | global-power-plants_fts_config | 1 1 |
| global-power-plants_fts_docsize | | 33643 |
| global-power-plants_fts_idx | global-power-plants_fts_idx | 199 40 1 |
| global-power-plants_fts_data | | 136 |
| global-power-plants | ""global-power-plants_owner"" | 33643 4 |
| global-power-plants | ""global-power-plants_country_long"" | 33643 202 |
> In each such row, the sqlite_stat.stat column will be a string consisting of a list of integers followed by zero or more arguments. The first integer in this list is the approximate number of rows in the index. (The number of rows in the index is the same as the number of rows in the table, except for partial indexes.) The second integer is the approximate number of rows in the index that have the same value in the first column of the index. The third integer is the number number of rows in the index that have the same value for the first two columns. The N-th integer (for N>1) is the estimated average number of rows in the index which have the same value for the first N-1 columns. For a K-column index, there will be K+1 integers in the stat column. If the index is unique, then the last integer will be 1. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE,
https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008155916,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008155916,IC_kwDOCGYnMM48Fz0M,9599,simonw,2022-01-08T21:16:46Z,2022-01-08T21:16:46Z,OWNER,"No, `chunks()` seems to work OK in the test I just added.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item,
https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008154873,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008154873,IC_kwDOCGYnMM48Fzj5,9599,simonw,2022-01-08T21:11:55Z,2022-01-08T21:11:55Z,OWNER,"I'm suspicious that the `chunks()` utility function may not be working correctly:
```pycon
In [10]: [list(d) for d in list(chunks('abc', 5))]
Out[10]: [['a'], ['b'], ['c']]
In [11]: [list(d) for d in list(chunks('abcdefghi', 5))]
Out[11]: [['a'], ['b'], ['c'], ['d'], ['e'], ['f'], ['g'], ['h'], ['i']]
In [12]: [list(d) for d in list(chunks('abcdefghi', 3))]
Out[12]: [['a'], ['b'], ['c'], ['d'], ['e'], ['f'], ['g'], ['h'], ['i']]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item,
https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008153586,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008153586,IC_kwDOCGYnMM48FzPy,9599,simonw,2022-01-08T21:06:15Z,2022-01-08T21:06:15Z,OWNER,"I added a print statement after `for query, params in queries_and_params` and confirmed that something in the code is waiting until 16 records are available to be inserted and then executing the inserts, even with `--batch-size 1`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item,
https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008151884,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008151884,IC_kwDOCGYnMM48Fy1M,9599,simonw,2022-01-08T20:59:21Z,2022-01-08T20:59:21Z,OWNER,"(That Heroku example doesn't record the timestamp, which limits its usefulness)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item,
https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008143248,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008143248,IC_kwDOCGYnMM48FwuQ,9599,simonw,2022-01-08T20:34:12Z,2022-01-08T20:34:12Z,OWNER,Built that tool: https://github.com/simonw/stream-delay and https://pypi.org/project/stream-delay/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item,
https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008129841,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008129841,IC_kwDOCGYnMM48Ftcx,9599,simonw,2022-01-08T20:04:42Z,2022-01-08T20:04:42Z,OWNER,"It would be easier to test this if I had a utility for streaming out a file one line at a time.
A few recipes for this in https://superuser.com/questions/526242/cat-file-to-terminal-at-particular-speed-of-lines-per-second - I'm going to build a quick `stream-delay` tool though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item,
https://github.com/simonw/datasette/pull/1574#issuecomment-1007844190,https://api.github.com/repos/simonw/datasette/issues/1574,1007844190,IC_kwDOBm6k_c48Ente,536941,fgregg,2022-01-08T00:42:12Z,2022-01-08T00:42:12Z,CONTRIBUTOR,is there a reason to not always use the slim option?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1084193403,introduce new option for datasette package to use a slim base image,
https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1007643254,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1007643254,IC_kwDOCGYnMM48D2p2,9599,simonw,2022-01-07T18:37:56Z,2022-01-07T18:37:56Z,OWNER,Or I could leave off `--no-analyze` and tell people that if they want to add an index without running analyze they can execute the `CREATE INDEX` themselves.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index,
https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1007642831,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1007642831,IC_kwDOCGYnMM48D2jP,9599,simonw,2022-01-07T18:37:18Z,2022-01-07T18:37:18Z,OWNER,"After implementing #366 I can make it so `sqlite-utils create-index` automatically runs `db.analyze(index_name)` afterwards, maybe with a `--no-analyze` option in case anyone wants to opt out of that for specific performance reasons.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index,
https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1007641634,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1007641634,IC_kwDOCGYnMM48D2Qi,9599,simonw,2022-01-07T18:35:35Z,2022-01-07T18:35:35Z,OWNER,"Since the existing CLI feature is this:
$ sqlite-utils analyze-tables github.db tags
I can add `sqlite-utils analyze` to reflect the Python library method.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE,
https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1007639860,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1007639860,IC_kwDOCGYnMM48D100,9599,simonw,2022-01-07T18:32:59Z,2022-01-07T18:33:07Z,OWNER,"From the SQLite docs:
> If no arguments are given, all attached databases are analyzed. If a schema name is given as the argument, then all tables and indices in that one database are analyzed. If the argument is a table name, then only that table and the indices associated with that table are analyzed. If the argument is an index name, then only that one index is analyzed.
So I think this becomes two methods:
- `db.analyze()` calls analyze on the whole database
- `db.analyze(name_of_table_or_index)` for a specific named table or index
- `table.analyze()` is a shortcut for `db.analyze(table.name)`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE,
https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1007637963,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1007637963,IC_kwDOCGYnMM48D1XL,9599,simonw,2022-01-07T18:30:13Z,2022-01-07T18:30:13Z,OWNER,"Annoyingly I use the word ""analyze"" to mean something else in the CLI - for these features:
- #207
- #320
there's only one method with a similar name in the Python library though and that's this one:
https://github.com/simonw/sqlite-utils/blob/6e46b9913411682f3a3ec66f4d58886c1db8654b/sqlite_utils/db.py#L2904-L2906","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE,
https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1007636709,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1007636709,IC_kwDOCGYnMM48D1Dl,536941,fgregg,2022-01-07T18:28:33Z,2022-01-07T18:29:43Z,CONTRIBUTOR,"i added an index to one table with sqlite-utils, and then a query that used to take about 1 second started taking hundreds of seconds.
running analyze got me back to sub second speed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index,
https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1007634999,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1007634999,IC_kwDOCGYnMM48D0o3,9599,simonw,2022-01-07T18:26:22Z,2022-01-07T18:26:22Z,OWNER,"I've not used the `ANALYZE` feature in SQLite at all before. Should probably add Python library methods for it.
Annoyingly I use the word ""analyze"" to mean something else in the CLI - for these features:
- #207
- #320","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index,
https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1007633376,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1007633376,IC_kwDOCGYnMM48D0Pg,9599,simonw,2022-01-07T18:24:07Z,2022-01-07T18:24:07Z,OWNER,Relevant documentation: https://www.sqlite.org/lang_analyze.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index,
https://github.com/dogsheep/dogsheep-photos/pull/36#issuecomment-1006708046,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/36,1006708046,IC_kwDOD079W848ASVO,71983,scoates,2022-01-06T16:04:46Z,2022-01-06T16:04:46Z,NONE,"This one got me, today, too. 👍","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",988493790,Correct naming of tool in readme,
https://github.com/simonw/sqlite-utils/issues/363#issuecomment-1006344080,https://api.github.com/repos/simonw/sqlite-utils/issues/363,1006344080,IC_kwDOCGYnMM47-5eQ,9599,simonw,2022-01-06T07:32:05Z,2022-01-06T07:32:05Z,OWNER,As part of this work I should add test coverage of this error message too: https://github.com/simonw/sqlite-utils/blob/413f8ed754e38d7b190de888c85fe8438336cb11/sqlite_utils/cli.py#L826,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094981339,Better error message if `--convert` code fails to return a dict,
https://github.com/simonw/sqlite-utils/issues/363#issuecomment-1006343303,https://api.github.com/repos/simonw/sqlite-utils/issues/363,1006343303,IC_kwDOCGYnMM47-5SH,9599,simonw,2022-01-06T07:30:20Z,2022-01-06T07:30:20Z,OWNER,This check should run inside the `.insert_all()` method. It should raise a custom exception which the CLI code can then catch and turn into a click error.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094981339,Better error message if `--convert` code fails to return a dict,
https://github.com/simonw/sqlite-utils/issues/356#issuecomment-1006318443,https://api.github.com/repos/simonw/sqlite-utils/issues/356,1006318443,IC_kwDOCGYnMM47-zNr,9599,simonw,2022-01-06T06:30:13Z,2022-01-06T06:30:13Z,OWNER,"Documentation:
- https://sqlite-utils.datasette.io/en/latest/cli.html#inserting-unstructured-data-with-lines-and-text
- https://sqlite-utils.datasette.io/en/latest/cli.html#applying-conversions-while-inserting-data","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option,
https://github.com/simonw/sqlite-utils/issues/356#issuecomment-1006318007,https://api.github.com/repos/simonw/sqlite-utils/issues/356,1006318007,IC_kwDOCGYnMM47-zG3,9599,simonw,2022-01-06T06:28:53Z,2022-01-06T06:28:53Z,OWNER,Implemented in #361.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option,
https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006219956,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006219956,IC_kwDOCGYnMM47-bK0,22429695,codecov[bot],2022-01-06T01:51:54Z,2022-01-06T06:22:25Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/361?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#361](https://codecov.io/gh/simonw/sqlite-utils/pull/361?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (b7f0b88) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/f3fd8613113d21d44238a6ec54b375f5aa72c4e0?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (f3fd861) will **decrease** coverage by `0.05%`.
> The diff coverage is `92.85%`.
[![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/361/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/361?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
```diff
@@ Coverage Diff @@
## main #361 +/- ##
==========================================
- Coverage 96.49% 96.44% -0.06%
==========================================
Files 5 5
Lines 2283 2306 +23
==========================================
+ Hits 2203 2224 +21
- Misses 80 82 +2
```
| [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/361?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [sqlite\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/361/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.49% <92.00%> (-0.11%)` | :arrow_down: |
| [sqlite\_utils/utils.py](https://codecov.io/gh/simonw/sqlite-utils/pull/361/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `94.23% <100.00%> (ø)` | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/361?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/361?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [f3fd861...b7f0b88](https://codecov.io/gh/simonw/sqlite-utils/pull/361?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import,
https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006315145,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006315145,IC_kwDOCGYnMM47-yaJ,9599,simonw,2022-01-06T06:20:51Z,2022-01-06T06:20:51Z,OWNER,This is all documented. I'm going to rebase-merge it to keep the individual commits.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import,
https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006311742,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006311742,IC_kwDOCGYnMM47-xk-,9599,simonw,2022-01-06T06:12:19Z,2022-01-06T06:12:19Z,OWNER,"Got that working:
```
% echo 'This is cool' | sqlite-utils insert words.db words - --text --convert '({""word"": w} for w in text.split())'
% sqlite-utils dump words.db
BEGIN TRANSACTION;
CREATE TABLE [words] (
[word] TEXT
);
INSERT INTO ""words"" VALUES('This');
INSERT INTO ""words"" VALUES('is');
INSERT INTO ""words"" VALUES('cool');
COMMIT;
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import,
https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006309834,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006309834,IC_kwDOCGYnMM47-xHK,9599,simonw,2022-01-06T06:08:01Z,2022-01-06T06:08:01Z,OWNER,"For `--text` the conversion function should be allowed to return an iterable instead of a dictionary, in which case it will be treated as the full list of records to be inserted.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import,
https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006301546,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006301546,IC_kwDOCGYnMM47-vFq,9599,simonw,2022-01-06T05:44:47Z,2022-01-06T05:44:47Z,OWNER,Just need documentation for `--convert` now against the various different types of input.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import,
https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006300280,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006300280,IC_kwDOCGYnMM47-ux4,9599,simonw,2022-01-06T05:40:45Z,2022-01-06T05:40:45Z,OWNER,"I'm going to rename `--all` to `--text`:
> - Use `--text` to write the entire input to a column called ""text""
To avoid that clash with Python's `all()` function.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import,
https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006299778,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006299778,IC_kwDOCGYnMM47-uqC,9599,simonw,2022-01-06T05:39:10Z,2022-01-06T05:39:10Z,OWNER,`all` is a bad variable name because it clashes with the Python `all()` built-in function.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import,
https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006295276,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006295276,IC_kwDOCGYnMM47-tjs,9599,simonw,2022-01-06T05:26:11Z,2022-01-06T05:26:11Z,OWNER,"Here's the traceback if your `--convert` function doesn't return a dict right now:
```
% sqlite-utils insert /tmp/all.db blah /tmp/log.log --convert 'all.upper()' --all
Traceback (most recent call last):
File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/bin/sqlite-utils"", line 33, in
sys.exit(load_entry_point('sqlite-utils', 'console_scripts', 'sqlite-utils')())
File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 1137, in __call__
return self.main(*args, **kwargs)
File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 1062, in main
rv = self.invoke(ctx)
File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 1668, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 763, in invoke
return __callback(*args, **kwargs)
File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py"", line 949, in insert
insert_upsert_implementation(
File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py"", line 834, in insert_upsert_implementation
db[table].insert_all(
File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 2602, in insert_all
first_record = next(records)
File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 3044, in fix_square_braces
for record in records:
File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py"", line 831, in
docs = (decode_base64_values(doc) for doc in docs)
File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py"", line 86, in decode_base64_values
to_fix = [
File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py"", line 89, in
if isinstance(doc[k], dict)
TypeError: string indices must be integers
```
I can live with that for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import,
https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006294777,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006294777,IC_kwDOCGYnMM47-tb5,9599,simonw,2022-01-06T05:24:54Z,2022-01-06T05:24:54Z,OWNER,"> I added a custom error message for if the user's `--convert` code doesn't return a dict.
That turned out to be a bad idea because it meant exhausting the iterator early for the check - before we got to the `.insert_all()` code that breaks the iterator up into chunks. I tried fixing that with `itertools.tee()` to run the generator twice but that's grossly memory-inefficient for large imports.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import,
https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006288444,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006288444,IC_kwDOCGYnMM47-r48,9599,simonw,2022-01-06T05:07:10Z,2022-01-06T05:07:10Z,OWNER,"And here's a demo of `--convert` used with `--all` - I added a custom error message for if the user's `--convert` code doesn't return a dict.
```
% sqlite-utils insert /tmp/all.db blah /tmp/log.log --convert 'all.upper()' --all
Error: Records returned by your --convert function must be dicts
% sqlite-utils insert /tmp/all.db blah /tmp/log.log --convert '{""all"": all.upper()}' --all
% sqlite-utils dump /tmp/all.db
BEGIN TRANSACTION;
CREATE TABLE [blah] (
[all] TEXT
);
INSERT INTO ""blah"" VALUES('INFO: 127.0.0.1:60581 - ""GET / HTTP/1.1"" 200 OK
INFO: 127.0.0.1:60581 - ""GET /FOO/-/STATIC/APP.CSS?CEAD5A HTTP/1.1"" 200 OK
INFO: 127.0.0.1:60581 - ""GET /FAVICON.ICO HTTP/1.1"" 200 OK
INFO: 127.0.0.1:60581 - ""GET /FOO/TIDDLYWIKI HTTP/1.1"" 200 OK
INFO: 127.0.0.1:60581 - ""GET /FOO/-/STATIC/APP.CSS?CEAD5A HTTP/1.1"" 200 OK
INFO: 127.0.0.1:60584 - ""GET /FOO/-/STATIC/SQL-FORMATTER-2.3.3.MIN.JS HTTP/1.1"" 200 OK
INFO: 127.0.0.1:60586 - ""GET /FOO/-/STATIC/CODEMIRROR-5.57.0.MIN.JS HTTP/1.1"" 200 OK
INFO: 127.0.0.1:60585 - ""GET /FOO/-/STATIC/CODEMIRROR-5.57.0.MIN.CSS HTTP/1.1"" 200 OK
INFO: 127.0.0.1:60588 - ""GET /FOO/-/STATIC/CODEMIRROR-5.57.0-SQL.MIN.JS HTTP/1.1"" 200 OK
INFO: 127.0.0.1:60587 - ""GET /FOO/-/STATIC/CM-RESIZE-1.0.1.MIN.JS HTTP/1.1"" 200 OK
INFO: 127.0.0.1:60586 - ""GET /FOO/TIDDLYWIKI/TIDDLERS HTTP/1.1"" 200 OK
INFO: 127.0.0.1:60586 - ""GET /FOO/-/STATIC/APP.CSS?CEAD5A HTTP/1.1"" 200 OK
INFO: 127.0.0.1:60584 - ""GET /FOO/-/STATIC/TABLE.JS HTTP/1.1"" 200 OK
');
COMMIT;
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import,
https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006284673,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006284673,IC_kwDOCGYnMM47-q-B,9599,simonw,2022-01-06T04:55:52Z,2022-01-06T04:55:52Z,OWNER,"Test code that just worked for me:
```
sqlite-utils insert /tmp/blah.db blah /tmp/log.log --convert '
bits = line.split()
return dict([(""b_{}"".format(i), bit) for i, bit in enumerate(bits)])' --lines
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import,
https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006232013,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006232013,IC_kwDOCGYnMM47-eHN,9599,simonw,2022-01-06T02:21:35Z,2022-01-06T02:21:35Z,OWNER,"I'm having second thoughts about this bit:
> Your Python code will be passed a ""row"" variable representing the imported row, and can return a modified row.
>
> If you are using `--lines` your code will be passed a ""line"" variable, and for `--all` an ""all"" variable.
The code in question is this:
https://github.com/simonw/sqlite-utils/blob/500a35ad4d91c8a6232134ce9406efec11bedff8/sqlite_utils/utils.py#L296-L303
Do I really want to add the complexity of supporting different variable names there? I think always using `value` might be better.
Except... `value` made sense for the existing `sqlite-utils convert` command where you are running a conversion function against the value for the column in the current row - is it confusing if applied to lines or documents or `all`?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import,
https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006230411,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006230411,IC_kwDOCGYnMM47-duL,9599,simonw,2022-01-06T02:17:35Z,2022-01-06T02:17:35Z,OWNER,"Documentation: https://github.com/simonw/sqlite-utils/blob/33223856ff7fe746b7b77750fbe5b218531d0545/docs/cli.rst#inserting-unstructured-data-with---lines-and---all - I went with a single section titled ""Inserting unstructured data with --lines and --all""","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import,
https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006220129,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006220129,IC_kwDOCGYnMM47-bNh,9599,simonw,2022-01-06T01:52:26Z,2022-01-06T01:52:26Z,OWNER,I'm going to refactor all of the tests for `sqlite-utils insert` into a new `test_cli_insert.py` module.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import,
https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006219848,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006219848,IC_kwDOCGYnMM47-bJI,9599,simonw,2022-01-06T01:51:36Z,2022-01-06T01:51:36Z,OWNER,"So far I've just implemented the new help:
```
% sqlite-utils insert --help
Usage: sqlite-utils insert [OPTIONS] PATH TABLE FILE
Insert records from FILE into a table, creating the table if it does not
already exist.
By default the input is expected to be a JSON array of objects. Or:
- Use --nl for newline-delimited JSON objects
- Use --csv or --tsv for comma-separated or tab-separated input
- Use --lines to write each incoming line to a column called ""line""
- Use --all to write the entire input to a column called ""all""
You can also use --convert to pass a fragment of Python code that will be
used to convert each input.
Your Python code will be passed a ""row"" variable representing the imported
row, and can return a modified row.
If you are using --lines your code will be passed a ""line"" variable, and for
--all an ""all"" variable.
Options:
--pk TEXT Columns to use as the primary key, e.g. id
--flatten Flatten nested JSON objects, so {""a"": {""b"": 1}}
becomes {""a_b"": 1}
--nl Expect newline-delimited JSON
-c, --csv Expect CSV input
--tsv Expect TSV input
--lines Treat each line as a single value called 'line'
--all Treat input as a single value called 'all'
--convert TEXT Python code to convert each item
--import TEXT Python modules to import
--delimiter TEXT Delimiter to use for CSV files
--quotechar TEXT Quote character to use for CSV/TSV
--sniff Detect delimiter and quote character
--no-headers CSV file has no header row
--batch-size INTEGER Commit every X records
--alter Alter existing table to add any missing columns
--not-null TEXT Columns that should be created as NOT NULL
--default ... Default value that should be set for a column
--encoding TEXT Character encoding for input, defaults to utf-8
-d, --detect-types Detect types for columns in CSV/TSV data
--load-extension TEXT SQLite extensions to load
--silent Do not show progress bar
--ignore Ignore records if pk already exists
--replace Replace records if pk already exists
--truncate Truncate table before inserting records, if table
already exists
-h, --help Show this message and exit.
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import,
https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997496626,https://api.github.com/repos/simonw/sqlite-utils/issues/356,997496626,IC_kwDOCGYnMM47dJcy,9599,simonw,2021-12-20T00:38:15Z,2022-01-06T01:29:03Z,OWNER,"The implementation of this gets a tiny bit complicated.
Ignoring `--convert`, the `--lines` option can internally produce `{""line"": ...}` records and the `--all` option can produce `{""all"": ...}` records.
But... when `--convert` is used, what should the code run against?
It could run against those already-converted records but that's a little bit strange, since you'd have to do this:
sqlite-utils insert blah.db blah myfile.txt --all --convert '{""item"": s for s in value[""all""].split(""-"")}'
Having to use `value[""all""]` there is unintuitive. It would be nicer to have a `all` variable to work against.
But then for `--lines` should the local variable be called `line`? And how best to summarize these different names for local variables in the inline help for the feature?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option,
https://github.com/simonw/sqlite-utils/issues/360#issuecomment-1006211113,https://api.github.com/repos/simonw/sqlite-utils/issues/360,1006211113,IC_kwDOCGYnMM47-ZAp,9599,simonw,2022-01-06T01:27:53Z,2022-01-06T01:27:53Z,OWNER,"It looks like you were using `sqlite-utils memory` - that works by loading the entire file into an in-memory database, so 170GB is very likely to run out of RAM.
The line of code there exhibits another problem: it's reading the entire JSON file into a Python string, so it looks like it's going to run out of RAM even before it gets to the SQLite in-memory database section.
To handle a file of this size you'd need to write it to a SQLite database on-disk first. The `sqlite-utils insert` command can do this, and it should be able to ""stream"" records in from a file without loading the entire thing into memory - but only for JSON-NL and CSV/TSV formats, not for JSON arrays.
The code in question is here:
https://github.com/simonw/sqlite-utils/blob/f3fd8613113d21d44238a6ec54b375f5aa72c4e0/sqlite_utils/cli.py#L738-L773
That's using Python generators for the CSV/TSV/JSON-NL variants... but it's doing this for regular JSON which requires reading the entire thing into memory:
https://github.com/simonw/sqlite-utils/blob/f3fd8613113d21d44238a6ec54b375f5aa72c4e0/sqlite_utils/cli.py#L767
If you have the ability to control how your 170GB file is generated you may have more luck converting it to CSV or TSV or newline-delimited JSON, then using `sqlite-utils insert` to insert it into a database file.
To be honest though I've never tested this tooling with anything nearly that big, so it's possible you'll still run into problems. If you do I'd love to hear about them!
I would be tempted to tackle this size of job by writing a custom Python script, either using the `sqlite_utils` Python library or even calling `sqlite3` directly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1091819089,MemoryError,
https://github.com/simonw/datasette/issues/1534#issuecomment-1005975080,https://api.github.com/repos/simonw/datasette/issues/1534,1005975080,IC_kwDOBm6k_c479fYo,9599,simonw,2022-01-05T18:29:06Z,2022-01-05T18:29:06Z,OWNER,"A really big downside to this is that it turns out many CDNs - apparently including Cloudflare - don't support the Vary header at all!
More in this thread: https://twitter.com/simonw/status/1478470282931163137","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065432388,Maybe return JSON from HTML pages if `Accept: application/json` is sent,
https://github.com/simonw/datasette/issues/1585#issuecomment-1003575286,https://api.github.com/repos/simonw/datasette/issues/1585,1003575286,IC_kwDOBm6k_c470Vf2,9599,simonw,2022-01-01T15:40:38Z,2022-01-01T15:40:38Z,OWNER,API tutorial: https://firebase.google.com/docs/hosting/api-deploy,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1091838742,Fire base caching for `publish cloudrun`,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/8#issuecomment-1003437288,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/8,1003437288,IC_kwDODFE5qs47zzzo,28565,maxhawkins,2021-12-31T19:06:20Z,2021-12-31T19:06:20Z,NONE,"> @maxhawkins how hard would it be to add an entry to the table that includes the HTML version of the email, if it exists? I just attempted your the PR branch on a very small mbox file, and it worked great. My use case is a research project and I need to access more than just the body plain text.
Shouldn't be hard. The easiest way is probably to remove the `if body.content_type == ""text/html""` clause from [utils.py:254](https://github.com/dogsheep/google-takeout-to-sqlite/pull/8/commits/8e6d487b697ce2e8ad885acf613a157bfba84c59#diff-25ad9dd1ced1b8bfc37fda8444819c803232c08891e4af3d4064aa205d8174eaR254) and just return content directly without parsing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",954546309,Add Gmail takeout mbox import (v2),
https://github.com/simonw/datasette/issues/1583#issuecomment-1002825217,https://api.github.com/repos/simonw/datasette/issues/1583,1002825217,IC_kwDOBm6k_c47xeYB,536941,fgregg,2021-12-30T00:34:16Z,2021-12-30T00:34:16Z,CONTRIBUTOR,"if that is not desirable, it might be good to document that users might want to set up a lifecycle rule to automatically delete these build artifacts. something like https://stackoverflow.com/questions/59937542/can-i-delete-container-images-from-google-cloud-storage-artifacts-bucket","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1090810196,consider adding deletion step of cloudbuild artifacts to gcloud publish,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/8#issuecomment-1002735370,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/8,1002735370,IC_kwDODFE5qs47xIcK,203343,Btibert3,2021-12-29T18:58:23Z,2021-12-29T18:58:23Z,NONE,"@maxhawkins how hard would it be to add an entry to the table that includes the HTML version of the email, if it exists? I just attempted your the PR branch on a very small mbox file, and it worked great. My use case is a research project and I need to access more than just the body plain text.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",954546309,Add Gmail takeout mbox import (v2),
https://github.com/simonw/datasette/issues/1152#issuecomment-1001791592,https://api.github.com/repos/simonw/datasette/issues/1152,1001791592,IC_kwDOBm6k_c47tiBo,9599,simonw,2021-12-27T23:04:31Z,2021-12-27T23:04:31Z,OWNER,Another option: rethink permissions to always work in terms of where clauses users as part of a SQL query that returns the overall allowed set of databases or tables. This would require rethinking existing permissions but it might be worthwhile prior to 1.0.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770598024,Efficiently calculate list of databases/tables a user can view,
https://github.com/simonw/datasette/issues/878#issuecomment-1001699559,https://api.github.com/repos/simonw/datasette/issues/878,1001699559,IC_kwDOBm6k_c47tLjn,9599,simonw,2021-12-27T18:53:04Z,2021-12-27T18:53:04Z,OWNER,"I'm going to see if I can come up with the simplest possible version of this pattern for the `/-/metadata` and `/-/metadata.json` page, then try it for the database query page, before tackling the much more complex table page.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/dogsheep/twitter-to-sqlite/issues/62#issuecomment-1001222213,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/62,1001222213,IC_kwDODEm0Qs47rXBF,6764957,swyxio,2021-12-26T17:59:25Z,2021-12-26T17:59:25Z,NONE,just confirmed that this error does not occur when i use my public main account. gets more interesting!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1088816961,KeyError: 'created_at' for private accounts?,
https://github.com/simonw/sqlite-utils/issues/228#issuecomment-1001115286,https://api.github.com/repos/simonw/sqlite-utils/issues/228,1001115286,IC_kwDOCGYnMM47q86W,1206106,agguser,2021-12-26T07:01:31Z,2021-12-26T07:01:31Z,NONE,"`--no-headers` does not work?
```
$ echo 'a,1\nb,2' | sqlite-utils memory --no-headers -t - 'select * from stdin'
a 1
--- ---
b 2
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807437089,--no-headers option for CSV and TSV,
https://github.com/simonw/datasette/issues/1576#issuecomment-1000935523,https://api.github.com/repos/simonw/datasette/issues/1576,1000935523,IC_kwDOBm6k_c47qRBj,9599,simonw,2021-12-24T21:33:05Z,2021-12-24T21:33:05Z,OWNER,"Another option would be to attempt to import `contextvars` and, if the import fails (for Python 3.6) continue using the current mechanism - then let Python 3.6 users know in the documentation that under Python 3.6 they will miss out on nested traces.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`,
https://github.com/simonw/datasette/issues/1577#issuecomment-1000673444,https://api.github.com/repos/simonw/datasette/issues/1577,1000673444,IC_kwDOBm6k_c47pRCk,9599,simonw,2021-12-24T06:08:58Z,2021-12-24T06:08:58Z,OWNER,"https://pypistats.org/packages/datasette shows a breakdown of downloads by Python version:
It looks like on a recent day I had 4,071 downloads from Python 3.7... and just 2 downloads from Python 3.6!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087913724,Drop support for Python 3.6,
https://github.com/simonw/datasette/issues/1534#issuecomment-1000535904,https://api.github.com/repos/simonw/datasette/issues/1534,1000535904,IC_kwDOBm6k_c47ovdg,9599,simonw,2021-12-23T21:44:31Z,2021-12-23T21:44:31Z,OWNER,A big downside to this is that I would need to use `Vary: Accept` for when Datasette is running behind a cache such as Cloudflare - would that greatly reduce overall cache efficiency due to subtle variations in the accept headers sent by common browsers?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065432388,Maybe return JSON from HTML pages if `Accept: application/json` is sent,
https://github.com/simonw/datasette/issues/1579#issuecomment-1000485719,https://api.github.com/repos/simonw/datasette/issues/1579,1000485719,IC_kwDOBm6k_c47ojNX,9599,simonw,2021-12-23T19:19:45Z,2021-12-23T19:19:45Z,OWNER,All of those removed `block=True` lines in 8c401ee0f054de2f568c3a8302c9223555146407 really help confirm to me that this was a good decision.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087931918,`.execute_write(... block=True)` should be the default behaviour,
https://github.com/simonw/datasette/issues/1579#issuecomment-1000485505,https://api.github.com/repos/simonw/datasette/issues/1579,1000485505,IC_kwDOBm6k_c47ojKB,9599,simonw,2021-12-23T19:19:13Z,2021-12-23T19:19:13Z,OWNER,Updated docs for `execute_write_fn()`: https://github.com/simonw/datasette/blob/75153ea9b94d09ec3d61f7c6ebdf378e0c0c7a0b/docs/internals.rst#await-dbexecute_write_fnfn-blocktrue,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087931918,`.execute_write(... block=True)` should be the default behaviour,
https://github.com/simonw/datasette/issues/1579#issuecomment-1000481686,https://api.github.com/repos/simonw/datasette/issues/1579,1000481686,IC_kwDOBm6k_c47oiOW,9599,simonw,2021-12-23T19:09:23Z,2021-12-23T19:09:23Z,OWNER,"Re-opening this because I missed updating some of the docs, and I also need to update Datasette's own code to not use `block=True` in a bunch of places.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087931918,`.execute_write(... block=True)` should be the default behaviour,
https://github.com/simonw/datasette/issues/1579#issuecomment-1000479737,https://api.github.com/repos/simonw/datasette/issues/1579,1000479737,IC_kwDOBm6k_c47ohv5,9599,simonw,2021-12-23T19:04:23Z,2021-12-23T19:04:23Z,OWNER,Updated documentation: https://github.com/simonw/datasette/blob/00a2895cd2dc42c63846216b36b2dc9f41170129/docs/internals.rst#await-dbexecute_writesql-paramsnone-blocktrue,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087931918,`.execute_write(... block=True)` should be the default behaviour,
https://github.com/simonw/datasette/issues/1579#issuecomment-1000477813,https://api.github.com/repos/simonw/datasette/issues/1579,1000477813,IC_kwDOBm6k_c47ohR1,9599,simonw,2021-12-23T18:59:41Z,2021-12-23T18:59:41Z,OWNER,"I'm going to go with `execute_write(..., block=False)` as the mechanism for fire-and-forget write queries.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087931918,`.execute_write(... block=True)` should be the default behaviour,
https://github.com/simonw/datasette/issues/1579#issuecomment-1000477621,https://api.github.com/repos/simonw/datasette/issues/1579,1000477621,IC_kwDOBm6k_c47ohO1,9599,simonw,2021-12-23T18:59:12Z,2021-12-23T18:59:12Z,OWNER,"The easiest way to change this would be to default to `block=True` such that you need to pass `block=False` to the APIs to have them do fire-and-forget.
An alternative would be to add new, separately named methods which do the fire-and-forget thing.
If I hadn't recently added `execute_write_script` and `execute_write_many` in #1570 I'd be more into this idea, but I don't want to end up with eight methods - `execute_write`, `execute_write_queue`, `execute_write_many`, `execute_write_many_queue`, `execute_write_script`, `execute_write_scrript_queue`, `execute_write_fn`, `execute_write_fn_queue`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087931918,`.execute_write(... block=True)` should be the default behaviour,
https://github.com/simonw/datasette/issues/1579#issuecomment-1000476413,https://api.github.com/repos/simonw/datasette/issues/1579,1000476413,IC_kwDOBm6k_c47og79,9599,simonw,2021-12-23T18:56:06Z,2021-12-23T18:56:06Z,OWNER,"This is technically a breaking change, but a GitHub code search at https://cs.github.com/?scopeName=All+repos&scope=&q=execute_write%20datasette%20-owner%3Asimonw shows only one repo not-owned-by-me using this, and they're using `block=True`: https://github.com/mfa/datasette-webhook-write/blob/e82440f372a2f2e3ed27d1bd34c9fa3a53b49b94/datasette_webhook_write/__init__.py#L88-L89","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087931918,`.execute_write(... block=True)` should be the default behaviour,
https://github.com/simonw/datasette/issues/1578#issuecomment-1000471782,https://api.github.com/repos/simonw/datasette/issues/1578,1000471782,IC_kwDOBm6k_c47ofzm,9599,simonw,2021-12-23T18:44:01Z,2021-12-23T18:44:01Z,OWNER,"The example nginx config on https://docs.datasette.io/en/stable/deploying.html#nginx-proxy-configuration is currently:
```
daemon off;
events {
worker_connections 1024;
}
http {
server {
listen 80;
location /my-datasette {
proxy_pass http://127.0.0.1:8009/my-datasette;
proxy_set_header Host $host;
}
}
}
```
This looks to me like it might exhibit the bug. Need to confirm that and figure out an alternative.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087919372,Confirm if documented nginx proxy config works for row pages with escaped characters in their primary key,
https://github.com/simonw/datasette/issues/1578#issuecomment-1000471371,https://api.github.com/repos/simonw/datasette/issues/1578,1000471371,IC_kwDOBm6k_c47oftL,9599,simonw,2021-12-23T18:42:50Z,2021-12-23T18:42:50Z,OWNER,"Confirmed, that fixed the bug for me on my server.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087919372,Confirm if documented nginx proxy config works for row pages with escaped characters in their primary key,
https://github.com/simonw/datasette/issues/1578#issuecomment-1000470652,https://api.github.com/repos/simonw/datasette/issues/1578,1000470652,IC_kwDOBm6k_c47ofh8,9599,simonw,2021-12-23T18:40:46Z,2021-12-23T18:40:46Z,OWNER,"[This StackOverflow answer](https://serverfault.com/a/463932) suggests that the fix is to change this:
proxy_pass http://127.0.0.1:8000/;
To this:
proxy_pass http://127.0.0.1:8000;
Quoting the nginx documentation: http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass
> A request URI is passed to the server as follows:
>
> - If the `proxy_pass` directive is specified with a URI, then when a request is passed to the server, the part of a [normalized](http://nginx.org/en/docs/http/ngx_http_core_module.html#location) request URI matching the location is replaced by a URI specified in the directive:
>
> location /name/ {
> proxy_pass http://127.0.0.1/remote/;
> }
>
> - If `proxy_pass` is specified without a URI, the request URI is passed to the server in the same form as sent by a client when the original request is processed, or the full normalized request URI is passed when processing the changed URI:
>
> location /some/path/ {
> proxy_pass http://127.0.0.1;
> }","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087919372,Confirm if documented nginx proxy config works for row pages with escaped characters in their primary key,
https://github.com/simonw/datasette/issues/1578#issuecomment-1000469107,https://api.github.com/repos/simonw/datasette/issues/1578,1000469107,IC_kwDOBm6k_c47ofJz,9599,simonw,2021-12-23T18:36:38Z,2021-12-23T18:36:38Z,OWNER,"This problem doesn't occur on my `localhost` running Uvicorn directly - but I'm seeing it in my production environment that runs Datasette behind an nginx proxy:
```
location / {
proxy_pass http://127.0.0.1:8000/;
proxy_set_header Host $host;
}
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087919372,Confirm if documented nginx proxy config works for row pages with escaped characters in their primary key,
https://github.com/simonw/datasette/issues/1577#issuecomment-1000462309,https://api.github.com/repos/simonw/datasette/issues/1577,1000462309,IC_kwDOBm6k_c47odfl,9599,simonw,2021-12-23T18:20:46Z,2021-12-23T18:20:46Z,OWNER,There are a lot of improvements to `asyncio` in 3.7: https://docs.python.org/3/whatsnew/3.7.html#whatsnew37-asyncio,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087913724,Drop support for Python 3.6,
https://github.com/simonw/datasette/issues/1577#issuecomment-1000461900,https://api.github.com/repos/simonw/datasette/issues/1577,1000461900,IC_kwDOBm6k_c47odZM,9599,simonw,2021-12-23T18:19:44Z,2021-12-23T18:19:44Z,OWNER,"The 3.7 feature I want to use today is [contextvars](https://docs.python.org/3/library/contextvars.html) - but I have a workaround for the moment, see https://github.com/simonw/datasette/issues/1576#issuecomment-999987418
So I'm going to hold off on dropping 3.6 for a little bit longer. I imagine I'll drop it before Datasette 1.0 though.
Leaving this issue open to gather thoughts and feedback on this issue from Datasette users and potential users.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087913724,Drop support for Python 3.6,
https://github.com/simonw/datasette/issues/1577#issuecomment-1000461275,https://api.github.com/repos/simonw/datasette/issues/1577,1000461275,IC_kwDOBm6k_c47odPb,9599,simonw,2021-12-23T18:18:11Z,2021-12-23T18:18:11Z,OWNER,"From the Twitter thread, there are still a decent amount of LTS Linux releases out there that are stuck on pre-3.7 Python.
Though many of those are 3.5 and Datasette dropped support for 3.5 in November 2019: cf7776d36fbacefa874cbd6e5fcdc9fff7661203","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087913724,Drop support for Python 3.6,
https://github.com/simonw/datasette/issues/1576#issuecomment-999990414,https://api.github.com/repos/simonw/datasette/issues/1576,999990414,IC_kwDOBm6k_c47mqSO,9599,simonw,2021-12-23T02:08:39Z,2021-12-23T18:16:35Z,OWNER,"It's tiny: I'm tempted to vendor it. https://github.com/Skyscanner/aiotask-context/blob/master/aiotask_context/__init__.py
No, I'll add it as a pinned dependency, which I can then drop when I drop 3.6 support.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`,
https://github.com/simonw/datasette/issues/1576#issuecomment-999987418,https://api.github.com/repos/simonw/datasette/issues/1576,999987418,IC_kwDOBm6k_c47mpja,9599,simonw,2021-12-23T01:59:58Z,2021-12-23T02:02:12Z,OWNER,"Another option: https://github.com/Skyscanner/aiotask-context - looks like it might be better as it's been updated for Python 3.7 in this commit https://github.com/Skyscanner/aiotask-context/commit/67108c91d2abb445655cc2af446fdb52ca7890c4
The Skyscanner one doesn't attempt to wrap any existing factories, but that's OK for my purposes since I don't need to handle arbitrary `asyncio` code written by other people.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`,
https://github.com/simonw/datasette/issues/1576#issuecomment-999876666,https://api.github.com/repos/simonw/datasette/issues/1576,999876666,IC_kwDOBm6k_c47mOg6,9599,simonw,2021-12-22T20:59:22Z,2021-12-22T21:18:09Z,OWNER,"This article is relevant: [Context information storage for asyncio](https://blog.sqreen.com/asyncio/) - in particular the section https://blog.sqreen.com/asyncio/#context-inheritance-between-tasks which describes exactly the problem I have and their solution, which involves this trickery:
```python
def request_task_factory(loop, coro):
child_task = asyncio.tasks.Task(coro, loop=loop)
parent_task = asyncio.Task.current_task(loop=loop)
current_request = getattr(parent_task, 'current_request', None)
setattr(child_task, 'current_request', current_request)
return child_task
loop = asyncio.get_event_loop()
loop.set_task_factory(request_task_factory)
```
They released their solution as a library: https://pypi.org/project/aiocontext/ and https://github.com/sqreen/AioContext - but that company was acquired by Datadog back in April and doesn't seem to be actively maintaining their open source stuff any more: https://twitter.com/SqreenIO/status/1384906075506364417","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`,
https://github.com/simonw/datasette/issues/1576#issuecomment-999878907,https://api.github.com/repos/simonw/datasette/issues/1576,999878907,IC_kwDOBm6k_c47mPD7,9599,simonw,2021-12-22T21:03:49Z,2021-12-22T21:10:46Z,OWNER,"`context_vars` can solve this but they were introduced in Python 3.7: https://www.python.org/dev/peps/pep-0567/
Python 3.6 support ends in a few days time, and it looks like Glitch has updated to 3.7 now - so maybe I can get away with Datasette needing 3.7 these days?
Tweeted about that here: https://twitter.com/simonw/status/1473761478155010048","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`,
https://github.com/simonw/datasette/issues/1576#issuecomment-999874886,https://api.github.com/repos/simonw/datasette/issues/1576,999874886,IC_kwDOBm6k_c47mOFG,9599,simonw,2021-12-22T20:55:42Z,2021-12-22T20:57:28Z,OWNER,"One way to solve this would be to introduce a `set_task_id()` method, which sets an ID which will be returned by `get_task_id()` instead of using `id(current_task(loop=loop))`.
It would be really nice if I could solve this using `with` syntax somehow. Something like:
```python
with trace_child_tasks():
(
suggested_facets,
(facet_results, facets_timed_out),
) = await asyncio.gather(
execute_suggested_facets(),
execute_facets(),
)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`,
https://github.com/simonw/datasette/issues/1576#issuecomment-999874484,https://api.github.com/repos/simonw/datasette/issues/1576,999874484,IC_kwDOBm6k_c47mN-0,9599,simonw,2021-12-22T20:54:52Z,2021-12-22T20:54:52Z,OWNER,"Here's the full current relevant code from `tracer.py`: https://github.com/simonw/datasette/blob/ace86566b28280091b3844cf5fbecd20158e9004/datasette/tracer.py#L8-L64
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`,
https://github.com/simonw/datasette/issues/1518#issuecomment-999870993,https://api.github.com/repos/simonw/datasette/issues/1518,999870993,IC_kwDOBm6k_c47mNIR,9599,simonw,2021-12-22T20:47:18Z,2021-12-22T20:50:24Z,OWNER,"The reason they aren't showing up in the traces is that traces are stored just for the currently executing `asyncio` task ID: https://github.com/simonw/datasette/blob/ace86566b28280091b3844cf5fbecd20158e9004/datasette/tracer.py#L13-L25
This is so traces for other incoming requests don't end up mixed together. But there's no current mechanism to track async tasks that are effectively ""child tasks"" of the current request, and hence should be tracked the same.
https://stackoverflow.com/a/69349501/6083 suggests that you pass the task ID as an argument to the child tasks that are executed using `asyncio.gather()` to work around this kind of problem.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/issues/1518#issuecomment-999870282,https://api.github.com/repos/simonw/datasette/issues/1518,999870282,IC_kwDOBm6k_c47mM9K,9599,simonw,2021-12-22T20:45:56Z,2021-12-22T20:46:08Z,OWNER,"> New short-term goal: get facets and suggested facets to execute in parallel with the main query. Generate a trace graph that proves that is happening using `datasette-pretty-traces`.
I wrote code to execute those in parallel using `asyncio.gather()` - which seems to work but causes the SQL run inside the parallel `async def` functions not to show up in the trace graph at all.
```diff
diff --git a/datasette/views/table.py b/datasette/views/table.py
index 9808fd2..ec9db64 100644
--- a/datasette/views/table.py
+++ b/datasette/views/table.py
@@ -1,3 +1,4 @@
+import asyncio
import urllib
import itertools
import json
@@ -615,44 +616,37 @@ class TableView(RowTableShared):
if request.args.get(""_timelimit""):
extra_args[""custom_time_limit""] = int(request.args.get(""_timelimit""))
- # Execute the main query!
- results = await db.execute(sql, params, truncate=True, **extra_args)
-
- # Calculate the total count for this query
- filtered_table_rows_count = None
- if (
- not db.is_mutable
- and self.ds.inspect_data
- and count_sql == f""select count(*) from {table} ""
- ):
- # We can use a previously cached table row count
- try:
- filtered_table_rows_count = self.ds.inspect_data[database][""tables""][
- table
- ][""count""]
- except KeyError:
- pass
-
- # Otherwise run a select count(*) ...
- if count_sql and filtered_table_rows_count is None and not nocount:
- try:
- count_rows = list(await db.execute(count_sql, from_sql_params))
- filtered_table_rows_count = count_rows[0][0]
- except QueryInterrupted:
- pass
-
- # Faceting
- if not self.ds.setting(""allow_facet"") and any(
- arg.startswith(""_facet"") for arg in request.args
- ):
- raise BadRequest(""_facet= is not allowed"")
+ async def execute_count():
+ # Calculate the total count for this query
+ filtered_table_rows_count = None
+ if (
+ not db.is_mutable
+ and self.ds.inspect_data
+ and count_sql == f""select count(*) from {table} ""
+ ):
+ # We can use a previously cached table row count
+ try:
+ filtered_table_rows_count = self.ds.inspect_data[database][
+ ""tables""
+ ][table][""count""]
+ except KeyError:
+ pass
+
+ if count_sql and filtered_table_rows_count is None and not nocount:
+ try:
+ count_rows = list(await db.execute(count_sql, from_sql_params))
+ filtered_table_rows_count = count_rows[0][0]
+ except QueryInterrupted:
+ pass
+
+ return filtered_table_rows_count
+
+ filtered_table_rows_count = await execute_count()
# pylint: disable=no-member
facet_classes = list(
itertools.chain.from_iterable(pm.hook.register_facet_classes())
)
- facet_results = {}
- facets_timed_out = []
facet_instances = []
for klass in facet_classes:
facet_instances.append(
@@ -668,33 +662,58 @@ class TableView(RowTableShared):
)
)
- if not nofacet:
- for facet in facet_instances:
- (
- instance_facet_results,
- instance_facets_timed_out,
- ) = await facet.facet_results()
- for facet_info in instance_facet_results:
- base_key = facet_info[""name""]
- key = base_key
- i = 1
- while key in facet_results:
- i += 1
- key = f""{base_key}_{i}""
- facet_results[key] = facet_info
- facets_timed_out.extend(instance_facets_timed_out)
-
- # Calculate suggested facets
- suggested_facets = []
- if (
- self.ds.setting(""suggest_facets"")
- and self.ds.setting(""allow_facet"")
- and not _next
- and not nofacet
- and not nosuggest
- ):
- for facet in facet_instances:
- suggested_facets.extend(await facet.suggest())
+ async def execute_suggested_facets():
+ # Calculate suggested facets
+ suggested_facets = []
+ if (
+ self.ds.setting(""suggest_facets"")
+ and self.ds.setting(""allow_facet"")
+ and not _next
+ and not nofacet
+ and not nosuggest
+ ):
+ for facet in facet_instances:
+ suggested_facets.extend(await facet.suggest())
+ return suggested_facets
+
+ async def execute_facets():
+ facet_results = {}
+ facets_timed_out = []
+ if not self.ds.setting(""allow_facet"") and any(
+ arg.startswith(""_facet"") for arg in request.args
+ ):
+ raise BadRequest(""_facet= is not allowed"")
+
+ if not nofacet:
+ for facet in facet_instances:
+ (
+ instance_facet_results,
+ instance_facets_timed_out,
+ ) = await facet.facet_results()
+ for facet_info in instance_facet_results:
+ base_key = facet_info[""name""]
+ key = base_key
+ i = 1
+ while key in facet_results:
+ i += 1
+ key = f""{base_key}_{i}""
+ facet_results[key] = facet_info
+ facets_timed_out.extend(instance_facets_timed_out)
+
+ return facet_results, facets_timed_out
+
+ # Execute the main query, facets and facet suggestions in parallel:
+ (
+ results,
+ suggested_facets,
+ (facet_results, facets_timed_out),
+ ) = await asyncio.gather(
+ db.execute(sql, params, truncate=True, **extra_args),
+ execute_suggested_facets(),
+ execute_facets(),
+ )
+
+ results = await db.execute(sql, params, truncate=True, **extra_args)
# Figure out columns and rows for the query
columns = [r[0] for r in results.description]
```
Here's the trace for `http://127.0.0.1:4422/fixtures/compound_three_primary_keys?_trace=1&_facet=pk1&_facet=pk2` with the missing facet and facet suggestion queries:
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/issues/1518#issuecomment-999863269,https://api.github.com/repos/simonw/datasette/issues/1518,999863269,IC_kwDOBm6k_c47mLPl,9599,simonw,2021-12-22T20:35:41Z,2021-12-22T20:37:13Z,OWNER,"It looks like the count has to be executed before facets can be, because the facet_class constructor needs that total count figure: https://github.com/simonw/datasette/blob/6b1384b2f529134998fb507e63307609a5b7f5c0/datasette/views/table.py#L660-L671
It's used in facet suggestion logic here: https://github.com/simonw/datasette/blob/ace86566b28280091b3844cf5fbecd20158e9004/datasette/facets.py#L172-L178","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/issues/1518#issuecomment-999850191,https://api.github.com/repos/simonw/datasette/issues/1518,999850191,IC_kwDOBm6k_c47mIDP,9599,simonw,2021-12-22T20:29:38Z,2021-12-22T20:29:38Z,OWNER,New short-term goal: get facets and suggested facets to execute in parallel with the main query. Generate a trace graph that proves that is happening using `datasette-pretty-traces`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/issues/1518#issuecomment-999837569,https://api.github.com/repos/simonw/datasette/issues/1518,999837569,IC_kwDOBm6k_c47mE-B,9599,simonw,2021-12-22T20:15:45Z,2021-12-22T20:15:45Z,OWNER,"Also the whole `special_args` v.s. `request.args` thing is pretty confusing, I think that might be an older code pattern back from when I was using Sanic.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/issues/1518#issuecomment-999837220,https://api.github.com/repos/simonw/datasette/issues/1518,999837220,IC_kwDOBm6k_c47mE4k,9599,simonw,2021-12-22T20:15:04Z,2021-12-22T20:15:04Z,OWNER,"I think I can move this much higher up in the method, it's a bit confusing having it half way through: https://github.com/simonw/datasette/blob/6b1384b2f529134998fb507e63307609a5b7f5c0/datasette/views/table.py#L414-L436","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/issues/1518#issuecomment-999831967,https://api.github.com/repos/simonw/datasette/issues/1518,999831967,IC_kwDOBm6k_c47mDmf,9599,simonw,2021-12-22T20:04:47Z,2021-12-22T20:10:11Z,OWNER,"I think I might be able to clean up a lot of the stuff in here using the `render_cell` plugin hook: https://github.com/simonw/datasette/blob/6b1384b2f529134998fb507e63307609a5b7f5c0/datasette/views/table.py#L87-L89
The catch with that hook - https://docs.datasette.io/en/stable/plugin_hooks.html#render-cell-value-column-table-database-datasette - is that it gets called for every single cell. I don't want the overhead of looking up the foreign key relationships etc once for every value in a specific column.
But maybe I could extend the hook to include a shared cache that gets used for all of the cells in a specific table? Something like this:
```python
render_cell(value, column, table, database, datasette, cache)
```
`cache` is a dictionary - and the same dictionary is passed to every call to that hook while rendering a specific page.
It's a bit of a gross hack though, and would it ever be useful for plugins outside of the default plugin in Datasette which does the foreign key stuff?
If I can think of one other potential application for this `cache` then I might implement it.
No, this optimization doesn't make sense: the most complex cell enrichment logic is the stuff that does a `select * from categories where id in (2, 5, 6)` query, using just the distinct set of IDs that are rendered on the current page. That's not going to fit in the `render_cell` hook no matter how hard I try to warp it into the right shape, because it needs full visibility of all of the results that are being rendered in order to collect those unique ID values.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/issues/1181#issuecomment-998999230,https://api.github.com/repos/simonw/datasette/issues/1181,998999230,IC_kwDOBm6k_c47i4S-,9308268,rayvoelker,2021-12-21T18:25:15Z,2021-12-21T18:25:15Z,NONE,"I wonder if I'm encountering the same bug (or something related). I had previously been using the .csv feature to run queries and then fetch results for the pandas `read_csv()` function, but it seems to have stopped working recently.
https://ilsweb.cincinnatilibrary.org/collection-analysis/collection-analysis/current_collection-3d56dbf.csv?sql=select%0D%0A++*%0D%0Afrom%0D%0A++bib%0D%0Alimit%0D%0A++100&_size=max
Datasette v0.59.4
![image](https://user-images.githubusercontent.com/9308268/146979957-66911877-2cd9-4022-bc76-fd54e4a3a6f7.png)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",781262510,"Certain database names results in 404: ""Database not found: None""",
https://github.com/simonw/datasette/pull/1554#issuecomment-998354538,https://api.github.com/repos/simonw/datasette/issues/1554,998354538,IC_kwDOBm6k_c47ga5q,9599,simonw,2021-12-20T23:52:04Z,2021-12-20T23:52:04Z,OWNER,Abandoning this since it didn't work how I wanted.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079129258,TableView refactor,
https://github.com/simonw/datasette/issues/1547#issuecomment-997519202,https://api.github.com/repos/simonw/datasette/issues/1547,997519202,IC_kwDOBm6k_c47dO9i,127565,wragge,2021-12-20T01:36:58Z,2021-12-20T01:36:58Z,CONTRIBUTOR,"Yep, that works -- thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1076388044,Writable canned queries fail to load custom templates,
https://github.com/simonw/datasette/issues/1547#issuecomment-997514220,https://api.github.com/repos/simonw/datasette/issues/1547,997514220,IC_kwDOBm6k_c47dNvs,9599,simonw,2021-12-20T01:26:25Z,2021-12-20T01:26:25Z,OWNER,"OK, this should hopefully fix that for you:
pip install https://github.com/simonw/datasette/archive/f36e010b3b69ada104b79d83c7685caf9359049e.zip","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1076388044,Writable canned queries fail to load custom templates,
https://github.com/simonw/datasette/issues/1547#issuecomment-997513369,https://api.github.com/repos/simonw/datasette/issues/1547,997513369,IC_kwDOBm6k_c47dNiZ,9599,simonw,2021-12-20T01:24:43Z,2021-12-20T01:24:43Z,OWNER,"@wragge thanks, that's a bug! Working on that in #1575.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1076388044,Writable canned queries fail to load custom templates,
https://github.com/simonw/datasette/issues/1575#issuecomment-997513177,https://api.github.com/repos/simonw/datasette/issues/1575,997513177,IC_kwDOBm6k_c47dNfZ,9599,simonw,2021-12-20T01:24:25Z,2021-12-20T01:24:25Z,OWNER,Looks like `specname` is new in Pluggy 1.0: https://github.com/pytest-dev/pluggy/blob/main/CHANGELOG.rst#pluggy-100-2021-08-25,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1084257842,__call__() got an unexpected keyword argument 'specname',
https://github.com/simonw/datasette/issues/1547#issuecomment-997511968,https://api.github.com/repos/simonw/datasette/issues/1547,997511968,IC_kwDOBm6k_c47dNMg,127565,wragge,2021-12-20T01:21:59Z,2021-12-20T01:21:59Z,CONTRIBUTOR,"I've installed the alpha version but get an error when starting up Datasette:
```
Traceback (most recent call last):
File ""/Users/tim/.pyenv/versions/stock-exchange/bin/datasette"", line 5, in
from datasette.cli import cli
File ""/Users/tim/.pyenv/versions/3.8.5/envs/stock-exchange/lib/python3.8/site-packages/datasette/cli.py"", line 15, in
from .app import Datasette, DEFAULT_SETTINGS, SETTINGS, SQLITE_LIMIT_ATTACHED, pm
File ""/Users/tim/.pyenv/versions/3.8.5/envs/stock-exchange/lib/python3.8/site-packages/datasette/app.py"", line 31, in
from .views.database import DatabaseDownload, DatabaseView
File ""/Users/tim/.pyenv/versions/3.8.5/envs/stock-exchange/lib/python3.8/site-packages/datasette/views/database.py"", line 25, in
from datasette.plugins import pm
File ""/Users/tim/.pyenv/versions/3.8.5/envs/stock-exchange/lib/python3.8/site-packages/datasette/plugins.py"", line 29, in
mod = importlib.import_module(plugin)
File ""/Users/tim/.pyenv/versions/3.8.5/lib/python3.8/importlib/__init__.py"", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File ""/Users/tim/.pyenv/versions/3.8.5/envs/stock-exchange/lib/python3.8/site-packages/datasette/filters.py"", line 9, in
@hookimpl(specname=""filters_from_request"")
TypeError: __call__() got an unexpected keyword argument 'specname'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1076388044,Writable canned queries fail to load custom templates,
https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997507074,https://api.github.com/repos/simonw/sqlite-utils/issues/356,997507074,IC_kwDOCGYnMM47dMAC,9599,simonw,2021-12-20T01:10:06Z,2021-12-20T01:16:11Z,OWNER,"Work-in-progress improved help:
```
Usage: sqlite-utils insert [OPTIONS] PATH TABLE FILE
Insert records from FILE into a table, creating the table if it does not
already exist.
By default the input is expected to be a JSON array of objects. Or:
- Use --nl for newline-delimited JSON objects
- Use --csv or --tsv for comma-separated or tab-separated input
- Use --lines to write each incoming line to a column called ""line""
- Use --all to write the entire input to a column called ""all""
You can also use --convert to pass a fragment of Python code that will be
used to convert each input.
Your Python code will be passed a ""row"" variable representing the imported
row, and can return a modified row.
If you are using --lines your code will be passed a ""line"" variable, and for
--all an ""all"" variable.
Options:
--pk TEXT Columns to use as the primary key, e.g. id
--flatten Flatten nested JSON objects, so {""a"": {""b"": 1}}
becomes {""a_b"": 1}
--nl Expect newline-delimited JSON
-c, --csv Expect CSV input
--tsv Expect TSV input
--lines Treat each line as a single value called 'line'
--all Treat input as a single value called 'all'
--convert TEXT Python code to convert each item
--import TEXT Python modules to import
--delimiter TEXT Delimiter to use for CSV files
--quotechar TEXT Quote character to use for CSV/TSV
--sniff Detect delimiter and quote character
--no-headers CSV file has no header row
--batch-size INTEGER Commit every X records
--alter Alter existing table to add any missing columns
--not-null TEXT Columns that should be created as NOT NULL
--default ... Default value that should be set for a column
--encoding TEXT Character encoding for input, defaults to utf-8
-d, --detect-types Detect types for columns in CSV/TSV data
--load-extension TEXT SQLite extensions to load
--silent Do not show progress bar
--ignore Ignore records if pk already exists
--replace Replace records if pk already exists
--truncate Truncate table before inserting records, if table
already exists
-h, --help Show this message and exit.
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option,
https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997508728,https://api.github.com/repos/simonw/sqlite-utils/issues/356,997508728,IC_kwDOCGYnMM47dMZ4,9599,simonw,2021-12-20T01:14:43Z,2021-12-20T01:14:43Z,OWNER,(This makes me want `--extract` from #352 even more.),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option,
https://github.com/simonw/sqlite-utils/issues/163#issuecomment-997502242,https://api.github.com/repos/simonw/sqlite-utils/issues/163,997502242,IC_kwDOCGYnMM47dK0i,9599,simonw,2021-12-20T00:56:45Z,2021-12-20T00:56:52Z,OWNER,"> Maybe `sqlite-utils` should absorb all of the functionality from `sqlite-transform` - having two separate tools doesn't necessarily make sense.
I implemented that in:
- #251","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706001517,Idea: conversions= could take Python functions,
https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997497262,https://api.github.com/repos/simonw/sqlite-utils/issues/356,997497262,IC_kwDOCGYnMM47dJmu,9599,simonw,2021-12-20T00:40:15Z,2021-12-20T00:40:15Z,OWNER,`--flatten` could do with a better description too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option,
https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997496931,https://api.github.com/repos/simonw/sqlite-utils/issues/356,997496931,IC_kwDOCGYnMM47dJhj,9599,simonw,2021-12-20T00:39:14Z,2021-12-20T00:39:52Z,OWNER,"```
% sqlite-utils insert --help
Usage: sqlite-utils insert [OPTIONS] PATH TABLE JSON_FILE
Insert records from JSON file into a table, creating the table if it does
not already exist.
Input should be a JSON array of objects, unless --nl or --csv is used.
Options:
--pk TEXT Columns to use as the primary key, e.g. id
--nl Expect newline-delimited JSON
--flatten Flatten nested JSON objects
-c, --csv Expect CSV
--tsv Expect TSV
--convert TEXT Python code to convert each item
--import TEXT Python modules to import
--delimiter TEXT Delimiter to use for CSV files
--quotechar TEXT Quote character to use for CSV/TSV
--sniff Detect delimiter and quote character
--no-headers CSV file has no header row
--batch-size INTEGER Commit every X records
--alter Alter existing table to add any missing columns
--not-null TEXT Columns that should be created as NOT NULL
--default ... Default value that should be set for a column
--encoding TEXT Character encoding for input, defaults to utf-8
-d, --detect-types Detect types for columns in CSV/TSV data
--load-extension TEXT SQLite extensions to load
--silent Do not show progress bar
--ignore Ignore records if pk already exists
--replace Replace records if pk already exists
--truncate Truncate table before inserting records, if table
already exists
-h, --help Show this message and exit.
```
I can add a bunch of extra help at the top there to explain all of this stuff. That ""Input should be a JSON array of objects"" bit could be expanded to several paragraphs.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option,
https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997492872,https://api.github.com/repos/simonw/sqlite-utils/issues/356,997492872,IC_kwDOCGYnMM47dIiI,9599,simonw,2021-12-20T00:23:31Z,2021-12-20T00:23:31Z,OWNER,"I think this should work on JSON, or CSV, or individual lines, or the entire content at once.
So I'll require `--lines --convert ...` to import individual lines, or `--all --convert` to run the conversion against the entire input at once.
What would `--lines` or `--all` do without `--convert`? Maybe insert records as `{""line"": ""line of text""}` or `{""all"": ""whole input}`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option,
https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997486156,https://api.github.com/repos/simonw/sqlite-utils/issues/356,997486156,IC_kwDOCGYnMM47dG5M,9599,simonw,2021-12-19T23:51:02Z,2021-12-19T23:51:02Z,OWNER,This is going to need a `--import` multi option too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option,
https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997485361,https://api.github.com/repos/simonw/sqlite-utils/issues/356,997485361,IC_kwDOCGYnMM47dGsx,9599,simonw,2021-12-19T23:45:30Z,2021-12-19T23:45:30Z,OWNER,Really interesting example input for this: https://blog.timac.org/2021/1219-state-of-swift-and-swiftui-ios15/iOS13.txt - see https://blog.timac.org/2021/1219-state-of-swift-and-swiftui-ios15/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option,
https://github.com/simonw/datasette/issues/1565#issuecomment-997474022,https://api.github.com/repos/simonw/datasette/issues/1565,997474022,IC_kwDOBm6k_c47dD7m,9599,simonw,2021-12-19T22:36:49Z,2021-12-19T22:37:29Z,OWNER,"No way with a tagged template literal to pass an extra database name argument, so instead I need a method that returns a callable that can be used for the tagged template literal for a specific database - or the default database.
This could work (bit weird looking though):
```javascript
var rows = await datasette.query(""fixtures"")`select * from foo`;
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins,
https://github.com/simonw/datasette/issues/1565#issuecomment-997473856,https://api.github.com/repos/simonw/datasette/issues/1565,997473856,IC_kwDOBm6k_c47dD5A,9599,simonw,2021-12-19T22:35:20Z,2021-12-19T22:35:20Z,OWNER,"Quick prototype of that tagged template `query` function:
```javascript
function query(pieces, ...parameters) {
var qs = new URLSearchParams();
var sql = pieces[0];
parameters.forEach((param, i) => {
sql += `:p${i}${pieces[i + 1]}`;
qs.append(`p${i}`, param);
});
qs.append(""sql"", sql);
return qs.toString();
}
var id = 4;
console.log(query`select * from ids where id > ${id}`);
```
Outputs:
```
p0=4&sql=select+*+from+ids+where+id+%3E+%3Ap0
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins,
https://github.com/simonw/datasette/issues/1565#issuecomment-997472639,https://api.github.com/repos/simonw/datasette/issues/1565,997472639,IC_kwDOBm6k_c47dDl_,9599,simonw,2021-12-19T22:25:50Z,2021-12-19T22:25:50Z,OWNER,"Or...
```javascript
rows = await datasette.query`select * from searchable where id > ${id}`;
```
And it knows how to turn that into a parameterized call using tagged template literals.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins,
https://github.com/simonw/datasette/issues/1565#issuecomment-997472509,https://api.github.com/repos/simonw/datasette/issues/1565,997472509,IC_kwDOBm6k_c47dDj9,9599,simonw,2021-12-19T22:24:50Z,2021-12-19T22:24:50Z,OWNER,"... huh, it could even expose a JavaScript function that can be called to execute a SQL query.
```javascript
datasette.query(""select * from blah"").then(...)
```
Maybe it takes an optional second argument that specifies the database - defaulting to the one for the current page.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins,
https://github.com/simonw/datasette/issues/1565#issuecomment-997472370,https://api.github.com/repos/simonw/datasette/issues/1565,997472370,IC_kwDOBm6k_c47dDhy,9599,simonw,2021-12-19T22:23:36Z,2021-12-19T22:23:36Z,OWNER,This should also expose the JSON API endpoints used to execute SQL against this database.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins,
https://github.com/simonw/datasette/issues/1518#issuecomment-997472214,https://api.github.com/repos/simonw/datasette/issues/1518,997472214,IC_kwDOBm6k_c47dDfW,9599,simonw,2021-12-19T22:22:08Z,2021-12-19T22:22:08Z,OWNER,"I sketched out a chained SQL builder pattern that might be useful for further tidying up this code - though with the new plugin hook I'm less excited about it than I was:
```python
class TableQuery:
def __init__(self, table, columns, pks, is_view=False, prev=None):
self.table = table
self.columns = columns
self.pks = pks
self.is_view = is_view
self.prev = prev
# These can be changed for different instances in the chain:
self._where_clauses = None
self._order_by = None
self._page_size = None
self._offset = None
self._select_columns = None
self.select_all_columns = '*'
self.select_specified_columns = '*'
@property
def where_clauses(self):
wheres = []
current = self
while current:
if current._where_clauses is not None:
wheres.extend(current._where_clauses)
current = current.prev
return list(reversed(wheres))
def where(self, where):
new_cls = TableQuery(self.table, self.columns, self.pks, self.is_view, self)
new_cls._where_clauses = [where]
return new_cls
@classmethod
async def introspect(cls, db, table):
return cls(
table,
columns = await db.table_columns(table),
pks = await db.primary_keys(table),
is_view = bool(await db.get_view_definition(table))
)
@property
def sql_from(self):
return f""from {self.table}{self.sql_where}""
@property
def sql_where(self):
if not self.where_clauses:
return """"
else:
return f"" where {' and '.join(self.where_clauses)}""
@property
def sql_no_order_no_limit(self):
return f""select {self.select_all_columns} from {self.table}{self.sql_where}""
@property
def sql(self):
return f""select {self.select_specified_columns} from {self.table} {self.sql_where}{self._order_by} limit {self._page_size}{self._offset}""
@property
def sql_count(self):
return f""select count(*) {self.sql_from}""
def __repr__(self):
return f""""
```
Usage:
```python
from datasette.app import Datasette
ds = Datasette(memory=True, files=[""/Users/simon/Dropbox/Development/datasette/fixtures.db""])
db = ds.get_database(""fixtures"")
query = await TableQuery.introspect(db, ""facetable"")
print(query.where(""foo = bar"").where(""baz = 1"").sql_count)
# 'select count(*) from facetable where foo = bar and baz = 1'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/issues/1547#issuecomment-997471672,https://api.github.com/repos/simonw/datasette/issues/1547,997471672,IC_kwDOBm6k_c47dDW4,9599,simonw,2021-12-19T22:18:26Z,2021-12-19T22:18:26Z,OWNER,"I released this [in an alpha](https://github.com/simonw/datasette/releases/tag/0.60a1), so you can try out this fix using:
pip install datasette==0.60a1","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1076388044,Writable canned queries fail to load custom templates,
https://github.com/simonw/datasette/issues/1566#issuecomment-997470633,https://api.github.com/repos/simonw/datasette/issues/1566,997470633,IC_kwDOBm6k_c47dDGp,9599,simonw,2021-12-19T22:12:00Z,2021-12-19T22:12:00Z,OWNER,"Released another alpha, 0.60a1: https://github.com/simonw/datasette/releases/tag/0.60a1","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083669410,Release Datasette 0.60,
https://github.com/simonw/datasette/issues/1545#issuecomment-997462604,https://api.github.com/repos/simonw/datasette/issues/1545,997462604,IC_kwDOBm6k_c47dBJM,9599,simonw,2021-12-19T21:17:08Z,2021-12-19T21:17:08Z,OWNER,"Here's the relevant code: https://github.com/simonw/datasette/blob/4094741c2881c2ada3f3f878b532fdaec7914953/datasette/app.py#L1204-L1219
It's using `route_path.split(""/"")` which should be OK because that's the incoming `request.path` path - which I would expect to use `/` even on Windows. Then it uses `os.path.join` which should do the right thing.
I need to get myself a proper Windows development environment setup to investigate this one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1075893249,Custom pages don't work on windows,
https://github.com/simonw/datasette/issues/1573#issuecomment-997462117,https://api.github.com/repos/simonw/datasette/issues/1573,997462117,IC_kwDOBm6k_c47dBBl,9599,simonw,2021-12-19T21:13:13Z,2021-12-19T21:13:13Z,OWNER,This might also be the impetus I need to bring the https://datasette.io/plugins/datasette-pretty-traces plugin into Datasette core itself.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1084185188,Make trace() a documented internal API,
https://github.com/simonw/datasette/issues/1547#issuecomment-997460731,https://api.github.com/repos/simonw/datasette/issues/1547,997460731,IC_kwDOBm6k_c47dAr7,9599,simonw,2021-12-19T21:02:15Z,2021-12-19T21:02:15Z,OWNER,"Yes, this is a bug. It looks like the problem is with the `if write:` branch in this code here: https://github.com/simonw/datasette/blob/5fac26aa221a111d7633f2dd92014641f7c0ade9/datasette/views/database.py#L252-L327
Is missing this bit of code:
https://github.com/simonw/datasette/blob/5fac26aa221a111d7633f2dd92014641f7c0ade9/datasette/views/database.py#L343-L347","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1076388044,Writable canned queries fail to load custom templates,
https://github.com/simonw/datasette/issues/1570#issuecomment-997460061,https://api.github.com/repos/simonw/datasette/issues/1570,997460061,IC_kwDOBm6k_c47dAhd,9599,simonw,2021-12-19T20:56:54Z,2021-12-19T20:56:54Z,OWNER,Documentation: https://docs.datasette.io/en/latest/internals.html#await-db-execute-write-sql-params-none-block-false,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083921371,Separate db.execute_write() into three methods,
https://github.com/simonw/datasette/issues/1555#issuecomment-997459958,https://api.github.com/repos/simonw/datasette/issues/1555,997459958,IC_kwDOBm6k_c47dAf2,9599,simonw,2021-12-19T20:55:59Z,2021-12-19T20:55:59Z,OWNER,"Closing this issue because I've optimized this a whole bunch, and it's definitely good enough for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list,
https://github.com/simonw/datasette/issues/1555#issuecomment-997325189,https://api.github.com/repos/simonw/datasette/issues/1555,997325189,IC_kwDOBm6k_c47cfmF,9599,simonw,2021-12-19T03:55:01Z,2021-12-19T20:54:51Z,OWNER,"It's a bit annoying that the queries no longer show up in the trace at all now, thanks to running in `.execute_fn()`. I wonder if there's something smart I can do about that - maybe have `trace()` record that function with a traceback even though it doesn't have the executed SQL string?
5fac26aa221a111d7633f2dd92014641f7c0ade9 has the same problem.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list,
https://github.com/simonw/datasette/issues/1555#issuecomment-997459637,https://api.github.com/repos/simonw/datasette/issues/1555,997459637,IC_kwDOBm6k_c47dAa1,9599,simonw,2021-12-19T20:53:46Z,2021-12-19T20:53:46Z,OWNER,Using #1571 showed me that the `DELETE FROM columns/foreign_keys/indexes WHERE database_name = ? and table_name = ?` queries were running way more times than I expected. I came up with a new optimization that just does `DELETE FROM columns/foreign_keys/indexes WHERE database_name = ?` instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list,
https://github.com/simonw/datasette/issues/1566#issuecomment-997457790,https://api.github.com/repos/simonw/datasette/issues/1566,997457790,IC_kwDOBm6k_c47c_9-,9599,simonw,2021-12-19T20:40:50Z,2021-12-19T20:40:57Z,OWNER,"Also release new version of `datasette-pretty-traces` with this feature:
- https://github.com/simonw/datasette-pretty-traces/issues/7","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083669410,Release Datasette 0.60,
https://github.com/simonw/datasette/issues/1555#issuecomment-997342494,https://api.github.com/repos/simonw/datasette/issues/1555,997342494,IC_kwDOBm6k_c47cj0e,9599,simonw,2021-12-19T07:22:04Z,2021-12-19T07:22:04Z,OWNER,"Another option would be to provide an abstraction that makes it easier to run a group of SQL queries in the same thread at the same time, and have them traced correctly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list,
https://github.com/simonw/datasette/issues/1555#issuecomment-997324666,https://api.github.com/repos/simonw/datasette/issues/1555,997324666,IC_kwDOBm6k_c47cfd6,9599,simonw,2021-12-19T03:47:51Z,2021-12-19T03:48:09Z,OWNER,"Here's a hacked together prototype of running all of that stuff inside a single function passed to `.execute_fn()`:
```diff
diff --git a/datasette/utils/internal_db.py b/datasette/utils/internal_db.py
index 95055d8..58f9982 100644
--- a/datasette/utils/internal_db.py
+++ b/datasette/utils/internal_db.py
@@ -1,4 +1,5 @@
import textwrap
+from datasette.utils import table_column_details
async def init_internal_db(db):
@@ -70,49 +71,70 @@ async def populate_schema_tables(internal_db, db):
""DELETE FROM tables WHERE database_name = ?"", [database_name], block=True
)
tables = (await db.execute(""select * from sqlite_master WHERE type = 'table'"")).rows
- tables_to_insert = []
- columns_to_delete = []
- columns_to_insert = []
- foreign_keys_to_delete = []
- foreign_keys_to_insert = []
- indexes_to_delete = []
- indexes_to_insert = []
- for table in tables:
- table_name = table[""name""]
- tables_to_insert.append(
- (database_name, table_name, table[""rootpage""], table[""sql""])
- )
- columns_to_delete.append((database_name, table_name))
- columns = await db.table_column_details(table_name)
- columns_to_insert.extend(
- {
- **{""database_name"": database_name, ""table_name"": table_name},
- **column._asdict(),
- }
- for column in columns
- )
- foreign_keys_to_delete.append((database_name, table_name))
- foreign_keys = (
- await db.execute(f""PRAGMA foreign_key_list([{table_name}])"")
- ).rows
- foreign_keys_to_insert.extend(
- {
- **{""database_name"": database_name, ""table_name"": table_name},
- **dict(foreign_key),
- }
- for foreign_key in foreign_keys
- )
- indexes_to_delete.append((database_name, table_name))
- indexes = (await db.execute(f""PRAGMA index_list([{table_name}])"")).rows
- indexes_to_insert.extend(
- {
- **{""database_name"": database_name, ""table_name"": table_name},
- **dict(index),
- }
- for index in indexes
+ def collect_info(conn):
+ tables_to_insert = []
+ columns_to_delete = []
+ columns_to_insert = []
+ foreign_keys_to_delete = []
+ foreign_keys_to_insert = []
+ indexes_to_delete = []
+ indexes_to_insert = []
+
+ for table in tables:
+ table_name = table[""name""]
+ tables_to_insert.append(
+ (database_name, table_name, table[""rootpage""], table[""sql""])
+ )
+ columns_to_delete.append((database_name, table_name))
+ columns = table_column_details(conn, table_name)
+ columns_to_insert.extend(
+ {
+ **{""database_name"": database_name, ""table_name"": table_name},
+ **column._asdict(),
+ }
+ for column in columns
+ )
+ foreign_keys_to_delete.append((database_name, table_name))
+ foreign_keys = conn.execute(
+ f""PRAGMA foreign_key_list([{table_name}])""
+ ).fetchall()
+ foreign_keys_to_insert.extend(
+ {
+ **{""database_name"": database_name, ""table_name"": table_name},
+ **dict(foreign_key),
+ }
+ for foreign_key in foreign_keys
+ )
+ indexes_to_delete.append((database_name, table_name))
+ indexes = conn.execute(f""PRAGMA index_list([{table_name}])"").fetchall()
+ indexes_to_insert.extend(
+ {
+ **{""database_name"": database_name, ""table_name"": table_name},
+ **dict(index),
+ }
+ for index in indexes
+ )
+ return (
+ tables_to_insert,
+ columns_to_delete,
+ columns_to_insert,
+ foreign_keys_to_delete,
+ foreign_keys_to_insert,
+ indexes_to_delete,
+ indexes_to_insert,
)
+ (
+ tables_to_insert,
+ columns_to_delete,
+ columns_to_insert,
+ foreign_keys_to_delete,
+ foreign_keys_to_insert,
+ indexes_to_delete,
+ indexes_to_insert,
+ ) = await db.execute_fn(collect_info)
+
await internal_db.execute_write_many(
""""""
INSERT INTO tables (database_name, table_name, rootpage, sql)
```
First impressions: it looks like this helps **a lot** - as far as I can tell this is now taking around 21ms to get to the point at which all of those internal databases have been populated, where previously it took more than 180ms.
![CleanShot 2021-12-18 at 19 47 22@2x](https://user-images.githubusercontent.com/9599/146663192-bba098d5-e7bd-4e2e-b525-2270867888a0.png)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list,
https://github.com/simonw/datasette/issues/1555#issuecomment-997324156,https://api.github.com/repos/simonw/datasette/issues/1555,997324156,IC_kwDOBm6k_c47cfV8,9599,simonw,2021-12-19T03:40:05Z,2021-12-19T03:40:05Z,OWNER,"Using the prototype of this:
- https://github.com/simonw/datasette-pretty-traces/issues/5
I'm seeing about 180ms spent running all of these queries on startup!
![CleanShot 2021-12-18 at 19 38 37@2x](https://user-images.githubusercontent.com/9599/146663045-46bda669-90de-474f-8870-345182725dc1.png)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list,
https://github.com/simonw/datasette/issues/1555#issuecomment-997321767,https://api.github.com/repos/simonw/datasette/issues/1555,997321767,IC_kwDOBm6k_c47cewn,9599,simonw,2021-12-19T03:10:58Z,2021-12-19T03:10:58Z,OWNER,"I wonder how much overhead there is switching between the `async` event loop main code and the thread that runs the SQL queries.
Would there be a performance boost if I gathered all of the column/index information in a single function run on the thread using `db.execute_fn()` I wonder? It would eliminate a bunch of switching between threads.
Would be great to understand how much of an impact that would have.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list,
https://github.com/simonw/datasette/issues/1555#issuecomment-997321653,https://api.github.com/repos/simonw/datasette/issues/1555,997321653,IC_kwDOBm6k_c47ceu1,9599,simonw,2021-12-19T03:09:43Z,2021-12-19T03:09:43Z,OWNER,"On that same documentation page I just spotted this:
> This feature is experimental and is subject to change. Further documentation will become available if and when the table-valued functions for PRAGMAs feature becomes officially supported.
This makes me nervous to rely on pragma function optimizations in Datasette itself.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list,
https://github.com/simonw/datasette/issues/1555#issuecomment-997321477,https://api.github.com/repos/simonw/datasette/issues/1555,997321477,IC_kwDOBm6k_c47cesF,9599,simonw,2021-12-19T03:07:33Z,2021-12-19T03:07:33Z,OWNER,"If I want to continue supporting SQLite prior to 3.16.0 (2017-01-02) I'll need this optimization to only kick in with versions that support table-valued PRAGMA functions, while keeping the old `PRAGMA foreign_key_list(table)` stuff working for those older versions.
That's feasible, but it's a bit more work - and I need to make sure I have robust testing in place for SQLite 3.15.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list,
https://github.com/simonw/datasette/issues/1555#issuecomment-997321327,https://api.github.com/repos/simonw/datasette/issues/1555,997321327,IC_kwDOBm6k_c47cepv,9599,simonw,2021-12-19T03:05:39Z,2021-12-19T03:05:44Z,OWNER,"This caught me out once before in:
- https://github.com/simonw/datasette/issues/1276
Turns out Glitch was running SQLite 3.11.0 from 2016-02-15.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list,
https://github.com/simonw/datasette/issues/1555#issuecomment-997321217,https://api.github.com/repos/simonw/datasette/issues/1555,997321217,IC_kwDOBm6k_c47ceoB,9599,simonw,2021-12-19T03:04:16Z,2021-12-19T03:04:16Z,OWNER,"One thing to watch out for though, from https://sqlite.org/pragma.html#pragfunc
> The table-valued functions for PRAGMA feature was added in SQLite version 3.16.0 (2017-01-02). Prior versions of SQLite cannot use this feature. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list,
https://github.com/simonw/datasette/issues/1555#issuecomment-997321115,https://api.github.com/repos/simonw/datasette/issues/1555,997321115,IC_kwDOBm6k_c47cemb,9599,simonw,2021-12-19T03:03:12Z,2021-12-19T03:03:12Z,OWNER,"Table columns is a bit harder, because `table_xinfo` is only in SQLite 3.26.0 or higher: https://github.com/simonw/datasette/blob/d637ed46762fdbbd8e32b86f258cd9a53c1cfdc7/datasette/utils/__init__.py#L565-L581
So if that function is available: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++table_xinfo.*%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_table_xinfo%28sqlite_master.name%29+AS+table_xinfo%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27
```sql
SELECT
sqlite_master.name,
table_xinfo.*
FROM
sqlite_master,
pragma_table_xinfo(sqlite_master.name) AS table_xinfo
WHERE
sqlite_master.type = 'table'
```
And otherwise, using `table_info`: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++table_info.*%2C%0D%0A++0+as+hidden%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_table_info%28sqlite_master.name%29+AS+table_info%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27
```sql
SELECT
sqlite_master.name,
table_info.*,
0 as hidden
FROM
sqlite_master,
pragma_table_info(sqlite_master.name) AS table_info
WHERE
sqlite_master.type = 'table'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list,
https://github.com/simonw/datasette/issues/1555#issuecomment-997320824,https://api.github.com/repos/simonw/datasette/issues/1555,997320824,IC_kwDOBm6k_c47ceh4,9599,simonw,2021-12-19T02:59:57Z,2021-12-19T03:00:44Z,OWNER,"To list all indexes: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++index_list.*%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_index_list%28sqlite_master.name%29+AS+index_list%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27
```sql
SELECT
sqlite_master.name,
index_list.*
FROM
sqlite_master,
pragma_index_list(sqlite_master.name) AS index_list
WHERE
sqlite_master.type = 'table'
```
Foreign keys: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++foreign_key_list.*%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_foreign_key_list%28sqlite_master.name%29+AS+foreign_key_list%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27
```sql
SELECT
sqlite_master.name,
foreign_key_list.*
FROM
sqlite_master,
pragma_foreign_key_list(sqlite_master.name) AS foreign_key_list
WHERE
sqlite_master.type = 'table'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list,
https://github.com/simonw/datasette/issues/1566#issuecomment-997272328,https://api.github.com/repos/simonw/datasette/issues/1566,997272328,IC_kwDOBm6k_c47cSsI,9599,simonw,2021-12-18T19:18:01Z,2021-12-18T19:18:01Z,OWNER,"Added some useful new documented internal methods in:
- #1570","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083669410,Release Datasette 0.60,
https://github.com/simonw/datasette/issues/1555#issuecomment-997272223,https://api.github.com/repos/simonw/datasette/issues/1555,997272223,IC_kwDOBm6k_c47cSqf,9599,simonw,2021-12-18T19:17:13Z,2021-12-18T19:17:13Z,OWNER,That's a good optimization. Still need to deal with the huge flurry of `PRAGMA` queries though before I can consider this done.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list,
https://github.com/simonw/datasette/issues/1570#issuecomment-997267583,https://api.github.com/repos/simonw/datasette/issues/1570,997267583,IC_kwDOBm6k_c47cRh_,9599,simonw,2021-12-18T18:46:05Z,2021-12-18T18:46:12Z,OWNER,This will replace the work done in #1569.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083921371,Separate db.execute_write() into three methods,
https://github.com/simonw/datasette/issues/1555#issuecomment-997267416,https://api.github.com/repos/simonw/datasette/issues/1555,997267416,IC_kwDOBm6k_c47cRfY,9599,simonw,2021-12-18T18:44:53Z,2021-12-18T18:45:28Z,OWNER,"Rather than adding a `executemany=True` parameter, I'm now thinking a better design might be to have three methods:
- `db.execute_write(sql, params=None, block=False)`
- `db.execute_writescript(sql, block=False)`
- `db.execute_writemany(sql, params_seq, block=False)`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list,
https://github.com/simonw/datasette/issues/1569#issuecomment-997266687,https://api.github.com/repos/simonw/datasette/issues/1569,997266687,IC_kwDOBm6k_c47cRT_,9599,simonw,2021-12-18T18:41:40Z,2021-12-18T18:41:40Z,OWNER,Updated documentation: https://docs.datasette.io/en/latest/internals.html#await-db-execute-write-sql-params-none-executescript-false-block-false,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083895395,"db.execute_write(..., executescript=True) parameter",
https://github.com/simonw/datasette/issues/1555#issuecomment-997266100,https://api.github.com/repos/simonw/datasette/issues/1555,997266100,IC_kwDOBm6k_c47cRK0,9599,simonw,2021-12-18T18:40:02Z,2021-12-18T18:40:02Z,OWNER,The implementation of `cursor.executemany()` looks very efficient - it turns into a call to this C function with `multiple` set to `1`: https://github.com/python/cpython/blob/e002bbc6cce637171fb2b1391ffeca8643a13843/Modules/_sqlite/cursor.c#L468-L469,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list,
https://github.com/simonw/datasette/issues/1555#issuecomment-997262475,https://api.github.com/repos/simonw/datasette/issues/1555,997262475,IC_kwDOBm6k_c47cQSL,9599,simonw,2021-12-18T18:34:18Z,2021-12-18T18:34:18Z,OWNER,"
Using `executescript=True` that call now takes 1.89ms to create all of those tables.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list,
https://github.com/simonw/datasette/issues/1569#issuecomment-997249563,https://api.github.com/repos/simonw/datasette/issues/1569,997249563,IC_kwDOBm6k_c47cNIb,9599,simonw,2021-12-18T18:21:23Z,2021-12-18T18:21:23Z,OWNER,"Goal here is to gain the ability to use `conn.executescript()` and still have it show up in the tracer.
https://docs.python.org/3/library/sqlite3.html#sqlite3.Cursor.executescript","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083895395,"db.execute_write(..., executescript=True) parameter",
https://github.com/simonw/datasette/issues/1555#issuecomment-997248364,https://api.github.com/repos/simonw/datasette/issues/1555,997248364,IC_kwDOBm6k_c47cM1s,9599,simonw,2021-12-18T18:20:10Z,2021-12-18T18:20:10Z,OWNER,"Idea: teach `execute_write` to accept an optional `executescript=True` parameter, like this:
```diff
diff --git a/datasette/database.py b/datasette/database.py
index 468e936..1a424f5 100644
--- a/datasette/database.py
+++ b/datasette/database.py
@@ -94,10 +94,14 @@ class Database:
f""file:{self.path}{qs}"", uri=True, check_same_thread=False
)
- async def execute_write(self, sql, params=None, block=False):
+ async def execute_write(self, sql, params=None, executescript=False, block=False):
+ assert not executescript and params, ""Cannot use params with executescript=True""
def _inner(conn):
with conn:
- return conn.execute(sql, params or [])
+ if executescript:
+ return conn.executescript(sql)
+ else:
+ return conn.execute(sql, params or [])
with trace(""sql"", database=self.name, sql=sql.strip(), params=params):
results = await self.execute_write_fn(_inner, block=block)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list,
https://github.com/simonw/datasette/issues/1555#issuecomment-997245301,https://api.github.com/repos/simonw/datasette/issues/1555,997245301,IC_kwDOBm6k_c47cMF1,9599,simonw,2021-12-18T18:17:04Z,2021-12-18T18:17:04Z,OWNER,"One downside of `conn.executescript()` is that it won't be picked up by the tracing mechanism - in fact nothing that uses `await db.execute_write_fn(fn, block=True)` or `await db.execute_fn(fn, block=True)` gets picked up by tracing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list,
https://github.com/simonw/datasette/issues/1555#issuecomment-997241969,https://api.github.com/repos/simonw/datasette/issues/1555,997241969,IC_kwDOBm6k_c47cLRx,9599,simonw,2021-12-18T18:13:04Z,2021-12-18T18:13:04Z,OWNER,Also: running all of those `CREATE TABLE IF NOT EXISTS` in a single call to `conn.executescript()` rather than as separate queries may speed things up too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list,
https://github.com/simonw/datasette/issues/1555#issuecomment-997241645,https://api.github.com/repos/simonw/datasette/issues/1555,997241645,IC_kwDOBm6k_c47cLMt,9599,simonw,2021-12-18T18:12:26Z,2021-12-18T18:12:26Z,OWNER,"A simpler optimization would be just to turn all of those column and index reads into a single efficient UNION query against each database, then figure out the most efficient pattern to send them all as writes in one go as opposed to calling `.execute_write()` in a loop.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list,
https://github.com/simonw/datasette/issues/1566#issuecomment-997235388,https://api.github.com/repos/simonw/datasette/issues/1566,997235388,IC_kwDOBm6k_c47cJq8,9599,simonw,2021-12-18T17:32:07Z,2021-12-18T17:32:07Z,OWNER,I can release a new version of `datasette-leaflet-freedraw` as soon as this is out.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083669410,Release Datasette 0.60,
https://github.com/simonw/datasette/issues/1555#issuecomment-997235086,https://api.github.com/repos/simonw/datasette/issues/1555,997235086,IC_kwDOBm6k_c47cJmO,9599,simonw,2021-12-18T17:30:13Z,2021-12-18T17:30:13Z,OWNER,"Now that trace sees write queries (#1568) it's clear that there is a whole lot more DB activity then I had realized:
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list,
https://github.com/simonw/datasette/issues/1555#issuecomment-997234858,https://api.github.com/repos/simonw/datasette/issues/1555,997234858,IC_kwDOBm6k_c47cJiq,9599,simonw,2021-12-18T17:28:44Z,2021-12-18T17:28:44Z,OWNER,"Maybe it would be worth exploring attaching each DB in turn to the _internal connection in order to perform these queries faster.
I'm a bit worried about leaks though: the internal database isn't meant to be visible, even temporarily attaching another DB to it could cause SQL queries against that DB to be able to access the internal data.
So maybe instead the _internal connection gets to connect to the other DBs? There's a maximum of ten there I think, which is good for most but not all cases. But the cases with the most connected databases will see the worst performance!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list,
https://github.com/simonw/datasette/issues/1568#issuecomment-997153253,https://api.github.com/repos/simonw/datasette/issues/1568,997153253,IC_kwDOBm6k_c47b1nl,9599,simonw,2021-12-18T06:20:23Z,2021-12-18T06:20:23Z,OWNER,Now running at https://latest-with-plugins.datasette.io/github/commits?_trace=1,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083726550,Trace should show queries on the write connection too,
https://github.com/simonw/datasette/issues/1568#issuecomment-997128950,https://api.github.com/repos/simonw/datasette/issues/1568,997128950,IC_kwDOBm6k_c47bvr2,9599,simonw,2021-12-18T02:38:01Z,2021-12-18T02:38:01Z,OWNER,"Prototype:
```diff
diff --git a/datasette/database.py b/datasette/database.py
index 0a0c104..468e936 100644
--- a/datasette/database.py
+++ b/datasette/database.py
@@ -99,7 +99,9 @@ class Database:
with conn:
return conn.execute(sql, params or [])
- return await self.execute_write_fn(_inner, block=block)
+ with trace(""sql"", database=self.name, sql=sql.strip(), params=params):
+ results = await self.execute_write_fn(_inner, block=block)
+ return results
async def execute_write_fn(self, fn, block=False):
task_id = uuid.uuid5(uuid.NAMESPACE_DNS, ""datasette.io"")
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083726550,Trace should show queries on the write connection too,
https://github.com/simonw/datasette/issues/1561#issuecomment-997128712,https://api.github.com/repos/simonw/datasette/issues/1561,997128712,IC_kwDOBm6k_c47bvoI,536941,fgregg,2021-12-18T02:35:48Z,2021-12-18T02:35:48Z,CONTRIBUTOR,interesting! i love this feature. this + full caching with cloudflare is really super!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082765654,"add hash id to ""_memory"" url if hashed url mode is turned on and crossdb is also turned on",
https://github.com/simonw/datasette/issues/1555#issuecomment-997128508,https://api.github.com/repos/simonw/datasette/issues/1555,997128508,IC_kwDOBm6k_c47bvk8,9599,simonw,2021-12-18T02:33:57Z,2021-12-18T02:33:57Z,OWNER,"Here's why - `trace` only applies to read, not write SQL operations: https://github.com/simonw/datasette/blob/7c8f8aa209e4ba7bf83976f8495d67c28fbfca24/datasette/database.py#L209-L211","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list,
https://github.com/simonw/datasette/issues/1555#issuecomment-997128368,https://api.github.com/repos/simonw/datasette/issues/1555,997128368,IC_kwDOBm6k_c47bviw,9599,simonw,2021-12-18T02:32:43Z,2021-12-18T02:32:43Z,OWNER,I wonder why the `INSERT INTO` queries don't show up in that `?trace=1` view?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list,
https://github.com/simonw/datasette/issues/1555#issuecomment-997128251,https://api.github.com/repos/simonw/datasette/issues/1555,997128251,IC_kwDOBm6k_c47bvg7,9599,simonw,2021-12-18T02:31:51Z,2021-12-18T02:31:51Z,OWNER,"I was thinking it might even be possible to convert this into a `insert into tables select from ...` query:
https://github.com/simonw/datasette/blob/c00f29affcafce8314366852ba1a0f5a7dd25690/datasette/utils/internal_db.py#L102-L112
But the `SELECT` runs against a separate database from the `INSERT INTO`, so I would have to setup a cross-database connection for this which feels a little too complicated.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list,
https://github.com/simonw/datasette/issues/1555#issuecomment-997128080,https://api.github.com/repos/simonw/datasette/issues/1555,997128080,IC_kwDOBm6k_c47bveQ,9599,simonw,2021-12-18T02:30:19Z,2021-12-18T02:30:19Z,OWNER,"I think all of these queries happen in one place - in the `populate_schema_tables()` function - so optimizing them might be localized to just that area of the code, which would be nice:
https://github.com/simonw/datasette/blob/c00f29affcafce8314366852ba1a0f5a7dd25690/datasette/utils/internal_db.py#L97-L183","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list,
https://github.com/simonw/datasette/issues/1561#issuecomment-997127784,https://api.github.com/repos/simonw/datasette/issues/1561,997127784,IC_kwDOBm6k_c47bvZo,9599,simonw,2021-12-18T02:27:56Z,2021-12-18T02:27:56Z,OWNER,"Oh that's an interesting solution, combining the hashes of all of the individual databases.
I'm actually not a big fan of `hashed_url` mode - I implemented it right at the start of the project because it felt like a clever hack, and then ended up making it not-the-default a few years ago:
- #418
- #419
- #421
I've since not found myself wanting to use it at all for any of my projects - which makes me nervous, because it means there's a pretty complex feature that I'm not using at all, so it's only really protected by the existing unit tests for it.
What I'd really like to do is figure out how to have hashed URL mode work entirely as a plugin - then I could extract it from Datasette core entirely (which would simplify a bunch of stuff) but people who find the optimization useful would be able to access it.
I'm not sure that the existing plugin hooks are robust enough to do that yet though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082765654,"add hash id to ""_memory"" url if hashed url mode is turned on and crossdb is also turned on",
https://github.com/simonw/datasette/issues/1563#issuecomment-997127084,https://api.github.com/repos/simonw/datasette/issues/1563,997127084,IC_kwDOBm6k_c47bvOs,9599,simonw,2021-12-18T02:22:30Z,2021-12-18T02:22:30Z,OWNER,Docs here: https://docs.datasette.io/en/latest/internals.html#datasette-class,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083573206,Datasette(... files=) should not be a required argument,
https://github.com/simonw/datasette/issues/1563#issuecomment-997125191,https://api.github.com/repos/simonw/datasette/issues/1563,997125191,IC_kwDOBm6k_c47buxH,9599,simonw,2021-12-18T02:10:20Z,2021-12-18T02:10:20Z,OWNER,I should document the usage of this constructor in https://docs.datasette.io/en/stable/internals.html#datasette-class,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083573206,Datasette(... files=) should not be a required argument,
https://github.com/simonw/datasette/issues/1546#issuecomment-997124280,https://api.github.com/repos/simonw/datasette/issues/1546,997124280,IC_kwDOBm6k_c47bui4,9599,simonw,2021-12-18T02:05:16Z,2021-12-18T02:05:16Z,OWNER,"Sure - there are actually several levels to this.
The code that creates connections to the database is this: https://github.com/simonw/datasette/blob/83bacfa9452babe7bd66e3579e23af988d00f6ac/datasette/database.py#L72-L95
For files on disk, it does this:
```python
# For read-only connections
conn = sqlite3.connect( ""file:my.db?mode=ro"", uri=True, check_same_thread=False)
# For connections that should be treated as immutable:
conn = sqlite3.connect( ""file:my.db?immutable=1"", uri=True, check_same_thread=False)
```
For in-memory databases it runs this after the connection has been created:
```python
conn.execute(""PRAGMA query_only=1"")
```
SQLite `PRAGMA` queries are treated as dangerous: someone could run `PRAGMA query_only=0` to turn that previous option off for example.
So this function runs against any incoming SQL to verify that it looks like a `SELECT ...` and doesn't have anything like that in it.
https://github.com/simonw/datasette/blob/83bacfa9452babe7bd66e3579e23af988d00f6ac/datasette/utils/__init__.py#L195-L204
You can see the tests for that here: https://github.com/simonw/datasette/blob/b1fed48a95516ae84c0f020582303ab50ab817e2/tests/test_utils.py#L136-L170","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1076057610,validating the sql,
https://github.com/simonw/datasette/issues/1564#issuecomment-997122938,https://api.github.com/repos/simonw/datasette/issues/1564,997122938,IC_kwDOBm6k_c47buN6,9599,simonw,2021-12-18T01:55:25Z,2021-12-18T01:55:46Z,OWNER,"Made this change while working on this issue:
- #1567
I'm going to write a test for this that uses that `sleep()` SQL function from c35b84a2aabe2f14aeacf6cda4110ae1e94d6059.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083581011,_prepare_connection not called on write connections,
https://github.com/simonw/datasette/issues/1565#issuecomment-997121215,https://api.github.com/repos/simonw/datasette/issues/1565,997121215,IC_kwDOBm6k_c47bty_,9599,simonw,2021-12-18T01:45:44Z,2021-12-18T01:45:44Z,OWNER,I want to get this into Datasette 0.60 - #1566 - it's a small change that can unlock a lot of potential.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins,
https://github.com/simonw/datasette/issues/621#issuecomment-997120723,https://api.github.com/repos/simonw/datasette/issues/621,997120723,IC_kwDOBm6k_c47btrT,9599,simonw,2021-12-18T01:42:33Z,2021-12-18T01:42:33Z,OWNER,I refactored this code out into the `filters.py` module in aa7f0037a46eb76ae6fe9bf2a1f616c58738ecdf,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520681725,Syntax for ?_through= that works as a form field,
https://github.com/simonw/datasette/issues/617#issuecomment-552253893,https://api.github.com/repos/simonw/datasette/issues/617,552253893,MDEyOklzc3VlQ29tbWVudDU1MjI1Mzg5Mw==,9599,simonw,2019-11-11T00:46:42Z,2021-12-18T01:41:47Z,OWNER,"As noted in https://github.com/simonw/datasette/issues/621#issuecomment-552253208 a common pattern in this method is blocks of code that append new items to the `where_clauses`, `params` and `extra_human_descriptions` arrays. This is a useful refactoring opportunity.
Code that fits this pattern:
* The code that builds based on the filters: `where_clauses, params = filters.build_where_clauses(table)` and `human_description_en = filters.human_description_en(extra=extra_human_descriptions)`
* Code that handles `?_where=`: `where_clauses.extend(request.args[""_where""])` - though note that this also appends to a `extra_wheres_for_ui` array which nothing else uses
* The `_through=` code, see #621 for details
* The code that deals with `?_search=` FTS
The keyset pagination code modifies `where_clauses` and `params` too, but I don't think it's quite going to work with the same abstraction that would cover the above examples.
[UPDATE December 2021 - this comment became the basis for a new `filters_from_request` plugin hook, see also #473]","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",519613116,Refactor TableView.data() method,
https://github.com/simonw/datasette/issues/1518#issuecomment-981153060,https://api.github.com/repos/simonw/datasette/issues/1518,981153060,IC_kwDOBm6k_c46ezUk,9599,simonw,2021-11-28T21:13:09Z,2021-12-17T23:37:08Z,OWNER,"Two new requirements inspired by work on the `datasette-table` (and `datasette-notebook`) projects:
- #1533
- #1534","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/sqlite-utils/issues/358#issuecomment-996482595,https://api.github.com/repos/simonw/sqlite-utils/issues/358,996482595,IC_kwDOCGYnMM47ZR4j,11597658,luxint,2021-12-17T06:57:51Z,2021-12-17T23:24:16Z,NONE,"> This goes beyond the `transform()` method - the curious methods that create new SQL tables could benefit from the ability to add `CHECK` constraints too.
>
> I haven't used these myself, do you have any `CREATE TABLE` examples that use them that you can share?
I'm using them myself for the first time as well, this is a tutorial of how to use (and change) them in sqlite: https://www.sqlitetutorial.net/sqlite-check-constraint/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082651698,Support for CHECK constraints,
https://github.com/simonw/datasette/issues/1518#issuecomment-997082845,https://api.github.com/repos/simonw/datasette/issues/1518,997082845,IC_kwDOBm6k_c47bkbd,9599,simonw,2021-12-17T23:10:09Z,2021-12-17T23:10:17Z,OWNER,These changes so far are now in the 0.60a0 alpha: https://github.com/simonw/datasette/releases/tag/0.60a0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/pull/1559#issuecomment-997082676,https://api.github.com/repos/simonw/datasette/issues/1559,997082676,IC_kwDOBm6k_c47bkY0,9599,simonw,2021-12-17T23:09:41Z,2021-12-17T23:09:41Z,OWNER,This is now available to try out in Datasette 0.60a0: https://github.com/simonw/datasette/releases/tag/0.60a0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082743068,"filters_from_request plugin hook, now used in TableView",
https://github.com/simonw/datasette/pull/1562#issuecomment-997082189,https://api.github.com/repos/simonw/datasette/issues/1562,997082189,IC_kwDOBm6k_c47bkRN,9599,simonw,2021-12-17T23:08:14Z,2021-12-17T23:08:14Z,OWNER,"Oh that makes sense: In Python 3.6 this happens:
```
Collecting janus<1.1,>=0.6.2
Using cached janus-0.7.0-py3-none-any.whl (6.9 kB)
```
While in Python 3.7 or higher this happens:
```
Collecting janus<1.1,>=0.6.2
Downloading janus-1.0.0-py3-none-any.whl (6.9 kB)
```
So this is safe to apply because `pip` is smart enough to pick the version of Janus that works for that Python version.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083246400,"Update janus requirement from <0.8,>=0.6.2 to >=0.6.2,<1.1",
https://github.com/simonw/datasette/pull/1562#issuecomment-997081673,https://api.github.com/repos/simonw/datasette/issues/1562,997081673,IC_kwDOBm6k_c47bkJJ,9599,simonw,2021-12-17T23:06:38Z,2021-12-17T23:06:38Z,OWNER,"From this diff between `0.7.0` and `1.0`: https://github.com/aio-libs/janus/compare/v0.7.0...v1.0.0
It looks like the only change relevant to compatibility is `loop = asyncio.get_running_loop()` directly instead of falling back to `asyncio.get_event_loop()` if `get_running_loop` isn't available.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083246400,"Update janus requirement from <0.8,>=0.6.2 to >=0.6.2,<1.1",
https://github.com/simonw/datasette/pull/1562#issuecomment-997080352,https://api.github.com/repos/simonw/datasette/issues/1562,997080352,IC_kwDOBm6k_c47bj0g,9599,simonw,2021-12-17T23:03:08Z,2021-12-17T23:03:08Z,OWNER,"They say they've dropped 3.6 support, but Datasette's tests against 3.6 are still passing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083246400,"Update janus requirement from <0.8,>=0.6.2 to >=0.6.2,<1.1",
https://github.com/simonw/datasette/issues/1566#issuecomment-997078812,https://api.github.com/repos/simonw/datasette/issues/1566,997078812,IC_kwDOBm6k_c47bjcc,9599,simonw,2021-12-17T22:58:55Z,2021-12-17T22:58:55Z,OWNER,The release notes for the 0.60a0 alpha will be useful here: https://github.com/simonw/datasette/releases/tag/0.60a0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083669410,Release Datasette 0.60,
https://github.com/simonw/datasette/issues/1565#issuecomment-997077410,https://api.github.com/repos/simonw/datasette/issues/1565,997077410,IC_kwDOBm6k_c47bjGi,9599,simonw,2021-12-17T22:54:45Z,2021-12-17T22:54:45Z,OWNER,"The table page should expose the query both with and without the `limit` clause. The above gave me back:
```sql
select id, ACCESS_TYP, UNIT_ID, UNIT_NAME, SUID_NMA, AGNCY_ID, AGNCY_NAME, AGNCY_LEV,
AGNCY_TYP, AGNCY_WEB, LAYER, MNG_AG_ID, MNG_AGENCY, MNG_AG_LEV, MNG_AG_TYP,
PARK_URL, COUNTY, ACRES, LABEL_NAME, YR_EST, DES_TP, GAP_STS, geometry
from CPAD_2020a_Units where ""AGNCY_LEV"" = :p0 order by id limit 101
```
But I actually wanted to run a `fetch()` against a version of that without the `order by id limit 101` bit (I wanted to figure out the `Extent()` of the `geometry` column) - so I need something like `datasette.table_sql_no_order_no_limit`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins,
https://github.com/simonw/datasette/issues/1565#issuecomment-997069128,https://api.github.com/repos/simonw/datasette/issues/1565,997069128,IC_kwDOBm6k_c47bhFI,9599,simonw,2021-12-17T22:31:18Z,2021-12-17T22:31:18Z,OWNER,This should aim to be as consistent as possible with the various arguments to hooks on https://docs.datasette.io/en/stable/plugin_hooks.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins,
https://github.com/simonw/datasette/pull/1559#issuecomment-996961196,https://api.github.com/repos/simonw/datasette/issues/1559,996961196,IC_kwDOBm6k_c47bGus,9599,simonw,2021-12-17T19:00:53Z,2021-12-17T19:00:53Z,OWNER,"I'm going to merge this to `main` now. I can continue the refactoring there, but having it in `main` means I can put out an alpha release with the new hook which will unblock me from running tests against it in this repo: https://github.com/simonw/datasette-leaflet-freedraw/pull/8","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082743068,"filters_from_request plugin hook, now used in TableView",
https://github.com/simonw/datasette/pull/1559#issuecomment-996959325,https://api.github.com/repos/simonw/datasette/issues/1559,996959325,IC_kwDOBm6k_c47bGRd,9599,simonw,2021-12-17T18:59:54Z,2021-12-17T18:59:54Z,OWNER,I've convinced myself that this plugin hook design is good through this `datasette-leaflet-freedraw` prototype: https://github.com/simonw/datasette-leaflet-freedraw/blob/e8a16a0fe90656b8d655c02881d23a2b9833281d/datasette_leaflet_freedraw/__init__.py,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082743068,"filters_from_request plugin hook, now used in TableView",
https://github.com/simonw/datasette/issues/473#issuecomment-996958442,https://api.github.com/repos/simonw/datasette/issues/473,996958442,IC_kwDOBm6k_c47bGDq,9599,simonw,2021-12-17T18:59:27Z,2021-12-17T18:59:27Z,OWNER,I'm happy with how the prototype that used this plugin in `datasette-leaflet-freedraw` turned out: https://github.com/simonw/datasette-leaflet-freedraw/blob/e8a16a0fe90656b8d655c02881d23a2b9833281d/datasette_leaflet_freedraw/__init__.py,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",445850934,Plugin hook: filters_from_request,
https://github.com/simonw/datasette/issues/473#issuecomment-996345233,https://api.github.com/repos/simonw/datasette/issues/473,996345233,IC_kwDOBm6k_c47YwWR,9599,simonw,2021-12-17T01:20:31Z,2021-12-17T18:13:01Z,OWNER,I could use this hook to add table filtering on a map to the existing `datasette-leaflet-freedraw` plugin.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",445850934,Plugin hook: filters_from_request,
https://github.com/simonw/datasette/pull/1559#issuecomment-996289541,https://api.github.com/repos/simonw/datasette/issues/1559,996289541,IC_kwDOBm6k_c47YiwF,22429695,codecov[bot],2021-12-17T00:07:42Z,2021-12-17T17:28:54Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1559?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1559](https://codecov.io/gh/simonw/datasette/pull/1559?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (71af58d) into [main](https://codecov.io/gh/simonw/datasette/commit/0663d5525cc41e9260ac7d1f6386d3a6eb5ad2a9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0663d55) will **increase** coverage by `0.09%`.
> The diff coverage is `97.97%`.
[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1559/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1559?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
```diff
@@ Coverage Diff @@
## main #1559 +/- ##
==========================================
+ Coverage 91.96% 92.05% +0.09%
==========================================
Files 34 34
Lines 4442 4493 +51
==========================================
+ Hits 4085 4136 +51
Misses 357 357
```
| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1559?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/plugins.py](https://codecov.io/gh/simonw/datasette/pull/1559/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3BsdWdpbnMucHk=) | `82.35% <ø> (ø)` | |
| [datasette/filters.py](https://codecov.io/gh/simonw/datasette/pull/1559/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2ZpbHRlcnMucHk=) | `95.69% <97.67%> (+1.33%)` | :arrow_up: |
| [datasette/hookspecs.py](https://codecov.io/gh/simonw/datasette/pull/1559/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2hvb2tzcGVjcy5weQ==) | `100.00% <100.00%> (ø)` | |
| [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1559/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `96.21% <100.00%> (+0.13%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1559?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1559?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [0663d55...71af58d](https://codecov.io/gh/simonw/datasette/pull/1559?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082743068,"filters_from_request plugin hook, now used in TableView",
https://github.com/simonw/datasette/pull/1559#issuecomment-996895423,https://api.github.com/repos/simonw/datasette/issues/1559,996895423,IC_kwDOBm6k_c47a2q_,9599,simonw,2021-12-17T17:28:44Z,2021-12-17T17:28:44Z,OWNER,"Before I land this I'm going to build one prototype plugin against it to confirm that the new hook is useful in its current shape.
I'll add support for filtering a table by drawing on a map to https://datasette.io/plugins/datasette-leaflet-freedraw","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082743068,"filters_from_request plugin hook, now used in TableView",
https://github.com/simonw/datasette/pull/1562#issuecomment-996716158,https://api.github.com/repos/simonw/datasette/issues/1562,996716158,IC_kwDOBm6k_c47aK5-,22429695,codecov[bot],2021-12-17T13:18:49Z,2021-12-17T13:18:49Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1562?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1562](https://codecov.io/gh/simonw/datasette/pull/1562?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (2f008e8) into [main](https://codecov.io/gh/simonw/datasette/commit/0663d5525cc41e9260ac7d1f6386d3a6eb5ad2a9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0663d55) will **not change** coverage.
> The diff coverage is `n/a`.
[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1562/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1562?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
```diff
@@ Coverage Diff @@
## main #1562 +/- ##
=======================================
Coverage 91.96% 91.96%
=======================================
Files 34 34
Lines 4442 4442
=======================================
Hits 4085 4085
Misses 357 357
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1562?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1562?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [0663d55...2f008e8](https://codecov.io/gh/simonw/datasette/pull/1562?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083246400,"Update janus requirement from <0.8,>=0.6.2 to >=0.6.2,<1.1",
https://github.com/simonw/datasette/pull/1204#issuecomment-996488925,https://api.github.com/repos/simonw/datasette/issues/1204,996488925,IC_kwDOBm6k_c47ZTbd,9599,simonw,2021-12-17T07:10:48Z,2021-12-17T07:10:48Z,OWNER,I think this is missing the `_macro.html` template file but I have that in my Dropbox.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",793002853,WIP: Plugin includes,
https://github.com/simonw/datasette/issues/473#issuecomment-996484551,https://api.github.com/repos/simonw/datasette/issues/473,996484551,IC_kwDOBm6k_c47ZSXH,9599,simonw,2021-12-17T07:02:21Z,2021-12-17T07:04:23Z,OWNER,"The one slightly weird thing about this hook is how it adds `extra_context` without an obvious way for plugins to add extra HTML to the templates based on that context.
Maybe I need the proposed mechanism from
- #1191
Which has an in-progress PR:
- #1204","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",445850934,Plugin hook: filters_from_request,
https://github.com/simonw/datasette/issues/1191#issuecomment-761104933,https://api.github.com/repos/simonw/datasette/issues/1191,761104933,MDEyOklzc3VlQ29tbWVudDc2MTEwNDkzMw==,9599,simonw,2021-01-15T18:21:26Z,2021-12-17T07:03:02Z,OWNER,"Also related: #857 (comprehensive documentation of variables available to templates) - since then the plugin hook could be fed the full template context and use that to do its thing.
Or maybe the plugin hooks gets to return the name of a template that should be `{% include %}` into the page at that point? But the plugin may want to add extra context that is available to that template include.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",787098345,Ability for plugins to collaborate when adding extra HTML to blocks in default templates,
https://github.com/simonw/datasette/pull/1559#issuecomment-996286808,https://api.github.com/repos/simonw/datasette/issues/1559,996286808,IC_kwDOBm6k_c47YiFY,9599,simonw,2021-12-17T00:01:43Z,2021-12-17T00:01:43Z,OWNER,"This already has tests and documentation, and I've used it to refactor out the logic for `?_where=` and `?_search=` and `?_through=`.
Do I like this enough to land it on `main`?
Also, I think I can still use it to refactor out the `Filters` code that implements `?col=x` and `?col__lt=5` and suchlike.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082743068,"filters_from_request plugin hook, now used in TableView",
https://github.com/simonw/datasette/issues/473#issuecomment-996286199,https://api.github.com/repos/simonw/datasette/issues/473,996286199,IC_kwDOBm6k_c47Yh73,9599,simonw,2021-12-17T00:00:22Z,2021-12-17T00:00:22Z,OWNER,Documentation for that hook in the PR branch: https://github.com/simonw/datasette/blob/54e9b3972f277431a001e685f78e5dd6403a6d8d/docs/plugin_hooks.rst#filters_from_requestrequest-database-table-datasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",445850934,Plugin hook: filters_from_request,
https://github.com/simonw/datasette/issues/1518#issuecomment-996286104,https://api.github.com/repos/simonw/datasette/issues/1518,996286104,IC_kwDOBm6k_c47Yh6Y,9599,simonw,2021-12-17T00:00:07Z,2021-12-17T00:00:07Z,OWNER,Documentation of the new hook in the PR: https://github.com/simonw/datasette/blob/54e9b3972f277431a001e685f78e5dd6403a6d8d/docs/plugin_hooks.rst#filters_from_requestrequest-database-table-datasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/issues/473#issuecomment-996275108,https://api.github.com/repos/simonw/datasette/issues/473,996275108,IC_kwDOBm6k_c47YfOk,9599,simonw,2021-12-16T23:32:22Z,2021-12-16T23:32:30Z,OWNER,This filter design can only influence the `where` component of the SQL clause - it's not able to modify the `SELECT` columns or adjust the `ORDER BY` or `OFFSET LIMIT` parts. I think that's OK.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",445850934,Plugin hook: filters_from_request,
https://github.com/simonw/datasette/issues/1518#issuecomment-996272906,https://api.github.com/repos/simonw/datasette/issues/1518,996272906,IC_kwDOBm6k_c47YesK,9599,simonw,2021-12-16T23:27:42Z,2021-12-16T23:27:42Z,OWNER,Got a TIL out of this: https://til.simonwillison.net/pluggy/multiple-hooks-same-file,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/issues/473#issuecomment-996267817,https://api.github.com/repos/simonw/datasette/issues/473,996267817,IC_kwDOBm6k_c47Ydcp,9599,simonw,2021-12-16T23:17:52Z,2021-12-16T23:19:00Z,OWNER,"I revisited this idea in #1518 and came up with a slightly different name and design for the hook:
```python
@hookspec
def filters_from_request(request, database, table, datasette):
""""""
Return FilterArguments(
where_clauses=[str, str, str],
params={},
human_descriptions=[str, str, str],
extra_context={}
) based on the request""""""
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",445850934,Plugin hook: filters_from_request,
https://github.com/simonw/datasette/issues/1518#issuecomment-996264617,https://api.github.com/repos/simonw/datasette/issues/1518,996264617,IC_kwDOBm6k_c47Ycqp,9599,simonw,2021-12-16T23:11:12Z,2021-12-16T23:11:12Z,OWNER,I managed to extract both `_search=` and `_where=` out using a prototype of that hook. I wonder if it could extract the complex code for `?_next` too?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/issues/1518#issuecomment-996250585,https://api.github.com/repos/simonw/datasette/issues/1518,996250585,IC_kwDOBm6k_c47YZPZ,9599,simonw,2021-12-16T22:43:37Z,2021-12-16T22:45:07Z,OWNER,"Ran into a problem prototyping that hook up for handling `?_where=` - that feature also adds a little bit of extra template context in order to show the interface for removing wheres - the `extra_wheres_for_ui` variable: https://github.com/simonw/datasette/blob/0663d5525cc41e9260ac7d1f6386d3a6eb5ad2a9/datasette/views/table.py#L457-L463
Maybe change to this?
```python
class FilterArguments(NamedTuple):
where_clauses: List[str]
params: Dict[str, Union[str, int, float]]
human_descriptions: List[str]
extra_context: Dict[str, Any]
```
That might be necessary for `_search` too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/issues/1518#issuecomment-996248713,https://api.github.com/repos/simonw/datasette/issues/1518,996248713,IC_kwDOBm6k_c47YYyJ,9599,simonw,2021-12-16T22:39:47Z,2021-12-16T22:39:47Z,OWNER,"The hook could return a named tuple like this one:
```python
from typing import NamedTuple, List, Optional, Union, Dict
class FilterArguments(NamedTuple):
where_clauses: List[str]
params: Dict[str, Union[str, int, float]]
human_descriptions: List[str]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/issues/1518#issuecomment-996240802,https://api.github.com/repos/simonw/datasette/issues/1518,996240802,IC_kwDOBm6k_c47YW2i,9599,simonw,2021-12-16T22:25:00Z,2021-12-16T22:36:04Z,OWNER,"I think that plugin hook would get given the `request` object (and `datasette` and the name of the database and table) and returns a list of SQL fragments, a dictionary of lookup arguments and a list of human-description fragments - or an awaitable.
`filters_from_request(request, database, table, datasette)` perhaps? (Similar in name to `actor_from_request`).
```python
@hookspec
def filters_from_request(request, database, table, datasette):
""""""Return (where_clauses, params_dict, human_descriptions) based on the request""""""
```
Turns out that's pretty much exactly what I implemented in 5116c4ec8aed5091e1f75415424b80f613518dc6 for #473:
```python
@hookspec
def table_filter():
""Custom filtering of the current table based on the request""
```
```python
TableFilter = namedtuple(""TableFilter"", (
""human_description_extras"", ""where_clauses"", ""params"")
)
```
```python
# filter_arguments plugin hook support
for awaitable_fn in pm.hook.table_filter():
extras = await awaitable_fn(
view=self, name=name, table=table, request=request
)
human_description_extras.extend(extras.human_description_extras)
where_clauses.extend(extras.where_clauses)
params.update(extras.params)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/sqlite-utils/issues/358#issuecomment-996232461,https://api.github.com/repos/simonw/sqlite-utils/issues/358,996232461,IC_kwDOCGYnMM47YU0N,9599,simonw,2021-12-16T22:10:39Z,2021-12-16T22:10:39Z,OWNER,"This goes beyond the `transform()` method - the curious methods that create new SQL tables could benefit from the ability to add `CHECK` constraints too.
I haven't used these myself, do you have any `CREATE TABLE` examples that use them that you can share?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082651698,Support for CHECK constraints,
https://github.com/simonw/datasette/issues/1552#issuecomment-996229007,https://api.github.com/repos/simonw/datasette/issues/1552,996229007,IC_kwDOBm6k_c47YT-P,3556,davidbgk,2021-12-16T22:04:39Z,2021-12-16T22:04:39Z,CONTRIBUTOR,"Wow, that was fast, thank you so much @simonw !
> I'm also not convinced that this configuration syntax is right. It's a bit weird having a `""facets""` list that can either by column-name-strings or `{""type-of-facet"": ""column-name""}` objects. Maybe there's a better design for this?
I agree that it's not ideal, my initial naive approach was to detect if it's an array, like what is done here:
https://github.com/simonw/datasette/blob/2c07327d23d9c5cf939ada9ba4091c1b8b2ba42d/datasette/facets.py#L312-L313
But it requires an extra query to determine the type, which is a bit problematic, especially for big tables I guess.
Taking a look at #510, I wonder if a `facet_delimiter` should be defined for that kind of columns (that would help our team not to have an intermediary conversion step from `foo|bar` to `[""foo"",""bar""]` for instance).
To be consistent with the `--extract-column` parameter, maybe an explicit casting/delimiter would be useful: `--set-column 'Foo:Array:|'`.
Throwing a lot of ideas without knowing the big picture… but sometimes newcomers have superpowers :).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`),
https://github.com/simonw/datasette/issues/1518#issuecomment-996227713,https://api.github.com/repos/simonw/datasette/issues/1518,996227713,IC_kwDOBm6k_c47YTqB,9599,simonw,2021-12-16T22:02:35Z,2021-12-16T22:03:55Z,OWNER,"Is there an opportunity to refactor things using a new plugin hook here? Maybe the `register_filters` hook from #473, where the hook becomes responsible for building where clauses (and human descriptions of them) based on the incoming query string.
That version dealt with `Filter` classes, but those might be a bit too low-level for this.
`?_spatial_within=GEOJSON` was an interesting idea attached to that issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/issues/1518#issuecomment-996225889,https://api.github.com/repos/simonw/datasette/issues/1518,996225889,IC_kwDOBm6k_c47YTNh,9599,simonw,2021-12-16T21:59:32Z,2021-12-16T22:00:42Z,OWNER,I added a ton of comments to the `data()` method which really helps get a better feel for how this all works: https://github.com/simonw/datasette/blob/0663d5525cc41e9260ac7d1f6386d3a6eb5ad2a9/datasette/views/table.py#L322,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/issues/1518#issuecomment-996225235,https://api.github.com/repos/simonw/datasette/issues/1518,996225235,IC_kwDOBm6k_c47YTDT,9599,simonw,2021-12-16T21:58:24Z,2021-12-16T21:58:41Z,OWNER,"A fundamental operation of this view is to construct the SQL query and accompanying human description based on the incoming query string parameters.
The human description is the bit at the top of https://latest.datasette.io/fixtures/searchable?_search=dog&_sort=pk&_facet=text2&text2=sara+weasel that says:
> 1 row where search matches ""dog"" and text2 = ""sara weasel"" sorted by pk
(Also used in the page ``).
The code actually gathers three things:
- Fragments of the `where` clause, for example ` ""text2"" = :p0`
- Parameters, e.g. `{""p0"": ""sara weasel""}`
- Human description components, e.g. `text2 = ""sara weasel""`
Some operations such as `?_where=` don't currently provide an extra human description component.
`_where=` also doesn't populate a parameter, but maybe it could? Would be neat if in the future `?_where=foo+=+:bar` worked and added a `bar` input field to the screen, as seen with custom queries.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/issues/1518#issuecomment-996219117,https://api.github.com/repos/simonw/datasette/issues/1518,996219117,IC_kwDOBm6k_c47YRjt,9599,simonw,2021-12-16T21:47:51Z,2021-12-16T21:49:24Z,OWNER,"Should facets really not be displayed on pages past page one (where `?_next=` is set)? That made sense to me at the time, but I'm now having second thoughts about it.
I guess it's a useful performance tweak for when crawlers keep hitting the `?_next=` link.
Actually it looks like facets DO display on subsequent pages, e.g. on https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_next=200 - but facet suggestions do not, thanks to this code: https://github.com/simonw/datasette/blob/2c07327d23d9c5cf939ada9ba4091c1b8b2ba42d/datasette/views/table.py#L777-L785
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/issues/1558#issuecomment-996204369,https://api.github.com/repos/simonw/datasette/issues/1558,996204369,IC_kwDOBm6k_c47YN9R,9599,simonw,2021-12-16T21:23:25Z,2021-12-16T21:23:25Z,OWNER,"Related: Following the fix for #625 I noticed that `facets_timed_out` gives you just the column name, but doesn't let you know which particular type of facet (`date` or `array` for example) suffered the timeout: https://github.com/simonw/datasette/blob/0d4145d0f4d8b2a7edc1ba4aac1be56cd536a10a/datasette/facets.py#L269-L270
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082584499,Redesign `facet_results` JSON structure prior to Datasette 1.0,
https://github.com/simonw/sqlite-utils/issues/357#issuecomment-996179930,https://api.github.com/repos/simonw/sqlite-utils/issues/357,996179930,IC_kwDOCGYnMM47YH_a,9599,simonw,2021-12-16T20:43:19Z,2021-12-16T20:43:19Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079422215,pytest-runner is not required,
https://github.com/simonw/datasette/issues/625#issuecomment-996170510,https://api.github.com/repos/simonw/datasette/issues/625,996170510,IC_kwDOBm6k_c47YFsO,9599,simonw,2021-12-16T20:27:41Z,2021-12-16T20:27:41Z,OWNER,"And here's the new JSON: https://latest.datasette.io/fixtures/facetable.json?_facet=created&_facet_date=created&_facet=tags&_facet_array=tags&_nosuggest=1
```
{
""database"": ""fixtures"",
""table"": ""facetable"",
""is_view"": false,
""human_description_en"": """",
...
""facet_results"": {
""created"": {
""name"": ""created"",
""type"": ""column"",
...
},
""tags"": {
""name"": ""tags"",
""type"": ""column"",
...
},
""created_2"": {
""name"": ""created"",
""type"": ""date"",
...
},
""tags_2"": {
""name"": ""tags"",
""type"": ""array"",
...
}
}
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing,
https://github.com/simonw/datasette/issues/625#issuecomment-996165659,https://api.github.com/repos/simonw/datasette/issues/625,996165659,IC_kwDOBm6k_c47YEgb,9599,simonw,2021-12-16T20:19:53Z,2021-12-16T20:19:53Z,OWNER,Demo of the fix: https://latest.datasette.io/fixtures/facetable?_facet=created&_facet_date=created&_facet=tags&_facet_array=tags#facet-tags,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing,
https://github.com/simonw/datasette/issues/625#issuecomment-996161380,https://api.github.com/repos/simonw/datasette/issues/625,996161380,IC_kwDOBm6k_c47YDdk,9599,simonw,2021-12-16T20:13:05Z,2021-12-16T20:13:05Z,OWNER,I updated the example code in the facet plugin hook documentation: https://github.com/simonw/datasette/blob/95d0dd7a1cf6be6b7da41e1404184217eb93f64a/docs/plugin_hooks.rst#register_facet_classes,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing,
https://github.com/simonw/datasette/issues/625#issuecomment-996152213,https://api.github.com/repos/simonw/datasette/issues/625,996152213,IC_kwDOBm6k_c47YBOV,9599,simonw,2021-12-16T19:59:46Z,2021-12-16T20:00:05Z,OWNER,"Since no-one is using that plugin hook I'm going to alter its contract slightly. I'll still keep the existing JSON format working though (until 1.0), since it's much more likely that people are using that JSON somewhere.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing,
https://github.com/simonw/datasette/issues/830#issuecomment-996151246,https://api.github.com/repos/simonw/datasette/issues/830,996151246,IC_kwDOBm6k_c47YA_O,9599,simonw,2021-12-16T19:58:22Z,2021-12-16T19:58:22Z,OWNER,"As of today, 16 December 2021, I'm still not seeing any evidence that anyone is using this hook (yet) according to GitHub code search: https://cs.github.com/?scopeName=All+repos&scope=&q=register_facet_classes%20-repo%3Asimonw%2Fdatasette","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636511683,Redesign register_facet_classes plugin hook,
https://github.com/simonw/datasette/issues/625#issuecomment-996150904,https://api.github.com/repos/simonw/datasette/issues/625,996150904,IC_kwDOBm6k_c47YA54,9599,simonw,2021-12-16T19:57:52Z,2021-12-16T19:57:52Z,OWNER,Good news - GitHub's new code search doesn't show ANYONE using that plugin hook - not surprising since it has that documentation warning plus it's just not a very clearly usable hook: https://cs.github.com/?scopeName=All+repos&scope=&q=register_facet_classes%20-repo%3Asimonw%2Fdatasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing,
https://github.com/simonw/datasette/issues/625#issuecomment-996149720,https://api.github.com/repos/simonw/datasette/issues/625,996149720,IC_kwDOBm6k_c47YAnY,9599,simonw,2021-12-16T19:56:14Z,2021-12-16T19:56:14Z,OWNER,"This bad design is even covered in the plugin hooks documentation: https://docs.datasette.io/en/0.59.4/plugin_hooks.html#register-facet-classes
It does at least have the following warning:
> **Warning**
>
> The design of this plugin hook is unstable and may change. See [issue 830](https://github.com/simonw/datasette/issues/830).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing,
https://github.com/simonw/datasette/issues/625#issuecomment-996146762,https://api.github.com/repos/simonw/datasette/issues/625,996146762,IC_kwDOBm6k_c47X_5K,9599,simonw,2021-12-16T19:51:44Z,2021-12-16T19:51:44Z,OWNER,"Here's where `facet_results` is built up: https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/views/table.py#L752-L758
So the decision to key things based on column name is actually embedded deep in the existing facet classes here:
https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/facets.py#L224-L226
https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/facets.py#L395-L397
https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/facets.py#L510-L512","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing,
https://github.com/simonw/datasette/issues/1558#issuecomment-996134716,https://api.github.com/repos/simonw/datasette/issues/1558,996134716,IC_kwDOBm6k_c47X888,9599,simonw,2021-12-16T19:46:21Z,2021-12-16T19:46:21Z,OWNER,"The flaw in the current design is illustrated by this example:
```
""facet_results"": {
""tags"": {
""name"": ""tags"",
""type"": ""array"",
""results"": [...],
""hideable"": false,
""toggle_url"": ""/fixtures/facetable.json?_facet=tags&_trace=1&_nosuggest=1"",
""truncated"": false
},
""created"": {
""name"": ""created"",
""type"": ""date"",
""results"": [...]
```
This was the cause of the bug in #625 - the each of those objects is keyed by the name of the column, which left no room for faceting the same column once by date and once by column value.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082584499,Redesign `facet_results` JSON structure prior to Datasette 1.0,
https://github.com/simonw/datasette/issues/625#issuecomment-996130862,https://api.github.com/repos/simonw/datasette/issues/625,996130862,IC_kwDOBm6k_c47X8Au,9599,simonw,2021-12-16T19:44:48Z,2021-12-16T19:44:48Z,OWNER,"Decision: as an initial fix I'm going to de-duplicate those keys by using `tags__array` etc - with a `_2` on the end if that key is already used.
I'll open a separate issue to redesign this better for Datasette 1.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing,
https://github.com/simonw/datasette/issues/625#issuecomment-996121736,https://api.github.com/repos/simonw/datasette/issues/625,996121736,IC_kwDOBm6k_c47X5yI,9599,simonw,2021-12-16T19:37:08Z,2021-12-16T19:37:08Z,OWNER,"Really `facet_results` here should be an array of objects, not an object that maps poorly designed string keys to those objects.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing,
https://github.com/simonw/datasette/issues/625#issuecomment-996119954,https://api.github.com/repos/simonw/datasette/issues/625,996119954,IC_kwDOBm6k_c47X5WS,9599,simonw,2021-12-16T19:36:01Z,2021-12-16T19:36:11Z,OWNER,"Datasette's own HTML rendering code doesn't actually use the keys in `facet_results` - it instead loops through `sorted_facet_results` which is defined like this: https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/views/table.py#L937-L941
And used like this:
https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/templates/table.html#L154-L156","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing,
https://github.com/simonw/datasette/issues/625#issuecomment-996118401,https://api.github.com/repos/simonw/datasette/issues/625,996118401,IC_kwDOBm6k_c47X4-B,9599,simonw,2021-12-16T19:34:28Z,2021-12-16T19:34:55Z,OWNER,"The big question here is do I break any existing clients of the `""facet_results""` JSON API?
It's still pre-1.0 so I could break them, but I've also built my own code against this in the past so it's likely other people have too.
If I don't break them, I will instead need to come up with a naming convention for those keys - something like `""tags__array""` for example. As well as a way to ensure that a column called `tags__array` doesn't end up conflicting with the `tags__array` key!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing,
https://github.com/simonw/datasette/issues/1557#issuecomment-996115949,https://api.github.com/repos/simonw/datasette/issues/1557,996115949,IC_kwDOBm6k_c47X4Xt,9599,simonw,2021-12-16T19:30:55Z,2021-12-16T19:30:55Z,OWNER,"Demo: compare https://latest.datasette.io/fixtures/facetable?_facet=_city_id&_nosuggest=1 to https://latest.datasette.io/fixtures/facetable?_facet=_city_id
Documentation: bottom of https://docs.datasette.io/en/latest/json_api.html#special-table-arguments","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082564912,`?_nosuggest=1` parameter for disabling facet suggestions on table view,
https://github.com/simonw/datasette/issues/1556#issuecomment-996104214,https://api.github.com/repos/simonw/datasette/issues/1556,996104214,IC_kwDOBm6k_c47X1gW,9599,simonw,2021-12-16T19:15:00Z,2021-12-16T19:15:28Z,OWNER,"Demo: https://latest.datasette.io/fixtures/facetable?_facet=planet_int&_facet=_city_id&_facet=created#facet-created
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1081318247,"Show count of facet values always, not just for `?_facet_size=max`",
https://github.com/simonw/datasette/issues/1553#issuecomment-996103956,https://api.github.com/repos/simonw/datasette/issues/1553,996103956,IC_kwDOBm6k_c47X1cU,9599,simonw,2021-12-16T19:14:38Z,2021-12-16T19:14:38Z,OWNER,This is a really interesting idea - kind of similar to how many APIs include custom HTTP headers informing of rate-limits.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1079111498,if csv export is truncated in non streaming mode set informative response header,
https://github.com/simonw/datasette/issues/625#issuecomment-996100774,https://api.github.com/repos/simonw/datasette/issues/625,996100774,IC_kwDOBm6k_c47X0qm,9599,simonw,2021-12-16T19:10:01Z,2021-12-16T19:10:48Z,OWNER,"I think the problem here may be in the design of the JSON returned by facets. It looks like this:
```
""facet_results"": {
""tags"": {
""name"": ""tags"",
""type"": ""array"",
""results"": [...],
""hideable"": false,
""toggle_url"": ""/fixtures/facetable.json?_facet=tags&_trace=1&_nosuggest=1"",
""truncated"": false
},
""created"": {
""name"": ""created"",
""type"": ""date"",
""results"": [...]
```
The problem then is that the `tags` key is over-ridden by the second facet with a different type against the same column name!
https://latest-with-plugins.datasette.io/fixtures/facetable?_trace=1&_facet=created&_facet_date=created&_facet_array=tags&_facet=tags confirms that the SQL queries for those facets are being executed - but the final JSON doesn't show them on https://latest-with-plugins.datasette.io/fixtures/facetable.json?_trace=1&_facet=created&_facet_date=created&_facet_array=tags&_facet=tags
They're not available in the template context either: https://latest-with-plugins.datasette.io/fixtures/facetable?_facet=created&_facet_date=created&_facet_array=tags&_facet=tags&_context=1","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing,
https://github.com/simonw/datasette/issues/625#issuecomment-996093884,https://api.github.com/repos/simonw/datasette/issues/625,996093884,IC_kwDOBm6k_c47Xy-8,9599,simonw,2021-12-16T19:00:28Z,2021-12-16T19:00:28Z,OWNER,Implementing #1552 has made a fix for this bug even more important.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing,
https://github.com/simonw/datasette/issues/1552#issuecomment-996084899,https://api.github.com/repos/simonw/datasette/issues/1552,996084899,IC_kwDOBm6k_c47Xwyj,9599,simonw,2021-12-16T18:48:14Z,2021-12-16T18:48:14Z,OWNER,Updated documentation: https://github.com/simonw/datasette/blob/20a2ed6bec367d2f6759be4a879364a72780b59d/docs/facets.rst#facets-in-metadatajson,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`),
https://github.com/simonw/datasette/issues/1552#issuecomment-996077053,https://api.github.com/repos/simonw/datasette/issues/1552,996077053,IC_kwDOBm6k_c47Xu39,9599,simonw,2021-12-16T18:36:41Z,2021-12-16T18:36:41Z,OWNER,"... actually no, I WILL document this, because not documenting this is what got us to this point in the first place!","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 1, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`),
https://github.com/simonw/datasette/issues/1552#issuecomment-996076373,https://api.github.com/repos/simonw/datasette/issues/1552,996076373,IC_kwDOBm6k_c47XutV,9599,simonw,2021-12-16T18:35:40Z,2021-12-16T18:35:40Z,OWNER,"I'm going to ship your fix now, but I'm not going to add this to the documentation yet because I hope to improve the design prior to Datasette 1.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`),
https://github.com/simonw/datasette/issues/1552#issuecomment-996046304,https://api.github.com/repos/simonw/datasette/issues/1552,996046304,IC_kwDOBm6k_c47XnXg,9599,simonw,2021-12-16T17:53:40Z,2021-12-16T18:16:12Z,OWNER,"I'm also not convinced that this configuration syntax is right. It's a bit weird having a `""facets""` list that can either by column-name-strings or `{""type-of-facet"": ""column-name""}` objects. Maybe there's a better design for this?
Part of the problem here is that facets were designed to accept optional extra configuration - partly to support `m2m` facets in #495 - but I haven't actually shipped any facets that use that ability.
Facet by delimiter would be a good one to exercise that ability:
- #510","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`),
https://github.com/simonw/datasette/issues/1552#issuecomment-996045776,https://api.github.com/repos/simonw/datasette/issues/1552,996045776,IC_kwDOBm6k_c47XnPQ,9599,simonw,2021-12-16T17:52:54Z,2021-12-16T17:52:54Z,OWNER,"I tried that fix you suggested and now this `metadata.json` does the right thing:
```json
{
""databases"": {
""fixtures"": {
""tables"": {
""facetable"": {
""facets"": [
{
""array"": ""tags""
}
]
}
}
}
}
}
```
It does further highlight the bug in #625 though - since then if you try to add `?_facet=tags` to facet by tags treating them NOT as an array your request to do so is ignored.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`),
https://github.com/simonw/datasette/issues/1552#issuecomment-996034408,https://api.github.com/repos/simonw/datasette/issues/1552,996034408,IC_kwDOBm6k_c47Xkdo,9599,simonw,2021-12-16T17:37:37Z,2021-12-16T17:37:37Z,OWNER,"I think you're right! I had completely forgotten that piece of code.
This just turned into a bug fix and a documentation update. Thanks for the research!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`),
https://github.com/simonw/datasette/issues/1552#issuecomment-995296725,https://api.github.com/repos/simonw/datasette/issues/1552,995296725,IC_kwDOBm6k_c47UwXV,3556,davidbgk,2021-12-15T23:29:32Z,2021-12-15T23:29:32Z,CONTRIBUTOR,"@simonw thank you for your fast answer and your guidance!
While digging into the code, I found an undocumented way of doing it:
```yaml
facets: [""Facet for a column"", {""array"": ""Facet for an array""}]
```
The only remaining problem with that solution is here: https://github.com/simonw/datasette/blob/250db8192cb8aba5eb8cd301ccc2a49525bc3d24/datasette/facets.py#L33
We have:
```python
type, metadata_config = metadata_config.items()[0]
```
But it requires to cast the `dict_items` as a list prior to access the first element:
```python
type, metadata_config = list(metadata_config.items())[0]
```
I guess it's an unspotted bug? (I mean, independently of the facets-with-arrays issue.)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`),
https://github.com/simonw/datasette/issues/262#issuecomment-995034911,https://api.github.com/repos/simonw/datasette/issues/262,995034911,IC_kwDOBm6k_c47Twcf,9599,simonw,2021-12-15T18:03:46Z,2021-12-15T18:03:56Z,OWNER,"This is relevant to the big refactor in:
- #1518","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,Add ?_extra= mechanism for requesting extra properties in JSON,
https://github.com/simonw/datasette/issues/1552#issuecomment-995034143,https://api.github.com/repos/simonw/datasette/issues/1552,995034143,IC_kwDOBm6k_c47TwQf,9599,simonw,2021-12-15T18:02:53Z,2021-12-15T18:02:53Z,OWNER,"This is definitely a missing feature. The ""different types of facet"" stuff feels incomplete to me generally - this is one issue, but this one as well:
- #625","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`),
https://github.com/simonw/datasette/issues/1423#issuecomment-995023410,https://api.github.com/repos/simonw/datasette/issues/1423,995023410,IC_kwDOBm6k_c47Ttoy,9599,simonw,2021-12-15T17:48:40Z,2021-12-15T17:48:40Z,OWNER,You've caused me to rethink this feature - I no longer think there's value in only showing these numbers if `?_facet_size=max` as opposed to all of the time. New issue coming up.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 1, ""eyes"": 0}",962391325,Show count of facet values if ?_facet_size=max,
https://github.com/simonw/datasette/issues/1542#issuecomment-995022217,https://api.github.com/repos/simonw/datasette/issues/1542,995022217,IC_kwDOBm6k_c47TtWJ,9599,simonw,2021-12-15T17:47:07Z,2021-12-15T17:47:07Z,OWNER,"This does make sense to me. I've been hoping to significantly improve the way JavaScript plugins work - there are some notes on that here:
- #983
Encouraging plugins such as `datasette-cluster-map` to emit events that can then be listened to by other plugins is a really interesting idea that I hadn't considered.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1072106103,feature request: order and dependency of plugins (that use js),
https://github.com/simonw/datasette/issues/1518#issuecomment-994085710,https://api.github.com/repos/simonw/datasette/issues/1518,994085710,IC_kwDOBm6k_c47QItO,9599,simonw,2021-12-14T22:03:16Z,2021-12-14T22:04:28Z,OWNER,"There are actually four forms of SQL query used by the table page:
- `from_sql` - just the `from table_name where ...`
- `sql_no_order_no_limit` - used for faceting, `""select {select_all_columns} from {table_name} {where}""`
- `sql` - the above but with order and limit clauses: `""select {select_specified_columns} from {table_name} {where}{order_by} limit {page_size}{offset}""`
- `count_sql` used for the count, built out of `from_sql`: `""select count(*) {from_sql}""`
I'm tempted to encapsulate those in a `Query` class.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/issues/1518#issuecomment-994042389,https://api.github.com/repos/simonw/datasette/issues/1518,994042389,IC_kwDOBm6k_c47P-IV,9599,simonw,2021-12-14T21:35:53Z,2021-12-14T21:35:53Z,OWNER,"Maybe a better way to approach this would be to focus on the JSON side of things - try to get a basic JSON version with `?_extra=` support working, then eventually build that up to the point where it can power the HTML version.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/issues/621#issuecomment-994005634,https://api.github.com/repos/simonw/datasette/issues/621,994005634,IC_kwDOBm6k_c47P1KC,9599,simonw,2021-12-14T21:02:50Z,2021-12-14T21:02:50Z,OWNER,"This would also mean that an extra text input box could be easily shown on the page.
https://latest-with-plugins.datasette.io/fixtures/roadside_attractions?_through={""table"":""roadside_attraction_characteristics"",""column"":""characteristic_id"",""value"":""1""} but with the annotated box added (and made to look good):
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520681725,Syntax for ?_through= that works as a form field,
https://github.com/simonw/datasette/issues/621#issuecomment-993958242,https://api.github.com/repos/simonw/datasette/issues/621,993958242,IC_kwDOBm6k_c47Ppli,9599,simonw,2021-12-14T20:33:25Z,2021-12-14T20:33:56Z,OWNER,"Alternative idea: since current syntax is:
`?_through={""table"":""roadside_attraction_characteristics"",""column"":""characteristic_id"",""value"":""1""}`
The form-encoding-friendly syntax could be:
`?_through.{""table"":""roadside_attraction_characteristics"",""column"":""characteristic_id""}=1`
Which is more consistent than the array proposal:
`?_through.[""roadside_attraction_characteristics"",""characteristic_id""]=1`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520681725,Syntax for ?_through= that works as a form field,
https://github.com/simonw/datasette/issues/621#issuecomment-993813210,https://api.github.com/repos/simonw/datasette/issues/621,993813210,IC_kwDOBm6k_c47PGLa,9599,simonw,2021-12-14T17:30:13Z,2021-12-14T20:23:57Z,OWNER,"Might be able to create a web form that's unambiguous using:
`https://latest.datasette.io/fixtures/roadside_attractions?_through.[""roadside_attraction_characteristics"",""characteristic_id""]=1`
So:
```html
```
I'm pretty confident this is allowed by the HTML specification. This works:
```html
```
ASGI parsing seems to work too: https://latest-with-plugins.datasette.io/-/asgi-scope?_through.[%22roadside_attraction_characteristics%22%2C%22characteristic_id%22]=1","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520681725,Syntax for ?_through= that works as a form field,
https://github.com/simonw/datasette/issues/1423#issuecomment-993876599,https://api.github.com/repos/simonw/datasette/issues/1423,993876599,IC_kwDOBm6k_c47PVp3,6165713,plpxsk,2021-12-14T18:48:09Z,2021-12-14T18:48:09Z,NONE,"Great feature.
But what is the right way to enable this to show up? Currently, it seems I need to edit the URL to add, in the right place, `&_facet_size=max`
Is there another (easier) way to enable this feature?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",962391325,Show count of facet values if ?_facet_size=max,
https://github.com/simonw/datasette/issues/1518#issuecomment-993794247,https://api.github.com/repos/simonw/datasette/issues/1518,993794247,IC_kwDOBm6k_c47PBjH,9599,simonw,2021-12-14T17:09:40Z,2021-12-14T17:09:40Z,OWNER,- `table_actions` should be an extra.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/issues/1518#issuecomment-993000787,https://api.github.com/repos/simonw/datasette/issues/1518,993000787,IC_kwDOBm6k_c47L_1T,9599,simonw,2021-12-13T23:19:20Z,2021-12-14T17:06:05Z,OWNER,"Useful old comment here: https://github.com/simonw/datasette/issues/617#issuecomment-552253893
> As noted in [#621 (comment)](https://github.com/simonw/datasette/issues/621#issuecomment-552253208) a common pattern in this method is blocks of code that append new items to the `where_clauses`, `params` and `extra_human_descriptions` arrays. This is a useful refactoring opportunity.
>
> Code that fits this pattern:
>
> * The code that builds based on the filters: `where_clauses, params = filters.build_where_clauses(table)` and `human_description_en = filters.human_description_en(extra=extra_human_descriptions)`
> * Code that handles `?_where=`: `where_clauses.extend(request.args[""_where""])` - though note that this also appends to a `extra_wheres_for_ui` array which nothing else uses
> * The `_through=` code, see [Syntax for ?_through= that works as a form field #621](https://github.com/simonw/datasette/issues/621) for details
> * The code that deals with `?_search=` FTS
>
> The keyset pagination code modifies `where_clauses` and `params` too, but I don't think it's quite going to work with the same abstraction that would cover the above examples.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/issues/526#issuecomment-993078038,https://api.github.com/repos/simonw/datasette/issues/526,993078038,IC_kwDOBm6k_c47MSsW,536941,fgregg,2021-12-14T01:46:52Z,2021-12-14T01:46:52Z,CONTRIBUTOR,"the nested query idea is very nice, and i stole if for [my client side paginator](https://observablehq.com/d/1d5da3a3c3f2f347#DatasetteClient). However, it won't do the right thing if the original query orders by random().
If you go the nested query route, maybe raise a 4XX status code if the query has such a clause?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459882902,Stream all results for arbitrary SQL and canned queries,
https://github.com/simonw/datasette/issues/1553#issuecomment-993014772,https://api.github.com/repos/simonw/datasette/issues/1553,993014772,IC_kwDOBm6k_c47MDP0,536941,fgregg,2021-12-13T23:46:18Z,2021-12-13T23:46:18Z,CONTRIBUTOR,these headers would also be relevant for json exports of custom queries,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079111498,if csv export is truncated in non streaming mode set informative response header,
https://github.com/simonw/datasette/pull/1554#issuecomment-993006521,https://api.github.com/repos/simonw/datasette/issues/1554,993006521,IC_kwDOBm6k_c47MBO5,9599,simonw,2021-12-13T23:28:47Z,2021-12-13T23:28:47Z,OWNER,"That's frustrating: you can only attach comments to lines that were changed in the PR or are within about 3-4 lines of them:
![comments](https://user-images.githubusercontent.com/9599/145905357-5d8873f5-99c9-4b46-b4d5-35d38f5cb686.gif)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079129258,TableView refactor,
https://github.com/simonw/datasette/pull/1554#issuecomment-993002933,https://api.github.com/repos/simonw/datasette/issues/1554,993002933,IC_kwDOBm6k_c47MAW1,22429695,codecov[bot],2021-12-13T23:22:58Z,2021-12-13T23:22:58Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1554?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1554](https://codecov.io/gh/simonw/datasette/pull/1554?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (1d08b46) into [main](https://codecov.io/gh/simonw/datasette/commit/a6ff123de5464806441f6a6f95145c9a83b7f20b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (a6ff123) will **not change** coverage.
> The diff coverage is `n/a`.
[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1554/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1554?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
```diff
@@ Coverage Diff @@
## main #1554 +/- ##
=======================================
Coverage 91.84% 91.84%
=======================================
Files 34 34
Lines 4437 4437
=======================================
Hits 4075 4075
Misses 362 362
```
| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1554?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1554/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `96.04% <ø> (ø)` | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1554?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1554?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [a6ff123...1d08b46](https://codecov.io/gh/simonw/datasette/pull/1554?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079129258,TableView refactor,
https://github.com/simonw/datasette/issues/1553#issuecomment-992986587,https://api.github.com/repos/simonw/datasette/issues/1553,992986587,IC_kwDOBm6k_c47L8Xb,536941,fgregg,2021-12-13T22:57:04Z,2021-12-13T22:57:04Z,CONTRIBUTOR,would also be good if the header said the what the max row limit was,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079111498,if csv export is truncated in non streaming mode set informative response header,
https://github.com/simonw/datasette/issues/526#issuecomment-992971072,https://api.github.com/repos/simonw/datasette/issues/526,992971072,IC_kwDOBm6k_c47L4lA,536941,fgregg,2021-12-13T22:29:34Z,2021-12-13T22:29:34Z,CONTRIBUTOR,just came by to open this issue. would make my data analysis in observable a lot better!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459882902,Stream all results for arbitrary SQL and canned queries,
https://github.com/simonw/datasette/issues/1518#issuecomment-992833868,https://api.github.com/repos/simonw/datasette/issues/1518,992833868,IC_kwDOBm6k_c47LXFM,9599,simonw,2021-12-13T19:59:17Z,2021-12-13T19:59:17Z,OWNER,"Built a new plugin to help with this work by improving the display of `?_trace=1` output: https://datasette.io/plugins/datasette-pretty-traces
![image](https://user-images.githubusercontent.com/9599/145879751-36621f43-ba68-4ccd-b14b-379ed8f2111a.png)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/issues/1518#issuecomment-991978789,https://api.github.com/repos/simonw/datasette/issues/1518,991978789,IC_kwDOBm6k_c47IGUl,9599,simonw,2021-12-12T22:04:19Z,2021-12-12T22:04:19Z,OWNER,Idea: in JSON output include a `warnings` block listing any _ parameters that were not recognized.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/issues/1551#issuecomment-991960719,https://api.github.com/repos/simonw/datasette/issues/1551,991960719,IC_kwDOBm6k_c47IB6P,9599,simonw,2021-12-12T19:58:17Z,2021-12-12T19:58:17Z,OWNER,"Here's an example of the difference that causes:
```pycon
>>> import urllib.parse
>>> urllib.parse.parse_qs(""foo=bar"")
{'foo': ['bar']}
>>> urllib.parse.parse_qs(""foo=bar&baz="")
{'foo': ['bar']}
>>> urllib.parse.parse_qs(""foo=bar&baz="", keep_blank_values=True)
{'foo': ['bar'], 'baz': ['']}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077893013,`keep_blank_values=True` when parsing `request.args`,
https://github.com/simonw/datasette/issues/1551#issuecomment-991960416,https://api.github.com/repos/simonw/datasette/issues/1551,991960416,IC_kwDOBm6k_c47IB1g,9599,simonw,2021-12-12T19:56:12Z,2021-12-12T19:56:12Z,OWNER,"Python documentation for `parse_qs`: https://docs.python.org/3/library/urllib.parse.html#urllib.parse.parse_qs
> The optional argument *keep_blank_values* is a flag indicating whether blank values in percent-encoded queries should be treated as blank strings. A true value indicates that blanks should be retained as blank strings. The default false value indicates that blank values are to be ignored and treated as if they were not included.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077893013,`keep_blank_values=True` when parsing `request.args`,
https://github.com/simonw/datasette/issues/1551#issuecomment-991960179,https://api.github.com/repos/simonw/datasette/issues/1551,991960179,IC_kwDOBm6k_c47IBxz,9599,simonw,2021-12-12T19:54:45Z,2021-12-12T19:54:45Z,OWNER,This is technically a backwards-incompatible for any plugins that use `request.args` - but it's unlikely to break anything. At any rate this needs to happen before Datasette 1.0!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077893013,`keep_blank_values=True` when parsing `request.args`,
https://github.com/simonw/datasette/issues/1518#issuecomment-991828014,https://api.github.com/repos/simonw/datasette/issues/1518,991828014,IC_kwDOBm6k_c47Hhgu,9599,simonw,2021-12-12T03:21:35Z,2021-12-12T03:21:35Z,OWNER,"No, removing that gave me the following test failure:
```
tests/test_table_api.py::test_table_filter_queries[/fixtures/simple_primary_key.json?content__exact=-expected_rows2] FAILED [100%]
=============================================================================== FAILURES ================================================================================
______________________________________ test_table_filter_queries[/fixtures/simple_primary_key.json?content__exact=-expected_rows2] ______________________________________
app_client = , path = '/fixtures/simple_primary_key.json?content__exact=', expected_rows = [['3', '']]
@pytest.mark.parametrize(
""path,expected_rows"",
[
(""/fixtures/simple_primary_key.json?content=hello"", [[""1"", ""hello""]]),
(
""/fixtures/simple_primary_key.json?content__contains=o"",
[
[""1"", ""hello""],
[""2"", ""world""],
[""4"", ""RENDER_CELL_DEMO""],
],
),
(""/fixtures/simple_primary_key.json?content__exact="", [[""3"", """"]]),
(
""/fixtures/simple_primary_key.json?content__not=world"",
[
[""1"", ""hello""],
[""3"", """"],
[""4"", ""RENDER_CELL_DEMO""],
[""5"", ""RENDER_CELL_ASYNC""],
],
),
],
)
def test_table_filter_queries(app_client, path, expected_rows):
response = app_client.get(path)
> assert expected_rows == response.json[""rows""]
E AssertionError: assert [['3', '']] == [['1', 'hello'],\n ['2', 'world'],\n ['3', ''],\n ['4', 'RENDER_CELL_DEMO'],\n ['5', 'RENDER_CELL_ASYNC']]
E At index 0 diff: ['3', ''] != ['1', 'hello']
E Right contains 4 more items, first extra item: ['2', 'world']
E Full diff:
E [
E - ['1',
E - 'hello'],
E - ['2',
E - 'world'],
E ['3',
E ''],
E - ['4',
E - 'RENDER_CELL_DEMO'],
E - ['5',
E - 'RENDER_CELL_ASYNC'],
E ]
/Users/simon/Dropbox/Development/datasette/tests/test_table_api.py:511: AssertionError
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/issues/1518#issuecomment-991827468,https://api.github.com/repos/simonw/datasette/issues/1518,991827468,IC_kwDOBm6k_c47HhYM,9599,simonw,2021-12-12T03:15:00Z,2021-12-12T03:15:00Z,OWNER," I don't think this code is necessary any more: https://github.com/simonw/datasette/blob/492f9835aa7e90540dd0c6324282b109f73df71b/datasette/views/table.py#L396-L399
That dates back from when Datasette was built on top of Sanic and Sanic didn't preserve those query parameters the way I needed it to:
https://github.com/simonw/datasette/blob/1f69269fe93e4cd42e56890126cc0dbcf719c6cb/datasette/views/table.py#L202-L206","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/issues/1518#issuecomment-991823001,https://api.github.com/repos/simonw/datasette/issues/1518,991823001,IC_kwDOBm6k_c47HgSZ,9599,simonw,2021-12-12T02:25:32Z,2021-12-12T02:25:32Z,OWNER,The tests for `TableView` are currently mixed in with everything else in `tests/test_api.py` and `tests/html.py` - might be good to split those out into `test_table_html.py` and `test_table_api.py` since they're such a key part of how Datasette works.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/issues/1518#issuecomment-991822853,https://api.github.com/repos/simonw/datasette/issues/1518,991822853,IC_kwDOBm6k_c47HgQF,9599,simonw,2021-12-12T02:24:00Z,2021-12-12T02:24:00Z,OWNER,Rebuilding `TableView` from the ground up is proving not to be much fun. I'm going to explore starting the refactor of the existing code by separating out the bit that generates the SQL query from the rest of it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/issues/1518#issuecomment-991819781,https://api.github.com/repos/simonw/datasette/issues/1518,991819781,IC_kwDOBm6k_c47HfgF,9599,simonw,2021-12-12T01:53:10Z,2021-12-12T01:53:10Z,OWNER,"I have a hunch that the conclusion of this experiment may end up being that the `asyncinject` trick is kinda neat but the code will be easier to maintain (while still executing in parallel) if it's written using `asyncio.gather` directly instead.
It's possible `asyncinject` will end up being neat enough that I'll want to keep it though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/issues/1550#issuecomment-991805516,https://api.github.com/repos/simonw/datasette/issues/1550,991805516,IC_kwDOBm6k_c47HcBM,9599,simonw,2021-12-11T23:43:24Z,2021-12-11T23:43:24Z,OWNER,"I built a tiny Starlette app to experiment with this a bit:
```python
import asyncio
import janus
from starlette.applications import Starlette
from starlette.responses import JSONResponse, HTMLResponse, StreamingResponse
from starlette.routing import Route
import sqlite3
from concurrent import futures
executor = futures.ThreadPoolExecutor(max_workers=10)
async def homepage(request):
return HTMLResponse(
""""""
SQL CSV Server
SQL CSV Server
""""""
)
def run_query_in_thread(sql, sync_q):
db = sqlite3.connect(""../datasette/covid.db"")
cursor = db.cursor()
cursor.arraysize = 100 # Default is 1 apparently?
cursor.execute(sql)
columns = [d[0] for d in cursor.description]
sync_q.put([columns])
# Now start putting batches of rows
while True:
rows = cursor.fetchmany()
if rows:
sync_q.put(rows)
else:
break
# Let queue know we are finished\
sync_q.put(None)
async def csv_query(request):
sql = request.query_params[""sql""]
queue = janus.Queue()
loop = asyncio.get_running_loop()
async def csv_generator():
loop.run_in_executor(None, run_query_in_thread, sql, queue.sync_q)
while True:
rows = await queue.async_q.get()
if rows is not None:
for row in rows:
yield "","".join(map(str, row)) + ""\n ""
queue.async_q.task_done()
else:
# Cleanup
queue.close()
await queue.wait_closed()
break
return StreamingResponse(csv_generator(), media_type='text/plain')
app = Starlette(
debug=True,
routes=[
Route(""/"", homepage),
Route(""/csv"", csv_query),
],
)
```
But.. if I run this in a terminal window:
```
/tmp % wget 'http://127.0.0.1:8000/csv?sql=select+*+from+ny_times_us_counties'
```
it takes about 20 seconds to run and returns a 50MB file - but while it is running no other requests can be served by that server - not even the homepage! So something is blocking the event loop.
Maybe I should be using `fut = loop.run_in_executor(None, run_query_in_thread, sql, queue.sync_q)` and then awaiting `fut` somewhere, like in the Janus documentation? Don't think that's needed though. Needs more work to figure out why this is blocking.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077628073,Research option for returning all rows from arbitrary query,
https://github.com/simonw/datasette/issues/1550#issuecomment-991761635,https://api.github.com/repos/simonw/datasette/issues/1550,991761635,IC_kwDOBm6k_c47HRTj,9599,simonw,2021-12-11T19:39:01Z,2021-12-11T19:39:01Z,OWNER,"I wonder if this could work for public instances too with some kind of queuing mechanism?
I really need to use benchmarking to figure out what the right number of maximum SQLite connections is. I'm just guessing at the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077628073,Research option for returning all rows from arbitrary query,
https://github.com/simonw/datasette/issues/1549#issuecomment-991755245,https://api.github.com/repos/simonw/datasette/issues/1549,991755245,IC_kwDOBm6k_c47HPvt,9599,simonw,2021-12-11T19:17:54Z,2021-12-11T19:17:54Z,OWNER,"Also relevant:
- #1062 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077620955,Redesign CSV export to improve usability,
https://github.com/simonw/datasette/issues/617#issuecomment-991755013,https://api.github.com/repos/simonw/datasette/issues/617,991755013,IC_kwDOBm6k_c47HPsF,9599,simonw,2021-12-11T19:17:11Z,2021-12-11T19:17:11Z,OWNER,This work is now happening in #1518 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",519613116,Refactor TableView.data() method,
https://github.com/simonw/datasette/issues/1549#issuecomment-991754794,https://api.github.com/repos/simonw/datasette/issues/1549,991754794,IC_kwDOBm6k_c47HPoq,9599,simonw,2021-12-11T19:16:33Z,2021-12-11T19:16:33Z,OWNER,Good call! I'm doing a refactor #1518 right now which will hopefully bring the functionality of those two much closer - I'll make a note to consider this there too.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077620955,Redesign CSV export to improve usability,
https://github.com/simonw/datasette/issues/1549#issuecomment-991754237,https://api.github.com/repos/simonw/datasette/issues/1549,991754237,IC_kwDOBm6k_c47HPf9,536941,fgregg,2021-12-11T19:14:39Z,2021-12-11T19:14:39Z,CONTRIBUTOR,"that option is not available on [custom queries](https://labordata.bunkum.us/odpr-962a140?sql=with+local_union_filings+as+%28%0D%0A++select+*+from+lm_data+%0D%0A++where%0D%0A++++yr_covered+%3E+cast%28strftime%28%27%25Y%27%2C+%27now%27%2C+%27-5+years%27%29+as+int%29%0D%0A++++and+desig_name+%3D+%27LU%27%0D%0A++order+by+yr_covered+desc%0D%0A%29%2C%0D%0Amost_recent_filing+as+%28%0D%0A++select%0D%0A++++*%0D%0A++from+local_union_filings%0D%0A++group+by%0D%0A++++f_num%0D%0A%29%0D%0Aselect%0D%0A++*%0D%0Afrom%0D%0A++most_recent_filing%0D%0Awhere%0D%0A++next_election+%3E%3D+strftime%28%27%25Y-%25m%27%2C+%27now%27%29%0D%0A++and+next_election+%3C+strftime%28%27%25Y-%25m%27%2C+%27now%27%2C+%27%2B1+year%27%29%0D%0Aorder+by%0D%0A++members+desc%3B).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077620955,Redesign CSV export to improve usability,
https://github.com/simonw/datasette/issues/1549#issuecomment-991752486,https://api.github.com/repos/simonw/datasette/issues/1549,991752486,IC_kwDOBm6k_c47HPEm,9599,simonw,2021-12-11T19:09:15Z,2021-12-11T19:09:15Z,OWNER,"That's what this option does:
![EAB1B9E8-38E9-4C6D-8854-BD1935F163D9](https://user-images.githubusercontent.com/9599/145688531-668bafa1-e287-4bbd-84d6-157241fb1f68.jpeg)
The usability of this is pretty terrible though (including ""stream all rows"" - how are people meant to understand what that does?) so it can definitely do with some rethinking.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077620955,Redesign CSV export to improve usability,
https://github.com/simonw/sqlite-utils/issues/356#issuecomment-991517209,https://api.github.com/repos/simonw/sqlite-utils/issues/356,991517209,IC_kwDOCGYnMM47GVoZ,9599,simonw,2021-12-11T07:46:41Z,2021-12-11T07:46:41Z,OWNER,"By default this will accept single lines, but maybe there could be a `--all` option which instead grabs all of stdin into a single string against which the conversion function runs - like `git-history file`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option,
https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991405755,https://api.github.com/repos/simonw/sqlite-utils/issues/353,991405755,IC_kwDOCGYnMM47F6a7,536941,fgregg,2021-12-11T01:38:29Z,2021-12-11T01:38:29Z,CONTRIBUTOR,"wow! that's awesome! thanks so much, @simonw!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077102934,"Allow passing a file of code to ""sqlite-utils convert""",
https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991400016,https://api.github.com/repos/simonw/sqlite-utils/issues/353,991400016,IC_kwDOCGYnMM47F5BQ,9599,simonw,2021-12-11T01:10:52Z,2021-12-11T01:11:02Z,OWNER,"This won't be in a release for a little while, but you can install it to try it out using:
pip install https://github.com/simonw/sqlite-utils/archive/ee13f98c2c.zip","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077102934,"Allow passing a file of code to ""sqlite-utils convert""",
https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991399782,https://api.github.com/repos/simonw/sqlite-utils/issues/353,991399782,IC_kwDOCGYnMM47F49m,9599,simonw,2021-12-11T01:09:37Z,2021-12-11T01:09:37Z,OWNER,"OK, this is implemented. Updated documentation is here: https://sqlite-utils.datasette.io/en/latest/cli.html#converting-data-in-columns","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077102934,"Allow passing a file of code to ""sqlite-utils convert""",
https://github.com/simonw/sqlite-utils/issues/354#issuecomment-991399604,https://api.github.com/repos/simonw/sqlite-utils/issues/354,991399604,IC_kwDOCGYnMM47F460,9599,simonw,2021-12-11T01:08:46Z,2021-12-11T01:08:46Z,OWNER,That passed!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077243232,Test failure in test_rebuild_fts,
https://github.com/simonw/sqlite-utils/issues/354#issuecomment-991398367,https://api.github.com/repos/simonw/sqlite-utils/issues/354,991398367,IC_kwDOCGYnMM47F4nf,9599,simonw,2021-12-11T01:03:14Z,2021-12-11T01:03:14Z,OWNER,The new test: https://github.com/simonw/sqlite-utils/blob/ee13f98c2c7ca3b819bd0fc55da3108cb6a6434a/tests/test_fts.py#L270-L277,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077243232,Test failure in test_rebuild_fts,
https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982123183,https://api.github.com/repos/simonw/sqlite-utils/issues/347,982123183,IC_kwDOCGYnMM46igKv,22429695,codecov[bot],2021-11-29T23:20:35Z,2021-12-11T01:02:19Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/347?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#347](https://codecov.io/gh/simonw/sqlite-utils/pull/347?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (71b6c38) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/213a0ff177f23a35f3b235386366ff132eb879f1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (213a0ff) will **increase** coverage by `0.00%`.
> The diff coverage is `100.00%`.
> :exclamation: Current head 71b6c38 differs from pull request most recent head 1a7ef2f. Consider uploading reports for the commit 1a7ef2f to get more accurate results
[![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/347/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/347?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
```diff
@@ Coverage Diff @@
## main #347 +/- ##
=======================================
Coverage 96.51% 96.52%
=======================================
Files 5 5
Lines 2270 2271 +1
=======================================
+ Hits 2191 2192 +1
Misses 79 79
```
| [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/347?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [sqlite\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/347/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.73% <100.00%> (ø)` | |
| [sqlite\_utils/utils.py](https://codecov.io/gh/simonw/sqlite-utils/pull/347/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `93.68% <100.00%> (+0.03%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/347?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/347?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [213a0ff...1a7ef2f](https://codecov.io/gh/simonw/sqlite-utils/pull/347?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066603133,Test against pysqlite3 running SQLite 3.37,
https://github.com/simonw/sqlite-utils/pull/347#issuecomment-991397907,https://api.github.com/repos/simonw/sqlite-utils/issues/347,991397907,IC_kwDOCGYnMM47F4gT,9599,simonw,2021-12-11T01:01:40Z,2021-12-11T01:01:40Z,OWNER,The change I made to that test in #354 might help with this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066603133,Test against pysqlite3 running SQLite 3.37,
https://github.com/simonw/sqlite-utils/issues/354#issuecomment-991395919,https://api.github.com/repos/simonw/sqlite-utils/issues/354,991395919,IC_kwDOCGYnMM47F4BP,9599,simonw,2021-12-11T00:52:31Z,2021-12-11T00:52:31Z,OWNER,"It turns out `rebuild` does indeed work against content tables, so I can put that in the test instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077243232,Test failure in test_rebuild_fts,
https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991395494,https://api.github.com/repos/simonw/sqlite-utils/issues/355,991395494,IC_kwDOCGYnMM47F36m,9599,simonw,2021-12-11T00:50:22Z,2021-12-11T00:51:15Z,OWNER,"Here's an example of the new (slightly confusing) error message:
```bash
sqlite-utils convert fixtures.db roadside_attractions name '
def foo(value)
bar
baz
'
Error: Syntax error in code:
def foo(value)
invalid syntax
```
Another:
```
sqlite-utils convert fixtures.db roadside_attractions name '$'
Error: Syntax error in code:
return $
invalid syntax
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077322009,Allow users to pass a full convert() function definition,
https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991393684,https://api.github.com/repos/simonw/sqlite-utils/issues/355,991393684,IC_kwDOCGYnMM47F3eU,9599,simonw,2021-12-11T00:42:19Z,2021-12-11T00:49:49Z,OWNER,"Ideally I'd like to show the perfect syntax error messages to the user - but I don't know if it's possible to do this cleanly because the error might occur with their originally entered code OR it might occur after I add `def fn(value)` to it.
I'm going to punt on that for the moment and tolerate slightly confusing syntax errors.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077322009,Allow users to pass a full convert() function definition,
https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991386841,https://api.github.com/repos/simonw/sqlite-utils/issues/355,991386841,IC_kwDOCGYnMM47F1zZ,9599,simonw,2021-12-11T00:14:11Z,2021-12-11T00:15:15Z,OWNER,"Relevant code: https://github.com/simonw/sqlite-utils/blob/7a43af232e4bc00bd227307665163614e225948b/sqlite_utils/cli.py#L2128-L2135
One way to implement this would be to look to see if the code starts with `def ...` - but that's not going to work for proper module that start with a docstring or imports.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077322009,Allow users to pass a full convert() function definition,
https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991387044,https://api.github.com/repos/simonw/sqlite-utils/issues/355,991387044,IC_kwDOCGYnMM47F12k,9599,simonw,2021-12-11T00:14:45Z,2021-12-11T00:14:45Z,OWNER,"Maybe attempt to compile their code, and if it fails try again after adding `def fn(value):` to the start?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077322009,Allow users to pass a full convert() function definition,
https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991381679,https://api.github.com/repos/simonw/sqlite-utils/issues/353,991381679,IC_kwDOCGYnMM47F0iv,9599,simonw,2021-12-10T23:58:43Z,2021-12-10T23:59:35Z,OWNER,"I think the fix for this is to change the rules about what code is accepted in both the `-` mode and the literal code string mode: you can pass in a Python expression, OR a fragment that gets turned into a function, OR code that implements its own `def convert(value)` function. So this would work too:
```sh
sqlite-utils convert my.db mytable col1 '
def convert(value):
return value.upper()
'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077102934,"Allow passing a file of code to ""sqlite-utils convert""",
https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991381281,https://api.github.com/repos/simonw/sqlite-utils/issues/353,991381281,IC_kwDOCGYnMM47F0ch,9599,simonw,2021-12-10T23:57:26Z,2021-12-10T23:57:26Z,OWNER,"My first attempt at building this looked a little bit strange, because you would end up having a file like this `convert.py`:
```
value = value.upper()
return value
```
Which gets used like this:
cat convert.py | sqlite-utils convert my.db mytable col1 -
But... that `convert.py` code isn't actually valid Python - it's a weird thing where you have a partial snippet of Python code that gets wrapped in a function automatically.
It would be better if you could write `convert.py` as a valid Python file with a function in it, something like this:
```python
def convert(value):
value = value.upper()
return value
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077102934,"Allow passing a file of code to ""sqlite-utils convert""",
https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991378346,https://api.github.com/repos/simonw/sqlite-utils/issues/353,991378346,IC_kwDOCGYnMM47Fzuq,9599,simonw,2021-12-10T23:48:28Z,2021-12-10T23:48:28Z,OWNER,"One option: allow `CODE` to be a special value of `-` which means ""read from standard input"". It's a tiny bit of a hack but I think it would work here.
If you wanted to replace a column entirely with hyphens you would still be able to do this:
sqlite-utils convert my.db mytable col1 '""-""'","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077102934,"Allow passing a file of code to ""sqlite-utils convert""",
https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991377288,https://api.github.com/repos/simonw/sqlite-utils/issues/353,991377288,IC_kwDOCGYnMM47FzeI,9599,simonw,2021-12-10T23:45:53Z,2021-12-10T23:45:53Z,OWNER,"One challenge here: the current signature looks like this:
```
% sqlite-utils convert --help
Usage: sqlite-utils convert [OPTIONS] DB_PATH TABLE COLUMNS... CODE
```
`CODE` is a positional argument which comes last - and since `COLUMNS` can be one or more items, making `CODE` optional isn't easy.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077102934,"Allow passing a file of code to ""sqlite-utils convert""",
https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991376639,https://api.github.com/repos/simonw/sqlite-utils/issues/353,991376639,IC_kwDOCGYnMM47FzT_,9599,simonw,2021-12-10T23:43:45Z,2021-12-10T23:43:45Z,OWNER,"There's a very non-obvious workaround for this at the moment. You can save your code in e.g. a file called` transform.py` - my test one looks like this:
```python
def upper(value):
return value.upper()
```
Then you can run the following to import and use that function:
`PYTHONPATH=. sqlite-utils convert fixtures.db roadside_attractions name 'transform.upper(value)' --import transform`
That `PYTHONPATH=. bit is necessary because otherwise the script won't look in the current directory for that `transform.py` module.
Now that I've written this down, it's obviously bad! I think your suggestion here is a good idea.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077102934,"Allow passing a file of code to ""sqlite-utils convert""",
https://github.com/simonw/sqlite-utils/issues/354#issuecomment-991309759,https://api.github.com/repos/simonw/sqlite-utils/issues/354,991309759,IC_kwDOCGYnMM47Fi-_,9599,simonw,2021-12-10T21:33:18Z,2021-12-10T21:33:18Z,OWNER,"https://www.sqlite.org/fts5.html#the_rebuild_command says:
> This command first deletes the entire full-text index, then rebuilds it based on the contents of the table or [content table](https://www.sqlite.org/fts5.html#external_content_tables). It is not available with [contentless tables](https://www.sqlite.org/fts5.html#contentless_tables).
>
> `INSERT INTO ft(ft) VALUES('rebuild');`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077243232,Test failure in test_rebuild_fts,
https://github.com/simonw/sqlite-utils/issues/354#issuecomment-991309002,https://api.github.com/repos/simonw/sqlite-utils/issues/354,991309002,IC_kwDOCGYnMM47FizK,9599,simonw,2021-12-10T21:32:14Z,2021-12-10T21:32:14Z,OWNER,"Here's what the method does: https://github.com/simonw/sqlite-utils/blob/e328db8eba1fbf29a69eda95dfec861954f9e771/sqlite_utils/db.py#L1941-L1952
Maybe I don't need a test that deliberately corrupts the database here? Not sure how to test that `rebuild` has been called though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077243232,Test failure in test_rebuild_fts,
https://github.com/simonw/sqlite-utils/issues/354#issuecomment-991307422,https://api.github.com/repos/simonw/sqlite-utils/issues/354,991307422,IC_kwDOCGYnMM47Fiae,9599,simonw,2021-12-10T21:29:34Z,2021-12-10T21:29:34Z,OWNER,Here's the test in question. The way it works is a bit weird (deleting everything in the `_fts_data` table in order to force errors that can be fixed with `.rebuild_fts()`): https://github.com/simonw/sqlite-utils/blob/8ae77a6961fed94ef2c9cc81fcfc7c81d222d9a2/tests/test_fts.py#L257-L285,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077243232,Test failure in test_rebuild_fts,
https://github.com/simonw/sqlite-utils/issues/354#issuecomment-991306712,https://api.github.com/repos/simonw/sqlite-utils/issues/354,991306712,IC_kwDOCGYnMM47FiPY,9599,simonw,2021-12-10T21:28:27Z,2021-12-10T21:28:27Z,OWNER,"Failures started with this commit, which only touches documentation so is completely unrelated: https://github.com/simonw/sqlite-utils/commit/e328db8eba1fbf29a69eda95dfec861954f9e771","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077243232,Test failure in test_rebuild_fts,
https://github.com/simonw/datasette/issues/1518#issuecomment-991285527,https://api.github.com/repos/simonw/datasette/issues/1518,991285527,IC_kwDOBm6k_c47FdEX,9599,simonw,2021-12-10T20:52:00Z,2021-12-10T20:52:00Z,OWNER,"If I break this up into `@inject` methods, what methods could I have and what would they do?
- `resolve_path`: Use request path to resolve the database and table. Could handle hash URLs too (if I don't manage to extract those to a plugin) - would be nice if this could raise a redirect, but I think that will instead have to be one of the things it returns
- `build_sql`: Builds the SQL query based on the querystring (and some DB introspection)
- `execute_count`: Execute the `count(*)`
- `execute_rows`: Execute the `limit 101` to fetch the rows
- `execute_facets`: Execute all requested facets (could this do its own `asyncio.gather()` to run facets in parallel?)
- `suggest_facets`: Execute facet suggestions
Are there any plugin hooks that would make sense to execute in parallel? Actually there might be: I don't think `extra_template_vars`, `extra_css_urls`, `extra_js_urls`, `extra_body_script` depend on each other so it might be possible to execute them in a parallel chunk (at least any of them that return awaitables).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/pull/1548#issuecomment-990967417,https://api.github.com/repos/simonw/datasette/issues/1548,990967417,IC_kwDOBm6k_c47EPZ5,22429695,codecov[bot],2021-12-10T13:19:00Z,2021-12-10T13:19:00Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1548?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1548](https://codecov.io/gh/simonw/datasette/pull/1548?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (68383f5) into [main](https://codecov.io/gh/simonw/datasette/commit/737115ea14cd51ffb55dea886e6a684c148db2c9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (737115e) will **not change** coverage.
> The diff coverage is `n/a`.
[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1548/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1548?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
```diff
@@ Coverage Diff @@
## main #1548 +/- ##
=======================================
Coverage 91.84% 91.84%
=======================================
Files 34 34
Lines 4438 4438
=======================================
Hits 4076 4076
Misses 362 362
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1548?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1548?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [737115e...68383f5](https://codecov.io/gh/simonw/datasette/pull/1548?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1076834768,"Update pytest-xdist requirement from <2.5,>=2.2.1 to >=2.2.1,<2.6",
https://github.com/simonw/datasette/issues/1540#issuecomment-984801331,https://api.github.com/repos/simonw/datasette/issues/1540,984801331,IC_kwDOBm6k_c46suAz,9599,simonw,2021-12-02T16:42:02Z,2021-12-09T23:38:39Z,OWNER,"I'm going to wrap this up in a plugin for the moment - I want it in Datasette core but I'd like to improve the implementation first with things like support for `base_url` which will likely depend on #1533 or similar.
Here's the plugin: https://github.com/simonw/datasette-hovercards","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1068791148,Idea: hover to reveal details of linked row,
https://github.com/simonw/datasette/issues/1528#issuecomment-988468238,https://api.github.com/repos/simonw/datasette/issues/1528,988468238,IC_kwDOBm6k_c466tQO,30934,20after4,2021-12-08T03:35:45Z,2021-12-08T03:35:45Z,NONE,"FWIW I implemented something similar with a bit of plugin code:
```python
@hookimpl
def canned_queries(datasette: Datasette, database: str) -> Mapping[str, str]:
# load ""canned queries"" from the filesystem under
# www/sql/db/query_name.sql
queries = {}
sqldir = Path(__file__).parent.parent / ""sql""
if database:
sqldir = sqldir / database
if not sqldir.is_dir():
return queries
for f in sqldir.glob('*.sql'):
try:
sql = f.read_text('utf8').strip()
if not len(sql):
log(f""Skipping empty canned query file: {f}"")
continue
queries[f.stem] = { ""sql"": sql }
except OSError as err:
log(err)
return queries
```","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1060631257,"Add new `""sql_file""` key to Canned Queries in metadata?",
https://github.com/simonw/datasette/issues/1304#issuecomment-988463455,https://api.github.com/repos/simonw/datasette/issues/1304,988463455,IC_kwDOBm6k_c466sFf,30934,20after4,2021-12-08T03:23:14Z,2021-12-08T03:23:14Z,NONE,I actually think it would be a useful thing to add support for in datasette. It wouldn't be difficult to unwind an array of params and add the placeholders automatically.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",863884805,"Document how to send multiple values for ""Named parameters"" ",
https://github.com/simonw/datasette/issues/1304#issuecomment-988461884,https://api.github.com/repos/simonw/datasette/issues/1304,988461884,IC_kwDOBm6k_c466rs8,30934,20after4,2021-12-08T03:20:26Z,2021-12-08T03:20:26Z,NONE,"The easiest or most straightforward thing to do is to use named parameters like:
```sql
select * where key IN (:p1, :p2, :p3)
```
And simply construct the list of placeholders dynamically based on the number of values.
Doing this is possible with datasette if you forgo ""canned queries"" and just use the raw query endpoint and pass the query sql, along with p1, p2 ... in the request.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",863884805,"Document how to send multiple values for ""Named parameters"" ",
https://github.com/simonw/datasette/issues/1304#issuecomment-988459453,https://api.github.com/repos/simonw/datasette/issues/1304,988459453,IC_kwDOBm6k_c466rG9,9308268,rayvoelker,2021-12-08T03:15:27Z,2021-12-08T03:15:27Z,NONE,"I was thinking if there were a way to use some sort of sting function to ""unpack"" the values and convert them into ints... hm","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",863884805,"Document how to send multiple values for ""Named parameters"" ",
https://github.com/simonw/datasette/issues/1544#issuecomment-988226938,https://api.github.com/repos/simonw/datasette/issues/1544,988226938,IC_kwDOBm6k_c465yV6,9599,simonw,2021-12-07T20:02:44Z,2021-12-07T20:02:44Z,OWNER,I'm feeling rushed today so I'm going to fix this without adding a test!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1073712378,Code that detects the label column for a table is case-sensitive,
https://github.com/simonw/datasette/issues/1544#issuecomment-988226523,https://api.github.com/repos/simonw/datasette/issues/1544,988226523,IC_kwDOBm6k_c465yPb,9599,simonw,2021-12-07T20:02:00Z,2021-12-07T20:02:00Z,OWNER,Here's the code at fault: https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/database.py#L288-L291,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1073712378,Code that detects the label column for a table is case-sensitive,
https://github.com/simonw/datasette/issues/1527#issuecomment-988154238,https://api.github.com/repos/simonw/datasette/issues/1527,988154238,IC_kwDOBm6k_c465gl-,9599,simonw,2021-12-07T18:05:26Z,2021-12-07T18:05:26Z,OWNER,"Found a new case of this bug: click the ""Apply"" button on https://latest.datasette.io/fixtures/facetable?_sort=pk&_city_id__gt=1
![apply-bug](https://user-images.githubusercontent.com/9599/145082760-6947c769-480f-45c7-9916-b6cc7f5834f8.gif)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059555791,Columns starting with an underscore behave poorly in filters,
https://github.com/simonw/sqlite-utils/issues/349#issuecomment-987461427,https://api.github.com/repos/simonw/sqlite-utils/issues/349,987461427,IC_kwDOCGYnMM4623cz,9599,simonw,2021-12-07T01:03:43Z,2021-12-07T01:04:37Z,OWNER,"In terms of types, I think that means it looks like this:
```python
IndexesType = Iterable[
Union[str, Iterable[str]]
]
def create(
self,
columns: Dict[str, Any],
pk: Optional[Any] = None,
...
indexes: Optional[IndexesType] = None,
):
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1071531082,A way of creating indexes on newly created tables,
https://github.com/simonw/sqlite-utils/issues/349#issuecomment-987458772,https://api.github.com/repos/simonw/sqlite-utils/issues/349,987458772,IC_kwDOCGYnMM4622zU,9599,simonw,2021-12-07T01:00:41Z,2021-12-07T01:00:41Z,OWNER,"I think the syntax design of this looks like:
```python
item_pk = db[item_table].lookup(
{""_item_id"": item_id},
item_to_insert,
column_order=(""_id"", ""_item_id""),
pk=""_id"",
indexes=(""_version"",),
)
```
So it's a sequence of column names... or a sequence of tuples for creating compound indexes:
```python
db[""dogs""].insert(
{""name"": ""Cleo"", ""species"": ""Mutt"", ""hobbies"": ""Raiding picnics""},
indexes=((""name"", ""species""), ""hobbies""),
)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1071531082,A way of creating indexes on newly created tables,
https://github.com/simonw/sqlite-utils/issues/352#issuecomment-987454872,https://api.github.com/repos/simonw/sqlite-utils/issues/352,987454872,IC_kwDOCGYnMM46212Y,9599,simonw,2021-12-07T00:56:29Z,2021-12-07T00:56:29Z,OWNER,"Thought about this due to this issue, which should stay consistent with how the `sqlite-utils` CLI works:
- https://github.com/simonw/git-history/issues/41","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1072792507,`sqlite-utils insert --extract colname`,
https://github.com/simonw/sqlite-utils/issues/351#issuecomment-987437043,https://api.github.com/repos/simonw/sqlite-utils/issues/351,987437043,IC_kwDOCGYnMM462xfz,9599,simonw,2021-12-07T00:41:02Z,2021-12-07T00:41:56Z,OWNER,"The fix there was:
```diff
@@ -463,7 +473,7 @@ def compile_convert(convert, imports):
locals = {}
globals = {""json"": json}
for import_ in imports:
- globals[import_] = __import__(import_)
+ globals[import_.split(""."")[0]] = __import__(import_)
exec(code_o, globals, locals)
```
With this example (needs to be modified here):
```
git-history file items.xml --convert '
tree = xml.etree.ElementTree.fromstring(content)
return [el.attrib for el in tree.iter(""item"")]
' --import xml.etree.ElementTree
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1072780607,Support `--import xml.etree.ElementTree` in `sqlite-utils convert`,
https://github.com/simonw/sqlite-utils/issues/349#issuecomment-987349633,https://api.github.com/repos/simonw/sqlite-utils/issues/349,987349633,IC_kwDOCGYnMM462cKB,9599,simonw,2021-12-06T23:19:28Z,2021-12-06T23:19:28Z,OWNER,(I ended up not needing this here since `.lookup()` already creates a unique index on `_item_id` for you. Still could be a useful feature though.),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1071531082,A way of creating indexes on newly created tables,
https://github.com/simonw/sqlite-utils/issues/350#issuecomment-987016019,https://api.github.com/repos/simonw/sqlite-utils/issues/350,987016019,IC_kwDOCGYnMM461KtT,9599,simonw,2021-12-06T17:56:57Z,2021-12-06T17:56:57Z,OWNER,"Would be interesting to micro-benchmark this to get an idea for how much of a performance boost it is, since the indexed SQLite lookups used by `table.lookup()` should be really fast already.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1072435124,Optional caching mechanism for table.lookup(),
https://github.com/simonw/sqlite-utils/issues/350#issuecomment-987015327,https://api.github.com/repos/simonw/sqlite-utils/issues/350,987015327,IC_kwDOCGYnMM461Kif,9599,simonw,2021-12-06T17:56:05Z,2021-12-06T17:56:05Z,OWNER,Should I implement this remember to apply the optimization in `git-history`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1072435124,Optional caching mechanism for table.lookup(),
https://github.com/simonw/sqlite-utils/issues/350#issuecomment-987015063,https://api.github.com/repos/simonw/sqlite-utils/issues/350,987015063,IC_kwDOCGYnMM461KeX,9599,simonw,2021-12-06T17:55:42Z,2021-12-06T17:55:42Z,OWNER,"API could be this:
```python
id = db[""columns""].lookup(
{""namespace"": namespace_id, ""name"": column},
cache=True
)
```
This could default to a 100 item LRU cache. You could perhaps modify that with `cache_size=500` or with `cache_size=None` to disable the size limit on that cache.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1072435124,Optional caching mechanism for table.lookup(),
https://github.com/simonw/datasette/pull/1543#issuecomment-986768401,https://api.github.com/repos/simonw/datasette/issues/1543,986768401,IC_kwDOBm6k_c460OQR,22429695,codecov[bot],2021-12-06T13:18:48Z,2021-12-06T13:18:48Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1543?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1543](https://codecov.io/gh/simonw/datasette/pull/1543?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (353a851) into [main](https://codecov.io/gh/simonw/datasette/commit/7c02be2ee94cc64b120cc58b7a72cd387031f287?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (7c02be2) will **not change** coverage.
> The diff coverage is `n/a`.
[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1543/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1543?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
```diff
@@ Coverage Diff @@
## main #1543 +/- ##
=======================================
Coverage 91.84% 91.84%
=======================================
Files 34 34
Lines 4438 4438
=======================================
Hits 4076 4076
Misses 362 362
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1543?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1543?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [7c02be2...353a851](https://codecov.io/gh/simonw/datasette/pull/1543?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1072135269,Bump black from 21.11b1 to 21.12b0,
https://github.com/simonw/datasette/issues/1426#issuecomment-985982668,https://api.github.com/repos/simonw/datasette/issues/1426,985982668,IC_kwDOBm6k_c46xObM,95520595,knowledgecamp12,2021-12-04T07:11:29Z,2021-12-04T07:11:29Z,NONE,You can generate xml site map from the online tools using https://tools4seo.site/xml-sitemap-generator. ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",964322136,"Manage /robots.txt in Datasette core, block robots by default",
https://github.com/dogsheep/github-to-sqlite/issues/69#issuecomment-985928838,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/69,985928838,IC_kwDODFdgUs46xBSG,9599,simonw,2021-12-04T00:34:52Z,2021-12-04T00:34:52Z,MEMBER,"First attempt at this:
```sql
select 'issues' as ""table"", id, node_id, title, user, created_at, body, repo
from issues
union all
select 'issue_comments' as ""table"", issue_comments.id, issue_comments.node_id, '' as title, issue_comments.user, issue_comments.created_at, issue_comments.body, issues.repo
from issue_comments join issues on issues.id = issue_comments.issue
order by created_at desc
```
https://github-to-sqlite.dogsheep.net/github?sql=select+%27issues%27+as+%22table%22%2C+id%2C+node_id%2C+title%2C+user%2C+created_at%2C+body%2C+repo%0D%0Afrom+issues%0D%0Aunion+all%0D%0Aselect+%27issue_comments%27+as+%22table%22%2C+issue_comments.id%2C+issue_comments.node_id%2C+%27%27+as+title%2C+issue_comments.user%2C+issue_comments.created_at%2C+issue_comments.body%2C+issues.repo%0D%0Afrom+issue_comments+join+issues+on+issues.id+%3D+issue_comments.issue%0D%0Aorder+by+created_at+desc","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1071071397,View that combines issues and issue comments,
https://github.com/simonw/datasette/issues/1541#issuecomment-984908185,https://api.github.com/repos/simonw/datasette/issues/1541,984908185,IC_kwDOBm6k_c46tIGZ,9599,simonw,2021-12-02T18:56:54Z,2021-12-02T18:56:54Z,OWNER,Also it should link to foreign keys like the table page does.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1069881276,Different default layout for row page,
https://github.com/simonw/datasette/issues/1175#issuecomment-984569477,https://api.github.com/repos/simonw/datasette/issues/1175,984569477,IC_kwDOBm6k_c46r1aF,24821294,AnkitKundariya,2021-12-02T12:09:30Z,2021-12-02T12:09:30Z,NONE,"@hannseman
I have tried the above suggestion given by you but somehow I'm getting the below error.
_note : I'm running my application with Docker._
`app_1 | {""event"": ""Exception in ASGI application\n"", ""exc_info"": ["""", ""RuntimeError('no running event loop')"", """"], ""logger"": ""uvicorn.error"", ""level"": ""error"", ""timestamp"": ""2021-12-02T12:06:36.011448Z""}
`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779156520,Use structlog for logging,
https://github.com/simonw/datasette/issues/1540#issuecomment-984053760,https://api.github.com/repos/simonw/datasette/issues/1540,984053760,IC_kwDOBm6k_c46p3gA,9599,simonw,2021-12-01T21:05:20Z,2021-12-01T21:05:20Z,OWNER,"I realized you couldn't click the links any more because the hovercard overlapped them, so I changed it to this instead. Need to reconsider the when-to-hide logic though.
```javascript
hovercard.style.top = (ev.pageY + 15) + 'px';
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1068791148,Idea: hover to reveal details of linked row,
https://github.com/simonw/datasette/issues/1540#issuecomment-984051925,https://api.github.com/repos/simonw/datasette/issues/1540,984051925,IC_kwDOBm6k_c46p3DV,9599,simonw,2021-12-01T21:03:16Z,2021-12-01T21:03:16Z,OWNER,Needs `pageX` not `clientX` because otherwise it doesn't work when you scroll down the page.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1068791148,Idea: hover to reveal details of linked row,
https://github.com/simonw/datasette/issues/1540#issuecomment-984048965,https://api.github.com/repos/simonw/datasette/issues/1540,984048965,IC_kwDOBm6k_c46p2VF,9599,simonw,2021-12-01T20:59:26Z,2021-12-01T21:02:58Z,OWNER,"This is a bit of a mess but it does keep the hovercard around for a moment and then fade it away when you mouse out of it:
```html+jinja
{% extends ""base.html"" %}
{% block content %}
Hovercards demo
Here is a link to a row
{% endblock %}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1068791148,Idea: hover to reveal details of linked row,
https://github.com/simonw/datasette/issues/1540#issuecomment-984037711,https://api.github.com/repos/simonw/datasette/issues/1540,984037711,IC_kwDOBm6k_c46pzlP,9599,simonw,2021-12-01T20:42:17Z,2021-12-01T20:43:14Z,OWNER,"A first prototype (saved as `templates/pages/hovercard.html` and run with `datasette fixtures.db --template-dir=templates`):
```html+jinja
{% extends ""base.html"" %}
{% block content %}
Hovercards demo
Here is a link to a row
{% endblock %}
```
![hovercard](https://user-images.githubusercontent.com/9599/144310888-6db71bad-b6f6-4d8a-a737-81a618022bbe.gif)
Lots of decisions to make here. Most importantly, when should it be hidden again?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1068791148,Idea: hover to reveal details of linked row,
https://github.com/simonw/datasette/issues/1540#issuecomment-983985330,https://api.github.com/repos/simonw/datasette/issues/1540,983985330,IC_kwDOBm6k_c46pmyy,9599,simonw,2021-12-01T19:29:05Z,2021-12-01T19:29:05Z,OWNER,"The layout of the hover card could be similar to the one used by `datasette-cluster-map`:
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1068791148,Idea: hover to reveal details of linked row,
https://github.com/simonw/datasette/issues/1519#issuecomment-983890815,https://api.github.com/repos/simonw/datasette/issues/1519,983890815,IC_kwDOBm6k_c46pPt_,157158,phubbard,2021-12-01T17:50:09Z,2021-12-01T17:50:09Z,NONE,"thanks so very much for the prompt attention and fix! Plus, the animated GIF showing the bug is just extra and I love it. Interactions like this are why I love open source.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views,
https://github.com/simonw/sqlite-utils/issues/348#issuecomment-983155079,https://api.github.com/repos/simonw/sqlite-utils/issues/348,983155079,IC_kwDOCGYnMM46mcGH,25778,eyeseast,2021-12-01T00:28:40Z,2021-12-01T00:28:40Z,CONTRIBUTOR,"I'd use this. Right now, I tend to do `touch my.db` and then `enable-wal` or whatever else, but I'm never sure if that's a bad idea.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1067771698,Command for creating an empty database,
https://github.com/simonw/sqlite-utils/issues/348#issuecomment-983122733,https://api.github.com/repos/simonw/sqlite-utils/issues/348,983122733,IC_kwDOCGYnMM46mUMt,9599,simonw,2021-11-30T23:31:17Z,2021-11-30T23:31:17Z,OWNER,"Potential other options could include:
- `--page-size` https://www.sqlite.org/pragma.html#pragma_page_size","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1067771698,Command for creating an empty database,
https://github.com/simonw/sqlite-utils/issues/348#issuecomment-983120066,https://api.github.com/repos/simonw/sqlite-utils/issues/348,983120066,IC_kwDOCGYnMM46mTjC,9599,simonw,2021-11-30T23:25:52Z,2021-11-30T23:26:11Z,OWNER,"Maybe this:
sqlite-utils create-database my.db
With options that include `--enable-wal` (actually that's the only option I can think of).
This is consistent with the existing `create-table` and `create-view` commands.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1067771698,Command for creating an empty database,
https://github.com/simonw/datasette/issues/1532#issuecomment-982745406,https://api.github.com/repos/simonw/datasette/issues/1532,982745406,IC_kwDOBm6k_c46k4E-,30934,20after4,2021-11-30T15:28:57Z,2021-11-30T15:28:57Z,NONE,"It's a really great API and the documentation is really great too. Honestly, in more than 20 years of professional experience, I haven't worked with any software API that was more of a joy to use. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065429936,Use datasette-table Web Component to guide the design of the JSON API for 1.0,
https://github.com/simonw/datasette/issues/1525#issuecomment-982331602,https://api.github.com/repos/simonw/datasette/issues/1525,982331602,IC_kwDOBm6k_c46jTDS,9599,simonw,2021-11-30T06:39:00Z,2021-11-30T06:39:00Z,OWNER,"These two pages now help demonstrate the fix:
- https://latest.datasette.io/fixtures/facet_cities/1
- https://latest.datasette.io/fixtures/attraction_characteristic/2
I added a new test for these here:
https://github.com/simonw/datasette/blob/35b12746ba2bf9f254791bddac03d25b19be9b77/tests/test_html.py#L823-L848
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059509927,"""Links from other tables"" broken for columns starting with underscore",
https://github.com/simonw/datasette/issues/1532#issuecomment-982319210,https://api.github.com/repos/simonw/datasette/issues/1532,982319210,IC_kwDOBm6k_c46jQBq,9599,simonw,2021-11-30T06:12:19Z,2021-11-30T06:12:19Z,OWNER,That's really cool to hear - I've not seen many people actively building on top of the JSON API.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065429936,Use datasette-table Web Component to guide the design of the JSON API for 1.0,
https://github.com/simonw/datasette/issues/1527#issuecomment-982318745,https://api.github.com/repos/simonw/datasette/issues/1527,982318745,IC_kwDOBm6k_c46jP6Z,9599,simonw,2021-11-30T06:11:21Z,2021-11-30T06:11:21Z,OWNER,"Manually tested this too, looks like that fixed it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059555791,Columns starting with an underscore behave poorly in filters,
https://github.com/simonw/datasette/issues/1527#issuecomment-982235541,https://api.github.com/repos/simonw/datasette/issues/1527,982235541,IC_kwDOBm6k_c46i7mV,9599,simonw,2021-11-30T02:57:34Z,2021-11-30T02:58:44Z,OWNER,"I started fiddling with a test for this which extracts the `