html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app https://github.com/simonw/sqlite-utils/issues/506#issuecomment-1296358636,https://api.github.com/repos/simonw/sqlite-utils/issues/506,1296358636,IC_kwDOCGYnMM5NRNzs,9599,simonw,2022-10-30T21:52:11Z,2022-10-30T21:52:11Z,OWNER,This could work in a similar way to `db.insert(...).last_rowid`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1429029604,Make `cursor.rowcount` accessible (wontfix), https://github.com/simonw/datasette/issues/1873#issuecomment-1296343716,https://api.github.com/repos/simonw/datasette/issues/1873,1296343716,IC_kwDOBm6k_c5NRKKk,9599,simonw,2022-10-30T20:24:55Z,2022-10-30T20:24:55Z,OWNER,"I think the key feature I need here is going to be the equivalent of `ignore=True` and `replace=True` for dealing with primary key collisions, see https://sqlite-utils.datasette.io/en/stable/reference.html#sqlite_utils.db.Table.insert","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/datasette/issues/1873#issuecomment-1296343317,https://api.github.com/repos/simonw/datasette/issues/1873,1296343317,IC_kwDOBm6k_c5NRKEV,9599,simonw,2022-10-30T20:22:40Z,2022-10-30T20:22:40Z,OWNER,"So maybe they're not actually worth worrying about separately, because they are guaranteed to have a primary key set.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/datasette/issues/1873#issuecomment-1296343173,https://api.github.com/repos/simonw/datasette/issues/1873,1296343173,IC_kwDOBm6k_c5NRKCF,9599,simonw,2022-10-30T20:21:54Z,2022-10-30T20:22:20Z,OWNER,"One last case to consider: `WITHOUT ROWID` tables. https://www.sqlite.org/withoutrowid.html > By default, every row in SQLite has a special column, usually called the ""[rowid](https://www.sqlite.org/lang_createtable.html#rowid)"", that uniquely identifies that row within the table. However if the phrase ""WITHOUT ROWID"" is added to the end of a [CREATE TABLE](https://www.sqlite.org/lang_createtable.html) statement, then the special ""rowid"" column is omitted. There are sometimes space and performance advantages to omitting the rowid. > > ... > > Every WITHOUT ROWID table must have a [PRIMARY KEY](https://www.sqlite.org/lang_createtable.html#primkeyconst). An error is raised if a CREATE TABLE statement with the WITHOUT ROWID clause lacks a PRIMARY KEY.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/datasette/issues/1873#issuecomment-1296343014,https://api.github.com/repos/simonw/datasette/issues/1873,1296343014,IC_kwDOBm6k_c5NRJ_m,9599,simonw,2022-10-30T20:21:01Z,2022-10-30T20:21:01Z,OWNER,"Actually, for simplicity I'm going to say that you can always set the primary key, even for auto-incrementing primary key columns... but you cannot set it on pure `rowid` columns.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/datasette/issues/1873#issuecomment-1296342814,https://api.github.com/repos/simonw/datasette/issues/1873,1296342814,IC_kwDOBm6k_c5NRJ8e,9599,simonw,2022-10-30T20:20:05Z,2022-10-30T20:20:05Z,OWNER,"Some notes on what Datasette does already https://latest.datasette.io/fixtures/tags.json?_shape=array returns: ```json [ { ""tag"": ""canine"" }, { ""tag"": ""feline"" } ] ``` That table is defined [like this](https://latest.datasette.io/fixtures/tags): ```sql CREATE TABLE tags ( tag TEXT PRIMARY KEY ); ``` Here's a `rowid` table with no explicit primary key: https://latest.datasette.io/fixtures/binary_data https://latest.datasette.io/fixtures/binary_data.json?_shape=array ```json [ { ""rowid"": 1, ""data"": { ""$base64"": true, ""encoded"": ""FRwCx60F/g=="" } }, { ""rowid"": 2, ""data"": { ""$base64"": true, ""encoded"": ""FRwDx60F/g=="" } }, { ""rowid"": 3, ""data"": null } ] ``` ```sql CREATE TABLE binary_data ( data BLOB ); ``` https://latest.datasette.io/fixtures/simple_primary_key has a text primary key: https://latest.datasette.io/fixtures/simple_primary_key.json?_shape=array ```json [ { ""id"": ""1"", ""content"": ""hello"" }, { ""id"": ""2"", ""content"": ""world"" }, { ""id"": ""3"", ""content"": """" }, { ""id"": ""4"", ""content"": ""RENDER_CELL_DEMO"" }, { ""id"": ""5"", ""content"": ""RENDER_CELL_ASYNC"" } ] ``` ```sql CREATE TABLE simple_primary_key ( id varchar(30) primary key, content text ); ``` https://latest.datasette.io/fixtures/compound_primary_key is a compound primary key. https://latest.datasette.io/fixtures/compound_primary_key.json?_shape=array ```json [ { ""pk1"": ""a"", ""pk2"": ""b"", ""content"": ""c"" }, { ""pk1"": ""a/b"", ""pk2"": "".c-d"", ""content"": ""c"" } ] ``` ```sql CREATE TABLE compound_primary_key ( pk1 varchar(30), pk2 varchar(30), content text, PRIMARY KEY (pk1, pk2) ); ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/datasette/issues/1873#issuecomment-1296341469,https://api.github.com/repos/simonw/datasette/issues/1873,1296341469,IC_kwDOBm6k_c5NRJnd,9599,simonw,2022-10-30T20:13:50Z,2022-10-30T20:13:50Z,OWNER,"I checked and SQLite itself does allow you to set the `rowid` on that kind of table - it then increments from whatever you inserted: ``` % sqlite3 /tmp/t.db SQLite version 3.39.4 2022-09-07 20:51:41 Enter "".help"" for usage hints. sqlite> create table docs (title text); sqlite> insert into docs (title) values ('one'); sqlite> select rowid, title from docs; 1|one sqlite> insert into docs (rowid, title) values (3, 'three'); sqlite> select rowid, title from docs; 1|one 3|three sqlite> insert into docs (title) values ('another'); sqlite> select rowid, title from docs; 1|one 3|three 4|another ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/datasette/issues/1873#issuecomment-1296341055,https://api.github.com/repos/simonw/datasette/issues/1873,1296341055,IC_kwDOBm6k_c5NRJg_,9599,simonw,2022-10-30T20:11:47Z,2022-10-30T20:12:30Z,OWNER,"If a table has an auto-incrementing primary key, should you be allowed to insert records with an explicit key into it? I'm torn on this one. It's something you can do with direct database access, but it's something I very rarely want to do. I'm inclined to disallow it and say that if you want that you can get it using a writable canned query instead. Likewise, I'm not going to provide a way to set the `rowid` explicitly on a freshly inserted row.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/datasette/issues/1871#issuecomment-1296339386,https://api.github.com/repos/simonw/datasette/issues/1871,1296339386,IC_kwDOBm6k_c5NRJG6,9599,simonw,2022-10-30T20:03:04Z,2022-10-30T20:03:04Z,OWNER,"I do need to skip CSRF for these API calls. I'm going to start out by doing that using the `skip_csrf()` hook to skip CSRF checks on anything with a `content-type: application/json` request header. ```python @hookimpl def skip_csrf(scope): if scope[""type""] == ""http"": headers = scope.get(""headers"") if dict(headers).get(b'content-type') == b'application/json': return True ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1296339205,https://api.github.com/repos/simonw/datasette/issues/1871,1296339205,IC_kwDOBm6k_c5NRJEF,9599,simonw,2022-10-30T20:02:05Z,2022-10-30T20:02:05Z,OWNER,"Realized the API explorer doesn't need the API key piece at all - it can work with standard cookie-based auth. This also reflects how most plugins are likely to use this API, where they'll be adding JavaScript that uses `fetch()` to call the write API directly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1296131872,https://api.github.com/repos/simonw/datasette/issues/1871,1296131872,IC_kwDOBm6k_c5NQWcg,9599,simonw,2022-10-30T06:27:56Z,2022-10-30T06:27:56Z,OWNER,Initial prototype API explorer is now live at https://latest-1-0-dev.datasette.io/-/api,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1873#issuecomment-1296131681,https://api.github.com/repos/simonw/datasette/issues/1873,1296131681,IC_kwDOBm6k_c5NQWZh,9599,simonw,2022-10-30T06:27:12Z,2022-10-30T06:27:12Z,OWNER,Relevant TODO: https://github.com/simonw/datasette/blob/c35859ae3df163406f1a1895ccf9803e933b2d8e/datasette/views/table.py#L1131-L1135,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/datasette/issues/1872#issuecomment-1296131343,https://api.github.com/repos/simonw/datasette/issues/1872,1296131343,IC_kwDOBm6k_c5NQWUP,9599,simonw,2022-10-30T06:26:01Z,2022-10-30T06:26:01Z,OWNER,"Good spot fixing that! Sorry about this - it was a change in Datasette 0.63 which should have been better called out. My goal for Datasette 1.0 (which I aim to have out by the end of the year) is to introduce a formal process for avoiding problems like this, with very clear documentation when something like this might happen.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428560020,"SITE-BUSTING ERROR: ""render_template() called before await ds.invoke_startup()""", https://github.com/simonw/datasette/issues/1871#issuecomment-1296130073,https://api.github.com/repos/simonw/datasette/issues/1871,1296130073,IC_kwDOBm6k_c5NQWAZ,9599,simonw,2022-10-30T06:20:56Z,2022-10-30T06:20:56Z,OWNER,"That initial prototype looks like this: It currently shows the returned JSON from the API in an `alert()`. Next I should make that part of the page instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1296126389,https://api.github.com/repos/simonw/datasette/issues/1871,1296126389,IC_kwDOBm6k_c5NQVG1,9599,simonw,2022-10-30T06:04:48Z,2022-10-30T06:04:48Z,OWNER,"This is even more important now I have pushed: - #1866","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1296114136,https://api.github.com/repos/simonw/datasette/issues/1871,1296114136,IC_kwDOBm6k_c5NQSHY,9599,simonw,2022-10-30T05:15:40Z,2022-10-30T05:15:40Z,OWNER,"Host it at `/-/api` It's an input box with a path in and a textarea you can put JSON in, plus a submit button to post the request. It lists the API endpoints you can use - click on a link to populate the form field plus a example. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/pull/1870#issuecomment-1295660092,https://api.github.com/repos/simonw/datasette/issues/1870,1295660092,IC_kwDOBm6k_c5NOjQ8,9599,simonw,2022-10-29T00:25:26Z,2022-10-29T00:25:26Z,OWNER,"Saw your comment here too: https://github.com/simonw/datasette/issues/1480#issuecomment-1271101072 > switching from `immutable=1` to `mode=ro` completely addressed this. see https://github.com/simonw/datasette/issues/1836#issuecomment-1271100651 for details. So maybe we need a special case for containers that are intended to be run using Docker - the ones produced by `datasette package` and `datasette publish cloudrun`? Those are cases where the `-i` option should actually be opened in read-only mode, not immutable mode. Maybe a `datasette serve --irw data.db` option for opening a file in immutable-but-actually-read-only mode? Bit ugly though. I should run some benchmarks to figure out if `immutable` really does offer significant performance benefits.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426379903,"don't use immutable=1, only mode=ro", https://github.com/simonw/datasette/pull/1870#issuecomment-1295657771,https://api.github.com/repos/simonw/datasette/issues/1870,1295657771,IC_kwDOBm6k_c5NOisr,9599,simonw,2022-10-29T00:19:03Z,2022-10-29T00:19:03Z,OWNER,"Just saw your comment here: https://github.com/simonw/datasette/issues/1836#issuecomment-1272357976 > when you are running from docker, you **always** will want to run as `mode=ro` because the same thing that is causing duplication in the inspect layer will cause duplication in the final container read/write layer when `datasette serve` runs. I don't understand this. My mental model of how Docker works is that the image itself is created using `docker build`... but then when the image runs later on (`docker run`) the image itself isn't touched at all. Are you saying that I can build a container, but then when I run it and it does `datasette serve -i data.db ...` it will somehow modify the image, or create a new modified filesystem layer in the runtime environment, as a result of running that `serve` command?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426379903,"don't use immutable=1, only mode=ro", https://github.com/simonw/datasette/issues/1866#issuecomment-1295200988,https://api.github.com/repos/simonw/datasette/issues/1866,1295200988,IC_kwDOBm6k_c5NMzLc,9599,simonw,2022-10-28T16:29:55Z,2022-10-28T16:29:55Z,OWNER,"I wonder if there's something clever I could do here within a transaction? Start a transaction. Write out a temporary in-memory table with all of the existing primary keys in the table. Run the bulk insert. Then run `select pk from table where pk not in (select pk from old_pks)` to see what has changed. I don't think that's going to work well for large tables. I'm going to go with not returning inserted rows by default, unless you pass a special option requesting that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table, https://github.com/simonw/datasette/issues/1866#issuecomment-1294316640,https://api.github.com/repos/simonw/datasette/issues/1866,1294316640,IC_kwDOBm6k_c5NJbRg,9599,simonw,2022-10-28T01:51:40Z,2022-10-28T01:51:40Z,OWNER,"This needs to support the following: - Rows do not include a primary key - one is assigned by the database - Rows provide their own primary key, any clashes are errors - Rows provide their own primary key, clashes are silently ignored - Rows provide their own primary key, replacing any existing records","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table, https://github.com/simonw/datasette/issues/1866#issuecomment-1294306071,https://api.github.com/repos/simonw/datasette/issues/1866,1294306071,IC_kwDOBm6k_c5NJYsX,9599,simonw,2022-10-28T01:37:14Z,2022-10-28T01:37:59Z,OWNER,"Quick crude benchmark: ```python import sqlite3 db = sqlite3.connect("":memory:"") def create_table(db, name): db.execute(f""create table {name} (id integer primary key, title text)"") create_table(db, ""single"") create_table(db, ""multi"") create_table(db, ""bulk"") def insert_singles(titles): inserted = [] for title in titles: cursor = db.execute(f""insert into single (title) values (?)"", [title]) inserted.append((cursor.lastrowid, title)) return inserted def insert_many(titles): db.executemany(f""insert into multi (title) values (?)"", ((t,) for t in titles)) def insert_bulk(titles): db.execute(""insert into bulk (title) values {}"".format( "", "".join(""(?)"" for _ in titles) ), titles) titles = [""title {}"".format(i) for i in range(1, 10001)] ``` Then in iPython I ran these: ``` In [14]: %timeit insert_singles(titles) 23.8 ms ± 535 µs per loop (mean ± std. dev. of 7 runs, 10 loops each) In [13]: %timeit insert_many(titles) 12 ms ± 520 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) In [12]: %timeit insert_bulk(titles) 2.59 ms ± 25 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) ``` So the bulk insert really is a lot faster - 3ms compared to 24ms for single inserts, so ~8x faster.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table, https://github.com/simonw/datasette/issues/1866#issuecomment-1294296767,https://api.github.com/repos/simonw/datasette/issues/1866,1294296767,IC_kwDOBm6k_c5NJWa_,9599,simonw,2022-10-28T01:22:25Z,2022-10-28T01:23:09Z,OWNER,"Nasty catch on this one: I wanted to return the IDs of the freshly inserted rows. But... the `insert_all()` method I was planning to use from `sqlite-utils` doesn't appear to have a way of doing that: https://github.com/simonw/sqlite-utils/blob/529110e7d8c4a6b1bbf5fb61f2e29d72aa95a611/sqlite_utils/db.py#L2813-L2835 SQLite itself added a `RETURNING` statement which might help, but that is only available from version 3.35 released in March 2021: https://www.sqlite.org/lang_returning.html - which isn't commonly available yet. https://latest.datasette.io/-/versions right now shows 3.34, and https://lite.datasette.io/#/-/versions shows 3.27.2 (from Feb 2019). Two options then: 1. Even for bulk inserts do one insert at a time so I can use `cursor.lastrowid` to get the ID of the inserted record. This isn't terrible since SQLite is very fast, but it may still be a big performance hit for large inserts. 2. Don't return the list of inserted rows for bulk inserts 3. Default to not returning the list of inserted rows for bulk inserts, but allow the user to request that - in which case we use the slower path That third option might be the way to go here. I should benchmark first to figure out how much of a difference this actually makes.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table, https://github.com/simonw/datasette/issues/1866#issuecomment-1294282263,https://api.github.com/repos/simonw/datasette/issues/1866,1294282263,IC_kwDOBm6k_c5NJS4X,9599,simonw,2022-10-28T01:00:42Z,2022-10-28T01:00:42Z,OWNER,"I'm going to set the limit at 1,000 rows inserted at a time. I'll make this configurable using a new `max_insert_rows` setting (for consistency with `max_returned_rows`).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table, https://github.com/simonw/datasette/issues/1851#issuecomment-1294281451,https://api.github.com/repos/simonw/datasette/issues/1851,1294281451,IC_kwDOBm6k_c5NJSrr,9599,simonw,2022-10-28T00:59:25Z,2022-10-28T00:59:25Z,OWNER,"I'm going to use this endpoint for bulk inserts too, so I'm closing this issue and continuing the work here: - #1866","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1851#issuecomment-1289712350,https://api.github.com/repos/simonw/datasette/issues/1851,1289712350,IC_kwDOBm6k_c5M33Le,9599,simonw,2022-10-24T22:28:39Z,2022-10-27T23:18:48Z,OWNER,"API design: (**UPDATE: this was [later changed to POST /db/table/-/insert](https://github.com/simonw/datasette/issues/1851#issuecomment-1294224185)) ``` POST /db/table Authorization: Bearer xxx Content-Type: application/json { ""row"": { ""id"": 1, ""name"": ""New record"" } } ``` Returns: ``` 201 Created { ""row"": { ""id"": 1, ""name"": ""New record"" } } ``` You can omit optional fields in the input, including the ID field. The returned object will always include all fields - and will even include `rowid` if your object doesn't have a primary key of its own. I decided to use `""row""` as the key in both request and response, to preserve space for other future keys - one that tells you that the table has been created, for example.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1869#issuecomment-1294181485,https://api.github.com/repos/simonw/datasette/issues/1869,1294181485,IC_kwDOBm6k_c5NI6Rt,9599,simonw,2022-10-27T22:24:37Z,2022-10-27T22:24:37Z,OWNER,"https://docs.datasette.io/en/stable/changelog.html#v0-63 Annotated release notes: https://simonwillison.net/2022/Oct/27/datasette-0-63/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426253476,Release 0.63, https://github.com/simonw/datasette/issues/1786#issuecomment-1294116493,https://api.github.com/repos/simonw/datasette/issues/1786,1294116493,IC_kwDOBm6k_c5NIqaN,9599,simonw,2022-10-27T21:50:12Z,2022-10-27T21:50:12Z,OWNER,Demo in Datasette Lite: https://lite.datasette.io/#/fixtures?sql=select%0A++pk1%2C%0A++pk2%2C%0A++content%2C%0A++sortable%2C%0A++sortable_with_nulls%2C%0A++sortable_with_nulls_2%2C%0A++text%0Afrom%0A++sortable%0Aorder+by%0A++pk1%2C%0A++pk2%0Alimit%0A++101,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1342430983,Adjust height of textarea for no JS case, https://github.com/simonw/datasette/issues/1869#issuecomment-1294105558,https://api.github.com/repos/simonw/datasette/issues/1869,1294105558,IC_kwDOBm6k_c5NInvW,9599,simonw,2022-10-27T21:44:13Z,2022-10-27T21:44:13Z,OWNER,I'm going to do annotated release notes for this one.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426253476,Release 0.63, https://github.com/simonw/datasette/issues/1869#issuecomment-1294056552,https://api.github.com/repos/simonw/datasette/issues/1869,1294056552,IC_kwDOBm6k_c5NIbxo,9599,simonw,2022-10-27T21:00:02Z,2022-10-27T21:02:25Z,OWNER,"Those release notes as markdown: ### Features - Now tested against Python 3.11. Docker containers used by `datasette publish` and `datasette package` both now use that version of Python. ([#1853](https://github.com/simonw/datasette/issues/1853)) - `--load-extension` option now supports entrypoints. Thanks, Alex Garcia. ([#1789](https://github.com/simonw/datasette/pull/1789)) - Facet size can now be set per-table with the new `facet_size` table metadata option. ([#1804](https://github.com/simonw/datasette/issues/1804)) - The [truncate_cells_html](https://docs.datasette.io/en/stable/settings.html#setting-truncate-cells-html) setting now also affects long URLs in columns. ([#1805](https://github.com/simonw/datasette/issues/1805)) - The non-JavaScript SQL editor textarea now increases height to fit the SQL query. ([#1786](https://github.com/simonw/datasette/issues/1786)) - Facets are now displayed with better line-breaks in long values. Thanks, Daniel Rech. ([#1794](https://github.com/simonw/datasette/pull/1794)) - The `settings.json` file used in [Configuration directory mode](https://docs.datasette.io/en/stable/settings.html#config-dir) is now validated on startup. ([#1816](https://github.com/simonw/datasette/issues/1816)) - SQL queries can now include leading SQL comments, using `/* ... */` or `-- ...` syntax. Thanks, Charles Nepote. ([#1860](https://github.com/simonw/datasette/issues/1860)) - SQL query is now re-displayed when terminated with a time limit error. ([#1819](https://github.com/simonw/datasette/issues/1819)) - The [inspect data](https://docs.datasette.io/en/stable/performance.html#performance-inspect) mechanism is now used to speed up server startup - thanks, Forest Gregg. ([#1834](https://github.com/simonw/datasette/issues/1834)) - In [Configuration directory mode](https://docs.datasette.io/en/stable/settings.html#config-dir) databases with filenames ending in `.sqlite` or `.sqlite3` are now automatically added to the Datasette instance. ([#1646](https://github.com/simonw/datasette/issues/1646)) - Breadcrumb navigation display now respects the current user's permissions. ([#1831](https://github.com/simonw/datasette/issues/1831)) ### Plugin hooks and internals - The [prepare_jinja2_environment(env, datasette)](https://docs.datasette.io/en/stable/plugin_hooks.html#plugin-hook-prepare-jinja2-environment) plugin hook now accepts an optional `datasette` argument. Hook implementations can also now return an `async` function which will be awaited automatically. ([#1809](https://github.com/simonw/datasette/issues/1809)) - `Database(is_mutable=)` now defaults to `True`. ([#1808](https://github.com/simonw/datasette/issues/1808)) - The [datasette.check_visibility()](https://docs.datasette.io/en/stable/internals.html#datasette-check-visibility) method now accepts an optional `permissions=` list, allowing it to take multiple permissions into account at once when deciding if something should be shown as public or private. This has been used to correctly display padlock icons in more places in the Datasette interface. ([#1829](https://github.com/simonw/datasette/issues/1829)) - Datasette no longer enforces upper bounds on its dependencies. ([#1800](https://github.com/simonw/datasette/issues/1800)) ### Documentation - New tutorial: [Cleaning data with sqlite-utils and Datasette](https://datasette.io/tutorials/clean-data). - Screenshots in the documentation are now maintained using [shot-scraper](https://shot-scraper.datasette.io/), as described in [Automating screenshots for the Datasette documentation using shot-scraper](https://simonwillison.net/2022/Oct/14/automating-screenshots/). ([#1844](https://github.com/simonw/datasette/issues/1844)) - More detailed command descriptions on the [CLI reference](https://docs.datasette.io/en/stable/cli-reference.html#cli-reference) page. ([#1787](https://github.com/simonw/datasette/issues/1787)) - New documentation on [Running Datasette using OpenRC](https://docs.datasette.io/en/stable/deploying.html#deploying-openrc) - thanks, Adam Simpson. ([#1825](https://github.com/simonw/datasette/pull/1825))","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426253476,Release 0.63, https://github.com/simonw/datasette/pull/1835#issuecomment-1294049178,https://api.github.com/repos/simonw/datasette/issues/1835,1294049178,IC_kwDOBm6k_c5NIZ-a,9599,simonw,2022-10-27T20:51:30Z,2022-10-27T20:51:30Z,OWNER,"See also: - https://github.com/simonw/datasette/pull/1837","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1400121355,use inspect data for hash and file size, https://github.com/simonw/datasette/pull/1837#issuecomment-1294048849,https://api.github.com/repos/simonw/datasette/issues/1837,1294048849,IC_kwDOBm6k_c5NIZ5R,9599,simonw,2022-10-27T20:51:08Z,2022-10-27T20:51:08Z,OWNER,"Yeah this is better, thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1400431789,Make hash and size a lazy property, https://github.com/simonw/datasette/pull/1839#issuecomment-1294034011,https://api.github.com/repos/simonw/datasette/issues/1839,1294034011,IC_kwDOBm6k_c5NIWRb,9599,simonw,2022-10-27T20:34:37Z,2022-10-27T20:34:37Z,OWNER,@dependabot rebase,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1401155623,Bump black from 22.8.0 to 22.10.0, https://github.com/simonw/datasette/issues/1851#issuecomment-1294012583,https://api.github.com/repos/simonw/datasette/issues/1851,1294012583,IC_kwDOBm6k_c5NIRCn,9599,simonw,2022-10-27T20:11:22Z,2022-10-27T20:11:22Z,OWNER,"And the response to `""inserted"": [{...}]` - it will be the same for bulk inserts.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1851#issuecomment-1294012084,https://api.github.com/repos/simonw/datasette/issues/1851,1294012084,IC_kwDOBm6k_c5NIQ60,9599,simonw,2022-10-27T20:10:47Z,2022-10-27T20:10:47Z,OWNER,"I'm going to change the incoming JSON back to `{""row"": {...}}` - no need to POST `{""insert"": ...}` to something with `/-/insert` in the URL already.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1851#issuecomment-1294009354,https://api.github.com/repos/simonw/datasette/issues/1851,1294009354,IC_kwDOBm6k_c5NIQQK,9599,simonw,2022-10-27T20:07:42Z,2022-10-27T20:07:42Z,OWNER,"Need to implement the new URL design from: - #1868 This is now going to be `/db/table/-/insert` - and it will eventually handle bulk inserts as well as single inserts.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1868#issuecomment-1294008733,https://api.github.com/repos/simonw/datasette/issues/1868,1294008733,IC_kwDOBm6k_c5NIQGd,9599,simonw,2022-10-27T20:07:01Z,2022-10-27T20:07:01Z,OWNER,I'm happy with this `/db/table/-/action` design for the moment. Will review it once I've built it to see if I still like it!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426195437,Design URLs for the write API, https://github.com/simonw/datasette/issues/1868#issuecomment-1294008282,https://api.github.com/repos/simonw/datasette/issues/1868,1294008282,IC_kwDOBm6k_c5NIP_a,9599,simonw,2022-10-27T20:06:34Z,2022-10-27T20:06:34Z,OWNER,"I'm going to stick with one `/-/insert` endpoint which handles both single row inserts AND multiple row inserts I think - partly because I don't want to build both `/-/upsert` and `/-/upsert-many`, I'd rather just have `/-/upsert`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426195437,Design URLs for the write API, https://github.com/simonw/datasette/issues/1868#issuecomment-1294007024,https://api.github.com/repos/simonw/datasette/issues/1868,1294007024,IC_kwDOBm6k_c5NIPrw,9599,simonw,2022-10-27T20:05:44Z,2022-10-27T20:05:52Z,OWNER,"So given this scheme, the URL design would look like this: - `POST /db/table/-/insert` - insert a single row - `POST /db/table/-/insert-many` - insert multiple rows (might just keep that on `/-/insert` with a JSON array rather than object though) - `POST /db/table/-/drop` - drop a table - `POST /db/table/-/alter` - alter a table - `POST /db/table/-/upsert` - upsert, https://sqlite-utils.datasette.io/en/stable/python-api.html#upserting-data - `POST /db/table/-/create` - could be an endpoint for explicitly creating a table, or should that live at `/db/-/create` instead? And for rows (`pks` here since compound primary keys are supported): - `POST /db/table/pks/-/update` - update row - `POST /db/table/pks/-/delete` - delete row","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426195437,Design URLs for the write API, https://github.com/simonw/datasette/issues/1868#issuecomment-1294004308,https://api.github.com/repos/simonw/datasette/issues/1868,1294004308,IC_kwDOBm6k_c5NIPBU,9599,simonw,2022-10-27T20:03:08Z,2022-10-27T20:03:08Z,OWNER,The other option here would be to lean into custom HTTP verbs like `DELETE` and `PATCH`. I'm not sold on those: they've never given me any convincing wins over just using `POST` for the many times I've encountered them in my career to date.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426195437,Design URLs for the write API, https://github.com/simonw/datasette/issues/1868#issuecomment-1294003701,https://api.github.com/repos/simonw/datasette/issues/1868,1294003701,IC_kwDOBm6k_c5NIO31,9599,simonw,2022-10-27T20:02:26Z,2022-10-27T20:02:26Z,OWNER,"The problem with the above design is that I want to support a bunch of different actions that can be taken against a table: - insert a single row - insert multiple rows - bulk update rows - rename table - alter table - drop table I could have ALL of those be a `POST /db/table` with different JSON root keys (`{""drop"": true}` for example, but this raises two problems: 1. Server logs that only show `POST /db/table` will be less useful, they won't reveal what action was performed 2. What happens if you send `{""insert"": {""title"": ""New record""}, ""drop"": true}`? Does that return an error, or does it perform both of those actions? This is already slightly confusing in that `POST /db/name-of-query` is the existing API for executing a writable canned query: https://docs.datasette.io/en/stable/sql_queries.html#json-api-for-writable-canned-queries So I'm ready to consider other design options. Initial thoughts on possible designs (for the single row insert case, but could be expanded to cover other verbs): - `POST /db/table?action=insert` - `POST /db/table?nsert` - `POST /db/table/-/insert` I quite like that third one: it feels consistent with the existing `/-/actor` etc pages that Datasette serves already. There's one slight confusion here in that it overlaps with the URL for a row with a primary key of `""-""` - which is currently at `/db/table/-` - but that might be OK. Especially if I say that child pages of rows must theselves use the `/-/` pattern. So to update or delet a row you would use: - `POST /db/table/row/-/update` - `POST /db/table/row/-/delete` So a row with primary key `-` would end up as `/db/table/row/-/-/update` - which I think is OK.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426195437,Design URLs for the write API, https://github.com/simonw/datasette/issues/1851#issuecomment-1293996735,https://api.github.com/repos/simonw/datasette/issues/1851,1293996735,IC_kwDOBm6k_c5NINK_,9599,simonw,2022-10-27T19:54:53Z,2022-10-27T19:54:53Z,OWNER,"Updated docs: https://docs.datasette.io/en/1.0-dev/json_api.html#inserting-a-single-row ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1851#issuecomment-1292997608,https://api.github.com/repos/simonw/datasette/issues/1851,1292997608,IC_kwDOBm6k_c5NEZPo,9599,simonw,2022-10-27T04:54:53Z,2022-10-27T19:05:50Z,OWNER,"I'm going to change the design of this to: ``` { ""insert"": { ""title"" :""..."" } } ``` Renaming `""row""` to `""insert""`. This will be consistent with adding `""drop"": true` for dropping a table, and maybe other verbs like for modifying the schema. The API response will look like this: ```json { ""inserted_row"": { ""id"": 1, ""title"": ""..."" } } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1860#issuecomment-1293939737,https://api.github.com/repos/simonw/datasette/issues/1860,1293939737,IC_kwDOBm6k_c5NH_QZ,9599,simonw,2022-10-27T18:57:37Z,2022-10-27T18:57:37Z,OWNER,The new code is now live at https://latest.datasette.io/fixtures,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1424378012,SQL query field can't begin by a comment, https://github.com/simonw/datasette/issues/1860#issuecomment-1293928738,https://api.github.com/repos/simonw/datasette/issues/1860,1293928738,IC_kwDOBm6k_c5NH8ki,9599,simonw,2022-10-27T18:46:31Z,2022-10-27T18:46:31Z,OWNER,I think mine has a better pattern for handling `/* ... anything in here that isn't */ ... */`,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1424378012,SQL query field can't begin by a comment, https://github.com/simonw/datasette/issues/1860#issuecomment-1293928230,https://api.github.com/repos/simonw/datasette/issues/1860,1293928230,IC_kwDOBm6k_c5NH8cm,9599,simonw,2022-10-27T18:46:03Z,2022-10-27T18:46:03Z,OWNER,"Here's yours on Debuggex: https://www.debuggex.com/r/HjdJryTy9ezGsuWK ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1424378012,SQL query field can't begin by a comment, https://github.com/simonw/datasette/issues/1860#issuecomment-1293926417,https://api.github.com/repos/simonw/datasette/issues/1860,1293926417,IC_kwDOBm6k_c5NH8AR,9599,simonw,2022-10-27T18:44:20Z,2022-10-27T18:45:21Z,OWNER,"Hah, I just came up with this one - we were clearly working on this at the same time! `^\s*((?:\-\-.*?\n\s*)|(?:\/\*((?!\*\/)[\s\S])*\*\/)\s*)*\s*select\b` https://www.debuggex.com/r/Rbw-UWD9PdOU2GyO ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1424378012,SQL query field can't begin by a comment, https://github.com/simonw/datasette/issues/1866#issuecomment-1293893789,https://api.github.com/repos/simonw/datasette/issues/1866,1293893789,IC_kwDOBm6k_c5NH0Cd,9599,simonw,2022-10-27T18:13:00Z,2022-10-27T18:13:00Z,OWNER,If people care about that kind of thing they could always push all of their inserts to a table called `_tablename` and then atomically rename that once they've uploaded all of the data (assuming I provide an atomic-rename-this-table mechanism).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table, https://github.com/simonw/datasette/issues/1866#issuecomment-1293892818,https://api.github.com/repos/simonw/datasette/issues/1866,1293892818,IC_kwDOBm6k_c5NHzzS,9599,simonw,2022-10-27T18:12:02Z,2022-10-27T18:12:02Z,OWNER,"There's one catch with batched inserts: if your CLI tool fails half way through you could end up with a partially populated table - since a bunch of batches will have succeeded first. I think that's OK. In the future I may want to come up with a way to run multiple batches of inserts inside a single transaction, but I can ignore that for the first release of this feature.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table, https://github.com/simonw/datasette/issues/1866#issuecomment-1293891876,https://api.github.com/repos/simonw/datasette/issues/1866,1293891876,IC_kwDOBm6k_c5NHzkk,9599,simonw,2022-10-27T18:11:05Z,2022-10-27T18:11:05Z,OWNER,Likewise for newline-delimited JSON. While it's tempting to want to accept that as an ingest format (because it's nice to generate and stream) I think it's better to have a client application that can turn a stream of newline-delimited JSON into batched JSON inserts.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table, https://github.com/simonw/datasette/issues/1866#issuecomment-1293891191,https://api.github.com/repos/simonw/datasette/issues/1866,1293891191,IC_kwDOBm6k_c5NHzZ3,9599,simonw,2022-10-27T18:10:22Z,2022-10-27T18:10:22Z,OWNER,"So for the moment I'm just going to concentrate on the JSON API. I can consider CSV variants later on, or as plugins, or both.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table, https://github.com/simonw/datasette/issues/1866#issuecomment-1293890684,https://api.github.com/repos/simonw/datasette/issues/1866,1293890684,IC_kwDOBm6k_c5NHzR8,9599,simonw,2022-10-27T18:09:52Z,2022-10-27T18:09:52Z,OWNER,"Should this API accept CSV/TSV etc in addition to JSON? I'm torn on this one. My initial instinct is that it should not - and there should instead be a Datasette client library / CLI tool you can use that knows how to turn CSV into batches of JSON calls for when you want to upload a CSV file. I don't think the usability of `curl https://datasette/db/table -F 'data=@path/to/file.csv' -H 'Authentication: Bearer xxx'` is particularly great compared to something like`datasette client insert https://datasette/ db table file.csv --csv` (where the command version could store API tokens for you too).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table, https://github.com/simonw/datasette/issues/1866#issuecomment-1293887808,https://api.github.com/repos/simonw/datasette/issues/1866,1293887808,IC_kwDOBm6k_c5NHylA,9599,simonw,2022-10-27T18:07:02Z,2022-10-27T18:07:02Z,OWNER,"Error handling is really important here. What should happen if you submit 100 records and one of them has some kind of validation error? How should that error be reported back to you? I'm inclined to say that it defaults to all-or-nothing in a transaction - but there should be a `""continue_on_error"": true` option (or similar) which causes it to insert the ones that are valid while reporting back the ones that are invalid.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table, https://github.com/simonw/datasette/issues/1862#issuecomment-1293857306,https://api.github.com/repos/simonw/datasette/issues/1862,1293857306,IC_kwDOBm6k_c5NHrIa,9599,simonw,2022-10-27T17:38:17Z,2022-10-27T17:38:17Z,OWNER,"Strongly related to: - #1866","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425011030,"Create a new table from one or more records, `sqlite-utils` style", https://github.com/simonw/datasette/issues/1865#issuecomment-1293568194,https://api.github.com/repos/simonw/datasette/issues/1865,1293568194,IC_kwDOBm6k_c5NGkjC,9599,simonw,2022-10-27T13:58:26Z,2022-10-27T13:58:26Z,OWNER,"Here's the issue where I started doing this: - #849","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425682079,Stop syncing main to master, https://github.com/simonw/datasette/issues/849#issuecomment-649908756,https://api.github.com/repos/simonw/datasette/issues/849,649908756,MDEyOklzc3VlQ29tbWVudDY0OTkwODc1Ng==,9599,simonw,2020-06-26T02:09:09Z,2022-10-27T13:57:08Z,OWNER,"I mentioned this issue here: https://simonwillison.net/2020/Jun/26/weeknotes-plugins-sqlite-generate/ Repositories created by following the README in https://github.com/simonw/datasette-template and https://github.com/simonw/click-app have a `main` branch instead of `master` so I have a few examples live now. https://github.com/simonw/datasette-saved-queries is one example.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639072811,Rename master branch to main, https://github.com/simonw/datasette/issues/1851#issuecomment-1292999579,https://api.github.com/repos/simonw/datasette/issues/1851,1292999579,IC_kwDOBm6k_c5NEZub,9599,simonw,2022-10-27T04:59:06Z,2022-10-27T04:59:12Z,OWNER,"I should probably refactor this to use `sqlite-utils`, since I'm going to want to use that later for the feature that automatically creates tables. Might make it easier to solve the rowid issues too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1851#issuecomment-1292996181,https://api.github.com/repos/simonw/datasette/issues/1851,1292996181,IC_kwDOBm6k_c5NEY5V,9599,simonw,2022-10-27T04:51:47Z,2022-10-27T04:51:47Z,OWNER,Also need a test for invalid JSON (currently triggers a 500 HTML error).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1855#issuecomment-1292962813,https://api.github.com/repos/simonw/datasette/issues/1855,1292962813,IC_kwDOBm6k_c5NEQv9,9599,simonw,2022-10-27T04:31:40Z,2022-10-27T04:31:40Z,OWNER,"My hunch on this is that anyone with that level of complex permissions requirements needs to be using a custom authentication plugin which includes much more concrete token rules, rather than the default signed stateless token implementation that ships with Datasette core.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336089,`datasette create-token` ability to create tokens with a reduced set of permissions, https://github.com/simonw/datasette/issues/1855#issuecomment-1292959886,https://api.github.com/repos/simonw/datasette/issues/1855,1292959886,IC_kwDOBm6k_c5NEQCO,9599,simonw,2022-10-27T04:30:07Z,2022-10-27T04:30:07Z,OWNER,"Here's an interesting edge-case to consider: what if a user creates themselves a token for a specific table, then deletes that table, and waits for another user to create a table of the same name... and then uses their previously created token to write to the table that someone else created? Not sure if this is a threat I need to actively consider, but it's worth thinking a little bit about the implications of such a thing - since there will be APIs that allow users to create tables, and there may be cases where people want to have a concept of users ""owning"" specific tables. This is probably something that could be left for plugins to solve, but it still needs to be understood and potentially documented. There may even be a world in which tracking the timestamp at which a table was created becomes useful - because that could then be baked into API tokens, such that a token created BEFORE the table was created does not grant access to that table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336089,`datasette create-token` ability to create tokens with a reduced set of permissions, https://github.com/simonw/datasette/issues/1851#issuecomment-1292952121,https://api.github.com/repos/simonw/datasette/issues/1851,1292952121,IC_kwDOBm6k_c5NEOI5,9599,simonw,2022-10-27T04:24:09Z,2022-10-27T04:24:20Z,OWNER,"And come up with a whole bunch of tests for weird table shapes, surprising column names, different types etc.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1851#issuecomment-1292951833,https://api.github.com/repos/simonw/datasette/issues/1851,1292951833,IC_kwDOBm6k_c5NEOEZ,9599,simonw,2022-10-27T04:23:40Z,2022-10-27T04:23:40Z,OWNER,Also need to think about transactions - it should use them!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1851#issuecomment-1292939146,https://api.github.com/repos/simonw/datasette/issues/1851,1292939146,IC_kwDOBm6k_c5NEK-K,9599,simonw,2022-10-27T04:00:17Z,2022-10-27T04:23:15Z,OWNER,"Documentation for this first draft of the API: https://docs.datasette.io/en/1.0-dev/json_api.html#inserting-a-single-row It currently returns errors as HTML - it needs to return errors as JSON. Also the errors need comprehensive test coverage. I'm also worried about what happens if you use it on a table that doesn't use an integer primary key - need to check that. I think this code may break: https://github.com/simonw/datasette/blob/51c436fed29205721dcf17fa31d7e7090d34ebb8/datasette/views/table.py#L155-L171 Plus will `rowid` tables without an explicit primary key return the `rowid` column? They should.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1850#issuecomment-1292940011,https://api.github.com/repos/simonw/datasette/issues/1850,1292940011,IC_kwDOBm6k_c5NELLr,9599,simonw,2022-10-27T04:01:59Z,2022-10-27T04:01:59Z,OWNER,"Working on that first ""insert row"" implementation: - https://github.com/simonw/datasette/issues/1851 Has made it very clear to me that I should go the whole hog and build the basic form-based interface for this as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421529723,Write API in Datasette core, https://github.com/simonw/datasette/issues/1858#issuecomment-1292709818,https://api.github.com/repos/simonw/datasette/issues/1858,1292709818,IC_kwDOBm6k_c5NDS-6,9599,simonw,2022-10-26T22:07:04Z,2022-10-26T22:07:04Z,OWNER,"New token design: ```json { ""a"": ""actor-id"", ""t"": ""creation timestamp as integer"", ""d"": ""intended duration in seconds, or blank if no duration set"" } ``` This is in place of the `""e"": ""expiry timestamp""` design I've built so far.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423364990,`max_signed_tokens_ttl` setting for a maximum duration on API tokens, https://github.com/simonw/datasette/issues/1858#issuecomment-1292708227,https://api.github.com/repos/simonw/datasette/issues/1858,1292708227,IC_kwDOBm6k_c5NDSmD,9599,simonw,2022-10-26T22:05:34Z,2022-10-26T22:05:34Z,OWNER,"I just realized this can't easily affect the `datasette create-token` command because it doesn't currently accept the `--setting` option, so it wouldn't know what `max_signed_tokens_ttl` was. More to the point: even if it did, someone could abuse their knowledge of the secret to create a signed non-expiring token even on servers that didn't want to support those. So I actually need to redesign the token format: it needs to store the timestamp when the token was created and the intended duration, NOT the timestamp that the token expires at. Otherwise it's not possible for servers to enforce `max_signed_tokens_ttl` - someone could always create a token with a custom `expires_at` timestamp on it outside of the configured limit.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423364990,`max_signed_tokens_ttl` setting for a maximum duration on API tokens, https://github.com/simonw/datasette/issues/1858#issuecomment-1292687774,https://api.github.com/repos/simonw/datasette/issues/1858,1292687774,IC_kwDOBm6k_c5NDNme,9599,simonw,2022-10-26T21:44:57Z,2022-10-26T21:44:57Z,OWNER,"I'm going for consistency with `max_csv_mb` and `max_returned_rows` and `allow_signed_tokens` and `default_cache_ttl`. So `max_signed_tokens_ttl`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423364990,`max_signed_tokens_ttl` setting for a maximum duration on API tokens, https://github.com/simonw/datasette/issues/1860#issuecomment-1292685478,https://api.github.com/repos/simonw/datasette/issues/1860,1292685478,IC_kwDOBm6k_c5NDNCm,9599,simonw,2022-10-26T21:42:35Z,2022-10-26T21:42:35Z,OWNER,"That's deployed to https://latest.datasette.io/ now - some examples: - https://latest.datasette.io/fixtures?sql=--+one+kind+of+comment%0D%0Aselect+*+from+searchable - https://latest.datasette.io/fixtures?sql=%2F*+Multi%0D%0A++line+comment+*%2F%0D%0Aselect+*+from+searchable - https://latest.datasette.io/fixtures?sql=%2F*+Both+kinds+*%2F%0D%0A--+of+comment%0D%0A%2F*+and+more+*%2F%0D%0A--+and+more+and+more%0D%0Aselect+*+from+searchable","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1424378012,SQL query field can't begin by a comment, https://github.com/simonw/datasette/issues/1860#issuecomment-1292679567,https://api.github.com/repos/simonw/datasette/issues/1860,1292679567,IC_kwDOBm6k_c5NDLmP,9599,simonw,2022-10-26T21:36:25Z,2022-10-26T21:36:25Z,OWNER,I'm never 100% sure how to tell if a regular expression includes a nasty denial of service attack - are there any inputs that could cause this new regex to execute in quadratic time or similar?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1424378012,SQL query field can't begin by a comment, https://github.com/simonw/datasette/issues/1860#issuecomment-1292678657,https://api.github.com/repos/simonw/datasette/issues/1860,1292678657,IC_kwDOBm6k_c5NDLYB,9599,simonw,2022-10-26T21:35:23Z,2022-10-26T21:35:37Z,OWNER,Here are the new tests - each of these should now work: https://github.com/simonw/datasette/blob/55a709c480a1e7401b4ff6208f37a2cf7c682183/tests/test_utils.py#L170-L175,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1424378012,SQL query field can't begin by a comment, https://github.com/simonw/datasette/issues/1860#issuecomment-1292674919,https://api.github.com/repos/simonw/datasette/issues/1860,1292674919,IC_kwDOBm6k_c5NDKdn,9599,simonw,2022-10-26T21:31:22Z,2022-10-26T21:31:22Z,OWNER,"I'm experimenting with this: ```python # Allow SQL to start with a /* */ or -- comment comment_re = ( # Start of string, then any amount of whitespace r'^(\s*' + # Comment that starts with -- and ends at a newline r'(?:\-\-.*?\n\s*)' + # Comment that starts with /* and ends with */ r'|(?:/\*[\s\S]*?\*/)' + # Whitespace r')*\s*' ) allowed_sql_res = [ re.compile(comment_re + r""select\b""), re.compile(comment_re + r""explain\s+select\b""), re.compile(comment_re + r""explain\s+query\s+plan\s+select\b""), re.compile(comment_re + r""with\b""), re.compile(comment_re + r""explain\s+with\b""), re.compile(comment_re + r""explain\s+query\s+plan\s+with\b""), ] ``` This should allow any number of comments of either type as a suffix to the allowed SQL patterns. Needs extensive unit tests! I'm not massively worried if it has a flaw in it though, since this is part of Datasette's defense in depth: if a non-SELECT query sneaks through it still shouldn't be able to cause any damage as the database connection is read-only or immutable.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1424378012,SQL query field can't begin by a comment, https://github.com/simonw/datasette/issues/1860#issuecomment-1292659986,https://api.github.com/repos/simonw/datasette/issues/1860,1292659986,IC_kwDOBm6k_c5NDG0S,9599,simonw,2022-10-26T21:14:26Z,2022-10-26T21:15:22Z,OWNER,"Yeah we should fix this. https://www.sqlite.org/lang_comment.html - SQLite also supports `-- style` comments. I like how explicit the documentation is here: > SQL comments begin with two consecutive ""-"" characters (ASCII 0x2d) and extend up to and including the next newline character (ASCII 0x0a) or until the end of input, whichever comes first. > > C-style comments begin with ""/*"" and extend up to and including the next ""*/"" character pair or until the end of input, whichever comes first. C-style comments can span multiple lines. ","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1424378012,SQL query field can't begin by a comment, https://github.com/simonw/datasette/issues/1849#issuecomment-1292654852,https://api.github.com/repos/simonw/datasette/issues/1849,1292654852,IC_kwDOBm6k_c5NDFkE,9599,simonw,2022-10-26T21:08:44Z,2022-10-26T21:08:44Z,OWNER,"Generally though we should expect that people might try to use `render_template(...)` without passing a `request`, so Datasette core should be able to handle this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1420174670,NoneType' object has no attribute 'actor', https://github.com/simonw/datasette/issues/1849#issuecomment-1292654522,https://api.github.com/repos/simonw/datasette/issues/1849,1292654522,IC_kwDOBm6k_c5NDFe6,9599,simonw,2022-10-26T21:08:20Z,2022-10-26T21:08:20Z,OWNER,"From the stack trace in Sentry: So this happened because a custom plugin tried to render `forbidden.html` without passing in the `request`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1420174670,NoneType' object has no attribute 'actor', https://github.com/simonw/datasette/issues/1849#issuecomment-1292653219,https://api.github.com/repos/simonw/datasette/issues/1849,1292653219,IC_kwDOBm6k_c5NDFKj,9599,simonw,2022-10-26T21:06:56Z,2022-10-26T21:06:56Z,OWNER,"This was a hit to an authenticated page where the incoming user WAS logged in but did not have permission to view that specific page. Code in question: https://github.com/simonw/datasette/blob/c7dd76c26257ded5bcdfd0570e12412531b8b88f/datasette/app.py#L634-L640","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1420174670,NoneType' object has no attribute 'actor', https://github.com/simonw/datasette/issues/1851#issuecomment-1292544296,https://api.github.com/repos/simonw/datasette/issues/1851,1292544296,IC_kwDOBm6k_c5NCqko,9599,simonw,2022-10-26T19:33:34Z,2022-10-26T19:33:34Z,OWNER,That trigger solution is pretty neat!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1855#issuecomment-1291485444,https://api.github.com/repos/simonw/datasette/issues/1855,1291485444,IC_kwDOBm6k_c5M-oEE,9599,simonw,2022-10-26T04:30:34Z,2022-10-26T04:30:34Z,OWNER,"I'm going to delay working on this until after I have some of the write APIs built to try it against: - #1851","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336089,`datasette create-token` ability to create tokens with a reduced set of permissions, https://github.com/simonw/datasette/issues/1859#issuecomment-1291484749,https://api.github.com/repos/simonw/datasette/issues/1859,1291484749,IC_kwDOBm6k_c5M-n5N,9599,simonw,2022-10-26T04:29:43Z,2022-10-26T04:29:43Z,OWNER,"Documentation: - https://docs.datasette.io/en/1.0-dev/authentication.html#datasette-create-token - https://docs.datasette.io/en/1.0-dev/cli-reference.html#datasette-create-token","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423369494,datasette create-token CLI command, https://github.com/simonw/datasette/issues/1843#issuecomment-1291467084,https://api.github.com/repos/simonw/datasette/issues/1843,1291467084,IC_kwDOBm6k_c5M-jlM,9599,simonw,2022-10-26T04:03:49Z,2022-10-26T04:03:49Z,OWNER,"This time I'm suspicious that there are open SQLite files tucked away in thread locals hidden inside my thread pool executor: https://github.com/simonw/datasette/blob/c7dd76c26257ded5bcdfd0570e12412531b8b88f/datasette/database.py#L24 https://github.com/simonw/datasette/blob/c7dd76c26257ded5bcdfd0570e12412531b8b88f/datasette/database.py#L204-L214","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705,"Intermittent ""Too many open files"" error running tests", https://github.com/simonw/datasette/issues/1843#issuecomment-1291466613,https://api.github.com/repos/simonw/datasette/issues/1843,1291466613,IC_kwDOBm6k_c5M-jd1,9599,simonw,2022-10-26T04:02:56Z,2022-10-26T04:02:56Z,OWNER,Just saw this error again!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705,"Intermittent ""Too many open files"" error running tests", https://github.com/simonw/datasette/issues/1859#issuecomment-1291439998,https://api.github.com/repos/simonw/datasette/issues/1859,1291439998,IC_kwDOBm6k_c5M-c9-,9599,simonw,2022-10-26T03:15:13Z,2022-10-26T03:15:13Z,OWNER,Reads from `DATASETTE_SECRET` or accepts `--secret` for the signing secret.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423369494,datasette create-token CLI command, https://github.com/simonw/datasette/issues/1859#issuecomment-1291439875,https://api.github.com/repos/simonw/datasette/issues/1859,1291439875,IC_kwDOBm6k_c5M-c8D,9599,simonw,2022-10-26T03:14:58Z,2022-10-26T03:14:58Z,OWNER,"Initial design: datasette create-token Or: datasette create-token --expire-after 10m","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423369494,datasette create-token CLI command, https://github.com/simonw/datasette/issues/1858#issuecomment-1291435464,https://api.github.com/repos/simonw/datasette/issues/1858,1291435464,IC_kwDOBm6k_c5M-b3I,9599,simonw,2022-10-26T03:07:16Z,2022-10-26T03:07:16Z,OWNER,"This setting will disable the ""Token never expires"" option: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423364990,`max_signed_tokens_ttl` setting for a maximum duration on API tokens, https://github.com/simonw/datasette/issues/1852#issuecomment-1291406219,https://api.github.com/repos/simonw/datasette/issues/1852,1291406219,IC_kwDOBm6k_c5M-UuL,9599,simonw,2022-10-26T02:19:54Z,2022-10-26T02:59:52Z,OWNER,"I'm going to split the remaining work into separate issues: - [x] #1856 - [ ] #1855 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291431132,https://api.github.com/repos/simonw/datasette/issues/1852,1291431132,IC_kwDOBm6k_c5M-azc,9599,simonw,2022-10-26T02:59:50Z,2022-10-26T02:59:50Z,OWNER,Documentation: https://docs.datasette.io/en/1.0-dev/authentication.html#api-tokens,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1850#issuecomment-1291430992,https://api.github.com/repos/simonw/datasette/issues/1850,1291430992,IC_kwDOBm6k_c5M-axQ,9599,simonw,2022-10-26T02:59:33Z,2022-10-26T02:59:33Z,OWNER,I started the documentation for the API tokens mechanism here: https://docs.datasette.io/en/1.0-dev/authentication.html#api-tokens,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421529723,Write API in Datasette core, https://github.com/simonw/datasette/issues/1857#issuecomment-1291418546,https://api.github.com/repos/simonw/datasette/issues/1857,1291418546,IC_kwDOBm6k_c5M-Xuy,9599,simonw,2022-10-26T02:38:35Z,2022-10-26T02:38:35Z,OWNER,"I'm going to set a convention that an actor signed in via a token should set `""token"": ""something""` as a key. Then the `/-/create-token` view can reject those actors.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423347412,Prevent API tokens from using /-/create-token to create more tokens, https://github.com/simonw/datasette/issues/1850#issuecomment-1291417755,https://api.github.com/repos/simonw/datasette/issues/1850,1291417755,IC_kwDOBm6k_c5M-Xib,9599,simonw,2022-10-26T02:36:52Z,2022-10-26T02:36:58Z,OWNER,"I'm going to set a convention that `""token"": ""something""` in an actor means that they were authenticated by a token. `""token"": ""dstok""` for example.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421529723,Write API in Datasette core, https://github.com/simonw/datasette/issues/1850#issuecomment-1291417100,https://api.github.com/repos/simonw/datasette/issues/1850,1291417100,IC_kwDOBm6k_c5M-XYM,9599,simonw,2022-10-26T02:35:32Z,2022-10-26T02:35:32Z,OWNER,"It strikes me that users should NOT be able to use a token to create additional tokens. The current design actually does allow that, since the `dstok_` Bearer token can be used to authenticate calls to the `/-/create-token` page. So I think I need a mechanism whereby that page can only allow access to users authenticated by cookie. Not obvious how to do that though, since Datasette's authentication actor system is designed to abstract that detail away!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421529723,Write API in Datasette core, https://github.com/simonw/datasette/issues/1856#issuecomment-1291410747,https://api.github.com/repos/simonw/datasette/issues/1856,1291410747,IC_kwDOBm6k_c5M-V07,9599,simonw,2022-10-26T02:27:05Z,2022-10-26T02:27:05Z,OWNER,"Because of that I think this is a better name: --setting allow_signed_tokens off","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336122,allow_signed_tokens setting for disabling API signed token mechanism, https://github.com/simonw/datasette/issues/1856#issuecomment-1291410331,https://api.github.com/repos/simonw/datasette/issues/1856,1291410331,IC_kwDOBm6k_c5M-Vub,9599,simonw,2022-10-26T02:26:19Z,2022-10-26T02:26:19Z,OWNER,"It's a bit confusing that a setting called `allow_create_tokens` also causes incoming `dstok_` tokens to be ignored. Is it confusing enough that I should pick a different name for the setting though?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336122,allow_signed_tokens setting for disabling API signed token mechanism, https://github.com/simonw/datasette/issues/1856#issuecomment-1291409312,https://api.github.com/repos/simonw/datasette/issues/1856,1291409312,IC_kwDOBm6k_c5M-Veg,9599,simonw,2022-10-26T02:24:49Z,2022-10-26T02:24:49Z,OWNER,"The effect of this setting will be: - `/-/create-tokens` interface is no longer available - Incoming `dstok_` tokens are no longer respected by the following code: https://github.com/simonw/datasette/blob/b29e487bc3fde6418bf45bda7cfed2e081ff03fb/datasette/default_permissions.py#L52-L72","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336122,allow_signed_tokens setting for disabling API signed token mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291397623,https://api.github.com/repos/simonw/datasette/issues/1852,1291397623,IC_kwDOBm6k_c5M-Sn3,9599,simonw,2022-10-26T02:11:40Z,2022-10-26T02:11:40Z,OWNER,"Built a prototype of the `actor_from_request()` hook for this and now: ``` % curl http://127.0.0.1:8001/-/actor.json -H 'Authorization: Bearer dstok_eyJhIjoicm9vdCIsImUiOm51bGx9.6O1OxgNTFkAU6uw7xNcmXYX949A' {""actor"": {""id"": ""root"", ""dstok"": true}} ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291392887,https://api.github.com/repos/simonw/datasette/issues/1852,1291392887,IC_kwDOBm6k_c5M-Rd3,9599,simonw,2022-10-26T02:04:48Z,2022-10-26T02:04:48Z,OWNER,"Implemented that `dstok_` prefix and the thing where only the `actor[""id""]` is copied to the `""a""` field.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291290451,https://api.github.com/repos/simonw/datasette/issues/1852,1291290451,IC_kwDOBm6k_c5M94dT,9599,simonw,2022-10-26T00:49:56Z,2022-10-26T00:49:56Z,OWNER,Prefix: `dstok_` - for Datasette signed token.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291289369,https://api.github.com/repos/simonw/datasette/issues/1852,1291289369,IC_kwDOBm6k_c5M94MZ,9599,simonw,2022-10-26T00:47:46Z,2022-10-26T00:47:46Z,OWNER,"The tokens also need something that can be used to differentiate them from alternative token mechanisms that other plugins might provide. Maybe a prefix before the signed value. Prefixes are also useful for scanning to check they were not accidentally committed to source control.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291272280,https://api.github.com/repos/simonw/datasette/issues/1852,1291272280,IC_kwDOBm6k_c5M90BY,9599,simonw,2022-10-26T00:16:09Z,2022-10-26T00:46:21Z,OWNER,"Other options: - `--setting default_api_tokens off` - `--setting signed_api_tokens off` - `--setting allow_create_token off` These feel inconsistent because they don't use the `allow_` prefix - but they're also a bit less ugly to look at. I like that last one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291281243,https://api.github.com/repos/simonw/datasette/issues/1852,1291281243,IC_kwDOBm6k_c5M92Nb,9599,simonw,2022-10-26T00:32:21Z,2022-10-26T00:32:21Z,OWNER,"Rather than duplicating the entire actor into the ""a"" field, maybe just copy the actor ID? Would need to restrict token creation to just actors with an ID set. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291274835,https://api.github.com/repos/simonw/datasette/issues/1852,1291274835,IC_kwDOBm6k_c5M90pT,9599,simonw,2022-10-26T00:20:48Z,2022-10-26T00:22:26Z,OWNER,"Tests failed because I added a view without also adding documentation! I forgot that the deploy still goes out for branches other than `main` even if the tests aren't passing: https://github.com/simonw/datasette/blob/c7dd76c26257ded5bcdfd0570e12412531b8b88f/.github/workflows/deploy-latest.yml#L34-L38","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291273609,https://api.github.com/repos/simonw/datasette/issues/1852,1291273609,IC_kwDOBm6k_c5M90WJ,9599,simonw,2022-10-26T00:18:40Z,2022-10-26T00:18:40Z,OWNER,"Another thought about tokens that can act on behalf of the user. Imagine a user has permission to access a table. They create a token that can create that table... but then their permission is revoked. It would be bad if they could still use that token they created earlier to access that table! On that basis, I think the model described above where tokens mainly work to provide an ""act on behalf of this actor"" - but with optional additional constraints - is a good one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291272612,https://api.github.com/repos/simonw/datasette/issues/1852,1291272612,IC_kwDOBm6k_c5M90Gk,9599,simonw,2022-10-26T00:16:53Z,2022-10-26T00:16:53Z,OWNER,Next step: make these tokens actually do something.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291272414,https://api.github.com/repos/simonw/datasette/issues/1852,1291272414,IC_kwDOBm6k_c5M90De,9599,simonw,2022-10-26T00:16:28Z,2022-10-26T00:16:28Z,OWNER,If I'm going to change the naming conventions for settings I should do it before Datasette 1.0.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291271580,https://api.github.com/repos/simonw/datasette/issues/1852,1291271580,IC_kwDOBm6k_c5M9z2c,9599,simonw,2022-10-26T00:14:49Z,2022-10-26T00:15:06Z,OWNER,"If I'm going to have a setting to disable this feature I need to decide what it will be called. Closest existing setting is this one, since it's for a feature that is turned on by default: datasette mydatabase.db --setting allow_download off So maybe this? datasette mydatabase.db --setting allow_signed_api_tokens off I like `allow_signed_api_tokens` more than `allow_api_tokens` because if you install a plugin such as https://datasette.io/plugins/datasette-auth-tokens then API tokens will work even though you disabled this default signed token feature. `allow_signed_api_tokens` does feel a bit clumsy/verbose though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291270227,https://api.github.com/repos/simonw/datasette/issues/1852,1291270227,IC_kwDOBm6k_c5M9zhT,9599,simonw,2022-10-26T00:12:18Z,2022-10-26T00:12:18Z,OWNER,Demo is now live at https://latest-1-0-dev.datasette.io/-/create-token - visit https://latest-1-0-dev.datasette.io/login-as-root first to sign in.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291269607,https://api.github.com/repos/simonw/datasette/issues/1852,1291269607,IC_kwDOBm6k_c5M9zXn,9599,simonw,2022-10-26T00:11:15Z,2022-10-26T00:11:15Z,OWNER,"If you click ""Create token"" for ""Token never expires"" multiple times you currently get exactly the same token each time, since it's just a signed token containing a copy of your actor dictionary. I'm not sure if I like that. I could give each token a random ID (maybe using `secrets.token_hex()`) such that different tokens have different identities, which would be useful for logging and auditing and maybe even revocation at some point in the future.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291268380,https://api.github.com/repos/simonw/datasette/issues/1852,1291268380,IC_kwDOBm6k_c5M9zEc,9599,simonw,2022-10-26T00:09:06Z,2022-10-26T00:09:06Z,OWNER,"Demo: ![token-demo](https://user-images.githubusercontent.com/9599/197904595-e5651d6c-bafc-4124-b762-71ad94c06ced.gif) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291243333,https://api.github.com/repos/simonw/datasette/issues/1852,1291243333,IC_kwDOBm6k_c5M9s9F,9599,simonw,2022-10-25T23:25:13Z,2022-10-25T23:25:13Z,OWNER,"A `/-/debug-token` page that can take a token and decode it to show you how long until it expires, what actor it represents and the permissions it has will be useful as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291234262,https://api.github.com/repos/simonw/datasette/issues/1852,1291234262,IC_kwDOBm6k_c5M9qvW,9599,simonw,2022-10-25T23:11:23Z,2022-10-25T23:11:23Z,OWNER,I'm going to build an initial `/-/create-token` interface which just bakes a token with the current actor in it and an optional expiry timestamp. I'll try the limited permissions thing later.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291233652,https://api.github.com/repos/simonw/datasette/issues/1852,1291233652,IC_kwDOBm6k_c5M9ql0,9599,simonw,2022-10-25T23:10:20Z,2022-10-25T23:10:44Z,OWNER,"In which case the token would need to duplicate the current `actor` and then add extra constraints. So maybe the token design looks like this: ```json { ""a"": { ""copy_of"": ""actor_creating_token""}, ""p"": { ""t"": ""... the thing designed earlier, with those permissions in it"" }, ""e"": ""integer timestamp when token expires"" } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291232589,https://api.github.com/repos/simonw/datasette/issues/1852,1291232589,IC_kwDOBm6k_c5M9qVN,9599,simonw,2022-10-25T23:08:37Z,2022-10-25T23:08:37Z,OWNER,"... so maybe there's a way to create a token that inherits the exact permissions of the actor that created the token? That could even be a default mode for tokens, with an option to then further restrict permissions if desired.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291231651,https://api.github.com/repos/simonw/datasette/issues/1852,1291231651,IC_kwDOBm6k_c5M9qGj,9599,simonw,2022-10-25T23:07:17Z,2022-10-25T23:07:17Z,OWNER,"Interesting challenge: what permissions should users be allowed to grant to tokens? Clearly a user should not be able to create a token with a permission that the user themselves does not have. And should there be a permission that allows people to create tokens? I think so.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291227942,https://api.github.com/repos/simonw/datasette/issues/1852,1291227942,IC_kwDOBm6k_c5M9pMm,9599,simonw,2022-10-25T23:01:18Z,2022-10-25T23:01:18Z,OWNER,"Datasette currently defaults to having everything public-readable by default, unless a permission plugin changes that default. In thinking more about this API mechanism, I realized that it might be good to have a mode where Datasette _doesn't_ default to public everything. Maybe `datasette --private` to start it like that? Might even be an opportunity to get rid of the current slightly confusing mechanism where permission checks can announce that they should default to true: https://github.com/simonw/datasette/blob/c7dd76c26257ded5bcdfd0570e12412531b8b88f/datasette/views/database.py#L152-L154","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1851#issuecomment-1291226367,https://api.github.com/repos/simonw/datasette/issues/1851,1291226367,IC_kwDOBm6k_c5M9oz_,9599,simonw,2022-10-25T22:58:30Z,2022-10-25T22:58:30Z,OWNER,"The `datasette insert` concept included plugin support, with the idea of being able to support things like SpatiaLite files: - #1160 I think this API mechanism is going to be a bit less exciting than that - it will be low-level for inserting rows, and if you want to do something fancier you can use a canned query that feeds incoming GeoJSON to a SpatiaLite function instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/sqlite-utils/issues/505#issuecomment-1291216193,https://api.github.com/repos/simonw/sqlite-utils/issues/505,1291216193,IC_kwDOCGYnMM5M9mVB,9599,simonw,2022-10-25T22:41:16Z,2022-10-25T22:41:16Z,OWNER,Tweeted about it here: https://twitter.com/simonw/status/1585038766678609921,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423182778,Release sqlite-utils 3.30, https://github.com/simonw/sqlite-utils/issues/505#issuecomment-1291203911,https://api.github.com/repos/simonw/sqlite-utils/issues/505,1291203911,IC_kwDOCGYnMM5M9jVH,9599,simonw,2022-10-25T22:21:02Z,2022-10-25T22:21:02Z,OWNER,"- Now tested against Python 3.11. ([#502](https://github.com/simonw/sqlite-utils/issues/502)) - New `table.search_sql(include_rank=True)` option, which adds a `rank` column to the generated SQL. Thanks, Jacob Chapman. ([#480](https://github.com/simonw/sqlite-utils/pull/480)) - Progress bars now display for newline-delimited JSON files using the `--nl` option. Thanks, Mischa Untaga. ([#485](https://github.com/simonw/sqlite-utils/issues/485)) - New `db.close()` method. ([#504](https://github.com/simonw/sqlite-utils/issues/504)) - Conversion functions passed to [table.convert(...)](https://sqlite-utils.datasette.io/en/stable/python-api.html#python-api-convert) can now return lists or dictionaries, which will be inserted into the database as JSON strings. ([#495](https://github.com/simonw/sqlite-utils/issues/495)) - `sqlite-utils install` and `sqlite-utils uninstall` commands for installing packages into the same virtual environment as `sqlite-utils`, [described here](https://sqlite-utils.datasette.io/en/stable/cli.html#cli-install). ([#483](https://github.com/simonw/sqlite-utils/issues/483)) - New [sqlite_utils.utils.flatten()](https://sqlite-utils.datasette.io/en/stable/reference.html#reference-utils-flatten) utility function. ([#500](https://github.com/simonw/sqlite-utils/issues/500)) - Documentation on [using Just](https://sqlite-utils.datasette.io/en/stable/contributing.html#contributing-just) to run tests, linters and build documentation. - Documentation now covers the [Release process](https://sqlite-utils.datasette.io/en/stable/contributing.html#release-process) for this package.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423182778,Release sqlite-utils 3.30, https://github.com/simonw/sqlite-utils/issues/496#issuecomment-1291170072,https://api.github.com/repos/simonw/sqlite-utils/issues/496,1291170072,IC_kwDOCGYnMM5M9bEY,9599,simonw,2022-10-25T21:36:12Z,2022-10-25T21:36:12Z,OWNER,"I was going to suggest using `db.table(name)` instead of `db[name]` - but it looks like that method will have the same problem: https://github.com/simonw/sqlite-utils/blob/defa2974c6d3abc19be28d6b319649b8028dc966/sqlite_utils/db.py#L497-L506 I could change `sqlite-utils` so `db.table(name)` always returns a table and you need to call `db.view(name)` if you want to access a view - that would require bumping to 4.0 though. I'm not convinced that's the best approach here either.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1393202060,devrel/python api: Pylance type hinting, https://github.com/simonw/sqlite-utils/issues/496#issuecomment-1291167887,https://api.github.com/repos/simonw/sqlite-utils/issues/496,1291167887,IC_kwDOCGYnMM5M9aiP,9599,simonw,2022-10-25T21:33:25Z,2022-10-25T21:33:25Z,OWNER,"I do care about this, but I'm not hugely experienced with types yet so I'm open to suggestions about how to do it!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1393202060,devrel/python api: Pylance type hinting, https://github.com/simonw/sqlite-utils/issues/493#issuecomment-1291166273,https://api.github.com/repos/simonw/sqlite-utils/issues/493,1291166273,IC_kwDOCGYnMM5M9aJB,9599,simonw,2022-10-25T21:31:15Z,2022-10-25T21:31:15Z,OWNER,"Based on the docs here I tried the following too: https://docutils.sourceforge.io/docs/user/smartquotes.html#description - `\--` - `\\--` - `\\-\\-` - `\-\-` But none of them had the desired effect in this particular piece of markup: the :ref:`insert \--convert ` I think because this is text inside a `:ref:` block, not regular text. Consider the following: The \--convert and the :ref:`insert \--convert ` and It's rendered like this: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1386562662,Tiny typographical error in install/uninstall docs, https://github.com/simonw/sqlite-utils/issues/495#issuecomment-1291159549,https://api.github.com/repos/simonw/sqlite-utils/issues/495,1291159549,IC_kwDOCGYnMM5M9Yf9,9599,simonw,2022-10-25T21:23:01Z,2022-10-25T21:23:01Z,OWNER,"I've decided not to explicitly document this, since it's consistent with how other parts of the library work already.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1392690202,Support JSON values returned from .convert() functions, https://github.com/simonw/sqlite-utils/issues/495#issuecomment-1291152433,https://api.github.com/repos/simonw/sqlite-utils/issues/495,1291152433,IC_kwDOCGYnMM5M9Wwx,9599,simonw,2022-10-25T21:14:54Z,2022-10-25T21:14:54Z,OWNER,"There is a case where the function can return a dictionary at the moment: `multi=True` ```python table.convert( ""title"", lambda v: {""upper"": v.upper(), ""lower"": v.lower()}, multi=True ) ``` But I think this change is still compatible with that. if you don't use `multi=True` then the return value will be stringified. If you DO use `multi=True` then something like this could work: ```python table.convert( ""title"", lambda v: {""upper"": {""str"": v.upper()}, ""lower"": {""str"": v.lower()}}, multi=True ) ``` This would result in a `upper` and `lower` column, each containing the JSON string `{""str"": ""UPPERCASE""}`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1392690202,Support JSON values returned from .convert() functions, https://github.com/simonw/sqlite-utils/issues/495#issuecomment-1291149509,https://api.github.com/repos/simonw/sqlite-utils/issues/495,1291149509,IC_kwDOCGYnMM5M9WDF,9599,simonw,2022-10-25T21:12:11Z,2022-10-25T21:12:11Z,OWNER,"This makes sense to me. There are other places in the codebase where JSON is automatically stringified: https://github.com/simonw/sqlite-utils/blob/c7e4308e6f49d929704163531632e558f9646e4a/sqlite_utils/db.py#L2759-L2766 I don't see why the return value from a convert function shouldn't do the same thing. Since this will result in previous errors working, I don't think it warrants a major version bump either.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1392690202,Support JSON values returned from .convert() functions, https://github.com/simonw/sqlite-utils/issues/497#issuecomment-1291146850,https://api.github.com/repos/simonw/sqlite-utils/issues/497,1291146850,IC_kwDOCGYnMM5M9VZi,9599,simonw,2022-10-25T21:09:28Z,2022-10-25T21:09:28Z,OWNER,"Yeah, `table.columns` and `table.columns_dict` are meant to handle this: https://sqlite-utils.datasette.io/en/stable/python-api.html#columns","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1393212964,column_names, https://github.com/simonw/sqlite-utils/issues/504#issuecomment-1291136971,https://api.github.com/repos/simonw/sqlite-utils/issues/504,1291136971,IC_kwDOCGYnMM5M9S_L,9599,simonw,2022-10-25T21:00:29Z,2022-10-25T21:00:29Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Database.close,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423069384,"db.close() method, calling db.conn.close()", https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291124413,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291124413,IC_kwDOCGYnMM5M9P69,9599,simonw,2022-10-25T20:47:34Z,2022-10-25T20:47:34Z,OWNER,TIL about this: https://til.simonwillison.net/python/os-remove-windows,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291122389,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291122389,IC_kwDOCGYnMM5M9PbV,9599,simonw,2022-10-25T20:45:43Z,2022-10-25T20:45:43Z,OWNER,That fixed it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291115986,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291115986,IC_kwDOCGYnMM5M9N3S,9599,simonw,2022-10-25T20:39:24Z,2022-10-25T20:39:24Z,OWNER,"Used `psutil` to confirm that closing a SQLite connection closes the underlying file: https://til.simonwillison.net/python/too-many-open-files-psutil ```pycon >>> import psutil >>> import sqlite3 >>> for f in psutil.Process().open_files(): print(f) ... >>> sqlite3.connect(""/tmp/blah.db"") >>> conn = _ >>> for f in psutil.Process().open_files(): print(f) ... popenfile(path='/private/tmp/blah.db', fd=3) >>> conn.close() >>> for f in psutil.Process().open_files(): print(f) ... >>> ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291111357,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291111357,IC_kwDOCGYnMM5M9Mu9,9599,simonw,2022-10-25T20:36:06Z,2022-10-25T20:36:06Z,OWNER,... or maybe Windows doesn't like attempts to remove a file that the process has opened?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291103021,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291103021,IC_kwDOCGYnMM5M9Kst,9599,simonw,2022-10-25T20:32:01Z,2022-10-25T20:32:01Z,OWNER,"This test reliably fails on Windows with Python 3.11. I'm going to skip the test for the moment to get back to green CI... but I'll leave this issue open. This is definitely concerning, I just don't have the right local environment to solve this at the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291093581,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291093581,IC_kwDOCGYnMM5M9IZN,9599,simonw,2022-10-25T20:23:00Z,2022-10-25T20:23:00Z,OWNER,"I'm not hugely happy with my fix there: https://github.com/simonw/sqlite-utils/blob/c5d7ec1dd71fa1dce829bc8bb82b639018befd63/sqlite_utils/db.py#L321-L328 The problem here was that in the case where the `os.remove()` failed the `self.conn` property was NOT being set to a valid connection - which caused `__repr__` to fail later on. So now I catch the `os.remove()` error, set `self.conn` to a memory connection, then raise the error again.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291088108,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291088108,IC_kwDOCGYnMM5M9HDs,9599,simonw,2022-10-25T20:17:36Z,2022-10-25T20:17:36Z,OWNER,"Now `mypy` is failing: ``` sqlite_utils/db.py:474: error: Item ""None"" of ""Optional[Any]"" has no attribute ""execute"" sqlite_utils/db.py:476: error: Item ""None"" of ""Optional[Any]"" has no attribute ""execute"" sqlite_utils/db.py:486: error: Item ""None"" of ""Optional[Any]"" has no attribute ""executescript"" sqlite_utils/db.py:603: error: Item ""None"" of ""Optional[Any]"" has no attribute ""__enter__"" sqlite_utils/db.py:603: error: Item ""None"" of ""Optional[Any]"" has no attribute ""__exit__"" sqlite_utils/db.py:604: error: Item ""None"" of ""Optional[Any]"" has no attribute ""execute"" sqlite_utils/db.py:607: error: Item ""None"" of ""Optional[Any]"" has no attribute ""execute"" sqlite_utils/db.py:1082: error: Item ""None"" of ""Optional[Any]"" has no attribute ""__enter__"" sqlite_utils/db.py:1082: error: Item ""None"" of ""Optional[Any]"" has no attribute ""__exit__"" sqlite_utils/db.py:1083: error: Item ""None"" of ""Optional[Any]"" has no attribute ""cursor"" sqlite_utils/db.py:1155: error: Item ""None"" of ""Optional[Any]"" has no attribute ""enable_load_extension"" sqlite_utils/db.py:1156: error: Item ""None"" of ""Optional[Any]"" has no attribute ""load_extension"" Found 12 errors in 1 file (checked 51 source files) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291083188,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291083188,IC_kwDOCGYnMM5M9F20,9599,simonw,2022-10-25T20:12:52Z,2022-10-25T20:12:52Z,OWNER,"Failed again, but just noticed this: https://github.com/simonw/sqlite-utils/actions/runs/3323932266/jobs/5494890223 ``` > Database(filepath, recreate=True)[""t2""].insert({""foo"": ""bar""}) tests\test_recreate.py:31: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError(""'Database' object has no attribute 'conn'"") raised in repr()] Database object at 0x29fc125aa90> ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291076031,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291076031,IC_kwDOCGYnMM5M9EG_,9599,simonw,2022-10-25T20:06:28Z,2022-10-25T20:06:28Z,OWNER,"This is the failing test: https://github.com/simonw/sqlite-utils/blob/7b2d1c0ffd0b874e280292b926f328a61cb31e2c/tests/test_recreate.py#L21-L32 I'm going to try a different way of creating the temporary file: https://docs.pytest.org/en/7.1.x/how-to/tmp_path.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291071627,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291071627,IC_kwDOCGYnMM5M9DCL,9599,simonw,2022-10-25T20:02:18Z,2022-10-25T20:02:18Z,OWNER,Passes on Windows with other Python versions for some reason.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/datasette/issues/1854#issuecomment-1291047214,https://api.github.com/repos/simonw/datasette/issues/1854,1291047214,IC_kwDOBm6k_c5M89Eu,9599,simonw,2022-10-25T19:39:36Z,2022-10-25T19:39:48Z,OWNER,"This pattern should work (for the http server at least): ```python # Loop until port 8041 serves traffic while True: try: httpx.get(""http://localhost:8041/"") break except httpx.ConnectError: time.sleep(0.1) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1422973111,Flaky test: test_serve_localhost_http, https://github.com/simonw/datasette/issues/1854#issuecomment-1291046958,https://api.github.com/repos/simonw/datasette/issues/1854,1291046958,IC_kwDOBm6k_c5M89Au,9599,simonw,2022-10-25T19:39:22Z,2022-10-25T19:39:22Z,OWNER,"Here's the code that starts those various servers: https://github.com/simonw/datasette/blob/613ad05c095f92653221db267ef53d54d00cdfbb/tests/conftest.py#L104-L177 I don't like those `time.sleep(1.5)` lines much - I'm going to try polling for readiness instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1422973111,Flaky test: test_serve_localhost_http, https://github.com/simonw/datasette/issues/1854#issuecomment-1291045997,https://api.github.com/repos/simonw/datasette/issues/1854,1291045997,IC_kwDOBm6k_c5M88xt,9599,simonw,2022-10-25T19:38:28Z,2022-10-25T19:38:28Z,OWNER,"Also: ``` @pytest.mark.serial @pytest.mark.skipif( not hasattr(socket, ""AF_UNIX""), reason=""Requires socket.AF_UNIX support"" ) def test_serve_unix_domain_socket(ds_unix_domain_socket_server): _, uds = ds_unix_domain_socket_server transport = httpx.HTTPTransport(uds=uds) client = httpx.Client(transport=transport) > response = client.get(""http://localhost/_memory.json"") /home/runner/work/datasette/datasette/tests/test_cli_serve_server.py:35: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/httpx/_client.py:1039: in get return self.request( /opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/httpx/_client.py:815: in request return self.send(request, auth=auth, follow_redirects=follow_redirects) /opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/httpx/_client.py:902: in send response = self._send_handling_auth( /opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/httpx/_client.py:930: in _send_handling_auth response = self._send_handling_redirects( /opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/httpx/_client.py:967: in _send_handling_redirects response = self._send_single_request(request) /opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/httpx/_client.py:1003: in _send_single_request response = transport.handle_request(request) /opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/httpx/_transports/default.py:217: in handle_request with map_httpcore_exceptions(): /opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/contextlib.py:153: in __exit__ self.gen.throw(typ, value, traceback) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ @contextlib.contextmanager def map_httpcore_exceptions() -> typing.Iterator[None]: try: yield except Exception as exc: # noqa: PIE-786 mapped_exc = None for from_exc, to_exc in HTTPCORE_EXC_MAP.items(): if not isinstance(exc, from_exc): continue # We want to map to the most specific exception we can find. # Eg if `exc` is an `httpcore.ReadTimeout`, we want to map to # `httpx.ReadTimeout`, not just `httpx.TimeoutException`. if mapped_exc is None or issubclass(to_exc, mapped_exc): mapped_exc = to_exc if mapped_exc is None: # pragma: nocover raise message = str(exc) > raise mapped_exc(message) from exc E httpx.ConnectError: [Errno 2] No such file or directory ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1422973111,Flaky test: test_serve_localhost_http, https://github.com/simonw/datasette/issues/1853#issuecomment-1291036623,https://api.github.com/repos/simonw/datasette/issues/1853,1291036623,IC_kwDOBm6k_c5M86fP,9599,simonw,2022-10-25T19:28:56Z,2022-10-25T19:28:56Z,OWNER,"Opened an issue here: - https://github.com/coleifer/pysqlite3/issues/43","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1422915587,Upgrade Datasette Docker to Python 3.11, https://github.com/simonw/datasette/issues/1853#issuecomment-1291032289,https://api.github.com/repos/simonw/datasette/issues/1853,1291032289,IC_kwDOBm6k_c5M85bh,9599,simonw,2022-10-25T19:24:27Z,2022-10-25T19:24:27Z,OWNER,https://latest.datasette.io/-/versions now shows 3.11.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1422915587,Upgrade Datasette Docker to Python 3.11, https://github.com/simonw/sqlite-utils/issues/502#issuecomment-1291029761,https://api.github.com/repos/simonw/sqlite-utils/issues/502,1291029761,IC_kwDOCGYnMM5M840B,9599,simonw,2022-10-25T19:21:44Z,2022-10-25T19:21:44Z,OWNER,"Replicated locally using a fresh virtual environment with Python 3.11 and: pytest -k test_query_invalid_function","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1422954582,Fix tests for Python 3.11, https://github.com/simonw/datasette/issues/1853#issuecomment-1291023926,https://api.github.com/repos/simonw/datasette/issues/1853,1291023926,IC_kwDOBm6k_c5M83Y2,9599,simonw,2022-10-25T19:15:49Z,2022-10-25T19:15:49Z,OWNER,"This broke the deploy of `https://latest.datasette.io/` - because it tries to install `pysqlite3-binary` which doesn't have a 3.11 release yet: https://github.com/simonw/datasette/blob/2e9751672d4fe329b3c359d5b7b1992283185820/.github/workflows/deploy-latest.yml#L77 I started using that for the `latest.datasette.io` demo in https://github.com/simonw/datasette/commit/a970276b9999687b96c5e11ea1c817d814f5d267 because I wanted a version of SQLite that supported generated columns. Those were added in [SQLite 3.31.0](https://www.sqlite.org/changes.html#version_3_31_0) - and the SQLite version in the new base image is 3.34.1 - so I don't actually need `pysqlite3-binary` any more.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1422915587,Upgrade Datasette Docker to Python 3.11, https://github.com/simonw/datasette/issues/1853#issuecomment-1291012637,https://api.github.com/repos/simonw/datasette/issues/1853,1291012637,IC_kwDOBm6k_c5M80od,9599,simonw,2022-10-25T19:04:03Z,2022-10-25T19:04:09Z,OWNER,"And tested `datasette package` like this: ``` datasette package fixtures.db -t datasette-package-python-upgrade-3-11 ``` Then: ``` docker run -p 8081:8001 datasette-package-python-upgrade-3-11 ``` And tested it like this: ``` curl http://localhost:8081/-/versions.json | jq ``` Output: ``` { ""python"": { ""version"": ""3.11.0"", ""full"": ""3.11.0 (main, Oct 25 2022, 05:00:36) [GCC 10.2.1 20210110]"" }, ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1422915587,Upgrade Datasette Docker to Python 3.11, https://github.com/simonw/datasette/issues/1853#issuecomment-1291009987,https://api.github.com/repos/simonw/datasette/issues/1853,1291009987,IC_kwDOBm6k_c5M8z_D,9599,simonw,2022-10-25T19:01:23Z,2022-10-25T19:01:23Z,OWNER,"Also tested by running this locally: datasette publish cloudrun fixtures.db --service issue-1853 https://issue-1853-j7hipcg4aq-uc.a.run.app/-/versions now shows Python 3.11.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1422915587,Upgrade Datasette Docker to Python 3.11, https://github.com/simonw/datasette/issues/1853#issuecomment-1291006149,https://api.github.com/repos/simonw/datasette/issues/1853,1291006149,IC_kwDOBm6k_c5M8zDF,9599,simonw,2022-10-25T18:57:33Z,2022-10-25T18:57:33Z,OWNER,"Ran the upgrade on the Datasette Cloud image first, works fine there. https://simon.datasette.cloud/-/versions shows me: ``` { ""python"": { ""version"": ""3.11.0"", ""full"": ""3.11.0 (main, Oct 25 2022, 05:00:36) [GCC 10.2.1 20210110]"" }, ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1422915587,Upgrade Datasette Docker to Python 3.11, https://github.com/simonw/datasette/issues/1853#issuecomment-1290995178,https://api.github.com/repos/simonw/datasette/issues/1853,1290995178,IC_kwDOBm6k_c5M8wXq,9599,simonw,2022-10-25T18:46:33Z,2022-10-25T18:46:33Z,OWNER,"I ran a very crude benchmark on my laptop using Locust (against the official macOS packages from www.python.org for Python 3.10 and Python 3.11) and saw a substantial speed increase: 533.89 requests/second on 3.11 413.56 requests/second on 3.10 That was from running Locust against this `locustfile.py`: ```python from locust import HttpUser, task class CounterOne(HttpUser): @task def hello(self): self.client.get(""/-/static/app.css"") ``` Using: locust --headless --users 4 --spawn-rate 4 -H http://127.0.0.1:8001","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1422915587,Upgrade Datasette Docker to Python 3.11, https://github.com/simonw/datasette/issues/1851#issuecomment-1289865317,https://api.github.com/repos/simonw/datasette/issues/1851,1289865317,IC_kwDOBm6k_c5M4chl,9599,simonw,2022-10-25T01:42:47Z,2022-10-25T01:42:47Z,OWNER,"This is going to tie into Datasette's existing permissions mechanism, so plugins will be able to define their own custom mechanisms for tokens to be attached to a specific identity: https://docs.datasette.io/en/stable/authentication.html There's only one plugin for API tokens at the moment, which is this one: https://datasette.io/plugins/datasette-auth-tokens I'm actually planning on adding another, default token mechanism to Datasette itself as part of this work: - #1852 It may well be that `datasette-sandstorm-support` needs to add something custom here too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1852#issuecomment-1289776707,https://api.github.com/repos/simonw/datasette/issues/1852,1289776707,IC_kwDOBm6k_c5M4G5D,9599,simonw,2022-10-24T23:29:03Z,2022-10-24T23:29:03Z,OWNER,"I'm going to implement the first version of this token mechanism using permissions that exist already. Right now that's: https://docs.datasette.io/en/0.62/authentication.html#built-in-permissions Here are the shortcuts I'll use for them: - `view-instance` - `vi` - `view-database` - `vd` - `view-database-download` - `vdd` - `view-table` - `vt` - `view-query` - `vq` - `execute-sql` - `es` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1289775162,https://api.github.com/repos/simonw/datasette/issues/1852,1289775162,IC_kwDOBm6k_c5M4Gg6,9599,simonw,2022-10-24T23:27:00Z,2022-10-24T23:27:00Z,OWNER,"Might be neat for API tokens to be signed with an additional secret than can be rotated independently of `DATASETTE_SECRET` itself, in order to invalidate all tokens without needing to invalidate logged in users too. But again, I don't want to implement something like that until I see an actual need for it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1289774183,https://api.github.com/repos/simonw/datasette/issues/1852,1289774183,IC_kwDOBm6k_c5M4GRn,9599,simonw,2022-10-24T23:25:52Z,2022-10-24T23:25:52Z,OWNER,"... also, maybe there should be a UI (perhaps on that page) for resetting the Datasette secret? Useful for emergency invalidation of all tokens. No, I'm not going to build that unless someone asks for it. Restarting the server with a fresh secret should be easy enough.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1289773634,https://api.github.com/repos/simonw/datasette/issues/1852,1289773634,IC_kwDOBm6k_c5M4GJC,9599,simonw,2022-10-24T23:25:06Z,2022-10-24T23:25:06Z,OWNER,"If you start Datasette without providing a `DATASETTE_SECRET` environment variable of `--secret` option it creates a random signing secret that only lasts for the lifetime of the server. This means any signed API tokens you create will stop working if the server restarts. I think the `/-/create-token` UI should know when this happens and show a warning message about it, to avoid confusion.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1289766513,https://api.github.com/repos/simonw/datasette/issues/1852,1289766513,IC_kwDOBm6k_c5M4EZx,9599,simonw,2022-10-24T23:16:00Z,2022-10-24T23:16:00Z,OWNER,"Here's what that example looks like signed: ```python from datasette.app import Datasette ds = Datasette() ds.sign('{""t"":{""a"":[""ir"",""ur"",""dr""],""d"":{""fixtures"":[""ir"",""ur"",""dr""]},""t"":{""fixtures"":{""searchable"":[""ir""]}}}}') ``` ``` .eJxTqo5RKolRsgJSiUAqOkYpsyhGSSdGqRRCpQCpWBANUZOWWVFSWpRajFNprQ7cPCS1QF5xamJRckZiUk4qQm9sLRAoAQCC8yph.O0Gaej6-VOLbbtPq7xU6T77jEO0 ``` That's 129 characters. Note that Datasette doesn't have its own mechanism for signing things for a specific duration yet: https://docs.datasette.io/en/stable/internals.html#sign-value-namespace-default So I'll need to add a `""e"": 1666739744` field with the UTC timestamp at which the token should expire.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1289733483,https://api.github.com/repos/simonw/datasette/issues/1852,1289733483,IC_kwDOBm6k_c5M38Vr,9599,simonw,2022-10-24T22:54:37Z,2022-10-24T23:12:10Z,OWNER,"Token design concept: ```json { ""t"": { ""a"": [""ir"", ""ur"", ""dr""], ""d"": { ""fixtures"": [""ir"", ""ur"", ""dr""] }, ""t"": { ""fixtures"": { ""searchable"": [""ir""] } } } } ``` That JSON would be minified and signed. Minified version of the above looks like this (101 characters): `{""t"":{""a"":[""ir"",""ur"",""dr""],""d"":{""fixtures"":[""ir"",""ur"",""dr""]},""t"":{""fixtures"":{""searchable"":[""ir""]}}}}` The `""t""` key shows this is a token that as a default API key. `""a""` means ""all"" - these are permissions that have been granted on all tables and databases. `""d""` means ""databases"" - this is a way to set permissions for all tables in a specific database. `""t""` means ""tables"" - this lets you set permissions at a finely grained table level. Then the permissions themselves are two character codes which are shortened versions - so: - `ir` = `insert-row` - `ur` = `update-row` - `dr` = `delete-row`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1289718660,https://api.github.com/repos/simonw/datasette/issues/1852,1289718660,IC_kwDOBm6k_c5M34uE,9599,simonw,2022-10-24T22:35:01Z,2022-10-24T22:35:01Z,OWNER,"Maybe these tokens can be restricted to specific databases and tables when they are first created? Since they're signed tokens, I could bundle a bunch of extra stuff in them - this token is allowed to do these permissions against these tables/rows for example. General wisdom seems to be that 8KB is a sensible maximum length for this kind of token, which is easily long enough to fit in a bunch of database / table / permissions.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1851#issuecomment-1289713513,https://api.github.com/repos/simonw/datasette/issues/1851,1289713513,IC_kwDOBm6k_c5M33dp,9599,simonw,2022-10-24T22:29:58Z,2022-10-24T22:30:15Z,OWNER,"Interesting open question: how should validation errors (if any) be returned? The two forms of validation I can think of at first are: - Missing keys which are marked as `not null` in the schema - Keys that do not match to existing columns (if you didn't pass `""alter"": true`, an option I am considering)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1850#issuecomment-1289707357,https://api.github.com/repos/simonw/datasette/issues/1850,1289707357,IC_kwDOBm6k_c5M319d,9599,simonw,2022-10-24T22:23:12Z,2022-10-24T22:23:12Z,OWNER,Here's the implementation of `datasette-auth-tokens`: https://github.com/simonw/datasette-auth-tokens/blob/main/datasette_auth_tokens/__init__.py,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421529723,Write API in Datasette core, https://github.com/simonw/datasette/issues/1850#issuecomment-1289706439,https://api.github.com/repos/simonw/datasette/issues/1850,1289706439,IC_kwDOBm6k_c5M31vH,9599,simonw,2022-10-24T22:22:17Z,2022-10-24T22:22:17Z,OWNER,"API authentication will be via `Authorization: Bearer XXX` request headers. I'm inclined to add a default token mechanism to Datasette based on tokens that are signed with the `DATASETTE_SECRET`. Maybe the root user can access `/-/create-token` which provides a UI for generating a time-limited signed token? Could also have a `datasette create-token` command for creating such tokens at the command-line. Plugins can then define alternative ways of creating tokens, such as the existing https://datasette.io/plugins/datasette-auth-tokens plugin.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421529723,Write API in Datasette core, https://github.com/simonw/datasette/issues/1850#issuecomment-1289703432,https://api.github.com/repos/simonw/datasette/issues/1850,1289703432,IC_kwDOBm6k_c5M31AI,9599,simonw,2022-10-24T22:19:48Z,2022-10-24T22:19:48Z,OWNER,It may turn out that it makes sense to also add a UI for these actions as part of this project. That's still to be determined.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421529723,Write API in Datasette core, https://github.com/simonw/datasette/issues/1850#issuecomment-1289702146,https://api.github.com/repos/simonw/datasette/issues/1850,1289702146,IC_kwDOBm6k_c5M30sC,9599,simonw,2022-10-24T22:19:04Z,2022-10-24T22:19:04Z,OWNER,"This is going to need a whole bunch of new permissions. To review: the existing set of permissions are listed here: https://docs.datasette.io/en/0.62/authentication.html#built-in-permissions - `view-instance` - `view-database` - `view-database-download` - `view-table` - `view-query` - `execute-sql` - `permissions-debug` - `debug-menu` I'm going to reuse database terminology for the new permissions. So first draft of those is: - `insert-row` - `update-row` - `delete-row` - `create-table` - `drop-table` - `alter-table`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421529723,Write API in Datasette core, https://github.com/simonw/datasette/issues/1850#issuecomment-1289696171,https://api.github.com/repos/simonw/datasette/issues/1850,1289696171,IC_kwDOBm6k_c5M3zOr,9599,simonw,2022-10-24T22:15:57Z,2022-10-24T22:15:57Z,OWNER,"I'm going to treat this as a bit of a research spike, at least until I like the direction it is going enough to commit to it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421529723,Write API in Datasette core, https://github.com/simonw/datasette/issues/1849#issuecomment-1288384907,https://api.github.com/repos/simonw/datasette/issues/1849,1288384907,IC_kwDOBm6k_c5MyzGL,9599,simonw,2022-10-24T04:04:02Z,2022-10-24T04:04:02Z,OWNER,"Refs: - #1831 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1420174670,NoneType' object has no attribute 'actor', https://github.com/simonw/datasette/issues/1849#issuecomment-1288384098,https://api.github.com/repos/simonw/datasette/issues/1849,1288384098,IC_kwDOBm6k_c5Myy5i,9599,simonw,2022-10-24T04:03:09Z,2022-10-24T04:03:09Z,OWNER,"Looks like the new breadcrumbs code can't handle the case where `request` is `None`. Need a test that demonstrates this too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1420174670,NoneType' object has no attribute 'actor', https://github.com/simonw/datasette/issues/1848#issuecomment-1288340476,https://api.github.com/repos/simonw/datasette/issues/1848,1288340476,IC_kwDOBm6k_c5MyoP8,9599,simonw,2022-10-24T02:50:29Z,2022-10-24T02:50:29Z,OWNER,"https://latest.datasette.io/_internal now looks like this: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1420090659,Private database page should show padlock on every table, https://github.com/simonw/datasette/issues/1848#issuecomment-1288330238,https://api.github.com/repos/simonw/datasette/issues/1848,1288330238,IC_kwDOBm6k_c5Mylv-,9599,simonw,2022-10-24T02:34:41Z,2022-10-24T02:34:41Z,OWNER,"Tested my fix with this `metadata.yml`: ```yaml databases: fixtures: allow: id: root tables: 123_starts_with_digits: allow: true ``` Signed in as root I saw this - showing that the `123_starts_with_digits` table is public: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1420090659,Private database page should show padlock on every table, https://github.com/simonw/datasette/issues/1848#issuecomment-1288327467,https://api.github.com/repos/simonw/datasette/issues/1848,1288327467,IC_kwDOBm6k_c5MylEr,9599,simonw,2022-10-24T02:30:48Z,2022-10-24T02:31:04Z,OWNER,"Here's the code at fault: https://github.com/simonw/datasette/blob/78dad236df730212aa7172f885fd8ec575f0d3ad/datasette/views/database.py#L67-L116 Those checks aren't doing the new cascading permissions thing added in #1829 which means they can't tell that an anonymous user would not be able to se those tbles and queries and views. Should do something like this instead: ```python view_visible, view_private = await self.ds.check_visibility( request.actor, permissions=[ (""view-table"", (database, view_name)), (""view-database"", database), ""view-instance"", ], ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1420090659,Private database page should show padlock on every table, https://github.com/simonw/datasette/issues/1829#issuecomment-1288321630,https://api.github.com/repos/simonw/datasette/issues/1829,1288321630,IC_kwDOBm6k_c5Myjpe,9599,simonw,2022-10-24T02:22:49Z,2022-10-24T02:23:46Z,OWNER,"Visit https://latest.datasette.io/login-as-root and then: https://latest.datasette.io/ https://latest.datasette.io/_internal/columns https://latest.datasette.io/_internal/columns/_internal,columns,cid https://latest.datasette.io/_internal/from_hook That's all as it should be.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1396948693,Table/database that is private due to inherited permissions does not show padlock, https://github.com/simonw/datasette/issues/1829#issuecomment-1288320411,https://api.github.com/repos/simonw/datasette/issues/1829,1288320411,IC_kwDOBm6k_c5MyjWb,9599,simonw,2022-10-24T02:21:19Z,2022-10-24T02:21:19Z,OWNER,Updated docs for `check_visibility()`: https://docs.datasette.io/en/latest/internals.html#await-check-visibility-actor-action-none-resource-none-permissions-none,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1396948693,Table/database that is private due to inherited permissions does not show padlock, https://github.com/simonw/datasette/pull/1842#issuecomment-1288311852,https://api.github.com/repos/simonw/datasette/issues/1842,1288311852,IC_kwDOBm6k_c5MyhQs,9599,simonw,2022-10-24T02:11:12Z,2022-10-24T02:11:12Z,OWNER,"I'm going to construct a `metadata.yml` which makes various databases and tables visible or invisible, then browse them using the root user. `block-instance.yml`: ```yaml allow: id: root ``` `block-database.yml`: ```yaml databases: fixtures: allow: id: root ``` `block-table.yml`: ```yaml databases: fixtures: tables: searchable: allow: id: root ``` `block-query.yml`: ```yaml databases: fixtures: queries: two: sql: select 1 + 1 allow: id: root ``` https://gist.github.com/simonw/2d007ebe43de46d44499c77a2a291756 - checkout that Gist to get all four. I manually tested all four scenarios with root and non-root users and confirmed that they worked correctly and padlocks were shown in the right places.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408561039,check_visibility can now take multiple permissions into account, https://github.com/simonw/datasette/issues/1829#issuecomment-1288308945,https://api.github.com/repos/simonw/datasette/issues/1829,1288308945,IC_kwDOBm6k_c5MygjR,9599,simonw,2022-10-24T02:07:50Z,2022-10-24T02:07:50Z,OWNER,"Useful test: if you sign in as root to https://latest.datasette.io/_internal/columns/_internal,columns,database_name you can see there's no padlock icon on that page or on https://latest.datasette.io/_internal/columns - fixing this bug should fix that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1396948693,Table/database that is private due to inherited permissions does not show padlock, https://github.com/simonw/datasette/pull/1842#issuecomment-1288304224,https://api.github.com/repos/simonw/datasette/issues/1842,1288304224,IC_kwDOBm6k_c5MyfZg,9599,simonw,2022-10-24T02:00:14Z,2022-10-24T02:00:14Z,OWNER,I need to do one last round of manual testing before I merge this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408561039,check_visibility can now take multiple permissions into account, https://github.com/simonw/datasette/issues/1847#issuecomment-1288296235,https://api.github.com/repos/simonw/datasette/issues/1847,1288296235,IC_kwDOBm6k_c5Mydcr,9599,simonw,2022-10-24T01:45:56Z,2022-10-24T01:45:56Z,OWNER,This bug here: https://github.com/simonw/datasette/blob/85d5d2762c13d2b5a8bd9c5ec81c77fe6577121f/tests/test_permissions.py#L485,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1420055377,Both _local_metadata and _metadata_local?, https://github.com/simonw/datasette/issues/1847#issuecomment-1288295713,https://api.github.com/repos/simonw/datasette/issues/1847,1288295713,IC_kwDOBm6k_c5MydUh,9599,simonw,2022-10-24T01:45:13Z,2022-10-24T01:45:13Z,OWNER,"Turns out that was a bug I had introduced while working on that test, and it was the reason I was blocked on finishing work on: - #1829","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1420055377,Both _local_metadata and _metadata_local?, https://github.com/simonw/datasette/issues/1843#issuecomment-1288214953,https://api.github.com/repos/simonw/datasette/issues/1843,1288214953,IC_kwDOBm6k_c5MyJmp,9599,simonw,2022-10-23T22:22:52Z,2022-10-23T22:22:52Z,OWNER,This seems to have fixed it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705,"Intermittent ""Too many open files"" error running tests", https://github.com/simonw/sqlite-utils/issues/501#issuecomment-1282830806,https://api.github.com/repos/simonw/sqlite-utils/issues/501,1282830806,IC_kwDOCGYnMM5MdnHW,9599,simonw,2022-10-18T18:23:36Z,2022-10-18T18:23:36Z,OWNER,"Tests pass now. Updated docs: - https://sqlite-utils.datasette.io/en/latest/cli.html#table-formatted-output - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#query - and many other places on that page","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1413641049,Tests failing due to updated tabulate library, https://github.com/simonw/sqlite-utils/issues/501#issuecomment-1282819035,https://api.github.com/repos/simonw/sqlite-utils/issues/501,1282819035,IC_kwDOCGYnMM5MdkPb,9599,simonw,2022-10-18T18:15:05Z,2022-10-18T18:15:05Z,OWNER,I'm going to skip the cog test on Python 3.6 to address this. The documentation on the website will show the available list of options for 3.7 and higher.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1413641049,Tests failing due to updated tabulate library, https://github.com/simonw/sqlite-utils/issues/501#issuecomment-1282817901,https://api.github.com/repos/simonw/sqlite-utils/issues/501,1282817901,IC_kwDOCGYnMM5Mdj9t,9599,simonw,2022-10-18T18:14:35Z,2022-10-18T18:14:35Z,OWNER,"Now the 3.6 tests fail - because the new release of tabulate dropped support for that Python version (so on Python 3.6 you get an older version): https://github.com/simonw/sqlite-utils/actions/runs/3275842849/jobs/5391181675 https://github.com/astanin/python-tabulate/blame/20c6370d5da2dae89b305bfb6c7f12a0f8b7236c/pyproject.toml#L22 shows minimum is 3.7 now.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1413641049,Tests failing due to updated tabulate library, https://github.com/simonw/sqlite-utils/issues/501#issuecomment-1282813168,https://api.github.com/repos/simonw/sqlite-utils/issues/501,1282813168,IC_kwDOCGYnMM5Mdizw,9599,simonw,2022-10-18T18:12:15Z,2022-10-18T18:12:15Z,OWNER,"Here's the new Tabulate release: - https://github.com/astanin/python-tabulate/releases/tag/v0.9.0 - https://github.com/astanin/python-tabulate/compare/v0.8.10...v0.9.0","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1413641049,Tests failing due to updated tabulate library, https://github.com/simonw/sqlite-utils/issues/500#issuecomment-1282800547,https://api.github.com/repos/simonw/sqlite-utils/issues/500,1282800547,IC_kwDOCGYnMM5Mdfuj,9599,simonw,2022-10-18T18:02:09Z,2022-10-18T18:02:09Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite-utils-utils-flatten,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 1, ""eyes"": 0}",1413610718,Turn --flatten into a documented utility function, https://github.com/simonw/sqlite-utils/issues/500#issuecomment-1282780770,https://api.github.com/repos/simonw/sqlite-utils/issues/500,1282780770,IC_kwDOCGYnMM5Mda5i,9599,simonw,2022-10-18T17:45:56Z,2022-10-18T17:46:05Z,OWNER,I think the public interface is a `flatten(row)` function that does `dict(_flatten(row))`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1413610718,Turn --flatten into a documented utility function, https://github.com/simonw/sqlite-utils/issues/500#issuecomment-1282779755,https://api.github.com/repos/simonw/sqlite-utils/issues/500,1282779755,IC_kwDOCGYnMM5Mdapr,9599,simonw,2022-10-18T17:45:10Z,2022-10-18T17:45:10Z,OWNER,It should go in `sqlite_utils.utils` - documented here: https://sqlite-utils.datasette.io/en/stable/reference.html#sqlite-utils-utils,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1413610718,Turn --flatten into a documented utility function, https://github.com/simonw/sqlite-utils/issues/500#issuecomment-1282778928,https://api.github.com/repos/simonw/sqlite-utils/issues/500,1282778928,IC_kwDOCGYnMM5Mdacw,9599,simonw,2022-10-18T17:44:20Z,2022-10-18T17:44:20Z,OWNER,"Here's how it works: https://github.com/simonw/sqlite-utils/blob/d792dad1cf5f16525da81b1e162fb71d469995f3/sqlite_utils/cli.py#L1847-L1848 https://github.com/simonw/sqlite-utils/blob/d792dad1cf5f16525da81b1e162fb71d469995f3/sqlite_utils/cli.py#L1082-L1088","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1413610718,Turn --flatten into a documented utility function, https://github.com/simonw/datasette/issues/1845#issuecomment-1279842912,https://api.github.com/repos/simonw/datasette/issues/1845,1279842912,IC_kwDOBm6k_c5MSNpg,9599,simonw,2022-10-15T22:22:58Z,2022-10-15T22:22:58Z,OWNER,"I think this mechanism could go a long way towards helping here: - https://github.com/simonw/datasette/issues/1160 It was part of a larger idea I was exploring around ensuring Datasette could be used to start interacting with CSV/JSON data out-of-the-box, without needing to first convert that data into SQLite using separate tools.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1410305897,Reconsider the Datasette first-run experience, https://github.com/simonw/datasette/issues/1844#issuecomment-1279598878,https://api.github.com/repos/simonw/datasette/issues/1844,1279598878,IC_kwDOBm6k_c5MRSEe,9599,simonw,2022-10-14T23:51:46Z,2022-10-14T23:51:46Z,OWNER,Blogged about this here: https://simonwillison.net/2022/Oct/14/automating-screenshots/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1409679008,Update screenshots in documentation to match latest designs, https://github.com/simonw/datasette/issues/1844#issuecomment-1279427618,https://api.github.com/repos/simonw/datasette/issues/1844,1279427618,IC_kwDOBm6k_c5MQoQi,9599,simonw,2022-10-14T20:25:45Z,2022-10-14T20:25:45Z,OWNER,Extracted a TIL: https://til.simonwillison.net/shot-scraper/subset-of-table-columns,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1409679008,Update screenshots in documentation to match latest designs, https://github.com/simonw/datasette/issues/1844#issuecomment-1279415365,https://api.github.com/repos/simonw/datasette/issues/1844,1279415365,IC_kwDOBm6k_c5MQlRF,9599,simonw,2022-10-14T20:11:55Z,2022-10-14T20:11:55Z,OWNER,Twitter thread about this issue: https://twitter.com/simonw/status/1581012617526595584,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1409679008,Update screenshots in documentation to match latest designs, https://github.com/simonw/datasette/issues/1844#issuecomment-1279406134,https://api.github.com/repos/simonw/datasette/issues/1844,1279406134,IC_kwDOBm6k_c5MQjA2,9599,simonw,2022-10-14T20:01:13Z,2022-10-14T20:01:13Z,OWNER,"Here's the YAML I added to https://github.com/simonw/datasette-screenshots/blob/main/shots.yml for this issue: ```yaml - url: https://register-of-members-interests.datasettes.com/regmem/items?_search=hamper&_sort_desc=date height: 585 width: 960 output: regmem-search.png - url: https://register-of-members-interests.datasettes.com/regmem/items?_search=hamper selector: ""#export"" output: advanced-export.png padding: 10 - url: https://congress-legislators.datasettes.com/legislators/legislator_terms?_facet=type&_facet=party&_facet=state&_facet_size=10 selectors_all: - .suggested-facets a - tr:not(tr:nth-child(n+4)) td:not(:nth-child(n+11)) padding: 10 output: faceting-details.png - url: https://latest.datasette.io/fixtures/binary_data selector: table javascript: |- Array.from( document.querySelectorAll('tr:nth-child(n+3)'), el => el.parentNode.removeChild(el) ); padding: 10 output: binary-data.png ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1409679008,Update screenshots in documentation to match latest designs, https://github.com/simonw/datasette/issues/1844#issuecomment-1279405429,https://api.github.com/repos/simonw/datasette/issues/1844,1279405429,IC_kwDOBm6k_c5MQi11,9599,simonw,2022-10-14T20:00:26Z,2022-10-14T20:00:26Z,OWNER,"New images are now live on these pages: - https://docs.datasette.io/en/latest/csv_export.html - https://docs.datasette.io/en/latest/binary_data.html - https://docs.datasette.io/en/latest/facets.html - https://docs.datasette.io/en/latest/full_text_search.html - https://docs.datasette.io/en/latest/changelog.html#v0-23 (was a duplicate of the advanced export image)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1409679008,Update screenshots in documentation to match latest designs, https://github.com/simonw/datasette/issues/1844#issuecomment-1279392717,https://api.github.com/repos/simonw/datasette/issues/1844,1279392717,IC_kwDOBm6k_c5MQfvN,9599,simonw,2022-10-14T19:44:44Z,2022-10-14T19:45:54Z,OWNER,"OK, the URLs to use in the docs are: * https://raw.githubusercontent.com/simonw/datasette-screenshots/0.62/advanced-export.png (retina) * https://raw.githubusercontent.com/simonw/datasette-screenshots/0.62/non-retina/regmem-search.png * https://raw.githubusercontent.com/simonw/datasette-screenshots/0.62/binary-data.png (retina) * https://raw.githubusercontent.com/simonw/datasette-screenshots/0.62/non-retina/faceting-details.png ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1409679008,Update screenshots in documentation to match latest designs, https://github.com/simonw/datasette/issues/1844#issuecomment-1279349314,https://api.github.com/repos/simonw/datasette/issues/1844,1279349314,IC_kwDOBm6k_c5MQVJC,9599,simonw,2022-10-14T18:50:42Z,2022-10-14T19:34:37Z,OWNER,"I'm going to link the documentation screenshots directly to the images in the https://github.com/simonw/datasette-screenshots repository - but I don't want those images to reflect `main` when the documentation may reflect a specific version. So I'm going to start tagging releases of `datasette-screenshots` so I can get permanent URLs to those images.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1409679008,Update screenshots in documentation to match latest designs, https://github.com/simonw/datasette/issues/1844#issuecomment-1279383121,https://api.github.com/repos/simonw/datasette/issues/1844,1279383121,IC_kwDOBm6k_c5MQdZR,9599,simonw,2022-10-14T19:33:49Z,2022-10-14T19:33:49Z,OWNER,"I'm going to tag `datasette-screenshots` with the current Datasette version, `0.62`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1409679008,Update screenshots in documentation to match latest designs, https://github.com/simonw/datasette/issues/1844#issuecomment-1279311487,https://api.github.com/repos/simonw/datasette/issues/1844,1279311487,IC_kwDOBm6k_c5MQL5_,9599,simonw,2022-10-14T18:06:53Z,2022-10-14T19:33:24Z,OWNER,"I just spotted some other out-dated screenshots in the `docs/` directory: - [x] [advanced_export.png](https://github.com/simonw/datasette/blob/main/docs/advanced_export.png ""advanced_export.png"") - [x] [binary_data.png](https://github.com/simonw/datasette/blob/main/docs/binary_data.png ""binary_data.png"") - [x] [facets.png](https://github.com/simonw/datasette/blob/main/docs/facets.png ""facets.png"") - [x] [full_text_search.png](https://github.com/simonw/datasette/blob/main/docs/full_text_search.png ""full_text_search.png"") ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1409679008,Update screenshots in documentation to match latest designs, https://github.com/simonw/datasette/issues/1844#issuecomment-1279382674,https://api.github.com/repos/simonw/datasette/issues/1844,1279382674,IC_kwDOBm6k_c5MQdSS,9599,simonw,2022-10-14T19:33:16Z,2022-10-14T19:33:16Z,OWNER,"That's the last two screenshots: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1409679008,Update screenshots in documentation to match latest designs, https://github.com/simonw/datasette/issues/1844#issuecomment-1279348239,https://api.github.com/repos/simonw/datasette/issues/1844,1279348239,IC_kwDOBm6k_c5MQU4P,9599,simonw,2022-10-14T18:49:22Z,2022-10-14T18:49:22Z,OWNER,"This works: ``` shot-scraper 'https://congress-legislators.datasettes.com/legislators/legislator_terms?_facet=type&_facet=party&_facet=state&_facet_size=10' \ -s '.suggested-facets a' \ --selector-all 'tr:not(tr:nth-child(n+4)) td:not(:nth-child(n+11))' \ -p 10 ``` ![congress-legislators-datasettes-com-legislators-legislator_terms 6](https://user-images.githubusercontent.com/9599/195919422-97616694-3ec0-4e05-afc2-c509275c767c.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1409679008,Update screenshots in documentation to match latest designs, https://github.com/simonw/datasette/issues/1844#issuecomment-1279339124,https://api.github.com/repos/simonw/datasette/issues/1844,1279339124,IC_kwDOBm6k_c5MQSp0,9599,simonw,2022-10-14T18:38:22Z,2022-10-14T18:42:58Z,OWNER,"This seems to get every table cell in that table for the first 3 rows and the columns up to `party`: document.querySelectorAll('tr:not(:nth-child(n+4)) td:not(:nth-child(n+10))')","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1409679008,Update screenshots in documentation to match latest designs, https://github.com/simonw/datasette/issues/1844#issuecomment-1279334694,https://api.github.com/repos/simonw/datasette/issues/1844,1279334694,IC_kwDOBm6k_c5MQRkm,9599,simonw,2022-10-14T18:33:41Z,2022-10-14T18:34:32Z,OWNER,"I'm going to use this page for the facets screenshot: https://congress-legislators.datasettes.com/legislators/legislator_terms?_facet=type&_facet=party&_facet=state&_facet_size=10 Trying for this bit: Which incorporates `.suggested-facets` but also the first 3 rows and 10 columns of the table, I wonder if I can specify that in a single selector?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1409679008,Update screenshots in documentation to match latest designs, https://github.com/simonw/datasette/issues/1844#issuecomment-1279325685,https://api.github.com/repos/simonw/datasette/issues/1844,1279325685,IC_kwDOBm6k_c5MQPX1,9599,simonw,2022-10-14T18:23:22Z,2022-10-14T18:23:22Z,OWNER,"Here's the new advanced export image: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1409679008,Update screenshots in documentation to match latest designs, https://github.com/simonw/datasette/issues/1844#issuecomment-1279325003,https://api.github.com/repos/simonw/datasette/issues/1844,1279325003,IC_kwDOBm6k_c5MQPNL,9599,simonw,2022-10-14T18:22:35Z,2022-10-14T18:22:35Z,OWNER,"So: ``` shot-scraper 'https://latest.datasette.io/fixtures/binary_data' \ -j ""Array.from(document.querySelectorAll('tr:nth-child(n+3)'), el => el.parentNode.removeChild(el));"" \ -s table -p 10 ``` ![latest-datasette-io-fixtures-binary_data 1](https://user-images.githubusercontent.com/9599/195915092-be81db43-5672-4375-bd66-4316211b1afc.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1409679008,Update screenshots in documentation to match latest designs, https://github.com/simonw/datasette/issues/1844#issuecomment-1279323714,https://api.github.com/repos/simonw/datasette/issues/1844,1279323714,IC_kwDOBm6k_c5MQO5C,9599,simonw,2022-10-14T18:21:03Z,2022-10-14T18:21:03Z,OWNER,"For this image: https://latest.datasette.io/fixtures/binary_data has an extra row these days: This deletes every row past the first two (first three including the header row): ```javascipt Array.from(document.querySelectorAll('tr:nth-child(n+3)'), el => el.parentNode.removeChild(el)); ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1409679008,Update screenshots in documentation to match latest designs, https://github.com/simonw/datasette/issues/1844#issuecomment-1279314400,https://api.github.com/repos/simonw/datasette/issues/1844,1279314400,IC_kwDOBm6k_c5MQMng,9599,simonw,2022-10-14T18:10:07Z,2022-10-14T18:13:36Z,OWNER,"For the advanced export one: shot-scraper 'https://register-of-members-interests.datasettes.com/regmem/items?_search=hamper' -s '#export' -p 10 Produces: ![register-of-members-interests-datasettes-com-regmem-items 2](https://user-images.githubusercontent.com/9599/195913614-448557aa-5ec6-4a83-98cf-8837d3117204.png) Current image is: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1409679008,Update screenshots in documentation to match latest designs, https://github.com/simonw/datasette/issues/1844#issuecomment-1279316670,https://api.github.com/repos/simonw/datasette/issues/1844,1279316670,IC_kwDOBm6k_c5MQNK-,9599,simonw,2022-10-14T18:12:39Z,2022-10-14T18:12:39Z,OWNER,"New screenshot of FTS, from https://register-of-members-interests.datasettes.com/regmem/items?_search=hamper&_sort_desc=date ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1409679008,Update screenshots in documentation to match latest designs, https://github.com/simonw/datasette/issues/1843#issuecomment-1278537920,https://api.github.com/repos/simonw/datasette/issues/1843,1278537920,IC_kwDOBm6k_c5MNPDA,9599,simonw,2022-10-14T06:19:55Z,2022-10-14T06:20:06Z,OWNER,Maybe I need to explicitly close those SQLite connections held by the Datasette instance after this line: https://github.com/simonw/datasette/blob/79aa0de083d38a9975915d5a4cc68ca6c74fbe3d/tests/fixtures.py#L165,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705,"Intermittent ""Too many open files"" error running tests", https://github.com/simonw/datasette/issues/1843#issuecomment-1278480437,https://api.github.com/repos/simonw/datasette/issues/1843,1278480437,IC_kwDOBm6k_c5MNBA1,9599,simonw,2022-10-14T04:51:10Z,2022-10-14T04:51:10Z,OWNER,Extracted a TIL: https://til.simonwillison.net/python/too-many-open-files-psutil,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705,"Intermittent ""Too many open files"" error running tests", https://github.com/simonw/datasette/issues/1843#issuecomment-1278478042,https://api.github.com/repos/simonw/datasette/issues/1843,1278478042,IC_kwDOBm6k_c5MNAba,9599,simonw,2022-10-14T04:46:29Z,2022-10-14T04:46:29Z,OWNER,"I did `pip install psutil` and then ran this in the debugger for one of these errors: ```python import psutil for f in psutil.Process().open_files(): print(f) ``` The output looked like this: ``` popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpq31d2af1/fixtures.db', fd=11) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpoxdpxj6w/fixtures.db', fd=12) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfd3oyo10/fixtures.dot.db', fd=13) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpezwfu7w8/fixtures.db', fd=14) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpq31d2af1/fixtures.db', fd=15) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpq31d2af1/fixtures.db', fd=16) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpz6e2anqw/fixtures.db', fd=17) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpoxdpxj6w/fixtures.db', fd=18) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpdp4we7hb/fixtures.db', fd=19) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfd3oyo10/fixtures.dot.db', fd=20) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp4ljq_ai0/fixtures.db', fd=21) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpezwfu7w8/fixtures.db', fd=22) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp907xmnzb/fixtures.db', fd=24) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpz6e2anqw/extra database.db', fd=25) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpz6e2anqw/fixtures.db', fd=26) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpzlwn6bqm/fixtures.db', fd=27) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpdp4we7hb/fixtures.db', fd=28) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp42e6vyj_/fixtures.db', fd=29) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp4ljq_ai0/fixtures.db', fd=31) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpw32vwkjq/fixtures.db', fd=32) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp907xmnzb/extra database.db', fd=33) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp907xmnzb/fixtures.db', fd=34) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpw32vwkjq/fixtures.db', fd=35) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpzlwn6bqm/foo-bar.db', fd=36) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpzlwn6bqm/foo.db', fd=37) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpzlwn6bqm/fixtures.db', fd=38) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpcl1edmyv/fixtures.db', fd=39) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp42e6vyj_/fixtures.db', fd=40) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpoxdpxj6w/fixtures.db', fd=41) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp0w5jugqk/fixtures.db', fd=42) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpmb9y9fba/fixtures.db', fd=43) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfwsbx941/fixtures.db', fd=44) popenfile(path='/Users/simon/Dropbox/Development/datasette/tests/spatialite.db', fd=45) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp0w5jugqk/fixtures.db', fd=46) popenfile(path='/Users/simon/Dropbox/Development/datasette/tests/spatialite.db', fd=47) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpmb9y9fba/fixtures.db', fd=48) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_serve_create0/does_not_exist_yet.db', fd=49) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfwsbx941/data.db', fd=92) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfwsbx941/fixtures.db', fd=93) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfwsbx941/data.db', fd=94) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_serve_duplicate_database_0/db.db', fd=95) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_serve_duplicate_database_0/nested/db.db', fd=99) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_serve_deduplicate_same_da0/db.db', fd=100) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_weird_database_names_test0/test-database (1).sqlite', fd=101) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_weird_database_names_test0/test-database (1).sqlite', fd=102) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_serve_duplicate_database_0/db.db', fd=103) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_serve_duplicate_database_0/nested/db.db', fd=104) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_weird_database_names_data0/database (1).sqlite', fd=105) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_weird_database_names_data0/database (1).sqlite', fd=106) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_weird_database_names_test0/test-database (1).sqlite', fd=107) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_weird_database_names_test0/test-database (1).sqlite', fd=109) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_weird_database_names_data0/database (1).sqlite', fd=111) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_weird_database_names_data0/database (1).sqlite', fd=113) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmppju9w34z/fixtures.db', fd=117) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpky1jamnv/fixtures.db', fd=118) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_0.db', fd=119) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/config-dir0/demo.db', fd=120) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/config-dir0/immutable.db', fd=121) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/config-dir0/k.sqlite', fd=122) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/config-dir0/j.sqlite3', fd=123) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_1.db', fd=124) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmppju9w34z/extra database.db', fd=125) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmppju9w34z/fixtures.db', fd=126) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmppju9w34z/extra database.db', fd=127) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmppju9w34z/fixtures.db', fd=128) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_2.db', fd=129) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_3.db', fd=130) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_4.db', fd=131) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_5.db', fd=132) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_6.db', fd=133) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_7.db', fd=134) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_8.db', fd=135) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_9.db', fd=136) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_0.db', fd=137) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_1.db', fd=138) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_2.db', fd=139) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_3.db', fd=140) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_4.db', fd=141) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_5.db', fd=142) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_6.db', fd=143) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_7.db', fd=144) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_8.db', fd=145) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_9.db', fd=146) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_10.db', fd=147) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpmzu4jbdx/fixtures.db', fd=148) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpskyh32wh/fixtures.db', fd=149) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp966705ec/fixtures.db', fd=150) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_0.db', fd=151) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_1.db', fd=152) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_2.db', fd=153) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_3.db', fd=154) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_4.db', fd=155) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_5.db', fd=156) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_6.db', fd=157) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_7.db', fd=158) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_8.db', fd=159) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_9.db', fd=160) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_10.db', fd=161) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_0.db', fd=162) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_1.db', fd=163) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_2.db', fd=164) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_3.db', fd=165) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_4.db', fd=166) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_5.db', fd=167) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_6.db', fd=168) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_7.db', fd=169) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_8.db', fd=170) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_9.db', fd=171) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpky1jamnv/fixtures.db', fd=172) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpwmewpy8e/fixtures.db', fd=173) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpmzu4jbdx/fixtures.db', fd=174) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpkyn60obe/fixtures.db', fd=175) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpskyh32wh/fixtures.db', fd=176) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpcl1edmyv/fixtures.db', fd=177) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp966705ec/fixtures.db', fd=178) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp4gctb4i6/fixtures.db', fd=179) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpwmewpy8e/fixtures.db', fd=180) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp4gctb4i6/fixtures.db', fd=181) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpkyn60obe/fixtures.db', fd=182) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp4gctb4i6/fixtures.db', fd=183) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmps7p_ee_e/fixtures.db', fd=184) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpf3_6i1d6/fixtures.db', fd=185) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp6_9vzq4o/fixtures.db', fd=187) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpqtubcoah/fixtures.db', fd=188) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmps7p_ee_e/extra database.db', fd=189) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmps7p_ee_e/fixtures.db', fd=190) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp2gauogmy/fixtures.db', fd=191) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpf3_6i1d6/fixtures.db', fd=192) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmprh373nt1/fixtures.db', fd=193) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp6_9vzq4o/fixtures.db', fd=197) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmprh373nt1/extra database.db', fd=198) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpqtubcoah/fixtures.db', fd=211) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmprh373nt1/fixtures.db', fd=212) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp2gauogmy/fixtures.db', fd=215) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpd2tleqni/fixtures.db', fd=216) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpd2tleqni/fixtures.db', fd=217) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmplpdc3wfl/fixtures.db', fd=219) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpdnsscl5v/fixtures.db', fd=222) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmptjayd3pn/fixtures.db', fd=223) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp4u5oo5ne/fixtures.db', fd=224) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp0dgqc4wx/fixtures.db', fd=225) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpdnsscl5v/fixtures.db', fd=226) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpoehh1_tg/fixtures.db', fd=227) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp4u5oo5ne/fixtures.db', fd=247) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp0dgqc4wx/fixtures.db', fd=251) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpoehh1_tg/fixtures.db', fd=253) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmptjayd3pn/fixtures.db', fd=254) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test-view-names0/fixtures.db', fd=255) ``` Clearly something is bad with the way fixtures work in the tests - a huge number of `fixtures.db` database files are being created and left open!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705,"Intermittent ""Too many open files"" error running tests", https://github.com/simonw/datasette/issues/1829#issuecomment-1278302478,https://api.github.com/repos/simonw/datasette/issues/1829,1278302478,IC_kwDOBm6k_c5MMVkO,9599,simonw,2022-10-14T00:06:19Z,2022-10-14T00:06:19Z,OWNER,"I'll finish this in a PR: - https://github.com/simonw/datasette/pull/1842","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1396948693,Table/database that is private due to inherited permissions does not show padlock, https://github.com/simonw/datasette/issues/1829#issuecomment-1278300241,https://api.github.com/repos/simonw/datasette/issues/1829,1278300241,IC_kwDOBm6k_c5MMVBR,9599,simonw,2022-10-14T00:03:52Z,2022-10-14T00:04:28Z,OWNER,"Here's what I've got so far: ```diff diff --git a/datasette/app.py b/datasette/app.py index 5fa4955c..df9eae49 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -1,5 +1,5 @@ import asyncio -from typing import Sequence, Union, Tuple +from typing import Sequence, Union, Tuple, Optional import asgi_csrf import collections import datetime @@ -707,7 +707,7 @@ class Datasette: Raises datasette.Forbidden() if any of the checks fail """""" - assert actor is None or isinstance(actor, dict) + assert actor is None or isinstance(actor, dict), ""actor must be None or a dict"" for permission in permissions: if isinstance(permission, str): action = permission @@ -732,23 +732,34 @@ class Datasette: else: raise Forbidden(action) - async def check_visibility(self, actor, action, resource): + async def check_visibility( + self, + actor: dict, + action: Optional[str] = None, + resource: Optional[str] = None, + permissions: Optional[ + Sequence[Union[Tuple[str, Union[str, Tuple[str, str]]], str]] + ] = None, + ): """"""Returns (visible, private) - visible = can you see it, private = can others see it too"""""" - visible = await self.permission_allowed( - actor, - action, - resource=resource, - default=True, - ) - if not visible: + if permissions: + assert ( + not action and not resource + ), ""Can't use action= or resource= with permissions="" + else: + permissions = [(action, resource)] + try: + await self.ensure_permissions(actor, permissions) + except Forbidden: return False, False - private = not await self.permission_allowed( - None, - action, - resource=resource, - default=True, - ) - return visible, private + # User can see it, but can the anonymous user see it? + try: + await self.ensure_permissions(None, permissions) + except Forbidden: + # It's visible but private + return True, True + # It's visible to everyone + return True, False async def execute( self, diff --git a/datasette/views/table.py b/datasette/views/table.py index 60c092f9..f73b0957 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -28,7 +28,7 @@ from datasette.utils import ( urlsafe_components, value_as_boolean, ) -from datasette.utils.asgi import BadRequest, NotFound +from datasette.utils.asgi import BadRequest, Forbidden, NotFound from datasette.filters import Filters from .base import DataView, DatasetteError, ureg from .database import QueryView @@ -213,18 +213,16 @@ class TableView(DataView): raise NotFound(f""Table not found: {table_name}"") # Ensure user has permission to view this table - await self.ds.ensure_permissions( + visible, private = await self.ds.check_visibility( request.actor, - [ + permissions=[ (""view-table"", (database_name, table_name)), (""view-database"", database_name), ""view-instance"", ], ) - - private = not await self.ds.permission_allowed( - None, ""view-table"", (database_name, table_name), default=True - ) + if not visible: + raise Forbidden(""You do not have permission to view this table"") # Handle ?_filter_column and redirect, if present redirect_params = filters_should_redirect(request.args) ``` Still needs tests and a documentation update. Also this fix is currently only applied on the table page - needs to be applied on database, row and query pages too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1396948693,Table/database that is private due to inherited permissions does not show padlock, https://github.com/simonw/datasette/issues/1829#issuecomment-1278237331,https://api.github.com/repos/simonw/datasette/issues/1829,1278237331,IC_kwDOBm6k_c5MMFqT,9599,simonw,2022-10-13T22:17:45Z,2022-10-13T22:19:22Z,OWNER,I think `check_visibility` should be changed to optionally accept `permissions=` which is the same list of tuples that can be passed to `ensure_permissions`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1396948693,Table/database that is private due to inherited permissions does not show padlock, https://github.com/simonw/datasette/issues/1831#issuecomment-1278198700,https://api.github.com/repos/simonw/datasette/issues/1831,1278198700,IC_kwDOBm6k_c5ML8Os,9599,simonw,2022-10-13T21:29:09Z,2022-10-13T21:29:09Z,OWNER,"I'm going to commit the code now, but then I need to add some extra tests to ensure the breadcrumb permission display logic works correctly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1397084281,If user can see table but NOT database/instance nav links should not display, https://github.com/simonw/datasette/issues/1831#issuecomment-1278198145,https://api.github.com/repos/simonw/datasette/issues/1831,1278198145,IC_kwDOBm6k_c5ML8GB,9599,simonw,2022-10-13T21:28:30Z,2022-10-13T21:28:30Z,OWNER,"This has turned into a full refactor of how breadcrumbs work. I'm using my first ever Jinja macro for this - I import that at the top of `base.html` so that it will be available everywhere else: ```html+jinja {% import ""_crumbs.html"" as crumbs with context %} ``` The `with context` bit is needed so the macro can see the new `crumb_items()` function that I'm adding to the global template rendering scope. Here's the full content of `_crumbs.html`: ```html+jinja {% macro nav(request, database=None, table=None) -%} {% set items=crumb_items(request=request, database=database, table=table) %} {% if items %}

{% for item in items %} {{ item.label }} {% if not loop.last %} / {% endif %} {% endfor %}

{% endif %} {%- endmacro %} ``` This means custom template authors can use their own `_crumbs.html` template to do something different with the breadcrumbs. In the actual templates I display breadcrumbs like this: ```html+jinja {% block crumbs %} {{ crumbs.nav(request=request, database=database) }} {% endblock %} ``` Pass `database=` to get `home / database_name` - pass `table=` as well to get `home / database_name / table_name` - if you just send `request=` you just get `home`. I've also made the default base template show the `home` breadcrumbs - other pages such as `table.html` and `row.html` can then over-ride `{% block crumbs %}` to get different breadcrumbs. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1397084281,If user can see table but NOT database/instance nav links should not display, https://github.com/simonw/datasette/issues/1831#issuecomment-1278175798,https://api.github.com/repos/simonw/datasette/issues/1831,1278175798,IC_kwDOBm6k_c5ML2o2,9599,simonw,2022-10-13T21:02:52Z,2022-10-13T21:02:52Z,OWNER,This patch to `default_permissions.py` made debugging easier: https://gist.github.com/simonw/daddf022e75a98ea6246ac1e12dc8759,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1397084281,If user can see table but NOT database/instance nav links should not display, https://github.com/simonw/datasette/issues/1831#issuecomment-1276882359,https://api.github.com/repos/simonw/datasette/issues/1831,1276882359,IC_kwDOBm6k_c5MG623,9599,simonw,2022-10-13T00:36:09Z,2022-10-13T00:36:19Z,OWNER,"It's important that, however this works, it supports custom templates changing how the breadcrumbs are displayed. Probably needs a `_crumbs.html` template.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1397084281,If user can see table but NOT database/instance nav links should not display, https://github.com/simonw/datasette/issues/1831#issuecomment-1276856836,https://api.github.com/repos/simonw/datasette/issues/1831,1276856836,IC_kwDOBm6k_c5MG0oE,9599,simonw,2022-10-12T23:57:28Z,2022-10-12T23:57:28Z,OWNER,"As part of this I think I want `request` to always be available in the template context, which will remove the need for https://datasette.io/plugins/datasette-template-request","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1397084281,If user can see table but NOT database/instance nav links should not display, https://github.com/simonw/datasette/issues/1831#issuecomment-1276706181,https://api.github.com/repos/simonw/datasette/issues/1831,1276706181,IC_kwDOBm6k_c5MGP2F,9599,simonw,2022-10-12T20:34:00Z,2022-10-12T23:43:39Z,OWNER,"Maybe new template functions: `table_crumbs(table, database)` and `database_crumbs(database)` and `instance_crumbs()` - which know how to both check the permissions and display the `