html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app https://github.com/simonw/datasette/issues/1668#issuecomment-1073125334,https://api.github.com/repos/simonw/datasette/issues/1668,1073125334,IC_kwDOBm6k_c4_9pfW,9599,2022-03-19T22:53:55Z,2022-03-19T22:53:55Z,OWNER,"Need to update documentation in a few places - e.g. https://docs.datasette.io/en/stable/internals.html#remove-database-name > This removes a database that has been previously added. `name=` is the unique name of that database, used in its URL path.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154, https://github.com/dogsheep/healthkit-to-sqlite/issues/14#issuecomment-1073123231,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/14,1073123231,IC_kwDOC8tyDs4_9o-f,343884,2022-03-19T22:39:29Z,2022-03-19T22:39:29Z,NONE,"I have this issue, too, with a fresh export. None of my `Workout` entries in `export.xml` have an `id` key, though [the sample `export.xml` in the tests folder doesn’t either](https://github.com/dogsheep/healthkit-to-sqlite/blob/main/tests/zip_contents/apple_health_export/export.xml#L14-L21), so I don’t think this is the culprit. Indeed, it seems @simonw is using the [`hash_id` function from `sqlite_utils`](https://sqlite-utils.datasette.io/en/stable/python-api.html#setting-an-id-based-on-the-hash-of-the-row-contents), which creates a column (`id`, in this case) based on a hash of the row’s contents. When I run the script, a `workouts` table is created, with one entry: my first workout. No `workout_points` table is created, as [I’d expect from `utils.py`](https://github.com/dogsheep/healthkit-to-sqlite/blob/main/healthkit_to_sqlite/utils.py#L89-L90). I then get essentially the same error as noted in this thread: ```Importing from HealthKit [###################################-] 98% 00:00:01 Traceback (most recent call last): File ""/Users/lchski/.pyenv/versions/3.10.3/bin/healthkit-to-sqlite"", line 8, in sys.exit(cli()) File ""/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/click/core.py"", line 1128, in __call__ return self.main(*args, **kwargs) File ""/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/click/core.py"", line 1053, in main rv = self.invoke(ctx) File ""/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/click/core.py"", line 1395, in invoke return ctx.invoke(self.callback, **ctx.params) File ""/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/click/core.py"", line 754, in invoke return __callback(*args, **kwargs) File ""/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/healthkit_to_sqlite/cli.py"", line 57, in cli convert_xml_to_sqlite(fp, db, progress_callback=bar.update, zipfile=zf) File ""/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/healthkit_to_sqlite/utils.py"", line 34, in convert_xml_to_sqlite workout_to_db(el, db, zipfile) File ""/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/healthkit_to_sqlite/utils.py"", line 57, in workout_to_db pk = db[""workouts""].insert(record, alter=True, hash_id=""id"").last_pk File ""/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/sqlite_utils/db.py"", line 2822, in insert return self.insert_all( File ""/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/sqlite_utils/db.py"", line 2950, in insert_all self.insert_chunk( File ""/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/sqlite_utils/db.py"", line 2715, in insert_chunk result = self.db.execute(query, params) File ""/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/sqlite_utils/db.py"", line 458, in execute return self.conn.execute(sql, parameters) sqlite3.IntegrityError: UNIQUE constraint failed: workouts.id ``` Are there maybe duplicate workouts in the data, which’d cause multiple rows to share the same `id`? It’s strange, though, that no `workout_points` is created at all. Export created from iOS 15.3.1.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771608692, https://github.com/simonw/datasette/issues/1668#issuecomment-1073112104,https://api.github.com/repos/simonw/datasette/issues/1668,1073112104,IC_kwDOBm6k_c4_9mQo,9599,2022-03-19T21:08:21Z,2022-03-19T21:08:21Z,OWNER,"I think I've got this working but I need to write a test for it that covers the rare case when the route is not the same thing as the database name. I'll do that with a new test.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154, https://github.com/simonw/datasette/issues/1668#issuecomment-1073097394,https://api.github.com/repos/simonw/datasette/issues/1668,1073097394,IC_kwDOBm6k_c4_9iqy,9599,2022-03-19T20:56:35Z,2022-03-19T20:56:35Z,OWNER,"I'm trying to think if there's any reason not to use `route` for this. Would I possibly want to use that noun for something else in the future? I like it more than `route_path` because it has no underscore. Decision made: I'm going with `route`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154, https://github.com/simonw/datasette/issues/1667#issuecomment-1073076624,https://api.github.com/repos/simonw/datasette/issues/1667,1073076624,IC_kwDOBm6k_c4_9dmQ,9599,2022-03-19T20:31:44Z,2022-03-19T20:31:44Z,OWNER,I can now read `format` from `request.url_vars` and delete this code entirely: https://github.com/simonw/datasette/blob/b9c2b1cfc8692b9700416db98721fa3ec982f6be/datasette/views/base.py#L375-L381,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174302994, https://github.com/simonw/datasette/issues/1668#issuecomment-1073076187,https://api.github.com/repos/simonw/datasette/issues/1668,1073076187,IC_kwDOBm6k_c4_9dfb,9599,2022-03-19T20:28:20Z,2022-03-19T20:28:20Z,OWNER,I'm going to keep `path` as the path to the file on disk. I'll pick a new name for what is currently `path` in that undocumented JSON API.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154, https://github.com/simonw/datasette/issues/1668#issuecomment-1073076136,https://api.github.com/repos/simonw/datasette/issues/1668,1073076136,IC_kwDOBm6k_c4_9deo,9599,2022-03-19T20:27:44Z,2022-03-19T20:27:44Z,OWNER,"Pretty sure changing it will break some existing plugins though, including likely Datasette Desktop.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154, https://github.com/simonw/datasette/issues/1668#issuecomment-1073076110,https://api.github.com/repos/simonw/datasette/issues/1668,1073076110,IC_kwDOBm6k_c4_9deO,9599,2022-03-19T20:27:22Z,2022-03-19T20:27:22Z,OWNER,"The docs do currently describe `path` as the filesystem path here: https://docs.datasette.io/en/stable/internals.html#database-class Good thing I'm not at 1.0 yet so I can change that!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154, https://github.com/simonw/datasette/issues/1668#issuecomment-1073076015,https://api.github.com/repos/simonw/datasette/issues/1668,1073076015,IC_kwDOBm6k_c4_9dcv,9599,2022-03-19T20:26:32Z,2022-03-19T20:26:32Z,OWNER,I'm inclined to redefine `ds.path` to `ds.file_path` to fix this. Or `ds.filepath`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154, https://github.com/simonw/datasette/issues/1668#issuecomment-1073075913,https://api.github.com/repos/simonw/datasette/issues/1668,1073075913,IC_kwDOBm6k_c4_9dbJ,9599,2022-03-19T20:25:46Z,2022-03-19T20:26:08Z,OWNER,"The output of `/.json` DOES use `path` to mean the URL path, not the path to the file on disk: ``` { ""fixtures.dot"": { ""name"": ""fixtures.dot"", ""hash"": null, ""color"": ""631f11"", ""path"": ""/fixtures~2Edot"", ``` So that's a problem already: having `db.path` refer to something different from that JSON is inconsistent.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154, https://github.com/simonw/datasette/issues/1668#issuecomment-1073075697,https://api.github.com/repos/simonw/datasette/issues/1668,1073075697,IC_kwDOBm6k_c4_9dXx,9599,2022-03-19T20:24:06Z,2022-03-19T20:24:06Z,OWNER,"Right now if a database has a `.` in its name e.g. `fixtures.dot` the URL to that database is: /fixtures~2Edot But the output on `/-/databases` doesn't reflect that, it still shows the name with the dot.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154, https://github.com/simonw/datasette/issues/1660#issuecomment-1073073599,https://api.github.com/repos/simonw/datasette/issues/1660,1073073599,IC_kwDOBm6k_c4_9c2_,9599,2022-03-19T20:06:40Z,2022-03-19T20:06:40Z,OWNER,"This blocks: - #1668","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170144879, https://github.com/simonw/datasette/issues/1668#issuecomment-1073073579,https://api.github.com/repos/simonw/datasette/issues/1668,1073073579,IC_kwDOBm6k_c4_9c2r,9599,2022-03-19T20:06:27Z,2022-03-19T20:06:27Z,OWNER,Marking this as blocked until #1660 is done.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154, https://github.com/simonw/datasette/issues/1668#issuecomment-1073073547,https://api.github.com/repos/simonw/datasette/issues/1668,1073073547,IC_kwDOBm6k_c4_9c2L,9599,2022-03-19T20:06:07Z,2022-03-19T20:06:07Z,OWNER,"Implementing this is a little tricky because there's a whole lot of code that expects the `database` captured by the URL routing to be the name used to look up the database in `datasette.databases` - or via `.get_database()`. The `DataView.get()` method is a good example of the trickyness here. It even has code that dispatches out to plugin hooks that take `database` as a parameter. https://github.com/simonw/datasette/blob/61419388c134001118aaf7dfb913562d467d7913/datasette/views/base.py#L383-L555 All the more reason to get rid of that `BaseView -> DataView -> TableView` hierarchy entirely: - #1660","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154, https://github.com/simonw/datasette/issues/1668#issuecomment-1073043433,https://api.github.com/repos/simonw/datasette/issues/1668,1073043433,IC_kwDOBm6k_c4_9Vfp,9599,2022-03-19T16:54:55Z,2022-03-19T20:01:19Z,OWNER,"Options: - `route_path` - `url_path` - `route` I like `route_path`, or maybe `route`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154, https://github.com/simonw/datasette/issues/1668#issuecomment-1073043713,https://api.github.com/repos/simonw/datasette/issues/1668,1073043713,IC_kwDOBm6k_c4_9VkB,9599,2022-03-19T16:56:19Z,2022-03-19T16:56:19Z,OWNER,"Worth noting that the `name` right now is picked automatically to avoid conflicts: https://github.com/simonw/datasette/blob/61419388c134001118aaf7dfb913562d467d7913/datasette/app.py#L397-L413","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154, https://github.com/simonw/datasette/issues/1668#issuecomment-1073043350,https://api.github.com/repos/simonw/datasette/issues/1668,1073043350,IC_kwDOBm6k_c4_9VeW,9599,2022-03-19T16:54:26Z,2022-03-19T16:54:26Z,OWNER,"The `Database` class already has a `path` property but it means something else - it's the path to the `.db` file on disk: https://github.com/simonw/datasette/blob/61419388c134001118aaf7dfb913562d467d7913/datasette/database.py#L29-L50 So need a different name for the path-that-is-used-in-the-URL.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154, https://github.com/simonw/datasette/issues/1667#issuecomment-1073042554,https://api.github.com/repos/simonw/datasette/issues/1667,1073042554,IC_kwDOBm6k_c4_9VR6,9599,2022-03-19T16:50:01Z,2022-03-19T16:52:35Z,OWNER,"OK, I've made this more consistent - I still need to address the fact that `format` can be `.json` or `json` or not used at all before I close this issue. https://github.com/simonw/datasette/blob/61419388c134001118aaf7dfb913562d467d7913/tests/test_routes.py#L15-L35","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174302994, https://github.com/simonw/datasette/issues/1667#issuecomment-1073040072,https://api.github.com/repos/simonw/datasette/issues/1667,1073040072,IC_kwDOBm6k_c4_9UrI,9599,2022-03-19T16:34:02Z,2022-03-19T16:34:02Z,OWNER,"I called it `as_format` to avoid clashing with the Python built-in `format()` function when these things were turned into keyword arguments, but now that they're not I can use `format` instead. I think I'm going to go with `database`, `table`, `format` and `pks`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174302994, https://github.com/simonw/datasette/issues/1666#issuecomment-1073039670,https://api.github.com/repos/simonw/datasette/issues/1666,1073039670,IC_kwDOBm6k_c4_9Uk2,9599,2022-03-19T16:31:08Z,2022-03-19T16:31:57Z,OWNER,"This does make it more interesting - it also highlights how inconsistent the way the capturing works is. Especially `as_format` which can be `None` or `""""` or `.json` or `json` or not used at all in the case of `TableView`. https://github.com/simonw/datasette/blob/764738dfcb16cd98b0987d443f59d5baa9d3c332/tests/test_routes.py#L12-L36","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174162781, https://github.com/simonw/datasette/issues/1666#issuecomment-1073039241,https://api.github.com/repos/simonw/datasette/issues/1666,1073039241,IC_kwDOBm6k_c4_9UeJ,9599,2022-03-19T16:28:15Z,2022-03-19T16:28:15Z,OWNER,This is more interesting if it also asserts against the captured matches from the pattern.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174162781, https://github.com/simonw/datasette/issues/878#issuecomment-1073037939,https://api.github.com/repos/simonw/datasette/issues/878,1073037939,IC_kwDOBm6k_c4_9UJz,9599,2022-03-19T16:19:30Z,2022-03-19T16:19:30Z,OWNER,"On revisiting https://gist.github.com/simonw/281eac9c73b062c3469607ad86470eb2 a few months later I'm having second thoughts about using `@inject` on the `main()` method. But I still like the pattern as a way to resolve more complex cases like ""to generate GeoJSON of the expanded view with labels, the label expansion code needs to run once at some before the GeoJSON formatting code does"". So I'm going to stick with it a tiny bit longer, but maybe try to make it a lot more explicit when it's going to happen rather than having the main view methods themselves also use async DI.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885, https://github.com/simonw/datasette/issues/1228#issuecomment-1072954795,https://api.github.com/repos/simonw/datasette/issues/1228,1072954795,IC_kwDOBm6k_c4_8_2r,7107523,2022-03-19T06:44:40Z,2022-03-19T06:44:40Z,NONE,"> ... unless your data had a column called `n`? Exactly, that's highly likely even though I can't double check from this computer just now. Thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810397025, https://github.com/simonw/datasette/issues/1561#issuecomment-1072939780,https://api.github.com/repos/simonw/datasette/issues/1561,1072939780,IC_kwDOBm6k_c4_88ME,9599,2022-03-19T04:45:40Z,2022-03-19T04:45:40Z,OWNER,"I ended up moving hashed URL mode out to a plugin in: - #647 If you're still interested in using it with `_memory` please open an issue in that repo here: https://github.com/simonw/datasette-hashed-urls","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082765654, https://github.com/simonw/datasette/issues/1666#issuecomment-1072933875,https://api.github.com/repos/simonw/datasette/issues/1666,1072933875,IC_kwDOBm6k_c4_86vz,9599,2022-03-19T04:03:42Z,2022-03-19T04:03:42Z,OWNER,Tests so far: https://github.com/simonw/datasette/blob/711767bcd3c1e76a0861fe7f24069ff1c8efc97a/tests/test_routes.py#L12-L34,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174162781, https://github.com/simonw/datasette/issues/1228#issuecomment-1072915936,https://api.github.com/repos/simonw/datasette/issues/1228,1072915936,IC_kwDOBm6k_c4_82Xg,9599,2022-03-19T01:50:27Z,2022-03-19T01:50:27Z,OWNER,Demo: https://latest.datasette.io/fixtures/facetable - which now has a column called `n`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810397025, https://github.com/simonw/datasette/issues/1228#issuecomment-1072908029,https://api.github.com/repos/simonw/datasette/issues/1228,1072908029,IC_kwDOBm6k_c4_80b9,9599,2022-03-19T00:57:54Z,2022-03-19T00:57:54Z,OWNER,"Yes! That's the problem. I was able to replicate it like so: ``` echo '[{ ""n"": ""one"", ""abc"": 1 }, { ""n"": ""one"", ""abc"": 2 }, { ""n"": ""two"", ""abc"": 3 }]' | sqlite-utils insert column-called-n.db t - ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810397025, https://github.com/simonw/datasette/issues/1228#issuecomment-1072907680,https://api.github.com/repos/simonw/datasette/issues/1228,1072907680,IC_kwDOBm6k_c4_80Wg,9599,2022-03-19T00:55:48Z,2022-03-19T00:55:48Z,OWNER,... unless your data had a column called `n`?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810397025, https://github.com/simonw/datasette/issues/1228#issuecomment-1072907610,https://api.github.com/repos/simonw/datasette/issues/1228,1072907610,IC_kwDOBm6k_c4_80Va,9599,2022-03-19T00:55:29Z,2022-03-19T00:55:29Z,OWNER,"It looks to me like something is causing the faceting query here to return a string when it was expected to return a number: https://github.com/simonw/datasette/blob/32963018e7edfab1233de7c7076c428d0e5c7813/datasette/facets.py#L153-L170 I can't think of any way that a `count(*) as n` would turn into a string though!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810397025, https://github.com/simonw/datasette/issues/1605#issuecomment-1072907200,https://api.github.com/repos/simonw/datasette/issues/1605,1072907200,IC_kwDOBm6k_c4_80PA,9599,2022-03-19T00:52:54Z,2022-03-19T00:53:45Z,OWNER,"Had a thought about the implementation of this: it could make a really neat plugin. Something like `datasette-export` which adds a `export` command using https://docs.datasette.io/en/stable/plugin_hooks.html#register-commands-cli - then you could run: datasette export my-export-dir mydatabase.db -m metadata.json --template-dir templates/ And the command would then: - Create a `Datasette()` instance with those databases/metadata/etc - Execute`await datasette.client.get(""/"")` to get the homepage HTML - Parse the HTML using BeautifulSoup to find all `a[href]`, `link[href]`, `script[src]`, `img[src]` elements that reference a relative path as opposed to one that starts with `http://` - Write out the homepage to `my-export-dir/index.html` - Recursively fetch and dump all of the other pages and assets that it found too All of that HTML parsing may be over-complicating things. It could alternatively accept options for which pages you want to export: ``` datasette export my-export-dir \ mydatabase.db -m metadata.json --template-dir templates/ \ --path / \ --path /mydatabase ... ``` Or a really wild option: it could allow you to define the paths you want to export using a SQL query: ``` datasette export my-export-dir \ mydatabase.db -m metadata.json --template-dir templates/ \ --sql "" select '/' as path, 'index.html' as filename union all select '/mydatabase/articles/' || id as path, 'article-' || id || '.html' as filename from articles union all select '/mydatabase/tags/' || tag as path, 'tag-' || tag || '.html' as filename from tags "" ``` Which would save these files: - `index.html` as the content of `/` - `article-1.html` (and more) as the content of `/mydatabase/articles/1` - `tag-python.html` (and more) as the content of `/mydatabase/tags/python`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108671952, https://github.com/simonw/datasette/issues/1662#issuecomment-1072905467,https://api.github.com/repos/simonw/datasette/issues/1662,1072905467,IC_kwDOBm6k_c4_8zz7,9599,2022-03-19T00:42:23Z,2022-03-19T00:42:23Z,OWNER,"Those client-side SQLite tricks are _really_ neat. `datasette publish` defaults to configuring it so the raw SQLite database can be downloaded from `/fixtures.db` - and this issue updated it to be served with a CORS header that would allow client-side scripts to load the file: - #1057 If you're not going to run any server-side code at all you don't need Datasette for this - you can upload the SQLite database file to any static hosting with CORS headers and load it into the client that way. In terms of static publishing, I do think there's something interesting about using Datasette to generate static sites. There's an issue discussing options for that over here: - #1605","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170497629, https://github.com/simonw/datasette/issues/1661#issuecomment-1072904703,https://api.github.com/repos/simonw/datasette/issues/1661,1072904703,IC_kwDOBm6k_c4_8zn_,9599,2022-03-19T00:37:36Z,2022-03-19T00:37:36Z,OWNER,Updated docs: https://docs.datasette.io/en/latest/performance.html#datasette-hashed-urls,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774, https://github.com/simonw/datasette/issues/1661#issuecomment-1072901159,https://api.github.com/repos/simonw/datasette/issues/1661,1072901159,IC_kwDOBm6k_c4_8ywn,9599,2022-03-19T00:20:27Z,2022-03-19T00:20:27Z,OWNER,I can remove the `default_cache_ttl_hashed` setting too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774, https://github.com/simonw/datasette/pull/1664#issuecomment-1072898923,https://api.github.com/repos/simonw/datasette/issues/1664,1072898923,IC_kwDOBm6k_c4_8yNr,9599,2022-03-19T00:11:33Z,2022-03-19T00:11:33Z,OWNER,I'm going to land this and handle those in separate commits.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173017980, https://github.com/simonw/datasette/pull/1664#issuecomment-1072898797,https://api.github.com/repos/simonw/datasette/issues/1664,1072898797,IC_kwDOBm6k_c4_8yLt,9599,2022-03-19T00:11:09Z,2022-03-19T00:11:09Z,OWNER,Still need to remove it from the documentation and do something about that `hash_urls` setting.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173017980, https://github.com/simonw/datasette/pull/1664#issuecomment-1072890524,https://api.github.com/repos/simonw/datasette/issues/1664,1072890524,IC_kwDOBm6k_c4_8wKc,9599,2022-03-18T23:44:33Z,2022-03-19T00:06:51Z,OWNER,Looks like that was set here: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L490-L492,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173017980, https://github.com/simonw/datasette/pull/1664#issuecomment-1072890205,https://api.github.com/repos/simonw/datasette/issues/1664,1072890205,IC_kwDOBm6k_c4_8wFd,9599,2022-03-18T23:43:15Z,2022-03-18T23:43:15Z,OWNER,"Now almost everything is working except for foreign key expansion: ![CleanShot 2022-03-18 at 16 41 39@2x](https://user-images.githubusercontent.com/9599/159097349-6f41dfdf-5bab-449b-a148-5cda3df6534c.png) Using the debugger I tracked it down to this code: https://github.com/simonw/datasette/blob/30e5f0e67c38054a8087a2a4eae3fc4d1779af90/datasette/views/table.py#L708-L715 Turns out `default_labels` there is `None` - and it's a parameter to that `data()` method: https://github.com/simonw/datasette/blob/30e5f0e67c38054a8087a2a4eae3fc4d1779af90/datasette/views/table.py#L325-L334 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173017980, https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1072834273,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1072834273,IC_kwDOCGYnMM4_8ibh,9599,2022-03-18T21:36:05Z,2022-03-18T21:36:05Z,OWNER,"Python's `str.encode()` method has a `errors=` parameter that does something along these lines: https://docs.python.org/3/library/stdtypes.html#str.encode > *errors* may be given to set a different error handling scheme. The default for *errors* is `'strict'`, meaning that encoding errors raise a [`UnicodeError`](https://docs.python.org/3/library/exceptions.html#UnicodeError ""UnicodeError""). Other possible values are `'ignore'`, `'replace'`, `'xmlcharrefreplace'`, `'backslashreplace'` and any other name registered via [`codecs.register_error()`](https://docs.python.org/3/library/codecs.html#codecs.register_error ""codecs.register_error""), Imitating this might be the way to go.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272, https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1072833174,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1072833174,IC_kwDOCGYnMM4_8iKW,9599,2022-03-18T21:34:06Z,2022-03-18T21:34:06Z,OWNER,"Good call-out: right now the `parsedate()` and `parsedatetime()` functions both terminate with an exception if they hit something invalid: https://sqlite-utils.datasette.io/en/stable/cli.html#sqlite-utils-convert-recipes It would be better if this was configurable by the user (and properly documented) - options could include ""set null if date is invalid"" and ""leave the value as it is if invalid"" in addition to throwing an error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272, https://github.com/simonw/datasette/pull/1664#issuecomment-1071813296,https://api.github.com/repos/simonw/datasette/issues/1664,1071813296,IC_kwDOBm6k_c4_4pKw,9599,2022-03-17T23:26:22Z,2022-03-17T23:26:22Z,OWNER,Probably caused by the convoluted code is `get_format()`: https://github.com/simonw/datasette/blob/30e5f0e67c38054a8087a2a4eae3fc4d1779af90/datasette/views/base.py#L466-L481,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173017980, https://github.com/simonw/datasette/pull/1664#issuecomment-1071809988,https://api.github.com/repos/simonw/datasette/issues/1664,1071809988,IC_kwDOBm6k_c4_4oXE,9599,2022-03-17T23:24:57Z,2022-03-17T23:24:57Z,OWNER,"My hunch is that this is broken because of this: https://github.com/simonw/datasette/blob/30e5f0e67c38054a8087a2a4eae3fc4d1779af90/datasette/app.py#L1098-L1107 Note how the table uses `table_and_format` but the row uses just `table` - I think there's code that's getting confused by this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173017980, https://github.com/simonw/datasette/pull/1664#issuecomment-1071803114,https://api.github.com/repos/simonw/datasette/issues/1664,1071803114,IC_kwDOBm6k_c4_4mrq,9599,2022-03-17T23:22:00Z,2022-03-17T23:22:00Z,OWNER,"Surprisingly I managed to break https://latest.datasette.io/fixtures/custom_foreign_key_label while working on this change: ![CleanShot 2022-03-17 at 16 16 54@2x](https://user-images.githubusercontent.com/9599/158909271-717b65e8-cfcc-44c4-b1cc-f34478b0f803.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173017980, https://github.com/simonw/datasette/issues/1661#issuecomment-1071797707,https://api.github.com/repos/simonw/datasette/issues/1661,1071797707,IC_kwDOBm6k_c4_4lXL,9599,2022-03-17T23:19:24Z,2022-03-17T23:19:24Z,OWNER,"Moving this to PR so I can comment on individual lines: - #1664","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774, https://github.com/simonw/datasette/issues/1661#issuecomment-1071793307,https://api.github.com/repos/simonw/datasette/issues/1661,1071793307,IC_kwDOBm6k_c4_4kSb,9599,2022-03-17T23:17:32Z,2022-03-17T23:17:32Z,OWNER,"Surprisingly I managed to break https://latest.datasette.io/fixtures/custom_foreign_key_label while working on this change: ![CleanShot 2022-03-17 at 16 16 54@2x](https://user-images.githubusercontent.com/9599/158909271-717b65e8-cfcc-44c4-b1cc-f34478b0f803.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774, https://github.com/simonw/datasette/issues/1661#issuecomment-1071706993,https://api.github.com/repos/simonw/datasette/issues/1661,1071706993,IC_kwDOBm6k_c4_4PNx,9599,2022-03-17T22:42:21Z,2022-03-17T22:42:21Z,OWNER,"As part of this I'm going to get rid of this mechanism: https://github.com/simonw/datasette/blob/30e5f0e67c38054a8087a2a4eae3fc4d1779af90/datasette/views/base.py#L170-L173 Unwrapping `request.scope[""url_route""][""kwargs""]` into keyword argument to view functions just made the code harder to follow.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774, https://github.com/simonw/datasette/issues/1663#issuecomment-1071519407,https://api.github.com/repos/simonw/datasette/issues/1663,1071519407,IC_kwDOBm6k_c4_3hav,9599,2022-03-17T21:32:35Z,2022-03-17T21:32:35Z,OWNER,"Updated docs: - https://docs.datasette.io/en/latest/internals.html#datasette-class - https://docs.datasette.io/en/latest/internals.html#db-hash","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170554975, https://github.com/simonw/datasette/issues/1532#issuecomment-1069570893,https://api.github.com/repos/simonw/datasette/issues/1532,1069570893,IC_kwDOBm6k_c4_wFtN,9599,2022-03-16T20:11:41Z,2022-03-16T20:13:34Z,OWNER,"Could also build a CLI Rich/Textual app to exercise the API - which could embed Datasette as a dependency and work using `datasette.client.get(...)` calls. Could be a plugin that adds a `datasette tui` command.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065429936, https://github.com/simonw/datasette/issues/1663#issuecomment-1068742624,https://api.github.com/repos/simonw/datasette/issues/1663,1068742624,IC_kwDOBm6k_c4_s7fg,9599,2022-03-16T05:17:45Z,2022-03-16T05:17:45Z,OWNER,Should be documented here: https://docs.datasette.io/en/stable/internals.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170554975, https://github.com/simonw/datasette/issues/1661#issuecomment-1068728484,https://api.github.com/repos/simonw/datasette/issues/1661,1068728484,IC_kwDOBm6k_c4_s4Ck,9599,2022-03-16T04:47:39Z,2022-03-16T04:47:39Z,OWNER,https://datasette.io/plugins/datasette-hashed-urls is released now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774, https://github.com/simonw/datasette/issues/1661#issuecomment-1068630353,https://api.github.com/repos/simonw/datasette/issues/1661,1068630353,IC_kwDOBm6k_c4_sgFR,9599,2022-03-16T01:24:56Z,2022-03-16T01:25:49Z,OWNER,"Here's the only bit of code that references that `_hash` mechanism: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L259-L265 And here's the test: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/tests/test_api.py#L828-L854 Related issue: - #471","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774, https://github.com/simonw/datasette/issues/1661#issuecomment-1068628839,https://api.github.com/repos/simonw/datasette/issues/1661,1068628839,IC_kwDOBm6k_c4_sftn,9599,2022-03-16T01:21:36Z,2022-03-16T01:21:48Z,OWNER,"From https://docs.datasette.io/en/0.60.2/performance.html#hashed-url-mode > You can enable these hashed URLs in two ways: using the [hash_urls](https://docs.datasette.io/en/0.60.2/settings.html#setting-hash-urls) configuration setting (which affects all requests to Datasette) or via the `?_hash=1` query string parameter (which only applies to the current request). I'm going to drop` ?_hash=1` entirely. I'd actually forgotten that feature existed!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774, https://github.com/simonw/datasette/issues/1661#issuecomment-1068554827,https://api.github.com/repos/simonw/datasette/issues/1661,1068554827,IC_kwDOBm6k_c4_sNpL,9599,2022-03-15T23:16:58Z,2022-03-15T23:18:58Z,OWNER,"If you attempt to use the [old setting](https://docs.datasette.io/en/stable/settings.html#hash-urls): datasette mydatabase.db --setting hash_urls 1 It should error with a message saying that the feature has been moved to a plugin. I'll do this with a `deprecated_settings` mechanism so the error can be detected even though `datasette --help-settings` will no longer return the setting. https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/cli.py#L479-L489","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774, https://github.com/simonw/datasette/issues/1661#issuecomment-1068553454,https://api.github.com/repos/simonw/datasette/issues/1661,1068553454,IC_kwDOBm6k_c4_sNTu,9599,2022-03-15T23:14:37Z,2022-03-15T23:14:37Z,OWNER,"This is going to simplify the code in the various view classes substantially: - #1660","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774, https://github.com/simonw/datasette/issues/647#issuecomment-1068552696,https://api.github.com/repos/simonw/datasette/issues/647,1068552696,IC_kwDOBm6k_c4_sNH4,9599,2022-03-15T23:13:06Z,2022-03-15T23:13:06Z,OWNER,"The plugin works. I'm going to implement one last feature for it: - https://github.com/simonw/datasette-hashed-urls/issues/3 Then I can remove hashed URL mode in a separate issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",531755959, https://github.com/simonw/datasette/issues/647#issuecomment-1068539404,https://api.github.com/repos/simonw/datasette/issues/647,1068539404,IC_kwDOBm6k_c4_sJ4M,9599,2022-03-15T22:49:01Z,2022-03-15T22:49:01Z,OWNER,"I shipped the first version of this: https://github.com/simonw/datasette-hashed-urls Next step: test it with a live demo: - https://github.com/simonw/datasette-hashed-urls/issues/2","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",531755959, https://github.com/simonw/datasette/issues/1439#issuecomment-1068461449,https://api.github.com/repos/simonw/datasette/issues/1439,1068461449,IC_kwDOBm6k_c4_r22J,9599,2022-03-15T20:51:26Z,2022-03-15T20:51:26Z,OWNER,I'm happy with this now that I've landed Tilde encoding in #1657.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047, https://github.com/simonw/datasette/issues/857#issuecomment-1068450483,https://api.github.com/repos/simonw/datasette/issues/857,1068450483,IC_kwDOBm6k_c4_r0Kz,9599,2022-03-15T20:43:55Z,2022-03-15T20:43:55Z,OWNER,Dupe of #1510.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642297505, https://github.com/simonw/datasette/issues/1509#issuecomment-1068445412,https://api.github.com/repos/simonw/datasette/issues/1509,1068445412,IC_kwDOBm6k_c4_ry7k,9599,2022-03-15T20:37:50Z,2022-03-15T20:38:56Z,OWNER,"... maybe Datasette itself should include interactive API documentation, in addition to documenting it in the manual? `/dbname/table/-/apidocs` could return documentation about the specific table, taking into account columns and types.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1054243511, https://github.com/simonw/datasette/issues/1509#issuecomment-1068444767,https://api.github.com/repos/simonw/datasette/issues/1509,1068444767,IC_kwDOBm6k_c4_ryxf,9599,2022-03-15T20:37:03Z,2022-03-15T20:37:03Z,OWNER,"Idea: I could add Pydantic https://pydantic-docs.helpmanual.io/usage/schema/ as an optional test dependency and use it to generate JSON schemas and run validation against examples in the API documentation. Maybe generate API documentation from it too?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1054243511, https://github.com/simonw/datasette/issues/1510#issuecomment-1068443509,https://api.github.com/repos/simonw/datasette/issues/1510,1068443509,IC_kwDOBm6k_c4_ryd1,9599,2022-03-15T20:35:29Z,2022-03-15T20:35:29Z,OWNER,If I set a rule that everything available in the template context MUST also be available via the JSON API (maybe through an extras mechanism) I can combine this with API documentation and solve both at once.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1054244712, https://github.com/simonw/datasette/issues/870#issuecomment-650696054,https://api.github.com/repos/simonw/datasette/issues/870,650696054,MDEyOklzc3VlQ29tbWVudDY1MDY5NjA1NA==,9599,2020-06-28T04:52:41Z,2022-03-15T20:07:17Z,OWNER,"This would be a lot easier if I had extracted out the hash logic to a plugin, see: - #647","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",646737558, https://github.com/simonw/datasette/issues/1660#issuecomment-1068418619,https://api.github.com/repos/simonw/datasette/issues/1660,1068418619,IC_kwDOBm6k_c4_rsY7,9599,2022-03-15T20:06:19Z,2022-03-15T20:06:19Z,OWNER,"Also related: - #878 - #1512 - #1518 - #870 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170144879, https://github.com/simonw/datasette/issues/1660#issuecomment-1068417357,https://api.github.com/repos/simonw/datasette/issues/1660,1068417357,IC_kwDOBm6k_c4_rsFN,9599,2022-03-15T20:05:08Z,2022-03-15T20:05:08Z,OWNER,"`DataView` is used as the base class for: - `DatabaseView` - `DatabaseDownload` (just so the permissions checks can be called) - `QueryView` - which isn't routed to directly, it's called from `DatabaseView` if `?sql=` is available and `TableView` for canned queries - `RowTableShared` which is the base class for `TableView` and `RowView`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170144879, https://github.com/simonw/datasette/issues/1660#issuecomment-1068415072,https://api.github.com/repos/simonw/datasette/issues/1660,1068415072,IC_kwDOBm6k_c4_rrhg,9599,2022-03-15T20:02:36Z,2022-03-15T20:02:36Z,OWNER,"This is one of the worst bits - the `get_format()` method on the `DataView` base class actually modifies `args`, including removing keys! Really confusing: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L454-L482 Then `BaseView` has some surprising responsibilities. It has a utility helper for checking multiple permissions at once: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L81-L105 And its own render method that adds extra stuff to the template context and handles the rel: alternate header: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L131-L157 Then `DataView` does all sorts of weird stuff - from handling database hashes (which I want to remove, see #647): https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L206-L219 To streaming CSV responses: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L286-L308 To handling SQLite exceptions: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L514-L526 And a ton more. It' s a big mess.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170144879, https://github.com/simonw/datasette/issues/1062#issuecomment-1068327874,https://api.github.com/repos/simonw/datasette/issues/1062,1068327874,IC_kwDOBm6k_c4_rWPC,9599,2022-03-15T18:33:49Z,2022-03-15T18:33:49Z,OWNER,"I can get regular `.json` to stream too, using the pattern described in this TIL: https://til.simonwillison.net/python/output-json-array-streaming","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732674148, https://github.com/simonw/datasette/issues/1651#issuecomment-1068319530,https://api.github.com/repos/simonw/datasette/issues/1651,1068319530,IC_kwDOBm6k_c4_rUMq,9599,2022-03-15T18:25:42Z,2022-03-15T18:25:42Z,OWNER,"Done: - https://latest.datasette.io/fixtures/table~2Fwith~2Fslashes~2Ecsv - https://latest.datasette.io/fixtures/table~2Fwith~2Fslashes~2Ecsv.csv - https://latest.datasette.io/fixtures/table~2Fwith~2Fslashes~2Ecsv.json","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161584460, https://github.com/simonw/datasette/issues/1657#issuecomment-1068318454,https://api.github.com/repos/simonw/datasette/issues/1657,1068318454,IC_kwDOBm6k_c4_rT72,9599,2022-03-15T18:25:11Z,2022-03-15T18:25:11Z,OWNER,"Demo: - https://latest.datasette.io/fixtures/table~2Fwith~2Fslashes~2Ecsv - https://latest.datasette.io/fixtures/table~2Fwith~2Fslashes~2Ecsv.csv - https://latest.datasette.io/fixtures/table~2Fwith~2Fslashes~2Ecsv.json","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756, https://github.com/simonw/datasette/issues/1657#issuecomment-1068306916,https://api.github.com/repos/simonw/datasette/issues/1657,1068306916,IC_kwDOBm6k_c4_rRHk,9599,2022-03-15T18:15:11Z,2022-03-15T18:15:11Z,OWNER,Now live here: https://fivethirtyeight.datasettes.com/fivethirtyeight/august-senate-polls~2Faugust_senate_polls,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756, https://github.com/simonw/datasette/issues/1657#issuecomment-1068296042,https://api.github.com/repos/simonw/datasette/issues/1657,1068296042,IC_kwDOBm6k_c4_rOdq,9599,2022-03-15T18:05:54Z,2022-03-15T18:05:54Z,OWNER,Documentation: https://docs.datasette.io/en/latest/internals.html#tilde-encoding,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756, https://github.com/simonw/datasette/pull/1659#issuecomment-1068193035,https://api.github.com/repos/simonw/datasette/issues/1659,1068193035,IC_kwDOBm6k_c4_q1UL,22429695,2022-03-15T16:28:25Z,2022-03-15T17:56:09Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1659?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1659](https://codecov.io/gh/simonw/datasette/pull/1659?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (85dde28) into [main](https://codecov.io/gh/simonw/datasette/commit/c10cd48baf106659bf3f129ad7bfb2226be73821?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (c10cd48) will **increase** coverage by `0.03%`. > The diff coverage is `100.00%`. > :exclamation: Current head 85dde28 differs from pull request most recent head 99b8263. Consider uploading reports for the commit 99b8263 to get more accurate results ```diff @@ Coverage Diff @@ ## main #1659 +/- ## ========================================== + Coverage 92.06% 92.10% +0.03% ========================================== Files 34 34 Lines 4576 4584 +8 ========================================== + Hits 4213 4222 +9 + Misses 363 362 -1 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1659?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1659/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.36% <100.00%> (ø)` | | | [datasette/url\_builder.py](https://codecov.io/gh/simonw/datasette/pull/1659/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3VybF9idWlsZGVyLnB5) | `100.00% <100.00%> (ø)` | | | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1659/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.84% <100.00%> (-0.13%)` | :arrow_down: | | [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1659/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `96.07% <100.00%> (+0.58%)` | :arrow_up: | | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1659/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `96.21% <100.00%> (+0.01%)` | :arrow_up: | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1659?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1659?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [c10cd48...99b8263](https://codecov.io/gh/simonw/datasette/pull/1659?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1169895600, https://github.com/simonw/datasette/issues/1657#issuecomment-1068181623,https://api.github.com/repos/simonw/datasette/issues/1657,1068181623,IC_kwDOBm6k_c4_qyh3,9599,2022-03-15T16:18:23Z,2022-03-15T16:18:23Z,OWNER,Moving this to a PR.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756, https://github.com/simonw/datasette/pull/1656#issuecomment-1068154183,https://api.github.com/repos/simonw/datasette/issues/1656,1068154183,IC_kwDOBm6k_c4_qr1H,22429695,2022-03-15T15:55:34Z,2022-03-15T15:55:34Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1656?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1656](https://codecov.io/gh/simonw/datasette/pull/1656?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (5d9883f) into [main](https://codecov.io/gh/simonw/datasette/commit/c10cd48baf106659bf3f129ad7bfb2226be73821?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (c10cd48) will **not change** coverage. > The diff coverage is `n/a`. ```diff @@ Coverage Diff @@ ## main #1656 +/- ## ======================================= Coverage 92.06% 92.06% ======================================= Files 34 34 Lines 4576 4576 ======================================= Hits 4213 4213 Misses 363 363 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1656?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1656?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [c10cd48...5d9883f](https://codecov.io/gh/simonw/datasette/pull/1656?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168357113, https://github.com/simonw/datasette/issues/1657#issuecomment-1068148013,https://api.github.com/repos/simonw/datasette/issues/1657,1068148013,IC_kwDOBm6k_c4_qqUt,9599,2022-03-15T15:50:15Z,2022-03-15T15:50:15Z,OWNER,"The thing that broke everything was this change: I'm going to bring back the horrible `get_format()` method for the moment, with its weird mutations of the `args` object, then try and get rid of it again later.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756, https://github.com/simonw/datasette/issues/1658#issuecomment-1068138578,https://api.github.com/repos/simonw/datasette/issues/1658,1068138578,IC_kwDOBm6k_c4_qoBS,9599,2022-03-15T15:42:49Z,2022-03-15T15:42:49Z,OWNER,"Easiest way to do this was with three reverts, then cherry-pick back the code of conduct.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1169840669, https://github.com/simonw/datasette/issues/1657#issuecomment-1068126821,https://api.github.com/repos/simonw/datasette/issues/1657,1068126821,IC_kwDOBm6k_c4_qlJl,9599,2022-03-15T15:31:54Z,2022-03-15T15:31:54Z,OWNER,The state I had got to prior to that revert is in https://github.com/simonw/datasette/tree/issue-1657-wip,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756, https://github.com/simonw/datasette/issues/1657#issuecomment-1068125636,https://api.github.com/repos/simonw/datasette/issues/1657,1068125636,IC_kwDOBm6k_c4_qk3E,9599,2022-03-15T15:30:54Z,2022-03-15T15:30:54Z,OWNER,I've made a real mess of this. I'm going to revert Datasette`main` back to the last commit that passed the tests and try this again in a branch.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756, https://github.com/simonw/sqlite-utils/issues/131#issuecomment-1067981656,https://api.github.com/repos/simonw/sqlite-utils/issues/131,1067981656,IC_kwDOCGYnMM4_qBtY,25778,2022-03-15T13:21:42Z,2022-03-15T13:21:42Z,CONTRIBUTOR,"Just ran into this issue last night. I have a big table that's _mostly_ numbers, but also a zip code column in a state where ZIP codes start with 0. Would be great to run something like this: ```sh sqlite-utils insert data.db places file.csv --csv --detect-types --type zipcode text ``` Maybe I'll take a crack at this one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675753042, https://github.com/simonw/datasette/issues/1657#issuecomment-1067423720,https://api.github.com/repos/simonw/datasette/issues/1657,1067423720,IC_kwDOBm6k_c4_n5fo,9599,2022-03-14T23:59:56Z,2022-03-14T23:59:56Z,OWNER,"Updated test: ```python @pytest.mark.parametrize( ""original,expected"", ( (""abc"", ""abc""), (""/foo/bar"", ""~2Ffoo~2Fbar""), (""/-/bar"", ""~2F-~2Fbar""), (""-/db-/table.csv"", ""-~2Fdb-~2Ftable~2Ecsv""), (r""%~-/"", ""~25~7E-~2F""), (""~25~7E~2D~2F"", ""~7E25~7E7E~7E2D~7E2F""), ), ) def test_tilde_encoding(original, expected): actual = utils.tilde_encode(original) assert actual == expected # And test round-trip assert original == utils.tilde_decode(actual) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756, https://github.com/simonw/datasette/issues/1657#issuecomment-1067414156,https://api.github.com/repos/simonw/datasette/issues/1657,1067414156,IC_kwDOBm6k_c4_n3KM,9599,2022-03-14T23:38:41Z,2022-03-14T23:38:41Z,OWNER,"And in https://datatracker.ietf.org/doc/html/rfc3986#section-2.3 ""Unreserved Characters"": unreserved = ALPHA / DIGIT / ""-"" / ""."" / ""_"" / ""~""","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756, https://github.com/simonw/datasette/issues/1657#issuecomment-1067413691,https://api.github.com/repos/simonw/datasette/issues/1657,1067413691,IC_kwDOBm6k_c4_n3C7,9599,2022-03-14T23:37:42Z,2022-03-14T23:37:42Z,OWNER,"Relevant: https://datatracker.ietf.org/doc/html/rfc3986#section-2.1 ``` reserved = gen-delims / sub-delims gen-delims = "":"" / ""/"" / ""?"" / ""#"" / ""["" / ""]"" / ""@"" sub-delims = ""!"" / ""$"" / ""&"" / ""'"" / ""("" / "")"" / ""*"" / ""+"" / "","" / "";"" / ""="" ``` Notably `~` is not in either of those lists.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756, https://github.com/simonw/datasette/issues/1651#issuecomment-1067382442,https://api.github.com/repos/simonw/datasette/issues/1651,1067382442,IC_kwDOBm6k_c4_nvaq,9599,2022-03-14T22:59:10Z,2022-03-14T22:59:10Z,OWNER,"This work is now blocked on: - https://github.com/simonw/datasette/issues/1657","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161584460, https://github.com/simonw/datasette/issues/1657#issuecomment-1067382232,https://api.github.com/repos/simonw/datasette/issues/1657,1067382232,IC_kwDOBm6k_c4_nvXY,9599,2022-03-14T22:58:47Z,2022-03-14T22:58:47Z,OWNER,"Asked about this [on Twitter](https://twitter.com/simonw/status/1503499169775849473): > Anyone ever seen a proxy or other URL handling system do anything surprising with the tilde ""~"" character? > > I'm considering it as an escaping character, in place of ""-"" as described in Replies so far seem like it should be OK - Apache has supported this for home directories for a couple of decades now without any problems.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756, https://github.com/simonw/datasette/issues/1657#issuecomment-1067381556,https://api.github.com/repos/simonw/datasette/issues/1657,1067381556,IC_kwDOBm6k_c4_nvM0,9599,2022-03-14T22:57:27Z,2022-03-14T22:57:45Z,OWNER,"The problem with the [dash encoding mechanism](https://simonwillison.net/2022/Mar/5/dash-encoding/) is that it turns out dashes are used in a LOT of existing Datasette instances - much of https://fivethirtyeight.datasettes.com/fivethirtyeight for example, and even https://datasette.io/ itself: https://datasette.io/dogsheep-index It's pretty ugly to force all of those to change to their dash-encoded equivalent - and in fact it broke https://datasette.io/ in a subtle way: - https://github.com/simonw/datasette.io/issues/94 I'm going to try using `~` instead and see if that works as well and causes less breakage to existing sites.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756, https://github.com/simonw/datasette/issues/1384#issuecomment-1066222323,https://api.github.com/repos/simonw/datasette/issues/1384,1066222323,IC_kwDOBm6k_c4_jULz,2670795,2022-03-14T00:36:42Z,2022-03-14T00:36:42Z,CONTRIBUTOR,"> Ah, sorry, I didn't get what you were saying you the first time. Using _metadata_local in that way makes total sense -- I agree, refreshing metadata each cell was seeming quite excessive. Now I'm on the same page! :) All good. Report back any issues you find with this stuff. Metadata/dynamic config hasn't been tested widely outside of what I've done AFAIK. If you find a strong use case for async meta, it's going to be better to know sooner rather than later!","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135, https://github.com/simonw/datasette/issues/1384#issuecomment-1066194130,https://api.github.com/repos/simonw/datasette/issues/1384,1066194130,IC_kwDOBm6k_c4_jNTS,167160,2022-03-13T22:23:04Z,2022-03-13T22:23:04Z,NONE,"Ah, sorry, I didn't get what you were saying you the first time. Using _metadata_local in that way makes total sense -- I agree, refreshing metadata each cell was seeming quite excessive. Now I'm on the same page! :)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135, https://github.com/simonw/datasette/issues/1384#issuecomment-1066169718,https://api.github.com/repos/simonw/datasette/issues/1384,1066169718,IC_kwDOBm6k_c4_jHV2,2670795,2022-03-13T19:48:49Z,2022-03-13T19:48:49Z,CONTRIBUTOR,"> For my reference, did you include a `render_cell` plugin calling `get_metadata` in those tests? You shouldn't need to do this, as I mentioned previously. The code inside `render_cell` hook already has access to the most recently sync'd metadata via `datasette._metadata_local`. Refreshing the metadata for every cell seems ... excessive.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135, https://github.com/simonw/datasette/issues/1384#issuecomment-1066143991,https://api.github.com/repos/simonw/datasette/issues/1384,1066143991,IC_kwDOBm6k_c4_jBD3,167160,2022-03-13T17:13:09Z,2022-03-13T17:13:09Z,NONE,"Thanks for taking the time to reply @brandonrobertz , this is really helpful info. > See ""Many small queries are efficient in sqlite"" for more information on the rationale here. Also note that in the datasette-live-config reference plugin, the DB connection is cached, so that eliminated most of the performance worries we had. Ah, that's nifty! Yeah, then caching on the python side is likely a waste :) I'm new to working with sqlite so this is super good to know the many-small-queries is a common pattern > I tested on very large Datasette deployments (hundreds of DBs, millions of rows). For my reference, did you include a `render_cell` plugin calling `get_metadata` in those tests? I'm less concerned now that I know a little more about sqlite's caching, but that special situation will jump you to a few orders of magnitude above what the sqlite article describes (e.g. 200 vs 20,000 queries+metadata merges for a page displaying 100 rows of a 200 column table). It wouldn't scale with db size as much as # of visible cells being rendered on the page, although they would be identical queries I suppose so will cache well. (If you didn't test this specific situation, no worries -- I'm just trying to calibrate my intuition on this and can do my own benchmarks at some point.) > Simon talked about eventually making something like this a standard feature of Datasette Yeah, getting metadata (and static pages as well for that matter) from internal tables definitely has my vote for including as a standard feature! Its really nice to be able to distribute a single *.db with all the metadata and static pages bundled. My metadata are sufficiently complex/domain specific that it makes sense to continue on my own plugin for now, but I'll be thinking about more general parts I can spin off as possible contributions to liveconfig (if you're open to them) or other plugins in this ecosystem.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135, https://github.com/simonw/sqlite-utils/issues/408#issuecomment-1066139147,https://api.github.com/repos/simonw/sqlite-utils/issues/408,1066139147,IC_kwDOCGYnMM4_i_4L,24938923,2022-03-13T16:45:00Z,2022-03-13T16:54:09Z,NONE,"@simonw Now I get this: ``` (app-root) sqlite-utils indexes global.db --table Error: near ""("": syntax error (app-root) sqlite-utils --version sqlite-utils, version 3.25.1 (app-root) sqlite3 --version 3.36.0 2021-06-18 18:36:39 (app-root) python --version Python 3.8.11 ``` Dockerfile ``` FROM centos/python-38-centos7 USER root RUN yum update -y RUN yum upgrade -y # epel RUN yum -y install epel-release && yum clean all # SQLite RUN yum -y install zlib-devel geos geos-devel proj proj-devel freexl freexl-devel libxml2-devel WORKDIR /build/ COPY sqlite-autoconf-3360000.tar.gz ./ RUN tar -zxf sqlite-autoconf-3360000.tar.gz WORKDIR /build/sqlite-autoconf-3360000 RUN ./configure RUN make RUN make install # RUN /opt/app-root/bin/python3.8 -m pip install --upgrade pip RUN pip install sqlite-utils ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1145882578, https://github.com/simonw/datasette/issues/1384#issuecomment-1066006292,https://api.github.com/repos/simonw/datasette/issues/1384,1066006292,IC_kwDOBm6k_c4_ifcU,2670795,2022-03-13T02:09:44Z,2022-03-13T02:09:44Z,CONTRIBUTOR,"> If I'm understanding your plugin code correctly, you query the db using the sync handle every time `get_metdata` is called, right? Won't this become a pretty big bottleneck if a hook into `render_cell` is trying to read metadata / plugin config? Reading from sqlite DBs is pretty quick and I didn't notice significant performance issues when I was benchmarking. I tested on very large Datasette deployments (hundreds of DBs, millions of rows). See [""Many small queries are efficient in sqlite""](https://sqlite.org/np1queryprob.html) for more information on the rationale here. Also note that in the [datasette-live-config](https://github.com/next-LI/datasette-live-config) reference plugin, the DB connection is cached, so that eliminated most of the performance worries we had. If you need to ensure fresh metadata is being read inside of a `render_cell` hook specifically, you don't need to do anything further! `get_metadata` gets called before `render_cell` every request, so it already has access to the synced meta. There shouldn't be a need to call `get_metadata(...)` or `metadata(...)` inside `render_cell`, you can just use `datasette._metadata_local` if you're really worried about performance. > The plugin is close, but looks like it only grabs remote metadata, is that right? Instead what I'm wanting is to grab metadata embedded in the attached databases. Yes correct, the datadette-remote-metadata plugin doesn't do that. But the datasette-live-config plugin does. [It supports a `__metadata` table](https://github.com/next-LI/datasette-live-config/blob/main/datasette_live_config/__init__.py#L107-L138) that, when it exists on an attached DB, gets pulled into the Datasette internal `_metadata` and is also accessible via `get_metadata`. Updating is instantaneous so there's no gotchas for users or security issues for users relying on the metadata-based permissions. Simon talked about eventually making something like this a standard feature of Datasette, but I'm not sure what the status is on that! Good luck!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135, https://github.com/simonw/datasette/issues/1439#issuecomment-1065988403,https://api.github.com/repos/simonw/datasette/issues/1439,1065988403,IC_kwDOBm6k_c4_ibEz,9599,2022-03-13T00:06:38Z,2022-03-13T00:07:19Z,OWNER,"If I want to reserve `-` as a character that CAN be used in URLs, the only remaining character that might make sense for escape sequences is `~` - based on this last line of characters that are escape from percentage encoding: ```python _ALWAYS_SAFE = frozenset(b'ABCDEFGHIJKLMNOPQRSTUVWXYZ' b'abcdefghijklmnopqrstuvwxyz' b'0123456789' b'_.-~') ``` So I'd add both `-` and `_` back to the safe list, but use `~` to escape `.` and `/` and suchlike.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047, https://github.com/simonw/datasette/issues/1439#issuecomment-1065987808,https://api.github.com/repos/simonw/datasette/issues/1439,1065987808,IC_kwDOBm6k_c4_ia7g,9599,2022-03-13T00:02:32Z,2022-03-13T00:02:32Z,OWNER,"OK, this has broken a lot more than I expected it would. Turns out `-` is a very common character in existing Datasette database names! https://datasette.io/-/databases for example has two: ```json [ { ""name"": ""docs-index"", ""path"": ""docs-index.db"", ""size"": 1007616, ""is_mutable"": false, ""is_memory"": false, ""hash"": ""0ac6c3de2762fcd174fd249fed8a8fa6046ea345173d22c2766186bf336462b2"" }, { ""name"": ""dogsheep-index"", ""path"": ""dogsheep-index.db"", ""size"": 5496832, ""is_mutable"": false, ""is_memory"": false, ""hash"": ""d1ea238d204e5b9ae783c86e4af5bcdf21267c1f391de3e468d9665494ee012a"" } ] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047, https://github.com/simonw/datasette/issues/1384#issuecomment-1065951744,https://api.github.com/repos/simonw/datasette/issues/1384,1065951744,IC_kwDOBm6k_c4_iSIA,167160,2022-03-12T19:47:17Z,2022-03-12T19:47:17Z,NONE,"Awesome, thanks @brandonrobertz ! The plugin is close, but looks like it only grabs remote metadata, is that right? Instead what I'm wanting is to grab metadata embedded in the attached databases. Rather than extending that plugin, at this point I've realized I need a lot more flexibility in metadata for my data model (esp around formatting cell values and custom file exports) so rather than extending that I'll continue working on a plugin specific to my app. If I'm understanding your plugin code correctly, you query the db using the sync handle every time `get_metdata` is called, right? Won't this become a pretty big bottleneck if a hook into `render_cell` is trying to read metadata / plugin config? > Making the get_metadata async won't improve the situation by itself as only some of the code paths accessing metadata use that hook. The other paths use the internal metadata dict. I agree -- because things like `render_cell` will potentially want to read metadata/config, `get_metadata` should really remain sync and lightweight, which we can do with something like the remote-metadata plugin that could also poll metadata tables in attached databases. That leaves your app, where it sounds like you want changes made by the user in the browser in to be immediately reflected, rather than have to wait for the next metadata refresh. In this case I wonder if you could have your app make a sync write to the datasette object so the change would have the immediate effect, but then have a separate async polling mechanism to eventually write that change out to the database for long-term persistence. Then you'd have the best of both worlds, I think? But probably not worth the trouble if your use cases are small (and/or you're not reading metadata/config from tight loops like render_cell).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135, https://github.com/simonw/datasette/issues/1384#issuecomment-1065940779,https://api.github.com/repos/simonw/datasette/issues/1384,1065940779,IC_kwDOBm6k_c4_iPcr,2670795,2022-03-12T18:49:29Z,2022-03-12T18:50:07Z,CONTRIBUTOR,"Hello! Just wanted to chime in and note that there's a plugin to have Datasette [watch for updates to an external metadata.yaml/json and update the internal settings accordingly](https://datasette.io/plugins/datasette-remote-metadata), so I think the cache/poll use case is already covered. @khusmann If you don't need truly dynamic metadata then what you've come up with or the plugin ought to work fine. Making the get_metadata async won't improve the situation by itself as only some of the code paths accessing metadata use that hook. The other paths use the internal metadata dict. Trying to force all paths through a async hook would have performance ramifications and making everything use the internal meta will cause problems for users that need changes to take effect immediately. This is why I came to the non-async solution as it was the path of least change within Datasette. As always, open to new ideas, etc!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135, https://github.com/simonw/datasette/issues/1384#issuecomment-1065929510,https://api.github.com/repos/simonw/datasette/issues/1384,1065929510,IC_kwDOBm6k_c4_iMsm,167160,2022-03-12T17:49:59Z,2022-03-12T17:49:59Z,NONE,"Ok, I'm taking a slightly different approach, which I think is sort of close to the in-memory _metadata table idea. I'm using a startup hook to load metadata / other info from the database, which I store in the datasette object for later: ``` @hookimpl def startup(datasette): async def inner(): datasette._mypluginmetadata = # await db query return inner ``` Then, I can use this in other plugins: ``` @hookimpl def render_cell(value, column, table, database, datasette): # use datasette._mypluginmetadata ``` For my app I don't need anything to update dynamically so it's fine to pre-populate everything on startup. It's also good to have things precached especially for a hook like render_cell, which would otherwise require a ton of redundant db queries. Makes me wonder if we could take a sort of similar caching approach with the internal _metadata table. Like have a little watchdog that could query all of the attached dbs for their _metadata tables every 5min or so, which then could be merged into the in memory _metadata table which then could be accessed sync by the plugins, or something like that. For most the use cases I can think of, live updates don't need to take into effect immediately; refreshing a cache every 5min or on some other trigger (adjustable w a config setting) would be just fine. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135, https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065597709,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065597709,IC_kwDOCGYnMM4_g7sN,9599,2022-03-11T22:32:43Z,2022-03-11T22:32:43Z,OWNER,"Trying to figure out what that extra field in `table_info` compared to `table_xinfo` is: ``` >>> list(db.query(""PRAGMA table_xinfo('t')"")) [{'cid': 0, 'name': 'body', 'type': 'TEXT', 'notnull': 0, 'dflt_value': None, 'pk': 0, 'hidden': 0}, {'cid': 1, 'name': 'd', 'type': 'INT', 'notnull': 0, 'dflt_value': None, 'pk': 0, 'hidden': 2}] `` Presumably `hidden` 0 v.s 2 v.s. other values has meaning.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160034488, https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065596417,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065596417,IC_kwDOCGYnMM4_g7YB,9599,2022-03-11T22:30:15Z,2022-03-11T22:30:15Z,OWNER,"I tried it out in Jupyter and it works as advertised: Introspection is a bit weird: there doesn't seem to be a way to introspect generated columns outside of parsing the stored SQL schema for the columns at the moment! And the `.columns` method doesn't return them at all: https://github.com/simonw/sqlite-utils/blob/433813612ff9b4b501739fd7543bef0040dd51fe/sqlite_utils/db.py#L1207-L1213 Here's why: ``` >>> db.execute(""PRAGMA table_info('t')"").fetchall() [(0, 'body', 'TEXT', 0, None, 0)] >>> db.execute(""PRAGMA table_xinfo('t')"").fetchall() [(0, 'body', 'TEXT', 0, None, 0, 0), (1, 'd', 'INT', 0, None, 0, 2)] ``` So `table_xinfo()` is needed to get back columns including generated columns: https://www.sqlite.org/pragma.html#pragma_table_xinfo > **PRAGMA** *schema.***table_xinfo(***table-name***);** > > This pragma returns one row for each column in the named table, including [hidden columns](https://www.sqlite.org/vtab.html#hiddencol) in virtual tables. The output is the same as for [PRAGMA table_info](https://www.sqlite.org/pragma.html#pragma_table_info) except that hidden columns are shown rather than being omitted.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160034488, https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065402557,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065402557,IC_kwDOCGYnMM4_gMC9,9599,2022-03-11T19:01:08Z,2022-03-11T21:42:25Z,OWNER,"Just spotted this in https://www.sqlite.org/gencol.html > The only functional difference is that one cannot add new STORED columns using the [ALTER TABLE ADD COLUMN](https://www.sqlite.org/lang_altertable.html#altertabaddcol) command. Only VIRTUAL columns can be added using ALTER TABLE. So to add stored columns to an existing table we would need to use the `.transform()` trick. Which implies that this should actually be a capability of the various `.create()` methods, since transform works by creating a new table with those and then copying across the old data. Here's where `.transform()` calls `.create_table_sql()` under the hood: https://github.com/simonw/sqlite-utils/blob/9388edf57aa15719095e3cf0952c1653cd070c9b/sqlite_utils/db.py#L1627-L1637","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160034488, https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065389386,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065389386,IC_kwDOCGYnMM4_gI1K,9599,2022-03-11T18:42:53Z,2022-03-11T21:40:51Z,OWNER,"The Python API could be: ```python db[table_name].add_generated_column(""field"", str, ""json_extract(data, '$.field')"", stored=True) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160034488, https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065477258,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065477258,IC_kwDOCGYnMM4_geSK,25778,2022-03-11T20:14:59Z,2022-03-11T20:14:59Z,CONTRIBUTOR,"Good call on adding this to `create-table`, especially for stored columns. Having the stored/virtual split might make this tricky to implement, but I haven't gone any farther than thinking about what the CLI looks like. I'm going to try making the SQL side work first and figure that'll tell me more about what it needs.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160034488, https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065458729,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065458729,IC_kwDOCGYnMM4_gZwp,9599,2022-03-11T19:58:50Z,2022-03-11T20:00:25Z,OWNER,"I'm coming round to your suggestion to have this as extra arguments to `sqlite-utils add-column` now, especially since you also need to pass a column type. I'd like to come up with syntax for `sqlite-utils create-table` as well. https://sqlite-utils.datasette.io/en/stable/cli-reference.html#create-table Maybe extra `--generated-stored colname expression` (and `--generated`) options would work there.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160034488, https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065440445,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065440445,IC_kwDOCGYnMM4_gVS9,9599,2022-03-11T19:52:15Z,2022-03-11T19:52:15Z,OWNER,"Two new parameters to `.create_table()` and friends: - `generated={...}` - generated column definitions - `generated_stored={...}` generated stored column definitions These columns will be added at the end of the table, but you can use the `column_order=` parameter to apply a different order.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1160034488,