html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app https://github.com/simonw/datasette/issues/1746#issuecomment-1133254599,https://api.github.com/repos/simonw/datasette/issues/1746,1133254599,IC_kwDOBm6k_c5DjBfH,9599,simonw,2022-05-20T19:33:08Z,2022-05-20T19:33:08Z,OWNER,Actually maybe I don't? I just noticed that on other pages on https://docs.datasette.io/en/stable/installation.html the only way to get back to that useful table of context / index page at https://docs.datasette.io/en/stable/index.html is by clicking the tiny house icon. Can I do better or should I have the logo do that?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1243498298,Switch documentation theme to Furo, https://github.com/simonw/datasette/issues/1746#issuecomment-1133252598,https://api.github.com/repos/simonw/datasette/issues/1746,1133252598,IC_kwDOBm6k_c5DjA_2,9599,simonw,2022-05-20T19:31:30Z,2022-05-20T19:31:30Z,OWNER,"I'd also like to bring back this stable / latest / version indicator: ![CleanShot 2022-05-20 at 12 30 49@2x](https://user-images.githubusercontent.com/9599/169598732-e2093ec1-7eaf-40dd-acfa-1a7c31091ff1.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1243498298,Switch documentation theme to Furo, https://github.com/simonw/datasette/issues/1746#issuecomment-1133250151,https://api.github.com/repos/simonw/datasette/issues/1746,1133250151,IC_kwDOBm6k_c5DjAZn,9599,simonw,2022-05-20T19:29:37Z,2022-05-20T19:29:37Z,OWNER,"I want the Datasette logo in the sidebar to link to https://datasette.io/ Looks like I can do that by dropping in my own `sidebar/brand.html` template based on this: https://github.com/pradyunsg/furo/blob/0c2acbbd23f8146dd0ae50a2ba57258c1f63ea9f/src/furo/theme/furo/sidebar/brand.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1243498298,Switch documentation theme to Furo, https://github.com/simonw/datasette/issues/1746#issuecomment-1133246791,https://api.github.com/repos/simonw/datasette/issues/1746,1133246791,IC_kwDOBm6k_c5Di_lH,9599,simonw,2022-05-20T19:26:49Z,2022-05-20T19:26:49Z,OWNER,"Putting this in the `css/custom.css` file seems to work for fixing that logo problem: ```css body[data-theme=""dark""] .sidebar-logo-container { background-color: white; padding: 5px; opacity: 0.6; } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1243498298,Switch documentation theme to Furo, https://github.com/simonw/datasette/issues/1746#issuecomment-1133242063,https://api.github.com/repos/simonw/datasette/issues/1746,1133242063,IC_kwDOBm6k_c5Di-bP,9599,simonw,2022-05-20T19:22:49Z,2022-05-20T19:22:49Z,OWNER,"I have some custom CSS in this file: https://github.com/simonw/datasette/blob/1465fea4798599eccfe7e8f012bd8d9adfac3039/docs/_static/css/custom.css#L1-L7 I tested and the `overflow-wrap: anywhere` is still needed for this fix: - #828 The `.wy-side-nav-search` bit is no longer needed with the new theme.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1243498298,Switch documentation theme to Furo, https://github.com/simonw/datasette/issues/1748#issuecomment-1133232301,https://api.github.com/repos/simonw/datasette/issues/1748,1133232301,IC_kwDOBm6k_c5Di8Ct,9599,simonw,2022-05-20T19:15:00Z,2022-05-20T19:15:00Z,OWNER,"Now live on https://docs.datasette.io/en/latest/testing_plugins.html ![copy](https://user-images.githubusercontent.com/9599/169596586-396eb6c7-ef5a-405a-bb21-348499478d9b.gif) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1243517592,Add copy buttons next to code examples in the documentation, https://github.com/simonw/datasette/issues/1747#issuecomment-1133229196,https://api.github.com/repos/simonw/datasette/issues/1747,1133229196,IC_kwDOBm6k_c5Di7SM,9599,simonw,2022-05-20T19:12:30Z,2022-05-20T19:12:30Z,OWNER,https://docs.datasette.io/en/latest/getting_started.html#follow-a-tutorial,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1243512344,Add tutorials to the getting started guide, https://github.com/simonw/datasette/issues/1748#issuecomment-1133225441,https://api.github.com/repos/simonw/datasette/issues/1748,1133225441,IC_kwDOBm6k_c5Di6Xh,9599,simonw,2022-05-20T19:09:13Z,2022-05-20T19:09:13Z,OWNER,I'm going to add this Sphinx plugin: https://github.com/executablebooks/sphinx-copybutton,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1243517592,Add copy buttons next to code examples in the documentation, https://github.com/simonw/datasette/issues/1153#issuecomment-1133222848,https://api.github.com/repos/simonw/datasette/issues/1153,1133222848,IC_kwDOBm6k_c5Di5vA,9599,simonw,2022-05-20T19:07:10Z,2022-05-20T19:07:10Z,OWNER,I could use https://github.com/pradyunsg/sphinx-inline-tabs for this - recommended by https://pradyunsg.me/furo/recommendations/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON", https://github.com/simonw/datasette/issues/1746#issuecomment-1133217219,https://api.github.com/repos/simonw/datasette/issues/1746,1133217219,IC_kwDOBm6k_c5Di4XD,9599,simonw,2022-05-20T18:58:54Z,2022-05-20T18:58:54Z,OWNER,"Need to address other customizations I've made in https://github.com/simonw/datasette/blob/0.62a0/docs/_templates/layout.html - such as Plausible analytics and some custom JavaScript. https://github.com/simonw/datasette/blob/943aa2e1f7341cb51e60332cde46bde650c64217/docs/_templates/layout.html#L1-L61","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1243498298,Switch documentation theme to Furo, https://github.com/simonw/datasette/issues/1746#issuecomment-1133215684,https://api.github.com/repos/simonw/datasette/issues/1746,1133215684,IC_kwDOBm6k_c5Di3_E,9599,simonw,2022-05-20T18:56:29Z,2022-05-20T18:56:29Z,OWNER,"One other problem: in dark mode the Datasette logo looks bad: This helps a bit: ```css .sidebar-logo-container { background-color: white; padding: 5px; opacity: 0.6; } ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1243498298,Switch documentation theme to Furo, https://github.com/simonw/datasette/issues/1746#issuecomment-1133210942,https://api.github.com/repos/simonw/datasette/issues/1746,1133210942,IC_kwDOBm6k_c5Di20-,9599,simonw,2022-05-20T18:49:40Z,2022-05-20T18:49:40Z,OWNER,"And for those local table of contents, do this: ```rst .. contents:: :local: :class: this-will-duplicate-information-and-it-is-still-useful-here ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1243498298,Switch documentation theme to Furo, https://github.com/simonw/datasette/issues/1746#issuecomment-1133210651,https://api.github.com/repos/simonw/datasette/issues/1746,1133210651,IC_kwDOBm6k_c5Di2wb,9599,simonw,2022-05-20T18:49:11Z,2022-05-20T18:49:11Z,OWNER,"I found a workaround for the no-longer-nested left hand navigation: drop this into `_templates/sidebar/navigation.html`: ```html+jinja
{{ toctree( collapse=True, titles_only=False, maxdepth=3, includehidden=True, ) }}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1243498298,Switch documentation theme to Furo, https://github.com/simonw/datasette/issues/1746#issuecomment-1133210032,https://api.github.com/repos/simonw/datasette/issues/1746,1133210032,IC_kwDOBm6k_c5Di2mw,9599,simonw,2022-05-20T18:48:17Z,2022-05-20T18:48:17Z,OWNER,"A couple of changes I want to make. First, I don't really like the way Furo keeps the in-page titles in a separate menu on the right rather than expanding them on the left. I like this: ![CleanShot 2022-05-20 at 11 43 33@2x](https://user-images.githubusercontent.com/9599/169592611-ac0f9bd2-ff99-49b6-88d3-92dace9d85a6.png) Furo wants to do this instead: I also still want to include those inline tables of contents on the two pages that have them: - https://docs.datasette.io/en/stable/installation.html - https://docs.datasette.io/en/stable/plugin_hooks.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1243498298,Switch documentation theme to Furo, https://github.com/simonw/datasette/issues/1744#issuecomment-1129251699,https://api.github.com/repos/simonw/datasette/issues/1744,1129251699,IC_kwDOBm6k_c5DTwNz,9599,simonw,2022-05-17T19:44:47Z,2022-05-17T19:46:38Z,OWNER,Updated docs: https://docs.datasette.io/en/latest/getting_started.html#using-datasette-on-your-own-computer and https://docs.datasette.io/en/latest/cli-reference.html#datasette-serve-help,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1239008850,`--nolock` feature for opening locked databases, https://github.com/simonw/datasette/issues/1745#issuecomment-1129252603,https://api.github.com/repos/simonw/datasette/issues/1745,1129252603,IC_kwDOBm6k_c5DTwb7,9599,simonw,2022-05-17T19:45:51Z,2022-05-17T19:45:51Z,OWNER,Now documented here: https://docs.datasette.io/en/latest/contributing.html#running-cog,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1239080102,Documentation on running cog, https://github.com/simonw/datasette/issues/1744#issuecomment-1129243427,https://api.github.com/repos/simonw/datasette/issues/1744,1129243427,IC_kwDOBm6k_c5DTuMj,9599,simonw,2022-05-17T19:35:02Z,2022-05-17T19:35:02Z,OWNER,"One thing to note is that the `datasette-copy-to-memory` plugin broke with a locked file, because it does this: https://github.com/simonw/datasette-copy-to-memory/blob/d541c18a78ae6f707a8f9b1e7fc4c020a9f68f2e/datasette_copy_to_memory/__init__.py#L27 ```python tmp.execute(""ATTACH DATABASE ? AS _copy_from"", [db.path]) ``` That would need to use a URI filename too for it to work with locked files.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1239008850,`--nolock` feature for opening locked databases, https://github.com/simonw/datasette/issues/1744#issuecomment-1129241873,https://api.github.com/repos/simonw/datasette/issues/1744,1129241873,IC_kwDOBm6k_c5DTt0R,9599,simonw,2022-05-17T19:33:16Z,2022-05-17T19:33:16Z,OWNER,"I'm going to skip adding a test for this - the test logic would have to be pretty convoluted to exercise it properly, and it's a pretty minor and low-risk feature in the scheme of things.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1239008850,`--nolock` feature for opening locked databases, https://github.com/simonw/datasette/issues/1744#issuecomment-1129241283,https://api.github.com/repos/simonw/datasette/issues/1744,1129241283,IC_kwDOBm6k_c5DTtrD,9599,simonw,2022-05-17T19:32:35Z,2022-05-17T19:32:35Z,OWNER,"I tried writing a test like this: ```python @pytest.mark.parametrize(""locked"", (True, False)) def test_locked_sqlite_db(tmp_path_factory, locked): dir = tmp_path_factory.mktemp(""test_locked_sqlite_db"") test_db = str(dir / ""test.db"") sqlite3.connect(test_db).execute(""create table t (id integer primary key)"") if locked: fp = open(test_db, ""w"") fcntl.lockf(fp.fileno(), fcntl.LOCK_EX) runner = CliRunner() result = runner.invoke( cli, [ ""serve"", ""--memory"", ""--get"", ""/test"", ], catch_exceptions=False, ) ``` But it didn't work, because the test runs in the same process - so taking an exclusive lock on that file didn't cause an error when the test later tried to access it via Datasette!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1239008850,`--nolock` feature for opening locked databases, https://github.com/simonw/datasette/issues/1744#issuecomment-1129187486,https://api.github.com/repos/simonw/datasette/issues/1744,1129187486,IC_kwDOBm6k_c5DTgie,9599,simonw,2022-05-17T18:28:49Z,2022-05-17T18:28:49Z,OWNER,I think I do that with `fcntl.flock()`: https://docs.python.org/3/library/fcntl.html#fcntl.flock,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1239008850,`--nolock` feature for opening locked databases, https://github.com/simonw/datasette/issues/1744#issuecomment-1129185356,https://api.github.com/repos/simonw/datasette/issues/1744,1129185356,IC_kwDOBm6k_c5DTgBM,9599,simonw,2022-05-17T18:26:26Z,2022-05-17T18:26:26Z,OWNER,Not sure how to test this - I'd need to open my own lock against a database file somehow.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1239008850,`--nolock` feature for opening locked databases, https://github.com/simonw/datasette/issues/1744#issuecomment-1129184908,https://api.github.com/repos/simonw/datasette/issues/1744,1129184908,IC_kwDOBm6k_c5DTf6M,9599,simonw,2022-05-17T18:25:57Z,2022-05-17T18:25:57Z,OWNER,"I knocked out a quick prototype of this and it worked! datasette ~/Library/Application\ Support/Google/Chrome/Default/History --nolock Here's the prototype diff: ```diff diff --git a/datasette/app.py b/datasette/app.py index b7b8437..f43700d 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -213,6 +213,7 @@ class Datasette: config_dir=None, pdb=False, crossdb=False, + nolock=False, ): assert config_dir is None or isinstance( config_dir, Path @@ -238,6 +239,7 @@ class Datasette: self.databases = collections.OrderedDict() self._refresh_schemas_lock = asyncio.Lock() self.crossdb = crossdb + self.nolock = nolock if memory or crossdb or not self.files: self.add_database(Database(self, is_memory=True), name=""_memory"") # memory_name is a random string so that each Datasette instance gets its own diff --git a/datasette/cli.py b/datasette/cli.py index 3c6e1b2..7e44665 100644 --- a/datasette/cli.py +++ b/datasette/cli.py @@ -452,6 +452,11 @@ def uninstall(packages, yes): is_flag=True, help=""Enable cross-database joins using the /_memory database"", ) +@click.option( + ""--nolock"", + is_flag=True, + help=""Ignore locking and open locked files in read-only mode"", +) @click.option( ""--ssl-keyfile"", help=""SSL key file"", @@ -486,6 +491,7 @@ def serve( open_browser, create, crossdb, + nolock, ssl_keyfile, ssl_certfile, return_instance=False, @@ -545,6 +551,7 @@ def serve( version_note=version_note, pdb=pdb, crossdb=crossdb, + nolock=nolock, ) # if files is a single directory, use that as config_dir= diff --git a/datasette/database.py b/datasette/database.py index 44d3266..fa55804 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -89,6 +89,8 @@ class Database: # mode=ro or immutable=1? if self.is_mutable: qs = ""?mode=ro"" + if self.ds.nolock: + qs += ""&nolock=1"" else: qs = ""?immutable=1"" assert not (write and not self.is_mutable) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1239008850,`--nolock` feature for opening locked databases, https://github.com/simonw/datasette/issues/1742#issuecomment-1128052948,https://api.github.com/repos/simonw/datasette/issues/1742,1128052948,IC_kwDOBm6k_c5DPLjU,9599,simonw,2022-05-16T19:28:31Z,2022-05-16T19:28:31Z,OWNER,"The trace mechanism is a bit gnarly - it's actually done by some ASGI middleware I wrote, so I'm pretty sure the bug is in there somewhere: https://github.com/simonw/datasette/blob/280ff372ab30df244f6c54f6f3002da57334b3d7/datasette/tracer.py#L73","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1237586379,?_trace=1 fails with datasette-geojson for some reason, https://github.com/simonw/datasette/issues/1742#issuecomment-1128033018,https://api.github.com/repos/simonw/datasette/issues/1742,1128033018,IC_kwDOBm6k_c5DPGr6,9599,simonw,2022-05-16T19:06:38Z,2022-05-16T19:06:38Z,OWNER,The same URL with `.json` instead works fine: https://calands.datasettes.com/calands/CPAD_2020a_SuperUnits.json?_sort=id&id__exact=4&_labels=on&_trace=1,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1237586379,?_trace=1 fails with datasette-geojson for some reason, https://github.com/simonw/datasette/issues/1739#issuecomment-1117662420,https://api.github.com/repos/simonw/datasette/issues/1739,1117662420,IC_kwDOBm6k_c5CnizU,9599,simonw,2022-05-04T18:21:18Z,2022-05-04T18:21:18Z,OWNER,That prototype is now public: https://github.com/simonw/datasette-lite,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223699280,.db downloads should be served with an ETag, https://github.com/simonw/datasette/issues/1739#issuecomment-1116215371,https://api.github.com/repos/simonw/datasette/issues/1739,1116215371,IC_kwDOBm6k_c5CiBhL,9599,simonw,2022-05-03T15:12:16Z,2022-05-03T15:12:16Z,OWNER,"That worked - both DBs are 304 for me now on a subsequent load of the page: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223699280,.db downloads should be served with an ETag, https://github.com/simonw/datasette/issues/1739#issuecomment-1116183369,https://api.github.com/repos/simonw/datasette/issues/1739,1116183369,IC_kwDOBm6k_c5Ch5tJ,9599,simonw,2022-05-03T14:43:14Z,2022-05-03T14:43:14Z,OWNER,Relevant tests start here: https://github.com/simonw/datasette/blob/d60f163528f466b1127b2935c3b6869c34fd6545/tests/test_html.py#L395,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223699280,.db downloads should be served with an ETag, https://github.com/simonw/datasette/issues/1739#issuecomment-1116180599,https://api.github.com/repos/simonw/datasette/issues/1739,1116180599,IC_kwDOBm6k_c5Ch5B3,9599,simonw,2022-05-03T14:40:32Z,2022-05-03T14:40:32Z,OWNER,"Database downloads are served here: https://github.com/simonw/datasette/blob/d60f163528f466b1127b2935c3b6869c34fd6545/datasette/views/database.py#L186-L192 Here's `AsgiFileDownload`: https://github.com/simonw/datasette/blob/d60f163528f466b1127b2935c3b6869c34fd6545/datasette/utils/asgi.py#L410-L430 I can add an `etag=` parameter to that and populate it with `db.hash`, if it is populated (which it always should be for immutable databases that can be downloaded).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223699280,.db downloads should be served with an ETag, https://github.com/simonw/datasette/issues/1739#issuecomment-1116178727,https://api.github.com/repos/simonw/datasette/issues/1739,1116178727,IC_kwDOBm6k_c5Ch4kn,9599,simonw,2022-05-03T14:38:46Z,2022-05-03T14:38:46Z,OWNER,"Reminded myself how this works by reviewing `conditional-get`: https://github.com/simonw/conditional-get/blob/db6dfec0a296080aaf68fcd80e55fb3f0714e738/conditional_get/cli.py#L33-L52 Simply add a `If-None-Match: last-known-etag` header to the request and check that the response is a status 304 with an empty body.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223699280,.db downloads should be served with an ETag, https://github.com/simonw/datasette/issues/1739#issuecomment-1115760104,https://api.github.com/repos/simonw/datasette/issues/1739,1115760104,IC_kwDOBm6k_c5CgSXo,9599,simonw,2022-05-03T05:50:19Z,2022-05-03T05:50:19Z,OWNER,Here's how Starlette does it: https://github.com/encode/starlette/blob/830f3486537916bae6b46948ff922adc14a22b7c/starlette/staticfiles.py#L213,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223699280,.db downloads should be served with an ETag, https://github.com/simonw/datasette/issues/1732#issuecomment-1115533820,https://api.github.com/repos/simonw/datasette/issues/1732,1115533820,IC_kwDOBm6k_c5CfbH8,9599,simonw,2022-05-03T01:42:25Z,2022-05-03T01:42:25Z,OWNER,"Thanks, this definitely sounds like a bug. Do you have simple steps to reproduce this?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1221849746,Custom page variables aren't decoded, https://github.com/simonw/datasette/issues/1737#issuecomment-1115470180,https://api.github.com/repos/simonw/datasette/issues/1737,1115470180,IC_kwDOBm6k_c5CfLlk,9599,simonw,2022-05-02T23:39:29Z,2022-05-02T23:39:29Z,OWNER,"Test ran in 38 seconds and passed! https://github.com/simonw/datasette/runs/6265954274?check_suite_focus=true I'm going to have it run on every commit and PR.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223459734,Automated test for Pyodide compatibility, https://github.com/simonw/datasette/issues/1737#issuecomment-1115468193,https://api.github.com/repos/simonw/datasette/issues/1737,1115468193,IC_kwDOBm6k_c5CfLGh,9599,simonw,2022-05-02T23:35:26Z,2022-05-02T23:35:26Z,OWNER,"https://github.com/simonw/datasette/runs/6265915080?check_suite_focus=true failed but looks like it passed because I forgot to use `set -e` at the start of the bash script. It failed because it didn't have `build` available.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223459734,Automated test for Pyodide compatibility, https://github.com/simonw/datasette/issues/1737#issuecomment-1115464097,https://api.github.com/repos/simonw/datasette/issues/1737,1115464097,IC_kwDOBm6k_c5CfKGh,9599,simonw,2022-05-02T23:27:40Z,2022-05-02T23:27:40Z,OWNER,"I'm going to start off by running this manually - I may run it on every commit once this is all a little bit more stable. I can base the workflow on https://github.com/simonw/scrape-hacker-news-by-domain/blob/main/.github/workflows/scrape.yml","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223459734,Automated test for Pyodide compatibility, https://github.com/simonw/datasette/issues/1737#issuecomment-1115462720,https://api.github.com/repos/simonw/datasette/issues/1737,1115462720,IC_kwDOBm6k_c5CfJxA,9599,simonw,2022-05-02T23:25:03Z,2022-05-02T23:25:03Z,OWNER,"Here's a script that seems to work. It builds the wheel, starts a Python web server that serves the wheel, runs a test with `shot-scraper` and then shuts down the server again. ```bash #!/bin/bash # Build the wheel python3 -m build # Find name of wheel wheel=$(basename $(ls dist/*.whl)) # strip off the dist/ # Create a blank index page echo ' ' > dist/index.html # Run a server for that dist/ folder cd dist python3 -m http.server 8529 & cd .. shot-scraper javascript http://localhost:8529/ "" async () => { let pyodide = await loadPyodide(); await pyodide.loadPackage(['micropip', 'ssl', 'setuptools']); let output = await pyodide.runPythonAsync(\` import micropip await micropip.install('h11==0.12.0') await micropip.install('http://localhost:8529/$wheel') import ssl import setuptools from datasette.app import Datasette ds = Datasette(memory=True, settings={'num_sql_threads': 0}) (await ds.client.get('/_memory.json?sql=select+55+as+itworks&_shape=array')).text \`); if (JSON.parse(output)[0].itworks != 55) { throw 'Got ' + output + ', expected itworks: 55'; } return 'Test passed!'; } "" # Shut down the server pkill -f 'http.server 8529' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223459734,Automated test for Pyodide compatibility, https://github.com/simonw/datasette/issues/1733#issuecomment-1115404729,https://api.github.com/repos/simonw/datasette/issues/1733,1115404729,IC_kwDOBm6k_c5Ce7m5,9599,simonw,2022-05-02T21:49:01Z,2022-05-02T21:49:38Z,OWNER,"That alpha release works! https://pyodide.org/en/stable/console.html ```pycon Welcome to the Pyodide terminal emulator 🐍 Python 3.10.2 (main, Apr 9 2022 20:52:01) on WebAssembly VM Type ""help"", ""copyright"", ""credits"" or ""license"" for more information. >>> import micropip >>> await micropip.install(""datasette==0.62a0"") >>> import ssl >>> import setuptools >>> from datasette.app import Datasette >>> ds = Datasette(memory=True, settings={""num_sql_threads"": 0}) >>> await ds.client.get(""/.json"") >>> (await ds.client.get(""/.json"")).json() {'_memory': {'name': '_memory', 'hash': None, 'color': 'a6c7b9', 'path': '/_memory', 'tables_and_views_truncated': [], 'tab les_and_views_more': False, 'tables_count': 0, 'table_rows_sum': 0, 'show_table_row_counts': False, 'hidden_table_rows_sum' : 0, 'hidden_tables_count': 0, 'views_count': 0, 'private': False}} >>> ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223234932,Get Datasette compatible with Pyodide, https://github.com/simonw/datasette/issues/1733#issuecomment-1115318417,https://api.github.com/repos/simonw/datasette/issues/1733,1115318417,IC_kwDOBm6k_c5CemiR,9599,simonw,2022-05-02T20:13:43Z,2022-05-02T20:13:43Z,OWNER,This is good enough to push an alpha.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223234932,Get Datasette compatible with Pyodide, https://github.com/simonw/datasette/issues/1733#issuecomment-1115318303,https://api.github.com/repos/simonw/datasette/issues/1733,1115318303,IC_kwDOBm6k_c5Cemgf,9599,simonw,2022-05-02T20:13:36Z,2022-05-02T20:13:36Z,OWNER,"I got a build from the `pyodide` branch to work! ``` Welcome to the Pyodide terminal emulator 🐍 Python 3.10.2 (main, Apr 9 2022 20:52:01) on WebAssembly VM Type ""help"", ""copyright"", ""credits"" or ""license"" for more information. >>> import micropip >>> await micropip.install(""https://s3.amazonaws.com/simonwillison-cors-allowed-public/datasette-0.62a0-py3-none-any.whl"") Traceback (most recent call last): File """", line 1, in File ""/lib/python3.10/asyncio/futures.py"", line 284, in __await__ yield self # This tells Task to wait for completion. File ""/lib/python3.10/asyncio/tasks.py"", line 304, in __wakeup future.result() File ""/lib/python3.10/asyncio/futures.py"", line 201, in result raise self._exception File ""/lib/python3.10/asyncio/tasks.py"", line 234, in __step result = coro.throw(exc) File ""/lib/python3.10/site-packages/micropip/_micropip.py"", line 183, in install transaction = await self.gather_requirements(requirements, ctx, keep_going) File ""/lib/python3.10/site-packages/micropip/_micropip.py"", line 173, in gather_requirements await gather(*requirement_promises) File ""/lib/python3.10/asyncio/futures.py"", line 284, in __await__ yield self # This tells Task to wait for completion. File ""/lib/python3.10/asyncio/tasks.py"", line 304, in __wakeup future.result() File ""/lib/python3.10/asyncio/futures.py"", line 201, in result raise self._exception File ""/lib/python3.10/asyncio/tasks.py"", line 232, in __step result = coro.send(None) File ""/lib/python3.10/site-packages/micropip/_micropip.py"", line 245, in add_requirement await self.add_wheel(name, wheel, version, (), ctx, transaction) File ""/lib/python3.10/site-packages/micropip/_micropip.py"", line 316, in add_wheel await self.add_requirement(recurs_req, ctx, transaction) File ""/lib/python3.10/site-packages/micropip/_micropip.py"", line 291, in add_requirement await self.add_wheel( File ""/lib/python3.10/site-packages/micropip/_micropip.py"", line 316, in add_wheel await self.add_requirement(recurs_req, ctx, transaction) File ""/lib/python3.10/site-packages/micropip/_micropip.py"", line 291, in add_requirement await self.add_wheel( File ""/lib/python3.10/site-packages/micropip/_micropip.py"", line 316, in add_wheel await self.add_requirement(recurs_req, ctx, transaction) File ""/lib/python3.10/site-packages/micropip/_micropip.py"", line 276, in add_requirement raise ValueError( ValueError: Requested 'h11<0.13,>=0.11', but h11==0.13.0 is already installed >>> await micropip.install(""https://s3.amazonaws.com/simonwillison-cors-allowed-public/datasette-0.62a0-py3-none-any.whl"") Traceback (most recent call last): File """", line 1, in File ""/lib/python3.10/asyncio/futures.py"", line 284, in __await__ yield self # This tells Task to wait for completion. File ""/lib/python3.10/asyncio/tasks.py"", line 304, in __wakeup future.result() File ""/lib/python3.10/asyncio/futures.py"", line 201, in result raise self._exception File ""/lib/python3.10/asyncio/tasks.py"", line 234, in __step result = coro.throw(exc) File ""/lib/python3.10/site-packages/micropip/_micropip.py"", line 183, in install transaction = await self.gather_requirements(requirements, ctx, keep_going) File ""/lib/python3.10/site-packages/micropip/_micropip.py"", line 173, in gather_requirements await gather(*requirement_promises) File ""/lib/python3.10/asyncio/futures.py"", line 284, in __await__ yield self # This tells Task to wait for completion. File ""/lib/python3.10/asyncio/tasks.py"", line 304, in __wakeup future.result() File ""/lib/python3.10/asyncio/futures.py"", line 201, in result raise self._exception File ""/lib/python3.10/asyncio/tasks.py"", line 232, in __step result = coro.send(None) File ""/lib/python3.10/site-packages/micropip/_micropip.py"", line 245, in add_requirement await self.add_wheel(name, wheel, version, (), ctx, transaction) File ""/lib/python3.10/site-packages/micropip/_micropip.py"", line 316, in add_wheel await self.add_requirement(recurs_req, ctx, transaction) File ""/lib/python3.10/site-packages/micropip/_micropip.py"", line 291, in add_requirement await self.add_wheel( File ""/lib/python3.10/site-packages/micropip/_micropip.py"", line 316, in add_wheel await self.add_requirement(recurs_req, ctx, transaction) File ""/lib/python3.10/site-packages/micropip/_micropip.py"", line 291, in add_requirement await self.add_wheel( File ""/lib/python3.10/site-packages/micropip/_micropip.py"", line 316, in add_wheel await self.add_requirement(recurs_req, ctx, transaction) File ""/lib/python3.10/site-packages/micropip/_micropip.py"", line 276, in add_requirement raise ValueError( ValueError: Requested 'h11<0.13,>=0.11', but h11==0.13.0 is already installed >>> await micropip.install(""h11==0.12"") >>> await micropip.install(""https://s3.amazonaws.com/simonwillison-cors-allowed-public/datasette-0.62a0-py3-none-any.whl"") >>> import datasette >>> from datasette.app import Datasette Traceback (most recent call last): File """", line 1, in File ""/lib/python3.10/site-packages/datasette/app.py"", line 9, in import httpx File ""/lib/python3.10/site-packages/httpx/__init__.py"", line 2, in from ._api import delete, get, head, options, patch, post, put, request, stream File ""/lib/python3.10/site-packages/httpx/_api.py"", line 4, in from ._client import Client File ""/lib/python3.10/site-packages/httpx/_client.py"", line 9, in from ._auth import Auth, BasicAuth, FunctionAuth File ""/lib/python3.10/site-packages/httpx/_auth.py"", line 10, in from ._models import Request, Response File ""/lib/python3.10/site-packages/httpx/_models.py"", line 16, in from ._content import ByteStream, UnattachedStream, encode_request, encode_response File ""/lib/python3.10/site-packages/httpx/_content.py"", line 17, in from ._multipart import MultipartStream File ""/lib/python3.10/site-packages/httpx/_multipart.py"", line 7, in from ._types import ( File ""/lib/python3.10/site-packages/httpx/_types.py"", line 5, in import ssl File ""/lib/python3.10/ssl.py"", line 98, in import _ssl # if we can't import it, let the error propagate ModuleNotFoundError: No module named '_ssl' >>> import ssl >>> from datasette.app import Datasette Traceback (most recent call last): File """", line 1, in File ""/lib/python3.10/site-packages/datasette/app.py"", line 14, in import pkg_resources ModuleNotFoundError: No module named 'pkg_resources' >>> import setuptools >>> from datasette.app import Datasette >>> ds = Datasette(memory=True) >>> ds >>> await ds.client.get(""/"") Traceback (most recent call last): File ""/lib/python3.10/site-packages/datasette/app.py"", line 1268, in route_path response = await view(request, send) File ""/lib/python3.10/site-packages/datasette/views/base.py"", line 134, in view return await self.dispatch_request(request) File ""/lib/python3.10/site-packages/datasette/views/base.py"", line 89, in dispatch_request await self.ds.refresh_schemas() File ""/lib/python3.10/site-packages/datasette/app.py"", line 353, in refresh_schemas await self._refresh_schemas() File ""/lib/python3.10/site-packages/datasette/app.py"", line 358, in _refresh_schemas await init_internal_db(internal_db) File ""/lib/python3.10/site-packages/datasette/utils/internal_db.py"", line 65, in init_internal_db await db.execute_write_script(create_tables_sql) File ""/lib/python3.10/site-packages/datasette/database.py"", line 116, in execute_write_script results = await self.execute_write_fn(_inner, block=block) File ""/lib/python3.10/site-packages/datasette/database.py"", line 155, in execute_write_fn self._write_thread.start() File ""/lib/python3.10/threading.py"", line 928, in start _start_new_thread(self._bootstrap, ()) RuntimeError: can't start new thread >>> ds = Datasette(memory=True, settings={""num_sql_threads"": 0}) >>> await ds.client.get(""/"") >>> (await ds.client.get(""/"")).text '\n\n\n Datasette: _memory\n \n \n\n\n\n
\n
\n\n\n\n \n\n\n\n
\n\n

Datasette

\n\n\n\n\n\n

r detailsClickedWithin = null;\n while (target && target.tagName != \'DETAILS\') {\n target = target.parentNode;\ n }\n if (target && target.tagName == \'DETAILS\') {\n detailsClickedWithin = target;\n }\n Array.from(d ocument.getElementsByTagName(\'details\')).filter(\n (details) => details.open && details != detailsClickedWithin\n ).forEach(details => details.open = false);\n});\n\n\n\n\n\n\n ' >>> ``` That `ValueError: Requested 'h11<0.13,>=0.11', but h11==0.13.0 is already installed` error is annoying. I assume it's a `uvicorn` dependency clash of some sort, because I wasn't getting that when I removed `uvicorn` as a dependency. I can avoid it by running this first though: await micropip.install(""h11==0.12"")","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223234932,Get Datasette compatible with Pyodide, https://github.com/simonw/datasette/issues/1735#issuecomment-1115301733,https://api.github.com/repos/simonw/datasette/issues/1735,1115301733,IC_kwDOBm6k_c5Ceidl,9599,simonw,2022-05-02T19:57:19Z,2022-05-02T19:59:03Z,OWNER,"This code breaks if that setting is 0: https://github.com/simonw/datasette/blob/a29c1277896b6a7905ef5441c42a37bc15f67599/datasette/app.py#L291-L293 It's used here: https://github.com/simonw/datasette/blob/a29c1277896b6a7905ef5441c42a37bc15f67599/datasette/database.py#L188-L190","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223263540,Datasette setting to disable threading (for Pyodide), https://github.com/simonw/datasette/issues/1733#issuecomment-1115288284,https://api.github.com/repos/simonw/datasette/issues/1733,1115288284,IC_kwDOBm6k_c5CefLc,9599,simonw,2022-05-02T19:40:33Z,2022-05-02T19:40:33Z,OWNER,"I'll release this as a `0.62a0` as soon as it's ready, so I can start testing it out in Pyodide for real.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223234932,Get Datasette compatible with Pyodide, https://github.com/simonw/datasette/issues/1734#issuecomment-1115283922,https://api.github.com/repos/simonw/datasette/issues/1734,1115283922,IC_kwDOBm6k_c5CeeHS,9599,simonw,2022-05-02T19:35:32Z,2022-05-02T19:35:32Z,OWNER,I'll use my original from 2009: https://www.djangosnippets.org/snippets/1431/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223241647,Remove python-baseconv dependency, https://github.com/simonw/datasette/issues/1734#issuecomment-1115282773,https://api.github.com/repos/simonw/datasette/issues/1734,1115282773,IC_kwDOBm6k_c5Ced1V,9599,simonw,2022-05-02T19:34:15Z,2022-05-02T19:34:15Z,OWNER,I'm going to vendor it and update the documentation.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223241647,Remove python-baseconv dependency, https://github.com/simonw/datasette/issues/1733#issuecomment-1115278325,https://api.github.com/repos/simonw/datasette/issues/1733,1115278325,IC_kwDOBm6k_c5Cecv1,9599,simonw,2022-05-02T19:29:05Z,2022-05-02T19:29:05Z,OWNER,"I'm going to add a Datasette setting to disable threading entirely, designed for usage in this particular case. I thought about adding a new setting, then I noticed this: datasette mydatabase.db --setting num_sql_threads 10 I'm going to let users set that to `0` to disable threaded execution of SQL queries.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223234932,Get Datasette compatible with Pyodide, https://github.com/simonw/datasette/issues/1733#issuecomment-1115268245,https://api.github.com/repos/simonw/datasette/issues/1733,1115268245,IC_kwDOBm6k_c5CeaSV,9599,simonw,2022-05-02T19:18:11Z,2022-05-02T19:18:11Z,OWNER,"Maybe I can leave `uvicorn` as a dependency? Installing it works OK, it only generates errors when you try to import it: ```pycon Welcome to the Pyodide terminal emulator 🐍 Python 3.10.2 (main, Apr 9 2022 20:52:01) on WebAssembly VM Type ""help"", ""copyright"", ""credits"" or ""license"" for more information. >>> import micropip >>> await micropip.install(""uvicorn"") >>> import uvicorn Traceback (most recent call last): File """", line 1, in File ""/lib/python3.10/site-packages/uvicorn/__init__.py"", line 1, in from uvicorn.config import Config File ""/lib/python3.10/site-packages/uvicorn/config.py"", line 8, in import ssl File ""/lib/python3.10/ssl.py"", line 98, in import _ssl # if we can't import it, let the error propagate ModuleNotFoundError: No module named '_ssl' >>> import ssl >>> import uvicorn Traceback (most recent call last): File """", line 1, in File ""/lib/python3.10/site-packages/uvicorn/__init__.py"", line 2, in from uvicorn.main import Server, main, run File ""/lib/python3.10/site-packages/uvicorn/main.py"", line 24, in from uvicorn.supervisors import ChangeReload, Multiprocess File ""/lib/python3.10/site-packages/uvicorn/supervisors/__init__.py"", line 3, in from uvicorn.supervisors.basereload import BaseReload File ""/lib/python3.10/site-packages/uvicorn/supervisors/basereload.py"", line 12, in from uvicorn.subprocess import get_subprocess File ""/lib/python3.10/site-packages/uvicorn/subprocess.py"", line 14, in multiprocessing.allow_connection_pickling() File ""/lib/python3.10/multiprocessing/context.py"", line 170, in allow_connection_pickling from . import connection File ""/lib/python3.10/multiprocessing/connection.py"", line 21, in import _multiprocessing ModuleNotFoundError: No module named '_multiprocessing' >>> import multiprocessing >>> import uvicorn Traceback (most recent call last): File """", line 1, in File ""/lib/python3.10/site-packages/uvicorn/__init__.py"", line 2, in from uvicorn.main import Server, main, run File ""/lib/python3.10/site-packages/uvicorn/main.py"", line 24, in from uvicorn.supervisors import ChangeReload, Multiprocess File ""/lib/python3.10/site-packages/uvicorn/supervisors/__init__.py"", line 3, in from uvicorn.supervisors.basereload import BaseReload File ""/lib/python3.10/site-packages/uvicorn/supervisors/basereload.py"", line 12, in from uvicorn.subprocess import get_subprocess File ""/lib/python3.10/site-packages/uvicorn/subprocess.py"", line 14, in multiprocessing.allow_connection_pickling() File ""/lib/python3.10/multiprocessing/context.py"", line 170, in allow_connection_pickling from . import connection File ""/lib/python3.10/multiprocessing/connection.py"", line 21, in import _multiprocessing ModuleNotFoundError: No module named '_multiprocessing' >>> ``` Since the `import ssl` trick fixed the `_ssl` error I was hopeful that `import multiprocessing` could fix the `_multiprocessing` one, but sadly it did not. But it looks like i can address this issue just by making `import uvicorn` in `app.py` an optional import.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223234932,Get Datasette compatible with Pyodide, https://github.com/simonw/datasette/issues/1733#issuecomment-1115262218,https://api.github.com/repos/simonw/datasette/issues/1733,1115262218,IC_kwDOBm6k_c5CeY0K,9599,simonw,2022-05-02T19:11:51Z,2022-05-02T19:14:01Z,OWNER,"Here's the full diff I applied to Datasette to get it fully working in Pyodide: https://github.com/simonw/datasette/compare/94a3171b01fde5c52697aeeff052e3ad4bab5391...8af32bc5b03c30b1f7a4a8cc4bd80eb7e2ee7b81 And as a visible diff: ```diff diff --git a/datasette/app.py b/datasette/app.py index d269372..6c0c5fc 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -15,7 +15,6 @@ import pkg_resources import re import secrets import sys -import threading import traceback import urllib.parse from concurrent import futures @@ -26,7 +25,6 @@ from itsdangerous import URLSafeSerializer from jinja2 import ChoiceLoader, Environment, FileSystemLoader, PrefixLoader from jinja2.environment import Template from jinja2.exceptions import TemplateNotFound -import uvicorn from .views.base import DatasetteError, ureg from .views.database import DatabaseDownload, DatabaseView @@ -813,7 +811,6 @@ class Datasette: }, ""datasette"": datasette_version, ""asgi"": ""3.0"", - ""uvicorn"": uvicorn.__version__, ""sqlite"": { ""version"": sqlite_version, ""fts_versions"": fts_versions, @@ -854,23 +851,7 @@ class Datasette: ] def _threads(self): - threads = list(threading.enumerate()) - d = { - ""num_threads"": len(threads), - ""threads"": [ - {""name"": t.name, ""ident"": t.ident, ""daemon"": t.daemon} for t in threads - ], - } - # Only available in Python 3.7+ - if hasattr(asyncio, ""all_tasks""): - tasks = asyncio.all_tasks() - d.update( - { - ""num_tasks"": len(tasks), - ""tasks"": [_cleaner_task_str(t) for t in tasks], - } - ) - return d + return {""num_threads"": 0, ""threads"": []} def _actor(self, request): return {""actor"": request.actor} diff --git a/datasette/database.py b/datasette/database.py index ba594a8..b50142d 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -4,7 +4,6 @@ from pathlib import Path import janus import queue import sys -import threading import uuid from .tracer import trace @@ -21,8 +20,6 @@ from .utils import ( ) from .inspect import inspect_hash -connections = threading.local() - AttachedDatabase = namedtuple(""AttachedDatabase"", (""seq"", ""name"", ""file"")) @@ -43,12 +40,12 @@ class Database: self.hash = None self.cached_size = None self._cached_table_counts = None - self._write_thread = None - self._write_queue = None if not self.is_mutable and not self.is_memory: p = Path(path) self.hash = inspect_hash(p) self.cached_size = p.stat().st_size + self._read_connection = None + self._write_connection = None @property def cached_table_counts(self): @@ -134,60 +131,17 @@ class Database: return results async def execute_write_fn(self, fn, block=True): - task_id = uuid.uuid5(uuid.NAMESPACE_DNS, ""datasette.io"") - if self._write_queue is None: - self._write_queue = queue.Queue() - if self._write_thread is None: - self._write_thread = threading.Thread( - target=self._execute_writes, daemon=True - ) - self._write_thread.start() - reply_queue = janus.Queue() - self._write_queue.put(WriteTask(fn, task_id, reply_queue)) - if block: - result = await reply_queue.async_q.get() - if isinstance(result, Exception): - raise result - else: - return result - else: - return task_id - - def _execute_writes(self): - # Infinite looping thread that protects the single write connection - # to this database - conn_exception = None - conn = None - try: - conn = self.connect(write=True) - self.ds._prepare_connection(conn, self.name) - except Exception as e: - conn_exception = e - while True: - task = self._write_queue.get() - if conn_exception is not None: - result = conn_exception - else: - try: - result = task.fn(conn) - except Exception as e: - sys.stderr.write(""{}\n"".format(e)) - sys.stderr.flush() - result = e - task.reply_queue.sync_q.put(result) + # We always treat it as if block=True now + if self._write_connection is None: + self._write_connection = self.connect(write=True) + self.ds._prepare_connection(self._write_connection, self.name) + return fn(self._write_connection) async def execute_fn(self, fn): - def in_thread(): - conn = getattr(connections, self.name, None) - if not conn: - conn = self.connect() - self.ds._prepare_connection(conn, self.name) - setattr(connections, self.name, conn) - return fn(conn) - - return await asyncio.get_event_loop().run_in_executor( - self.ds.executor, in_thread - ) + if self._read_connection is None: + self._read_connection = self.connect() + self.ds._prepare_connection(self._read_connection, self.name) + return fn(self._read_connection) async def execute( self, diff --git a/setup.py b/setup.py index 7f0562f..c41669c 100644 --- a/setup.py +++ b/setup.py @@ -44,20 +44,20 @@ setup( install_requires=[ ""asgiref>=3.2.10,<3.6.0"", ""click>=7.1.1,<8.2.0"", - ""click-default-group~=1.2.2"", + # ""click-default-group~=1.2.2"", ""Jinja2>=2.10.3,<3.1.0"", ""hupper~=1.9"", ""httpx>=0.20"", ""pint~=0.9"", ""pluggy>=1.0,<1.1"", - ""uvicorn~=0.11"", + # ""uvicorn~=0.11"", ""aiofiles>=0.4,<0.9"", ""janus>=0.6.2,<1.1"", ""asgi-csrf>=0.9"", ""PyYAML>=5.3,<7.0"", ""mergedeep>=1.1.1,<1.4.0"", ""itsdangerous>=1.1,<3.0"", - ""python-baseconv==1.2.2"", + # ""python-baseconv==1.2.2"", ], entry_points="""""" [console_scripts] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223234932,Get Datasette compatible with Pyodide, https://github.com/simonw/datasette/issues/1734#issuecomment-1115260999,https://api.github.com/repos/simonw/datasette/issues/1734,1115260999,IC_kwDOBm6k_c5CeYhH,9599,simonw,2022-05-02T19:10:34Z,2022-05-02T19:10:34Z,OWNER,"This is actually mostly a documentation thing: here: https://docs.datasette.io/en/0.61.1/authentication.html#including-an-expiry-time In the code it's only used in these two places: https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/actor_auth_cookie.py#L16-L20 https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/tests/test_auth.py#L56-L60","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223241647,Remove python-baseconv dependency, https://github.com/simonw/datasette/issues/1733#issuecomment-1115258737,https://api.github.com/repos/simonw/datasette/issues/1733,1115258737,IC_kwDOBm6k_c5CeX9x,9599,simonw,2022-05-02T19:08:17Z,2022-05-02T19:08:17Z,OWNER,"I was going to vendor `baseconv.py`, but then I reconsidered - what if there are plugins out there that expect `import baseconv` to work because they have dependend on Datasette? I used https://cs.github.com/ and as far as I can tell there aren't any! So I'm going to remove that dependency and work out a smarter way to do this - probably by providing a utility function within Datasette itself.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223234932,Get Datasette compatible with Pyodide, https://github.com/simonw/datasette/issues/1733#issuecomment-1115256318,https://api.github.com/repos/simonw/datasette/issues/1733,1115256318,IC_kwDOBm6k_c5CeXX-,9599,simonw,2022-05-02T19:05:55Z,2022-05-02T19:05:55Z,OWNER,"I released a `click-default-group-wheel` package to solve that dependency issue. I've already upgraded `sqlite-utils` to that, so now you can use that in Pyodide: - https://github.com/simonw/sqlite-utils/pull/429 `python-baseconv` is only used for actor cookie expiration times: https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/actor_auth_cookie.py#L16-L20 Datasette never actually sets that cookie itself - it instead encourages plugins to set it in the authentication documentation here: https://docs.datasette.io/en/0.61.1/authentication.html#including-an-expiry-time","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223234932,Get Datasette compatible with Pyodide, https://github.com/simonw/sqlite-utils/pull/429#issuecomment-1115196863,https://api.github.com/repos/simonw/sqlite-utils/issues/429,1115196863,IC_kwDOCGYnMM5CeI2_,9599,simonw,2022-05-02T18:03:47Z,2022-05-02T18:52:42Z,OWNER,"I made a build of this branch and tested it like this: https://pyodide.org/en/stable/console.html ```pycon >>> import micropip >>> await micropip.install(""https://s3.amazonaws.com/simonwillison-cors-allowed-public/sqlite_utils-3.26-py3-none-any.whl"") >>> import sqlite_utils >>> db = sqlite_utils.Database(memory=True) >>> list(db.query(""select 32443 + 55"")) [{'32443 + 55': 32498}] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223177069,Depend on click-default-group-wheel, https://github.com/simonw/sqlite-utils/pull/429#issuecomment-1115197644,https://api.github.com/repos/simonw/sqlite-utils/issues/429,1115197644,IC_kwDOCGYnMM5CeJDM,9599,simonw,2022-05-02T18:04:28Z,2022-05-02T18:04:28Z,OWNER,I'm going to ship this straight away as `3.26.1`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223177069,Depend on click-default-group-wheel, https://github.com/simonw/datasette/issues/1727#issuecomment-1114058210,https://api.github.com/repos/simonw/datasette/issues/1727,1114058210,IC_kwDOBm6k_c5CZy3i,9599,simonw,2022-04-30T21:39:34Z,2022-04-30T21:39:34Z,OWNER,"Something to consider if I look into subprocesses for parallel query execution: https://sqlite.org/howtocorrupt.html#_carrying_an_open_database_connection_across_a_fork_ > Do not open an SQLite database connection, then fork(), then try to use that database connection in the child process. All kinds of locking problems will result and you can easily end up with a corrupt database. SQLite is not designed to support that kind of behavior. Any database connection that is used in a child process must be opened in the child process, not inherited from the parent. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile, https://github.com/simonw/datasette/issues/1729#issuecomment-1114038259,https://api.github.com/repos/simonw/datasette/issues/1729,1114038259,IC_kwDOBm6k_c5CZt_z,9599,simonw,2022-04-30T19:06:03Z,2022-04-30T19:06:03Z,OWNER,"> but actually the facet results would be better if they were a list rather than a dictionary I think `facet_results` in the JSON should match this (used by the HTML) instead: https://github.com/simonw/datasette/blob/942411ef946e9a34a2094944d3423cddad27efd3/datasette/views/table.py#L737-L741 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1219385669,Implement ?_extra and new API design for TableView, https://github.com/simonw/datasette/issues/1729#issuecomment-1114036946,https://api.github.com/repos/simonw/datasette/issues/1729,1114036946,IC_kwDOBm6k_c5CZtrS,9599,simonw,2022-04-30T18:56:25Z,2022-04-30T19:04:03Z,OWNER,"Related: - #1558 Which talks about how there was confusion in this example: https://latest.datasette.io/fixtures/facetable.json?_facet=created&_facet_date=created&_facet=tags&_facet_array=tags&_nosuggest=1&_size=0 Which I fixed in #625 by introducing `tags` and `tags_2` keys, but actually the facet results would be better if they were a list rather than a dictionary.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1219385669,Implement ?_extra and new API design for TableView, https://github.com/simonw/datasette/issues/1729#issuecomment-1114037521,https://api.github.com/repos/simonw/datasette/issues/1729,1114037521,IC_kwDOBm6k_c5CZt0R,9599,simonw,2022-04-30T19:01:07Z,2022-04-30T19:01:07Z,OWNER,"I had to look up what `hideable` means - it means that you can't hide the current facet because it was defined in metadata, not as a `?_facet=` parameter: https://github.com/simonw/datasette/blob/4e47a2d894b96854348343374c8e97c9d7055cf6/datasette/facets.py#L228 That's a bit of a weird thing to expose in the API. Maybe change that to `source` so it can be `metadata` or `request`? That's very slightly less coupled to how the UI works.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1219385669,Implement ?_extra and new API design for TableView, https://github.com/simonw/datasette/issues/1729#issuecomment-1114013757,https://api.github.com/repos/simonw/datasette/issues/1729,1114013757,IC_kwDOBm6k_c5CZoA9,9599,simonw,2022-04-30T16:15:51Z,2022-04-30T18:54:39Z,OWNER,"Deployed a preview of this here: https://latest-1-0-alpha.datasette.io/ Examples: - https://latest-1-0-alpha.datasette.io/fixtures/facetable.json - https://latest-1-0-alpha.datasette.io/fixtures/facetable.json?_facet=state&_size=0&_extra=facet_results&_extra=count Second example produces: ```json { ""rows"": [], ""next"": null, ""next_url"": null, ""count"": 15, ""facet_results"": { ""state"": { ""name"": ""state"", ""type"": ""column"", ""hideable"": true, ""toggle_url"": ""/fixtures/facetable.json?_size=0&_extra=facet_results&_extra=count"", ""results"": [ { ""value"": ""CA"", ""label"": ""CA"", ""count"": 10, ""toggle_url"": ""https://latest-1-0-alpha.datasette.io/fixtures/facetable.json?_facet=state&_size=0&_extra=facet_results&_extra=count&state=CA"", ""selected"": false }, { ""value"": ""MI"", ""label"": ""MI"", ""count"": 4, ""toggle_url"": ""https://latest-1-0-alpha.datasette.io/fixtures/facetable.json?_facet=state&_size=0&_extra=facet_results&_extra=count&state=MI"", ""selected"": false }, { ""value"": ""MC"", ""label"": ""MC"", ""count"": 1, ""toggle_url"": ""https://latest-1-0-alpha.datasette.io/fixtures/facetable.json?_facet=state&_size=0&_extra=facet_results&_extra=count&state=MC"", ""selected"": false } ], ""truncated"": false } } } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1219385669,Implement ?_extra and new API design for TableView, https://github.com/simonw/datasette/issues/1727#issuecomment-1112889800,https://api.github.com/repos/simonw/datasette/issues/1727,1112889800,IC_kwDOBm6k_c5CVVnI,9599,simonw,2022-04-29T05:29:38Z,2022-04-29T05:29:38Z,OWNER,"OK, I just got the most incredible result with that! I started up a container running `bash` like this, from my `datasette` checkout. I'm mapping port 8005 on my laptop to port 8001 inside the container because laptop port 8001 was already doing something else: ``` docker run -it --rm --name my-running-script -p 8005:8001 -v ""$PWD"":/usr/src/myapp \ -w /usr/src/myapp nogil/python bash ``` Then in `bash` I ran the following commands to install Datasette and its dependencies: ``` pip install -e '.[test]' pip install datasette-pretty-traces # For debug tracing ``` Then I started Datasette against my `github.db` database (from github-to-sqlite.dogsheep.net/github.db) like this: ``` datasette github.db -h 0.0.0.0 --setting trace_debug 1 ``` I hit the following two URLs to compare the parallel v.s. not parallel implementations: - `http://127.0.0.1:8005/github/issues?_facet=milestone&_facet=repo&_trace=1&_size=10` - `http://127.0.0.1:8005/github/issues?_facet=milestone&_facet=repo&_trace=1&_size=10&_noparallel=1` And... the parallel one beat the non-parallel one decisively, on multiple page refreshes! Not parallel: 77ms Parallel: 47ms So yeah, I'm very confident this is a problem with the GIL. And I am absolutely **stunned** that @colesbury's fork ran Datasette (which has some reasonably tricky threading and async stuff going on) out of the box!","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile, https://github.com/simonw/datasette/issues/1727#issuecomment-1112879463,https://api.github.com/repos/simonw/datasette/issues/1727,1112879463,IC_kwDOBm6k_c5CVTFn,9599,simonw,2022-04-29T05:03:58Z,2022-04-29T05:03:58Z,OWNER,"It would be _really_ fun to try running this with the in-development `nogil` Python from https://github.com/colesbury/nogil There's a Docker container for it: https://hub.docker.com/r/nogil/python It suggests you can run something like this: docker run -it --rm --name my-running-script -v ""$PWD"":/usr/src/myapp \ -w /usr/src/myapp nogil/python python your-daemon-or-script.py","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile, https://github.com/simonw/datasette/issues/1727#issuecomment-1112878955,https://api.github.com/repos/simonw/datasette/issues/1727,1112878955,IC_kwDOBm6k_c5CVS9r,9599,simonw,2022-04-29T05:02:40Z,2022-04-29T05:02:40Z,OWNER,"Here's a very useful (recent) article about how the GIL works and how to think about it: https://pythonspeed.com/articles/python-gil/ - via https://lobste.rs/s/9hj80j/when_python_can_t_thread_deep_dive_into_gil From that article: > For example, let's consider an extension module written in C or Rust that lets you talk to a PostgreSQL database server. > > Conceptually, handling a SQL query with this library will go through three steps: > > 1. Deserialize from Python to the internal library representation. Since this will be reading Python objects, it needs to hold the GIL. > 2. Send the query to the database server, and wait for a response. This doesn't need the GIL. > 3. Convert the response into Python objects. This needs the GIL again. > > As you can see, how much parallelism you can get depends on how much time is spent in each step. If the bulk of time is spent in step 2, you'll get parallelism there. But if, for example, you run a `SELECT` and get a large number of rows back, the library will need to create many Python objects, and step 3 will have to hold GIL for a while. That explains what I'm seeing here. I'm pretty convinced now that the reason I'm not getting a performance boost from parallel queries is that there's more time spent in Python code assembling the results than in SQLite C code executing the query.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile, https://github.com/simonw/datasette/issues/1729#issuecomment-1112734577,https://api.github.com/repos/simonw/datasette/issues/1729,1112734577,IC_kwDOBm6k_c5CUvtx,9599,simonw,2022-04-28T23:08:42Z,2022-04-28T23:08:42Z,OWNER,"That prototype is a very small amount of code so far: ```diff diff --git a/datasette/renderer.py b/datasette/renderer.py index 4508949..b600e1b 100644 --- a/datasette/renderer.py +++ b/datasette/renderer.py @@ -28,6 +28,10 @@ def convert_specific_columns_to_json(rows, columns, json_cols): def json_renderer(args, data, view_name): """"""Render a response as JSON"""""" + from pprint import pprint + + pprint(data) + status_code = 200 # Handle the _json= parameter which may modify data[""rows""] @@ -43,6 +47,41 @@ def json_renderer(args, data, view_name): if ""rows"" in data and not value_as_boolean(args.get(""_json_infinity"", ""0"")): data[""rows""] = [remove_infinites(row) for row in data[""rows""]] + # Start building the default JSON here + columns = data[""columns""] + next_url = data.get(""next_url"") + output = { + ""rows"": [dict(zip(columns, row)) for row in data[""rows""]], + ""next"": data[""next""], + ""next_url"": next_url, + } + + extras = set(args.getlist(""_extra"")) + + extras_map = { + # _extra= : data[field] + ""count"": ""filtered_table_rows_count"", + ""facet_results"": ""facet_results"", + ""suggested_facets"": ""suggested_facets"", + ""columns"": ""columns"", + ""primary_keys"": ""primary_keys"", + ""query_ms"": ""query_ms"", + ""query"": ""query"", + } + for extra_key, data_key in extras_map.items(): + if extra_key in extras: + output[extra_key] = data[data_key] + + body = json.dumps(output, cls=CustomJSONEncoder) + content_type = ""application/json; charset=utf-8"" + headers = {} + if next_url: + headers[""link""] = f'<{next_url}>; rel=""next""' + return Response( + body, status=status_code, headers=headers, content_type=content_type + ) + + # Deal with the _shape option shape = args.get(""_shape"", ""arrays"") # if there's an error, ignore the shape entirely ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1219385669,Implement ?_extra and new API design for TableView, https://github.com/simonw/datasette/issues/1729#issuecomment-1112732563,https://api.github.com/repos/simonw/datasette/issues/1729,1112732563,IC_kwDOBm6k_c5CUvOT,9599,simonw,2022-04-28T23:05:03Z,2022-04-28T23:05:03Z,OWNER,"OK, the prototype of this is looking really good - it's very pleasant to use. `http://127.0.0.1:8001/github_memory/issue_comments.json?_search=simon&_sort=id&_size=5&_extra=query_ms&_extra=count&_col=body` returns this: ```json { ""rows"": [ { ""id"": 338854988, ""body"": "" /database-name/table-name?name__contains=simon&sort=id+desc\r\n\r\nNote that if there's a column called \""sort\"" you can still do sort__exact=blah\r\n\r\n"" }, { ""id"": 346427794, ""body"": ""Thanks. There is a way to use pip to grab apsw, which also let's you configure it (flags to build extensions, use an internal sqlite, etc). Don't know how that works as a dependency for another package, though.\n\nOn November 22, 2017 11:38:06 AM EST, Simon Willison wrote:\n>I have a solution for FTS already, but I'm interested in apsw as a\n>mechanism for allowing custom virtual tables to be written in Python\n>(pysqlite only lets you write custom functions)\n>\n>Not having PyPI support is pretty tough though. I'm planning a\n>plugin/extension system which would be ideal for things like an\n>optional apsw mode, but that's a lot harder if apsw isn't in PyPI.\n>\n>-- \n>You are receiving this because you authored the thread.\n>Reply to this email directly or view it on GitHub:\n>https://github.com/simonw/datasette/issues/144#issuecomment-346405660\n"" }, { ""id"": 348252037, ""body"": ""WOW!\n\n--\nPaul Ford // (646) 369-7128 // @ftrain\n\nOn Thu, Nov 30, 2017 at 11:47 AM, Simon Willison \nwrote:\n\n> Remaining work on this now lives in a milestone:\n> https://github.com/simonw/datasette/milestone/6\n>\n> —\n> You are receiving this because you were mentioned.\n> Reply to this email directly, view it on GitHub\n> ,\n> or mute the thread\n> \n> .\n>\n"" }, { ""id"": 391141391, ""body"": ""I'm going to clean this up for consistency tomorrow morning so hold off\nmerging until then please\n\nOn Tue, May 22, 2018 at 6:34 PM, Simon Willison \nwrote:\n\n> Yeah let's try this without pysqlite3 and see if we still get the correct\n> version.\n>\n> —\n> You are receiving this because you authored the thread.\n> Reply to this email directly, view it on GitHub\n> , or mute\n> the thread\n> \n> .\n>\n"" }, { ""id"": 391355030, ""body"": ""No objections;\r\nIt's good to go @simonw\r\n\r\nOn Wed, 23 May 2018, 14:51 Simon Willison, wrote:\r\n\r\n> @r4vi any objections to me merging this?\r\n>\r\n> —\r\n> You are receiving this because you were mentioned.\r\n> Reply to this email directly, view it on GitHub\r\n> , or mute\r\n> the thread\r\n> \r\n> .\r\n>\r\n"" } ], ""next"": ""391355030,391355030"", ""next_url"": ""http://127.0.0.1:8001/github_memory/issue_comments.json?_search=simon&_size=5&_extra=query_ms&_extra=count&_col=body&_next=391355030%2C391355030&_sort=id"", ""count"": 57, ""query_ms"": 21.780223003588617 } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1219385669,Implement ?_extra and new API design for TableView, https://github.com/simonw/datasette/issues/1729#issuecomment-1112730416,https://api.github.com/repos/simonw/datasette/issues/1729,1112730416,IC_kwDOBm6k_c5CUusw,9599,simonw,2022-04-28T23:01:21Z,2022-04-28T23:01:21Z,OWNER,"I'm not sure what to do about the `""truncated"": true/false` key. It's not really relevant to table results, since they are paginated whether or not you ask for them to be. It plays a role in query results, where you might run `select * from table` and get back 1000 results because Datasette truncates at that point rather than returning everything. Adding it to every table result and always setting it to `""truncated"": false` feels confusing. I think I'm going to keep it exclusively in the default representation for the `/db?sql=...` query endpoint, and not return it at all for tables.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1219385669,Implement ?_extra and new API design for TableView, https://github.com/simonw/datasette/issues/1729#issuecomment-1112721321,https://api.github.com/repos/simonw/datasette/issues/1729,1112721321,IC_kwDOBm6k_c5CUsep,9599,simonw,2022-04-28T22:44:05Z,2022-04-28T22:44:14Z,OWNER,I may be able to implement this mostly in the `json_renderer()` function: https://github.com/simonw/datasette/blob/94a3171b01fde5c52697aeeff052e3ad4bab5391/datasette/renderer.py#L29-L34,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1219385669,Implement ?_extra and new API design for TableView, https://github.com/simonw/datasette/issues/1729#issuecomment-1112717745,https://api.github.com/repos/simonw/datasette/issues/1729,1112717745,IC_kwDOBm6k_c5CUrmx,9599,simonw,2022-04-28T22:38:39Z,2022-04-28T22:39:05Z,OWNER,"(I remain keen on the idea of shipping a plugin that restores the old default API shape to people who have written pre-Datasette-1.0 code against it, but I'll tackle that much later. I really like how jQuery has a culture of doing this.)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1219385669,Implement ?_extra and new API design for TableView, https://github.com/simonw/datasette/issues/1729#issuecomment-1112717210,https://api.github.com/repos/simonw/datasette/issues/1729,1112717210,IC_kwDOBm6k_c5CUrea,9599,simonw,2022-04-28T22:37:37Z,2022-04-28T22:37:37Z,OWNER,"This means `filtered_table_rows_count` is going to become `count`. I had originally picked that terrible name to avoid confusion between the count of all rows in the table and the count of rows that were filtered. I'll add `?_extra=table_count` for getting back the full table count instead. I think `count` is clear enough!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1219385669,Implement ?_extra and new API design for TableView, https://github.com/simonw/datasette/issues/1729#issuecomment-1112716611,https://api.github.com/repos/simonw/datasette/issues/1729,1112716611,IC_kwDOBm6k_c5CUrVD,9599,simonw,2022-04-28T22:36:24Z,2022-04-28T22:36:24Z,OWNER,"Then I'm going to implement the following `?_extra=` options: - `?_extra=facet_results` - to see facet results - `?_extra=suggested_facets` - for suggested facets - `?_extra=count` - for the count of total rows - `?_extra=columns` - for a list of column names - `?_extra=primary_keys` - for a list of primary keys - `?_extra=query` - a `{""sql"" ""select ..."", ""params"": {}}` object I thought about having `?_extra=facet_results` returned automatically if the user specifies at least one `?_facet` - but that doesn't work for default facets configured in `metadata.json` - how can the user opt out of those being returned? So I'm going to say you don't see facets at all if you don't include `?_extra=facet_results`. I'm tempted to add `?_extra=_all` to return everything, but I can decide if that's a good idea later.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1219385669,Implement ?_extra and new API design for TableView, https://github.com/simonw/datasette/issues/1729#issuecomment-1112713581,https://api.github.com/repos/simonw/datasette/issues/1729,1112713581,IC_kwDOBm6k_c5CUqlt,9599,simonw,2022-04-28T22:31:11Z,2022-04-28T22:31:11Z,OWNER,"I'm going to change the default API response to look like this: ```json { ""rows"": [ { ""pk"": 1, ""created"": ""2019-01-14 08:00:00"", ""planet_int"": 1, ""on_earth"": 1, ""state"": ""CA"", ""_city_id"": 1, ""_neighborhood"": ""Mission"", ""tags"": ""[\""tag1\"", \""tag2\""]"", ""complex_array"": ""[{\""foo\"": \""bar\""}]"", ""distinct_some_null"": ""one"", ""n"": ""n1"" }, { ""pk"": 2, ""created"": ""2019-01-14 08:00:00"", ""planet_int"": 1, ""on_earth"": 1, ""state"": ""CA"", ""_city_id"": 1, ""_neighborhood"": ""Dogpatch"", ""tags"": ""[\""tag1\"", \""tag3\""]"", ""complex_array"": ""[]"", ""distinct_some_null"": ""two"", ""n"": ""n2"" } ], ""next"": null, ""next_url"": null } ``` Basically https://latest.datasette.io/fixtures/facetable.json?_shape=objects but with just the `rows`, `next` and `next_url` fields returned by default.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1219385669,Implement ?_extra and new API design for TableView, https://github.com/simonw/datasette/issues/1715#issuecomment-1112711115,https://api.github.com/repos/simonw/datasette/issues/1715,1112711115,IC_kwDOBm6k_c5CUp_L,9599,simonw,2022-04-28T22:26:56Z,2022-04-28T22:26:56Z,OWNER,"I'm not going to use `asyncinject` in this refactor - at least not until I really need it. My research in these issues has put me off the idea ( in favour of `asyncio.gather()` or even not trying for parallel execution at all): - #1727","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1212823665,Refactor TableView to use asyncinject, https://github.com/simonw/datasette/issues/1727#issuecomment-1112668411,https://api.github.com/repos/simonw/datasette/issues/1727,1112668411,IC_kwDOBm6k_c5CUfj7,9599,simonw,2022-04-28T21:25:34Z,2022-04-28T21:25:44Z,OWNER,"The two most promising theories at the moment, from here and Twitter and the SQLite forum, are: - SQLite is I/O bound - it generally only goes as fast as it can load data from disk. Multiple connections all competing for the same file on disk are going to end up blocked at the file system layer. But maybe this means in-memory databases will perform better? - It's the GIL. The sqlite3 C code may release the GIL, but the bits that do things like assembling `Row` objects to return still happen in Python, and that Python can only run on a single core. A couple of ways to research the in-memory theory: - Use a RAM disk on macOS (or Linux). https://stackoverflow.com/a/2033417/6083 has instructions - short version: hdiutil attach -nomount ram://$((2 * 1024 * 100)) diskutil eraseVolume HFS+ RAMDisk name-returned-by-previous-command (was `/dev/disk2` when I tried it) cd /Volumes/RAMDisk cp ~/fixtures.db . - Copy Datasette databases into an in-memory database on startup. I built a new plugin to do that here: https://github.com/simonw/datasette-copy-to-memory I need to do some more, better benchmarks using these different approaches. https://twitter.com/laurencerowe/status/1519780174560169987 also suggests: > Maybe try: > 1. Copy the sqlite file to /dev/shm and rerun (all in ram.) > 2. Create a CTE which calculates Fibonacci or similar so you can test something completely cpu bound (only return max value or something to avoid crossing between sqlite/Python.) I like that second idea a lot - I could use the mandelbrot example from https://www.sqlite.org/lang_with.html#outlandish_recursive_query_examples","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile, https://github.com/simonw/datasette/issues/1727#issuecomment-1111726586,https://api.github.com/repos/simonw/datasette/issues/1727,1111726586,IC_kwDOBm6k_c5CQ5n6,9599,simonw,2022-04-28T04:17:16Z,2022-04-28T04:19:31Z,OWNER,"I could experiment with the `await asyncio.run_in_executor(processpool_executor, fn)` mechanism described in https://stackoverflow.com/a/29147750 Code examples: https://cs.github.com/?scopeName=All+repos&scope=&q=run_in_executor+ProcessPoolExecutor","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile, https://github.com/simonw/datasette/issues/1727#issuecomment-1111725638,https://api.github.com/repos/simonw/datasette/issues/1727,1111725638,IC_kwDOBm6k_c5CQ5ZG,9599,simonw,2022-04-28T04:15:15Z,2022-04-28T04:15:15Z,OWNER,"Useful theory from Keith Medcalf https://sqlite.org/forum/forumpost/e363c69d3441172e > This is true, but the concurrency is limited to the execution which occurs with the GIL released (that is, in the native C sqlite3 library itself). Each row (for example) can be retrieved in parallel but ""constructing the python return objects for each row"" will be serialized (by the GIL). > > That is to say that if your have two python threads each with their own connection, and each one is performing a select that returns 1,000,000 rows (lets say that is 25% of the candidates for each select) then the difference in execution time between executing two python threads in parallel vs a single serial thead will not be much different (if even detectable at all). In fact it is possible that the multiple-threaded version takes longer to run both queries to completion because of the increased contention over a shared resource (the GIL). So maybe this is a GIL thing. I should test with some expensive SQL queries (maybe big aggregations against large tables) and see if I can spot an improvement there.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile, https://github.com/simonw/datasette/issues/1728#issuecomment-1111714665,https://api.github.com/repos/simonw/datasette/issues/1728,1111714665,IC_kwDOBm6k_c5CQ2tp,9599,simonw,2022-04-28T03:52:47Z,2022-04-28T03:52:58Z,OWNER,"Nice custom template/theme! Yeah, for that I'd recommend hosting elsewhere - on a regular VPS (I use `systemd` like this: https://docs.datasette.io/en/stable/deploying.html#running-datasette-using-systemd ) or using Fly if you want to tub containers without managing a full server.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1218133366,Writable canned queries fail with useless non-error against immutable databases, https://github.com/simonw/datasette/issues/1728#issuecomment-1111708206,https://api.github.com/repos/simonw/datasette/issues/1728,1111708206,IC_kwDOBm6k_c5CQ1Iu,9599,simonw,2022-04-28T03:38:56Z,2022-04-28T03:38:56Z,OWNER,"In terms of this bug, there are a few potential fixes: 1. Detect the write to a immutable database and show the user a proper, meaningful error message in the red error box at the top of the page 2. Don't allow the user to even submit the form - show a message saying that this canned query is unavailable because the database cannot be written to 3. Don't even allow Datasette to start running at all - if there's a canned query configured in `metadata.yml` and the database it refers to is in `-i` immutable mode throw an error on startup I'm not keen on that last one because it would be frustrating if you couldn't launch Datasette just because you had an old canned query lying around in your metadata file. So I'm leaning towards option 2.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1218133366,Writable canned queries fail with useless non-error against immutable databases, https://github.com/simonw/datasette/issues/1728#issuecomment-1111707384,https://api.github.com/repos/simonw/datasette/issues/1728,1111707384,IC_kwDOBm6k_c5CQ074,9599,simonw,2022-04-28T03:36:46Z,2022-04-28T03:36:56Z,OWNER,"A more realistic solution (which I've been using on several of my own projects) is to keep the data itself in GitHub and encourage users to edit it there - using the GitHub web interface to edit YAML files or similar. Needs your users to be comfortable hand-editing YAML though! You can at least guard against critical errors by having CI run tests against their YAML before deploying. I have a dream of building a more friendly web forms interface which edits the YAML back on GitHub for the user, but that's just a concept at the moment. Even more fun would be if a user-friendly form could submit PRs for review without the user having to know what a PR is!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1218133366,Writable canned queries fail with useless non-error against immutable databases, https://github.com/simonw/datasette/issues/1728#issuecomment-1111706519,https://api.github.com/repos/simonw/datasette/issues/1728,1111706519,IC_kwDOBm6k_c5CQ0uX,9599,simonw,2022-04-28T03:34:49Z,2022-04-28T03:34:49Z,OWNER,"I've wanted to do stuff like that on Cloud Run too. So far I've assumed that it's not feasible, but recently I've been wondering how hard it would be to have a small (like less than 100KB or so) Datasette instance which persists data to a backing GitHub repository such that when it starts up it can pull the latest copy and any time someone edits it can push their changes. I'm still not sure it would work well on Cloud Run due to the uncertainty at what would happen if Cloud Run decided to boot up a second instance - but it's still an interesting thought exercise.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1218133366,Writable canned queries fail with useless non-error against immutable databases, https://github.com/simonw/datasette/issues/1728#issuecomment-1111705069,https://api.github.com/repos/simonw/datasette/issues/1728,1111705069,IC_kwDOBm6k_c5CQ0Xt,9599,simonw,2022-04-28T03:31:33Z,2022-04-28T03:31:33Z,OWNER,"Confirmed - this is a bug where immutable databases fail to show a useful error if you write to them with a canned query. Steps to reproduce: ``` echo ' databases: writable: queries: add_name: sql: insert into names(name) values (:name) write: true ' > write-metadata.yml echo '{""name"": ""Simon""}' | sqlite-utils insert writable.db names - datasette writable.db -m write-metadata.yml ``` Then visit http://127.0.0.1:8001/writable/add_name - adding names works. Now do this instead: ``` datasette -i writable.db -m write-metadata.yml ``` And I'm getting a broken error: ![error](https://user-images.githubusercontent.com/9599/165670823-6604dd69-9905-475c-8098-5da22ab026a1.gif) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1218133366,Writable canned queries fail with useless non-error against immutable databases, https://github.com/simonw/datasette/issues/1727#issuecomment-1111699175,https://api.github.com/repos/simonw/datasette/issues/1727,1111699175,IC_kwDOBm6k_c5CQy7n,9599,simonw,2022-04-28T03:19:48Z,2022-04-28T03:20:08Z,OWNER,"I ran `py-spy` and then hammered refresh a bunch of times on the `http://127.0.0.1:8856/github/commits?_facet=repo&_facet=committer&_trace=1&_noparallel=` page - it generated this SVG profile for me. The area on the right is the threads running the DB queries: ![profile](https://user-images.githubusercontent.com/9599/165669677-5461ede5-3dc4-4b49-8319-bfe5fd8a723d.svg) Interactive version here: https://static.simonwillison.net/static/2022/datasette-parallel-profile.svg","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile, https://github.com/simonw/datasette/issues/1728#issuecomment-1111698307,https://api.github.com/repos/simonw/datasette/issues/1728,1111698307,IC_kwDOBm6k_c5CQyuD,9599,simonw,2022-04-28T03:18:02Z,2022-04-28T03:18:02Z,OWNER,If the behaviour you are seeing is because the database is running in immutable mode then that's a bug - you should get a useful error message instead!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1218133366,Writable canned queries fail with useless non-error against immutable databases, https://github.com/simonw/datasette/issues/1728#issuecomment-1111697985,https://api.github.com/repos/simonw/datasette/issues/1728,1111697985,IC_kwDOBm6k_c5CQypB,9599,simonw,2022-04-28T03:17:20Z,2022-04-28T03:17:20Z,OWNER,"How did you deploy to Cloud Run? `datasette publish cloudrun` defaults to running databases there in `-i` immutable mode, because if you managed to change a file on disk on Cloud Run those changes would be lost the next time your container restarted there. That's why I upgraded `datasette-publish-fly` to provide a way of working with their volumes support - they're the best option I know of right now for running Datasette in a container with a persistent volume that can accept writes: https://simonwillison.net/2022/Feb/15/fly-volumes/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1218133366,Writable canned queries fail with useless non-error against immutable databases, https://github.com/simonw/datasette/issues/1727#issuecomment-1111683539,https://api.github.com/repos/simonw/datasette/issues/1727,1111683539,IC_kwDOBm6k_c5CQvHT,9599,simonw,2022-04-28T02:47:57Z,2022-04-28T02:47:57Z,OWNER,"Maybe this is the Python GIL after all? I've been hoping that the GIL won't be an issue because the `sqlite3` module releases the GIL for the duration of the execution of a SQL query - see https://github.com/python/cpython/blob/f348154c8f8a9c254503306c59d6779d4d09b3a9/Modules/_sqlite/cursor.c#L749-L759 So I've been hoping this means that SQLite code itself can run concurrently on multiple cores even when Python threads cannot. But maybe I'm misunderstanding how that works?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile, https://github.com/simonw/datasette/issues/1727#issuecomment-1111681513,https://api.github.com/repos/simonw/datasette/issues/1727,1111681513,IC_kwDOBm6k_c5CQunp,9599,simonw,2022-04-28T02:44:26Z,2022-04-28T02:44:26Z,OWNER,"I could try `py-spy top`, which I previously used here: - https://github.com/simonw/datasette/issues/1673","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile, https://github.com/simonw/datasette/issues/1727#issuecomment-1111661331,https://api.github.com/repos/simonw/datasette/issues/1727,1111661331,IC_kwDOBm6k_c5CQpsT,9599,simonw,2022-04-28T02:07:31Z,2022-04-28T02:07:31Z,OWNER,Asked on the SQLite forum about this here: https://sqlite.org/forum/forumpost/ffbfa9f38e,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile, https://github.com/simonw/datasette/issues/1727#issuecomment-1111602802,https://api.github.com/repos/simonw/datasette/issues/1727,1111602802,IC_kwDOBm6k_c5CQbZy,9599,simonw,2022-04-28T00:21:35Z,2022-04-28T00:21:35Z,OWNER,"Tried this but I'm getting back an empty JSON array of traces at the bottom of the page most of the time (intermittently it works correctly): ```diff diff --git a/datasette/database.py b/datasette/database.py index ba594a8..d7f9172 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -7,7 +7,7 @@ import sys import threading import uuid -from .tracer import trace +from .tracer import trace, trace_child_tasks from .utils import ( detect_fts, detect_primary_keys, @@ -207,30 +207,31 @@ class Database: time_limit_ms = custom_time_limit with sqlite_timelimit(conn, time_limit_ms): - try: - cursor = conn.cursor() - cursor.execute(sql, params if params is not None else {}) - max_returned_rows = self.ds.max_returned_rows - if max_returned_rows == page_size: - max_returned_rows += 1 - if max_returned_rows and truncate: - rows = cursor.fetchmany(max_returned_rows + 1) - truncated = len(rows) > max_returned_rows - rows = rows[:max_returned_rows] - else: - rows = cursor.fetchall() - truncated = False - except (sqlite3.OperationalError, sqlite3.DatabaseError) as e: - if e.args == (""interrupted"",): - raise QueryInterrupted(e, sql, params) - if log_sql_errors: - sys.stderr.write( - ""ERROR: conn={}, sql = {}, params = {}: {}\n"".format( - conn, repr(sql), params, e + with trace(""sql"", database=self.name, sql=sql.strip(), params=params): + try: + cursor = conn.cursor() + cursor.execute(sql, params if params is not None else {}) + max_returned_rows = self.ds.max_returned_rows + if max_returned_rows == page_size: + max_returned_rows += 1 + if max_returned_rows and truncate: + rows = cursor.fetchmany(max_returned_rows + 1) + truncated = len(rows) > max_returned_rows + rows = rows[:max_returned_rows] + else: + rows = cursor.fetchall() + truncated = False + except (sqlite3.OperationalError, sqlite3.DatabaseError) as e: + if e.args == (""interrupted"",): + raise QueryInterrupted(e, sql, params) + if log_sql_errors: + sys.stderr.write( + ""ERROR: conn={}, sql = {}, params = {}: {}\n"".format( + conn, repr(sql), params, e + ) ) - ) - sys.stderr.flush() - raise + sys.stderr.flush() + raise if truncate: return Results(rows, truncated, cursor.description) @@ -238,9 +239,8 @@ class Database: else: return Results(rows, False, cursor.description) - with trace(""sql"", database=self.name, sql=sql.strip(), params=params): - results = await self.execute_fn(sql_operation_in_thread) - return results + with trace_child_tasks(): + return await self.execute_fn(sql_operation_in_thread) @property def size(self): ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile, https://github.com/simonw/datasette/issues/1727#issuecomment-1111597176,https://api.github.com/repos/simonw/datasette/issues/1727,1111597176,IC_kwDOBm6k_c5CQaB4,9599,simonw,2022-04-28T00:11:44Z,2022-04-28T00:11:44Z,OWNER,"Though it would be interesting to also have the trace reveal how much time is spent in the functions that wrap that core SQL - the stuff that is being measured at the moment. I have a hunch that this could help solve the over-arching performance mystery.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile, https://github.com/simonw/datasette/issues/1727#issuecomment-1111595319,https://api.github.com/repos/simonw/datasette/issues/1727,1111595319,IC_kwDOBm6k_c5CQZk3,9599,simonw,2022-04-28T00:09:45Z,2022-04-28T00:11:01Z,OWNER,"Here's where read queries are instrumented: https://github.com/simonw/datasette/blob/7a6654a253dee243518dc542ce4c06dbb0d0801d/datasette/database.py#L241-L242 So the instrumentation is actually capturing quite a bit of Python activity before it gets to SQLite: https://github.com/simonw/datasette/blob/7a6654a253dee243518dc542ce4c06dbb0d0801d/datasette/database.py#L179-L190 And then: https://github.com/simonw/datasette/blob/7a6654a253dee243518dc542ce4c06dbb0d0801d/datasette/database.py#L204-L233 Ideally I'd like that `trace()` block to wrap just the `cursor.execute()` and `cursor.fetchmany(...)` or `cursor.fetchall()` calls.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile, https://github.com/simonw/datasette/issues/1727#issuecomment-1111558204,https://api.github.com/repos/simonw/datasette/issues/1727,1111558204,IC_kwDOBm6k_c5CQQg8,9599,simonw,2022-04-27T22:58:39Z,2022-04-27T22:58:39Z,OWNER,"I should check my timing mechanism. Am I capturing the time taken just in SQLite or does it include time spent in Python crossing between async and threaded world and waiting for a thread pool worker to become available? That could explain the longer query times.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile, https://github.com/simonw/datasette/issues/1727#issuecomment-1111553029,https://api.github.com/repos/simonw/datasette/issues/1727,1111553029,IC_kwDOBm6k_c5CQPQF,9599,simonw,2022-04-27T22:48:21Z,2022-04-27T22:48:21Z,OWNER,I wonder if it would be worth exploring multiprocessing here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile, https://github.com/simonw/datasette/issues/1727#issuecomment-1111551076,https://api.github.com/repos/simonw/datasette/issues/1727,1111551076,IC_kwDOBm6k_c5CQOxk,9599,simonw,2022-04-27T22:44:51Z,2022-04-27T22:45:04Z,OWNER,Really wild idea: what if I created three copies of the SQLite database file - as three separate file names - and then balanced the parallel queries across all these? Any chance that could avoid any mysterious locking issues?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile, https://github.com/simonw/datasette/issues/1727#issuecomment-1111535818,https://api.github.com/repos/simonw/datasette/issues/1727,1111535818,IC_kwDOBm6k_c5CQLDK,9599,simonw,2022-04-27T22:18:45Z,2022-04-27T22:18:45Z,OWNER,"Another avenue: https://twitter.com/weargoggles/status/1519426289920270337 > SQLite has its own mutexes to provide thread safety, which as another poster noted are out of play in multi process setups. Perhaps downgrading from the “serializable” to “multi-threaded” safety would be okay for Datasette? https://sqlite.org/c3ref/c_config_covering_index_scan.html#sqliteconfigmultithread Doesn't look like there's an obvious way to access that from Python via the `sqlite3` module though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile, https://github.com/simonw/datasette/issues/1727#issuecomment-1111485722,https://api.github.com/repos/simonw/datasette/issues/1727,1111485722,IC_kwDOBm6k_c5CP-0a,9599,simonw,2022-04-27T21:08:20Z,2022-04-27T21:08:20Z,OWNER,"Tried that and it didn't seem to make a difference either. I really need a much deeper view of what's going on here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile, https://github.com/simonw/datasette/issues/1727#issuecomment-1111462442,https://api.github.com/repos/simonw/datasette/issues/1727,1111462442,IC_kwDOBm6k_c5CP5Iq,9599,simonw,2022-04-27T20:40:59Z,2022-04-27T20:42:49Z,OWNER,"This looks VERY relevant: [SQLite Shared-Cache Mode](https://www.sqlite.org/sharedcache.html): > SQLite includes a special ""shared-cache"" mode (disabled by default) intended for use in embedded servers. If shared-cache mode is enabled and a thread establishes multiple connections to the same database, the connections share a single data and schema cache. This can significantly reduce the quantity of memory and IO required by the system. Enabled as part of the URI filename: ATTACH 'file:aux.db?cache=shared' AS aux; Turns out I'm already using this for in-memory databases that have `.memory_name` set, but not (yet) for regular file-backed databases: https://github.com/simonw/datasette/blob/7a6654a253dee243518dc542ce4c06dbb0d0801d/datasette/database.py#L73-L75 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile, https://github.com/simonw/datasette/issues/1727#issuecomment-1111460068,https://api.github.com/repos/simonw/datasette/issues/1727,1111460068,IC_kwDOBm6k_c5CP4jk,9599,simonw,2022-04-27T20:38:32Z,2022-04-27T20:38:32Z,OWNER,WAL mode didn't seem to make a difference. I thought there was a chance it might help multiple read connections operate at the same time but it looks like it really does only matter for when writes are going on.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile, https://github.com/simonw/datasette/issues/1727#issuecomment-1111456500,https://api.github.com/repos/simonw/datasette/issues/1727,1111456500,IC_kwDOBm6k_c5CP3r0,9599,simonw,2022-04-27T20:36:01Z,2022-04-27T20:36:01Z,OWNER,"Yeah all of this is pretty much assuming read-only connections. Datasette has a separate mechanism for ensuring that writes are executed one at a time against a dedicated connection from an in-memory queue: - https://github.com/simonw/datasette/issues/682","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile, https://github.com/simonw/datasette/issues/1727#issuecomment-1111442012,https://api.github.com/repos/simonw/datasette/issues/1727,1111442012,IC_kwDOBm6k_c5CP0Jc,9599,simonw,2022-04-27T20:19:00Z,2022-04-27T20:19:00Z,OWNER,"Something worth digging into: are these parallel queries running against the same SQLite connection or are they each rubbing against a separate SQLite connection? Just realized I know the answer: they're running against separate SQLite connections, because that's how the time limit mechanism works: it installs a progress handler for each connection which terminates it after a set time. This means that if SQLite benefits from multiple threads using the same connection (due to shared caches or similar) then Datasette will not be seeing those benefits. It also means that if there's some mechanism within SQLite that penalizes you for having multiple parallel connections to a single file (just guessing here, maybe there's some kind of locking going on?) then Datasette will suffer those penalties. I should try seeing what happens with WAL mode enabled.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile, https://github.com/simonw/datasette/issues/1727#issuecomment-1111432375,https://api.github.com/repos/simonw/datasette/issues/1727,1111432375,IC_kwDOBm6k_c5CPxy3,9599,simonw,2022-04-27T20:07:57Z,2022-04-27T20:07:57Z,OWNER,Also useful: https://avi.im/blag/2021/fast-sqlite-inserts/ - from a tip on Twitter: https://twitter.com/ricardoanderegg/status/1519402047556235264,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile, https://github.com/simonw/datasette/issues/1727#issuecomment-1111431785,https://api.github.com/repos/simonw/datasette/issues/1727,1111431785,IC_kwDOBm6k_c5CPxpp,9599,simonw,2022-04-27T20:07:16Z,2022-04-27T20:07:16Z,OWNER,"I think I need some much more in-depth tracing tricks for this. https://www.maartenbreddels.com/perf/jupyter/python/tracing/gil/2021/01/14/Tracing-the-Python-GIL.html looks relevant - uses the `perf` tool on Linux.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile, https://github.com/simonw/datasette/issues/1727#issuecomment-1111408273,https://api.github.com/repos/simonw/datasette/issues/1727,1111408273,IC_kwDOBm6k_c5CPr6R,9599,simonw,2022-04-27T19:40:51Z,2022-04-27T19:42:17Z,OWNER,"Relevant: here's the code that sets up a Datasette SQLite connection: https://github.com/simonw/datasette/blob/7a6654a253dee243518dc542ce4c06dbb0d0801d/datasette/database.py#L73-L96 It's using `check_same_thread=False` - here's [the Python docs on that](https://docs.python.org/3/library/sqlite3.html#sqlite3.connect): > By default, *check_same_thread* is [`True`](https://docs.python.org/3/library/constants.html#True ""True"") and only the creating thread may use the connection. If set [`False`](https://docs.python.org/3/library/constants.html#False ""False""), the returned connection may be shared across multiple threads. When using multiple threads with the same connection writing operations should be serialized by the user to avoid data corruption. This is why Datasette reserves a single connection for write queries and queues them up in memory, [as described here](https://simonwillison.net/2020/Feb/26/weeknotes-datasette-writes/).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile, https://github.com/simonw/datasette/issues/1727#issuecomment-1111390433,https://api.github.com/repos/simonw/datasette/issues/1727,1111390433,IC_kwDOBm6k_c5CPnjh,9599,simonw,2022-04-27T19:21:02Z,2022-04-27T19:21:02Z,OWNER,"One weird thing: I noticed that in the parallel trace above the SQL query bars are wider. Mousover shows duration in ms, and I got 13ms for this query: select message as value, count(*) as n from ( But in the `?_noparallel=1` version that some query took 2.97ms. Given those numbers though I would expect the overall page time to be MUCH worse for the parallel version - but the page load times are instead very close to each other, with parallel often winning. This is super-weird.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile, https://github.com/simonw/datasette/issues/1727#issuecomment-1111385875,https://api.github.com/repos/simonw/datasette/issues/1727,1111385875,IC_kwDOBm6k_c5CPmcT,9599,simonw,2022-04-27T19:16:57Z,2022-04-27T19:16:57Z,OWNER,"I just remembered the `--setting num_sql_threads` option... which defaults to 3! https://github.com/simonw/datasette/blob/942411ef946e9a34a2094944d3423cddad27efd3/datasette/app.py#L109-L113 Would explain why the first trace never seems to show more than three SQL queries executing at once.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile, https://github.com/simonw/datasette/issues/1727#issuecomment-1111380282,https://api.github.com/repos/simonw/datasette/issues/1727,1111380282,IC_kwDOBm6k_c5CPlE6,9599,simonw,2022-04-27T19:10:27Z,2022-04-27T19:10:27Z,OWNER,"Wrote more about that here: https://simonwillison.net/2022/Apr/27/parallel-queries/ Compare https://latest-with-plugins.datasette.io/github/commits?_facet=repo&_facet=committer&_trace=1 ![image](https://user-images.githubusercontent.com/9599/165601503-2083c5d2-d740-405c-b34d-85570744ca82.png) With the same thing but with parallel execution disabled: https://latest-with-plugins.datasette.io/github/commits?_facet=repo&_facet=committer&_trace=1&_noparallel=1 ![image](https://user-images.githubusercontent.com/9599/165601525-98abbfb1-5631-4040-b6bd-700948d1db6e.png) Those total page load time numbers are very similar. Is this parallel optimization worthwhile? Maybe it's only worth it on larger databases? Or maybe larger databases perform worse with this?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile, https://github.com/simonw/datasette/issues/1724#issuecomment-1110585475,https://api.github.com/repos/simonw/datasette/issues/1724,1110585475,IC_kwDOBm6k_c5CMjCD,9599,simonw,2022-04-27T06:15:14Z,2022-04-27T06:15:14Z,OWNER,"Yeah, that page is 438K (but only 20K gzipped).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1216619276,?_trace=1 doesn't work on Global Power Plants demo, https://github.com/simonw/datasette/issues/1724#issuecomment-1110370095,https://api.github.com/repos/simonw/datasette/issues/1724,1110370095,IC_kwDOBm6k_c5CLucv,9599,simonw,2022-04-27T00:18:30Z,2022-04-27T00:18:30Z,OWNER,"So this isn't a bug here, it's working as intended.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1216619276,?_trace=1 doesn't work on Global Power Plants demo, https://github.com/simonw/datasette/issues/1724#issuecomment-1110369004,https://api.github.com/repos/simonw/datasette/issues/1724,1110369004,IC_kwDOBm6k_c5CLuLs,9599,simonw,2022-04-27T00:16:35Z,2022-04-27T00:17:04Z,OWNER,"I bet this is because it's exceeding the size limit: https://github.com/simonw/datasette/blob/da53e0360da4771ffb56a8e3eb3f7476f3168299/datasette/tracer.py#L80-L88 https://github.com/simonw/datasette/blob/da53e0360da4771ffb56a8e3eb3f7476f3168299/datasette/tracer.py#L102-L113","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1216619276,?_trace=1 doesn't work on Global Power Plants demo, https://github.com/simonw/datasette/issues/1723#issuecomment-1110330554,https://api.github.com/repos/simonw/datasette/issues/1723,1110330554,IC_kwDOBm6k_c5CLky6,9599,simonw,2022-04-26T23:06:20Z,2022-04-26T23:06:20Z,OWNER,Deployed here: https://latest-with-plugins.datasette.io/github/commits?_facet=repo&_trace=1&_facet=committer,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1216508080,Research running SQL in table view in parallel using `asyncio.gather()`, https://github.com/simonw/datasette/issues/1723#issuecomment-1110305790,https://api.github.com/repos/simonw/datasette/issues/1723,1110305790,IC_kwDOBm6k_c5CLev-,9599,simonw,2022-04-26T22:19:04Z,2022-04-26T22:19:04Z,OWNER,"I realized that seeing the total time in queries wasn't enough to understand this, because if the queries were executed in serial or parallel it should still sum up to the same amount of SQL time (roughly). Instead I need to know how long the page took to render. But that's hard to display on the page since you can't measure it until rendering has finished! So I built an ASGI plugin to handle that measurement: https://github.com/simonw/datasette-total-page-time And with that plugin installed, `http://127.0.0.1:8001/global-power-plants/global-power-plants?_facet=primary_fuel&_facet=other_fuel2&_facet=other_fuel1&_parallel=1` (the parallel version) takes 377ms: While `http://127.0.0.1:8001/global-power-plants/global-power-plants?_facet=primary_fuel&_facet=other_fuel2&_facet=other_fuel1` (the serial version) takes 762ms: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1216508080,Research running SQL in table view in parallel using `asyncio.gather()`, https://github.com/simonw/datasette/issues/1723#issuecomment-1110279869,https://api.github.com/repos/simonw/datasette/issues/1723,1110279869,IC_kwDOBm6k_c5CLYa9,9599,simonw,2022-04-26T21:45:39Z,2022-04-26T21:45:39Z,OWNER,"Getting some nice traces out of this: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1216508080,Research running SQL in table view in parallel using `asyncio.gather()`, https://github.com/simonw/datasette/issues/1723#issuecomment-1110278577,https://api.github.com/repos/simonw/datasette/issues/1723,1110278577,IC_kwDOBm6k_c5CLYGx,9599,simonw,2022-04-26T21:44:04Z,2022-04-26T21:44:04Z,OWNER,"And some simple benchmarks with `ab` - using the `?_parallel=1` hack to try it with and without a parallel `asyncio.gather()`: ``` ~ % ab -n 100 'http://127.0.0.1:8001/global-power-plants/global-power-plants?_facet=primary_fuel&_facet=other_fuel1&_facet=other_fuel3&_facet=other_fuel2' This is ApacheBench, Version 2.3 <$Revision: 1879490 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking 127.0.0.1 (be patient).....done Server Software: uvicorn Server Hostname: 127.0.0.1 Server Port: 8001 Document Path: /global-power-plants/global-power-plants?_facet=primary_fuel&_facet=other_fuel1&_facet=other_fuel3&_facet=other_fuel2 Document Length: 314187 bytes Concurrency Level: 1 Time taken for tests: 68.279 seconds Complete requests: 100 Failed requests: 13 (Connect: 0, Receive: 0, Length: 13, Exceptions: 0) Total transferred: 31454937 bytes HTML transferred: 31418437 bytes Requests per second: 1.46 [#/sec] (mean) Time per request: 682.787 [ms] (mean) Time per request: 682.787 [ms] (mean, across all concurrent requests) Transfer rate: 449.89 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.0 0 0 Processing: 621 683 68.0 658 993 Waiting: 620 682 68.0 657 992 Total: 621 683 68.0 658 993 Percentage of the requests served within a certain time (ms) 50% 658 66% 678 75% 687 80% 711 90% 763 95% 879 98% 926 99% 993 100% 993 (longest request) ---- In parallel: ~ % ab -n 100 'http://127.0.0.1:8001/global-power-plants/global-power-plants?_facet=primary_fuel&_facet=other_fuel1&_facet=other_fuel3&_facet=other_fuel2&_parallel=1' This is ApacheBench, Version 2.3 <$Revision: 1879490 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking 127.0.0.1 (be patient).....done Server Software: uvicorn Server Hostname: 127.0.0.1 Server Port: 8001 Document Path: /global-power-plants/global-power-plants?_facet=primary_fuel&_facet=other_fuel1&_facet=other_fuel3&_facet=other_fuel2&_parallel=1 Document Length: 315703 bytes Concurrency Level: 1 Time taken for tests: 34.763 seconds Complete requests: 100 Failed requests: 11 (Connect: 0, Receive: 0, Length: 11, Exceptions: 0) Total transferred: 31607988 bytes HTML transferred: 31570288 bytes Requests per second: 2.88 [#/sec] (mean) Time per request: 347.632 [ms] (mean) Time per request: 347.632 [ms] (mean, across all concurrent requests) Transfer rate: 887.93 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.0 0 0 Processing: 311 347 28.0 338 450 Waiting: 311 347 28.0 338 450 Total: 312 348 28.0 338 451 Percentage of the requests served within a certain time (ms) 50% 338 66% 348 75% 361 80% 367 90% 396 95% 408 98% 436 99% 451 100% 451 (longest request) ---- With concurrency 10, not parallel: ~ % ab -c 10 -n 100 'http://127.0.0.1:8001/global-power-plants/global-power-plants?_facet=primary_fuel&_facet=other_fuel1&_facet=other_fuel3&_facet=other_fuel2&_parallel=' This is ApacheBench, Version 2.3 <$Revision: 1879490 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking 127.0.0.1 (be patient).....done Server Software: uvicorn Server Hostname: 127.0.0.1 Server Port: 8001 Document Path: /global-power-plants/global-power-plants?_facet=primary_fuel&_facet=other_fuel1&_facet=other_fuel3&_facet=other_fuel2&_parallel= Document Length: 314346 bytes Concurrency Level: 10 Time taken for tests: 38.408 seconds Complete requests: 100 Failed requests: 93 (Connect: 0, Receive: 0, Length: 93, Exceptions: 0) Total transferred: 31471333 bytes HTML transferred: 31433733 bytes Requests per second: 2.60 [#/sec] (mean) Time per request: 3840.829 [ms] (mean) Time per request: 384.083 [ms] (mean, across all concurrent requests) Transfer rate: 800.18 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.1 0 1 Processing: 685 3719 354.0 3774 4096 Waiting: 684 3707 353.7 3750 4095 Total: 685 3719 354.0 3774 4096 Percentage of the requests served within a certain time (ms) 50% 3774 66% 3832 75% 3855 80% 3878 90% 3944 95% 4006 98% 4057 99% 4096 100% 4096 (longest request) ---- Concurrency 10 parallel: ~ % ab -c 10 -n 100 'http://127.0.0.1:8001/global-power-plants/global-power-plants?_facet=primary_fuel&_facet=other_fuel1&_facet=other_fuel3&_facet=other_fuel2&_parallel=1' This is ApacheBench, Version 2.3 <$Revision: 1879490 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking 127.0.0.1 (be patient).....done Server Software: uvicorn Server Hostname: 127.0.0.1 Server Port: 8001 Document Path: /global-power-plants/global-power-plants?_facet=primary_fuel&_facet=other_fuel1&_facet=other_fuel3&_facet=other_fuel2&_parallel=1 Document Length: 315703 bytes Concurrency Level: 10 Time taken for tests: 36.762 seconds Complete requests: 100 Failed requests: 89 (Connect: 0, Receive: 0, Length: 89, Exceptions: 0) Total transferred: 31606516 bytes HTML transferred: 31568816 bytes Requests per second: 2.72 [#/sec] (mean) Time per request: 3676.182 [ms] (mean) Time per request: 367.618 [ms] (mean, across all concurrent requests) Transfer rate: 839.61 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.1 0 0 Processing: 381 3602 419.6 3609 4458 Waiting: 381 3586 418.7 3607 4457 Total: 381 3603 419.6 3609 4458 Percentage of the requests served within a certain time (ms) 50% 3609 66% 3741 75% 3791 80% 3821 90% 3972 95% 4074 98% 4386 99% 4458 100% 4458 (longest request) Trying -c 3 instead. Non parallel: ~ % ab -c 3 -n 100 'http://127.0.0.1:8001/global-power-plants/global-power-plants?_facet=primary_fuel&_facet=other_fuel1&_facet=other_fuel3&_facet=other_fuel2&_parallel=' This is ApacheBench, Version 2.3 <$Revision: 1879490 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking 127.0.0.1 (be patient).....done Server Software: uvicorn Server Hostname: 127.0.0.1 Server Port: 8001 Document Path: /global-power-plants/global-power-plants?_facet=primary_fuel&_facet=other_fuel1&_facet=other_fuel3&_facet=other_fuel2&_parallel= Document Length: 314346 bytes Concurrency Level: 3 Time taken for tests: 39.365 seconds Complete requests: 100 Failed requests: 83 (Connect: 0, Receive: 0, Length: 83, Exceptions: 0) Total transferred: 31470808 bytes HTML transferred: 31433208 bytes Requests per second: 2.54 [#/sec] (mean) Time per request: 1180.955 [ms] (mean) Time per request: 393.652 [ms] (mean, across all concurrent requests) Transfer rate: 780.72 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.0 0 0 Processing: 731 1153 126.2 1189 1359 Waiting: 730 1151 125.9 1188 1358 Total: 731 1153 126.2 1189 1359 Percentage of the requests served within a certain time (ms) 50% 1189 66% 1221 75% 1234 80% 1247 90% 1296 95% 1309 98% 1343 99% 1359 100% 1359 (longest request) ---- Parallel: ~ % ab -c 3 -n 100 'http://127.0.0.1:8001/global-power-plants/global-power-plants?_facet=primary_fuel&_facet=other_fuel1&_facet=other_fuel3&_facet=other_fuel2&_parallel=1' This is ApacheBench, Version 2.3 <$Revision: 1879490 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking 127.0.0.1 (be patient).....done Server Software: uvicorn Server Hostname: 127.0.0.1 Server Port: 8001 Document Path: /global-power-plants/global-power-plants?_facet=primary_fuel&_facet=other_fuel1&_facet=other_fuel3&_facet=other_fuel2&_parallel=1 Document Length: 315703 bytes Concurrency Level: 3 Time taken for tests: 34.530 seconds Complete requests: 100 Failed requests: 18 (Connect: 0, Receive: 0, Length: 18, Exceptions: 0) Total transferred: 31606179 bytes HTML transferred: 31568479 bytes Requests per second: 2.90 [#/sec] (mean) Time per request: 1035.902 [ms] (mean) Time per request: 345.301 [ms] (mean, across all concurrent requests) Transfer rate: 893.87 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.0 0 0 Processing: 412 1020 104.4 1018 1280 Waiting: 411 1018 104.1 1014 1275 Total: 412 1021 104.4 1018 1280 Percentage of the requests served within a certain time (ms) 50% 1018 66% 1041 75% 1061 80% 1079 90% 1136 95% 1176 98% 1251 99% 1280 100% 1280 (longest request) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1216508080,Research running SQL in table view in parallel using `asyncio.gather()`, https://github.com/simonw/datasette/issues/1723#issuecomment-1110278182,https://api.github.com/repos/simonw/datasette/issues/1723,1110278182,IC_kwDOBm6k_c5CLYAm,9599,simonw,2022-04-26T21:43:34Z,2022-04-26T21:43:34Z,OWNER,"Here's the diff I'm using: ```diff diff --git a/datasette/views/table.py b/datasette/views/table.py index d66adb8..f15ef1e 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -1,3 +1,4 @@ +import asyncio import itertools import json @@ -5,6 +6,7 @@ import markupsafe from datasette.plugins import pm from datasette.database import QueryInterrupted +from datasette import tracer from datasette.utils import ( await_me_maybe, CustomRow, @@ -150,6 +152,16 @@ class TableView(DataView): default_labels=False, _next=None, _size=None, + ): + with tracer.trace_child_tasks(): + return await self._data_traced(request, default_labels, _next, _size) + + async def _data_traced( + self, + request, + default_labels=False, + _next=None, + _size=None, ): database_route = tilde_decode(request.url_vars[""database""]) table_name = tilde_decode(request.url_vars[""table""]) @@ -159,6 +171,20 @@ class TableView(DataView): raise NotFound(""Database not found: {}"".format(database_route)) database_name = db.name + # For performance profiling purposes, ?_parallel=1 turns on asyncio.gather + async def _gather_parallel(*args): + return await asyncio.gather(*args) + + async def _gather_sequential(*args): + results = [] + for fn in args: + results.append(await fn) + return results + + gather = ( + _gather_parallel if request.args.get(""_parallel"") else _gather_sequential + ) + # If this is a canned query, not a table, then dispatch to QueryView instead canned_query = await self.ds.get_canned_query( database_name, table_name, request.actor @@ -174,8 +200,12 @@ class TableView(DataView): write=bool(canned_query.get(""write"")), ) - is_view = bool(await db.get_view_definition(table_name)) - table_exists = bool(await db.table_exists(table_name)) + is_view, table_exists = map( + bool, + await gather( + db.get_view_definition(table_name), db.table_exists(table_name) + ), + ) # If table or view not found, return 404 if not is_view and not table_exists: @@ -497,33 +527,44 @@ class TableView(DataView): ) ) - if not nofacet: - for facet in facet_instances: - ( + async def execute_facets(): + if not nofacet: + # Run them in parallel + facet_awaitables = [facet.facet_results() for facet in facet_instances] + facet_awaitable_results = await gather(*facet_awaitables) + for ( instance_facet_results, instance_facets_timed_out, - ) = await facet.facet_results() - for facet_info in instance_facet_results: - base_key = facet_info[""name""] - key = base_key - i = 1 - while key in facet_results: - i += 1 - key = f""{base_key}_{i}"" - facet_results[key] = facet_info - facets_timed_out.extend(instance_facets_timed_out) - - # Calculate suggested facets + ) in facet_awaitable_results: + for facet_info in instance_facet_results: + base_key = facet_info[""name""] + key = base_key + i = 1 + while key in facet_results: + i += 1 + key = f""{base_key}_{i}"" + facet_results[key] = facet_info + facets_timed_out.extend(instance_facets_timed_out) + suggested_facets = [] - if ( - self.ds.setting(""suggest_facets"") - and self.ds.setting(""allow_facet"") - and not _next - and not nofacet - and not nosuggest - ): - for facet in facet_instances: - suggested_facets.extend(await facet.suggest()) + + async def execute_suggested_facets(): + # Calculate suggested facets + if ( + self.ds.setting(""suggest_facets"") + and self.ds.setting(""allow_facet"") + and not _next + and not nofacet + and not nosuggest + ): + # Run them in parallel + facet_suggest_awaitables = [ + facet.suggest() for facet in facet_instances + ] + for suggest_result in await gather(*facet_suggest_awaitables): + suggested_facets.extend(suggest_result) + + await gather(execute_facets(), execute_suggested_facets()) # Figure out columns and rows for the query columns = [r[0] for r in results.description] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1216508080,Research running SQL in table view in parallel using `asyncio.gather()`, https://github.com/simonw/datasette/issues/1715#issuecomment-1110265087,https://api.github.com/repos/simonw/datasette/issues/1715,1110265087,IC_kwDOBm6k_c5CLUz_,9599,simonw,2022-04-26T21:26:17Z,2022-04-26T21:26:17Z,OWNER,"Running facets and facet suggestions in parallel using `asyncio.gather()` turns out to be a lot less hassle than I had thought - maybe I don't need `asyncinject` for this at all? ```diff if not nofacet: - for facet in facet_instances: - ( - instance_facet_results, - instance_facets_timed_out, - ) = await facet.facet_results() + # Run them in parallel + facet_awaitables = [facet.facet_results() for facet in facet_instances] + facet_awaitable_results = await asyncio.gather(*facet_awaitables) + for ( + instance_facet_results, + instance_facets_timed_out, + ) in facet_awaitable_results: for facet_info in instance_facet_results: base_key = facet_info[""name""] key = base_key @@ -522,8 +540,10 @@ class TableView(DataView): and not nofacet and not nosuggest ): - for facet in facet_instances: - suggested_facets.extend(await facet.suggest()) + # Run them in parallel + facet_suggest_awaitables = [facet.suggest() for facet in facet_instances] + for suggest_result in await asyncio.gather(*facet_suggest_awaitables): + suggested_facets.extend(suggest_result) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1212823665,Refactor TableView to use asyncinject, https://github.com/simonw/datasette/issues/1715#issuecomment-1110246593,https://api.github.com/repos/simonw/datasette/issues/1715,1110246593,IC_kwDOBm6k_c5CLQTB,9599,simonw,2022-04-26T21:03:56Z,2022-04-26T21:03:56Z,OWNER,"Well this is fun... I applied this change: ```diff diff --git a/datasette/views/table.py b/datasette/views/table.py index d66adb8..85f9e44 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -1,3 +1,4 @@ +import asyncio import itertools import json @@ -5,6 +6,7 @@ import markupsafe from datasette.plugins import pm from datasette.database import QueryInterrupted +from datasette import tracer from datasette.utils import ( await_me_maybe, CustomRow, @@ -174,8 +176,11 @@ class TableView(DataView): write=bool(canned_query.get(""write"")), ) - is_view = bool(await db.get_view_definition(table_name)) - table_exists = bool(await db.table_exists(table_name)) + with tracer.trace_child_tasks(): + is_view, table_exists = map(bool, await asyncio.gather( + db.get_view_definition(table_name), + db.table_exists(table_name) + )) # If table or view not found, return 404 if not is_view and not table_exists: ``` And now using https://datasette.io/plugins/datasette-pretty-traces I get this: ![CleanShot 2022-04-26 at 14 03 33@2x](https://user-images.githubusercontent.com/9599/165392009-84c4399d-3e94-46d4-ba7b-a64a116cac5c.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1212823665,Refactor TableView to use asyncinject, https://github.com/simonw/datasette/issues/1715#issuecomment-1110219185,https://api.github.com/repos/simonw/datasette/issues/1715,1110219185,IC_kwDOBm6k_c5CLJmx,9599,simonw,2022-04-26T20:28:40Z,2022-04-26T20:56:48Z,OWNER,"The refactor I did in #1719 pretty much clashes with all of the changes in https://github.com/simonw/datasette/commit/5053f1ea83194ecb0a5693ad5dada5b25bf0f7e6 so I'll probably need to start my `api-extras` branch again from scratch. Using a new `tableview-asyncinject` branch.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1212823665,Refactor TableView to use asyncinject, https://github.com/simonw/datasette/issues/1715#issuecomment-1110239536,https://api.github.com/repos/simonw/datasette/issues/1715,1110239536,IC_kwDOBm6k_c5CLOkw,9599,simonw,2022-04-26T20:54:53Z,2022-04-26T20:54:53Z,OWNER,`pytest tests/test_table_*` runs the tests quickly.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1212823665,Refactor TableView to use asyncinject, https://github.com/simonw/datasette/issues/1715#issuecomment-1110238896,https://api.github.com/repos/simonw/datasette/issues/1715,1110238896,IC_kwDOBm6k_c5CLOaw,9599,simonw,2022-04-26T20:53:59Z,2022-04-26T20:53:59Z,OWNER,I'm going to rename `database` to `database_name` and `table` to `table_name` to avoid confusion with the `Database` object as opposed to the string name for the database.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1212823665,Refactor TableView to use asyncinject, https://github.com/simonw/datasette/issues/1715#issuecomment-1110229319,https://api.github.com/repos/simonw/datasette/issues/1715,1110229319,IC_kwDOBm6k_c5CLMFH,9599,simonw,2022-04-26T20:41:32Z,2022-04-26T20:44:38Z,OWNER,"This time I'm not going to bother with the `filter_args` thing - I'm going to just try to use `asyncinject` to execute some big high level things in parallel - facets, suggested facets, counts, the query - and then combine it with the `extras` mechanism I'm trying to introduce too. Most importantly: I want that `extra_template()` function that adds more template context for the HTML to be executed as part of an `asyncinject` flow!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1212823665,Refactor TableView to use asyncinject, https://github.com/simonw/datasette/issues/1720#issuecomment-1110212021,https://api.github.com/repos/simonw/datasette/issues/1720,1110212021,IC_kwDOBm6k_c5CLH21,9599,simonw,2022-04-26T20:20:27Z,2022-04-26T20:20:27Z,OWNER,Closing this because I have a good enough idea of the design for now - the details of the parameters can be figured out when I implement this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1215174094,Design plugin hook for extras, https://github.com/simonw/datasette/issues/1720#issuecomment-1109309683,https://api.github.com/repos/simonw/datasette/issues/1720,1109309683,IC_kwDOBm6k_c5CHrjz,9599,simonw,2022-04-26T04:12:39Z,2022-04-26T04:12:39Z,OWNER,"I think the rough shape of the three plugin hooks is right. The detailed decisions that are needed concern what the parameters should be, which I think will mainly happen as part of: - #1715","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1215174094,Design plugin hook for extras, https://github.com/simonw/datasette/issues/1720#issuecomment-1109306070,https://api.github.com/repos/simonw/datasette/issues/1720,1109306070,IC_kwDOBm6k_c5CHqrW,9599,simonw,2022-04-26T04:05:20Z,2022-04-26T04:05:20Z,OWNER,"The proposed plugin for annotations - allowing users to attach comments to database tables, columns and rows - would be a great application for all three of those `?_extra=` plugin hooks.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1215174094,Design plugin hook for extras, https://github.com/simonw/datasette/issues/1720#issuecomment-1109305184,https://api.github.com/repos/simonw/datasette/issues/1720,1109305184,IC_kwDOBm6k_c5CHqdg,9599,simonw,2022-04-26T04:03:35Z,2022-04-26T04:03:35Z,OWNER,I bet there's all kinds of interesting potential extras that could be calculated by loading the results of the query into a Pandas DataFrame.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1215174094,Design plugin hook for extras, https://github.com/simonw/datasette/issues/1720#issuecomment-1109200774,https://api.github.com/repos/simonw/datasette/issues/1720,1109200774,IC_kwDOBm6k_c5CHQ-G,9599,simonw,2022-04-26T01:25:43Z,2022-04-26T01:26:15Z,OWNER,"Had a thought: if a custom HTML template is going to make use of stuff generated using these extras, it will need a way to tell Datasette to execute those extras even in the absence of the `?_extra=...` URL parameters. Is that necessary? Or should those kinds of plugins use the existing `extra_template_vars` hook instead? Or maybe the `extra_template_vars` hook gets redesigned so it can depend on other `extras` in some way?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1215174094,Design plugin hook for extras, https://github.com/simonw/datasette/issues/1720#issuecomment-1109200335,https://api.github.com/repos/simonw/datasette/issues/1720,1109200335,IC_kwDOBm6k_c5CHQ3P,9599,simonw,2022-04-26T01:24:47Z,2022-04-26T01:24:47Z,OWNER,"Sketching out a `?_extra=statistics` table plugin: ```python from datasette import hookimpl @hookimpl def register_table_extras(datasette): return [statistics] async def statistics(datasette, query, columns, sql): # ... need to figure out which columns are integer/floats # then build and execute a SQL query that calculates sum/avg/etc for each column ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1215174094,Design plugin hook for extras, https://github.com/simonw/sqlite-utils/issues/428#issuecomment-1109190401,https://api.github.com/repos/simonw/sqlite-utils/issues/428,1109190401,IC_kwDOCGYnMM5CHOcB,9599,simonw,2022-04-26T01:05:29Z,2022-04-26T01:05:29Z,OWNER,Django makes extensive use of savepoints for nested transactions: https://docs.djangoproject.com/en/4.0/topics/db/transactions/#savepoints,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1215216249,Research adding support for savepoints, https://github.com/simonw/datasette/issues/1720#issuecomment-1109174715,https://api.github.com/repos/simonw/datasette/issues/1720,1109174715,IC_kwDOBm6k_c5CHKm7,9599,simonw,2022-04-26T00:40:13Z,2022-04-26T00:43:33Z,OWNER,"Some of the things I'd like to use `?_extra=` for, that may or not make sense as plugins: - Performance breakdown information, maybe including explain output for a query/table - Information about the tables that were consulted in a query - imagine pulling in additional table metadata - Statistical aggregates against the full set of results. This may well be a Datasette core feature at some point in the future, but being able to provide it early as a plugin would be really cool. - For tables, what are the other tables they can join against? - Suggested facets - Facet results themselves - New custom facets I haven't thought of - though the `register_facet_classes` hook covers that already - Table schema - Table metadata - Analytics - how many times has this table been queried? Would be a plugin thing - For geospatial data, how about a GeoJSON polygon that represents the bounding box for all returned results? Effectively this is an extra aggregation. Looking at https://github-to-sqlite.dogsheep.net/github/commits.json?_labels=on&_shape=objects for inspiration. I think there's a separate potential mechanism in the future that lets you add custom columns to a table. This would affect `.csv` and the HTML presentation too, which makes it a different concept from the `?_extra=` hook that affects the JSON export (and the context that is fed to the HTML templates).","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1215174094,Design plugin hook for extras, https://github.com/simonw/datasette/issues/1720#issuecomment-1109171871,https://api.github.com/repos/simonw/datasette/issues/1720,1109171871,IC_kwDOBm6k_c5CHJ6f,9599,simonw,2022-04-26T00:34:48Z,2022-04-26T00:34:48Z,OWNER,"Let's try sketching out a `register_table_extras` plugin for something new. The first idea I came up with suggests adding new fields to the individual row records that come back - my mental model for extras so far has been that they add new keys to the root object. So if a table result looked like this: ```json { ""rows"": [ {""id"": 1, ""name"": ""Cleo""}, {""id"": 2, ""name"": ""Suna""} ], ""next_url"": null } ``` I was initially thinking that `?_extra=facets` would add a `""facets"": {...}` key to that root object. Here's a plugin idea I came up with that would probably justify adding to the individual row objects instead: - `?_extra=check404s` - does an async `HEAD` request against every column value that looks like a URL and checks if it returns a 404 This could also work by adding a `""check404s"": {""url-here"": 200}` key to the root object though. I think I need some better plugin concepts before committing to this new hook. There's overlap between this and how I want the enrichments mechanism ([see here](https://simonwillison.net/2021/Jan/17/weeknotes-still-pretty-distracted/)) to work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1215174094,Design plugin hook for extras, https://github.com/simonw/datasette/issues/1720#issuecomment-1109165411,https://api.github.com/repos/simonw/datasette/issues/1720,1109165411,IC_kwDOBm6k_c5CHIVj,9599,simonw,2022-04-26T00:22:42Z,2022-04-26T00:22:42Z,OWNER,Passing `pk_values` to the plugin hook feels odd. I think I'd pass a `row` object instead and let the code look up the primary key values on that row (by introspecting the primary keys for the table).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1215174094,Design plugin hook for extras, https://github.com/simonw/datasette/issues/1720#issuecomment-1109164803,https://api.github.com/repos/simonw/datasette/issues/1720,1109164803,IC_kwDOBm6k_c5CHIMD,9599,simonw,2022-04-26T00:21:40Z,2022-04-26T00:21:40Z,OWNER,"What would the existing https://latest.datasette.io/fixtures/simple_primary_key/1.json?_extras=foreign_key_tables feature look like if it was re-imagined as a `register_row_extras()` plugin? Rough sketch, copying most of the code from https://github.com/simonw/datasette/blob/579f59dcec43a91dd7d404e00b87a00afd8515f2/datasette/views/row.py#L98 ```python from datasette import hookimpl @hookimpl def register_row_extras(datasette): return [foreign_key_tables] async def foreign_key_tables(datasette, database, table, pk_values): if len(pk_values) != 1: return [] db = datasette.get_database(database) all_foreign_keys = await db.get_all_foreign_keys() foreign_keys = all_foreign_keys[table][""incoming""] if len(foreign_keys) == 0: return [] sql = ""select "" + "", "".join( [ ""(select count(*) from {table} where {column}=:id)"".format( table=escape_sqlite(fk[""other_table""]), column=escape_sqlite(fk[""other_column""]), ) for fk in foreign_keys ] ) try: rows = list(await db.execute(sql, {""id"": pk_values[0]})) except QueryInterrupted: # Almost certainly hit the timeout return [] foreign_table_counts = dict( zip( [(fk[""other_table""], fk[""other_column""]) for fk in foreign_keys], list(rows[0]), ) ) foreign_key_tables = [] for fk in foreign_keys: count = ( foreign_table_counts.get((fk[""other_table""], fk[""other_column""])) or 0 ) key = fk[""other_column""] if key.startswith(""_""): key += ""__exact"" link = ""{}?{}={}"".format( self.ds.urls.table(database, fk[""other_table""]), key, "","".join(pk_values), ) foreign_key_tables.append({**fk, **{""count"": count, ""link"": link}}) return foreign_key_tables ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1215174094,Design plugin hook for extras, https://github.com/simonw/datasette/issues/1720#issuecomment-1109162123,https://api.github.com/repos/simonw/datasette/issues/1720,1109162123,IC_kwDOBm6k_c5CHHiL,9599,simonw,2022-04-26T00:16:42Z,2022-04-26T00:16:51Z,OWNER,"Actually I'm going to imitate the existing `register_*` hooks: - `def register_output_renderer(datasette)` - `def register_facet_classes()` - `def register_routes(datasette)` - `def register_commands(cli)` - `def register_magic_parameters(datasette)` So I'm going to call the new hooks: - `register_table_extras(datasette)` - `register_row_extras(datasette)` - `register_query_extras(datasette)` They'll return a list of `async def` functions. The names of those functions will become the names of the extras.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1215174094,Design plugin hook for extras, https://github.com/simonw/datasette/issues/1720#issuecomment-1109160226,https://api.github.com/repos/simonw/datasette/issues/1720,1109160226,IC_kwDOBm6k_c5CHHEi,9599,simonw,2022-04-26T00:14:11Z,2022-04-26T00:14:11Z,OWNER,"There are four existing plugin hooks that include the word ""extra"" but use it to mean something else - to mean additional CSS/JS/variables to be injected into the page: - `def extra_css_urls(...)` - `def extra_js_urls(...)` - `def extra_body_script(...)` - `def extra_template_vars(...)` I think `extra_*` and `*_extras` are different enough that they won't be confused with each other.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1215174094,Design plugin hook for extras, https://github.com/simonw/datasette/issues/1720#issuecomment-1109159307,https://api.github.com/repos/simonw/datasette/issues/1720,1109159307,IC_kwDOBm6k_c5CHG2L,9599,simonw,2022-04-26T00:12:28Z,2022-04-26T00:12:28Z,OWNER,"I'm going to keep table and row separate. So I think I need to add three new plugin hooks: - `table_extras()` - `row_extras()` - `query_extras()`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1215174094,Design plugin hook for extras, https://github.com/simonw/datasette/issues/1720#issuecomment-1109158903,https://api.github.com/repos/simonw/datasette/issues/1720,1109158903,IC_kwDOBm6k_c5CHGv3,9599,simonw,2022-04-26T00:11:42Z,2022-04-26T00:11:42Z,OWNER,"Places this plugin hook (or hooks?) should be able to affect: - JSON for a table/view - JSON for a row - JSON for a canned query - JSON for a custom arbitrary query I'm going to combine those last two, which means there are three places. But maybe I can combine the table one and the row one as well?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1215174094,Design plugin hook for extras, https://github.com/simonw/datasette/issues/1719#issuecomment-1108907238,https://api.github.com/repos/simonw/datasette/issues/1719,1108907238,IC_kwDOBm6k_c5CGJTm,9599,simonw,2022-04-25T18:34:21Z,2022-04-25T18:34:21Z,OWNER,Well this refactor turned out to be pretty quick and really does greatly simplify both the `RowView` and `TableView` classes. Very happy with this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1214859703,Refactor `RowView` and remove `RowTableShared`, https://github.com/simonw/datasette/issues/262#issuecomment-1108890170,https://api.github.com/repos/simonw/datasette/issues/262,1108890170,IC_kwDOBm6k_c5CGFI6,9599,simonw,2022-04-25T18:17:09Z,2022-04-25T18:18:39Z,OWNER,"I spotted in https://github.com/simonw/datasette/issues/1719#issuecomment-1108888494 that there's actually already an undocumented implementation of `?_extras=foreign_key_tables` - https://latest.datasette.io/fixtures/simple_primary_key/1.json?_extras=foreign_key_tables I added that feature all the way back in November 2017! https://github.com/simonw/datasette/commit/a30c5b220c15360d575e94b0e67f3255e120b916","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,Add ?_extra= mechanism for requesting extra properties in JSON, https://github.com/simonw/datasette/issues/1719#issuecomment-1108888494,https://api.github.com/repos/simonw/datasette/issues/1719,1108888494,IC_kwDOBm6k_c5CGEuu,9599,simonw,2022-04-25T18:15:42Z,2022-04-25T18:15:42Z,OWNER,"Here's an undocumented feature I forgot existed: https://latest.datasette.io/fixtures/simple_primary_key/1.json?_extras=foreign_key_tables `?_extras=foreign_key_tables` https://github.com/simonw/datasette/blob/0bc5186b7bb4fc82392df08f99a9132f84dcb331/datasette/views/table.py#L1021-L1024 It's even covered by the tests: https://github.com/simonw/datasette/blob/b9c2b1cfc8692b9700416db98721fa3ec982f6be/tests/test_api.py#L691-L703","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1214859703,Refactor `RowView` and remove `RowTableShared`, https://github.com/simonw/datasette/issues/1719#issuecomment-1108884171,https://api.github.com/repos/simonw/datasette/issues/1719,1108884171,IC_kwDOBm6k_c5CGDrL,9599,simonw,2022-04-25T18:10:46Z,2022-04-25T18:12:45Z,OWNER,"It looks like the only class method from that shared class needed by `RowView` is `self.display_columns_and_rows()`. Which I've been wanting to refactor to provide to `QueryView` too: - #715","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1214859703,Refactor `RowView` and remove `RowTableShared`, https://github.com/simonw/datasette/issues/1715#issuecomment-1108875068,https://api.github.com/repos/simonw/datasette/issues/1715,1108875068,IC_kwDOBm6k_c5CGBc8,9599,simonw,2022-04-25T18:03:13Z,2022-04-25T18:06:33Z,OWNER,"The `RowTableShared` class is making this a whole lot more complicated. I'm going to split the `RowView` view out into an entirely separate `views/row.py` module.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1212823665,Refactor TableView to use asyncinject, https://github.com/simonw/datasette/issues/1715#issuecomment-1108877454,https://api.github.com/repos/simonw/datasette/issues/1715,1108877454,IC_kwDOBm6k_c5CGCCO,9599,simonw,2022-04-25T18:04:27Z,2022-04-25T18:04:27Z,OWNER,Pushed my WIP on this to the `api-extras` branch: 5053f1ea83194ecb0a5693ad5dada5b25bf0f7e6,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1212823665,Refactor TableView to use asyncinject, https://github.com/simonw/datasette/issues/1718#issuecomment-1107873311,https://api.github.com/repos/simonw/datasette/issues/1718,1107873311,IC_kwDOBm6k_c5CCM4f,9599,simonw,2022-04-24T16:24:14Z,2022-04-24T16:24:14Z,OWNER,Wrote up what I learned in a TIL: https://til.simonwillison.net/sphinx/blacken-docs,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1213683988,Code examples in the documentation should be formatted with Black, https://github.com/simonw/datasette/issues/1718#issuecomment-1107873271,https://api.github.com/repos/simonw/datasette/issues/1718,1107873271,IC_kwDOBm6k_c5CCM33,9599,simonw,2022-04-24T16:23:57Z,2022-04-24T16:23:57Z,OWNER,"Turns out I didn't need that `git diff-index` trick after all - the `blacken-docs` command returns a non-zero exit code if it changes any files. Submitted a documentation PR to that project instead: - https://github.com/asottile/blacken-docs/pull/162","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1213683988,Code examples in the documentation should be formatted with Black, https://github.com/simonw/datasette/issues/1718#issuecomment-1107870788,https://api.github.com/repos/simonw/datasette/issues/1718,1107870788,IC_kwDOBm6k_c5CCMRE,9599,simonw,2022-04-24T16:09:23Z,2022-04-24T16:09:23Z,OWNER,One more attempt at testing the `git diff-index` trick.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1213683988,Code examples in the documentation should be formatted with Black, https://github.com/simonw/datasette/issues/1718#issuecomment-1107869884,https://api.github.com/repos/simonw/datasette/issues/1718,1107869884,IC_kwDOBm6k_c5CCMC8,9599,simonw,2022-04-24T16:04:03Z,2022-04-24T16:04:03Z,OWNER,"OK, I'm expecting this one to fail at the `git diff-index --quiet HEAD --` check.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1213683988,Code examples in the documentation should be formatted with Black, https://github.com/simonw/datasette/issues/1718#issuecomment-1107869556,https://api.github.com/repos/simonw/datasette/issues/1718,1107869556,IC_kwDOBm6k_c5CCL90,9599,simonw,2022-04-24T16:02:27Z,2022-04-24T16:02:27Z,OWNER,"Looking at that first error it appears to be a place where I had deliberately omitted the body of the function: https://github.com/simonw/datasette/blob/36573638b0948174ae237d62e6369b7d55220d7f/docs/internals.rst#L196-L211 I can use `...` as the function body here to get it to pass. Fixing those warnings actually helped me spot a couple of bugs, so I'm glad this happened.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1213683988,Code examples in the documentation should be formatted with Black, https://github.com/simonw/datasette/issues/1718#issuecomment-1107868585,https://api.github.com/repos/simonw/datasette/issues/1718,1107868585,IC_kwDOBm6k_c5CCLup,9599,simonw,2022-04-24T15:57:10Z,2022-04-24T15:57:19Z,OWNER,"The tests failed there because of what I thought were warnings but turn out to be treated as errors: ``` % blacken-docs -l 60 docs/*.rst docs/internals.rst:196: code block parse error Cannot parse: 14:0: docs/json_api.rst:449: code block parse error Cannot parse: 1:0: docs/testing_plugins.rst:135: code block parse error Cannot parse: 5:0: % echo $? 1 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1213683988,Code examples in the documentation should be formatted with Black, https://github.com/simonw/datasette/issues/1718#issuecomment-1107867281,https://api.github.com/repos/simonw/datasette/issues/1718,1107867281,IC_kwDOBm6k_c5CCLaR,9599,simonw,2022-04-24T15:49:23Z,2022-04-24T15:49:23Z,OWNER,I'm going to push the first commit with a deliberate missing formatting to check that the tests fail.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1213683988,Code examples in the documentation should be formatted with Black, https://github.com/simonw/datasette/issues/1718#issuecomment-1107866013,https://api.github.com/repos/simonw/datasette/issues/1718,1107866013,IC_kwDOBm6k_c5CCLGd,9599,simonw,2022-04-24T15:42:07Z,2022-04-24T15:42:07Z,OWNER,"In the absence of `--check` I can use this to detect if changes are applied: ```zsh % git diff-index --quiet HEAD -- % echo $? 0 % blacken-docs -l 60 docs/*.rst docs/authentication.rst: Rewriting... ... % git diff-index --quiet HEAD -- % echo $? 1 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1213683988,Code examples in the documentation should be formatted with Black, https://github.com/simonw/datasette/issues/1718#issuecomment-1107865493,https://api.github.com/repos/simonw/datasette/issues/1718,1107865493,IC_kwDOBm6k_c5CCK-V,9599,simonw,2022-04-24T15:39:02Z,2022-04-24T15:39:02Z,OWNER,"There's no `blacken-docs --check` option so I filed a feature request: - https://github.com/asottile/blacken-docs/issues/161","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1213683988,Code examples in the documentation should be formatted with Black, https://github.com/simonw/datasette/issues/1718#issuecomment-1107863924,https://api.github.com/repos/simonw/datasette/issues/1718,1107863924,IC_kwDOBm6k_c5CCKl0,9599,simonw,2022-04-24T15:30:03Z,2022-04-24T15:30:03Z,OWNER,"On the one hand, I'm not crazy about some of the indentation decisions Black made here - in particular this one, which I had indented deliberately for readability: ```diff diff --git a/docs/authentication.rst b/docs/authentication.rst index 0d98cf8..8008023 100644 --- a/docs/authentication.rst +++ b/docs/authentication.rst @@ -381,11 +381,7 @@ Authentication plugins can set signed ``ds_actor`` cookies themselves like so: .. code-block:: python response = Response.redirect(""/"") - response.set_cookie(""ds_actor"", datasette.sign({ - ""a"": { - ""id"": ""cleopaws"" - } - }, ""actor"")) + response.set_cookie(""ds_actor"", datasette.sign({""a"": {""id"": ""cleopaws""}}, ""actor"")) ``` But... consistency is a virtue. Maybe I'm OK with just this one disagreement? Also: I've been mentally trying to keep the line lengths a bit shorter to help them be more readable on mobile devices. I'll try a different line length using `blacken-docs -l 60 docs/*.rst` instead. I like this more - here's the result for that example: ```diff diff --git a/docs/authentication.rst b/docs/authentication.rst index 0d98cf8..2496073 100644 --- a/docs/authentication.rst +++ b/docs/authentication.rst @@ -381,11 +381,10 @@ Authentication plugins can set signed ``ds_actor`` cookies themselves like so: .. code-block:: python response = Response.redirect(""/"") - response.set_cookie(""ds_actor"", datasette.sign({ - ""a"": { - ""id"": ""cleopaws"" - } - }, ""actor"")) + response.set_cookie( + ""ds_actor"", + datasette.sign({""a"": {""id"": ""cleopaws""}}, ""actor""), + ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1213683988,Code examples in the documentation should be formatted with Black, https://github.com/simonw/datasette/issues/1718#issuecomment-1107863365,https://api.github.com/repos/simonw/datasette/issues/1718,1107863365,IC_kwDOBm6k_c5CCKdF,9599,simonw,2022-04-24T15:26:41Z,2022-04-24T15:26:41Z,OWNER,"Tried this: ``` pip install blacken-docs blacken-docs docs/*.rst git diff | pbcopy ``` Got this: ```diff diff --git a/docs/authentication.rst b/docs/authentication.rst index 0d98cf8..8008023 100644 --- a/docs/authentication.rst +++ b/docs/authentication.rst @@ -381,11 +381,7 @@ Authentication plugins can set signed ``ds_actor`` cookies themselves like so: .. code-block:: python response = Response.redirect(""/"") - response.set_cookie(""ds_actor"", datasette.sign({ - ""a"": { - ""id"": ""cleopaws"" - } - }, ""actor"")) + response.set_cookie(""ds_actor"", datasette.sign({""a"": {""id"": ""cleopaws""}}, ""actor"")) Note that you need to pass ``""actor""`` as the namespace to :ref:`datasette_sign`. @@ -412,12 +408,16 @@ To include an expiry, add a ``""e""`` key to the cookie value containing a `base62 expires_at = int(time.time()) + (24 * 60 * 60) response = Response.redirect(""/"") - response.set_cookie(""ds_actor"", datasette.sign({ - ""a"": { - ""id"": ""cleopaws"" - }, - ""e"": baseconv.base62.encode(expires_at), - }, ""actor"")) + response.set_cookie( + ""ds_actor"", + datasette.sign( + { + ""a"": {""id"": ""cleopaws""}, + ""e"": baseconv.base62.encode(expires_at), + }, + ""actor"", + ), + ) The resulting cookie will encode data that looks something like this: diff --git a/docs/spatialite.rst b/docs/spatialite.rst index d1b300b..556bad8 100644 --- a/docs/spatialite.rst +++ b/docs/spatialite.rst @@ -58,19 +58,22 @@ Here's a recipe for taking a table with existing latitude and longitude columns, .. code-block:: python import sqlite3 - conn = sqlite3.connect('museums.db') + + conn = sqlite3.connect(""museums.db"") # Lead the spatialite extension: conn.enable_load_extension(True) - conn.load_extension('/usr/local/lib/mod_spatialite.dylib') + conn.load_extension(""/usr/local/lib/mod_spatialite.dylib"") # Initialize spatial metadata for this database: - conn.execute('select InitSpatialMetadata(1)') + conn.execute(""select InitSpatialMetadata(1)"") # Add a geometry column called point_geom to our museums table: conn.execute(""SELECT AddGeometryColumn('museums', 'point_geom', 4326, 'POINT', 2);"") # Now update that geometry column with the lat/lon points - conn.execute(''' + conn.execute( + """""" UPDATE museums SET point_geom = GeomFromText('POINT('||""longitude""||' '||""latitude""||')',4326); - ''') + """""" + ) # Now add a spatial index to that column conn.execute('select CreateSpatialIndex(""museums"", ""point_geom"");') # If you don't commit your changes will not be persisted: @@ -186,13 +189,14 @@ Here's Python code to create a SQLite database, enable SpatiaLite, create a plac .. code-block:: python import sqlite3 - conn = sqlite3.connect('places.db') + + conn = sqlite3.connect(""places.db"") # Enable SpatialLite extension conn.enable_load_extension(True) - conn.load_extension('/usr/local/lib/mod_spatialite.dylib') + conn.load_extension(""/usr/local/lib/mod_spatialite.dylib"") # Create the masic countries table - conn.execute('select InitSpatialMetadata(1)') - conn.execute('create table places (id integer primary key, name text);') + conn.execute(""select InitSpatialMetadata(1)"") + conn.execute(""create table places (id integer primary key, name text);"") # Add a MULTIPOLYGON Geometry column conn.execute(""SELECT AddGeometryColumn('places', 'geom', 4326, 'MULTIPOLYGON', 2);"") # Add a spatial index against the new column @@ -201,13 +205,17 @@ Here's Python code to create a SQLite database, enable SpatiaLite, create a plac from shapely.geometry.multipolygon import MultiPolygon from shapely.geometry import shape import requests - geojson = requests.get('https://data.whosonfirst.org/404/227/475/404227475.geojson').json() + + geojson = requests.get( + ""https://data.whosonfirst.org/404/227/475/404227475.geojson"" + ).json() # Convert to ""Well Known Text"" format - wkt = shape(geojson['geometry']).wkt + wkt = shape(geojson[""geometry""]).wkt # Insert and commit the record - conn.execute(""INSERT INTO places (id, name, geom) VALUES(null, ?, GeomFromText(?, 4326))"", ( - ""Wales"", wkt - )) + conn.execute( + ""INSERT INTO places (id, name, geom) VALUES(null, ?, GeomFromText(?, 4326))"", + (""Wales"", wkt), + ) conn.commit() Querying polygons using within() diff --git a/docs/writing_plugins.rst b/docs/writing_plugins.rst index bd60a4b..5af01f6 100644 --- a/docs/writing_plugins.rst +++ b/docs/writing_plugins.rst @@ -18,9 +18,10 @@ The quickest way to start writing a plugin is to create a ``my_plugin.py`` file from datasette import hookimpl + @hookimpl def prepare_connection(conn): - conn.create_function('hello_world', 0, lambda: 'Hello world!') + conn.create_function(""hello_world"", 0, lambda: ""Hello world!"") If you save this in ``plugins/my_plugin.py`` you can then start Datasette like this:: @@ -60,22 +61,18 @@ The example consists of two files: a ``setup.py`` file that defines the plugin: from setuptools import setup - VERSION = '0.1' + VERSION = ""0.1"" setup( - name='datasette-plugin-demos', - description='Examples of plugins for Datasette', - author='Simon Willison', - url='https://github.com/simonw/datasette-plugin-demos', - license='Apache License, Version 2.0', + name=""datasette-plugin-demos"", + description=""Examples of plugins for Datasette"", + author=""Simon Willison"", + url=""https://github.com/simonw/datasette-plugin-demos"", + license=""Apache License, Version 2.0"", version=VERSION, - py_modules=['datasette_plugin_demos'], - entry_points={ - 'datasette': [ - 'plugin_demos = datasette_plugin_demos' - ] - }, - install_requires=['datasette'] + py_modules=[""datasette_plugin_demos""], + entry_points={""datasette"": [""plugin_demos = datasette_plugin_demos""]}, + install_requires=[""datasette""], ) And a Python module file, ``datasette_plugin_demos.py``, that implements the plugin: @@ -88,12 +85,12 @@ And a Python module file, ``datasette_plugin_demos.py``, that implements the plu @hookimpl def prepare_jinja2_environment(env): - env.filters['uppercase'] = lambda u: u.upper() + env.filters[""uppercase""] = lambda u: u.upper() @hookimpl def prepare_connection(conn): - conn.create_function('random_integer', 2, random.randint) + conn.create_function(""random_integer"", 2, random.randint) Having built a plugin in this way you can turn it into an installable package using the following command:: @@ -123,11 +120,13 @@ To bundle the static assets for a plugin in the package that you publish to PyPI .. code-block:: python - package_data={ - 'datasette_plugin_name': [ - 'static/plugin.js', - ], - }, + package_data = ( + { + ""datasette_plugin_name"": [ + ""static/plugin.js"", + ], + }, + ) Where ``datasette_plugin_name`` is the name of the plugin package (note that it uses underscores, not hyphens) and ``static/plugin.js`` is the path within that package to the static file. @@ -152,11 +151,13 @@ Templates should be bundled for distribution using the same ``package_data`` mec .. code-block:: python - package_data={ - 'datasette_plugin_name': [ - 'templates/my_template.html', - ], - }, + package_data = ( + { + ""datasette_plugin_name"": [ + ""templates/my_template.html"", + ], + }, + ) You can also use wildcards here such as ``templates/*.html``. See `datasette-edit-schema `__ for an example of this pattern. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1213683988,Code examples in the documentation should be formatted with Black, https://github.com/simonw/datasette/issues/1718#issuecomment-1107862882,https://api.github.com/repos/simonw/datasette/issues/1718,1107862882,IC_kwDOBm6k_c5CCKVi,9599,simonw,2022-04-24T15:23:56Z,2022-04-24T15:23:56Z,OWNER,"Found https://github.com/asottile/blacken-docs via - https://github.com/psf/black/issues/294","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1213683988,Code examples in the documentation should be formatted with Black, https://github.com/simonw/datasette/pull/1717#issuecomment-1107848097,https://api.github.com/repos/simonw/datasette/issues/1717,1107848097,IC_kwDOBm6k_c5CCGuh,9599,simonw,2022-04-24T14:02:37Z,2022-04-24T14:02:37Z,OWNER,"This is a neat feature, thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1213281044,Add timeout option to Cloudrun build, https://github.com/simonw/datasette/issues/1715#issuecomment-1106989581,https://api.github.com/repos/simonw/datasette/issues/1715,1106989581,IC_kwDOBm6k_c5B-1IN,9599,simonw,2022-04-22T23:03:29Z,2022-04-22T23:03:29Z,OWNER,I'm having second thoughts about injecting `request` - might be better to have the view function pull the relevant pieces out of the request before triggering the rest of the resolution.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1212823665,Refactor TableView to use asyncinject, https://github.com/simonw/datasette/issues/1715#issuecomment-1106947168,https://api.github.com/repos/simonw/datasette/issues/1715,1106947168,IC_kwDOBm6k_c5B-qxg,9599,simonw,2022-04-22T22:25:57Z,2022-04-22T22:26:06Z,OWNER,"```python async def database(request: Request, datasette: Datasette) -> Database: database_route = tilde_decode(request.url_vars[""database""]) try: return datasette.get_database(route=database_route) except KeyError: raise NotFound(""Database not found: {}"".format(database_route)) async def table_name(request: Request) -> str: return tilde_decode(request.url_vars[""table""]) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1212823665,Refactor TableView to use asyncinject, https://github.com/simonw/datasette/issues/1715#issuecomment-1106945876,https://api.github.com/repos/simonw/datasette/issues/1715,1106945876,IC_kwDOBm6k_c5B-qdU,9599,simonw,2022-04-22T22:24:29Z,2022-04-22T22:24:29Z,OWNER,"Looking at the start of `TableView.data()`: https://github.com/simonw/datasette/blob/d57c347f35bcd8cff15f913da851b4b8eb030867/datasette/views/table.py#L333-L346 I'm going to resolve `table_name` and `database` from the URL - `table_name` will be a string, `database` will be the DB object returned by `datasette.get_database()`. Then those can be passed in separately too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1212823665,Refactor TableView to use asyncinject, https://github.com/simonw/datasette/issues/1716#issuecomment-1106923258,https://api.github.com/repos/simonw/datasette/issues/1716,1106923258,IC_kwDOBm6k_c5B-k76,9599,simonw,2022-04-22T22:02:07Z,2022-04-22T22:02:07Z,OWNER,"https://github.com/simonw/datasette/blame/main/datasette/views/base.py ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1212838949,Configure git blame to ignore Black commit, https://github.com/simonw/datasette/issues/1715#issuecomment-1106908642,https://api.github.com/repos/simonw/datasette/issues/1715,1106908642,IC_kwDOBm6k_c5B-hXi,9599,simonw,2022-04-22T21:47:55Z,2022-04-22T21:47:55Z,OWNER,"I need a `asyncio.Registry` with functions registered to perform the role of the table view. Something like this perhaps: ```python def table_html_context(facet_results, query, datasette, rows): return {...} ``` That then gets called like this: ```python async def view(request): registry = Registry(facet_results, query, datasette, rows) context = await registry.resolve(table_html, request=request, datasette=datasette) return Reponse.html(await datasette.render(""table.html"", context) ``` It's also interesting to start thinking about this from a Python client library point of view. If I'm writing code outside of the HTTP request cycle, what would it look like? One thing I could do: break out is the code that turns a request into a list of pairs extracted from the request - this code here: https://github.com/simonw/datasette/blob/8338c66a57502ef27c3d7afb2527fbc0663b2570/datasette/views/table.py#L442-L449 I could turn that into a typed dependency injection function like this: ```python def filter_args(request: Request) -> List[Tuple[str, str]]: # Arguments that start with _ and don't contain a __ are # special - things like ?_search= - and should not be # treated as filters. filter_args = [] for key in request.args: if not (key.startswith(""_"") and ""__"" not in key): for v in request.args.getlist(key): filter_args.append((key, v)) return filter_args ``` Then I can either pass a `request` into a `.resolve()` call, or I can instead skip that function by passing: ```python output = registry.resolve(table_context, filter_args=[(""foo"", ""bar"")]) ``` I do need to think about where plugins get executed in all of this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1212823665,Refactor TableView to use asyncinject, https://github.com/simonw/datasette/issues/1101#issuecomment-1105615625,https://api.github.com/repos/simonw/datasette/issues/1101,1105615625,IC_kwDOBm6k_c5B5lsJ,9599,simonw,2022-04-21T18:31:41Z,2022-04-21T18:32:22Z,OWNER,"The `datasette-geojson` plugin is actually an interesting case here, because of the way it converts SpatiaLite geometries into GeoJSON: https://github.com/eyeseast/datasette-geojson/blob/602c4477dc7ddadb1c0a156cbcd2ef6688a5921d/datasette_geojson/__init__.py#L61-L66 ```python if isinstance(geometry, bytes): results = await db.execute( ""SELECT AsGeoJSON(:geometry)"", {""geometry"": geometry} ) return geojson.loads(results.single_value()) ``` That actually seems to work really well as-is, but it does worry me a bit that it ends up having to execute an extra `SELECT` query for every single returned row - especially in streaming mode where it might be asked to return 1m rows at once. My PostgreSQL/MySQL engineering brain says that this would be better handled by doing a chunk of these (maybe 100) at once, to avoid the per-query-overhead - but with SQLite that might not be necessary. At any rate, this is one of the reasons I'm interested in ""iterate over this sequence of chunks of 100 rows at a time"" as a potential option here. Of course, a better solution would be for `datasette-geojson` to have a way to influence the SQL query before it is executed, adding a `AsGeoJSON(geometry)` clause to it - so that's something I'm open to as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749283032,register_output_renderer() should support streaming data, https://github.com/simonw/datasette/issues/1101#issuecomment-1105608964,https://api.github.com/repos/simonw/datasette/issues/1101,1105608964,IC_kwDOBm6k_c5B5kEE,9599,simonw,2022-04-21T18:26:29Z,2022-04-21T18:26:29Z,OWNER,"I'm questioning if the mechanisms should be separate at all now - a single response rendering is really just a case of a streaming response that only pulls the first N records from the iterator. It probably needs to be an `async for` iterator, which I've not worked with much before. Good opportunity to learn. This actually gets a fair bit more complicated due to the work I'm doing right now to improve the default JSON API: - #1709 I want to do things like make faceting results optionally available to custom renderers - which is a separate concern from streaming rows. I'm going to poke around with a bunch of prototypes and see what sticks.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749283032,register_output_renderer() should support streaming data, https://github.com/simonw/datasette/issues/1101#issuecomment-1105571003,https://api.github.com/repos/simonw/datasette/issues/1101,1105571003,IC_kwDOBm6k_c5B5ay7,9599,simonw,2022-04-21T18:10:38Z,2022-04-21T18:10:46Z,OWNER,"Maybe the simplest design for this is to add an optional `can_stream` to the contract: ```python @hookimpl def register_output_renderer(datasette): return { ""extension"": ""tsv"", ""render"": render_tsv, ""can_render"": lambda: True, ""can_stream"": lambda: True } ``` When streaming, a new parameter could be passed to the render function - maybe `chunks` - which is an iterator/generator over a sequence of chunks of rows. Or it could use the existing `rows` parameter but treat that as an iterator?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749283032,register_output_renderer() should support streaming data, https://github.com/dogsheep/github-to-sqlite/issues/72#issuecomment-1105474232,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/72,1105474232,IC_kwDODFdgUs5B5DK4,9599,simonw,2022-04-21T17:02:15Z,2022-04-21T17:02:15Z,MEMBER,"That's interesting - yeah it looks like the number of pages can be derived from the `Link` header, which is enough information to show a progress bar, probably using Click just to avoid adding another dependency. https://docs.github.com/en/rest/guides/traversing-with-pagination","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1211283427,feature: display progress bar when downloading multi-page responses, https://github.com/simonw/sqlite-utils/issues/425#issuecomment-1101594549,https://api.github.com/repos/simonw/sqlite-utils/issues/425,1101594549,IC_kwDOCGYnMM5BqP-1,9599,simonw,2022-04-18T17:36:14Z,2022-04-18T17:36:14Z,OWNER,"Releated: - #408","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1203842656,`sqlite3.NotSupportedError`: deterministic=True requires SQLite 3.8.3 or higher, https://github.com/simonw/datasette/issues/1713#issuecomment-1098628334,https://api.github.com/repos/simonw/datasette/issues/1713,1098628334,IC_kwDOBm6k_c5Be7zu,9599,simonw,2022-04-14T01:43:00Z,2022-04-14T01:43:13Z,OWNER,"Current workaround for fast publishing to S3: datasette fixtures.db --get /fixtures/facetable.json | \ s3-credentials put-object my-bucket facetable.json -","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1203943272,Datasette feature for publishing snapshots of query results, https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098548931,https://api.github.com/repos/simonw/sqlite-utils/issues/421,1098548931,IC_kwDOCGYnMM5BeobD,9599,simonw,2022-04-13T22:41:59Z,2022-04-13T22:41:59Z,OWNER,"I'm going to close this ticket since it looks like this is a bug in the way the Dockerfile builds Python, but I'm going to ship a fix for that issue I found so the `LD_PRELOAD` workaround above should work OK with the next release of `sqlite-utils`. Thanks for the detailed bug report!","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180427792,"""Error: near ""("": syntax error"" when using sqlite-utils indexes CLI", https://github.com/simonw/sqlite-utils/issues/424#issuecomment-1098548090,https://api.github.com/repos/simonw/sqlite-utils/issues/424,1098548090,IC_kwDOCGYnMM5BeoN6,9599,simonw,2022-04-13T22:40:15Z,2022-04-13T22:40:15Z,OWNER,"New error: ```pycon >>> from sqlite_utils import Database >>> db = Database(memory=True) >>> db[""foo""].create({}) Traceback (most recent call last): File """", line 1, in File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1465, in create self.db.create_table( File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 885, in create_table sql = self.create_table_sql( File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 771, in create_table_sql assert columns, ""Tables must have at least one column"" AssertionError: Tables must have at least one column ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1200866134,Better error message if you try to create a table with no columns, https://github.com/simonw/sqlite-utils/issues/425#issuecomment-1098545390,https://api.github.com/repos/simonw/sqlite-utils/issues/425,1098545390,IC_kwDOCGYnMM5Benju,9599,simonw,2022-04-13T22:34:52Z,2022-04-13T22:34:52Z,OWNER,"That broke Python 3.7 because it doesn't support `deterministic=True` even being passed: > function takes at most 3 arguments (4 given)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1203842656,`sqlite3.NotSupportedError`: deterministic=True requires SQLite 3.8.3 or higher, https://github.com/simonw/sqlite-utils/issues/425#issuecomment-1098537000,https://api.github.com/repos/simonw/sqlite-utils/issues/425,1098537000,IC_kwDOCGYnMM5Belgo,9599,simonw,2022-04-13T22:18:22Z,2022-04-13T22:18:22Z,OWNER,"I figured out a workaround in https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098535531 The current `register(fn)` method looks like this: https://github.com/simonw/sqlite-utils/blob/95522ad919f96eb6cc8cd3cd30389b534680c717/sqlite_utils/db.py#L389-L403 This alternative implementation worked in the environment where that failed: ```python def register(fn): name = fn.__name__ arity = len(inspect.signature(fn).parameters) if not replace and (name, arity) in self._registered_functions: return fn kwargs = {} done = False if deterministic: # Try this, but fall back if sqlite3.NotSupportedError try: self.conn.create_function(name, arity, fn, **dict(kwargs, deterministic=True)) done = True except sqlite3.NotSupportedError: pass if not done: self.conn.create_function(name, arity, fn, **kwargs) self._registered_functions.add((name, arity)) return fn ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1203842656,`sqlite3.NotSupportedError`: deterministic=True requires SQLite 3.8.3 or higher, https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098535531,https://api.github.com/repos/simonw/sqlite-utils/issues/421,1098535531,IC_kwDOCGYnMM5BelJr,9599,simonw,2022-04-13T22:15:48Z,2022-04-13T22:15:48Z,OWNER,"Trying this alternative implementation of the `register()` method: ```python def register(fn): name = fn.__name__ arity = len(inspect.signature(fn).parameters) if not replace and (name, arity) in self._registered_functions: return fn kwargs = {} done = False if deterministic: # Try this, but fall back if sqlite3.NotSupportedError try: self.conn.create_function(name, arity, fn, **dict(kwargs, deterministic=True)) done = True except sqlite3.NotSupportedError: pass if not done: self.conn.create_function(name, arity, fn, **kwargs) self._registered_functions.add((name, arity)) return fn ``` With that fix, the following worked! ``` LD_PRELOAD=./build/sqlite-autoconf-3360000/.libs/libsqlite3.so sqlite-utils indexes /tmp/global.db --table table index_name seqno cid name desc coll key --------- -------------------------- ------- ----- ------- ------ ------ ----- countries idx_countries_country_name 0 1 country 0 BINARY 1 countries idx_countries_country_name 1 2 name 0 BINARY 1 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180427792,"""Error: near ""("": syntax error"" when using sqlite-utils indexes CLI", https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098532220,https://api.github.com/repos/simonw/sqlite-utils/issues/421,1098532220,IC_kwDOCGYnMM5BekV8,9599,simonw,2022-04-13T22:09:52Z,2022-04-13T22:09:52Z,OWNER,That error is weird - it's not supposed to happen according to this code here: https://github.com/simonw/sqlite-utils/blob/95522ad919f96eb6cc8cd3cd30389b534680c717/sqlite_utils/db.py#L389-L400,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180427792,"""Error: near ""("": syntax error"" when using sqlite-utils indexes CLI", https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098531354,https://api.github.com/repos/simonw/sqlite-utils/issues/421,1098531354,IC_kwDOCGYnMM5BekIa,9599,simonw,2022-04-13T22:08:20Z,2022-04-13T22:08:20Z,OWNER,"OK I figured out what's going on here. First I added an extra `print(sql)` statement to the `indexes` command to see what SQL it was running: ``` (app-root) sqlite-utils indexes global.db --table select sqlite_master.name as ""table"", indexes.name as index_name, xinfo.* from sqlite_master join pragma_index_list(sqlite_master.name) indexes join pragma_index_xinfo(index_name) xinfo where sqlite_master.type = 'table' and xinfo.key = 1 Error: near ""("": syntax error ``` This made me suspicious that the SQLite version being used here didn't support joining against the `pragma_index_list(...)` table-valued functions in that way. So I checked the version: ``` (app-root) sqlite3 SQLite version 3.36.0 2021-06-18 18:36:39 ``` That version should be fine - it's the one you compiled in the Dockerfile. Then I checked the version that `sqlite-utils` itself was using: ``` (app-root) sqlite-utils memory 'select sqlite_version()' [{""sqlite_version()"": ""3.7.17""}] ``` It's running SQLite 3.7.17! So the problem here is that the Python in that Docker image is running a very old version of SQLite. I tried using the trick in https://til.simonwillison.net/sqlite/ld-preload as a workaround, and it almost worked: ``` (app-root) python3 -c 'import sqlite3; print(sqlite3.connect("":memory"").execute(""select sqlite_version()"").fetchone())' ('3.7.17',) (app-root) LD_PRELOAD=./build/sqlite-autoconf-3360000/.libs/libsqlite3.so python3 -c 'import sqlite3; print(sqlite3.connect("":memory"").execute(""select sqlite_version()"").fetchone())' ('3.36.0',) ``` But when I try to run `sqlite-utils` like that I get an error: ``` (app-root) LD_PRELOAD=./build/sqlite-autoconf-3360000/.libs/libsqlite3.so sqlite-utils indexes /tmp/global.db ... File ""/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/cli.py"", line 1624, in query db.register_fts4_bm25() File ""/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py"", line 412, in register_fts4_bm25 self.register_function(rank_bm25, deterministic=True) File ""/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py"", line 408, in register_function register(fn) File ""/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py"", line 401, in register self.conn.create_function(name, arity, fn, **kwargs) sqlite3.NotSupportedError: deterministic=True requires SQLite 3.8.3 or higher ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180427792,"""Error: near ""("": syntax error"" when using sqlite-utils indexes CLI", https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098295517,https://api.github.com/repos/simonw/sqlite-utils/issues/421,1098295517,IC_kwDOCGYnMM5Bdqjd,9599,simonw,2022-04-13T17:16:20Z,2022-04-13T17:16:20Z,OWNER,"Aha! I was able to replicate the bug using your `Dockerfile` - thanks very much for providing that. ``` (app-root) sqlite-utils indexes global.db --table Error: near ""("": syntax error ``` (That wa sbefore I even ran the `extract` command.) To build your `Dockerfile` I copied it into an empty folder and ran the following: ``` wget https://www.sqlite.org/2021/sqlite-autoconf-3360000.tar.gz docker build . -t centos-sqlite-utils docker run -it centos-sqlite-utils /bin/bash ``` This gave me a shell in which I could replicate the bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180427792,"""Error: near ""("": syntax error"" when using sqlite-utils indexes CLI", https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098288158,https://api.github.com/repos/simonw/sqlite-utils/issues/421,1098288158,IC_kwDOCGYnMM5Bdowe,9599,simonw,2022-04-13T17:07:53Z,2022-04-13T17:07:53Z,OWNER,"I can't replicate the bug I'm afraid: ``` % wget ""https://github.com/wri/global-power-plant-database/blob/232a6666/output_database/global_power_plant_database.csv?raw=true"" ... 2022-04-13 10:06:29 (8.97 MB/s) - ‘global_power_plant_database.csv?raw=true’ saved [8856038/8856038] % sqlite-utils insert global.db power_plants \ 'global_power_plant_database.csv?raw=true' --csv [------------------------------------] 0% [###################################-] 99% 00:00:00% % sqlite-utils indexes global.db --table table index_name seqno cid name desc coll key ------- ------------ ------- ----- ------ ------ ------ ----- % sqlite-utils extract global.db power_plants country country_long \ --table countries \ --fk-column country_id \ --rename country_long name % sqlite-utils indexes global.db --table table index_name seqno cid name desc coll key --------- -------------------------- ------- ----- ------- ------ ------ ----- countries idx_countries_country_name 0 1 country 0 BINARY 1 countries idx_countries_country_name 1 2 name 0 BINARY 1 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180427792,"""Error: near ""("": syntax error"" when using sqlite-utils indexes CLI", https://github.com/simonw/datasette/issues/1712#issuecomment-1097115034,https://api.github.com/repos/simonw/datasette/issues/1712,1097115034,IC_kwDOBm6k_c5BZKWa,9599,simonw,2022-04-12T19:12:21Z,2022-04-12T19:12:21Z,OWNER,Got a TIL out of this too: https://til.simonwillison.net/spatialite/gunion-to-combine-geometries,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1202227104,"Make """" easier to read", https://github.com/simonw/datasette/issues/1712#issuecomment-1097076622,https://api.github.com/repos/simonw/datasette/issues/1712,1097076622,IC_kwDOBm6k_c5BZA-O,9599,simonw,2022-04-12T18:42:04Z,2022-04-12T18:42:04Z,OWNER,I'm not going to show the tooltip if the formatted number is in bytes.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1202227104,"Make """" easier to read", https://github.com/simonw/datasette/issues/1712#issuecomment-1097068474,https://api.github.com/repos/simonw/datasette/issues/1712,1097068474,IC_kwDOBm6k_c5BY--6,9599,simonw,2022-04-12T18:38:18Z,2022-04-12T18:38:18Z,OWNER," ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1202227104,"Make """" easier to read", https://github.com/simonw/datasette/issues/1708#issuecomment-1095687566,https://api.github.com/repos/simonw/datasette/issues/1708,1095687566,IC_kwDOBm6k_c5BTt2O,9599,simonw,2022-04-11T23:24:30Z,2022-04-11T23:24:30Z,OWNER,"## Redesigned template context **Warning:** if you use any custom templates with your Datasette instance they are likely to break when you upgrade to 1.0. The template context has been redesigned to be based on the documented JSON API. This means that the template context can be considered stable going forward, so any custom templates you implement should continue to work when you upgrade Datasette in the future.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1200649124,Datasette 1.0 alpha upcoming release notes, https://github.com/simonw/datasette/issues/1705#issuecomment-1095673947,https://api.github.com/repos/simonw/datasette/issues/1705,1095673947,IC_kwDOBm6k_c5BTqhb,9599,simonw,2022-04-11T23:03:49Z,2022-04-11T23:03:49Z,OWNER,I'll also encourage testing against both Datasette 0.x and Datasette 1.0 using a GitHub Actions matrix.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1197926598,How to upgrade your plugin for 1.0 documentation, https://github.com/simonw/datasette/issues/1710#issuecomment-1095673670,https://api.github.com/repos/simonw/datasette/issues/1710,1095673670,IC_kwDOBm6k_c5BTqdG,9599,simonw,2022-04-11T23:03:25Z,2022-04-11T23:03:25Z,OWNER,"Dupe of: - #1705","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1200649889,Guide for plugin authors to upgrade their plugins for 1.0, https://github.com/simonw/datasette/issues/1709#issuecomment-1095671940,https://api.github.com/repos/simonw/datasette/issues/1709,1095671940,IC_kwDOBm6k_c5BTqCE,9599,simonw,2022-04-11T23:00:39Z,2022-04-11T23:01:41Z,OWNER,"- #262 - #782 - #1509","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1200649502,Redesigned JSON API with ?_extra= parameters, https://github.com/simonw/datasette/issues/1711#issuecomment-1095672127,https://api.github.com/repos/simonw/datasette/issues/1711,1095672127,IC_kwDOBm6k_c5BTqE_,9599,simonw,2022-04-11T23:00:58Z,2022-04-11T23:00:58Z,OWNER,- #1510,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1200650491,Template context powered entirely by the JSON API format, https://github.com/simonw/datasette/issues/1707#issuecomment-1095277937,https://api.github.com/repos/simonw/datasette/issues/1707,1095277937,IC_kwDOBm6k_c5BSJ1x,9599,simonw,2022-04-11T16:32:31Z,2022-04-11T16:33:00Z,OWNER,"That's a really interesting idea! That page is one of the least developed at the moment. There's plenty of room for it to grow new useful features. I like this suggestion because it feels like a good opportunity to introduce some unobtrusive JavaScript. Could use a details/summary element that uses `fetch()` to load in the extra data for example. Could even do something with the `` Web Component here... https://github.com/simonw/datasette-table","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1200224939,[feature] expanded detail page, https://github.com/simonw/datasette/issues/1706#issuecomment-1094152642,https://api.github.com/repos/simonw/datasette/issues/1706,1094152642,IC_kwDOBm6k_c5BN3HC,9599,simonw,2022-04-10T01:11:54Z,2022-04-10T01:11:54Z,OWNER,"This relates to this much larger vision: - #417 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1198822563,"[feature] immutable mode for a directory, not just individual sqlite file", https://github.com/simonw/datasette/issues/1706#issuecomment-1094152173,https://api.github.com/repos/simonw/datasette/issues/1706,1094152173,IC_kwDOBm6k_c5BN2_t,9599,simonw,2022-04-10T01:08:50Z,2022-04-10T01:08:50Z,OWNER,This is a good idea - it matches the way `datasette .` works for mutable database files already.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1198822563,"[feature] immutable mode for a directory, not just individual sqlite file", https://github.com/simonw/datasette/pull/1693#issuecomment-1093454899,https://api.github.com/repos/simonw/datasette/issues/1693,1093454899,IC_kwDOBm6k_c5BLMwz,9599,simonw,2022-04-08T23:07:04Z,2022-04-08T23:07:04Z,OWNER,"Tests failed here due to this issue: - https://github.com/psf/black/pull/2987 A future Black release should fix that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1184850337,Bump black from 22.1.0 to 22.3.0, https://github.com/simonw/datasette/issues/1699#issuecomment-1092361727,https://api.github.com/repos/simonw/datasette/issues/1699,1092361727,IC_kwDOBm6k_c5BHB3_,9599,simonw,2022-04-08T01:47:43Z,2022-04-08T01:47:43Z,OWNER,"A render mode for that plugin hook that writes to a stream is exactly what I have in mind: - #1062 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1193090967,Proposal: datasette query, https://github.com/simonw/datasette/issues/1699#issuecomment-1092321966,https://api.github.com/repos/simonw/datasette/issues/1699,1092321966,IC_kwDOBm6k_c5BG4Ku,9599,simonw,2022-04-08T00:20:32Z,2022-04-08T00:20:56Z,OWNER,"If we do this I'm keen to have it be more than just an alternative to the existing `sqlite-utils` command - especially since if I add `sqlite-utils` as a dependency of Datasette in the future that command will be installed as part of `pip install datasette` anyway. My best thought on how to differentiate them so far is plugins: if Datasette plugins that provide alternative outputs - like `.geojson` and `.yml` and suchlike - also work for the `datasette query` command that would make a lot of sense to me. One way that could work: a `--fmt geojson` option to this command which uses the plugin that was registered for the specified extension.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1193090967,Proposal: datasette query, https://github.com/simonw/datasette/issues/1698#issuecomment-1086784547,https://api.github.com/repos/simonw/datasette/issues/1698,1086784547,IC_kwDOBm6k_c5AxwQj,9599,simonw,2022-04-03T06:10:24Z,2022-04-03T06:10:24Z,OWNER,Warning added here: https://docs.datasette.io/en/latest/publish.html#publishing-to-google-cloud-run,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1190828163,Add a warning about bots and Cloud Run, https://github.com/simonw/datasette/issues/1697#issuecomment-1085323192,https://api.github.com/repos/simonw/datasette/issues/1697,1085323192,IC_kwDOBm6k_c5AsLe4,9599,simonw,2022-04-01T02:01:51Z,2022-04-01T02:01:51Z,OWNER,"Huh, turns out `Request.fake()` wasn't yet documented.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1189113609,"`Request.fake(..., url_vars={})`", https://github.com/simonw/datasette/issues/1696#issuecomment-1083351437,https://api.github.com/repos/simonw/datasette/issues/1696,1083351437,IC_kwDOBm6k_c5AkqGN,9599,simonw,2022-03-30T16:20:49Z,2022-03-30T16:21:02Z,OWNER,"Maybe like this: ```html

283 rows where dcode = 3 (Human Related: Other)

```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1186696202,Show foreign key label when filtering, https://github.com/simonw/datasette/issues/1692#issuecomment-1082663746,https://api.github.com/repos/simonw/datasette/issues/1692,1082663746,IC_kwDOBm6k_c5AiCNC,9599,simonw,2022-03-30T06:14:39Z,2022-03-30T06:14:51Z,OWNER,"I like your design, though I think it should be `""nomodule"": True` for consistency with the other options. I think `""async"": True` is worth supporting too.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1182227211,[plugins][feature request]: Support additional script tag attributes when loading custom JS, https://github.com/simonw/datasette/issues/1692#issuecomment-1082661795,https://api.github.com/repos/simonw/datasette/issues/1692,1082661795,IC_kwDOBm6k_c5AiBuj,9599,simonw,2022-03-30T06:11:41Z,2022-03-30T06:11:41Z,OWNER,This is a good idea.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1182227211,[plugins][feature request]: Support additional script tag attributes when loading custom JS, https://github.com/simonw/datasette/issues/1695#issuecomment-1082617386,https://api.github.com/repos/simonw/datasette/issues/1695,1082617386,IC_kwDOBm6k_c5Ah24q,9599,simonw,2022-03-30T04:46:18Z,2022-03-30T04:46:18Z,OWNER,"` selected = (column_qs, str(row[""value""])) in qs_pairs` is wrong.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1185868354,Option to un-filter facet not shown for `?col__exact=value`, https://github.com/simonw/datasette/issues/1695#issuecomment-1082617241,https://api.github.com/repos/simonw/datasette/issues/1695,1082617241,IC_kwDOBm6k_c5Ah22Z,9599,simonw,2022-03-30T04:45:55Z,2022-03-30T04:45:55Z,OWNER,"Relevant template: https://github.com/simonw/datasette/blob/e73fa72917ca28c152208d62d07a490c81cadf52/datasette/templates/table.html#L168-L172 Populated from here: https://github.com/simonw/datasette/blob/c496f2b663ff0cef908ffaaa68b8cb63111fb5f2/datasette/facets.py#L246-L253","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1185868354,Option to un-filter facet not shown for `?col__exact=value`, https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1081047053,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1081047053,IC_kwDOCGYnMM5Ab3gN,9599,simonw,2022-03-28T19:22:37Z,2022-03-28T19:22:37Z,OWNER,Wrote about this in my weeknotes: https://simonwillison.net/2022/Mar/28/datasette-auth0/#new-features-as-documentation,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,Document how to use a `--convert` function that runs initialization code first, https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1080141111,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1080141111,IC_kwDOCGYnMM5AYaU3,9599,simonw,2022-03-28T03:25:57Z,2022-03-28T03:54:37Z,OWNER,"So now this should solve your problem: ``` echo '[{""name"": ""notaword""}, {""name"": ""word""}] ' | python3 -m sqlite_utils insert listings.db listings - --convert ' import enchant d = enchant.Dict(""en_US"") def convert(row): global d row[""is_dictionary_word""] = d.check(row[""name""]) ' ```","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,Document how to use a `--convert` function that runs initialization code first, https://github.com/simonw/datasette/issues/1690#issuecomment-1079788375,https://api.github.com/repos/simonw/datasette/issues/1690,1079788375,IC_kwDOBm6k_c5AXENX,9599,simonw,2022-03-26T22:43:00Z,2022-03-26T22:43:00Z,OWNER,Then I can update this section of the documentation which currently recommends the above pattern: https://docs.datasette.io/en/stable/authentication.html#the-ds-actor-cookie,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1182141761,"Idea: `datasette.set_actor_cookie(response, actor)`", https://github.com/simonw/datasette/issues/1690#issuecomment-1079788346,https://api.github.com/repos/simonw/datasette/issues/1690,1079788346,IC_kwDOBm6k_c5AXEM6,9599,simonw,2022-03-26T22:42:40Z,2022-03-26T22:42:40Z,OWNER,"I don't want to do a `response.set_actor_cookie()` method because I like `Response` not to carry too many Datasette-specific features. So `datasette.set_actor_cookie(response, actor, expire_after=None)` would be a better place for this I think.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1182141761,"Idea: `datasette.set_actor_cookie(response, actor)`", https://github.com/simonw/datasette/issues/1689#issuecomment-1079779040,https://api.github.com/repos/simonw/datasette/issues/1689,1079779040,IC_kwDOBm6k_c5AXB7g,9599,simonw,2022-03-26T21:35:57Z,2022-03-26T21:35:57Z,OWNER,Fixed: https://docs.datasette.io/en/latest/internals.html#add-message-request-message-type-datasette-info,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1182065616,datasette.add_message() documentation is incorrect, https://github.com/simonw/datasette/issues/1688#issuecomment-1079582485,https://api.github.com/repos/simonw/datasette/issues/1688,1079582485,IC_kwDOBm6k_c5AWR8V,9599,simonw,2022-03-26T03:15:34Z,2022-03-26T03:15:34Z,OWNER,"Yup, you're right in what you figured out here: stand-alone plugins can't currently package static assets other then using the static folder. The `datasette-plugin` cookiecutter template should make creating a Python package pretty easy though: https://github.com/simonw/datasette-plugin You can run that yourself, or you can run it using this GitHub template repository: https://github.com/simonw/datasette-plugin-template-repository ","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1181432624,[plugins][documentation] Is it possible to serve per-plugin static folders when writing one-off (single file) plugins?, https://github.com/simonw/sqlite-utils/issues/417#issuecomment-1079441621,https://api.github.com/repos/simonw/sqlite-utils/issues/417,1079441621,IC_kwDOCGYnMM5AVvjV,9599,simonw,2022-03-25T21:18:37Z,2022-03-25T21:18:37Z,OWNER,Updated documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#inserting-newline-delimited-json,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175744654,insert fails on JSONL with whitespace, https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1079407962,https://api.github.com/repos/simonw/sqlite-utils/issues/421,1079407962,IC_kwDOCGYnMM5AVnVa,9599,simonw,2022-03-25T20:25:10Z,2022-03-25T20:25:18Z,OWNER,"Can you share either your whole `global.db` table or a shrunk down example that illustrates the bug? My hunch is that you may have a table or column with a name that triggers the error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180427792,"""Error: near ""("": syntax error"" when using sqlite-utils indexes CLI", https://github.com/simonw/sqlite-utils/issues/422#issuecomment-1079406708,https://api.github.com/repos/simonw/sqlite-utils/issues/422,1079406708,IC_kwDOCGYnMM5AVnB0,9599,simonw,2022-03-25T20:23:21Z,2022-03-25T20:23:21Z,OWNER,"Fixing this would require a bump to 4.0 because it would break existing code. The alternative would be to introduce a new `ignore_nulls=True` parameter which users can change to `ignore_nulls=False`. Or come up with better wording for that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1181236173,Reconsider not running convert functions against null values, https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1079404281,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1079404281,IC_kwDOCGYnMM5AVmb5,9599,simonw,2022-03-25T20:19:50Z,2022-03-25T20:19:50Z,OWNER,Now documented here: https://sqlite-utils.datasette.io/en/latest/cli.html#using-a-convert-function-to-execute-initialization,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,Document how to use a `--convert` function that runs initialization code first, https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1079384771,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1079384771,IC_kwDOCGYnMM5AVhrD,9599,simonw,2022-03-25T19:51:34Z,2022-03-25T19:53:01Z,OWNER,"This works: ``` % sqlite-utils insert dogs.db dogs dogs.json --convert ' import random print(""seeding"") random.seed(10) print(random.random()) def convert(row): global random print(row) row[""random_score""] = random.random() ' seeding 0.5714025946899135 {'id': 1, 'name': 'Cleo'} {'id': 2, 'name': 'Pancakes'} {'id': 3, 'name': 'New dog'} (sqlite-utils) sqlite-utils % sqlite-utils rows dogs.db dogs [{""id"": 1, ""name"": ""Cleo"", ""random_score"": 0.4288890546751146}, {""id"": 2, ""name"": ""Pancakes"", ""random_score"": 0.5780913011344704}, {""id"": 3, ""name"": ""New dog"", ""random_score"": 0.20609823213950174}] ``` Having to use `global random` inside the function is frustrating but apparently necessary. https://stackoverflow.com/a/56552138/6083","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,Document how to use a `--convert` function that runs initialization code first, https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1079376283,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1079376283,IC_kwDOCGYnMM5AVfmb,9599,simonw,2022-03-25T19:39:30Z,2022-03-25T19:43:35Z,OWNER,"Actually this doesn't work as I thought. This demo shows that the initialization code is run once per item, not a single time at the start of the run: ``` % sqlite-utils insert dogs.db dogs dogs.json --convert ' import random print(""seeding"") random.seed(10) print(random.random()) def convert(row): print(row) row[""random_score""] = random.random() ' seeding 0.5714025946899135 seeding 0.5714025946899135 seeding 0.5714025946899135 seeding 0.5714025946899135 ``` Also that `print(row)` line is not being printed anywhere that gets to the console for some reason. ... my mistake, that happened because I changed this line in order to try to get local imports to work: ```python try: exec(code, globals, locals) return globals[""convert""] except (AttributeError, SyntaxError, NameError, KeyError, TypeError): ``` It should be `locals[""convert""]`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,Document how to use a `--convert` function that runs initialization code first, https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1079243535,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1079243535,IC_kwDOCGYnMM5AU_MP,9599,simonw,2022-03-25T17:25:12Z,2022-03-25T17:25:12Z,OWNER,"That documentation is split across a few places. This is the only bit that talks about `def convert()` pattern right now: - https://sqlite-utils.datasette.io/en/stable/cli.html#converting-data-in-columns But that's for `sqlite-utils convert` - the documentation for `sqlite-utils insert --convert` at https://sqlite-utils.datasette.io/en/stable/cli.html#applying-conversions-while-inserting-data doesn't mention it. Since both `sqlite-utils convert` and `sqlite-utils insert --convert` apply the same rules to the code, they should link to a shared explanation in the documentation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,Document how to use a `--convert` function that runs initialization code first, https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1078343231,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1078343231,IC_kwDOCGYnMM5ARjY_,9599,simonw,2022-03-24T21:16:10Z,2022-03-24T21:17:20Z,OWNER,"Aha! This may be possible already: https://github.com/simonw/sqlite-utils/blob/396f80fcc60da8dd844577114f7920830a2e5403/sqlite_utils/utils.py#L311-L316 And yes, this does indeed work - you can do something like this: ``` echo '{""name"": ""harry""}' | sqlite-utils insert db.db people - --convert ' import time # Simulate something expensive time.sleep(1) def convert(row): row[""upper""] = row[""name""].upper() ' ``` And after running that: ``` sqlite-utils dump db.db BEGIN TRANSACTION; CREATE TABLE [people] ( [name] TEXT, [upper] TEXT ); INSERT INTO ""people"" VALUES('harry','HARRY'); COMMIT; ``` So this is a documentation issue - there's a trick for it but I didn't know what the trick was!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,Document how to use a `--convert` function that runs initialization code first, https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1078328774,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1078328774,IC_kwDOCGYnMM5ARf3G,9599,simonw,2022-03-24T21:12:33Z,2022-03-24T21:12:33Z,OWNER,"Here's how the `_compile_code()` mechanism works at the moment: https://github.com/simonw/sqlite-utils/blob/396f80fcc60da8dd844577114f7920830a2e5403/sqlite_utils/utils.py#L308-L342 At the end it does this: ```python return locals[""fn""] ``` So it's already building and then returning a function. The question is if there's a sensible way to allow people to further customize that function by executing some code first, in a way that's easy to explain.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,Document how to use a `--convert` function that runs initialization code first, https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1078322301,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1078322301,IC_kwDOCGYnMM5AReR9,9599,simonw,2022-03-24T21:10:52Z,2022-03-24T21:10:52Z,OWNER,"I can think of three ways forward: - Figure out a pattern that gets that local file import workaround to work - Add another option such as `--convert-init` that lets you pass code that will be executed once at the start - Come up with a pattern where the `--convert` code can run some initialization code and then return a function which will be called against each value I quite like the idea of that third option - I'm going to prototype it and see if I can work something out.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,Document how to use a `--convert` function that runs initialization code first, https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1078315922,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1078315922,IC_kwDOCGYnMM5ARcuS,9599,simonw,2022-03-24T21:09:27Z,2022-03-24T21:09:27Z,OWNER,"Yeah, this is WAY harder than it should be. There's a clumsy workaround you could use which looks something like this: create a file `my_enchant.py` containing: ```python import enchant d = enchant.Dict(""en_US"") def check(word): return d.check(word) ``` Then run `sqlite-utils` like this: ``` PYTHONPATH=. cat items.json | jq '.data' | sqlite-utils insert listings.db listings - --convert 'my_enchant.check(value)' --import my_enchant ``` Except I tried that and it doesn't work! I don't know the right pattern for getting `--import` to work with modules in the same directory. So yeah, this is definitely a big feature gap.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,Document how to use a `--convert` function that runs initialization code first, https://github.com/simonw/datasette/issues/1682#issuecomment-1076696791,https://api.github.com/repos/simonw/datasette/issues/1682,1076696791,IC_kwDOBm6k_c5ALRbX,9599,simonw,2022-03-23T18:45:49Z,2022-03-23T18:45:49Z,OWNER,"The problem is here in `QueryView`: https://github.com/simonw/datasette/blob/d7c793d7998388d915f8d270079c68a77a785051/datasette/views/database.py#L206-L238 It should be resolving `database` based on the route path, as seen in other methods like this one: https://github.com/simonw/datasette/blob/d7c793d7998388d915f8d270079c68a77a785051/datasette/views/table.py#L270-L279 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178521513,SQL queries against databases with different routes are broken, https://github.com/simonw/datasette/issues/1670#issuecomment-1076683297,https://api.github.com/repos/simonw/datasette/issues/1670,1076683297,IC_kwDOBm6k_c5ALOIh,9599,simonw,2022-03-23T18:32:32Z,2022-03-23T18:32:32Z,OWNER,Added this to news on https://datasette.io/ https://github.com/simonw/datasette.io/commit/fd3ec57cdd5b935f75cbf52a86b3aabf2c97d217,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174423568,Ship Datasette 0.61, https://github.com/simonw/datasette/issues/1670#issuecomment-1076666293,https://api.github.com/repos/simonw/datasette/issues/1670,1076666293,IC_kwDOBm6k_c5ALJ-1,9599,simonw,2022-03-23T18:16:29Z,2022-03-23T18:16:29Z,OWNER,https://docs.datasette.io/en/stable/changelog.html#v0-61,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174423568,Ship Datasette 0.61, https://github.com/simonw/datasette/issues/1670#issuecomment-1076665837,https://api.github.com/repos/simonw/datasette/issues/1670,1076665837,IC_kwDOBm6k_c5ALJ3t,9599,simonw,2022-03-23T18:16:01Z,2022-03-23T18:16:01Z,OWNER,"https://github.com/simonw/datasette/releases/tag/0.61 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174423568,Ship Datasette 0.61, https://github.com/simonw/datasette/issues/1670#issuecomment-1076652046,https://api.github.com/repos/simonw/datasette/issues/1670,1076652046,IC_kwDOBm6k_c5ALGgO,9599,simonw,2022-03-23T18:02:30Z,2022-03-23T18:02:30Z,OWNER,"Two new things to add to the release notes from https://github.com/simonw/datasette/compare/0.61a0...main - https://github.com/simonw/datasette/issues/1678 - https://github.com/simonw/datasette/issues/1675 (now also a documented API)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174423568,Ship Datasette 0.61, https://github.com/simonw/datasette/issues/1670#issuecomment-1076647495,https://api.github.com/repos/simonw/datasette/issues/1670,1076647495,IC_kwDOBm6k_c5ALFZH,9599,simonw,2022-03-23T17:58:16Z,2022-03-23T17:58:16Z,OWNER,"I think the release notes are fine, but they need an opening paragraph highlighting the changes that are most likely to break backwards compatibility.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174423568,Ship Datasette 0.61, https://github.com/simonw/datasette/pull/1574#issuecomment-1076645636,https://api.github.com/repos/simonw/datasette/issues/1574,1076645636,IC_kwDOBm6k_c5ALE8E,9599,simonw,2022-03-23T17:56:35Z,2022-03-23T17:56:35Z,OWNER,I'd actually like to switch to slim as the default - I think Datasette should ship the smallest possible container that can still support extra packages being installed using `apt-get install`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1084193403,introduce new option for datasette package to use a slim base image, https://github.com/simonw/datasette/pull/1665#issuecomment-1076644362,https://api.github.com/repos/simonw/datasette/issues/1665,1076644362,IC_kwDOBm6k_c5ALEoK,9599,simonw,2022-03-23T17:55:39Z,2022-03-23T17:55:39Z,OWNER,Thanks for the PR - I spotted an error about this and went through and fixed this in all of my repos the other day: https://github.com/search?o=desc&q=user%3Asimonw+google-github-actions%2Fsetup-gcloud%40v0&s=committer-date&type=Commits,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173828092,Pin setup-gcloud to v0 instead of master, https://github.com/simonw/datasette/issues/1670#issuecomment-1076638278,https://api.github.com/repos/simonw/datasette/issues/1670,1076638278,IC_kwDOBm6k_c5ALDJG,9599,simonw,2022-03-23T17:50:55Z,2022-03-23T17:50:55Z,OWNER,"Release notes are mostly written for the alpha, just need to clean them up a bit https://github.com/simonw/datasette/blob/c4c9dbd0386e46d2bf199f0ed34e4895c98cb78c/docs/changelog.rst#061a0-2022-03-19","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174423568,Ship Datasette 0.61, https://github.com/simonw/datasette/issues/1681#issuecomment-1075438684,https://api.github.com/repos/simonw/datasette/issues/1681,1075438684,IC_kwDOBm6k_c5AGeRc,9599,simonw,2022-03-22T17:45:50Z,2022-03-22T17:49:09Z,OWNER,"I would expect this to break against SQL views that include calculated columns though - something like this: ```sql create view this_will_break as select pk + 1 as pk_plus_one, 0.5 as score from searchable; ``` Confirmed: the filter interface for that view plain doesn't work for any comparison against that table - except for `score > 0` since `0` is converted to an integer. `0.1` breaks though because it doesn't get converted as it doesn't match `.isdigit()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1177101697,Potential bug in numeric handling where_clause for filters, https://github.com/simonw/datasette/issues/1681#issuecomment-1075437598,https://api.github.com/repos/simonw/datasette/issues/1681,1075437598,IC_kwDOBm6k_c5AGeAe,9599,simonw,2022-03-22T17:44:42Z,2022-03-22T17:45:04Z,OWNER,"My hunch is that this mechanism doesn't actually do anything useful at all, because of the type conversion that automatically happens for data from tables based on the column type affinities, see: - #1671 So either remove the `self.numeric` type conversion bit entirely, or prove that it is necessary and upgrade it to be able to handle floating point values too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1177101697,Potential bug in numeric handling where_clause for filters, https://github.com/simonw/datasette/issues/1671#issuecomment-1075432283,https://api.github.com/repos/simonw/datasette/issues/1671,1075432283,IC_kwDOBm6k_c5AGctb,9599,simonw,2022-03-22T17:39:04Z,2022-03-22T17:43:12Z,OWNER,"Note that Datasette does already have special logic to convert parameters to integers for numeric comparisons like `>`: https://github.com/simonw/datasette/blob/c4c9dbd0386e46d2bf199f0ed34e4895c98cb78c/datasette/filters.py#L203-L212 Though... it looks like there's a bug in that? It doesn't account for `float` values - `""3.5"".isdigit()` return `False` - probably for the best, because `int(3.5)` would break that value anyway.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174655187,Filters fail to work correctly against calculated numeric columns returned by SQL views because type affinity rules do not apply, https://github.com/simonw/datasette/issues/1671#issuecomment-1075435185,https://api.github.com/repos/simonw/datasette/issues/1671,1075435185,IC_kwDOBm6k_c5AGdax,9599,simonw,2022-03-22T17:42:09Z,2022-03-22T17:42:09Z,OWNER,"Also made me realize that this query: ```sql select * from sortable where sortable > :p0 ``` Only works here thanks to the column affinity thing kicking in too: https://latest.datasette.io/fixtures?sql=select+*+from+sortable+where+sortable+%3E+%3Ap0&p0=70","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174655187,Filters fail to work correctly against calculated numeric columns returned by SQL views because type affinity rules do not apply, https://github.com/simonw/datasette/issues/1671#issuecomment-1075428030,https://api.github.com/repos/simonw/datasette/issues/1671,1075428030,IC_kwDOBm6k_c5AGbq-,9599,simonw,2022-03-22T17:34:30Z,2022-03-22T17:34:30Z,OWNER,"No, I think I need to use `cast` - I can't think of any way to ask SQLite ""for this query, what types are the columns that will come back from it?"" Even the details from the `explain` trick explored in #1293 don't seem to come back with column type information: https://latest.datasette.io/fixtures?sql=explain+select+pk%2C+text1%2C+text2%2C+[name+with+.+and+spaces]+from+searchable_view+where+%22pk%22+%3D+%3Ap0&p0=1","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174655187,Filters fail to work correctly against calculated numeric columns returned by SQL views because type affinity rules do not apply, https://github.com/simonw/datasette/issues/1671#issuecomment-1075425513,https://api.github.com/repos/simonw/datasette/issues/1671,1075425513,IC_kwDOBm6k_c5AGbDp,9599,simonw,2022-03-22T17:31:53Z,2022-03-22T17:31:53Z,OWNER,"The alternative to using `cast` here would be for Datasette to convert the `""1""` to a `1` in Python code before passing it as a param. This feels a bit neater to me, but I still then need to solve the problem of how to identify the ""type"" of a column that I want to use in a query.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174655187,Filters fail to work correctly against calculated numeric columns returned by SQL views because type affinity rules do not apply, https://github.com/simonw/datasette/issues/339#issuecomment-1074479932,https://api.github.com/repos/simonw/datasette/issues/339,1074479932,IC_kwDOBm6k_c5AC0M8,9599,simonw,2022-03-21T22:22:34Z,2022-03-21T22:22:34Z,OWNER,Closing this as obsolete since Datasette no longer uses Sanic.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",340396247,Expose SANIC_RESPONSE_TIMEOUT config option in a sensible way, https://github.com/simonw/datasette/issues/276#issuecomment-1074479768,https://api.github.com/repos/simonw/datasette/issues/276,1074479768,IC_kwDOBm6k_c5AC0KY,9599,simonw,2022-03-21T22:22:20Z,2022-03-21T22:22:20Z,OWNER,"I'm closing this issue because this is now solved by a number of neat plugins: - https://datasette.io/plugins/datasette-geojson-map shows the geometry from SpatiaLite columns on a map - https://datasette.io/plugins/datasette-leaflet-geojson can be used to display inline maps next to each column","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",324835838,Handle spatialite geometry columns better, https://github.com/simonw/datasette/issues/1671#issuecomment-1074478299,https://api.github.com/repos/simonw/datasette/issues/1671,1074478299,IC_kwDOBm6k_c5ACzzb,9599,simonw,2022-03-21T22:20:26Z,2022-03-21T22:20:26Z,OWNER,"Thinking about options for fixing this... The following query works fine: ```sql select * from test_view where cast(has_expired as text) = '1' ``` I don't want to start using this for every query, because one of the goals of Datasette is to help people who are learning SQL: - #1613 If someone clicks on ""View and edit SQL"" from a filtered table page I don't want them to have to wonder why that `cast` is there. But... for querying views, the `cast` turns out to be necessary. So one fix would be to get the SQL generating logic to use casts like this any time it is operating against a view. An even better fix would be to detect which columns in a view come from a table and which ones might not, and only use casts for the columns that aren't definitely from a table. The trick I was exploring here might be able to help with that: - #1293 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174655187,Filters fail to work correctly against calculated numeric columns returned by SQL views because type affinity rules do not apply, https://github.com/simonw/datasette/issues/1671#issuecomment-1074470568,https://api.github.com/repos/simonw/datasette/issues/1671,1074470568,IC_kwDOBm6k_c5ACx6o,9599,simonw,2022-03-21T22:11:14Z,2022-03-21T22:12:49Z,OWNER,"I wonder if this will be a problem with generated columns, or with SQLite strict tables? My hunch is that strict tables will continue to work without any changes, because https://www.sqlite.org/stricttables.html says nothing about their impact on comparison operations. I should test this to make absolutely sure though. Generated columns have a type, so my hunch is they will continue to work fine too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174655187,Filters fail to work correctly against calculated numeric columns returned by SQL views because type affinity rules do not apply, https://github.com/simonw/datasette/issues/1671#issuecomment-1074468450,https://api.github.com/repos/simonw/datasette/issues/1671,1074468450,IC_kwDOBm6k_c5ACxZi,9599,simonw,2022-03-21T22:08:35Z,2022-03-21T22:10:00Z,OWNER,"Relevant section of the SQLite documentation: [3.2. Affinity Of Expressions](https://www.sqlite.org/datatype3.html#affinity_of_expressions): > When an expression is a simple reference to a column of a real table (not a [VIEW](https://www.sqlite.org/lang_createview.html) or subquery) then the expression has the same affinity as the table column. In your example, `has_expired` is no longer a simple reference to a column of a real table, hence the bug. Then [4.2. Type Conversions Prior To Comparison](https://www.sqlite.org/datatype3.html#type_conversions_prior_to_comparison) fills in the rest: > SQLite may attempt to convert values between the storage classes INTEGER, REAL, and/or TEXT before performing a comparison. Whether or not any conversions are attempted before the comparison takes place depends on the type affinity of the operands. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174655187,Filters fail to work correctly against calculated numeric columns returned by SQL views because type affinity rules do not apply, https://github.com/simonw/datasette/issues/1671#issuecomment-1074465536,https://api.github.com/repos/simonw/datasette/issues/1671,1074465536,IC_kwDOBm6k_c5ACwsA,9599,simonw,2022-03-21T22:04:31Z,2022-03-21T22:04:31Z,OWNER,"Oh this is fascinating! I replicated the bug (thanks for the steps to reproduce) and it looks like this is down to the following: Against views, `where has_expired = 1` returns different results from `where has_expired = '1'` This doesn't happen against tables because of SQLite's [type affinity](https://www.sqlite.org/datatype3.html#type_affinity) mechanism, which handles the type conversion automatically.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174655187,Filters fail to work correctly against calculated numeric columns returned by SQL views because type affinity rules do not apply, https://github.com/simonw/datasette/issues/1679#issuecomment-1074459746,https://api.github.com/repos/simonw/datasette/issues/1679,1074459746,IC_kwDOBm6k_c5ACvRi,9599,simonw,2022-03-21T21:55:45Z,2022-03-21T21:55:45Z,OWNER,I'm going to change the original logic to set n=1 for times that are `<= 20ms` - and update the comments to make it more obvious what is happening.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,Research: how much overhead does the n=1 time limit have?, https://github.com/simonw/datasette/issues/1679#issuecomment-1074458506,https://api.github.com/repos/simonw/datasette/issues/1679,1074458506,IC_kwDOBm6k_c5ACu-K,9599,simonw,2022-03-21T21:53:47Z,2022-03-21T21:53:47Z,OWNER,"Oh interesting, it turns out there is ONE place in the code that sets the `ms` to less than 20 - this test fixture: https://github.com/simonw/datasette/blob/4e47a2d894b96854348343374c8e97c9d7055cf6/tests/fixtures.py#L224-L226","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,Research: how much overhead does the n=1 time limit have?, https://github.com/simonw/datasette/issues/1679#issuecomment-1074454687,https://api.github.com/repos/simonw/datasette/issues/1679,1074454687,IC_kwDOBm6k_c5ACuCf,9599,simonw,2022-03-21T21:48:02Z,2022-03-21T21:48:02Z,OWNER,"Here's another microbenchmark that measures how many nanoseconds it takes to run 1,000 vmops: ```python import sqlite3 import time db = sqlite3.connect("":memory:"") i = 0 out = [] def count(): global i i += 1000 out.append(((i, time.perf_counter_ns()))) db.set_progress_handler(count, 1000) print(""Start:"", time.perf_counter_ns()) all = db.execute("""""" with recursive counter(x) as ( select 0 union select x + 1 from counter ) select * from counter limit 10000; """""").fetchall() print(""End:"", time.perf_counter_ns()) print() print(""So how long does it take to execute 1000 ops?"") prev_time_ns = None for i, time_ns in out: if prev_time_ns is not None: print(time_ns - prev_time_ns, ""ns"") prev_time_ns = time_ns ``` Running it: ``` % python nanobench.py Start: 330877620374821 End: 330877632515822 So how long does it take to execute 1000 ops? 47290 ns 49573 ns 48226 ns 45674 ns 53238 ns 47313 ns 52346 ns 48689 ns 47092 ns 87596 ns 69999 ns 52522 ns 52809 ns 53259 ns 52478 ns 53478 ns 65812 ns ``` 87596ns is 0.087596ms - so even a measure rate of every 1000 ops is easily finely grained enough to capture differences of less than 0.1ms. If anything I could bump that default 1000 up - and I can definitely eliminate the `if ms < 50` branch entirely.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,Research: how much overhead does the n=1 time limit have?, https://github.com/simonw/datasette/issues/1679#issuecomment-1074446576,https://api.github.com/repos/simonw/datasette/issues/1679,1074446576,IC_kwDOBm6k_c5ACsDw,9599,simonw,2022-03-21T21:38:27Z,2022-03-21T21:38:27Z,OWNER,"OK here's a microbenchmark script: ```python import sqlite3 import timeit db = sqlite3.connect("":memory:"") db_with_progress_handler_1 = sqlite3.connect("":memory:"") db_with_progress_handler_1000 = sqlite3.connect("":memory:"") db_with_progress_handler_1.set_progress_handler(lambda: None, 1) db_with_progress_handler_1000.set_progress_handler(lambda: None, 1000) def execute_query(db): cursor = db.execute("""""" with recursive counter(x) as ( select 0 union select x + 1 from counter ) select * from counter limit 10000; """""") list(cursor.fetchall()) print(""Without progress_handler"") print(timeit.timeit(lambda: execute_query(db), number=100)) print(""progress_handler every 1000 ops"") print(timeit.timeit(lambda: execute_query(db_with_progress_handler_1000), number=100)) print(""progress_handler every 1 op"") print(timeit.timeit(lambda: execute_query(db_with_progress_handler_1), number=100)) ``` Results: ``` % python3 bench.py Without progress_handler 0.8789225700311363 progress_handler every 1000 ops 0.8829826560104266 progress_handler every 1 op 2.8892734259716235 ``` So running every 1000 ops makes almost no difference at all, but running every single op is a 3.2x performance degradation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,Research: how much overhead does the n=1 time limit have?, https://github.com/simonw/datasette/issues/1679#issuecomment-1074439309,https://api.github.com/repos/simonw/datasette/issues/1679,1074439309,IC_kwDOBm6k_c5ACqSN,9599,simonw,2022-03-21T21:28:58Z,2022-03-21T21:28:58Z,OWNER,"David Raymond solved it there: https://sqlite.org/forum/forumpost/330c8532d8a88bcd > Don't forget to step through the results. All .execute() has done is prepared it. > > db.execute(query).fetchall() Sure enough, adding that gets the VM steps number up to 190,007 which is close enough that I'm happy.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,Research: how much overhead does the n=1 time limit have?, https://github.com/simonw/datasette/issues/1676#issuecomment-1074378472,https://api.github.com/repos/simonw/datasette/issues/1676,1074378472,IC_kwDOBm6k_c5ACbbo,9599,simonw,2022-03-21T20:18:10Z,2022-03-21T20:18:10Z,OWNER,Maybe there is a better name for this method that helps emphasize its cascading nature.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175690070,"Reconsider ensure_permissions() logic, can it be less confusing?", https://github.com/simonw/datasette/issues/1679#issuecomment-1074347023,https://api.github.com/repos/simonw/datasette/issues/1679,1074347023,IC_kwDOBm6k_c5ACTwP,9599,simonw,2022-03-21T19:48:59Z,2022-03-21T19:48:59Z,OWNER,Posed a question about that here: https://sqlite.org/forum/forumpost/de9ff10fa7,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,Research: how much overhead does the n=1 time limit have?, https://github.com/simonw/datasette/issues/1679#issuecomment-1074341924,https://api.github.com/repos/simonw/datasette/issues/1679,1074341924,IC_kwDOBm6k_c5ACSgk,9599,simonw,2022-03-21T19:42:08Z,2022-03-21T19:42:08Z,OWNER,"Here's the Python-C implementation of `set_progress_handler`: https://github.com/python/cpython/blob/4674fd4e938eb4a29ccd5b12c15455bd2a41c335/Modules/_sqlite/connection.c#L1177-L1201 It calls `sqlite3_progress_handler(self->db, n, progress_callback, ctx);` https://www.sqlite.org/c3ref/progress_handler.html says: > The parameter N is the approximate number of [virtual machine instructions](https://www.sqlite.org/opcode.html) that are evaluated between successive invocations of the callback X So maybe VM-steps and virtual machine instructions are different things?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,Research: how much overhead does the n=1 time limit have?, https://github.com/simonw/datasette/issues/1679#issuecomment-1074337997,https://api.github.com/repos/simonw/datasette/issues/1679,1074337997,IC_kwDOBm6k_c5ACRjN,9599,simonw,2022-03-21T19:37:08Z,2022-03-21T19:37:08Z,OWNER,"This is weird: ```python import sqlite3 db = sqlite3.connect("":memory:"") i = 0 def count(): global i i += 1 db.set_progress_handler(count, 1) db.execute("""""" with recursive counter(x) as ( select 0 union select x + 1 from counter ) select * from counter limit 10000; """""") print(i) ``` Outputs `24`. But if you try the same thing in the SQLite console: ``` sqlite> .stats vmstep sqlite> with recursive counter(x) as ( ...> select 0 ...> union ...> select x + 1 from counter ...> ) ...> select * from counter limit 10000; ... VM-steps: 200007 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,Research: how much overhead does the n=1 time limit have?, https://github.com/simonw/datasette/issues/1679#issuecomment-1074332718,https://api.github.com/repos/simonw/datasette/issues/1679,1074332718,IC_kwDOBm6k_c5ACQQu,9599,simonw,2022-03-21T19:31:10Z,2022-03-21T19:31:10Z,OWNER,How long does it take for SQLite to execute 1000 opcodes anyway?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,Research: how much overhead does the n=1 time limit have?, https://github.com/simonw/datasette/issues/1679#issuecomment-1074332325,https://api.github.com/repos/simonw/datasette/issues/1679,1074332325,IC_kwDOBm6k_c5ACQKl,9599,simonw,2022-03-21T19:30:44Z,2022-03-21T19:30:44Z,OWNER,So it looks like even for facet suggestion `n=1000` always - it's never reduced to `n=1`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,Research: how much overhead does the n=1 time limit have?, https://github.com/simonw/datasette/issues/1679#issuecomment-1074331743,https://api.github.com/repos/simonw/datasette/issues/1679,1074331743,IC_kwDOBm6k_c5ACQBf,9599,simonw,2022-03-21T19:30:05Z,2022-03-21T19:30:05Z,OWNER,"https://github.com/simonw/datasette/blob/1a7750eb29fd15dd2eea3b9f6e33028ce441b143/datasette/app.py#L118-L122 sets it to 50ms for facet suggestion but that's not going to pass `ms < 50`: ```python Setting( ""facet_suggest_time_limit_ms"", 50, ""Time limit for calculating a suggested facet"", ), ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,Research: how much overhead does the n=1 time limit have?, https://github.com/simonw/datasette/issues/1660#issuecomment-1074321862,https://api.github.com/repos/simonw/datasette/issues/1660,1074321862,IC_kwDOBm6k_c5ACNnG,9599,simonw,2022-03-21T19:19:01Z,2022-03-21T19:19:01Z,OWNER,I've simplified this a ton now. I'm going to keep working on this in the long-term but I think this issue can be closed.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170144879,Refactor and simplify Datasette routing and views, https://github.com/simonw/datasette/issues/1678#issuecomment-1074302559,https://api.github.com/repos/simonw/datasette/issues/1678,1074302559,IC_kwDOBm6k_c5ACI5f,9599,simonw,2022-03-21T19:04:03Z,2022-03-21T19:04:03Z,OWNER,Documentation: https://docs.datasette.io/en/latest/internals.html#await-check-visibility-actor-action-resource-none,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175715988,Make `check_visibility()` a documented API, https://github.com/simonw/datasette/issues/1660#issuecomment-1074287177,https://api.github.com/repos/simonw/datasette/issues/1660,1074287177,IC_kwDOBm6k_c5ACFJJ,9599,simonw,2022-03-21T18:51:42Z,2022-03-21T18:51:42Z,OWNER,`BaseView` is looking a LOT slimmer now that I've moved all of the permissions stuff out of it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170144879,Refactor and simplify Datasette routing and views, https://github.com/simonw/sqlite-utils/issues/417#issuecomment-1074243540,https://api.github.com/repos/simonw/sqlite-utils/issues/417,1074243540,IC_kwDOCGYnMM5AB6fU,9599,simonw,2022-03-21T18:08:03Z,2022-03-21T18:08:03Z,OWNER,"I've not really thought about standards as much here as I should. It looks like there are two competing specs for newline-delimited JSON! http://ndjson.org/ is the one I've been using in `sqlite-utils` - and https://github.com/ndjson/ndjson-spec#31-serialization says: > The JSON texts MUST NOT contain newlines or carriage returns. https://jsonlines.org/ is the other one. It is slightly less clear, but it does say this: > 2. Each Line is a Valid JSON Value > > The most common values will be objects or arrays, but any JSON value is permitted. My interpretation of both of these is that newlines in the middle of a JSON object shouldn't be allowed. So what's `jq` doing here? It looks to me like that `jq` format is its own thing - it's not actually compatible with either of those two loose specs described above. The `jq` docs seem to call this ""whitespace-separated JSON"": https://stedolan.github.io/jq/manual/v1.6/#Invokingjq The thing I like about newline-delimited JSON is that it's really trivial to parse - loop through each line, run it through `json.loads()` and that's it. No need to try and unwrap JSON objects that might span multiple lines. Unless someone has written a robust Python implementation of a `jq`-compatible whitespace-separated JSON parser, I'm inclined to leave this as is. I'd be fine adding some documentation that helps point people towards `jq -c` though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175744654,insert fails on JSONL with whitespace, https://github.com/simonw/datasette/issues/1677#issuecomment-1074184240,https://api.github.com/repos/simonw/datasette/issues/1677,1074184240,IC_kwDOBm6k_c5ABsAw,9599,simonw,2022-03-21T17:20:17Z,2022-03-21T17:20:17Z,OWNER,"https://github.com/simonw/datasette/blob/e627510b760198ccedba9e5af47a771e847785c9/datasette/views/base.py#L69-L77 This is weirdly different from how `check_permissions()` used to work, in that it doesn't differentiate between `None` and `False`. https://github.com/simonw/datasette/blob/4a4164b81191dec35e423486a208b05a9edc65e4/datasette/views/base.py#L79-L103","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175694248,Remove `check_permission()` from `BaseView`, https://github.com/simonw/datasette/issues/1676#issuecomment-1074180312,https://api.github.com/repos/simonw/datasette/issues/1676,1074180312,IC_kwDOBm6k_c5ABrDY,9599,simonw,2022-03-21T17:16:45Z,2022-03-21T17:16:45Z,OWNER,"When looking at this code earlier I assumed that the following would check each permission in turn and fail if any of them failed: ```python await self.ds.ensure_permissions( request.actor, [ (""view-table"", (database, table)), (""view-database"", database), ""view-instance"", ] ) ``` But it's not quite that simple: if any of them fail, it fails... but if an earlier one returns `True` the whole stack passes even if there would have been a failure later on! If that is indeed the right abstraction, I need to work to make the documentation as clear as possible.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175690070,"Reconsider ensure_permissions() logic, can it be less confusing?", https://github.com/simonw/datasette/issues/1676#issuecomment-1074178865,https://api.github.com/repos/simonw/datasette/issues/1676,1074178865,IC_kwDOBm6k_c5ABqsx,9599,simonw,2022-03-21T17:15:27Z,2022-03-21T17:15:27Z,OWNER,This method here: https://github.com/simonw/datasette/blob/e627510b760198ccedba9e5af47a771e847785c9/datasette/app.py#L632-L664,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175690070,"Reconsider ensure_permissions() logic, can it be less confusing?", https://github.com/simonw/datasette/issues/1675#issuecomment-1074177827,https://api.github.com/repos/simonw/datasette/issues/1675,1074177827,IC_kwDOBm6k_c5ABqcj,9599,simonw,2022-03-21T17:14:31Z,2022-03-21T17:14:31Z,OWNER,"Updated documentation: https://github.com/simonw/datasette/blob/e627510b760198ccedba9e5af47a771e847785c9/docs/internals.rst#await-ensure_permissionsactor-permissions > This method allows multiple permissions to be checked at onced. It raises a `datasette.Forbidden` exception if any of the checks are denied before one of them is explicitly granted. > > This is useful when you need to check multiple permissions at once. For example, an actor should be able to view a table if either one of the following checks returns `True` or not a single one of them returns `False`: That's pretty hard to understand! I'm going to open a separate issue to reconsider if this is a useful enough abstraction given how confusing it is.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175648453,Extract out `check_permissions()` from `BaseView, https://github.com/simonw/datasette/issues/1675#issuecomment-1074161523,https://api.github.com/repos/simonw/datasette/issues/1675,1074161523,IC_kwDOBm6k_c5ABmdz,9599,simonw,2022-03-21T16:59:55Z,2022-03-21T17:00:03Z,OWNER,Also calling that function `permissions_allowed()` is confusing because there is a plugin hook with a similar name already: https://docs.datasette.io/en/stable/plugin_hooks.html#permission-allowed-datasette-actor-action-resource,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175648453,Extract out `check_permissions()` from `BaseView, https://github.com/simonw/datasette/issues/1675#issuecomment-1074158890,https://api.github.com/repos/simonw/datasette/issues/1675,1074158890,IC_kwDOBm6k_c5ABl0q,9599,simonw,2022-03-21T16:57:15Z,2022-03-21T16:57:15Z,OWNER,"Idea: `ds.permission_allowed()` continues to just return `True` or `False`. A new `ds.ensure_permissions(...)` method is added which raises a `Forbidden` exception if a check fails (hence the different name)`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175648453,Extract out `check_permissions()` from `BaseView, https://github.com/simonw/datasette/issues/1675#issuecomment-1074156779,https://api.github.com/repos/simonw/datasette/issues/1675,1074156779,IC_kwDOBm6k_c5ABlTr,9599,simonw,2022-03-21T16:55:08Z,2022-03-21T16:56:02Z,OWNER,"One benefit of the current design of `check_permissions` that raises an exception is that the exception includes information on WHICH of the permission checks failed. Returning just `True` or `False` loses that information. I could return an object which evaluates to `False` but also carries extra information? Bit weird, I've never seen anything like that in other Python code.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175648453,Extract out `check_permissions()` from `BaseView, https://github.com/simonw/datasette/issues/1675#issuecomment-1074143209,https://api.github.com/repos/simonw/datasette/issues/1675,1074143209,IC_kwDOBm6k_c5ABh_p,9599,simonw,2022-03-21T16:46:05Z,2022-03-21T16:46:05Z,OWNER,"The other difference though is that `ds.permission_allowed(...)` works against an actor, while `check_permission()` works against a request (though just to access `request.actor`).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175648453,Extract out `check_permissions()` from `BaseView, https://github.com/simonw/datasette/issues/1675#issuecomment-1074142617,https://api.github.com/repos/simonw/datasette/issues/1675,1074142617,IC_kwDOBm6k_c5ABh2Z,9599,simonw,2022-03-21T16:45:27Z,2022-03-21T16:45:27Z,OWNER,"Though at that point `check_permission` is such a light wrapper around `self.ds.permission_allowed()` that there's little point in it existing at all. So maybe `check_permisions()` becomes `ds.permissions_allowed()`. `permission_allowed()` v.s. `permissions_allowed()` is a bit of a subtle naming difference, but I think it works.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175648453,Extract out `check_permissions()` from `BaseView, https://github.com/simonw/datasette/issues/1675#issuecomment-1074141457,https://api.github.com/repos/simonw/datasette/issues/1675,1074141457,IC_kwDOBm6k_c5ABhkR,9599,simonw,2022-03-21T16:44:09Z,2022-03-21T16:44:09Z,OWNER,"A slightly odd thing about these methods is that they either fail silently or they raise a `Forbidden` exception. Maybe they should instead return `True` or `False` and the calling code could decide if it wants to raise the exception? That would make them more usable and a little less surprising.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175648453,Extract out `check_permissions()` from `BaseView, https://github.com/simonw/datasette/issues/1660#issuecomment-1074136176,https://api.github.com/repos/simonw/datasette/issues/1660,1074136176,IC_kwDOBm6k_c5ABgRw,9599,simonw,2022-03-21T16:38:46Z,2022-03-21T16:38:46Z,OWNER,"I'm going to refactor this stuff out and document it so it can be easily used by plugins: https://github.com/simonw/datasette/blob/4a4164b81191dec35e423486a208b05a9edc65e4/datasette/views/base.py#L69-L103","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170144879,Refactor and simplify Datasette routing and views, https://github.com/simonw/datasette/issues/526#issuecomment-1074019047,https://api.github.com/repos/simonw/datasette/issues/526,1074019047,IC_kwDOBm6k_c5ABDrn,9599,simonw,2022-03-21T15:09:56Z,2022-03-21T15:09:56Z,OWNER,I should research how much overhead creating a new connection costs - it may be that an easy way to solve this is to create A dedicated connection for the query and then close that connection at the end.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459882902,Stream all results for arbitrary SQL and canned queries, https://github.com/simonw/datasette/issues/1177#issuecomment-1074017633,https://api.github.com/repos/simonw/datasette/issues/1177,1074017633,IC_kwDOBm6k_c5ABDVh,9599,simonw,2022-03-21T15:08:51Z,2022-03-21T15:08:51Z,OWNER,"Related: - #1062 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780153562,Ability to stream all rows as newline-delimited JSON, https://github.com/simonw/sqlite-utils/issues/415#issuecomment-1073468996,https://api.github.com/repos/simonw/sqlite-utils/issues/415,1073468996,IC_kwDOCGYnMM4_-9ZE,9599,simonw,2022-03-21T04:14:42Z,2022-03-21T04:14:42Z,OWNER,"I can fix this like so: ``` % sqlite-utils convert demo.db demo foo '{""foo"": ""bar""}' --multi --dry-run abc --- becomes: {""foo"": ""bar""} Would affect 1 row ``` Diff is this: ```diff diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py index 0cf0468..b2a0440 100644 --- a/sqlite_utils/cli.py +++ b/sqlite_utils/cli.py @@ -2676,7 +2676,10 @@ def convert( raise click.ClickException(str(e)) if dry_run: # Pull first 20 values for first column and preview them - db.conn.create_function(""preview_transform"", 1, lambda v: fn(v) if v else v) + preview = lambda v: fn(v) if v else v + if multi: + preview = lambda v: json.dumps(fn(v), default=repr) if v else v + db.conn.create_function(""preview_transform"", 1, preview) sql = """""" select [{column}] as value, ```","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1171599874,Convert with `--multi` and `--dry-run` flag does not work, https://github.com/simonw/sqlite-utils/issues/415#issuecomment-1073463375,https://api.github.com/repos/simonw/sqlite-utils/issues/415,1073463375,IC_kwDOCGYnMM4_-8BP,9599,simonw,2022-03-21T04:02:36Z,2022-03-21T04:02:36Z,OWNER,Thanks for the really clear steps to reproduce!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1171599874,Convert with `--multi` and `--dry-run` flag does not work, https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073456222,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073456222,IC_kwDOCGYnMM4_-6Re,9599,simonw,2022-03-21T03:45:52Z,2022-03-21T03:45:52Z,OWNER,Needs tests and documentation.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,Options for how `r.parsedate()` should handle invalid dates, https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073456155,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073456155,IC_kwDOCGYnMM4_-6Qb,9599,simonw,2022-03-21T03:45:37Z,2022-03-21T03:45:37Z,OWNER,"Prototype: ```diff diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py index 8255b56..0a3693e 100644 --- a/sqlite_utils/cli.py +++ b/sqlite_utils/cli.py @@ -2583,7 +2583,11 @@ def _generate_convert_help(): """""" ).strip() recipe_names = [ - n for n in dir(recipes) if not n.startswith(""_"") and n not in (""json"", ""parser"") + n + for n in dir(recipes) + if not n.startswith(""_"") + and n not in (""json"", ""parser"") + and callable(getattr(recipes, n)) ] for name in recipe_names: fn = getattr(recipes, name) diff --git a/sqlite_utils/recipes.py b/sqlite_utils/recipes.py index 6918661..569c30d 100644 --- a/sqlite_utils/recipes.py +++ b/sqlite_utils/recipes.py @@ -1,17 +1,38 @@ from dateutil import parser import json +IGNORE = object() +SET_NULL = object() -def parsedate(value, dayfirst=False, yearfirst=False): + +def parsedate(value, dayfirst=False, yearfirst=False, errors=None): ""Parse a date and convert it to ISO date format: yyyy-mm-dd"" - return ( - parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).date().isoformat() - ) + try: + return ( + parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst) + .date() + .isoformat() + ) + except parser.ParserError: + if errors is IGNORE: + return value + elif errors is SET_NULL: + return None + else: + raise -def parsedatetime(value, dayfirst=False, yearfirst=False): +def parsedatetime(value, dayfirst=False, yearfirst=False, errors=None): ""Parse a datetime and convert it to ISO datetime format: yyyy-mm-ddTHH:MM:SS"" - return parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).isoformat() + try: + return parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).isoformat() + except parser.ParserError: + if errors is IGNORE: + return value + elif errors is SET_NULL: + return None + else: + raise def jsonsplit(value, delimiter="","", type=str): ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,Options for how `r.parsedate()` should handle invalid dates, https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073455905,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073455905,IC_kwDOCGYnMM4_-6Mh,9599,simonw,2022-03-21T03:44:47Z,2022-03-21T03:45:00Z,OWNER,"This is quite nice: ``` % sqlite-utils convert test-dates.db dates date ""r.parsedate(value, errors=r.IGNORE)"" [####################################] 100% % sqlite-utils rows test-dates.db dates [{""id"": 1, ""date"": ""2016-03-15""}, {""id"": 2, ""date"": ""2016-03-16""}, {""id"": 3, ""date"": ""2016-03-17""}, {""id"": 4, ""date"": ""2016-03-18""}, {""id"": 5, ""date"": ""2016-03-19""}, {""id"": 6, ""date"": ""2016-03-20""}, {""id"": 7, ""date"": ""2016-03-21""}, {""id"": 8, ""date"": ""2016-03-22""}, {""id"": 9, ""date"": ""2016-03-23""}, {""id"": 10, ""date"": ""//""}, {""id"": 11, ""date"": ""2016-03-25""}, {""id"": 12, ""date"": ""2016-03-26""}, {""id"": 13, ""date"": ""2016-03-27""}, {""id"": 14, ""date"": ""2016-03-28""}, {""id"": 15, ""date"": ""2016-03-29""}, {""id"": 16, ""date"": ""2016-03-30""}, {""id"": 17, ""date"": ""2016-03-31""}, {""id"": 18, ""date"": ""2016-04-01""}] % sqlite-utils convert test-dates.db dates date ""r.parsedate(value, errors=r.SET_NULL)"" [####################################] 100% % sqlite-utils rows test-dates.db dates [{""id"": 1, ""date"": ""2016-03-15""}, {""id"": 2, ""date"": ""2016-03-16""}, {""id"": 3, ""date"": ""2016-03-17""}, {""id"": 4, ""date"": ""2016-03-18""}, {""id"": 5, ""date"": ""2016-03-19""}, {""id"": 6, ""date"": ""2016-03-20""}, {""id"": 7, ""date"": ""2016-03-21""}, {""id"": 8, ""date"": ""2016-03-22""}, {""id"": 9, ""date"": ""2016-03-23""}, {""id"": 10, ""date"": null}, {""id"": 11, ""date"": ""2016-03-25""}, {""id"": 12, ""date"": ""2016-03-26""}, {""id"": 13, ""date"": ""2016-03-27""}, {""id"": 14, ""date"": ""2016-03-28""}, {""id"": 15, ""date"": ""2016-03-29""}, {""id"": 16, ""date"": ""2016-03-30""}, {""id"": 17, ""date"": ""2016-03-31""}, {""id"": 18, ""date"": ""2016-04-01""}] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,Options for how `r.parsedate()` should handle invalid dates, https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073453370,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073453370,IC_kwDOCGYnMM4_-5k6,9599,simonw,2022-03-21T03:41:06Z,2022-03-21T03:41:06Z,OWNER,I'm going to try the `errors=r.IGNORE` option and see what that looks like once implemented.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,Options for how `r.parsedate()` should handle invalid dates, https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073453230,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073453230,IC_kwDOCGYnMM4_-5iu,9599,simonw,2022-03-21T03:40:37Z,2022-03-21T03:40:37Z,OWNER,"I think the options here should be: - On error, raise an exception and revert the transaction (the current default) - On error, leave the value as-is - On error, set the value to `None` These need to be indicated by parameters to the `r.parsedate()` function. Some design options: - `ignore=True` to ignore errors - but how does it know if it should leave the value or set it to `None`? This is similar to other `ignore=True` parameters elsewhere in the Python API. - `errors=""ignore""`, `errors=""set-null""` - I don't like magic string values very much, but this is similar to Python's `str.encode(errors=)` mechanism - `errors=r.IGNORE` - using constants, which at least avoids magic strings. The other one could be `errors=r.SET_NULL` - `error=lambda v: None` or `error=lambda v: v` - this is a bit confusing though, introducing another callback that gets to have a go at converting the error if the first callback failed? And what happens if that lambda itself raises an error?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,Options for how `r.parsedate()` should handle invalid dates, https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073451659,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073451659,IC_kwDOCGYnMM4_-5KL,9599,simonw,2022-03-21T03:35:01Z,2022-03-21T03:35:01Z,OWNER,"I confirmed that if it fails for any value ALL values are left alone, since it runs in a transaction. Here's the code that does that: https://github.com/simonw/sqlite-utils/blob/433813612ff9b4b501739fd7543bef0040dd51fe/sqlite_utils/db.py#L2523-L2526","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,Options for how `r.parsedate()` should handle invalid dates, https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073450588,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073450588,IC_kwDOCGYnMM4_-45c,9599,simonw,2022-03-21T03:32:58Z,2022-03-21T03:32:58Z,OWNER,"Then I ran this to convert `2016-03-27` etc to `2016/03/27` so I could see which ones were later converted: sqlite-utils convert test-dates.db dates date 'value.replace(""-"", ""/"")' ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,Options for how `r.parsedate()` should handle invalid dates, https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073448904,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073448904,IC_kwDOCGYnMM4_-4fI,9599,simonw,2022-03-21T03:28:12Z,2022-03-21T03:30:37Z,OWNER,"Generating a test database using a pattern from https://www.geekytidbits.com/date-range-table-sqlite/ ``` sqlite-utils create-database test-dates.db sqlite-utils create-table test-dates.db dates id integer date text --pk id sqlite-utils test-dates.db ""WITH RECURSIVE cnt(x) AS ( SELECT 0 UNION ALL SELECT x+1 FROM cnt LIMIT (SELECT ((julianday('2016-04-01') - julianday('2016-03-15'))) + 1) ) insert into dates (date) select date(julianday('2016-03-15'), '+' || x || ' days') as date FROM cnt;"" ``` After running that: ``` % sqlite-utils rows test-dates.db dates [{""id"": 1, ""date"": ""2016-03-15""}, {""id"": 2, ""date"": ""2016-03-16""}, {""id"": 3, ""date"": ""2016-03-17""}, {""id"": 4, ""date"": ""2016-03-18""}, {""id"": 5, ""date"": ""2016-03-19""}, {""id"": 6, ""date"": ""2016-03-20""}, {""id"": 7, ""date"": ""2016-03-21""}, {""id"": 8, ""date"": ""2016-03-22""}, {""id"": 9, ""date"": ""2016-03-23""}, {""id"": 10, ""date"": ""2016-03-24""}, {""id"": 11, ""date"": ""2016-03-25""}, {""id"": 12, ""date"": ""2016-03-26""}, {""id"": 13, ""date"": ""2016-03-27""}, {""id"": 14, ""date"": ""2016-03-28""}, {""id"": 15, ""date"": ""2016-03-29""}, {""id"": 16, ""date"": ""2016-03-30""}, {""id"": 17, ""date"": ""2016-03-31""}, {""id"": 18, ""date"": ""2016-04-01""}] ``` Then to make one of them invalid: sqlite-utils test-dates.db ""update dates set date = '//' where id = 10""","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,Options for how `r.parsedate()` should handle invalid dates, https://github.com/simonw/datasette/issues/1510#issuecomment-1073366630,https://api.github.com/repos/simonw/datasette/issues/1510,1073366630,IC_kwDOBm6k_c4_-kZm,9599,simonw,2022-03-20T22:59:33Z,2022-03-20T22:59:33Z,OWNER,"I really like the idea of making this effectively the same thing as the fully documented, stable JSON API that comes as part of 1.0. If you want to know what will be available to your templates, consult the API documentation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1054244712,Datasette 1.0 documented template context (maybe via API docs), https://github.com/simonw/datasette/issues/1674#issuecomment-1073366436,https://api.github.com/repos/simonw/datasette/issues/1674,1073366436,IC_kwDOBm6k_c4_-kWk,9599,simonw,2022-03-20T22:58:40Z,2022-03-20T22:58:40Z,OWNER,"This will probably happen as part of turning this into an officially documented API that serves the template context for the homepage: - #1510","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174717287,Tweak design of /.json, https://github.com/simonw/datasette/issues/1355#issuecomment-1073362979,https://api.github.com/repos/simonw/datasette/issues/1355,1073362979,IC_kwDOBm6k_c4_-jgj,9599,simonw,2022-03-20T22:38:53Z,2022-03-20T22:38:53Z,OWNER,"Built a research prototype: ```diff diff --git a/datasette/app.py b/datasette/app.py index 5c8101a..5cd3e63 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -1,6 +1,7 @@ import asyncio import asgi_csrf import collections +import contextlib import datetime import functools import glob @@ -1490,3 +1491,11 @@ class DatasetteClient: return await client.request( method, self._fix(path, avoid_path_rewrites), **kwargs ) + + @contextlib.asynccontextmanager + async def stream(self, method, path, **kwargs): + async with httpx.AsyncClient(app=self.app) as client: + print(""async with as client"") + async with client.stream(method, self._fix(path), **kwargs) as response: + print(""async with client.stream about to yield response"") + yield response diff --git a/datasette/cli.py b/datasette/cli.py index 3c6e1b2..3025ead 100644 --- a/datasette/cli.py +++ b/datasette/cli.py @@ -585,11 +585,19 @@ def serve( asyncio.get_event_loop().run_until_complete(check_databases(ds)) if get: - client = TestClient(ds) - response = client.get(get) - click.echo(response.text) - exit_code = 0 if response.status == 200 else 1 - sys.exit(exit_code) + + async def _run_get(): + print(""_run_get"") + async with ds.client.stream(""GET"", get) as response: + print(""Got response:"", response) + async for chunk in response.aiter_bytes(chunk_size=1024): + print("" chunk"") + sys.stdout.buffer.write(chunk) + sys.stdout.buffer.flush() + exit_code = 0 if response.status_code == 200 else 1 + sys.exit(exit_code) + + asyncio.get_event_loop().run_until_complete(_run_get()) return # Start the server ``` But for some reason it didn't appear to stream out the response - it would print this out: ``` % datasette covid.db --get '/covid/ny_times_us_counties.csv?_size=10&_stream=on' _run_get async with as client ``` And then hang. I would expect it to start printing out chunks of CSV data here, but instead it looks like it waited for everything to be generated before returning anything to the console. No idea why. I dropped this for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",910088936,datasette --get should efficiently handle streaming CSV, https://github.com/simonw/datasette/issues/1673#issuecomment-1073361986,https://api.github.com/repos/simonw/datasette/issues/1673,1073361986,IC_kwDOBm6k_c4_-jRC,9599,simonw,2022-03-20T22:31:41Z,2022-03-20T22:34:06Z,OWNER,"Maybe it's because `supports_table_xinfo()` creates a brand new in-memory SQLite connection every time you call it? https://github.com/simonw/datasette/blob/798f075ef9b98819fdb564f9f79c78975a0f71e8/datasette/utils/sqlite.py#L22-L35 Actually no, I'm caching that already: https://github.com/simonw/datasette/blob/798f075ef9b98819fdb564f9f79c78975a0f71e8/datasette/utils/sqlite.py#L12-L19","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174708375,Streaming CSV spends a lot of time in `table_column_details`, https://github.com/simonw/datasette/issues/1672#issuecomment-1073355818,https://api.github.com/repos/simonw/datasette/issues/1672,1073355818,IC_kwDOBm6k_c4_-hwq,9599,simonw,2022-03-20T21:52:38Z,2022-03-20T21:52:38Z,OWNER,"That means taking on these issues: - https://github.com/simonw/datasette/issues/1101 - https://github.com/simonw/datasette/issues/1096 - https://github.com/simonw/datasette/issues/1062","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174697144,Refactor CSV handling code out of DataView, https://github.com/simonw/datasette/issues/1660#issuecomment-1073355032,https://api.github.com/repos/simonw/datasette/issues/1660,1073355032,IC_kwDOBm6k_c4_-hkY,9599,simonw,2022-03-20T21:46:43Z,2022-03-20T21:46:43Z,OWNER,I think the way to get rid of most of the remaining complexity in `DataView` is to refactor how CSV stuff works - pulling it in line with other export factors and extracting the streaming mechanism. Opening a fresh issue for that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170144879,Refactor and simplify Datasette routing and views, https://github.com/simonw/sqlite-utils/issues/140#issuecomment-1073330388,https://api.github.com/repos/simonw/sqlite-utils/issues/140,1073330388,IC_kwDOCGYnMM4_-bjU,9599,simonw,2022-03-20T19:44:39Z,2022-03-20T19:45:45Z,OWNER,"Alternative idea for specifying types: accept a Python expression, then use Python type literal syntax. For example: ``` sqlite-utils insert-files gifs.db images *.gif \ -c path -c md5 -c last_modified:mtime \ -a file_type '""gif""' ``` Where `-a` indicates an additional column.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688351054,Idea: insert-files mechanism for adding extra columns with fixed values, https://github.com/simonw/datasette/issues/1669#issuecomment-1073143413,https://api.github.com/repos/simonw/datasette/issues/1669,1073143413,IC_kwDOBm6k_c4_9t51,9599,simonw,2022-03-20T01:24:36Z,2022-03-20T01:24:36Z,OWNER,https://github.com/simonw/datasette/releases/tag/0.61a0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174404647,Release 0.61 alpha, https://github.com/simonw/datasette/issues/1669#issuecomment-1073137170,https://api.github.com/repos/simonw/datasette/issues/1669,1073137170,IC_kwDOBm6k_c4_9sYS,9599,simonw,2022-03-20T00:35:52Z,2022-03-20T00:35:52Z,OWNER,https://github.com/simonw/datasette/compare/0.60.2...main,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174404647,Release 0.61 alpha, https://github.com/simonw/datasette/issues/1668#issuecomment-1073136896,https://api.github.com/repos/simonw/datasette/issues/1668,1073136896,IC_kwDOBm6k_c4_9sUA,9599,simonw,2022-03-20T00:33:23Z,2022-03-20T00:33:23Z,OWNER,I'm going to release this as a 0.61 alpha so I can more easily depend on it from `datasette-hashed-urls`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073136686,https://api.github.com/repos/simonw/datasette/issues/1668,1073136686,IC_kwDOBm6k_c4_9sQu,9599,simonw,2022-03-20T00:31:13Z,2022-03-20T00:31:13Z,OWNER,"That demo is now live: - https://latest.datasette.io/alternative-route - https://latest.datasette.io/alternative-route/attraction_characteristic","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073135433,https://api.github.com/repos/simonw/datasette/issues/1668,1073135433,IC_kwDOBm6k_c4_9r9J,9599,simonw,2022-03-20T00:20:36Z,2022-03-20T00:20:36Z,OWNER,"Building this plugin instantly revealed that all of the links - on the homepage and the database page and so on - are incorrect: ```python from datasette import hookimpl @hookimpl def startup(datasette): db = datasette.get_database(""fixtures2"") db.route = ""alternative-route"" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073134816,https://api.github.com/repos/simonw/datasette/issues/1668,1073134816,IC_kwDOBm6k_c4_9rzg,9599,simonw,2022-03-20T00:16:22Z,2022-03-20T00:16:22Z,OWNER,I'm going to add a `fixtures2.db` database which has that as the name but `alternative-route` as the route. I'll set that up using a custom plugin in the `plugins/` folder that gets deployed by https://github.com/simonw/datasette/blob/main/.github/workflows/deploy-latest.yml,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073134206,https://api.github.com/repos/simonw/datasette/issues/1668,1073134206,IC_kwDOBm6k_c4_9rp-,9599,simonw,2022-03-20T00:12:03Z,2022-03-20T00:12:03Z,OWNER,I'd like to have a live demo of this up on `latest.datasette.io` too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073126264,https://api.github.com/repos/simonw/datasette/issues/1668,1073126264,IC_kwDOBm6k_c4_9pt4,9599,simonw,2022-03-19T22:59:30Z,2022-03-19T22:59:30Z,OWNER,"Also need to update the `datasette.urls` methods that construct the URL to a database/table/row - they take the database name but they need to know to look for the route. Need to add tests that check the links in the HTML and can confirm this is working correctly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073125334,https://api.github.com/repos/simonw/datasette/issues/1668,1073125334,IC_kwDOBm6k_c4_9pfW,9599,simonw,2022-03-19T22:53:55Z,2022-03-19T22:53:55Z,OWNER,"Need to update documentation in a few places - e.g. https://docs.datasette.io/en/stable/internals.html#remove-database-name > This removes a database that has been previously added. `name=` is the unique name of that database, used in its URL path.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073112104,https://api.github.com/repos/simonw/datasette/issues/1668,1073112104,IC_kwDOBm6k_c4_9mQo,9599,simonw,2022-03-19T21:08:21Z,2022-03-19T21:08:21Z,OWNER,"I think I've got this working but I need to write a test for it that covers the rare case when the route is not the same thing as the database name. I'll do that with a new test.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073097394,https://api.github.com/repos/simonw/datasette/issues/1668,1073097394,IC_kwDOBm6k_c4_9iqy,9599,simonw,2022-03-19T20:56:35Z,2022-03-19T20:56:35Z,OWNER,"I'm trying to think if there's any reason not to use `route` for this. Would I possibly want to use that noun for something else in the future? I like it more than `route_path` because it has no underscore. Decision made: I'm going with `route`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1667#issuecomment-1073076624,https://api.github.com/repos/simonw/datasette/issues/1667,1073076624,IC_kwDOBm6k_c4_9dmQ,9599,simonw,2022-03-19T20:31:44Z,2022-03-19T20:31:44Z,OWNER,I can now read `format` from `request.url_vars` and delete this code entirely: https://github.com/simonw/datasette/blob/b9c2b1cfc8692b9700416db98721fa3ec982f6be/datasette/views/base.py#L375-L381,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174302994,Make route matched pattern groups more consistent, https://github.com/simonw/datasette/issues/1668#issuecomment-1073076187,https://api.github.com/repos/simonw/datasette/issues/1668,1073076187,IC_kwDOBm6k_c4_9dfb,9599,simonw,2022-03-19T20:28:20Z,2022-03-19T20:28:20Z,OWNER,I'm going to keep `path` as the path to the file on disk. I'll pick a new name for what is currently `path` in that undocumented JSON API.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073076136,https://api.github.com/repos/simonw/datasette/issues/1668,1073076136,IC_kwDOBm6k_c4_9deo,9599,simonw,2022-03-19T20:27:44Z,2022-03-19T20:27:44Z,OWNER,"Pretty sure changing it will break some existing plugins though, including likely Datasette Desktop.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073076110,https://api.github.com/repos/simonw/datasette/issues/1668,1073076110,IC_kwDOBm6k_c4_9deO,9599,simonw,2022-03-19T20:27:22Z,2022-03-19T20:27:22Z,OWNER,"The docs do currently describe `path` as the filesystem path here: https://docs.datasette.io/en/stable/internals.html#database-class Good thing I'm not at 1.0 yet so I can change that!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073076015,https://api.github.com/repos/simonw/datasette/issues/1668,1073076015,IC_kwDOBm6k_c4_9dcv,9599,simonw,2022-03-19T20:26:32Z,2022-03-19T20:26:32Z,OWNER,I'm inclined to redefine `ds.path` to `ds.file_path` to fix this. Or `ds.filepath`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073075913,https://api.github.com/repos/simonw/datasette/issues/1668,1073075913,IC_kwDOBm6k_c4_9dbJ,9599,simonw,2022-03-19T20:25:46Z,2022-03-19T20:26:08Z,OWNER,"The output of `/.json` DOES use `path` to mean the URL path, not the path to the file on disk: ``` { ""fixtures.dot"": { ""name"": ""fixtures.dot"", ""hash"": null, ""color"": ""631f11"", ""path"": ""/fixtures~2Edot"", ``` So that's a problem already: having `db.path` refer to something different from that JSON is inconsistent.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073075697,https://api.github.com/repos/simonw/datasette/issues/1668,1073075697,IC_kwDOBm6k_c4_9dXx,9599,simonw,2022-03-19T20:24:06Z,2022-03-19T20:24:06Z,OWNER,"Right now if a database has a `.` in its name e.g. `fixtures.dot` the URL to that database is: /fixtures~2Edot But the output on `/-/databases` doesn't reflect that, it still shows the name with the dot.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1660#issuecomment-1073073599,https://api.github.com/repos/simonw/datasette/issues/1660,1073073599,IC_kwDOBm6k_c4_9c2_,9599,simonw,2022-03-19T20:06:40Z,2022-03-19T20:06:40Z,OWNER,"This blocks: - #1668","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170144879,Refactor and simplify Datasette routing and views, https://github.com/simonw/datasette/issues/1668#issuecomment-1073073579,https://api.github.com/repos/simonw/datasette/issues/1668,1073073579,IC_kwDOBm6k_c4_9c2r,9599,simonw,2022-03-19T20:06:27Z,2022-03-19T20:06:27Z,OWNER,Marking this as blocked until #1660 is done.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073073547,https://api.github.com/repos/simonw/datasette/issues/1668,1073073547,IC_kwDOBm6k_c4_9c2L,9599,simonw,2022-03-19T20:06:07Z,2022-03-19T20:06:07Z,OWNER,"Implementing this is a little tricky because there's a whole lot of code that expects the `database` captured by the URL routing to be the name used to look up the database in `datasette.databases` - or via `.get_database()`. The `DataView.get()` method is a good example of the trickyness here. It even has code that dispatches out to plugin hooks that take `database` as a parameter. https://github.com/simonw/datasette/blob/61419388c134001118aaf7dfb913562d467d7913/datasette/views/base.py#L383-L555 All the more reason to get rid of that `BaseView -> DataView -> TableView` hierarchy entirely: - #1660","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073043433,https://api.github.com/repos/simonw/datasette/issues/1668,1073043433,IC_kwDOBm6k_c4_9Vfp,9599,simonw,2022-03-19T16:54:55Z,2022-03-19T20:01:19Z,OWNER,"Options: - `route_path` - `url_path` - `route` I like `route_path`, or maybe `route`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073043713,https://api.github.com/repos/simonw/datasette/issues/1668,1073043713,IC_kwDOBm6k_c4_9VkB,9599,simonw,2022-03-19T16:56:19Z,2022-03-19T16:56:19Z,OWNER,"Worth noting that the `name` right now is picked automatically to avoid conflicts: https://github.com/simonw/datasette/blob/61419388c134001118aaf7dfb913562d467d7913/datasette/app.py#L397-L413","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073043350,https://api.github.com/repos/simonw/datasette/issues/1668,1073043350,IC_kwDOBm6k_c4_9VeW,9599,simonw,2022-03-19T16:54:26Z,2022-03-19T16:54:26Z,OWNER,"The `Database` class already has a `path` property but it means something else - it's the path to the `.db` file on disk: https://github.com/simonw/datasette/blob/61419388c134001118aaf7dfb913562d467d7913/datasette/database.py#L29-L50 So need a different name for the path-that-is-used-in-the-URL.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1667#issuecomment-1073042554,https://api.github.com/repos/simonw/datasette/issues/1667,1073042554,IC_kwDOBm6k_c4_9VR6,9599,simonw,2022-03-19T16:50:01Z,2022-03-19T16:52:35Z,OWNER,"OK, I've made this more consistent - I still need to address the fact that `format` can be `.json` or `json` or not used at all before I close this issue. https://github.com/simonw/datasette/blob/61419388c134001118aaf7dfb913562d467d7913/tests/test_routes.py#L15-L35","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174302994,Make route matched pattern groups more consistent, https://github.com/simonw/datasette/issues/1667#issuecomment-1073040072,https://api.github.com/repos/simonw/datasette/issues/1667,1073040072,IC_kwDOBm6k_c4_9UrI,9599,simonw,2022-03-19T16:34:02Z,2022-03-19T16:34:02Z,OWNER,"I called it `as_format` to avoid clashing with the Python built-in `format()` function when these things were turned into keyword arguments, but now that they're not I can use `format` instead. I think I'm going to go with `database`, `table`, `format` and `pks`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174302994,Make route matched pattern groups more consistent, https://github.com/simonw/datasette/issues/1666#issuecomment-1073039670,https://api.github.com/repos/simonw/datasette/issues/1666,1073039670,IC_kwDOBm6k_c4_9Uk2,9599,simonw,2022-03-19T16:31:08Z,2022-03-19T16:31:57Z,OWNER,"This does make it more interesting - it also highlights how inconsistent the way the capturing works is. Especially `as_format` which can be `None` or `""""` or `.json` or `json` or not used at all in the case of `TableView`. https://github.com/simonw/datasette/blob/764738dfcb16cd98b0987d443f59d5baa9d3c332/tests/test_routes.py#L12-L36","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174162781,Refactor URL routing to enable testing, https://github.com/simonw/datasette/issues/1666#issuecomment-1073039241,https://api.github.com/repos/simonw/datasette/issues/1666,1073039241,IC_kwDOBm6k_c4_9UeJ,9599,simonw,2022-03-19T16:28:15Z,2022-03-19T16:28:15Z,OWNER,This is more interesting if it also asserts against the captured matches from the pattern.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174162781,Refactor URL routing to enable testing, https://github.com/simonw/datasette/issues/878#issuecomment-1073037939,https://api.github.com/repos/simonw/datasette/issues/878,1073037939,IC_kwDOBm6k_c4_9UJz,9599,simonw,2022-03-19T16:19:30Z,2022-03-19T16:19:30Z,OWNER,"On revisiting https://gist.github.com/simonw/281eac9c73b062c3469607ad86470eb2 a few months later I'm having second thoughts about using `@inject` on the `main()` method. But I still like the pattern as a way to resolve more complex cases like ""to generate GeoJSON of the expanded view with labels, the label expansion code needs to run once at some before the GeoJSON formatting code does"". So I'm going to stick with it a tiny bit longer, but maybe try to make it a lot more explicit when it's going to happen rather than having the main view methods themselves also use async DI.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/1561#issuecomment-1072939780,https://api.github.com/repos/simonw/datasette/issues/1561,1072939780,IC_kwDOBm6k_c4_88ME,9599,simonw,2022-03-19T04:45:40Z,2022-03-19T04:45:40Z,OWNER,"I ended up moving hashed URL mode out to a plugin in: - #647 If you're still interested in using it with `_memory` please open an issue in that repo here: https://github.com/simonw/datasette-hashed-urls","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082765654,"add hash id to ""_memory"" url if hashed url mode is turned on and crossdb is also turned on", https://github.com/simonw/datasette/issues/1666#issuecomment-1072933875,https://api.github.com/repos/simonw/datasette/issues/1666,1072933875,IC_kwDOBm6k_c4_86vz,9599,simonw,2022-03-19T04:03:42Z,2022-03-19T04:03:42Z,OWNER,Tests so far: https://github.com/simonw/datasette/blob/711767bcd3c1e76a0861fe7f24069ff1c8efc97a/tests/test_routes.py#L12-L34,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174162781,Refactor URL routing to enable testing, https://github.com/simonw/datasette/issues/1228#issuecomment-1072915936,https://api.github.com/repos/simonw/datasette/issues/1228,1072915936,IC_kwDOBm6k_c4_82Xg,9599,simonw,2022-03-19T01:50:27Z,2022-03-19T01:50:27Z,OWNER,Demo: https://latest.datasette.io/fixtures/facetable - which now has a column called `n`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810397025,500 error caused by faceting if a column called `n` exists, https://github.com/simonw/datasette/issues/1228#issuecomment-1072908029,https://api.github.com/repos/simonw/datasette/issues/1228,1072908029,IC_kwDOBm6k_c4_80b9,9599,simonw,2022-03-19T00:57:54Z,2022-03-19T00:57:54Z,OWNER,"Yes! That's the problem. I was able to replicate it like so: ``` echo '[{ ""n"": ""one"", ""abc"": 1 }, { ""n"": ""one"", ""abc"": 2 }, { ""n"": ""two"", ""abc"": 3 }]' | sqlite-utils insert column-called-n.db t - ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810397025,500 error caused by faceting if a column called `n` exists, https://github.com/simonw/datasette/issues/1228#issuecomment-1072907680,https://api.github.com/repos/simonw/datasette/issues/1228,1072907680,IC_kwDOBm6k_c4_80Wg,9599,simonw,2022-03-19T00:55:48Z,2022-03-19T00:55:48Z,OWNER,... unless your data had a column called `n`?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810397025,500 error caused by faceting if a column called `n` exists, https://github.com/simonw/datasette/issues/1228#issuecomment-1072907610,https://api.github.com/repos/simonw/datasette/issues/1228,1072907610,IC_kwDOBm6k_c4_80Va,9599,simonw,2022-03-19T00:55:29Z,2022-03-19T00:55:29Z,OWNER,"It looks to me like something is causing the faceting query here to return a string when it was expected to return a number: https://github.com/simonw/datasette/blob/32963018e7edfab1233de7c7076c428d0e5c7813/datasette/facets.py#L153-L170 I can't think of any way that a `count(*) as n` would turn into a string though!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810397025,500 error caused by faceting if a column called `n` exists, https://github.com/simonw/datasette/issues/1605#issuecomment-1072907200,https://api.github.com/repos/simonw/datasette/issues/1605,1072907200,IC_kwDOBm6k_c4_80PA,9599,simonw,2022-03-19T00:52:54Z,2022-03-19T00:53:45Z,OWNER,"Had a thought about the implementation of this: it could make a really neat plugin. Something like `datasette-export` which adds a `export` command using https://docs.datasette.io/en/stable/plugin_hooks.html#register-commands-cli - then you could run: datasette export my-export-dir mydatabase.db -m metadata.json --template-dir templates/ And the command would then: - Create a `Datasette()` instance with those databases/metadata/etc - Execute`await datasette.client.get(""/"")` to get the homepage HTML - Parse the HTML using BeautifulSoup to find all `a[href]`, `link[href]`, `script[src]`, `img[src]` elements that reference a relative path as opposed to one that starts with `http://` - Write out the homepage to `my-export-dir/index.html` - Recursively fetch and dump all of the other pages and assets that it found too All of that HTML parsing may be over-complicating things. It could alternatively accept options for which pages you want to export: ``` datasette export my-export-dir \ mydatabase.db -m metadata.json --template-dir templates/ \ --path / \ --path /mydatabase ... ``` Or a really wild option: it could allow you to define the paths you want to export using a SQL query: ``` datasette export my-export-dir \ mydatabase.db -m metadata.json --template-dir templates/ \ --sql "" select '/' as path, 'index.html' as filename union all select '/mydatabase/articles/' || id as path, 'article-' || id || '.html' as filename from articles union all select '/mydatabase/tags/' || tag as path, 'tag-' || tag || '.html' as filename from tags "" ``` Which would save these files: - `index.html` as the content of `/` - `article-1.html` (and more) as the content of `/mydatabase/articles/1` - `tag-python.html` (and more) as the content of `/mydatabase/tags/python`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108671952,Scripted exports, https://github.com/simonw/datasette/issues/1662#issuecomment-1072905467,https://api.github.com/repos/simonw/datasette/issues/1662,1072905467,IC_kwDOBm6k_c4_8zz7,9599,simonw,2022-03-19T00:42:23Z,2022-03-19T00:42:23Z,OWNER,"Those client-side SQLite tricks are _really_ neat. `datasette publish` defaults to configuring it so the raw SQLite database can be downloaded from `/fixtures.db` - and this issue updated it to be served with a CORS header that would allow client-side scripts to load the file: - #1057 If you're not going to run any server-side code at all you don't need Datasette for this - you can upload the SQLite database file to any static hosting with CORS headers and load it into the client that way. In terms of static publishing, I do think there's something interesting about using Datasette to generate static sites. There's an issue discussing options for that over here: - #1605","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170497629,[feature request] Publish to fully static website, https://github.com/simonw/datasette/issues/1661#issuecomment-1072904703,https://api.github.com/repos/simonw/datasette/issues/1661,1072904703,IC_kwDOBm6k_c4_8zn_,9599,simonw,2022-03-19T00:37:36Z,2022-03-19T00:37:36Z,OWNER,Updated docs: https://docs.datasette.io/en/latest/performance.html#datasette-hashed-urls,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774,Remove Hashed URL mode, https://github.com/simonw/datasette/issues/1661#issuecomment-1072901159,https://api.github.com/repos/simonw/datasette/issues/1661,1072901159,IC_kwDOBm6k_c4_8ywn,9599,simonw,2022-03-19T00:20:27Z,2022-03-19T00:20:27Z,OWNER,I can remove the `default_cache_ttl_hashed` setting too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774,Remove Hashed URL mode, https://github.com/simonw/datasette/pull/1664#issuecomment-1072898923,https://api.github.com/repos/simonw/datasette/issues/1664,1072898923,IC_kwDOBm6k_c4_8yNr,9599,simonw,2022-03-19T00:11:33Z,2022-03-19T00:11:33Z,OWNER,I'm going to land this and handle those in separate commits.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173017980,Remove hashed URL mode, https://github.com/simonw/datasette/pull/1664#issuecomment-1072898797,https://api.github.com/repos/simonw/datasette/issues/1664,1072898797,IC_kwDOBm6k_c4_8yLt,9599,simonw,2022-03-19T00:11:09Z,2022-03-19T00:11:09Z,OWNER,Still need to remove it from the documentation and do something about that `hash_urls` setting.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173017980,Remove hashed URL mode, https://github.com/simonw/datasette/pull/1664#issuecomment-1072890524,https://api.github.com/repos/simonw/datasette/issues/1664,1072890524,IC_kwDOBm6k_c4_8wKc,9599,simonw,2022-03-18T23:44:33Z,2022-03-19T00:06:51Z,OWNER,Looks like that was set here: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L490-L492,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173017980,Remove hashed URL mode, https://github.com/simonw/datasette/pull/1664#issuecomment-1072890205,https://api.github.com/repos/simonw/datasette/issues/1664,1072890205,IC_kwDOBm6k_c4_8wFd,9599,simonw,2022-03-18T23:43:15Z,2022-03-18T23:43:15Z,OWNER,"Now almost everything is working except for foreign key expansion: ![CleanShot 2022-03-18 at 16 41 39@2x](https://user-images.githubusercontent.com/9599/159097349-6f41dfdf-5bab-449b-a148-5cda3df6534c.png) Using the debugger I tracked it down to this code: https://github.com/simonw/datasette/blob/30e5f0e67c38054a8087a2a4eae3fc4d1779af90/datasette/views/table.py#L708-L715 Turns out `default_labels` there is `None` - and it's a parameter to that `data()` method: https://github.com/simonw/datasette/blob/30e5f0e67c38054a8087a2a4eae3fc4d1779af90/datasette/views/table.py#L325-L334 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173017980,Remove hashed URL mode, https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1072834273,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1072834273,IC_kwDOCGYnMM4_8ibh,9599,simonw,2022-03-18T21:36:05Z,2022-03-18T21:36:05Z,OWNER,"Python's `str.encode()` method has a `errors=` parameter that does something along these lines: https://docs.python.org/3/library/stdtypes.html#str.encode > *errors* may be given to set a different error handling scheme. The default for *errors* is `'strict'`, meaning that encoding errors raise a [`UnicodeError`](https://docs.python.org/3/library/exceptions.html#UnicodeError ""UnicodeError""). Other possible values are `'ignore'`, `'replace'`, `'xmlcharrefreplace'`, `'backslashreplace'` and any other name registered via [`codecs.register_error()`](https://docs.python.org/3/library/codecs.html#codecs.register_error ""codecs.register_error""), Imitating this might be the way to go.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,Options for how `r.parsedate()` should handle invalid dates, https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1072833174,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1072833174,IC_kwDOCGYnMM4_8iKW,9599,simonw,2022-03-18T21:34:06Z,2022-03-18T21:34:06Z,OWNER,"Good call-out: right now the `parsedate()` and `parsedatetime()` functions both terminate with an exception if they hit something invalid: https://sqlite-utils.datasette.io/en/stable/cli.html#sqlite-utils-convert-recipes It would be better if this was configurable by the user (and properly documented) - options could include ""set null if date is invalid"" and ""leave the value as it is if invalid"" in addition to throwing an error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,Options for how `r.parsedate()` should handle invalid dates, https://github.com/simonw/datasette/pull/1664#issuecomment-1071813296,https://api.github.com/repos/simonw/datasette/issues/1664,1071813296,IC_kwDOBm6k_c4_4pKw,9599,simonw,2022-03-17T23:26:22Z,2022-03-17T23:26:22Z,OWNER,Probably caused by the convoluted code is `get_format()`: https://github.com/simonw/datasette/blob/30e5f0e67c38054a8087a2a4eae3fc4d1779af90/datasette/views/base.py#L466-L481,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173017980,Remove hashed URL mode, https://github.com/simonw/datasette/pull/1664#issuecomment-1071809988,https://api.github.com/repos/simonw/datasette/issues/1664,1071809988,IC_kwDOBm6k_c4_4oXE,9599,simonw,2022-03-17T23:24:57Z,2022-03-17T23:24:57Z,OWNER,"My hunch is that this is broken because of this: https://github.com/simonw/datasette/blob/30e5f0e67c38054a8087a2a4eae3fc4d1779af90/datasette/app.py#L1098-L1107 Note how the table uses `table_and_format` but the row uses just `table` - I think there's code that's getting confused by this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173017980,Remove hashed URL mode, https://github.com/simonw/datasette/pull/1664#issuecomment-1071803114,https://api.github.com/repos/simonw/datasette/issues/1664,1071803114,IC_kwDOBm6k_c4_4mrq,9599,simonw,2022-03-17T23:22:00Z,2022-03-17T23:22:00Z,OWNER,"Surprisingly I managed to break https://latest.datasette.io/fixtures/custom_foreign_key_label while working on this change: ![CleanShot 2022-03-17 at 16 16 54@2x](https://user-images.githubusercontent.com/9599/158909271-717b65e8-cfcc-44c4-b1cc-f34478b0f803.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173017980,Remove hashed URL mode, https://github.com/simonw/datasette/issues/1661#issuecomment-1071797707,https://api.github.com/repos/simonw/datasette/issues/1661,1071797707,IC_kwDOBm6k_c4_4lXL,9599,simonw,2022-03-17T23:19:24Z,2022-03-17T23:19:24Z,OWNER,"Moving this to PR so I can comment on individual lines: - #1664","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774,Remove Hashed URL mode, https://github.com/simonw/datasette/issues/1661#issuecomment-1071793307,https://api.github.com/repos/simonw/datasette/issues/1661,1071793307,IC_kwDOBm6k_c4_4kSb,9599,simonw,2022-03-17T23:17:32Z,2022-03-17T23:17:32Z,OWNER,"Surprisingly I managed to break https://latest.datasette.io/fixtures/custom_foreign_key_label while working on this change: ![CleanShot 2022-03-17 at 16 16 54@2x](https://user-images.githubusercontent.com/9599/158909271-717b65e8-cfcc-44c4-b1cc-f34478b0f803.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774,Remove Hashed URL mode, https://github.com/simonw/datasette/issues/1661#issuecomment-1071706993,https://api.github.com/repos/simonw/datasette/issues/1661,1071706993,IC_kwDOBm6k_c4_4PNx,9599,simonw,2022-03-17T22:42:21Z,2022-03-17T22:42:21Z,OWNER,"As part of this I'm going to get rid of this mechanism: https://github.com/simonw/datasette/blob/30e5f0e67c38054a8087a2a4eae3fc4d1779af90/datasette/views/base.py#L170-L173 Unwrapping `request.scope[""url_route""][""kwargs""]` into keyword argument to view functions just made the code harder to follow.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774,Remove Hashed URL mode, https://github.com/simonw/datasette/issues/1663#issuecomment-1071519407,https://api.github.com/repos/simonw/datasette/issues/1663,1071519407,IC_kwDOBm6k_c4_3hav,9599,simonw,2022-03-17T21:32:35Z,2022-03-17T21:32:35Z,OWNER,"Updated docs: - https://docs.datasette.io/en/latest/internals.html#datasette-class - https://docs.datasette.io/en/latest/internals.html#db-hash","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170554975,Document the internals that were used in datasette-hashed-urls, https://github.com/simonw/datasette/issues/1532#issuecomment-1069570893,https://api.github.com/repos/simonw/datasette/issues/1532,1069570893,IC_kwDOBm6k_c4_wFtN,9599,simonw,2022-03-16T20:11:41Z,2022-03-16T20:13:34Z,OWNER,"Could also build a CLI Rich/Textual app to exercise the API - which could embed Datasette as a dependency and work using `datasette.client.get(...)` calls. Could be a plugin that adds a `datasette tui` command.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065429936,Use datasette-table Web Component to guide the design of the JSON API for 1.0, https://github.com/simonw/datasette/issues/1663#issuecomment-1068742624,https://api.github.com/repos/simonw/datasette/issues/1663,1068742624,IC_kwDOBm6k_c4_s7fg,9599,simonw,2022-03-16T05:17:45Z,2022-03-16T05:17:45Z,OWNER,Should be documented here: https://docs.datasette.io/en/stable/internals.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170554975,Document the internals that were used in datasette-hashed-urls, https://github.com/simonw/datasette/issues/1661#issuecomment-1068728484,https://api.github.com/repos/simonw/datasette/issues/1661,1068728484,IC_kwDOBm6k_c4_s4Ck,9599,simonw,2022-03-16T04:47:39Z,2022-03-16T04:47:39Z,OWNER,https://datasette.io/plugins/datasette-hashed-urls is released now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774,Remove Hashed URL mode, https://github.com/simonw/datasette/issues/1661#issuecomment-1068630353,https://api.github.com/repos/simonw/datasette/issues/1661,1068630353,IC_kwDOBm6k_c4_sgFR,9599,simonw,2022-03-16T01:24:56Z,2022-03-16T01:25:49Z,OWNER,"Here's the only bit of code that references that `_hash` mechanism: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L259-L265 And here's the test: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/tests/test_api.py#L828-L854 Related issue: - #471","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774,Remove Hashed URL mode, https://github.com/simonw/datasette/issues/1661#issuecomment-1068628839,https://api.github.com/repos/simonw/datasette/issues/1661,1068628839,IC_kwDOBm6k_c4_sftn,9599,simonw,2022-03-16T01:21:36Z,2022-03-16T01:21:48Z,OWNER,"From https://docs.datasette.io/en/0.60.2/performance.html#hashed-url-mode > You can enable these hashed URLs in two ways: using the [hash_urls](https://docs.datasette.io/en/0.60.2/settings.html#setting-hash-urls) configuration setting (which affects all requests to Datasette) or via the `?_hash=1` query string parameter (which only applies to the current request). I'm going to drop` ?_hash=1` entirely. I'd actually forgotten that feature existed!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774,Remove Hashed URL mode, https://github.com/simonw/datasette/issues/1661#issuecomment-1068554827,https://api.github.com/repos/simonw/datasette/issues/1661,1068554827,IC_kwDOBm6k_c4_sNpL,9599,simonw,2022-03-15T23:16:58Z,2022-03-15T23:18:58Z,OWNER,"If you attempt to use the [old setting](https://docs.datasette.io/en/stable/settings.html#hash-urls): datasette mydatabase.db --setting hash_urls 1 It should error with a message saying that the feature has been moved to a plugin. I'll do this with a `deprecated_settings` mechanism so the error can be detected even though `datasette --help-settings` will no longer return the setting. https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/cli.py#L479-L489","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774,Remove Hashed URL mode, https://github.com/simonw/datasette/issues/1661#issuecomment-1068553454,https://api.github.com/repos/simonw/datasette/issues/1661,1068553454,IC_kwDOBm6k_c4_sNTu,9599,simonw,2022-03-15T23:14:37Z,2022-03-15T23:14:37Z,OWNER,"This is going to simplify the code in the various view classes substantially: - #1660","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774,Remove Hashed URL mode, https://github.com/simonw/datasette/issues/647#issuecomment-1068552696,https://api.github.com/repos/simonw/datasette/issues/647,1068552696,IC_kwDOBm6k_c4_sNH4,9599,simonw,2022-03-15T23:13:06Z,2022-03-15T23:13:06Z,OWNER,"The plugin works. I'm going to implement one last feature for it: - https://github.com/simonw/datasette-hashed-urls/issues/3 Then I can remove hashed URL mode in a separate issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",531755959,Move hashed URL mode out to a plugin, https://github.com/simonw/datasette/issues/647#issuecomment-1068539404,https://api.github.com/repos/simonw/datasette/issues/647,1068539404,IC_kwDOBm6k_c4_sJ4M,9599,simonw,2022-03-15T22:49:01Z,2022-03-15T22:49:01Z,OWNER,"I shipped the first version of this: https://github.com/simonw/datasette-hashed-urls Next step: test it with a live demo: - https://github.com/simonw/datasette-hashed-urls/issues/2","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",531755959,Move hashed URL mode out to a plugin, https://github.com/simonw/datasette/issues/1439#issuecomment-1068461449,https://api.github.com/repos/simonw/datasette/issues/1439,1068461449,IC_kwDOBm6k_c4_r22J,9599,simonw,2022-03-15T20:51:26Z,2022-03-15T20:51:26Z,OWNER,I'm happy with this now that I've landed Tilde encoding in #1657.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/857#issuecomment-1068450483,https://api.github.com/repos/simonw/datasette/issues/857,1068450483,IC_kwDOBm6k_c4_r0Kz,9599,simonw,2022-03-15T20:43:55Z,2022-03-15T20:43:55Z,OWNER,Dupe of #1510.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642297505,Comprehensive documentation for variables made available to templates, https://github.com/simonw/datasette/issues/1509#issuecomment-1068445412,https://api.github.com/repos/simonw/datasette/issues/1509,1068445412,IC_kwDOBm6k_c4_ry7k,9599,simonw,2022-03-15T20:37:50Z,2022-03-15T20:38:56Z,OWNER,"... maybe Datasette itself should include interactive API documentation, in addition to documenting it in the manual? `/dbname/table/-/apidocs` could return documentation about the specific table, taking into account columns and types.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1054243511,Datasette 1.0 JSON API (and documentation), https://github.com/simonw/datasette/issues/1509#issuecomment-1068444767,https://api.github.com/repos/simonw/datasette/issues/1509,1068444767,IC_kwDOBm6k_c4_ryxf,9599,simonw,2022-03-15T20:37:03Z,2022-03-15T20:37:03Z,OWNER,"Idea: I could add Pydantic https://pydantic-docs.helpmanual.io/usage/schema/ as an optional test dependency and use it to generate JSON schemas and run validation against examples in the API documentation. Maybe generate API documentation from it too?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1054243511,Datasette 1.0 JSON API (and documentation), https://github.com/simonw/datasette/issues/1510#issuecomment-1068443509,https://api.github.com/repos/simonw/datasette/issues/1510,1068443509,IC_kwDOBm6k_c4_ryd1,9599,simonw,2022-03-15T20:35:29Z,2022-03-15T20:35:29Z,OWNER,If I set a rule that everything available in the template context MUST also be available via the JSON API (maybe through an extras mechanism) I can combine this with API documentation and solve both at once.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1054244712,Datasette 1.0 documented template context (maybe via API docs), https://github.com/simonw/datasette/issues/870#issuecomment-650696054,https://api.github.com/repos/simonw/datasette/issues/870,650696054,MDEyOklzc3VlQ29tbWVudDY1MDY5NjA1NA==,9599,simonw,2020-06-28T04:52:41Z,2022-03-15T20:07:17Z,OWNER,"This would be a lot easier if I had extracted out the hash logic to a plugin, see: - #647","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",646737558,Refactor default views to use register_routes, https://github.com/simonw/datasette/issues/1660#issuecomment-1068418619,https://api.github.com/repos/simonw/datasette/issues/1660,1068418619,IC_kwDOBm6k_c4_rsY7,9599,simonw,2022-03-15T20:06:19Z,2022-03-15T20:06:19Z,OWNER,"Also related: - #878 - #1512 - #1518 - #870 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170144879,Refactor and simplify Datasette routing and views, https://github.com/simonw/datasette/issues/1660#issuecomment-1068417357,https://api.github.com/repos/simonw/datasette/issues/1660,1068417357,IC_kwDOBm6k_c4_rsFN,9599,simonw,2022-03-15T20:05:08Z,2022-03-15T20:05:08Z,OWNER,"`DataView` is used as the base class for: - `DatabaseView` - `DatabaseDownload` (just so the permissions checks can be called) - `QueryView` - which isn't routed to directly, it's called from `DatabaseView` if `?sql=` is available and `TableView` for canned queries - `RowTableShared` which is the base class for `TableView` and `RowView`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170144879,Refactor and simplify Datasette routing and views, https://github.com/simonw/datasette/issues/1660#issuecomment-1068415072,https://api.github.com/repos/simonw/datasette/issues/1660,1068415072,IC_kwDOBm6k_c4_rrhg,9599,simonw,2022-03-15T20:02:36Z,2022-03-15T20:02:36Z,OWNER,"This is one of the worst bits - the `get_format()` method on the `DataView` base class actually modifies `args`, including removing keys! Really confusing: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L454-L482 Then `BaseView` has some surprising responsibilities. It has a utility helper for checking multiple permissions at once: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L81-L105 And its own render method that adds extra stuff to the template context and handles the rel: alternate header: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L131-L157 Then `DataView` does all sorts of weird stuff - from handling database hashes (which I want to remove, see #647): https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L206-L219 To streaming CSV responses: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L286-L308 To handling SQLite exceptions: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L514-L526 And a ton more. It' s a big mess.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170144879,Refactor and simplify Datasette routing and views, https://github.com/simonw/datasette/issues/1062#issuecomment-1068327874,https://api.github.com/repos/simonw/datasette/issues/1062,1068327874,IC_kwDOBm6k_c4_rWPC,9599,simonw,2022-03-15T18:33:49Z,2022-03-15T18:33:49Z,OWNER,"I can get regular `.json` to stream too, using the pattern described in this TIL: https://til.simonwillison.net/python/output-json-array-streaming","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732674148,Refactor .csv to be an output renderer - and teach register_output_renderer to stream all rows, https://github.com/simonw/datasette/issues/1651#issuecomment-1068319530,https://api.github.com/repos/simonw/datasette/issues/1651,1068319530,IC_kwDOBm6k_c4_rUMq,9599,simonw,2022-03-15T18:25:42Z,2022-03-15T18:25:42Z,OWNER,"Done: - https://latest.datasette.io/fixtures/table~2Fwith~2Fslashes~2Ecsv - https://latest.datasette.io/fixtures/table~2Fwith~2Fslashes~2Ecsv.csv - https://latest.datasette.io/fixtures/table~2Fwith~2Fslashes~2Ecsv.json","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161584460,Get rid of the no-longer necessary ?_format=json hack for tables called x.json, https://github.com/simonw/datasette/issues/1657#issuecomment-1068318454,https://api.github.com/repos/simonw/datasette/issues/1657,1068318454,IC_kwDOBm6k_c4_rT72,9599,simonw,2022-03-15T18:25:11Z,2022-03-15T18:25:11Z,OWNER,"Demo: - https://latest.datasette.io/fixtures/table~2Fwith~2Fslashes~2Ecsv - https://latest.datasette.io/fixtures/table~2Fwith~2Fslashes~2Ecsv.csv - https://latest.datasette.io/fixtures/table~2Fwith~2Fslashes~2Ecsv.json","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756,Tilde encoding: use ~ instead of - for dash-encoding, https://github.com/simonw/datasette/issues/1657#issuecomment-1068306916,https://api.github.com/repos/simonw/datasette/issues/1657,1068306916,IC_kwDOBm6k_c4_rRHk,9599,simonw,2022-03-15T18:15:11Z,2022-03-15T18:15:11Z,OWNER,Now live here: https://fivethirtyeight.datasettes.com/fivethirtyeight/august-senate-polls~2Faugust_senate_polls,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756,Tilde encoding: use ~ instead of - for dash-encoding, https://github.com/simonw/datasette/issues/1657#issuecomment-1068296042,https://api.github.com/repos/simonw/datasette/issues/1657,1068296042,IC_kwDOBm6k_c4_rOdq,9599,simonw,2022-03-15T18:05:54Z,2022-03-15T18:05:54Z,OWNER,Documentation: https://docs.datasette.io/en/latest/internals.html#tilde-encoding,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756,Tilde encoding: use ~ instead of - for dash-encoding, https://github.com/simonw/datasette/issues/1657#issuecomment-1068181623,https://api.github.com/repos/simonw/datasette/issues/1657,1068181623,IC_kwDOBm6k_c4_qyh3,9599,simonw,2022-03-15T16:18:23Z,2022-03-15T16:18:23Z,OWNER,Moving this to a PR.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756,Tilde encoding: use ~ instead of - for dash-encoding, https://github.com/simonw/datasette/issues/1657#issuecomment-1068148013,https://api.github.com/repos/simonw/datasette/issues/1657,1068148013,IC_kwDOBm6k_c4_qqUt,9599,simonw,2022-03-15T15:50:15Z,2022-03-15T15:50:15Z,OWNER,"The thing that broke everything was this change: I'm going to bring back the horrible `get_format()` method for the moment, with its weird mutations of the `args` object, then try and get rid of it again later.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756,Tilde encoding: use ~ instead of - for dash-encoding, https://github.com/simonw/datasette/issues/1658#issuecomment-1068138578,https://api.github.com/repos/simonw/datasette/issues/1658,1068138578,IC_kwDOBm6k_c4_qoBS,9599,simonw,2022-03-15T15:42:49Z,2022-03-15T15:42:49Z,OWNER,"Easiest way to do this was with three reverts, then cherry-pick back the code of conduct.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1169840669,Revert main to version that passes tests, https://github.com/simonw/datasette/issues/1657#issuecomment-1068126821,https://api.github.com/repos/simonw/datasette/issues/1657,1068126821,IC_kwDOBm6k_c4_qlJl,9599,simonw,2022-03-15T15:31:54Z,2022-03-15T15:31:54Z,OWNER,The state I had got to prior to that revert is in https://github.com/simonw/datasette/tree/issue-1657-wip,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756,Tilde encoding: use ~ instead of - for dash-encoding, https://github.com/simonw/datasette/issues/1657#issuecomment-1068125636,https://api.github.com/repos/simonw/datasette/issues/1657,1068125636,IC_kwDOBm6k_c4_qk3E,9599,simonw,2022-03-15T15:30:54Z,2022-03-15T15:30:54Z,OWNER,I've made a real mess of this. I'm going to revert Datasette`main` back to the last commit that passed the tests and try this again in a branch.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756,Tilde encoding: use ~ instead of - for dash-encoding, https://github.com/simonw/datasette/issues/1657#issuecomment-1067423720,https://api.github.com/repos/simonw/datasette/issues/1657,1067423720,IC_kwDOBm6k_c4_n5fo,9599,simonw,2022-03-14T23:59:56Z,2022-03-14T23:59:56Z,OWNER,"Updated test: ```python @pytest.mark.parametrize( ""original,expected"", ( (""abc"", ""abc""), (""/foo/bar"", ""~2Ffoo~2Fbar""), (""/-/bar"", ""~2F-~2Fbar""), (""-/db-/table.csv"", ""-~2Fdb-~2Ftable~2Ecsv""), (r""%~-/"", ""~25~7E-~2F""), (""~25~7E~2D~2F"", ""~7E25~7E7E~7E2D~7E2F""), ), ) def test_tilde_encoding(original, expected): actual = utils.tilde_encode(original) assert actual == expected # And test round-trip assert original == utils.tilde_decode(actual) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756,Tilde encoding: use ~ instead of - for dash-encoding, https://github.com/simonw/datasette/issues/1657#issuecomment-1067414156,https://api.github.com/repos/simonw/datasette/issues/1657,1067414156,IC_kwDOBm6k_c4_n3KM,9599,simonw,2022-03-14T23:38:41Z,2022-03-14T23:38:41Z,OWNER,"And in https://datatracker.ietf.org/doc/html/rfc3986#section-2.3 ""Unreserved Characters"": unreserved = ALPHA / DIGIT / ""-"" / ""."" / ""_"" / ""~""","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756,Tilde encoding: use ~ instead of - for dash-encoding, https://github.com/simonw/datasette/issues/1657#issuecomment-1067413691,https://api.github.com/repos/simonw/datasette/issues/1657,1067413691,IC_kwDOBm6k_c4_n3C7,9599,simonw,2022-03-14T23:37:42Z,2022-03-14T23:37:42Z,OWNER,"Relevant: https://datatracker.ietf.org/doc/html/rfc3986#section-2.1 ``` reserved = gen-delims / sub-delims gen-delims = "":"" / ""/"" / ""?"" / ""#"" / ""["" / ""]"" / ""@"" sub-delims = ""!"" / ""$"" / ""&"" / ""'"" / ""("" / "")"" / ""*"" / ""+"" / "","" / "";"" / ""="" ``` Notably `~` is not in either of those lists.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756,Tilde encoding: use ~ instead of - for dash-encoding, https://github.com/simonw/datasette/issues/1651#issuecomment-1067382442,https://api.github.com/repos/simonw/datasette/issues/1651,1067382442,IC_kwDOBm6k_c4_nvaq,9599,simonw,2022-03-14T22:59:10Z,2022-03-14T22:59:10Z,OWNER,"This work is now blocked on: - https://github.com/simonw/datasette/issues/1657","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161584460,Get rid of the no-longer necessary ?_format=json hack for tables called x.json, https://github.com/simonw/datasette/issues/1657#issuecomment-1067382232,https://api.github.com/repos/simonw/datasette/issues/1657,1067382232,IC_kwDOBm6k_c4_nvXY,9599,simonw,2022-03-14T22:58:47Z,2022-03-14T22:58:47Z,OWNER,"Asked about this [on Twitter](https://twitter.com/simonw/status/1503499169775849473): > Anyone ever seen a proxy or other URL handling system do anything surprising with the tilde ""~"" character? > > I'm considering it as an escaping character, in place of ""-"" as described in Replies so far seem like it should be OK - Apache has supported this for home directories for a couple of decades now without any problems.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756,Tilde encoding: use ~ instead of - for dash-encoding, https://github.com/simonw/datasette/issues/1657#issuecomment-1067381556,https://api.github.com/repos/simonw/datasette/issues/1657,1067381556,IC_kwDOBm6k_c4_nvM0,9599,simonw,2022-03-14T22:57:27Z,2022-03-14T22:57:45Z,OWNER,"The problem with the [dash encoding mechanism](https://simonwillison.net/2022/Mar/5/dash-encoding/) is that it turns out dashes are used in a LOT of existing Datasette instances - much of https://fivethirtyeight.datasettes.com/fivethirtyeight for example, and even https://datasette.io/ itself: https://datasette.io/dogsheep-index It's pretty ugly to force all of those to change to their dash-encoded equivalent - and in fact it broke https://datasette.io/ in a subtle way: - https://github.com/simonw/datasette.io/issues/94 I'm going to try using `~` instead and see if that works as well and causes less breakage to existing sites.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756,Tilde encoding: use ~ instead of - for dash-encoding, https://github.com/simonw/datasette/issues/1439#issuecomment-1065988403,https://api.github.com/repos/simonw/datasette/issues/1439,1065988403,IC_kwDOBm6k_c4_ibEz,9599,simonw,2022-03-13T00:06:38Z,2022-03-13T00:07:19Z,OWNER,"If I want to reserve `-` as a character that CAN be used in URLs, the only remaining character that might make sense for escape sequences is `~` - based on this last line of characters that are escape from percentage encoding: ```python _ALWAYS_SAFE = frozenset(b'ABCDEFGHIJKLMNOPQRSTUVWXYZ' b'abcdefghijklmnopqrstuvwxyz' b'0123456789' b'_.-~') ``` So I'd add both `-` and `_` back to the safe list, but use `~` to escape `.` and `/` and suchlike.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1065987808,https://api.github.com/repos/simonw/datasette/issues/1439,1065987808,IC_kwDOBm6k_c4_ia7g,9599,simonw,2022-03-13T00:02:32Z,2022-03-13T00:02:32Z,OWNER,"OK, this has broken a lot more than I expected it would. Turns out `-` is a very common character in existing Datasette database names! https://datasette.io/-/databases for example has two: ```json [ { ""name"": ""docs-index"", ""path"": ""docs-index.db"", ""size"": 1007616, ""is_mutable"": false, ""is_memory"": false, ""hash"": ""0ac6c3de2762fcd174fd249fed8a8fa6046ea345173d22c2766186bf336462b2"" }, { ""name"": ""dogsheep-index"", ""path"": ""dogsheep-index.db"", ""size"": 5496832, ""is_mutable"": false, ""is_memory"": false, ""hash"": ""d1ea238d204e5b9ae783c86e4af5bcdf21267c1f391de3e468d9665494ee012a"" } ] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065597709,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065597709,IC_kwDOCGYnMM4_g7sN,9599,simonw,2022-03-11T22:32:43Z,2022-03-11T22:32:43Z,OWNER,"Trying to figure out what that extra field in `table_info` compared to `table_xinfo` is: ``` >>> list(db.query(""PRAGMA table_xinfo('t')"")) [{'cid': 0, 'name': 'body', 'type': 'TEXT', 'notnull': 0, 'dflt_value': None, 'pk': 0, 'hidden': 0}, {'cid': 1, 'name': 'd', 'type': 'INT', 'notnull': 0, 'dflt_value': None, 'pk': 0, 'hidden': 2}] `` Presumably `hidden` 0 v.s 2 v.s. other values has meaning.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160034488,Support for generated columns, https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065596417,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065596417,IC_kwDOCGYnMM4_g7YB,9599,simonw,2022-03-11T22:30:15Z,2022-03-11T22:30:15Z,OWNER,"I tried it out in Jupyter and it works as advertised: Introspection is a bit weird: there doesn't seem to be a way to introspect generated columns outside of parsing the stored SQL schema for the columns at the moment! And the `.columns` method doesn't return them at all: https://github.com/simonw/sqlite-utils/blob/433813612ff9b4b501739fd7543bef0040dd51fe/sqlite_utils/db.py#L1207-L1213 Here's why: ``` >>> db.execute(""PRAGMA table_info('t')"").fetchall() [(0, 'body', 'TEXT', 0, None, 0)] >>> db.execute(""PRAGMA table_xinfo('t')"").fetchall() [(0, 'body', 'TEXT', 0, None, 0, 0), (1, 'd', 'INT', 0, None, 0, 2)] ``` So `table_xinfo()` is needed to get back columns including generated columns: https://www.sqlite.org/pragma.html#pragma_table_xinfo > **PRAGMA** *schema.***table_xinfo(***table-name***);** > > This pragma returns one row for each column in the named table, including [hidden columns](https://www.sqlite.org/vtab.html#hiddencol) in virtual tables. The output is the same as for [PRAGMA table_info](https://www.sqlite.org/pragma.html#pragma_table_info) except that hidden columns are shown rather than being omitted.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160034488,Support for generated columns, https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065402557,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065402557,IC_kwDOCGYnMM4_gMC9,9599,simonw,2022-03-11T19:01:08Z,2022-03-11T21:42:25Z,OWNER,"Just spotted this in https://www.sqlite.org/gencol.html > The only functional difference is that one cannot add new STORED columns using the [ALTER TABLE ADD COLUMN](https://www.sqlite.org/lang_altertable.html#altertabaddcol) command. Only VIRTUAL columns can be added using ALTER TABLE. So to add stored columns to an existing table we would need to use the `.transform()` trick. Which implies that this should actually be a capability of the various `.create()` methods, since transform works by creating a new table with those and then copying across the old data. Here's where `.transform()` calls `.create_table_sql()` under the hood: https://github.com/simonw/sqlite-utils/blob/9388edf57aa15719095e3cf0952c1653cd070c9b/sqlite_utils/db.py#L1627-L1637","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160034488,Support for generated columns, https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065389386,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065389386,IC_kwDOCGYnMM4_gI1K,9599,simonw,2022-03-11T18:42:53Z,2022-03-11T21:40:51Z,OWNER,"The Python API could be: ```python db[table_name].add_generated_column(""field"", str, ""json_extract(data, '$.field')"", stored=True) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160034488,Support for generated columns, https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065458729,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065458729,IC_kwDOCGYnMM4_gZwp,9599,simonw,2022-03-11T19:58:50Z,2022-03-11T20:00:25Z,OWNER,"I'm coming round to your suggestion to have this as extra arguments to `sqlite-utils add-column` now, especially since you also need to pass a column type. I'd like to come up with syntax for `sqlite-utils create-table` as well. https://sqlite-utils.datasette.io/en/stable/cli-reference.html#create-table Maybe extra `--generated-stored colname expression` (and `--generated`) options would work there.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160034488,Support for generated columns, https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065440445,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065440445,IC_kwDOCGYnMM4_gVS9,9599,simonw,2022-03-11T19:52:15Z,2022-03-11T19:52:15Z,OWNER,"Two new parameters to `.create_table()` and friends: - `generated={...}` - generated column definitions - `generated_stored={...}` generated stored column definitions These columns will be added at the end of the table, but you can use the `column_order=` parameter to apply a different order.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1160034488,Support for generated columns, https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065386352,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065386352,IC_kwDOCGYnMM4_gIFw,9599,simonw,2022-03-11T18:41:37Z,2022-03-11T18:41:37Z,OWNER,"I like `add-generated-column` - feels very clear to me, and is a nice place for adding logic that checks if the DB version supports it or not and shows a useful error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160034488,Support for generated columns, https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065384183,https://api.github.com/repos/simonw/sqlite-utils/issues/414,1065384183,IC_kwDOCGYnMM4_gHj3,9599,simonw,2022-03-11T18:40:39Z,2022-03-11T18:40:39Z,OWNER,"That fixed it: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166731361,I forgot to include the changelog in the 3.25.1 release, https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065382145,https://api.github.com/repos/simonw/sqlite-utils/issues/414,1065382145,IC_kwDOCGYnMM4_gHEB,9599,simonw,2022-03-11T18:39:05Z,2022-03-11T18:39:05Z,OWNER,"https://sqlite-utils.datasette.io/en/3.25.1/changelog.html is currently wrong: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166731361,I forgot to include the changelog in the 3.25.1 release, https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065381047,https://api.github.com/repos/simonw/sqlite-utils/issues/414,1065381047,IC_kwDOCGYnMM4_gGy3,9599,simonw,2022-03-11T18:38:27Z,2022-03-11T18:38:27Z,OWNER,"OK that fixed it here: https://sqlite-utils.datasette.io/en/stable/changelog.html I'm going to trigger a rebuild of `3.25.1` too: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166731361,I forgot to include the changelog in the 3.25.1 release, https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065380286,https://api.github.com/repos/simonw/sqlite-utils/issues/414,1065380286,IC_kwDOCGYnMM4_gGm-,9599,simonw,2022-03-11T18:37:23Z,2022-03-11T18:37:23Z,OWNER,"On ReadTheDocs that triggered a new `stable` build but it didn't seem to trigger a new build of `3.25.1`: https://readthedocs.org/projects/sqlite-utils/builds/ ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166731361,I forgot to include the changelog in the 3.25.1 release, https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065379528,https://api.github.com/repos/simonw/sqlite-utils/issues/414,1065379528,IC_kwDOCGYnMM4_gGbI,9599,simonw,2022-03-11T18:36:17Z,2022-03-11T18:36:17Z,OWNER,"I created a new tag and release: https://github.com/simonw/sqlite-utils/releases/tag/3.25.1 And I cancelled the publish workflow: https://github.com/simonw/sqlite-utils/actions/runs/1970415399","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166731361,I forgot to include the changelog in the 3.25.1 release, https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065378902,https://api.github.com/repos/simonw/sqlite-utils/issues/414,1065378902,IC_kwDOCGYnMM4_gGRW,9599,simonw,2022-03-11T18:35:26Z,2022-03-11T18:35:26Z,OWNER,I deleted both the release and the tag from GitHub using the web interface.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166731361,I forgot to include the changelog in the 3.25.1 release, https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065377926,https://api.github.com/repos/simonw/sqlite-utils/issues/414,1065377926,IC_kwDOCGYnMM4_gGCG,9599,simonw,2022-03-11T18:34:05Z,2022-03-11T18:34:05Z,OWNER,"Two options: - Delete and recreate the release on GitHub, triggering it to be fixed on Read The Docs (as the `stable` version) - but cancel the push to PyPI, since that platform doesn't allow package versions to be over-written and in this case since the changelog file isn't included in the PyPI package there should be no change at all - Push a 3.25.2 release. I'm going to try and do that first option.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166731361,I forgot to include the changelog in the 3.25.1 release, https://github.com/simonw/sqlite-utils/issues/413#issuecomment-1065357081,https://api.github.com/repos/simonw/sqlite-utils/issues/413,1065357081,IC_kwDOCGYnMM4_gA8Z,9599,simonw,2022-03-11T18:07:10Z,2022-03-11T18:07:10Z,OWNER,I'm really happy with this improvement.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166587040,Display autodoc type information more legibly, https://github.com/simonw/sqlite-utils/issues/413#issuecomment-1065345515,https://api.github.com/repos/simonw/sqlite-utils/issues/413,1065345515,IC_kwDOCGYnMM4_f-Hr,9599,simonw,2022-03-11T17:52:22Z,2022-03-11T17:52:22Z,OWNER,"Well this is a huge improvement! https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Table.insert I'm not crazy about the `extracts=` thing though - I wonder if there's a neat way to customize that to be less verbose?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166587040,Display autodoc type information more legibly, https://github.com/simonw/sqlite-utils/issues/413#issuecomment-1065249883,https://api.github.com/repos/simonw/sqlite-utils/issues/413,1065249883,IC_kwDOCGYnMM4_fmxb,9599,simonw,2022-03-11T16:03:35Z,2022-03-11T16:03:35Z,OWNER,"Applying this change fixes that: ```diff diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py index 3bc528f..2a79711 100644 --- a/sqlite_utils/db.py +++ b/sqlite_utils/db.py @@ -2293,18 +2293,18 @@ class Table(Queryable): """""" Apply conversion function ``fn`` to every value in the specified columns. - - ``columns`` - a single column or list of string column names to convert. - - ``fn`` - a callable that takes a single argument, ``value``, and returns it converted. - - ``output`` - optional string column name to write the results to (defaults to the input column). - - ``output_type`` - if the output column needs to be created, this is the type that will be used + :param columns: a single column or list of string column names to convert. + :param fn: a callable that takes a single argument, ``value``, and returns it converted. + :param output: optional string column name to write the results to (defaults to the input column). + :param output_type: if the output column needs to be created, this is the type that will be used for the new column. - - ``drop`` - boolean, should the original column be dropped once the conversion is complete? - - ``multi`` - boolean, if ``True`` the return value of ``fn(value)`` will be expected to be a + :param drop: boolean, should the original column be dropped once the conversion is complete? + :param multi: boolean, if ``True`` the return value of ``fn(value)`` will be expected to be a dictionary, and new columns will be created for each key of that dictionary. - - ``where`` - a SQL fragment to use as a ``WHERE`` clause to limit the rows to which the conversion + :param where: a SQL fragment to use as a ``WHERE`` clause to limit the rows to which the conversion is applied, for example ``age > ?`` or ``age > :age``. - - ``where_args`` - a list of arguments (if using ``?``) or a dictionary (if using ``:age``). - - ``show_progress`` - boolean, should a progress bar be displayed? + :param where_args: a list of arguments (if using ``?``) or a dictionary (if using ``:age``). + :param show_progress: boolean, should a progress bar be displayed? See :ref:`python_api_convert`. """""" ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166587040,Display autodoc type information more legibly, https://github.com/simonw/sqlite-utils/issues/413#issuecomment-1065247619,https://api.github.com/repos/simonw/sqlite-utils/issues/413,1065247619,IC_kwDOCGYnMM4_fmOD,9599,simonw,2022-03-11T16:01:20Z,2022-03-11T16:01:20Z,OWNER,"Definitely an improvement! It does highlight that I'm not currently using the `:param XXX: description` syntax though, which should move my descriptions of each parameter into that generated list.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166587040,Display autodoc type information more legibly, https://github.com/simonw/sqlite-utils/issues/413#issuecomment-1065245831,https://api.github.com/repos/simonw/sqlite-utils/issues/413,1065245831,IC_kwDOCGYnMM4_flyH,9599,simonw,2022-03-11T15:59:14Z,2022-03-11T15:59:14Z,OWNER,"Hint from https://twitter.com/AdamChainz/status/1502311047612575745 > Try: > > `autodoc_typehints = 'description'` > > For a list-of-arguments format > > https://sphinx-doc.org/en/master/usage/extensions/autodoc.html#confval-autodoc_typehints","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166587040,Display autodoc type information more legibly, https://github.com/simonw/datasette/issues/1655#issuecomment-1062445113,https://api.github.com/repos/simonw/datasette/issues/1655,1062445113,IC_kwDOBm6k_c4_U6A5,9599,simonw,2022-03-09T01:01:24Z,2022-03-09T01:01:24Z,OWNER,"https://labordata.bunkum.us/-/settings shows `max_returned_rows` had been increased to 5,000 for that instance - the default of 1,000 would help a bit here. Any thoughts on how Datasette could handle this kind of thing better?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1163369515,query result page is using 400mb of browser memory 40x size of html page and 400x size of csv data, https://github.com/simonw/datasette/issues/932#issuecomment-1061891851,https://api.github.com/repos/simonw/datasette/issues/932,1061891851,IC_kwDOBm6k_c4_Sy8L,9599,simonw,2022-03-08T15:20:48Z,2022-03-08T15:20:48Z,OWNER,Made a start on this here: https://datasette.io/tutorials ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",678760988,End-user documentation, https://github.com/simonw/datasette/issues/1651#issuecomment-1061359915,https://api.github.com/repos/simonw/datasette/issues/1651,1061359915,IC_kwDOBm6k_c4_QxEr,9599,simonw,2022-03-08T03:08:14Z,2022-03-08T03:09:24Z,OWNER,"A lot of the code complexity here is caused by `DataView` ([here](https://github.com/simonw/datasette/blob/c5791156d92615f25696ba93dae5bb2dcc192c98/datasette/views/base.py#L182-L669)), which has the logic for CSV streaming and plugin formats such that it can be shared between tables and custom queries. It would be good to get rid of that subclassed shared code, figure out how to do it via a utility function instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161584460,Get rid of the no-longer necessary ?_format=json hack for tables called x.json, https://github.com/simonw/datasette/issues/1645#issuecomment-1061355871,https://api.github.com/repos/simonw/datasette/issues/1645,1061355871,IC_kwDOBm6k_c4_QwFf,9599,simonw,2022-03-08T02:59:28Z,2022-03-08T02:59:28Z,OWNER,"Hah, found a TODO about this: https://github.com/simonw/datasette/blob/c5791156d92615f25696ba93dae5bb2dcc192c98/datasette/app.py#L997-L999","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 1, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1154399841,"Sensible `cache-control` headers for static assets, including those served by plugins", https://github.com/simonw/datasette/issues/647#issuecomment-1061282743,https://api.github.com/repos/simonw/datasette/issues/647,1061282743,IC_kwDOBm6k_c4_QeO3,9599,simonw,2022-03-08T00:32:34Z,2022-03-08T00:32:47Z,OWNER,It would be neat if the plugin could spot old-style hyphen hash URLs (maybe on 404) and redirect those too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",531755959,Move hashed URL mode out to a plugin, https://github.com/simonw/datasette/issues/647#issuecomment-1061276646,https://api.github.com/repos/simonw/datasette/issues/647,1061276646,IC_kwDOBm6k_c4_Qcvm,9599,simonw,2022-03-08T00:22:11Z,2022-03-08T00:22:11Z,OWNER,I'm now convinced this is feasible enough that it's worth doing in time for Datasette 1.0.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",531755959,Move hashed URL mode out to a plugin, https://github.com/simonw/datasette/issues/647#issuecomment-1061276399,https://api.github.com/repos/simonw/datasette/issues/647,1061276399,IC_kwDOBm6k_c4_Qcrv,9599,simonw,2022-03-08T00:21:47Z,2022-03-08T00:21:47Z,OWNER,"This seems to do the job: ```python @hookimpl def startup(datasette): for name, database in datasette.databases.items(): if database.hash: new_name = ""{}_{}"".format(name, database.hash[:7]) del datasette.databases[name] datasette.databases[new_name] = database ``` Would have to teach the rest of the plugin to split on `_` and to only redirect if the user seems to be hitting the URL for an old hash after which Datasette has been restarted with an updated database.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",531755959,Move hashed URL mode out to a plugin, https://github.com/simonw/datasette/issues/647#issuecomment-1061272544,https://api.github.com/repos/simonw/datasette/issues/647,1061272544,IC_kwDOBm6k_c4_Qbvg,9599,simonw,2022-03-08T00:14:42Z,2022-03-08T00:14:42Z,OWNER,Maybe the plugin should interfere with `datasette.databases` on startup and change the registered name for each one?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",531755959,Move hashed URL mode out to a plugin, https://github.com/simonw/datasette/issues/647#issuecomment-1061267615,https://api.github.com/repos/simonw/datasette/issues/647,1061267615,IC_kwDOBm6k_c4_Qaif,9599,simonw,2022-03-08T00:05:43Z,2022-03-08T00:05:43Z,OWNER,"Built a prototype of that plugin: ```python from datasette import hookimpl from functools import wraps @hookimpl def asgi_wrapper(datasette): def wrap_with_hashed_urls(app): @wraps(app) async def hashed_urls(scope, receive, send): # Only triggers on pages with a path not starting in /-/ # and where the first page component matches a database name if scope.get(""type"") != ""http"": await app(scope, receive, send) return path = scope[""path""].lstrip(""/"") if not path or path.startswith(""-/""): await app(scope, receive, send) return potential_database = path.split(""/"")[0] # It may or may not be already dbname~hash if ""~"" in potential_database: db_name, hash = potential_database.split(""~"", 1) else: db_name = potential_database hash = """" # Is db_name a database we have a hash for? try: db = datasette.get_database(db_name) except KeyError: await app(scope, receive, send) return if db.hash is not None: # TODO: make sure db.hash is documented if db.hash[:7] != hash: # Send a redirect path_bits = path.split(""/"") new_path = ""/"" + ""/"".join([""{}-{}"".format(db_name, db.hash[:7])] + path_bits[1:]) if scope.get(""query_string""): new_path += ""?"" + scope[""query_string""].decode(""latin-1"") await send({ ""type"": ""http.response.start"", ""status"": 302, ""headers"": [ [b""location"", new_path.encode(""latin1"")] ], }) await send({""type"": ""http.response.body"", ""body"": b""""}) return else: # Add a far-future cache header async def wrapped_send(event): if event[""type""] == ""http.response.start"": original_headers = event.get(""headers"") or [] event = { ""type"": event[""type""], ""status"": event[""status""], ""headers"": original_headers + [ [b""Cache-Control"", b""max-age=31536000""] ], } await send(event) await app(scope, receive, wrapped_send) return await app(scope, receive, send) return hashed_urls return wrap_with_hashed_urls ``` One catch: it doesn't affect the way URLs are generated - so every internal link within Datasette links to the non-hash version and then triggers a 302 redirect to the hashed version.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",531755959,Move hashed URL mode out to a plugin, https://github.com/simonw/datasette/issues/647#issuecomment-1061226942,https://api.github.com/repos/simonw/datasette/issues/647,1061226942,IC_kwDOBm6k_c4_QQm-,9599,simonw,2022-03-07T23:00:06Z,2022-03-07T23:00:06Z,OWNER,"This needs to take into account the changes made here: - #1439 In the new encoding scheme, `-` has a special meaning in a table name: https://docs.datasette.io/en/latest/internals.html#dash-encoding I think `~` is the right character to use to separate a database name from its hash. `~` should be a URL safe character according to Python's implementation of percent-encoding, see comment here: https://github.com/simonw/datasette/blob/c5791156d92615f25696ba93dae5bb2dcc192c98/datasette/utils/__init__.py#L1146-L1152 So the plugin could check for `dbname~hash` and react based on that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",531755959,Move hashed URL mode out to a plugin, https://github.com/simonw/datasette/issues/1651#issuecomment-1061223822,https://api.github.com/repos/simonw/datasette/issues/1651,1061223822,IC_kwDOBm6k_c4_QP2O,9599,simonw,2022-03-07T22:54:54Z,2022-03-07T22:54:54Z,OWNER,"I'm going to do a review of how URL routing works at the moment for the various views. I edited down [the full list](https://github.com/simonw/datasette/blob/c5791156d92615f25696ba93dae5bb2dcc192c98/datasette/app.py#L997-L1107) a bit - these are the most relevant: ```python add_route(IndexView.as_view(self), r""/(?P(\.jsono?)?$)"") add_route( DatabaseView.as_view(self), r""/(?P[^/]+?)(?P"" + renderer_regex + r""|.jsono|\.csv)?$"", ) add_route( TableView.as_view(self), r""/(?P[^/]+)/(?P[^/]+?$)"", ) add_route( RowView.as_view(self), r""/(?P[^/]+)/(?P[^/]+?)/(?P[^/]+?)(?P"" + renderer_regex + r"")?$"", ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161584460,Get rid of the no-longer necessary ?_format=json hack for tables called x.json, https://github.com/simonw/datasette/issues/1654#issuecomment-1061197133,https://api.github.com/repos/simonw/datasette/issues/1654,1061197133,IC_kwDOBm6k_c4_QJVN,9599,simonw,2022-03-07T22:19:35Z,2022-03-07T22:19:35Z,OWNER,"Also now live on https://datasette.io ![CleanShot 2022-03-07 at 14 18 30@2x](https://user-images.githubusercontent.com/9599/157127424-805b3166-f0a8-4fac-be87-c055740af580.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161969891,Adopt a code of conduct, https://github.com/simonw/datasette/issues/1654#issuecomment-1061184206,https://api.github.com/repos/simonw/datasette/issues/1654,1061184206,IC_kwDOBm6k_c4_QGLO,9599,simonw,2022-03-07T22:04:51Z,2022-03-07T22:04:51Z,OWNER,I'm going to add this to the main Datasette repo (done) and the `datasette.io` website too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161969891,Adopt a code of conduct, https://github.com/simonw/datasette/issues/1654#issuecomment-1061182132,https://api.github.com/repos/simonw/datasette/issues/1654,1061182132,IC_kwDOBm6k_c4_QFq0,9599,simonw,2022-03-07T22:02:43Z,2022-03-07T22:02:43Z,OWNER,"Neat, GitHub have a template for this https://github.com/simonw/datasette/community/code-of-conduct/new?template=contributor-covenant","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161969891,Adopt a code of conduct, https://github.com/simonw/datasette/issues/1654#issuecomment-1061181530,https://api.github.com/repos/simonw/datasette/issues/1654,1061181530,IC_kwDOBm6k_c4_QFha,9599,simonw,2022-03-07T22:02:06Z,2022-03-07T22:02:06Z,OWNER,https://docs.github.com/en/communities/setting-up-your-project-for-healthy-contributions/adding-a-code-of-conduct-to-your-project says this should be called `CODE_OF_CONDUCT.md` in order for GitHub to pick it up.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161969891,Adopt a code of conduct, https://github.com/simonw/datasette/issues/1654#issuecomment-1061181089,https://api.github.com/repos/simonw/datasette/issues/1654,1061181089,IC_kwDOBm6k_c4_QFah,9599,simonw,2022-03-07T22:01:38Z,2022-03-07T22:01:38Z,OWNER,"I'm going to use the [widely adopted](https://www.contributor-covenant.org/adopters/) Contributor Covenant: https://www.contributor-covenant.org/version/1/4/code-of-conduct/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161969891,Adopt a code of conduct, https://github.com/simonw/datasette/issues/1651#issuecomment-1061170897,https://api.github.com/repos/simonw/datasette/issues/1651,1061170897,IC_kwDOBm6k_c4_QC7R,9599,simonw,2022-03-07T21:48:35Z,2022-03-07T21:48:35Z,OWNER,"My attempts to simplify `get_format()` keep resulting in errors like this one: ``` File ""/Users/simon/Dropbox/Development/datasette/datasette/views/base.py"", line 474, in view_get response_or_template_contexts = await self.data( TypeError: TableView.data() missing 1 required positional argument: 'table' ``` I really need to clean this up.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161584460,Get rid of the no-longer necessary ?_format=json hack for tables called x.json, https://github.com/simonw/datasette/issues/1651#issuecomment-1061169528,https://api.github.com/repos/simonw/datasette/issues/1651,1061169528,IC_kwDOBm6k_c4_QCl4,9599,simonw,2022-03-07T21:47:01Z,2022-03-07T21:47:01Z,OWNER,"Wow, this code is difficult to follow! Look at this bit inside the `get_format()` method: https://github.com/simonw/datasette/blob/bb499942c15c4e2cfa4b6afab8f8debe5948c009/datasette/views/base.py#L469-L478 That's modifying the arguments that were extracted from the path by the routing regular expressions to have `table` as ` dash-decoded value! So calling `.get_format()` has the side effect of decoding the table names for you. Nasty.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161584460,Get rid of the no-longer necessary ?_format=json hack for tables called x.json, https://github.com/simonw/datasette/issues/1653#issuecomment-1061150672,https://api.github.com/repos/simonw/datasette/issues/1653,1061150672,IC_kwDOBm6k_c4_P9_Q,9599,simonw,2022-03-07T21:23:39Z,2022-03-07T21:23:39Z,OWNER,"There may be a short-term fix for this: table view could start accepting a `?_sort_sql=SQLfragment` parameter, similar to the `?_where=` parameter described here: https://docs.datasette.io/en/stable/json_api.html#special-table-arguments That fragment could then be pre-populated in `metadata`. Makes me think maybe that `?_where=` should be optionally settable in metadata too?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161937073,Mechanism to default a table to sorting by multiple columns, https://github.com/simonw/datasette/issues/1653#issuecomment-1061148807,https://api.github.com/repos/simonw/datasette/issues/1653,1061148807,IC_kwDOBm6k_c4_P9iH,9599,simonw,2022-03-07T21:21:23Z,2022-03-07T21:21:23Z,OWNER,"This is currently blocked on the fact that Datasette doesn't have a mechanism for sorting by more than one column: - #197","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161937073,Mechanism to default a table to sorting by multiple columns, https://github.com/simonw/datasette/issues/1651#issuecomment-1061053094,https://api.github.com/repos/simonw/datasette/issues/1651,1061053094,IC_kwDOBm6k_c4_PmKm,9599,simonw,2022-03-07T19:29:01Z,2022-03-07T19:29:01Z,OWNER,"I found an obscure bug in #1650 which I can fix with this too. The following test should pass: ```python @pytest.mark.parametrize( ""path,expected"", ( ( ""/fivethirtyeight/twitter-ratio%2Fsenators"", ""/fivethirtyeight/twitter-2Dratio-2Fsenators"", ), ( ""/fixtures/table%2Fwith%2Fslashes.csv"", ""/fixtures/table-2Fwith-2Fslashes-2Ecsv"", ), # query string should be preserved (""/foo/bar%2Fbaz?id=5"", ""/foo/bar-2Fbaz?id=5""), ), ) def test_redirect_percent_encoding_to_dash_encoding(app_client, path, expected): response = app_client.get(path) assert response.status == 302 assert response.headers[""location""] == expected ``` It currently fails like this: ``` > assert response.headers[""location""] == expected E AssertionError: assert '/fixtures/table-2Fwith-2Fslashes.csv?_nofacet=1&_nocount=1' == '/fixtures/table-2Fwith-2Fslashes-2Ecsv' E - /fixtures/table-2Fwith-2Fslashes-2Ecsv E + /fixtures/table-2Fwith-2Fslashes.csv?_nofacet=1&_nocount=1 ``` Because the logic in that `get_format()` function notices that the table exists, and then weird things happen here: https://github.com/simonw/datasette/blob/1baa030eca375f839f3471237547ab403523e643/datasette/views/base.py#L288-L303 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161584460,Get rid of the no-longer necessary ?_format=json hack for tables called x.json, https://github.com/simonw/datasette/issues/1650#issuecomment-1061041034,https://api.github.com/repos/simonw/datasette/issues/1650,1061041034,IC_kwDOBm6k_c4_PjOK,9599,simonw,2022-03-07T19:16:51Z,2022-03-07T19:16:51Z,OWNER,"Here's the problem: https://github.com/simonw/datasette/blob/020effe47bf89f35182960a9645f2383a42ebd54/datasette/utils/__init__.py#L1173-L1175 Which is called here: https://github.com/simonw/datasette/blob/1baa030eca375f839f3471237547ab403523e643/datasette/views/base.py#L469-L473 So `table%2Fwith%2Fslashes` ends up decoded as if it was using dash encoding.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160750713,Implement redirects from old % encoding to new dash encoding, https://github.com/simonw/datasette/issues/1650#issuecomment-1061038414,https://api.github.com/repos/simonw/datasette/issues/1650,1061038414,IC_kwDOBm6k_c4_PilO,9599,simonw,2022-03-07T19:14:04Z,2022-03-07T19:14:04Z,OWNER,"The problem seems to be that `http://127.0.0.1:8002/fixtures/table%2Fwith%2Fslashes.csv` doesn't result in a 404 at all. If it did, it would be redirected.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160750713,Implement redirects from old % encoding to new dash encoding, https://github.com/simonw/datasette/issues/1439#issuecomment-1060870237,https://api.github.com/repos/simonw/datasette/issues/1439,1060870237,IC_kwDOBm6k_c4_O5hd,9599,simonw,2022-03-07T16:19:22Z,2022-03-07T16:19:22Z,OWNER,"I didn't need to do any of the fancy regular expression routing stuff after all, since the new dash encoding format avoids using `/` so a simple `[^/]+` can capture the correct segments from the URL.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1650#issuecomment-1060864823,https://api.github.com/repos/simonw/datasette/issues/1650,1060864823,IC_kwDOBm6k_c4_O4M3,9599,simonw,2022-03-07T16:14:33Z,2022-03-07T16:14:33Z,OWNER,Same problem here: https://fivethirtyeight.datasettes.com/fivethirtyeight/ahca-2Dpolls%2Fahca_polls should redirect but doesn't.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160750713,Implement redirects from old % encoding to new dash encoding, https://github.com/simonw/datasette/issues/1650#issuecomment-1060863311,https://api.github.com/repos/simonw/datasette/issues/1650,1060863311,IC_kwDOBm6k_c4_O31P,9599,simonw,2022-03-07T16:13:17Z,2022-03-07T16:13:17Z,OWNER,"This doesn't seem to work. https://latest.datasette.io/fixtures/table%2Fwith%2Fslashes.csv should be redirecting now that this is deployed - which it is, because https://latest.datasette.io/-/versions shows 644d25d1de78a36b105cca479e7b3e4375a6eadc - but I'm not getting that redirect.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160750713,Implement redirects from old % encoding to new dash encoding, https://github.com/simonw/datasette/issues/1651#issuecomment-1060853226,https://api.github.com/repos/simonw/datasette/issues/1651,1060853226,IC_kwDOBm6k_c4_O1Xq,9599,simonw,2022-03-07T16:04:26Z,2022-03-07T16:04:26Z,OWNER,"Here's the relevant code: https://github.com/simonw/datasette/blob/1baa030eca375f839f3471237547ab403523e643/datasette/utils/__init__.py#L753-L772 https://github.com/simonw/datasette/blob/1baa030eca375f839f3471237547ab403523e643/datasette/views/base.py#L451-L479","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161584460,Get rid of the no-longer necessary ?_format=json hack for tables called x.json, https://github.com/simonw/datasette/issues/1650#issuecomment-1060836262,https://api.github.com/repos/simonw/datasette/issues/1650,1060836262,IC_kwDOBm6k_c4_OxOm,9599,simonw,2022-03-07T15:52:09Z,2022-03-07T15:52:09Z,OWNER,"This is a bit tricky. I tried this, sending a redirect only if a 404 happens: ```diff diff --git a/datasette/app.py b/datasette/app.py index 8c5480c..420664c 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -1211,6 +1211,10 @@ class DatasetteRouter: return await self.handle_404(request, send) async def handle_404(self, request, send, exception=None): + # If path contains % encoding, redirect to dash encoding + if '%' in request.scope[""path""]: + await asgi_send_redirect(send, request.scope[""path""].replace(""%"", ""-"")) + return # If URL has a trailing slash, redirect to URL without it path = request.scope.get( ""raw_path"", request.scope[""path""].encode(""utf8"") ``` But this URL didn't work: - http://127.0.0.1:8001/fivethirtyeight/twitter-ratio%2Fsenators I was expecting that to redirect to this page: - http://127.0.0.1:8001/fivethirtyeight/twitter-2Dratio-2Fsenators But instead it took me to another 404: - http://127.0.0.1:8001/fivethirtyeight/twitter-ratio%2Fsenators This is because that URL contains both a %-escaped `/` AND a plain `-` - which was not escaped in the old system but is escaped in the new system.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160750713,Implement redirects from old % encoding to new dash encoding, https://github.com/simonw/datasette/pull/1648#issuecomment-1060067031,https://api.github.com/repos/simonw/datasette/issues/1648,1060067031,IC_kwDOBm6k_c4_L1bX,9599,simonw,2022-03-06T23:50:40Z,2022-03-06T23:58:31Z,OWNER,"I may have to do extra work here ```python def database(self, database, format=None): db = self.ds.databases[database] if self.ds.setting(""hash_urls"") and db.hash: path = self.path( f""{dash_encode(database)}-{db.hash[:HASH_LENGTH]}"", format=format ) else: path = self.path(dash_encode(database), format=format) return path ``` The URLs that incorporate a hash have a `dbname-hash` format - will that `-` in the middle there mess up the dash decoding mechanism? I think it will. Might be able to solve that like so: ```python async def resolve_db_name(self, request, db_name, **kwargs): hash = None name = None decoded_name = dash_decode(db_name) if decoded_name not in self.ds.databases and ""-"" in db_name: # No matching DB found, maybe it's a name-hash? name_bit, hash_bit = db_name.rsplit(""-"", 1) if dash_decode(name_bit) not in self.ds.databases: raise NotFound(f""Database not found: {name}"") else: name = dash_decode(name_bit) hash = hash_bit else: name = decoded_name ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160432941,Use dash encoding for table names and row primary keys in URLs, https://github.com/simonw/datasette/pull/1648#issuecomment-1060065736,https://api.github.com/repos/simonw/datasette/issues/1648,1060065736,IC_kwDOBm6k_c4_L1HI,9599,simonw,2022-03-06T23:43:00Z,2022-03-06T23:43:11Z,OWNER,"> * Maybe use dash encoding for database name too? Yes, I'm going to do this. At the moment if a DB file is called `fixx%tures.db` when you run it in Datasette the path is `/fix%2525tures` - which is liable to break.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160432941,Use dash encoding for table names and row primary keys in URLs, https://github.com/simonw/datasette/pull/1648#issuecomment-1060056510,https://api.github.com/repos/simonw/datasette/issues/1648,1060056510,IC_kwDOBm6k_c4_Ly2-,9599,simonw,2022-03-06T23:02:05Z,2022-03-06T23:04:24Z,OWNER,"Just spotted this: https://github.com/simonw/datasette/blob/de810f49cc57a4f88e4a1553d26c579253ce4531/datasette/views/base.py#L203-L216 Maybe the db name should use dash encoding too? If so, relevant code includes this bit: https://github.com/simonw/datasette/blob/de810f49cc57a4f88e4a1553d26c579253ce4531/datasette/url_builder.py#L30-L38","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160432941,Use dash encoding for table names and row primary keys in URLs, https://github.com/simonw/datasette/pull/1648#issuecomment-1060044592,https://api.github.com/repos/simonw/datasette/issues/1648,1060044592,IC_kwDOBm6k_c4_Lv8w,9599,simonw,2022-03-06T21:42:35Z,2022-03-06T21:42:35Z,OWNER,"For consistency, I'm going to change how `?_next=` tokens work too. Right now they work like this: https://github.com/simonw/datasette/blob/de810f49cc57a4f88e4a1553d26c579253ce4531/datasette/views/table.py#L501-L507 https://github.com/simonw/datasette/blob/de810f49cc57a4f88e4a1553d26c579253ce4531/datasette/utils/__init__.py#L114-L116 I'm going to change those to use dash-encoding instead. I considered looking for `%` in those values and replacing that as `-` too, but since Datasette isn't 1.0 yet I'm going to risk breaking any pagination tokens that people might have saved away somewhere!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160432941,Use dash encoding for table names and row primary keys in URLs, https://github.com/simonw/datasette/issues/1439#issuecomment-1060044007,https://api.github.com/repos/simonw/datasette/issues/1439,1060044007,IC_kwDOBm6k_c4_Lvzn,9599,simonw,2022-03-06T21:38:15Z,2022-03-06T21:38:15Z,OWNER,"Test: https://github.com/simonw/datasette/blob/d2e3fe3facf0ed0abf8b00cd54463af90dd6904d/tests/test_utils.py#L651-L666 One big advantage to this scheme is that redirecting old links to `%2F` pages (e.g. https://fivethirtyeight.datasettes.com/fivethirtyeight/twitter-ratio%2Fsenators) is easy - if you see a `%` in the `raw_path`, redirect to that page with the `%` replaced by `-`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/pull/1648#issuecomment-1060034562,https://api.github.com/repos/simonw/datasette/issues/1648,1060034562,IC_kwDOBm6k_c4_LtgC,9599,simonw,2022-03-06T20:36:12Z,2022-03-06T20:36:12Z,OWNER,"Updated documentation: ![image](https://user-images.githubusercontent.com/9599/156941171-89778c12-41bc-4951-97f2-ecc805025a53.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160432941,Use dash encoding for table names and row primary keys in URLs, https://github.com/simonw/datasette/pull/1648#issuecomment-1060016221,https://api.github.com/repos/simonw/datasette/issues/1648,1060016221,IC_kwDOBm6k_c4_LpBd,9599,simonw,2022-03-06T18:37:59Z,2022-03-06T18:37:59Z,OWNER,"Change of plan: based on extensive conversations on Twitter - see https://github.com/simonw/datasette/issues/1439#issuecomment-1059851259 - I'm going to try a variant of this which is basically percent-encoding but with a hyphen instead of a percent symbol. Reason being that the current scheme doesn't handle the case of `%` being part of the table name, which could cause weird breakage due to some proxies decoding percent encoding before it gets to Datasette.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160432941,Use dash encoding for table names and row primary keys in URLs, https://github.com/simonw/datasette/issues/1439#issuecomment-1059903309,https://api.github.com/repos/simonw/datasette/issues/1439,1059903309,IC_kwDOBm6k_c4_LNdN,9599,simonw,2022-03-06T06:17:51Z,2022-03-06T06:17:51Z,OWNER,"Suggestion from a conversation with Seth Michael Larson: it would be neat if plugins could easily integrate with whatever scheme this ends up using, maybe with the `/db/table/-/plugin-name` standardized pattern or similar. Making it easy for plugins to do the right, consistent thing is a good idea.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/pull/1589#issuecomment-1059875687,https://api.github.com/repos/simonw/datasette/issues/1589,1059875687,IC_kwDOBm6k_c4_LGtn,9599,simonw,2022-03-06T01:58:25Z,2022-03-06T01:58:25Z,OWNER,Thanks for catching this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1098275181,Typo in docs about default redirect status code, https://github.com/simonw/datasette/issues/1439#issuecomment-1059864154,https://api.github.com/repos/simonw/datasette/issues/1439,1059864154,IC_kwDOBm6k_c4_LD5a,9599,simonw,2022-03-06T00:59:04Z,2022-03-06T00:59:04Z,OWNER,"Needs more testing, but this seems to work for decoding the percent-escaped-with-dashes format: `urllib.parse.unquote(s.replace('-', '%'))`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1059855418,https://api.github.com/repos/simonw/datasette/issues/1439,1059855418,IC_kwDOBm6k_c4_LBw6,9599,simonw,2022-03-06T00:00:53Z,2022-03-06T00:04:18Z,OWNER,"```python _ESCAPE_SAFE = frozenset( b'ABCDEFGHIJKLMNOPQRSTUVWXYZ' b'abcdefghijklmnopqrstuvwxyz' b'0123456789_' ) # I removed b'.-~') class Quoter(dict): # Keeps a cache internally, via __missing__ def __missing__(self, b): # Handle a cache miss. Store quoted string in cache and return. res = chr(b) if b in _ESCAPE_SAFE else '-{:02X}'.format(b) self[b] = res return res quoter = Quoter().__getitem__ ''.join([quoter(char) for char in b'foo/bar.csv']) # 'foo-2Fbar-2Ecsv' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1059854864,https://api.github.com/repos/simonw/datasette/issues/1439,1059854864,IC_kwDOBm6k_c4_LBoQ,9599,simonw,2022-03-05T23:59:05Z,2022-03-05T23:59:05Z,OWNER,"OK, for that percentage thing: the Python core implementation of URL percentage escaping deliberately ignores two of the characters we want to escape: `.` and `-`: https://github.com/python/cpython/blob/6927632492cbad86a250aa006c1847e03b03e70b/Lib/urllib/parse.py#L780-L783 ```python _ALWAYS_SAFE = frozenset(b'ABCDEFGHIJKLMNOPQRSTUVWXYZ' b'abcdefghijklmnopqrstuvwxyz' b'0123456789' b'_.-~') ``` It also defaults to skipping `/` (passed as a `safe=` parameter to various things). I'm going to try borrowing and modifying the core of the Python implementation: https://github.com/python/cpython/blob/6927632492cbad86a250aa006c1847e03b03e70b/Lib/urllib/parse.py#L795-L814 ```python class _Quoter(dict): """"""A mapping from bytes numbers (in range(0,256)) to strings. String values are percent-encoded byte values, unless the key < 128, and in either of the specified safe set, or the always safe set. """""" # Keeps a cache internally, via __missing__, for efficiency (lookups # of cached keys don't call Python code at all). def __init__(self, safe): """"""safe: bytes object."""""" self.safe = _ALWAYS_SAFE.union(safe) def __repr__(self): return f"""" def __missing__(self, b): # Handle a cache miss. Store quoted string in cache and return. res = chr(b) if b in self.safe else '%{:02X}'.format(b) self[b] = res return res ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1059853526,https://api.github.com/repos/simonw/datasette/issues/1439,1059853526,IC_kwDOBm6k_c4_LBTW,9599,simonw,2022-03-05T23:49:59Z,2022-03-05T23:49:59Z,OWNER,"I want to try regular percentage encoding, except that it also encodes both the `-` and the `.` characters, AND it uses `-` instead of `%` as the encoding character. Should check what it does with emoji too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1059851259,https://api.github.com/repos/simonw/datasette/issues/1439,1059851259,IC_kwDOBm6k_c4_LAv7,9599,simonw,2022-03-05T23:35:47Z,2022-03-05T23:35:59Z,OWNER,"This [comment from glyph](https://twitter.com/glyph/status/1500244937312329730) got me thinking: > Have you considered replacing % with some other character and then using percent-encoding? What happens if a table name includes a `%` character and that ends up getting mangled by a misbehaving proxy? I should consider `%` in the escaping system too. And maybe go with that suggestion of using percent-encoding directly but with a different character.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1059850369,https://api.github.com/repos/simonw/datasette/issues/1439,1059850369,IC_kwDOBm6k_c4_LAiB,9599,simonw,2022-03-05T23:28:56Z,2022-03-05T23:28:56Z,OWNER,"Lots of great conversations about the dash encoding implementation on Twitter: https://twitter.com/simonw/status/1500228316309061633 @dracos helped me figure out a simpler regex: https://twitter.com/dracos/status/1500236433809973248 `^/(?P[^/]+)/(?P
[^\/\-\.]*|\-/|\-\.|\-\-)*(?P\.\w+)?$` ![image](https://user-images.githubusercontent.com/9599/156903088-c01933ae-4713-4e91-8d71-affebf70b945.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1059836599,https://api.github.com/repos/simonw/datasette/issues/1439,1059836599,IC_kwDOBm6k_c4_K9K3,9599,simonw,2022-03-05T21:52:10Z,2022-03-05T21:52:10Z,OWNER,Blogged about this here: https://simonwillison.net/2022/Mar/5/dash-encoding/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1045069481,https://api.github.com/repos/simonw/datasette/issues/1439,1045069481,IC_kwDOBm6k_c4-Sn6p,9599,simonw,2022-02-18T19:34:41Z,2022-03-05T21:32:22Z,OWNER,"I think I got format extraction working! https://regex101.com/r/A0bW1D/1 ^/(?P[^/]+)/(?P
(?:[^\/\-\.]*|(?:\-/)*|(?:\-\.)*|(?:\-\-)*)*?)(?:(?\w+))?$ I had to make that crazy inner one even more complicated to stop it from capturing `.` that was not part of `-.`. (?:[^\/\-\.]*|(?:\-/)*|(?:\-\.)*|(?:\-\-)*)* Visualized: So now I have a regex which can extract out the dot-encoded table name AND spot if there is an optional `.format` at the end: If I end up using this in Datasette it's going to need VERY comprehensive unit tests and inline documentation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1647#issuecomment-1059823119,https://api.github.com/repos/simonw/datasette/issues/1647,1059823119,IC_kwDOBm6k_c4_K54P,9599,simonw,2022-03-05T19:56:27Z,2022-03-05T19:56:27Z,OWNER,Updated this TIL with extra patterns I figured out: https://til.simonwillison.net/sqlite/ld-preload,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160407071,Test failures with SQLite 3.37.0+ due to column affinity case, https://github.com/simonw/datasette/issues/1439#issuecomment-1059822391,https://api.github.com/repos/simonw/datasette/issues/1439,1059822391,IC_kwDOBm6k_c4_K5s3,9599,simonw,2022-03-05T19:50:12Z,2022-03-05T19:50:12Z,OWNER,I'm going to move this work to a PR.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1059822151,https://api.github.com/repos/simonw/datasette/issues/1439,1059822151,IC_kwDOBm6k_c4_K5pH,9599,simonw,2022-03-05T19:48:35Z,2022-03-05T19:48:35Z,OWNER,Those new docs: https://github.com/simonw/datasette/blob/d1cb73180b4b5a07538380db76298618a5fc46b6/docs/internals.rst#dash-encoding,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1647#issuecomment-1059821674,https://api.github.com/repos/simonw/datasette/issues/1647,1059821674,IC_kwDOBm6k_c4_K5hq,9599,simonw,2022-03-05T19:44:32Z,2022-03-05T19:44:32Z,OWNER,"I thought I'd need to introduce https://dirty-equals.helpmanual.io/types/string/ to help write tests for this, but I think I've found a good alternative that doesn't need a new dependency.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160407071,Test failures with SQLite 3.37.0+ due to column affinity case, https://github.com/simonw/datasette/issues/1647#issuecomment-1059819628,https://api.github.com/repos/simonw/datasette/issues/1647,1059819628,IC_kwDOBm6k_c4_K5Bs,9599,simonw,2022-03-05T19:28:54Z,2022-03-05T19:28:54Z,OWNER,"OK, using that trick worked for testing this: docker run -it -p 8001:8001 ubuntu Then inside that container: apt-get install -y python3 build-essential tcl wget python3-pip git python3.8-venv For each version of SQLite I wanted to test I needed to figure out the tarball URL - for example, for `3.38.0` I navigated to https://www.sqlite.org/src/timeline?t=version-3.38.0 and clicked the ""checkin"" link and copied the tarball link: https://www.sqlite.org/src/tarball/40fa792d/SQLite-40fa792d.tar.gz Then to build it (the `CPPFLAGS` took some trial and error): ``` cd /tmp wget https://www.sqlite.org/src/tarball/40fa792d/SQLite-40fa792d.tar.gz tar -xzvf SQLite-40fa792d.tar.gz cd SQLite-40fa792d CPPFLAGS=""-DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_RTREE=1"" ./configure make ``` Then to test with Datasette: ``` cd /tmp git clone https://github.com/simonw/datasette cd datasette python3 -m venv venv source venv/bin/activate pip install wheel # So bdist_wheel works in next step pip install -e '.[test]' LD_PRELOAD=/tmp/SQLite-40fa792d/.libs/libsqlite3.so pytest ``` After some trial and error I proved that those tests passed with 3.36.0: ``` cd /tmp wget https://www.sqlite.org/src/tarball/5c9a6c06/SQLite-5c9a6c06.tar.gz tar -xzvf SQLite-5c9a6c06.tar.gz cd SQLite-5c9a6c06 CPPFLAGS=""-DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_RTREE=1"" ./configure make cd /tmp/datasette LD_PRELOAD=/tmp/SQLite-5c9a6c06/.libs/libsqlite3.so pytest tests/test_internals_database.py ``` BUT failed with 3.37.0: ``` # 3.37.0 cd /tmp wget https://www.sqlite.org/src/tarball/bd41822c/SQLite-bd41822c.tar.gz tar -xzvf SQLite-bd41822c.tar.gz cd SQLite-bd41822c CPPFLAGS=""-DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_RTREE=1"" ./configure make cd /tmp/datasette LD_PRELOAD=/tmp/SQLite-bd41822c/.libs/libsqlite3.so pytest tests/test_internals_database.py ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160407071,Test failures with SQLite 3.37.0+ due to column affinity case, https://github.com/simonw/datasette/issues/1647#issuecomment-1059807598,https://api.github.com/repos/simonw/datasette/issues/1647,1059807598,IC_kwDOBm6k_c4_K2Fu,9599,simonw,2022-03-05T18:06:56Z,2022-03-05T18:08:00Z,OWNER,"Had a look through the commits in https://github.com/sqlite/sqlite/compare/version-3.37.2...version-3.38.0 but couldn't see anything obvious that might have caused this. Really wish I had a good mechanism for running the test suite against different SQLite versions! May have to revisit this old trick: https://til.simonwillison.net/sqlite/ld-preload","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160407071,Test failures with SQLite 3.37.0+ due to column affinity case, https://github.com/simonw/datasette/issues/1647#issuecomment-1059804577,https://api.github.com/repos/simonw/datasette/issues/1647,1059804577,IC_kwDOBm6k_c4_K1Wh,9599,simonw,2022-03-05T17:49:46Z,2022-03-05T17:49:46Z,OWNER,My best guess is that this is an undocumented change in SQLite 3.38 - I get that test failure with that SQLite version.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160407071,Test failures with SQLite 3.37.0+ due to column affinity case, https://github.com/simonw/datasette/issues/1439#issuecomment-1059802318,https://api.github.com/repos/simonw/datasette/issues/1439,1059802318,IC_kwDOBm6k_c4_K0zO,9599,simonw,2022-03-05T17:34:33Z,2022-03-05T17:34:33Z,OWNER,"Wrote documentation: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059652538,https://api.github.com/repos/simonw/sqlite-utils/issues/412,1059652538,IC_kwDOCGYnMM4_KQO6,9599,simonw,2022-03-05T02:13:17Z,2022-03-05T02:13:17Z,OWNER,"> It looks like the existing `pd.read_sql_query()` method has an optional dependency on SQLAlchemy: > > ``` > ... > import pandas as pd > pd.read_sql_query(db.conn, ""select * from articles"") > # ImportError: Using URI string without sqlalchemy installed. > ``` Hah, no I was wrong about this: SQLAlchemy is not needed for SQLite to work, I just had the arguments the wrong way round: ```python pd.read_sql_query(""select * from articles"", db.conn) # Shows a DateFrame ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160182768,Optional Pandas integration, https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059651306,https://api.github.com/repos/simonw/sqlite-utils/issues/412,1059651306,IC_kwDOCGYnMM4_KP7q,9599,simonw,2022-03-05T02:10:49Z,2022-03-05T02:10:49Z,OWNER,"I could teach `.insert_all()` and `.upsert_all()` to optionally accept a DataFrame. A challenge there is `mypy` - if Pandas is an optional dependency, is it possibly to declare types that accept a Union that includes DataFrame?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160182768,Optional Pandas integration, https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059651056,https://api.github.com/repos/simonw/sqlite-utils/issues/412,1059651056,IC_kwDOCGYnMM4_KP3w,9599,simonw,2022-03-05T02:09:38Z,2022-03-05T02:09:38Z,OWNER,"OK, so reading results from existing `sqlite-utils` into a Pandas DataFrame turns out to be trivial. How about writing a DataFrame to a database table? That feels like it could a lot more useful.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160182768,Optional Pandas integration, https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059650190,https://api.github.com/repos/simonw/sqlite-utils/issues/412,1059650190,IC_kwDOCGYnMM4_KPqO,9599,simonw,2022-03-05T02:04:43Z,2022-03-05T02:04:54Z,OWNER,"To be honest, I'm having second thoughts about this now mainly because the idiom for turning a generator of dicts into a DataFrame is SO simple: ```python df = pd.DataFrame(db.query(""select * from articles"")) ``` Given it's that simple, I'm questioning if there's any value to adding this to `sqlite-utils` at all. This likely becomes a documentation thing instead!","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160182768,Optional Pandas integration, https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059649803,https://api.github.com/repos/simonw/sqlite-utils/issues/412,1059649803,IC_kwDOCGYnMM4_KPkL,9599,simonw,2022-03-05T02:02:41Z,2022-03-05T02:02:41Z,OWNER,"It looks like the existing `pd.read_sql_query()` method has an optional dependency on SQLAlchemy: ``` ... import pandas as pd pd.read_sql_query(db.conn, ""select * from articles"") # ImportError: Using URI string without sqlalchemy installed. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160182768,Optional Pandas integration, https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059649213,https://api.github.com/repos/simonw/sqlite-utils/issues/412,1059649213,IC_kwDOCGYnMM4_KPa9,9599,simonw,2022-03-05T02:00:10Z,2022-03-05T02:00:10Z,OWNER,Requested feedback on Twitter here :https://twitter.com/simonw/status/1499927075930578948,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160182768,Optional Pandas integration, https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059649193,https://api.github.com/repos/simonw/sqlite-utils/issues/412,1059649193,IC_kwDOCGYnMM4_KPap,9599,simonw,2022-03-05T02:00:02Z,2022-03-05T02:00:02Z,OWNER,"Yeah, I imagine there are plenty of ways to do this with Pandas already - I'm opportunistically looking for a way to provide better integration with the rest of the Pandas situation from the work I've done in `sqlite-utils` already. Might be that this isn't worth doing at all.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160182768,Optional Pandas integration, https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059646645,https://api.github.com/repos/simonw/sqlite-utils/issues/412,1059646645,IC_kwDOCGYnMM4_KOy1,9599,simonw,2022-03-05T01:53:10Z,2022-03-05T01:53:10Z,OWNER,I'm not an experienced enough Pandas user to know if this design is right or not. I'm going to leave this open for a while and solicit some feedback.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160182768,Optional Pandas integration, https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059646543,https://api.github.com/repos/simonw/sqlite-utils/issues/412,1059646543,IC_kwDOCGYnMM4_KOxP,9599,simonw,2022-03-05T01:52:47Z,2022-03-05T01:52:47Z,OWNER,"I built a prototype of that second option and it looks pretty good: Here's the `pandas.py` prototype: ```python from .db import Database as _Database, Table as _Table, View as _View import pandas as pd from typing import ( Iterable, Union, Optional, ) class Database(_Database): def query( self, sql: str, params: Optional[Union[Iterable, dict]] = None ) -> pd.DataFrame: return pd.DataFrame(super().query(sql, params)) def table(self, table_name: str, **kwargs) -> Union[""Table"", ""View""]: ""Return a table object, optionally configured with default options."" klass = View if table_name in self.view_names() else Table return klass(self, table_name, **kwargs) class PandasQueryable: def rows_where( self, where: str = None, where_args: Optional[Union[Iterable, dict]] = None, order_by: str = None, select: str = ""*"", limit: int = None, offset: int = None, ) -> pd.DataFrame: return pd.DataFrame( super().rows_where( where, where_args, order_by=order_by, select=select, limit=limit, offset=offset, ) ) class Table(PandasQueryable, _Table): pass class View(PandasQueryable, _View): pass ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160182768,Optional Pandas integration, https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059646247,https://api.github.com/repos/simonw/sqlite-utils/issues/412,1059646247,IC_kwDOCGYnMM4_KOsn,9599,simonw,2022-03-05T01:51:03Z,2022-03-05T01:51:03Z,OWNER,"I considered two ways of doing this. First, have methods such as `db.query_df()` and `table.rows_df` which do the same as `.query()` and `table.rows` but return a DataFrame instead of a generator of dictionaries. Second, have a compatibility class that is imported separately such as: ```python from sqlite_utils.pandas import Database ``` Then have the `.query()` and `.rows` and other similar methods return dataframes.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160182768,Optional Pandas integration, https://github.com/simonw/datasette/issues/1640#issuecomment-1059638778,https://api.github.com/repos/simonw/datasette/issues/1640,1059638778,IC_kwDOBm6k_c4_KM36,9599,simonw,2022-03-05T01:19:00Z,2022-03-05T01:19:00Z,OWNER,"The reason I implemented it like this was to support things like the `curl` progress bar if users decide to serve up large files using the `--static` mechanism. Here's the code that hooks it up to the URL resolver: https://github.com/simonw/datasette/blob/458f03ad3a454d271f47a643f4530bd8b60ddb76/datasette/app.py#L1001-L1005 Which uses this function: https://github.com/simonw/datasette/blob/a6ff123de5464806441f6a6f95145c9a83b7f20b/datasette/utils/asgi.py#L285-L310 One option here would be to support a workaround that looks something like this: http://localhost:8001/my-static/log.txt?_unknown_size=1` The URL routing code could then look out for that `?_unknown_size=1` option and, if it's present, omit the `content-length` header entirely. It's a bit of a cludge, but it would be pretty straight-forward to implement. Would that work for you @broccolihighkicks?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1148725876,"Support static assets where file length may change, e.g. logs", https://github.com/simonw/datasette/issues/1640#issuecomment-1059636420,https://api.github.com/repos/simonw/datasette/issues/1640,1059636420,IC_kwDOBm6k_c4_KMTE,9599,simonw,2022-03-05T01:13:26Z,2022-03-05T01:13:26Z,OWNER,"Hah, this is certainly unexpected. It looks like this is the code in question: https://github.com/simonw/datasette/blob/a6ff123de5464806441f6a6f95145c9a83b7f20b/datasette/utils/asgi.py#L259-L266 You're right: it assumes that the file it is serving won't change length while it is serving it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1148725876,"Support static assets where file length may change, e.g. logs", https://github.com/simonw/datasette/issues/1642#issuecomment-1059635969,https://api.github.com/repos/simonw/datasette/issues/1642,1059635969,IC_kwDOBm6k_c4_KMMB,9599,simonw,2022-03-05T01:11:17Z,2022-03-05T01:11:17Z,OWNER,"`pip install datasette` in a fresh virtual environment doesn't show any warnings. Neither does `pip install -e '.'` in a fresh checkout. Or `pip install -e '.[test]'`. Closing this as can't reproduce.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1152072027,Dependency issue with asgiref and uvicorn, https://github.com/simonw/datasette/issues/1645#issuecomment-1059634688,https://api.github.com/repos/simonw/datasette/issues/1645,1059634688,IC_kwDOBm6k_c4_KL4A,9599,simonw,2022-03-05T01:06:08Z,2022-03-05T01:06:08Z,OWNER,"It sounds like you can workaround this with Varnish configuration for the moment, but I'm going to bump this up the list of things to fix - it's particularly relevant now as I'd like to get a solution in place before Datasette 1.0, since it's likely to be beneficial to plugins and hence should be part of the stable, documented plugin interface.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1154399841,"Sensible `cache-control` headers for static assets, including those served by plugins", https://github.com/simonw/datasette/issues/1645#issuecomment-1059634412,https://api.github.com/repos/simonw/datasette/issues/1645,1059634412,IC_kwDOBm6k_c4_KLzs,9599,simonw,2022-03-05T01:04:53Z,2022-03-05T01:04:53Z,OWNER,"The existing `app_css_hash` already isn't good enough, because I built that before `table.js` existed, and that file should obviously be smartly cached too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1154399841,"Sensible `cache-control` headers for static assets, including those served by plugins", https://github.com/simonw/datasette/issues/1645#issuecomment-1059633902,https://api.github.com/repos/simonw/datasette/issues/1645,1059633902,IC_kwDOBm6k_c4_KLru,9599,simonw,2022-03-05T01:03:06Z,2022-03-05T01:03:06Z,OWNER,"I agree: this is bad. Ideally, content served from `/static/` would apply best practices for static content serving - which to my mind means the following: - Where possible, serve with a far-future cache expiry header and use an asset URL that changes when the file itself changes - For assets without that, support conditional GET to avoid transferring the whole asset if it hasn't changed - Some kind of sensible mechanism for setting cache TTLs on assets that don't have a unique-file-per-version - in particular assets that might be served from plugins. Datasette half-implemented the first of these: if you view source on https://latest.datasette.io/ you'll see it links to `/-/static/app.css?cead5a` - which in the template looks like this: https://github.com/simonw/datasette/blob/dd94157f8958bdfe9f45575add934ccf1aba6d63/datasette/templates/base.html#L5 I had forgotten I had implemented this! Here is how it is calculated: https://github.com/simonw/datasette/blob/458f03ad3a454d271f47a643f4530bd8b60ddb76/datasette/app.py#L510-L516 So `app.css` right now could be safely served with a far-future cache header... only it isn't: ``` ~ % curl -i 'https://latest.datasette.io/-/static/app.css?cead5a' HTTP/2 200 content-type: text/css x-databases: _memory, _internal, fixtures, extra_database x-cloud-trace-context: 9ddc825620eb53d30fc127d1c750f342 date: Sat, 05 Mar 2022 01:01:53 GMT server: Google Frontend content-length: 16178 ``` The larger question though is what to do about other assets. I'm particularly interested in plugin assets, since visualization plugins like `datasette-vega` and `datasette-cluster-map` ship with large amounts of JavaScript and I'd really like that to be sensibly cached by default.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1154399841,"Sensible `cache-control` headers for static assets, including those served by plugins", https://github.com/simonw/sqlite-utils/issues/408#issuecomment-1056001414,https://api.github.com/repos/simonw/sqlite-utils/issues/408,1056001414,IC_kwDOCGYnMM4-8U2G,9599,simonw,2022-03-02T00:20:26Z,2022-03-02T00:20:26Z,OWNER,I need a `db.sqlite_version` property to implement this check.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1145882578,`deterministic=True` fails on versions of SQLite prior to 3.8.3, https://github.com/simonw/sqlite-utils/issues/408#issuecomment-1055996626,https://api.github.com/repos/simonw/sqlite-utils/issues/408,1055996626,IC_kwDOCGYnMM4-8TrS,9599,simonw,2022-03-02T00:12:21Z,2022-03-02T00:12:21Z,OWNER,Here's the SQLite changelog mentioning that it was added in 3.8.3: https://www.sqlite.org/changes.html#version_3_8_3,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1145882578,`deterministic=True` fails on versions of SQLite prior to 3.8.3, https://github.com/simonw/sqlite-utils/issues/408#issuecomment-1055995100,https://api.github.com/repos/simonw/sqlite-utils/issues/408,1055995100,IC_kwDOCGYnMM4-8TTc,9599,simonw,2022-03-02T00:10:41Z,2022-03-02T00:10:41Z,OWNER,"Here's the code in question: https://github.com/simonw/sqlite-utils/blob/521921b849003ed3742338f76f9d47ff3d95eaf3/sqlite_utils/db.py#L384-L394 It's checking for Python 3.8, because that's the version of Python that added the `deterministic=True` option: https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.create_function But from your error message it looks like it should be checking the SQLite version too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1145882578,`deterministic=True` fails on versions of SQLite prior to 3.8.3, https://github.com/simonw/sqlite-utils/issues/408#issuecomment-1055993700,https://api.github.com/repos/simonw/sqlite-utils/issues/408,1055993700,IC_kwDOCGYnMM4-8S9k,9599,simonw,2022-03-02T00:08:10Z,2022-03-02T00:08:10Z,OWNER,"I thought I'd made it so `deterministic=True` would be silently ignored in environments that don't support it, but clearly I missed a case here!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1145882578,`deterministic=True` fails on versions of SQLite prior to 3.8.3, https://github.com/simonw/sqlite-utils/issues/343#issuecomment-1055992544,https://api.github.com/repos/simonw/sqlite-utils/issues/343,1055992544,IC_kwDOCGYnMM4-8Srg,9599,simonw,2022-03-02T00:06:10Z,2022-03-02T00:06:10Z,OWNER,"Updated documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#setting-an-id-based-on-the-hash-of-the-row-contents Documentation for the renamed `utils.hash_record()` function: https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite-utils-utils-hash-record","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1063388037,Provide function to generate hash_id from specified columns, https://github.com/simonw/sqlite-utils/issues/343#issuecomment-1055991226,https://api.github.com/repos/simonw/sqlite-utils/issues/343,1055991226,IC_kwDOCGYnMM4-8SW6,9599,simonw,2022-03-02T00:03:47Z,2022-03-02T00:03:47Z,OWNER,"Oops, broke mypy: ``` sqlite_utils/db.py:2600: error: Incompatible default for argument ""hash_id_columns"" (default has type ""Default"", argument has type ""Optional[Iterable[str]]"") Found 1 error in 1 file (checked 49 source files) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1063388037,Provide function to generate hash_id from specified columns, https://github.com/simonw/sqlite-utils/issues/343#issuecomment-1055855845,https://api.github.com/repos/simonw/sqlite-utils/issues/343,1055855845,IC_kwDOCGYnMM4-7xTl,9599,simonw,2022-03-01T21:04:45Z,2022-03-01T22:43:38Z,OWNER,"I'm going to make that `_hash()` utility function a documented, non-underscore-prefixed function too - called `hash_record()`.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 1, ""eyes"": 0}",1063388037,Provide function to generate hash_id from specified columns, https://github.com/simonw/sqlite-utils/issues/409#issuecomment-1055930639,https://api.github.com/repos/simonw/sqlite-utils/issues/409,1055930639,IC_kwDOCGYnMM4-8DkP,9599,simonw,2022-03-01T22:40:15Z,2022-03-01T22:40:15Z,OWNER,"This test fails and I don't understand why: ```python from sqlite_utils import Database def test_transaction(): db1 = Database(memory_name=""transaction_test"", tracer=print) db2 = Database(memory_name=""transaction_test"", tracer=print) with db1.conn: db1[""t""].insert({""foo"": 1}) assert list(db2[""t""].rows) == [] assert list(db2[""t""].rows) == [{""foo"": 1}] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1149661489,`with db:` for transactions, https://github.com/simonw/sqlite-utils/pull/410#issuecomment-1055856441,https://api.github.com/repos/simonw/sqlite-utils/issues/410,1055856441,IC_kwDOCGYnMM4-7xc5,9599,simonw,2022-03-01T21:05:21Z,2022-03-01T21:05:21Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1149729902,Correct spelling mistakes (found with codespell), https://github.com/simonw/sqlite-utils/issues/343#issuecomment-1055854884,https://api.github.com/repos/simonw/sqlite-utils/issues/343,1055854884,IC_kwDOCGYnMM4-7xEk,9599,simonw,2022-03-01T21:03:45Z,2022-03-01T21:03:45Z,OWNER,"Just found myself needing this capability myself! Relevant code: https://github.com/simonw/sqlite-utils/blob/8f386a0d300d1b1c76132bb75972b755049fb742/sqlite_utils/db.py#L2297-L2307 https://github.com/simonw/sqlite-utils/blob/8f386a0d300d1b1c76132bb75972b755049fb742/sqlite_utils/db.py#L2996-L3001 So various functions could grow a `hash_id_columns=(""title"", ""date"")` argument which causes just those columns to be included in the hash. Bonus: if you use `hash_id_columns=...` without setting `hash_id=""id""` it could assume that you want the column to be called `id`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1063388037,Provide function to generate hash_id from specified columns, https://github.com/simonw/datasette/issues/1439#issuecomment-1053973425,https://api.github.com/repos/simonw/datasette/issues/1439,1053973425,IC_kwDOBm6k_c4-0lux,9599,simonw,2022-02-28T07:40:12Z,2022-02-28T07:40:12Z,OWNER,"If I make this change it will break existing links to one of the oldest Datasette demos: http://fivethirtyeight.datasettes.com/fivethirtyeight/avengers%2Favengers A plugin that fixes those by redirecting them on 404 would be neat.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1049126151,https://api.github.com/repos/simonw/datasette/issues/1439,1049126151,IC_kwDOBm6k_c4-iGUH,9599,simonw,2022-02-23T19:17:01Z,2022-02-23T19:17:01Z,OWNER,Actually the relevant code looks to be: https://github.com/simonw/datasette/blob/7d24fd405f3c60e4c852c5d746c91aa2ba23cf5b/datasette/views/base.py#L481-L498,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1049124390,https://api.github.com/repos/simonw/datasette/issues/1439,1049124390,IC_kwDOBm6k_c4-iF4m,9599,simonw,2022-02-23T19:15:00Z,2022-02-23T19:15:00Z,OWNER,"I'll start by modifying this function: https://github.com/simonw/datasette/blob/458f03ad3a454d271f47a643f4530bd8b60ddb76/datasette/utils/__init__.py#L732-L749 Later I want to move this to the routing layer to split out `format` automatically, as seen in the regexes here: https://github.com/simonw/datasette/issues/1439#issuecomment-1045069481","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1049114724,https://api.github.com/repos/simonw/datasette/issues/1439,1049114724,IC_kwDOBm6k_c4-iDhk,9599,simonw,2022-02-23T19:04:40Z,2022-02-23T19:04:40Z,OWNER,I'm going to try dash encoding for table names (and row IDs) in a branch and see how I like it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/928#issuecomment-672379897,https://api.github.com/repos/simonw/datasette/issues/928,672379897,MDEyOklzc3VlQ29tbWVudDY3MjM3OTg5Nw==,9599,simonw,2020-08-12T00:07:49Z,2022-02-23T16:19:47Z,OWNER,Made this into a TIL: https://til.simonwillison.net/python/call-pip-programatically,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677272618,Test failures caused by failed attempts to mock pip, https://github.com/simonw/datasette/issues/1439#issuecomment-1045269544,https://api.github.com/repos/simonw/datasette/issues/1439,1045269544,IC_kwDOBm6k_c4-TYwo,9599,simonw,2022-02-18T22:19:29Z,2022-02-18T22:19:29Z,OWNER,"Note that I've ruled out using `Accept: application/json` to return JSON because it turns out Cloudflare and potentially other CDNs ignore the `Vary: Accept` header entirely: - https://github.com/simonw/datasette/issues/1534","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1045134050,https://api.github.com/repos/simonw/datasette/issues/1439,1045134050,IC_kwDOBm6k_c4-S3ri,9599,simonw,2022-02-18T20:25:04Z,2022-02-18T20:25:04Z,OWNER,Here's a useful modern spec for how existing URL percentage encoding is supposed to work: https://url.spec.whatwg.org/#percent-encoded-bytes,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1045131086,https://api.github.com/repos/simonw/datasette/issues/1439,1045131086,IC_kwDOBm6k_c4-S29O,9599,simonw,2022-02-18T20:22:13Z,2022-02-18T20:22:47Z,OWNER,"Should it encode `%` symbols too, since they have a special meaning in URLs and we can't guarantee that every single web server / proxy out there will round-trip them safely using percentage encoding? If so, would need to pick a different encoding character for them. Maybe `%` becomes `-p` - and in that case `/` could become `-s` too. Is it worth expanding dash-encoding outside of just `/` and `-` and `.` though? Not sure.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1045117304,https://api.github.com/repos/simonw/datasette/issues/1439,1045117304,IC_kwDOBm6k_c4-Szl4,9599,simonw,2022-02-18T20:09:22Z,2022-02-18T20:09:22Z,OWNER,Adopting this could result in supporting database files with surprising characters in their filename too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1045108611,https://api.github.com/repos/simonw/datasette/issues/1439,1045108611,IC_kwDOBm6k_c4-SxeD,9599,simonw,2022-02-18T20:02:19Z,2022-02-18T20:08:34Z,OWNER,"One other potential variant: ```python def dash_encode(s): return s.replace(""-"", ""-dash-"").replace(""."", ""-dot-"").replace(""/"", ""-slash-"") def dash_decode(s): return s.replace(""-slash-"", ""/"").replace(""-dot-"", ""."").replace(""-dash-"", ""-"") ``` Except this has bugs - it doesn't round-trip safely, because it can get confused about things like `-dash-slash-` in terms of is that a `-dash-` or a `-slash-`? ```pycon >>> dash_encode(""/db/table-.csv.csv"") '-slash-db-slash-table-dash--dot-csv-dot-csv' >>> dash_decode('-slash-db-slash-table-dash--dot-csv-dot-csv') '/db/table-.csv.csv' >>> dash_encode('-slash-db-slash-table-dash--dot-csv-dot-csv') '-dash-slash-dash-db-dash-slash-dash-table-dash-dash-dash--dash-dot-dash-csv-dash-dot-dash-csv' >>> dash_decode('-dash-slash-dash-db-dash-slash-dash-table-dash-dash-dash--dash-dot-dash-csv-dash-dot-dash-csv') '-dash/dash-db-dash/dash-table-dash--dash.dash-csv-dash.dash-csv' ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1045111309,https://api.github.com/repos/simonw/datasette/issues/1439,1045111309,IC_kwDOBm6k_c4-SyIN,9599,simonw,2022-02-18T20:04:24Z,2022-02-18T20:05:40Z,OWNER,"This made me worry that my current `dash_decode()` implementation had unknown round-trip bugs, but thankfully this works OK: ```pycon >>> dash_encode(""/db/table-.csv.csv"") '-/db-/table---.csv-.csv' >>> dash_encode('-/db-/table---.csv-.csv') '---/db---/table-------.csv---.csv' >>> dash_decode('---/db---/table-------.csv---.csv') '-/db-/table---.csv-.csv' >>> dash_decode('-/db-/table---.csv-.csv') '/db/table-.csv.csv' ``` The regex still works against that double-encoded example too: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1045099290,https://api.github.com/repos/simonw/datasette/issues/1439,1045099290,IC_kwDOBm6k_c4-SvMa,9599,simonw,2022-02-18T19:56:18Z,2022-02-18T19:56:30Z,OWNER,"> ```python > def dash_encode(s): > return s.replace(""-"", ""--"").replace(""."", ""-."").replace(""/"", ""-/"") > > def dash_decode(s): > return s.replace(""-/"", ""/"").replace(""-."", ""."").replace(""--"", ""-"") > ``` I think **dash-encoding** (new name for this) is the right way forward here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1045024276,https://api.github.com/repos/simonw/datasette/issues/1439,1045024276,IC_kwDOBm6k_c4-Sc4U,9599,simonw,2022-02-18T19:01:42Z,2022-02-18T19:55:24Z,OWNER,"> Maybe I should use `-/` to encode forward slashes too, to defend against any ASGI servers that might not implement `raw_path` correctly. ```python def dash_encode(s): return s.replace(""-"", ""--"").replace(""."", ""-."").replace(""/"", ""-/"") def dash_decode(s): return s.replace(""-/"", ""/"").replace(""-."", ""."").replace(""--"", ""-"") ``` ```pycon >>> dash_encode(""foo/bar/baz.csv"") 'foo-/bar-/baz-.csv' >>> dash_decode('foo-/bar-/baz-.csv') 'foo/bar/baz.csv' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1045095348,https://api.github.com/repos/simonw/datasette/issues/1439,1045095348,IC_kwDOBm6k_c4-SuO0,9599,simonw,2022-02-18T19:53:48Z,2022-02-18T19:53:48Z,OWNER,"> Ugh, one disadvantage I just spotted with this: Datasette already has a `/-/versions.json` convention where ""system"" URLs are namespaced under `/-/` - but that could be confused under this new scheme with the `-/` escaping sequence. > > And I've thought about adding `/db/-/special` and `/db/table/-/special` URLs in the past too. I don't think this matters. The new regex does indeed capture that kind of page: But Datasette goes through configured route regular expressions in order - so I can have the regex that captures `/db/-/special` routes listed before the one that captures tables and formats.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1045081042,https://api.github.com/repos/simonw/datasette/issues/1439,1045081042,IC_kwDOBm6k_c4-SqvS,9599,simonw,2022-02-18T19:44:12Z,2022-02-18T19:51:34Z,OWNER,"```python def dot_encode(s): return s.replace(""."", "".."").replace(""/"", ""./"") def dot_decode(s): return s.replace(""./"", ""/"").replace("".."", ""."") ``` No need for hyphen encoding in this variant at all, which simplifies things a bit. (Update: this is flawed, see https://github.com/simonw/datasette/issues/1439#issuecomment-1045086033)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1045086033,https://api.github.com/repos/simonw/datasette/issues/1439,1045086033,IC_kwDOBm6k_c4-Sr9R,9599,simonw,2022-02-18T19:47:43Z,2022-02-18T19:51:11Z,OWNER,"- https://datasette.io/-/asgi-scope/db/./db./table-..csv..csv - https://til.simonwillison.net/-/asgi-scope/db/./db./table-..csv..csv Do both of those survive the round-trip to populate `raw_path` correctly? No! In both cases the `/./` bit goes missing. It looks like this might even be a client issue - `curl` shows me this: ``` ~ % curl -vv -i 'https://datasette.io/-/asgi-scope/db/./db./table-..csv..csv' * Trying 216.239.32.21:443... * Connected to datasette.io (216.239.32.21) port 443 (#0) * ALPN, offering http/1.1 * TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 * Server certificate: datasette.io * Server certificate: R3 * Server certificate: ISRG Root X1 > GET /-/asgi-scope/db/db./table-..csv..csv HTTP/1.1 ``` So `curl` decided to turn `/-/asgi-scope/db/./db./table` into `/-/asgi-scope/db/db./table` before even sending the request.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1045082891,https://api.github.com/repos/simonw/datasette/issues/1439,1045082891,IC_kwDOBm6k_c4-SrML,9599,simonw,2022-02-18T19:45:32Z,2022-02-18T19:45:32Z,OWNER,"```pycon >>> dot_encode(""/db/table-.csv.csv"") './db./table-..csv..csv' >>> dot_decode('./db./table-..csv..csv') '/db/table-.csv.csv' ``` I worry that web servers might treat `./` in a special way though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1045077590,https://api.github.com/repos/simonw/datasette/issues/1439,1045077590,IC_kwDOBm6k_c4-Sp5W,9599,simonw,2022-02-18T19:41:37Z,2022-02-18T19:42:41Z,OWNER,"Ugh, one disadvantage I just spotted with this: Datasette already has a `/-/versions.json` convention where ""system"" URLs are namespaced under `/-/` - but that could be confused under this new scheme with the `-/` escaping sequence. And I've thought about adding `/db/-/special` and `/db/table/-/special` URLs in the past too. Maybe change this system to use `.` as the escaping character instead of `-`?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1045075207,https://api.github.com/repos/simonw/datasette/issues/1439,1045075207,IC_kwDOBm6k_c4-SpUH,9599,simonw,2022-02-18T19:39:35Z,2022-02-18T19:40:13Z,OWNER,"> And if for some horific reason you had a table with the name `/db/table-.csv.csv` (so `/db/` was the first part of the actual table name in SQLite) the URLs would look like this: > > * `/db/%2Fdb%2Ftable---.csv-.csv` - the HTML version > * `/db/%2Fdb%2Ftable---.csv-.csv.csv` - the CSV version > * `/db/%2Fdb%2Ftable---.csv-.csv.json` - the JSON version Here's what those look like with the updated version of `dot_dash_encode()` that also encodes `/` as `-/`: - `/db/-/db-/table---.csv-.csv` - HTML - `/db/-/db-/table---.csv-.csv.csv` - CSV - `/db/-/db-/table---.csv-.csv.json` - JSON ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1045059427,https://api.github.com/repos/simonw/datasette/issues/1439,1045059427,IC_kwDOBm6k_c4-Sldj,9599,simonw,2022-02-18T19:26:25Z,2022-02-18T19:26:25Z,OWNER,"With this new pattern I could probably extract out the optional `.json` format string as part of the initial route capturing regex too, rather than the current `table_and_format` hack.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1045055772,https://api.github.com/repos/simonw/datasette/issues/1439,1045055772,IC_kwDOBm6k_c4-Skkc,9599,simonw,2022-02-18T19:23:33Z,2022-02-18T19:25:42Z,OWNER,"I want a match for this URL: /db/table-/with-/slashes-.csv Maybe this: ^/(?P[^/]+)/(?P([^/]*|(\-/)*|(\-\.)*|(\.\.)*)*$) Here we are matching a sequence of: ([^/]*|(\-/)*|(\-\.)*|(\-\-)*)* So a combination of not-slashes OR -/ or -. Or -- sequences ^/(?P[^/]+)/(?P([^/]*|(\-/)*|(\-\.)*|(\-\-)*)*$) Try that with non-capturing bits: ^/(?P[^/]+)/(?P(?:[^/]*|(?:\-/)*|(?:\-\.)*|(?:\-\-)*)*$) `(?:[^/]*|(?:\-/)*|(?:\-\.)*|(?:\-\-)*)*` visualized is: Here's the explanation on regex101.com https://regex101.com/r/CPnsIO/1 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1045032377,https://api.github.com/repos/simonw/datasette/issues/1439,1045032377,IC_kwDOBm6k_c4-Se25,9599,simonw,2022-02-18T19:06:50Z,2022-02-18T19:06:50Z,OWNER,"How does URL routing for https://latest.datasette.io/fixtures/table%2Fwith%2Fslashes.csv work? Right now it's https://github.com/simonw/datasette/blob/7d24fd405f3c60e4c852c5d746c91aa2ba23cf5b/datasette/app.py#L1098-L1101 That's not going to capture the dot-dash encoding version of that table name: ```pycon >>> dot_dash_encode(""table/with/slashes.csv"") 'table-/with-/slashes-.csv' ``` Probably needs a fancy regex trick like a negative lookbehind assertion or similar.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1045027067,https://api.github.com/repos/simonw/datasette/issues/1439,1045027067,IC_kwDOBm6k_c4-Sdj7,9599,simonw,2022-02-18T19:03:26Z,2022-02-18T19:03:26Z,OWNER,"(If I make this change it may break some existing Datasette installations when they upgrade - I could try and build a plugin for them which triggers on 404s and checks to see if the old format would return a 200 response, then returns that.)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/sqlite-utils/issues/406#issuecomment-1040978032,https://api.github.com/repos/simonw/sqlite-utils/issues/406,1040978032,IC_kwDOCGYnMM4-DBBw,9599,simonw,2022-02-16T01:10:31Z,2022-02-16T01:10:31Z,OWNER,"Allowing custom strings in the `create()` method, as you suggest in your example, feels like a reasonable way to support this. ```python db[""dummy""].create({ ""title"": str, ""vector"": ""array"", }) ``` I'm slightly nervous about that just because people might accidentally use this without realizig what they are doing - passing `""column-name"": ""string""` for example when they should have used `""column-name"": str` in order to get a `TEXT` column. Alternatively, this could work: ```python db[""dummy""].create({ ""title"": str, ""vector"": CustomColumnType(""array"") }) ``` This would play better with `mypy` too I think.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1128466114,Creating tables with custom datatypes, https://github.com/simonw/sqlite-utils/issues/406#issuecomment-1040974519,https://api.github.com/repos/simonw/sqlite-utils/issues/406,1040974519,IC_kwDOCGYnMM4-DAK3,9599,simonw,2022-02-16T01:08:17Z,2022-02-16T01:08:17Z,OWNER,"I had no idea this was possible! I guess SQLite will allow any text string as the column type, defaulting to `TEXT` as the underlying default representation if it doesn't recognize the type.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1128466114,Creating tables with custom datatypes, https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1040965672,https://api.github.com/repos/simonw/sqlite-utils/issues/398,1040965672,IC_kwDOCGYnMM4-C-Ao,9599,simonw,2022-02-16T01:02:29Z,2022-02-16T01:02:29Z,OWNER,"Documentation: - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#create-database - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#add-geometry-column - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#create-spatial-index - https://sqlite-utils.datasette.io/en/latest/cli.html#spatialite-helpers","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124237013,Add SpatiaLite helpers to CLI, https://github.com/simonw/sqlite-utils/pull/407#issuecomment-1040959312,https://api.github.com/repos/simonw/sqlite-utils/issues/407,1040959312,IC_kwDOCGYnMM4-C8dQ,9599,simonw,2022-02-16T00:58:32Z,2022-02-16T00:58:32Z,OWNER,This is honestly one of the most complete PRs I've ever seen for a feature of this size. Thanks so much for this!,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138948786,Add SpatiaLite helpers to CLI, https://github.com/simonw/sqlite-utils/pull/407#issuecomment-1040598665,https://api.github.com/repos/simonw/sqlite-utils/issues/407,1040598665,IC_kwDOCGYnMM4-BkaJ,9599,simonw,2022-02-15T17:58:11Z,2022-02-15T17:58:11Z,OWNER,"Wow, just found out I can edit files in this PR branch by hitting `.` on my keyboard while looking at the PR, then making changes in the VS Code for web on `github.dev`!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138948786,Add SpatiaLite helpers to CLI, https://github.com/simonw/sqlite-utils/pull/407#issuecomment-1040596969,https://api.github.com/repos/simonw/sqlite-utils/issues/407,1040596969,IC_kwDOCGYnMM4-Bj_p,9599,simonw,2022-02-15T17:56:22Z,2022-02-15T17:56:35Z,OWNER,"We should add SpatiaLite to the action that calculates code coverage - that way we can calculate coverage across the new GIS tests as well: https://github.com/simonw/sqlite-utils/blob/main/.github/workflows/test-coverage.yml Should just be a case of adding this to that workflow - we can do this in the same PR. ``` - name: Install SpatiaLite run: sudo apt-get install libsqlite3-mod-spatialite ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138948786,Add SpatiaLite helpers to CLI, https://github.com/simonw/sqlite-utils/pull/407#issuecomment-1040595572,https://api.github.com/repos/simonw/sqlite-utils/issues/407,1040595572,IC_kwDOCGYnMM4-Bjp0,9599,simonw,2022-02-15T17:54:58Z,2022-02-15T17:54:58Z,OWNER,This PR looks fantastic.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138948786,Add SpatiaLite helpers to CLI, https://github.com/simonw/datasette/issues/1143#issuecomment-1038289584,https://api.github.com/repos/simonw/datasette/issues/1143,1038289584,IC_kwDOBm6k_c494wqw,9599,simonw,2022-02-13T17:40:50Z,2022-02-13T17:41:17Z,OWNER,"The way Drupal does this is interesting; https://www.drupal.org/node/2715637 - it supports the following YAML: ```yaml # Configure Cross-Site HTTP requests (CORS). # Read https://developer.mozilla.org/en-US/docs/Web/HTTP/Access_control_CORS # for more information about the topic in general. # Note: By default the configuration is disabled. cors.config: enabled: false # Specify allowed headers, like 'x-allowed-header'. allowedHeaders: [] # Specify allowed request methods, specify ['*'] to allow all possible ones. allowedMethods: [] # Configure requests allowed from specific origins. allowedOrigins: ['*'] # Sets the Access-Control-Expose-Headers header. exposedHeaders: false # Sets the Access-Control-Max-Age header. maxAge: false # Sets the Access-Control-Allow-Credentials header. supportsCredentials: false ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",764059235,"More flexible CORS support in core, to encourage good security practices", https://github.com/simonw/datasette/issues/1634#issuecomment-1035667060,https://api.github.com/repos/simonw/datasette/issues/1634,1035667060,IC_kwDOBm6k_c49uwZ0,9599,simonw,2022-02-11T00:13:22Z,2022-02-11T00:13:22Z,OWNER,Looks like `3.10.2` is the latest: https://hub.docker.com/_/python?tab=tags&page=1&name=3.10.2-slim-bu,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1131295060,Update Dockerfile generated by `datasette publish`, https://github.com/simonw/datasette/issues/1634#issuecomment-1035664928,https://api.github.com/repos/simonw/datasette/issues/1634,1035664928,IC_kwDOBm6k_c49uv4g,9599,simonw,2022-02-11T00:10:07Z,2022-02-11T00:10:23Z,OWNER,Could also bump this up to Python 3.10: https://github.com/simonw/datasette/blob/5619069968ab39fd44c44a1888965e361c6f7fb9/Dockerfile#L1,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1131295060,Update Dockerfile generated by `datasette publish`, https://github.com/simonw/datasette/issues/1634#issuecomment-1035664412,https://api.github.com/repos/simonw/datasette/issues/1634,1035664412,IC_kwDOBm6k_c49uvwc,9599,simonw,2022-02-11T00:09:18Z,2022-02-11T00:09:18Z,OWNER,Starting it with `FROM datasetteproject/datasette` might be a good idea. ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1131295060,Update Dockerfile generated by `datasette publish`, https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1033366312,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1033366312,IC_kwDOCGYnMM49l-so,9599,simonw,2022-02-09T05:28:11Z,2022-02-09T07:28:48Z,OWNER,"My hunch is that the case where you want to consider input from more than one column will actually be pretty rare - the only case I can think of where I would want to do that is for latitude/longitude columns - everything else that I'd want to use it for (which admittedly is still mostly SpatiaLite stuff) works against a single value. The reason I'm leaning towards using the constructor for the values is that I really like the look of this variant for common conversions: ```python db[""places""].insert( { ""name"": ""London"", ""boundary"": GeometryFromGeoJSON({...}) } ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism, https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1033428967,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1033428967,IC_kwDOCGYnMM49mN_n,9599,simonw,2022-02-09T07:25:44Z,2022-02-09T07:28:11Z,OWNER,"The CLI version of this could perhaps look like this: sqlite-utils insert a.db places places.json \ --conversion boundary GeometryGeoJSON This will treat the boundary key as GeoJSON. It's equivalent to passing `conversions={""boundary"": geometryGeoJSON}` The combined latitude/longitude case here can be handled by combining this with the existing `--convert` mechanism. Any `Conversion` subclass will be available to the CLI in this way.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism, https://github.com/simonw/sqlite-utils/issues/405#issuecomment-1033425512,https://api.github.com/repos/simonw/sqlite-utils/issues/405,1033425512,IC_kwDOCGYnMM49mNJo,9599,simonw,2022-02-09T07:20:11Z,2022-02-09T07:20:11Z,OWNER,"Datasette's implementation: https://github.com/simonw/datasette/blob/458f03ad3a454d271f47a643f4530bd8b60ddb76/datasette/database.py#L73-L79 ```python if self.memory_name: uri = ""file:{}?mode=memory&cache=shared"".format(self.memory_name) conn = sqlite3.connect( uri, uri=True, check_same_thread=False, ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1128139375,"`Database(memory_name=""name"")` constructor argument", https://github.com/simonw/sqlite-utils/issues/405#issuecomment-1033424454,https://api.github.com/repos/simonw/sqlite-utils/issues/405,1033424454,IC_kwDOCGYnMM49mM5G,9599,simonw,2022-02-09T07:18:25Z,2022-02-09T07:18:25Z,OWNER,Writing tests against this is always a tiny bit fiddly since the created databases persist across the lifetime of the test run. Using randomly generated names helps.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1128139375,"`Database(memory_name=""name"")` constructor argument", https://github.com/simonw/sqlite-utils/issues/404#issuecomment-1033410970,https://api.github.com/repos/simonw/sqlite-utils/issues/404,1033410970,IC_kwDOCGYnMM49mJma,9599,simonw,2022-02-09T06:56:35Z,2022-02-09T06:56:35Z,OWNER,https://sqlite-utils.datasette.io/en/latest/cli-reference.html#insert,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1128120451,Add example of `--convert` to the help for `sqlite-utils insert`, https://github.com/simonw/sqlite-utils/issues/404#issuecomment-1033407778,https://api.github.com/repos/simonw/sqlite-utils/issues/404,1033407778,IC_kwDOCGYnMM49mI0i,9599,simonw,2022-02-09T06:50:26Z,2022-02-09T06:50:26Z,OWNER,"I'll use this: ``` sqlite-utils insert plants.db plants plants.csv --csv --convert ' return { ""name"": row[""name""].upper(), ""latitude"": float(row[""latitude""]), ""longitude"": float(row[""longitude""]), }' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1128120451,Add example of `--convert` to the help for `sqlite-utils insert`, https://github.com/simonw/datasette/issues/1607#issuecomment-1033403664,https://api.github.com/repos/simonw/datasette/issues/1607,1033403664,IC_kwDOBm6k_c49mH0Q,9599,simonw,2022-02-09T06:42:02Z,2022-02-09T06:42:02Z,OWNER,"Deployed a new build of https://github.com/simonw/calands-datasette/actions/workflows/build-and-deploy.yml for a live demo: https://calands.datasettes.com/-/versions","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109783030,More detailed information about installed SpatiaLite version, https://github.com/simonw/sqlite-utils/issues/403#issuecomment-1032987901,https://api.github.com/repos/simonw/sqlite-utils/issues/403,1032987901,IC_kwDOCGYnMM49kiT9,9599,simonw,2022-02-08T19:36:06Z,2022-02-08T19:36:06Z,OWNER,New documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#adding-a-primary-key-to-a-rowid-table,"{""total_count"": 3, ""+1"": 3, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1126692066,Document how to add a primary key to a rowid table using `sqlite-utils transform --pk`, https://github.com/simonw/sqlite-utils/issues/403#issuecomment-1032976720,https://api.github.com/repos/simonw/sqlite-utils/issues/403,1032976720,IC_kwDOCGYnMM49kflQ,9599,simonw,2022-02-08T19:23:05Z,2022-02-08T19:23:27Z,OWNER,"This is already possible using `sqlite-utils transform` like so: ``` % echo '[{""name"": ""Barry""}, {""name"": ""Sandra""}]' | sqlite-utils insert rowid.db records - % sqlite-utils schema rowid.db CREATE TABLE [records] ( [name] TEXT ); % sqlite-utils rows rowid.db records [{""name"": ""Barry""}, {""name"": ""Sandra""}] % sqlite-utils transform rowid.db records --pk id % sqlite-utils rows rowid.db records [{""id"": 1, ""name"": ""Barry""}, {""id"": 2, ""name"": ""Sandra""}] % sqlite-utils schema rowid.db CREATE TABLE ""records"" ( [id] INTEGER PRIMARY KEY, [name] TEXT ); % echo '[{""name"": ""Barry 2""}, {""name"": ""Sandra 2""}]' | sqlite-utils insert rowid.db records - % sqlite-utils rows rowid.db records [{""id"": 1, ""name"": ""Barry""}, {""id"": 2, ""name"": ""Sandra""}, {""id"": 3, ""name"": ""Barry 2""}, {""id"": 4, ""name"": ""Sandra 2""}] ``` It's not covered in the documentation though: https://sqlite-utils.datasette.io/en/3.23/cli.html#transforming-tables","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1126692066,Document how to add a primary key to a rowid table using `sqlite-utils transform --pk`, https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1030904948,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1030904948,IC_kwDOCGYnMM49clx0,9599,simonw,2022-02-06T20:09:42Z,2022-02-08T07:40:44Z,OWNER,"I think this is the code that needs to become aware of this system: https://github.com/simonw/sqlite-utils/blob/fea8c9bcc509bcae75e99ae8870f520103b9aa58/sqlite_utils/db.py#L2453-L2469 There's an earlier branch that runs for upserts which needs to be modified too: https://github.com/simonw/sqlite-utils/blob/fea8c9bcc509bcae75e99ae8870f520103b9aa58/sqlite_utils/db.py#L2417-L2440","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism, https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1030902102,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1030902102,IC_kwDOCGYnMM49clFW,9599,simonw,2022-02-06T19:53:34Z,2022-02-08T07:40:34Z,OWNER,"I like the idea that the contract for `Conversion` (or rather for its subclasses) is that it can wrap a Python value and then return both the SQL fragment - e.g. `GeomFromText(?, 4326)` - and the values that should be used as the SQL parameters.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism, https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1032296717,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1032296717,IC_kwDOCGYnMM49h5kN,9599,simonw,2022-02-08T07:35:46Z,2022-02-08T07:35:46Z,OWNER,"I'm going to write the documentation for this first, before the implementation, so I can see if it explains cleanly enough that the design appears to be sound.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism, https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1032294365,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1032294365,IC_kwDOCGYnMM49h4_d,9599,simonw,2022-02-08T07:32:09Z,2022-02-08T07:34:41Z,OWNER,"I have an idea for how that third option could work - the one that creates a new column using values from the existing ones: ```python db[""places""].insert( { ""name"": ""London"", ""lng"": -0.118092, ""lat"": 51.509865, }, conversions={""point"": LongitudeLatitude(""lng"", ""lat"")}, ) ``` How about specifying that the values in that `conversion=` dictionary can be: - A SQL string fragment (as currently implemented) - A subclass of `Conversion` as described above - Or... a callable function that takes the row as an argument and returns either a `Conversion` subclass instance or a literal value to be jnserted into the database (a string, int or float) Then you could do this: ```python db[""places""].insert( { ""name"": ""London"", ""lng"": -0.118092, ""lat"": 51.509865, }, conversions={ ""point"": lambda row: LongitudeLatitude( row[""lng""], row[""lat""] ) } ) ``` Something I really like about this is that it expands the abilities of `conversions=` beyond the slightly obscure need to customize the SQL fragment into something that can solve other data insertion cleanup problems too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism, https://github.com/simonw/datasette/issues/1632#issuecomment-1032057472,https://api.github.com/repos/simonw/datasette/issues/1632,1032057472,IC_kwDOBm6k_c49g_KA,9599,simonw,2022-02-07T23:50:01Z,2022-02-07T23:50:01Z,OWNER,Released in https://github.com/simonw/datasette/releases/tag/0.60.2,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1126604194,"datasette one.db one.db opens database twice, as one and one_2", https://github.com/simonw/datasette/issues/1632#issuecomment-1032050489,https://api.github.com/repos/simonw/datasette/issues/1632,1032050489,IC_kwDOBm6k_c49g9c5,9599,simonw,2022-02-07T23:39:11Z,2022-02-07T23:42:08Z,OWNER,"That implementation broke on Python 3.6 - which is still a supported Python version for the 0.60.x branch - `test_homepage` failed. ``` > assert ( ""2 rows in 1 table, 5 rows in 4 hidden tables, 1 view"" == counts_p.text.strip() ) E AssertionError: assert '2 rows in 1 ...ables, 1 view' == '1 table, 4 h...ables, 1 view' E - 1 table, 4 hidden tables, 1 view E + 2 rows in 1 table, 5 rows in 4 hidden tables, 1 view E ? ++++++++++ ++++++++++ ``` That's because this idiom isn't guaranteed to preserve order in versions earlier than Python 3.7: https://github.com/simonw/datasette/blob/fa5fc327adbbf70656ac533912f3fc0526a3873d/datasette/cli.py#L552-L553 I could say that `0.60.2` is the first version to require Python 3.7 - but that feels a little surprising. I'm going to use a different idiom for order-preserving de-duplication from [this StackOverflow](https://stackoverflow.com/questions/480214/how-do-you-remove-duplicates-from-a-list-whilst-preserving-order) instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1126604194,"datasette one.db one.db opens database twice, as one and one_2", https://github.com/simonw/datasette/issues/1632#issuecomment-1032037391,https://api.github.com/repos/simonw/datasette/issues/1632,1032037391,IC_kwDOBm6k_c49g6QP,9599,simonw,2022-02-07T23:21:07Z,2022-02-07T23:21:07Z,OWNER,"For the record, here's the code that picks the `one_2` name if that stem is already used as a database name: https://github.com/simonw/datasette/blob/03305ea183b1534bc4cef3a721fe5f3700273b84/datasette/app.py#L401-L417","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1126604194,"datasette one.db one.db opens database twice, as one and one_2", https://github.com/simonw/datasette/issues/1632#issuecomment-1032036525,https://api.github.com/repos/simonw/datasette/issues/1632,1032036525,IC_kwDOBm6k_c49g6Ct,9599,simonw,2022-02-07T23:19:59Z,2022-02-07T23:19:59Z,OWNER,"I'm going to fix this in the CLI code itself, rather than fixing it in the `Datasette` constructor. That way if someone has a truly weird reason to want this behaviour they can construct Datasette directly. https://github.com/simonw/datasette/blob/03305ea183b1534bc4cef3a721fe5f3700273b84/datasette/cli.py#L535-L550","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1126604194,"datasette one.db one.db opens database twice, as one and one_2", https://github.com/simonw/datasette/issues/1632#issuecomment-1032034015,https://api.github.com/repos/simonw/datasette/issues/1632,1032034015,IC_kwDOBm6k_c49g5bf,9599,simonw,2022-02-07T23:17:57Z,2022-02-07T23:17:57Z,OWNER,I'm going to fix this in a 0.60.2 bug fix release.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1126604194,"datasette one.db one.db opens database twice, as one and one_2", https://github.com/simonw/datasette/issues/1632#issuecomment-1032032686,https://api.github.com/repos/simonw/datasette/issues/1632,1032032686,IC_kwDOBm6k_c49g5Gu,9599,simonw,2022-02-07T23:16:10Z,2022-02-07T23:16:10Z,OWNER,"I found this bug while trying to get the following to work: datasette /data/one.db /data/two.db /data/*.db --create I want this to create any missing database files on startup out of that literal list of `one.db` and `two.db` and to also open any other `*.db` files in that folder - needed for `datasette-publish-fly` in https://github.com/simonw/datasette-publish-fly/pull/12#issuecomment-1032029874","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1126604194,"datasette one.db one.db opens database twice, as one and one_2", https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1031787865,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1031787865,IC_kwDOCGYnMM49f9VZ,9599,simonw,2022-02-07T18:33:27Z,2022-02-07T18:33:27Z,OWNER,"Hah, that's interesting - I've never used that mechanism before so it wasn't something that came to mind. They seem to be using a pretty surprising trick there that takes advantage of SQLite allowing you to define a column ""type"" using a made-up type name, which you can then introspect later.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism, https://github.com/simonw/datasette/issues/1439#issuecomment-1031141849,https://api.github.com/repos/simonw/datasette/issues/1439,1031141849,IC_kwDOBm6k_c49dfnZ,9599,simonw,2022-02-07T07:11:11Z,2022-02-07T07:11:11Z,OWNER,"I added a Link header to solve this problem for the JSON version in: - #1533 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1176#issuecomment-1031126801,https://api.github.com/repos/simonw/datasette/issues/1176,1031126801,IC_kwDOBm6k_c49db8R,9599,simonw,2022-02-07T06:43:31Z,2022-02-07T06:43:31Z,OWNER,Here's the new test: https://github.com/simonw/datasette/blob/03305ea183b1534bc4cef3a721fe5f3700273b84/tests/test_docs.py#L91-L104,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779691739,"Policy on documenting ""public"" datasette.utils functions", https://github.com/simonw/datasette/issues/1176#issuecomment-1031126547,https://api.github.com/repos/simonw/datasette/issues/1176,1031126547,IC_kwDOBm6k_c49db4T,9599,simonw,2022-02-07T06:42:58Z,2022-02-07T06:42:58Z,OWNER,"That fixed it: https://docs.datasette.io/en/latest/internals.html#parse-metadata-content ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779691739,"Policy on documenting ""public"" datasette.utils functions", https://github.com/simonw/datasette/issues/1176#issuecomment-1031125347,https://api.github.com/repos/simonw/datasette/issues/1176,1031125347,IC_kwDOBm6k_c49dblj,9599,simonw,2022-02-07T06:40:16Z,2022-02-07T06:40:16Z,OWNER,"Read The Docs error: > Problem in your project's configuration. Invalid ""python.version"": .readthedocs.yaml: Invalid configuration option: python.version. Make sure the key name is correct.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779691739,"Policy on documenting ""public"" datasette.utils functions", https://github.com/simonw/datasette/issues/1176#issuecomment-1031123719,https://api.github.com/repos/simonw/datasette/issues/1176,1031123719,IC_kwDOBm6k_c49dbMH,9599,simonw,2022-02-07T06:36:32Z,2022-02-07T06:36:32Z,OWNER,"https://github.com/simonw/sqlite-utils/blob/main/.readthedocs.yaml looks like this (it works correctly): ```yaml version: 2 sphinx: configuration: docs/conf.py python: version: ""3.8"" install: - method: pip path: . extra_requirements: - docs ``` Compare to the current Datasette one here: https://github.com/simonw/datasette/blob/d9b508ffaa91f9f1840b366f5d282712d445f16b/.readthedocs.yaml#L1-L13 Looks like I need this bit: ```python python: version: ""3.8"" install: - method: pip path: . extra_requirements: - docs ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779691739,"Policy on documenting ""public"" datasette.utils functions", https://github.com/simonw/datasette/issues/1176#issuecomment-1031122800,https://api.github.com/repos/simonw/datasette/issues/1176,1031122800,IC_kwDOBm6k_c49da9w,9599,simonw,2022-02-07T06:34:21Z,2022-02-07T06:34:21Z,OWNER,"New section is here: https://docs.datasette.io/en/latest/internals.html#the-datasette-utils-module But it's not correctly displaying the new autodoc stuff: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779691739,"Policy on documenting ""public"" datasette.utils functions", https://github.com/simonw/datasette/issues/1176#issuecomment-1031108559,https://api.github.com/repos/simonw/datasette/issues/1176,1031108559,IC_kwDOBm6k_c49dXfP,9599,simonw,2022-02-07T06:11:27Z,2022-02-07T06:11:27Z,OWNER,I'm going with `@documented` as the decorator for functions that should be documented.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779691739,"Policy on documenting ""public"" datasette.utils functions", https://github.com/simonw/datasette/issues/932#issuecomment-1030940407,https://api.github.com/repos/simonw/datasette/issues/932,1030940407,IC_kwDOBm6k_c49cub3,9599,simonw,2022-02-06T23:31:22Z,2022-02-06T23:31:22Z,OWNER,"Great argument for doing this from a conversation on Twitter about documentation-driven development: > Long ago, when the majority of commercial programs were desktop apps, I've read a very wise advice: The user manual should be written first, before even a single line if code. https://twitter.com/b11c/status/1490466703175823362","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",678760988,End-user documentation, https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030902158,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030902158,IC_kwDOCGYnMM49clGO,9599,simonw,2022-02-06T19:53:54Z,2022-02-06T19:53:54Z,OWNER,"Moving the design of this new `Conversion` subclass mechanism to: - https://github.com/simonw/sqlite-utils/issues/402","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1030901853,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1030901853,IC_kwDOCGYnMM49clBd,9599,simonw,2022-02-06T19:52:10Z,2022-02-06T19:52:10Z,OWNER,"So the key idea here is to introduce a new abstract base class, `Conversion`, which has the following abilities: - Can wrap one or more Python values (if called using the constructor) such that the `.insert_all()` method knows how to transform those into a format that can be included in an insert - something like `GeomFromText(?, 4326)` with input `POINT(-0.118092 51.509865)` - Can be passed to `conversions={""point"": LongitudeLatitude}` in a way that then knows to apply that conversion to every value in the `""point""` key of the data being inserted. - Maybe also extend `conversions=` to allow the definition of additional keys that use as input other rows? That's the `conversions={""point"": LongitudeLatitude(""lng"", ""lat"")}` example above - it may not be possible to get this working with the rest of the design though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism, https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1030901189,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1030901189,IC_kwDOCGYnMM49ck3F,9599,simonw,2022-02-06T19:48:36Z,2022-02-06T19:48:52Z,OWNER,"From [that thread](https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030739566), two extra ideas which it may be possible to support in a single implementation: ```python from sqlite_utils.conversions import LongitudeLatitude db[""places""].insert( { ""name"": ""London"", ""lng"": -0.118092, ""lat"": 51.509865, }, conversions={""point"": LongitudeLatitude(""lng"", ""lat"")}, ) ``` And ```python db[""places""].insert( { ""name"": ""London"", ""point"": LongitudeLatitude(-0.118092, 51.509865) } ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism, https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030871591,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030871591,IC_kwDOCGYnMM49cdon,9599,simonw,2022-02-06T16:57:22Z,2022-02-06T16:57:22Z,OWNER,"I wonder if I could implement the above such that this *also* works: ```python db[""places""].insert( { ""name"": ""London"", ""point"": LongitudeLatitude(-0.118092, 51.509865) } ) ``` This feels like a very natural way to work with single inserts. The challenge is writing the code inside `.insert_all()` such that it can handle these special objects in the input column values in addition to them being passed in `conversions=`. I'm feeling very good about this direction in general though, it feels like it takes the existing but not particularly elegant `conversions=` mechanism and upgrades it to be far more useful, while maintaining backwards compatibility.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 1}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/datasette/issues/1176#issuecomment-1030762279,https://api.github.com/repos/simonw/datasette/issues/1176,1030762279,IC_kwDOBm6k_c49cC8n,9599,simonw,2022-02-06T06:38:08Z,2022-02-06T06:41:37Z,OWNER,"Might do this using Sphinx auto-generated function and class documentation hooks, as seen here in `sqlite-utils`: https://sqlite-utils.datasette.io/en/stable/python-api.html#spatialite-helpers This would encourage me to add really good docstrings. ``` .. _python_api_gis_find_spatialite: Finding SpatiaLite ------------------ .. autofunction:: sqlite_utils.utils.find_spatialite ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779691739,"Policy on documenting ""public"" datasette.utils functions", https://github.com/simonw/datasette/issues/957#issuecomment-1030762140,https://api.github.com/repos/simonw/datasette/issues/957,1030762140,IC_kwDOBm6k_c49cC6c,9599,simonw,2022-02-06T06:36:41Z,2022-02-06T06:36:41Z,OWNER,Documented here: https://docs.datasette.io/en/latest/internals.html#import-shortcuts,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688622148,Simplify imports of common classes, https://github.com/simonw/datasette/issues/957#issuecomment-1030761625,https://api.github.com/repos/simonw/datasette/issues/957,1030761625,IC_kwDOBm6k_c49cCyZ,9599,simonw,2022-02-06T06:30:32Z,2022-02-06T06:31:44Z,OWNER,"I'm just going with: ```python from datasette import Response from datasette import Forbidden from datasette import NotFound from datasette import hookimpl from datasette import actor_matches_allow ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688622148,Simplify imports of common classes, https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030740963,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030740963,IC_kwDOCGYnMM49b9vj,9599,simonw,2022-02-06T03:00:33Z,2022-02-06T03:00:33Z,OWNER,"Yeah, having this be a general purpose mechanism which has a few canned examples for handling geospatial stuff is a lot neater than having a mechanism for this that's exclusive to SpatiaLite.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030740846,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030740846,IC_kwDOCGYnMM49b9tu,9599,simonw,2022-02-06T02:59:21Z,2022-02-06T02:59:21Z,OWNER,I wonder if there are any interesting non-geospatial canned conversions that it would be worth including?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030740771,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030740771,IC_kwDOCGYnMM49b9sj,9599,simonw,2022-02-06T02:58:29Z,2022-02-06T02:58:29Z,OWNER,That example you have there is really neat - I like the idea that they can also be used to populate completely new columns that are derived from the other column inputs.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030740570,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030740570,IC_kwDOCGYnMM49b9pa,9599,simonw,2022-02-06T02:56:17Z,2022-02-06T02:57:00Z,OWNER,"Thinking about types. The type of the `conversions` parameter right now is a bit lazy: ```python conversions: Optional[dict] = None, ``` That becomes: ```python Optional[Dict[str, Union[str, Conversion]]] ``` Where `Conversion` is an abstract base class which expects implementations to have a `.sql() -> str` and a `.convert(value) -> str` method.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030739566,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030739566,IC_kwDOCGYnMM49b9Zu,9599,simonw,2022-02-06T02:45:25Z,2022-02-06T02:50:27Z,OWNER,"Another idea - my favourite option so far: ```python from sqlite_utils.utils import LongitudeLatitude db[""places""].insert( { ""name"": ""London"", ""point"": (-0.118092, 51.509865) }, conversions={""point"": LongitudeLatitude}, ) ``` Here `LongitudeLatitude` is a magical value which does TWO things: it sets up the `GeomFromText(?, 4326)` SQL function, and it handles converting the `(51.509865, -0.118092)` tuple into a `POINT({} {})` string. This would involve a change to the `conversions=` contract - where it usually expects a SQL string fragment, but it can also take an object which combines that SQL string fragment with a Python conversion function. Best of all... this resolves the `lat, lon` v.s. `lon, lat` dilemma because you can use `from sqlite_utils.utils import LongitudeLatitude` OR `from sqlite_utils.utils import LatitudeLongitude` depending on which you prefer!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030738023,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030738023,IC_kwDOCGYnMM49b9Bn,9599,simonw,2022-02-06T02:28:05Z,2022-02-06T02:29:24Z,OWNER,"Here's the definitive guide to `latitude, longitude` v.s. `longitude, latitude`: https://macwright.com/lonlat/ > Which is right? > > Neither. This is an opinion with no right answer. Geographical tradition favors lat, lon. Math and software prefer lon, lat. I asked on Twitter here: https://twitter.com/simonw/status/1490148001569906688","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030736848,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030736848,IC_kwDOCGYnMM49b8vQ,9599,simonw,2022-02-06T02:17:35Z,2022-02-06T02:17:35Z,OWNER,"Note that GeoJSON itself uses `(longitude, latitude)` so I should probably stick to that order here too. https://datatracker.ietf.org/doc/html/rfc7946#section-3.1.1","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030736589,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030736589,IC_kwDOCGYnMM49b8rN,9599,simonw,2022-02-06T02:14:52Z,2022-02-06T02:14:52Z,OWNER,"Another idea: introduce a helper function transform pattern, something a bit like this: ```python transformer = make_transformer({ ""point"": lambda pair: ""POINT({} {})"".format(pair[1], pair[0]) }) db[""places""].insert_all( transformer([{""name"": ""London"", ""point"": (51.509865, -0.118092)}]) conversions={""point"": ""GeomFromText(?, 4326)""}, ) ``` The `make_transformer(...)` function builds an object that can work as a wrapping iterator, applying those transform functions to everything in the sequence that it wraps. So the above code would handle converting `(lat, lon)` to `POINT(lon lat)` - then the `conversions=` applies `GeomFromText`. Naming is a challenge here: `.transform()` and `.convert()` and `conversions=` all have existing meanings within the `sqlite-utils` Python library. It's also a bit of a messy way of solving this. It's not exactly a smooth API for inserting a bunch of lat/lon coordinate pairs!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030736047,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030736047,IC_kwDOCGYnMM49b8iv,9599,simonw,2022-02-06T02:10:18Z,2022-02-06T02:10:18Z,OWNER,"So maybe back to that earlier idea where the code introspects the table, figures out that `""point""` is a geometry table of type POINT, then applies the necessary conversions to the raw Python data? That feels overly-complicated to me, especially since nothing else in the `.insert()` method currently relies on table introspection.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030735774,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030735774,IC_kwDOCGYnMM49b8ee,9599,simonw,2022-02-06T02:08:19Z,2022-02-06T02:08:59Z,OWNER,"Maybe I should leave this entirely up to documented patterns in the `conversions={}` dictionary? But even that's not ideal for the co-ordinate case. Consider the following: ```python db[""places""].insert( {""name"": ""London"", ""point"": (51.509865, -0.118092)}, conversions={""point"": ""GeomFromText(?, 4326)""}, ) ``` The challenge here is that the SpatiaLite function `GeomFromText()` expects a WKT string, which looks like this: POINT(-0.118092 51.509865) The existing `conversions=` mechanism doesn't support applying Python code to convert the `(lat, lon)` tuple to that value. It doesn't even support passing a Python tuple as a `?` parameter - so I don't think I could come up with a SQL string that would do the right thing here either.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/401#issuecomment-1030735372,https://api.github.com/repos/simonw/sqlite-utils/issues/401,1030735372,IC_kwDOCGYnMM49b8YM,9599,simonw,2022-02-06T02:05:03Z,2022-02-06T02:05:03Z,OWNER,Improved version: https://sqlite-utils.datasette.io/en/latest/python-api.html#converting-column-values-using-sql-functions,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125081640,Update SpatiaLite example in the documentation, https://github.com/simonw/sqlite-utils/issues/401#issuecomment-1030734937,https://api.github.com/repos/simonw/sqlite-utils/issues/401,1030734937,IC_kwDOCGYnMM49b8RZ,9599,simonw,2022-02-06T02:02:24Z,2022-02-06T02:02:24Z,OWNER,The example also doesn't work right now - the code that fetches from Who's On First gets a 403 forbidden error.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125081640,Update SpatiaLite example in the documentation, https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030732909,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030732909,IC_kwDOCGYnMM49b7xt,9599,simonw,2022-02-06T01:47:06Z,2022-02-06T01:47:06Z,OWNER,"Here's an idea for an API design: ```python geojson_geometry = {} # ... GeoJSON goes here db[""places""].insert( {""name"": ""Wales"", ""geometry"": geojson_geometry}, geojson=""geometry"" ) ``` That `geojson=` parameter takes either a single column name or an iterable of column names. Any column in that list is expected to be a compatible `geometry` and the correct conversion functions will be applied. That solves for GeoJSON, but it's a bit ugly. Should I add `wkt=` and maybe even `kml=` and `gml=` and so-on too? Definitely not, that's way too many ugly and inscrutable new parameters. More importantly: if I want to support the following how would I do it? ```python db[""places""].insert( {""name"": ""London"", ""point"": (51.509865, -0.118092)} ) ``` Here I want to provide a `(latitude, longitude)` pair and have it inserted correctly into a `point` column. Could do this, but again it's messy: ```python db[""places""].insert( {""name"": ""London"", ""point"": (51.509865, -0.118092)}, point=""point"" ) ``` And again, what about those `(longitude, latitude)` people?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1030732222,https://api.github.com/repos/simonw/sqlite-utils/issues/398,1030732222,IC_kwDOCGYnMM49b7m-,9599,simonw,2022-02-06T01:42:19Z,2022-02-06T01:42:28Z,OWNER,"Adding some thoughts to: - #399 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124237013,Add SpatiaLite helpers to CLI, https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1030732093,https://api.github.com/repos/simonw/sqlite-utils/issues/398,1030732093,IC_kwDOCGYnMM49b7k9,9599,simonw,2022-02-06T01:41:37Z,2022-02-06T01:41:37Z,OWNER,Yeah I'd like to avoid adding any geo-dependencies to `sqlite-utils` if I can avoid it. I'm fine using stuff that's going to be available in SpatiaLite itself (provided it's available as a SQLite module) since then I don't need to add any extra Python dependencies.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124237013,Add SpatiaLite helpers to CLI, https://github.com/simonw/sqlite-utils/issues/400#issuecomment-1030730748,https://api.github.com/repos/simonw/sqlite-utils/issues/400,1030730748,IC_kwDOCGYnMM49b7P8,9599,simonw,2022-02-06T01:34:46Z,2022-02-06T01:34:46Z,OWNER,"Actually this is not needed - there is already an option that does this, it's just called `--ignore` rather than `--if-not-exists`. The lack of consistency here is a little annoying, but not annoying enough to justify making a backwards incompatible change. ``` % sqlite-utils create-table --help Usage: sqlite-utils create-table [OPTIONS] PATH TABLE COLUMNS... Add a table with the specified columns. Columns should be specified using name, type pairs, for example: sqlite-utils create-table my.db people \ id integer \ name text \ height float \ photo blob --pk id Options: --pk TEXT Column to use as primary key --not-null TEXT Columns that should be created as NOT NULL --default ... Default value that should be set for a column --fk ... Column, other table, other column to set as a foreign key --ignore If table already exists, do nothing --replace If table already exists, replace it --load-extension TEXT SQLite extensions to load -h, --help Show this message and exit. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125077063,`sqlite-utils create-table` ... `--if-not-exists`, https://github.com/simonw/sqlite-utils/issues/397#issuecomment-1030730108,https://api.github.com/repos/simonw/sqlite-utils/issues/397,1030730108,IC_kwDOCGYnMM49b7F8,9599,simonw,2022-02-06T01:30:46Z,2022-02-06T01:30:46Z,OWNER,Updated documentation is here: https://sqlite-utils.datasette.io/en/latest/python-api.html#explicitly-creating-a-table,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1123903919,Support IF NOT EXISTS for table creation, https://github.com/simonw/sqlite-utils/issues/397#issuecomment-1030727979,https://api.github.com/repos/simonw/sqlite-utils/issues/397,1030727979,IC_kwDOCGYnMM49b6kr,9599,simonw,2022-02-06T01:19:21Z,2022-02-06T01:19:21Z,OWNER,"Just noticed there's no explicit test coverage for the `db[""table""].create(...)` method.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1123903919,Support IF NOT EXISTS for table creation, https://github.com/simonw/sqlite-utils/issues/397#issuecomment-1030726991,https://api.github.com/repos/simonw/sqlite-utils/issues/397,1030726991,IC_kwDOCGYnMM49b6VP,9599,simonw,2022-02-06T01:13:58Z,2022-02-06T01:13:58Z,OWNER,This is a good idea. We already have that parameter for the `table.create_index()` method: https://sqlite-utils.datasette.io/en/stable/reference.html#sqlite_utils.db.Table.create_index,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1123903919,Support IF NOT EXISTS for table creation, https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030712129,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030712129,IC_kwDOCGYnMM49b2tB,9599,simonw,2022-02-05T23:08:45Z,2022-02-05T23:08:45Z,OWNER,"Useful thoughts on Twitter regarding making coordinate pairs easy and more complex shapes possible: https://twitter.com/dbreunig/status/1490099303888547843 > That is exactly where I was going: two modes. > > 1. Heuristics and assumptions to get coordinates as a pair (in tuple) or as columns (look for lat, lon, latitude, longitude, etc). > 2. GIS mode with projections, polys, etc > > Make it easy for people with csvs of coordinates. If you're using Geojson or shp files, you have to specify.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030468418,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030468418,IC_kwDOCGYnMM49a7NC,9599,simonw,2022-02-05T00:49:08Z,2022-02-05T22:59:06Z,OWNER,"I'm trying to think of ways to make this nicer from the perspective of someone calling the `.insert()` or `.insert_all()` methods against a table that has geometry columns. One option would be for the code to introspect the table (if it exists) before running the insert, looking for any geometry columns. This introspection isn't easy! The table schema just gives you `""name_of_column"" point` or similar - to figure out the SRID and suchlike you need to consult the `geometry_columns` table, I think - which throws a 500 error on https://calands.datasettes.com/calands/geometry_columns for some reason. Also does the shape of that table change between SpatiaLite versions? Assuming we can introspect the table, what would we do with that information? We could add code that detects if the user attempted to pass GeoJSON objects and automatically inserts a `GeomFromGeoJSON()` function call - but detecting GeoJSON is a bit weird, and GeoJSON also isn't necessarily the nicest format for populating e.g. latitude/longitude points. Maybe we just support the simplest possible case: a tuple of floats, which we assume is `latitude, longitude` (or should we expect `longitude, latitude`, the eternal debate?) - if those are used against a geometry table (especially a point table) we assume they are coordinates that need to be converted using `GeomFromText('POINT(...`. Not crazy about either of these ideas. Is there something better?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1030534868,https://api.github.com/repos/simonw/sqlite-utils/issues/398,1030534868,IC_kwDOCGYnMM49bLbU,9599,simonw,2022-02-05T06:03:38Z,2022-02-05T06:03:38Z,OWNER,@eyeseast how do you usually insert geometries at the moment?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124237013,Add SpatiaLite helpers to CLI, https://github.com/simonw/datasette/issues/1576#issuecomment-1030530071,https://api.github.com/repos/simonw/datasette/issues/1576,1030530071,IC_kwDOBm6k_c49bKQX,9599,simonw,2022-02-05T05:21:35Z,2022-02-05T05:21:35Z,OWNER,New documentation section: https://docs.datasette.io/en/latest/internals.html#datasette-tracer,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/1576#issuecomment-1030528532,https://api.github.com/repos/simonw/datasette/issues/1576,1030528532,IC_kwDOBm6k_c49bJ4U,9599,simonw,2022-02-05T05:09:57Z,2022-02-05T05:09:57Z,OWNER,Needs documentation. I'll document `from datasette.tracer import trace` too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/1576#issuecomment-1030525218,https://api.github.com/repos/simonw/datasette/issues/1576,1030525218,IC_kwDOBm6k_c49bJEi,9599,simonw,2022-02-05T04:45:11Z,2022-02-05T04:45:11Z,OWNER,"Got a prototype working with `contextvars` - it identified two parallel executing queries using the patch from above: ![CleanShot 2022-02-04 at 20 41 50@2x](https://user-images.githubusercontent.com/9599/152628949-cf766b13-13cf-4831-b48d-2f23cadb6a05.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/1576#issuecomment-1017112543,https://api.github.com/repos/simonw/datasette/issues/1576,1017112543,IC_kwDOBm6k_c48n-ff,9599,simonw,2022-01-20T04:35:00Z,2022-02-05T04:33:46Z,OWNER,I dropped support for Python 3.6 in fae3983c51f4a3aca8335f3e01ff85ef27076fbf so now free to use `contextvars` for this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1030521533,https://api.github.com/repos/simonw/sqlite-utils/issues/398,1030521533,IC_kwDOCGYnMM49bIK9,9599,simonw,2022-02-05T04:25:49Z,2022-02-05T04:25:49Z,OWNER,For ingesting geometry data from the command-line maybe GeoJSON would be the best route?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124237013,Add SpatiaLite helpers to CLI, https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030466255,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030466255,IC_kwDOCGYnMM49a6rP,9599,simonw,2022-02-05T00:41:35Z,2022-02-05T00:42:23Z,OWNER,"Wow, it was the newlines that broke it! This works fine: ```sql select AsWKT(SetSRID(GeomFromGeoJSON('{""type"": ""Point"",""coordinates"": [-94.921875,45.460130637921004]}'), 4326)) ``` https://calands.datasettes.com/calands?sql=select+AsWKT%28SetSRID%28GeomFromGeoJSON%28%27%7B%22type%22%3A+%22Point%22%2C%22coordinates%22%3A+%5B-94.921875%2C45.460130637921004%5D%7D%27%29%2C+4326%29%29 And removing `SetSRID()` returns exactly the same result: https://calands.datasettes.com/calands?sql=select+AsWKT%28GeomFromGeoJSON%28%27%7B%22type%22%3A+%22Point%22%2C%22coordinates%22%3A+%5B-94.921875%2C45.460130637921004%5D%7D%27%29%29","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030465557,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030465557,IC_kwDOCGYnMM49a6gV,9599,simonw,2022-02-05T00:39:09Z,2022-02-05T00:39:09Z,OWNER,"I can't seem to get `GeomFromGeoJSON()` to work - example: https://calands.datasettes.com/calands?sql=select+IsValid%28SetSRID%28GeomFromGeoJSON%28%27%7B%0D%0A++++++++%22type%22%3A+%22Point%22%2C%0D%0A++++++++%22coordinates%22%3A+%5B%0D%0A++++++++++-94.921875%2C%0D%0A++++++++++45.460130637921004%0D%0A++++++++%5D%0D%0A++++++%7D%27%29%2C+4326%29%29 ```sql select IsValid(SetSRID(GeomFromGeoJSON('{ ""type"": ""Point"", ""coordinates"": [ -94.921875, 45.460130637921004 ] }'), 4326)) ``` Returns `-1` suggesting the geometry is not valid. Just doing this (with or without that `SetSRID()` function) returns null: ```sql select SetSRID(GeomFromGeoJSON('{ ""type"": ""Point"", ""coordinates"": [ -94.921875, 45.460130637921004 ] }'), 4326) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030461163,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030461163,IC_kwDOCGYnMM49a5br,9599,simonw,2022-02-05T00:30:18Z,2022-02-05T00:30:18Z,OWNER,"I wonder what the most developer-friendly way to insert geometry data into SpatiaLite is? From https://www.gaia-gis.it/gaia-sins/spatialite-sql-latest.html it looks like these are the main options: - `GeomFromText( wkt String [ , SRID Integer] )` - `GeomFromWKB( wkbGeometry Binary [ , SRID Integer] )` - `GeomFromKml( KmlGeometry String )` - `GeomFromGML( gmlGeometry String )` - `GeomFromGeoJSON( geoJSONGeometry String )` - `GeomFromEWKB( ewkbGeometry String )` - `GeomFromEWKT( ewktGeometry String )` - `GeomFromFGF( fgfGeometry Binary [ , SRID Integer] )` - `GeomFromTWKB( twkbGeometry BLOB [ , SRID Integer] )` - `GeomFromGPB( geom GPKG Blob Geometry )` - GeoPackage format - `GeomFromExifGpsBlob( image BLOB )` Interesting that some accept an SRID and others do not - presumably `GeomFromGeoJSON()` always uses SRID=4326?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1030456717,https://api.github.com/repos/simonw/sqlite-utils/issues/398,1030456717,IC_kwDOCGYnMM49a4WN,9599,simonw,2022-02-05T00:16:42Z,2022-02-05T00:16:42Z,OWNER,"> The one thing worth highlighting in docs is that geometry columns can only be added to existing tables. Trying to add a geometry column to a table that doesn't exist yet might mean you have a schema like `{""rowid"": int, ""geometry"": bytes}`. Might be worth nudging people to explicitly create a table first, then add geometry columns. That's a good call. I'm happy for `sqlite-utils add-geometry-column` to throw an error if the table doesn't exist yet.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124237013,Add SpatiaLite helpers to CLI, https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030455715,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030455715,IC_kwDOCGYnMM49a4Gj,9599,simonw,2022-02-05T00:15:28Z,2022-02-05T00:15:28Z,OWNER,"The `conversions=` argument to `.insert()` and friends is designed to handle this case, but I don't think it's very elegant: https://sqlite-utils.datasette.io/en/stable/python-api.html#converting-column-values-using-sql-functions ```python db[""places""].insert( {""name"": ""Wales"", ""geometry"": wkt}, conversions={""geometry"": ""GeomFromText(?, 4326)""}, ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1030454114,https://api.github.com/repos/simonw/sqlite-utils/issues/398,1030454114,IC_kwDOCGYnMM49a3ti,9599,simonw,2022-02-05T00:14:47Z,2022-02-05T00:14:47Z,OWNER,"I like these designs a lot. I would suggest `sqlite-utils create database.db --init-spatialite` there for consistency with the `sqlite-utils init-spatialite database.db` command. The other part of this story is how we support actually inserting spatial data from the command-line. I opened an issue about the challenges in doing that for the Python API here - #399 - but we need a good answer for the CLI too. I don't yet have any good ideas here. The `conversions=` option in the Python library was designed to cover these kinds of cases but it's pretty clunky and I don't think it's very widely used: https://sqlite-utils.datasette.io/en/stable/python-api.html#converting-column-values-using-sql-functions","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124237013,Add SpatiaLite helpers to CLI, https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1029703503,https://api.github.com/repos/simonw/sqlite-utils/issues/79,1029703503,IC_kwDOCGYnMM49YAdP,9599,simonw,2022-02-04T06:46:32Z,2022-02-04T06:46:32Z,OWNER,Shipped in 3.23: https://sqlite-utils.datasette.io/en/stable/changelog.html#v3-23,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557842245,Helper methods for working with SpatiaLite, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029703216,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029703216,IC_kwDOCGYnMM49YAYw,9599,simonw,2022-02-04T06:45:43Z,2022-02-04T06:45:43Z,OWNER,Shipped this as `sqlite-utils` 3.23: https://sqlite-utils.datasette.io/en/stable/changelog.html#v3-23,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods, https://github.com/simonw/datasette/issues/1080#issuecomment-1029695083,https://api.github.com/repos/simonw/datasette/issues/1080,1029695083,IC_kwDOBm6k_c49X-Zr,9599,simonw,2022-02-04T06:24:40Z,2022-02-04T06:25:18Z,OWNER,"An initial prototype of that in my local `group-count` branch quickly started running into problems: ```diff diff --git a/datasette/views/table.py b/datasette/views/table.py index be9e9c3..d30efe1 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -105,8 +105,12 @@ class RowTableShared(DataView): type_ = ""integer"" notnull = 0 else: - type_ = column_details[r[0]].type - notnull = column_details[r[0]].notnull + try: + type_ = column_details[r[0]].type + notnull = column_details[r[0]].notnull + except KeyError: # Probably count(*) + type_ = ""integer"" + notnull = False columns.append( { ""name"": r[0], @@ -613,6 +617,15 @@ class TableView(RowTableShared): offset=offset, ) + # If ?_group_count we convert the SQL query here + group_count = request.args.getlist(""_group_count"") + if group_count: + wrapped_sql = ""select {cols}, count(*) from ({sql}) group by {cols}"".format( + cols="", "".join(group_count), + sql=sql, + ) + sql = wrapped_sql + if request.args.get(""_timelimit""): extra_args[""custom_time_limit""] = int(request.args.get(""_timelimit"")) ``` Resulted in errors like this one: ``` pk_path = path_from_row_pks(row, pks, not pks, False) File ""/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py"", line 82, in path_from_row_pks bits = [ File ""/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py"", line 83, in row[pk][""value""] if isinstance(row[pk], dict) else row[pk] for pk in pks IndexError: No item with that key ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",734777631,"""View all"" option for facets, to provide a (paginated) list of ALL of the facet counts plus a link to view them", https://github.com/simonw/datasette/issues/1080#issuecomment-1029691693,https://api.github.com/repos/simonw/datasette/issues/1080,1029691693,IC_kwDOBm6k_c49X9kt,9599,simonw,2022-02-04T06:16:45Z,2022-02-04T06:16:45Z,OWNER,"Had a new, different idea for how this could work: support a `?_group_count=colname` parameter to the table view, which turns the page into a `select colname, count(*) ... group by colname` query - but keeps things like the filter interface, facet selection, search box and so on.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",734777631,"""View all"" option for facets, to provide a (paginated) list of ALL of the facet counts plus a link to view them", https://github.com/simonw/sqlite-utils/issues/395#issuecomment-1029686150,https://api.github.com/repos/simonw/sqlite-utils/issues/395,1029686150,IC_kwDOCGYnMM49X8OG,9599,simonw,2022-02-04T06:03:51Z,2022-02-04T06:03:51Z,OWNER,I'm just going to run the SpatiaLite tests on Ubuntu.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1123849278,"""apt-get: command not found"" error on macOS", https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1029683977,https://api.github.com/repos/simonw/sqlite-utils/issues/79,1029683977,IC_kwDOCGYnMM49X7sJ,9599,simonw,2022-02-04T05:58:15Z,2022-02-04T05:58:15Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#spatialite-helpers,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557842245,Helper methods for working with SpatiaLite, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029682294,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029682294,IC_kwDOCGYnMM49X7R2,9599,simonw,2022-02-04T05:53:26Z,2022-02-04T05:53:26Z,OWNER,"This looks fantastic, thanks for all of the work you put into this!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods, https://github.com/simonw/sqlite-utils/issues/352#issuecomment-1029479388,https://api.github.com/repos/simonw/sqlite-utils/issues/352,1029479388,IC_kwDOCGYnMM49XJvc,9599,simonw,2022-02-03T22:59:35Z,2022-02-03T22:59:35Z,OWNER,"Ran into this bug again while writing tests for this: - #186","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1072792507,`sqlite-utils insert --extract colname`, https://github.com/simonw/sqlite-utils/issues/363#issuecomment-1029475387,https://api.github.com/repos/simonw/sqlite-utils/issues/363,1029475387,IC_kwDOCGYnMM49XIw7,9599,simonw,2022-02-03T22:52:30Z,2022-02-03T22:52:30Z,OWNER,"Demos: ``` % sqlite-utils insert /tmp/all.db blah /tmp/log.log --convert '[1]' --text Error: Rows must all be dictionaries, got: 1 % sqlite-utils insert /tmp/all.db blah /tmp/log.log --convert '1' --text Error: --convert must return dict or iterator ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094981339,Better error message if `--convert` code fails to return a dict, https://github.com/simonw/sqlite-utils/issues/363#issuecomment-1029469630,https://api.github.com/repos/simonw/sqlite-utils/issues/363,1029469630,IC_kwDOCGYnMM49XHW-,9599,simonw,2022-02-03T22:42:36Z,2022-02-03T22:42:36Z,OWNER,"> This check should run inside the `.insert_all()` method. It should raise a custom exception which the CLI code can then catch and turn into a click error. Actually no that doesn't work, because this line causes an error before we even get to `.insert_all()`: https://github.com/simonw/sqlite-utils/blob/7d928f83085fb285f294dbdaeb93fd94a44d5d44/sqlite_utils/cli.py#L1012-L1013","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094981339,Better error message if `--convert` code fails to return a dict, https://github.com/simonw/sqlite-utils/issues/393#issuecomment-1029450617,https://api.github.com/repos/simonw/sqlite-utils/issues/393,1029450617,IC_kwDOCGYnMM49XCt5,9599,simonw,2022-02-03T22:13:24Z,2022-02-03T22:13:24Z,OWNER,Much better: https://sqlite-utils.datasette.io/en/latest/python-api.html#insert-replacing-data,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1118585417,Better documentation for insert-replace, https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1029402837,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1029402837,IC_kwDOCGYnMM49W3DV,9599,simonw,2022-02-03T21:07:35Z,2022-02-03T21:07:35Z,OWNER,"Closing this - it was something I was curious about, but evidently not curious enough to actually do the work!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes, https://github.com/simonw/sqlite-utils/issues/394#issuecomment-1029402029,https://api.github.com/repos/simonw/sqlite-utils/issues/394,1029402029,IC_kwDOCGYnMM49W22t,9599,simonw,2022-02-03T21:06:35Z,2022-02-03T21:06:35Z,OWNER,"This broke on Windows: https://github.com/simonw/sqlite-utils/runs/5056912641 ``` if recreate and os.path.exists(filename_or_conn): > os.remove(filename_or_conn) E PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_recreate_False_True_0\\data.db' ``` I'm going to revert it from `main` for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1122446693,Test against Python 3.11-dev, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029335225,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029335225,IC_kwDOCGYnMM49Wmi5,9599,simonw,2022-02-03T19:39:40Z,2022-02-03T19:39:40Z,OWNER,"> I thought about adding these as methods on `Database` and `Table`, and I'm back and forth on it for the same reasons you are. It's certainly cleaner, and it's clearer what you're operating on. I could go either way. > > I do sort of like having all the Spatialite stuff in its own module, just because it's built around an extension you might not have or want, but I don't know if that's a good reason to have a different API. > > You could have `init_spatialite` add methods to `Database` and `Table`, so they're only there if you have Spatialite set up. Is that too clever? It feels too clever. Yeah that's too clever. You know what? I'm pretty confident we are both massively over-thinking this. We should put the methods on `Database` and `Table`! API simplicity and consistency matters more than vague concerns about purity.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029297971,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029297971,IC_kwDOCGYnMM49Wdcz,9599,simonw,2022-02-03T18:52:50Z,2022-02-03T18:52:50Z,OWNER,"I'm not sure I like `name=""geometry""` as the default argument to `add_geometry_column` - mainly because of this example here: ```python add_geometry_column(db[""locations""], ""POINT"") create_spatial_index(db[""locations""], ""geometry"") ``` I had to go and look at the code to figure out if `""POINT""` was the name of the column - and I don't like how inconsistent it looks next to the following `create_spatial_index()` call where you DO need to pass the column name.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029296782,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029296782,IC_kwDOCGYnMM49WdKO,9599,simonw,2022-02-03T18:51:21Z,2022-02-03T18:51:21Z,OWNER,"What do you think about adding these as methods on the `Database` class instead? Then you could do: ```python # This is with an optional argument, which if omitted runs find_spatialite() for you: db.init_spatialite() # Instead of: init_spatialite(db, find_spatialite()) ``` Likewise, the `add_geometry_column` and `create_spatial_index` methods could live on `Table`: ```python # Instead of this: add_geometry_column(db[""locations""], ""POINT"", ""geometry"") create_spatial_index(db[""locations""], ""geometry"") # Could have this: db[""locations""].add_geometry_column(""POINT"") db[""locations""].create_spatial_index(""geometry"") ``` On the one hand, this is much more consistent with the existing `sqlite-utils` Python API. But on the other hand... this is mixing SpatiaLite functionality directly into the core classes. Is that a good idea, seeing as SpatiaLite is both an optional extension (which can be tricky to install) AND something that has a very different release cadence and quality-of-documentation from SQLite itself? There's a third option: the SpatiaLite could exist on subclasses of `Database` and `Table` - so the above examples would look something like this: ```python from sqlite_utils.gis import SpatiaLiteDatabase db = SpatiaLiteDatabase(""geo.db"") db.init_spatialite() db[""locations""].add_geometry_column(""POINT"") db[""locations""].create_spatial_index(""geometry"") ``` On the one hand, this would keep the SpatiaLite-specific stuff out of the core Database/Table classes. But it feels a bit untidy to me, especially since it raises the spectre of someone who was already subclassing Database for some reason now needing to instead subclass `SpatiaLiteDatabase` (not too keen on that capitalization) - or even (horror) trying to dabble with multiple inheritance, which can only lead to pain. So I don't have a strong opinion formed on this question yet!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029285985,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029285985,IC_kwDOCGYnMM49Wahh,9599,simonw,2022-02-03T18:37:48Z,2022-02-03T18:37:48Z,OWNER,"`from sqlite_utils.utils import find_spatialite` is part of the documented API already: https://sqlite-utils.datasette.io/en/3.22.1/python-api.html#finding-spatialite To avoid needing to bump the major version number to 4 to indicate a backwards incompatible change, we should keep a `from .gis import find_spatialite` line at the top of `utils.py` such that any existing code with that documented import continues to work.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029273853,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029273853,IC_kwDOCGYnMM49WXj9,9599,simonw,2022-02-03T18:23:30Z,2022-02-03T18:31:21Z,OWNER,"OK, this change makes a bunch of sense to me - and also raises some interesting questions about future additions to `sqlite-utils` with regards to SpatiaLite. Would `sqlite-utils add-geometry-column ...` be a good CLI enhancement. for example? I see you've already talked about that in #79 - moving this conversation there!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029177015,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029177015,IC_kwDOCGYnMM49V_63,9599,simonw,2022-02-03T16:38:02Z,2022-02-03T16:38:02Z,OWNER,Sorry had missed this - tests should run now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods, https://github.com/simonw/datasette/issues/1545#issuecomment-1028535868,https://api.github.com/repos/simonw/datasette/issues/1545,1028535868,IC_kwDOBm6k_c49TjY8,9599,simonw,2022-02-03T02:08:30Z,2022-02-03T02:08:30Z,OWNER,"Filed an issue with Jinja suggesting a documentation update: - https://github.com/pallets/jinja/issues/1578","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1075893249,Custom pages don't work on windows, https://github.com/simonw/datasette/pull/1617#issuecomment-1028519382,https://api.github.com/repos/simonw/datasette/issues/1617,1028519382,IC_kwDOBm6k_c49TfXW,9599,simonw,2022-02-03T01:31:25Z,2022-02-03T01:31:25Z,OWNER,"I was hoping to get the test suite running on Windows before merging this PR but that looks like it will be a BIG job, see: - #1627 So I'm going to merge this one as-is for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1120990806,"Ensure template_path always uses ""/"" to match jinja", https://github.com/simonw/datasette/issues/1545#issuecomment-1028517268,https://api.github.com/repos/simonw/datasette/issues/1545,1028517268,IC_kwDOBm6k_c49Te2U,9599,simonw,2022-02-03T01:26:53Z,2022-02-03T01:26:53Z,OWNER,"I understand the problem now! https://github.com/pallets/jinja/issues/1378#issuecomment-812410922 > Jinja template names/paths are not always filesystem paths. So regardless of the OS Jinja always uses forward slashes.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1075893249,Custom pages don't work on windows, https://github.com/simonw/datasette/pull/1617#issuecomment-1028517073,https://api.github.com/repos/simonw/datasette/issues/1617,1028517073,IC_kwDOBm6k_c49TezR,9599,simonw,2022-02-03T01:26:32Z,2022-02-03T01:26:32Z,OWNER,"Aha I understand the problem now! https://github.com/pallets/jinja/issues/1378#issuecomment-812410922 > Jinja template names/paths are not always filesystem paths. So regardless of the OS Jinja always uses forward slashes.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1120990806,"Ensure template_path always uses ""/"" to match jinja", https://github.com/simonw/datasette/pull/1626#issuecomment-1028515161,https://api.github.com/repos/simonw/datasette/issues/1626,1028515161,IC_kwDOBm6k_c49TeVZ,9599,simonw,2022-02-03T01:22:43Z,2022-02-03T01:22:43Z,OWNER,"OK, the tests do NOT pass against Windows! https://github.com/simonw/datasette/runs/5044105941 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1122451096,Try test suite against macOS and Windows, https://github.com/simonw/datasette/issues/1534#issuecomment-1028461220,https://api.github.com/repos/simonw/datasette/issues/1534,1028461220,IC_kwDOBm6k_c49TRKk,9599,simonw,2022-02-02T23:39:33Z,2022-02-02T23:39:33Z,OWNER,"I've decided not to do this, because of the risk that Cloudflare could cache the JSON version for an HTML page or vice-versa.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065432388,Maybe return JSON from HTML pages if `Accept: application/json` is sent, https://github.com/simonw/datasette/pull/1626#issuecomment-1028420821,https://api.github.com/repos/simonw/datasette/issues/1626,1028420821,IC_kwDOBm6k_c49THTV,9599,simonw,2022-02-02T22:32:26Z,2022-02-02T22:33:31Z,OWNER,"That broke on a macOS test: https://github.com/simonw/datasette/runs/5044036993?check_suite_focus=true I'm going to remove macOS and Ubuntu and just try Windows purely to see what happens there.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1122451096,Try test suite against macOS and Windows, https://github.com/simonw/datasette/pull/1616#issuecomment-1028414871,https://api.github.com/repos/simonw/datasette/issues/1616,1028414871,IC_kwDOBm6k_c49TF2X,9599,simonw,2022-02-02T22:23:45Z,2022-02-02T22:23:45Z,OWNER,First stable Black release!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1119413338,Bump black from 21.12b0 to 22.1.0, https://github.com/simonw/datasette/issues/1623#issuecomment-1028397935,https://api.github.com/repos/simonw/datasette/issues/1623,1028397935,IC_kwDOBm6k_c49TBtv,9599,simonw,2022-02-02T21:59:43Z,2022-02-02T21:59:43Z,OWNER,Here's the new test: https://github.com/simonw/datasette/blob/23a09b0f6af33c52acf8c1d9002fe475b42fee10/tests/test_html.py#L927-L936,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1122416919,/-/patterns returns link: alternate JSON header to 404, https://github.com/simonw/datasette/issues/1624#issuecomment-1028396866,https://api.github.com/repos/simonw/datasette/issues/1624,1028396866,IC_kwDOBm6k_c49TBdC,9599,simonw,2022-02-02T21:58:06Z,2022-02-02T21:58:06Z,OWNER,"It looks like this is because `IndexView` extends `BaseView` rather than extending `DataView` which is where all that CORS stuff happens: https://github.com/simonw/datasette/blob/23a09b0f6af33c52acf8c1d9002fe475b42fee10/datasette/views/index.py#L18-L21 Another thing I should address with the refactor project in: - #878 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1122427321,Index page `/` has no CORS headers, https://github.com/simonw/datasette/issues/1620#issuecomment-1028393259,https://api.github.com/repos/simonw/datasette/issues/1620,1028393259,IC_kwDOBm6k_c49TAkr,9599,simonw,2022-02-02T21:53:02Z,2022-02-02T21:53:02Z,OWNER,"I ran the following on https://www.google.com/ in the console to demonstrate that these work as intended: ```javascript [ ""https://latest.datasette.io/fixtures"", ""https://latest.datasette.io/fixtures?sql=select+1"", ""https://latest.datasette.io/fixtures/facetable"" ].forEach(async (url) => { response = await fetch(url, {method: ""HEAD""}); console.log(response.headers.get(""Link"")); }); ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121618041,"Link: rel=""alternate"" to JSON for queries too", https://github.com/simonw/datasette/issues/1623#issuecomment-1028389953,https://api.github.com/repos/simonw/datasette/issues/1623,1028389953,IC_kwDOBm6k_c49S_xB,9599,simonw,2022-02-02T21:48:34Z,2022-02-02T21:48:34Z,OWNER,"A few other pages do that too, including: - https://latest.datasette.io/-/messages - https://latest.datasette.io/-/allow-debug","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1122416919,/-/patterns returns link: alternate JSON header to 404, https://github.com/simonw/datasette/issues/1620#issuecomment-1028385067,https://api.github.com/repos/simonw/datasette/issues/1620,1028385067,IC_kwDOBm6k_c49S-kr,9599,simonw,2022-02-02T21:42:23Z,2022-02-02T21:42:23Z,OWNER,"``` % curl -s -I 'https://latest.datasette.io/' | grep link link: https://latest.datasette.io/.json; rel=""alternate""; type=""application/json+datasette"" % curl -s -I 'https://latest.datasette.io/fixtures' | grep link link: https://latest.datasette.io/fixtures.json; rel=""alternate""; type=""application/json+datasette"" % curl -s -I 'https://latest.datasette.io/fixtures?sql=select+1' | grep link link: https://latest.datasette.io/fixtures.json?sql=select+1; rel=""alternate""; type=""application/json+datasette"" % curl -s -I 'https://latest.datasette.io/-/plugins' | grep link link: https://latest.datasette.io/-/plugins.json; rel=""alternate""; type=""application/json+datasette"" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121618041,"Link: rel=""alternate"" to JSON for queries too", https://github.com/simonw/datasette/issues/1620#issuecomment-1028374330,https://api.github.com/repos/simonw/datasette/issues/1620,1028374330,IC_kwDOBm6k_c49S786,9599,simonw,2022-02-02T21:28:16Z,2022-02-02T21:28:16Z,OWNER,I just realized I can refactor this to make it much simpler.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121618041,"Link: rel=""alternate"" to JSON for queries too", https://github.com/simonw/datasette/issues/1533#issuecomment-1027672617,https://api.github.com/repos/simonw/datasette/issues/1533,1027672617,IC_kwDOBm6k_c49QQop,9599,simonw,2022-02-02T07:56:51Z,2022-02-02T07:56:51Z,OWNER,"Demos - these pages both have ` Table-valued functions exist only for PRAGMAs that return results and that have no side-effects. So it's possible I'm being overly paranoid here after all: what I want to block here is people running things like `PRAGMA case_sensitive_like = 1` which could affect the global state for that connection and cause unexpected behaviour later on. So maybe I should allow all pragma functions. I previously allowed an allow-list of them in: - #761","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121121305,"Reconsider policy on blocking queries containing the string ""pragma""", https://github.com/simonw/datasette/issues/1618#issuecomment-1027653005,https://api.github.com/repos/simonw/datasette/issues/1618,1027653005,IC_kwDOBm6k_c49QL2N,9599,simonw,2022-02-02T07:22:13Z,2022-02-02T07:22:13Z,OWNER,"There's a workaround for this at the moment, which is to use parameterized SQL queries. For example, this: https://fivethirtyeight.datasettes.com/polls?sql=select+*+from+books+where+title+%3D+%3Atitle&title=The+Pragmatic+Programmer So the SQL query is `select * from books where title = :title` and then `&title=...` is added to the URL. The reason behind the quite aggressive pragma filtering is that SQLite allows you to execute pragmas using function calls, like this one: ```sql SELECT * FROM pragma_index_info('idx52'); ``` These can be nested arbitrarily deeply in sub-queries, so it's difficult to write a regular expression that will definitely catch them. I'm open to relaxing the regex a bit, but I need to be very confident that it's safe to do so. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121121305,"Reconsider policy on blocking queries containing the string ""pragma""", https://github.com/simonw/datasette/issues/1586#issuecomment-1027648180,https://api.github.com/repos/simonw/datasette/issues/1586,1027648180,IC_kwDOBm6k_c49QKq0,9599,simonw,2022-02-02T07:13:31Z,2022-02-02T07:13:31Z,OWNER,"Running it as part of `datasette publish` is a smart idea - I'm slightly nervous about modifying the database file that has been published though, since part of the undocumented contract right now is that the bytes served are the exact same bytes as the ones you ran the publish against. But there's no reason for that expectation to exist, and I doubt anyone is relying on that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096536240,run analyze on all databases as part of start up or publishing, https://github.com/simonw/datasette/issues/1619#issuecomment-1027647257,https://api.github.com/repos/simonw/datasette/issues/1619,1027647257,IC_kwDOBm6k_c49QKcZ,9599,simonw,2022-02-02T07:11:43Z,2022-02-02T07:11:43Z,OWNER,Weirdly the bug does NOT exhibit itself on this demo: https://datasette-apache-proxy-demo.datasette.io/prefix/fixtures/no_primary_key/1 - which correctly links to https://datasette-apache-proxy-demo.datasette.io/prefix/fixtures/no_primary_key/1.json,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121583414,JSON link on row page is 404 if base_url setting is used, https://github.com/simonw/datasette/issues/1619#issuecomment-1027646659,https://api.github.com/repos/simonw/datasette/issues/1619,1027646659,IC_kwDOBm6k_c49QKTD,9599,simonw,2022-02-02T07:10:37Z,2022-02-02T07:10:37Z,OWNER,It's not just the table with slashes in the name. Same thing on http://127.0.0.1:3344/foo/bar/fixtures/attraction_characteristic/1 - the `json` link goes to a JSON-rendered 404 on http://127.0.0.1:3344/foo/bar/foo/bar/fixtures/attraction_characteristic/1.json,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121583414,JSON link on row page is 404 if base_url setting is used, https://github.com/simonw/datasette/issues/1576#issuecomment-1027635925,https://api.github.com/repos/simonw/datasette/issues/1576,1027635925,IC_kwDOBm6k_c49QHrV,9599,simonw,2022-02-02T06:47:20Z,2022-02-02T06:47:20Z,OWNER,"Here's what I was hacking around with when I uncovered this problem: ```diff diff --git a/datasette/views/table.py b/datasette/views/table.py index 77fb285..8c57d08 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -1,3 +1,4 @@ +import asyncio import urllib import itertools import json @@ -615,44 +616,37 @@ class TableView(RowTableShared): if request.args.get(""_timelimit""): extra_args[""custom_time_limit""] = int(request.args.get(""_timelimit"")) - # Execute the main query! - results = await db.execute(sql, params, truncate=True, **extra_args) - - # Calculate the total count for this query - filtered_table_rows_count = None - if ( - not db.is_mutable - and self.ds.inspect_data - and count_sql == f""select count(*) from {table} "" - ): - # We can use a previously cached table row count - try: - filtered_table_rows_count = self.ds.inspect_data[database][""tables""][ - table - ][""count""] - except KeyError: - pass - - # Otherwise run a select count(*) ... - if count_sql and filtered_table_rows_count is None and not nocount: - try: - count_rows = list(await db.execute(count_sql, from_sql_params)) - filtered_table_rows_count = count_rows[0][0] - except QueryInterrupted: - pass - - # Faceting - if not self.ds.setting(""allow_facet"") and any( - arg.startswith(""_facet"") for arg in request.args - ): - raise BadRequest(""_facet= is not allowed"") + async def execute_count(): + # Calculate the total count for this query + filtered_table_rows_count = None + if ( + not db.is_mutable + and self.ds.inspect_data + and count_sql == f""select count(*) from {table} "" + ): + # We can use a previously cached table row count + try: + filtered_table_rows_count = self.ds.inspect_data[database][ + ""tables"" + ][table][""count""] + except KeyError: + pass + + if count_sql and filtered_table_rows_count is None and not nocount: + try: + count_rows = list(await db.execute(count_sql, from_sql_params)) + filtered_table_rows_count = count_rows[0][0] + except QueryInterrupted: + pass + + return filtered_table_rows_count + + filtered_table_rows_count = await execute_count() # pylint: disable=no-member facet_classes = list( itertools.chain.from_iterable(pm.hook.register_facet_classes()) ) - facet_results = {} - facets_timed_out = [] facet_instances = [] for klass in facet_classes: facet_instances.append( @@ -668,33 +662,58 @@ class TableView(RowTableShared): ) ) - if not nofacet: - for facet in facet_instances: - ( - instance_facet_results, - instance_facets_timed_out, - ) = await facet.facet_results() - for facet_info in instance_facet_results: - base_key = facet_info[""name""] - key = base_key - i = 1 - while key in facet_results: - i += 1 - key = f""{base_key}_{i}"" - facet_results[key] = facet_info - facets_timed_out.extend(instance_facets_timed_out) - - # Calculate suggested facets - suggested_facets = [] - if ( - self.ds.setting(""suggest_facets"") - and self.ds.setting(""allow_facet"") - and not _next - and not nofacet - and not nosuggest - ): - for facet in facet_instances: - suggested_facets.extend(await facet.suggest()) + async def execute_suggested_facets(): + # Calculate suggested facets + suggested_facets = [] + if ( + self.ds.setting(""suggest_facets"") + and self.ds.setting(""allow_facet"") + and not _next + and not nofacet + and not nosuggest + ): + for facet in facet_instances: + suggested_facets.extend(await facet.suggest()) + return suggested_facets + + async def execute_facets(): + facet_results = {} + facets_timed_out = [] + if not self.ds.setting(""allow_facet"") and any( + arg.startswith(""_facet"") for arg in request.args + ): + raise BadRequest(""_facet= is not allowed"") + + if not nofacet: + for facet in facet_instances: + ( + instance_facet_results, + instance_facets_timed_out, + ) = await facet.facet_results() + for facet_info in instance_facet_results: + base_key = facet_info[""name""] + key = base_key + i = 1 + while key in facet_results: + i += 1 + key = f""{base_key}_{i}"" + facet_results[key] = facet_info + facets_timed_out.extend(instance_facets_timed_out) + + return facet_results, facets_timed_out + + # Execute the main query, facets and facet suggestions in parallel: + ( + results, + suggested_facets, + (facet_results, facets_timed_out), + ) = await asyncio.gather( + db.execute(sql, params, truncate=True, **extra_args), + execute_suggested_facets(), + execute_facets(), + ) + + results = await db.execute(sql, params, truncate=True, **extra_args) # Figure out columns and rows for the query columns = [r[0] for r in results.description] ``` It's a hacky attempt at running some of the table page queries in parallel to see what happens.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/1611#issuecomment-1027635175,https://api.github.com/repos/simonw/datasette/issues/1611,1027635175,IC_kwDOBm6k_c49QHfn,9599,simonw,2022-02-02T06:45:47Z,2022-02-02T06:45:47Z,OWNER,"Prototype, not sure that this actually works yet: ```diff diff --git a/datasette/database.py b/datasette/database.py index 6ce8721..0c4aec7 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -256,18 +256,26 @@ class Database: # Try to get counts for each table, $limit timeout for each count counts = {} for table in await self.table_names(): - try: - table_count = ( - await self.execute( - f""select count(*) from [{table}]"", - custom_time_limit=limit, - ) - ).rows[0][0] - counts[table] = table_count - # In some cases I saw ""SQL Logic Error"" here in addition to - # QueryInterrupted - so we catch that too: - except (QueryInterrupted, sqlite3.OperationalError, sqlite3.DatabaseError): - counts[table] = None + print(table.lower()) + if table.lower() == ""knn"": + counts[table] = 0 + else: + try: + table_count = ( + await self.execute( + f""select count(*) from [{table}]"", + custom_time_limit=limit, + ) + ).rows[0][0] + counts[table] = table_count + # In some cases I saw ""SQL Logic Error"" here in addition to + # QueryInterrupted - so we catch that too: + except ( + QueryInterrupted, + sqlite3.OperationalError, + sqlite3.DatabaseError, + ): + counts[table] = None if not self.is_mutable: self._cached_table_counts = counts return counts ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1113384383,Avoid ever running count(*) against SpatiaLite KNN table, https://github.com/simonw/datasette/issues/1607#issuecomment-1027634490,https://api.github.com/repos/simonw/datasette/issues/1607,1027634490,IC_kwDOBm6k_c49QHU6,9599,simonw,2022-02-02T06:44:30Z,2022-02-02T06:44:30Z,OWNER,"Prototype: ```diff diff --git a/datasette/app.py b/datasette/app.py index 09d7d03..e2a5aea 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -724,6 +724,47 @@ class Datasette: sqlite_extensions[extension] = None except Exception: pass + # More details on SpatiaLite + if ""spatialite"" in sqlite_extensions: + spatialite_details = {} + fns = ( + ""spatialite_version"", + ""spatialite_target_cpu"", + ""rcheck_strict_sql_quoting"", + ""freexl_version"", + ""proj_version"", + ""geos_version"", + ""rttopo_version"", + ""libxml2_version"", + ""HasIconv"", + ""HasMathSQL"", + ""HasGeoCallbacks"", + ""HasProj"", + ""HasProj6"", + ""HasGeos"", + ""HasGeosAdvanced"", + ""HasGeosTrunk"", + ""HasGeosReentrant"", + ""HasGeosOnlyReentrant"", + ""HasMiniZip"", + ""HasRtTopo"", + ""HasLibXML2"", + ""HasEpsg"", + ""HasFreeXL"", + ""HasGeoPackage"", + ""HasGCP"", + ""HasTopology"", + ""HasKNN"", + ""HasRouting"", + ) + for fn in fns: + try: + result = conn.execute(""select {}()"".format(fn)) + spatialite_details[fn] = result.fetchone()[0] + except Exception: + pass + sqlite_extensions[""spatialite""] = spatialite_details + # Figure out supported FTS versions ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109783030,More detailed information about installed SpatiaLite version, https://github.com/simonw/datasette/issues/1533#issuecomment-1027633686,https://api.github.com/repos/simonw/datasette/issues/1533,1027633686,IC_kwDOBm6k_c49QHIW,9599,simonw,2022-02-02T06:42:53Z,2022-02-02T06:42:53Z,OWNER,"I'm going to apply the hack, then fix it again in: - #1518","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065431383,"Add `Link: rel=""alternate""` header pointing to JSON for a table/query", https://github.com/simonw/sqlite-utils/issues/352#issuecomment-1024727476,https://api.github.com/repos/simonw/sqlite-utils/issues/352,1024727476,IC_kwDOCGYnMM49FBm0,9599,simonw,2022-01-28T23:01:06Z,2022-01-28T23:01:06Z,OWNER,"Manual test run with that prototype: ``` % echo '{""foo"": ""bar"", ""id"": 1}' | sqlite-utils insert insert-extract.db rows - --pk id --extract foo % sqlite-utils dump insert-extract.db BEGIN TRANSACTION; CREATE TABLE [foo] ( [id] INTEGER PRIMARY KEY, [value] TEXT ); INSERT INTO ""foo"" VALUES(1,'bar'); CREATE TABLE [rows] ( [foo] INTEGER REFERENCES [foo]([id]), [id] INTEGER PRIMARY KEY ); INSERT INTO ""rows"" VALUES(1,1); CREATE UNIQUE INDEX [idx_foo_value] ON [foo] ([value]); COMMIT; ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1072792507,`sqlite-utils insert --extract colname`, https://github.com/simonw/sqlite-utils/issues/352#issuecomment-1024726879,https://api.github.com/repos/simonw/sqlite-utils/issues/352,1024726879,IC_kwDOCGYnMM49FBdf,9599,simonw,2022-01-28T22:59:44Z,2022-01-28T22:59:44Z,OWNER,"Rough prototype, seems to work:; ```diff diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py index 771d432..431b93e 100644 --- a/sqlite_utils/cli.py +++ b/sqlite_utils/cli.py @@ -867,6 +867,12 @@ def insert_upsert_options(*, require_pk=False): ), load_extension_option, click.option(""--silent"", is_flag=True, help=""Do not show progress bar""), + click.option( + ""--extract"", + ""extracts"", + multiple=True, + help=""Columns to extract to another table"", + ), ) ): fn = decorator(fn) @@ -906,6 +912,7 @@ def insert_upsert_implementation( load_extension=None, silent=False, bulk_sql=None, + extracts=None, ): db = sqlite_utils.Database(path) _load_extensions(db, load_extension) @@ -1008,6 +1015,8 @@ def insert_upsert_implementation( extra_kwargs[""defaults""] = dict(default) if upsert: extra_kwargs[""upsert""] = upsert + if extracts is not None: + extra_kwargs[""extracts""] = extracts # Apply {""$base64"": true, ...} decoding, if needed docs = (decode_base64_values(doc) for doc in docs) @@ -1117,6 +1126,7 @@ def insert( truncate, not_null, default, + extracts, ): """""" Insert records from FILE into a table, creating the table if it @@ -1174,6 +1184,7 @@ def insert( silent=silent, not_null=not_null, default=default, + extracts=extracts, ) except UnicodeDecodeError as ex: raise click.ClickException(UNICODE_ERROR.format(ex)) @@ -1207,6 +1218,7 @@ def upsert( analyze, load_extension, silent, + extracts, ): """""" Upsert records based on their primary key. Works like 'insert' but if ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1072792507,`sqlite-utils insert --extract colname`, https://github.com/simonw/sqlite-utils/issues/392#issuecomment-1022466476,https://api.github.com/repos/simonw/sqlite-utils/issues/392,1022466476,IC_kwDOCGYnMM488Zms,9599,simonw,2022-01-26T18:17:43Z,2022-01-26T18:17:43Z,OWNER,"Manually tested it like this: ``` # Create database with an empty ""lines"" table sqlite-utils create-table bulk-test.db lines line text # Stream records every 0.5s, commit every 5 records stream-delay docs/python-api.rst -d 500 | \ sqlite-utils bulk bulk-test.db 'insert into lines (line) values (:line)' - \ --lines --batch-size 5 ``` Running `datasette bulk-test.db` showed that records would show up about every 2.5s five at a time.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114640101,`sqlite-utils bulk --batch-size` option, https://github.com/simonw/datasette/issues/1613#issuecomment-1022381732,https://api.github.com/repos/simonw/datasette/issues/1613,1022381732,IC_kwDOBm6k_c488E6k,9599,simonw,2022-01-26T16:41:45Z,2022-01-26T16:41:45Z,OWNER,A better interface for modifying the columns used in the SELECT clause would be useful too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114628238,Improvements to help make Datasette a better tool for learning SQL, https://github.com/simonw/datasette/issues/1613#issuecomment-1022255862,https://api.github.com/repos/simonw/datasette/issues/1613,1022255862,IC_kwDOBm6k_c487mL2,9599,simonw,2022-01-26T14:35:31Z,2022-01-26T14:37:44Z,OWNER,"Joins are really hard. A mechanism for constructing them in the table view would help a lot: - https://github.com/simonw/datasette/issues/613","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114628238,Improvements to help make Datasette a better tool for learning SQL, https://github.com/simonw/datasette/issues/1613#issuecomment-1022254258,https://api.github.com/repos/simonw/datasette/issues/1613,1022254258,IC_kwDOBm6k_c487lyy,9599,simonw,2022-01-26T14:33:46Z,2022-01-26T14:37:31Z,OWNER,"Tool for setting up foreign key relationships. It could even verify the relationship before you apply it - checking that every value in the column does indeed correspond to a value in the other table. Could also detect and suggest possible ones.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114628238,Improvements to help make Datasette a better tool for learning SQL, https://github.com/simonw/datasette/issues/1613#issuecomment-1022257496,https://api.github.com/repos/simonw/datasette/issues/1613,1022257496,IC_kwDOBm6k_c487mlY,9599,simonw,2022-01-26T14:37:14Z,2022-01-26T14:37:14Z,OWNER,"Better contextual help on the SQL editor - like in Django SQL Dashboard which shows all available tables and columns. Fancy inline autocomplete would be great too, but that's pretty hard for SQL based on past research.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114628238,Improvements to help make Datasette a better tool for learning SQL, https://github.com/simonw/sqlite-utils/issues/392#issuecomment-1021877769,https://api.github.com/repos/simonw/sqlite-utils/issues/392,1021877769,IC_kwDOCGYnMM486J4J,9599,simonw,2022-01-26T05:19:48Z,2022-01-26T05:19:48Z,OWNER,Can use this utility function: https://github.com/simonw/sqlite-utils/blob/a9fca7efa4184fbb2a65ca1275c326950ed9d3c1/sqlite_utils/utils.py#L322-L325,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114640101,`sqlite-utils bulk --batch-size` option, https://github.com/simonw/sqlite-utils/issues/392#issuecomment-1021877058,https://api.github.com/repos/simonw/sqlite-utils/issues/392,1021877058,IC_kwDOCGYnMM486JtC,9599,simonw,2022-01-26T05:18:12Z,2022-01-26T05:18:18Z,OWNER,"Help for `insert` says: ``` --batch-size INTEGER Commit every X records ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114640101,`sqlite-utils bulk --batch-size` option, https://github.com/simonw/sqlite-utils/issues/392#issuecomment-1021876914,https://api.github.com/repos/simonw/sqlite-utils/issues/392,1021876914,IC_kwDOCGYnMM486Jqy,9599,simonw,2022-01-26T05:17:49Z,2022-01-26T05:17:49Z,OWNER,Relevant code: https://github.com/simonw/sqlite-utils/blob/a9fca7efa4184fbb2a65ca1275c326950ed9d3c1/sqlite_utils/cli.py#L1014-L1018,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114640101,`sqlite-utils bulk --batch-size` option, https://github.com/simonw/sqlite-utils/issues/391#issuecomment-1021876463,https://api.github.com/repos/simonw/sqlite-utils/issues/391,1021876463,IC_kwDOCGYnMM486Jjv,9599,simonw,2022-01-26T05:16:51Z,2022-01-26T05:16:51Z,OWNER,"Actually adding a progress bar may not make sense here: it's designed to work with streaming input from stdin, in which case it's impossible for it to know the overall number of rows to be processed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114638930,`sqlite-utils bulk` progress bar, https://github.com/simonw/sqlite-utils/issues/391#issuecomment-1021876055,https://api.github.com/repos/simonw/sqlite-utils/issues/391,1021876055,IC_kwDOCGYnMM486JdX,9599,simonw,2022-01-26T05:15:58Z,2022-01-26T05:15:58Z,OWNER,Could add support for `--batch-size` as seen in `insert`/`upsert` too - causing it to break the list up into batches and commit for each one.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114638930,`sqlite-utils bulk` progress bar, https://github.com/simonw/datasette/issues/1613#issuecomment-1021860694,https://api.github.com/repos/simonw/datasette/issues/1613,1021860694,IC_kwDOBm6k_c486FtW,9599,simonw,2022-01-26T04:57:53Z,2022-01-26T04:57:53Z,OWNER,"The existing flow where you can apply filters to a table and then click ""View and edit SQL"" to see the query is a good starting point. Group by queries are both crucially important and difficult to assemble for beginners. Providing a way to see the query that was used by a facet (since facets are really just group-by-counts) would be very useful, which could come out of this: - #1080","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114628238,Improvements to help make Datasette a better tool for learning SQL, https://github.com/simonw/sqlite-utils/issues/390#issuecomment-1021825322,https://api.github.com/repos/simonw/sqlite-utils/issues/390,1021825322,IC_kwDOCGYnMM4859Eq,9599,simonw,2022-01-26T03:20:12Z,2022-01-26T03:20:25Z,OWNER,"``` % sqlite-utils upsert trees.db blah - Usage: sqlite-utils upsert [OPTIONS] PATH TABLE FILE Try 'sqlite-utils upsert -h' for help. Error: Missing option '--pk'. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114557284,`sqlite-utils upsert` should require `--pk` more elegantly, https://github.com/simonw/sqlite-utils/issues/48#issuecomment-1021790707,https://api.github.com/repos/simonw/sqlite-utils/issues/48,1021790707,IC_kwDOCGYnMM4850nz,9599,simonw,2022-01-26T02:08:39Z,2022-01-26T02:08:39Z,OWNER,This is a dupe of #308.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",471818939,"Jupyter notebook demo of the library, launchable on Binder", https://github.com/simonw/sqlite-utils/issues/389#issuecomment-1021790136,https://api.github.com/repos/simonw/sqlite-utils/issues/389,1021790136,IC_kwDOCGYnMM4850e4,9599,simonw,2022-01-26T02:07:41Z,2022-01-26T02:07:41Z,OWNER,That fixed it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114544727,Plausible analytics for documentation, https://github.com/simonw/sqlite-utils/issues/389#issuecomment-1021789016,https://api.github.com/repos/simonw/sqlite-utils/issues/389,1021789016,IC_kwDOCGYnMM4850NY,9599,simonw,2022-01-26T02:05:41Z,2022-01-26T02:05:41Z,OWNER,"Oops, wrong domain in that - https://plausible.io/sqlite-utils.datasette.io is not yet showing data.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114544727,Plausible analytics for documentation, https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1021788260,https://api.github.com/repos/simonw/sqlite-utils/issues/388,1021788260,IC_kwDOCGYnMM4850Bk,9599,simonw,2022-01-26T02:04:17Z,2022-01-26T02:04:17Z,OWNER,https://sqlite-utils.datasette.io/en/3.20/ now also shows the version warning banner.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114543475,Link to stable docs from older versions, https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1021787822,https://api.github.com/repos/simonw/sqlite-utils/issues/388,1021787822,IC_kwDOCGYnMM485z6u,9599,simonw,2022-01-26T02:03:25Z,2022-01-26T02:03:25Z,OWNER,"https://sqlite-utils.datasette.io/en/latest/ ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114543475,Link to stable docs from older versions, https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1021785268,https://api.github.com/repos/simonw/sqlite-utils/issues/388,1021785268,IC_kwDOCGYnMM485zS0,9599,simonw,2022-01-26T01:58:19Z,2022-01-26T01:58:19Z,OWNER,"While I'm modifying that template I'm also going to add Plausible analytics: ```html ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114543475,Link to stable docs from older versions, https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1021784593,https://api.github.com/repos/simonw/sqlite-utils/issues/388,1021784593,IC_kwDOCGYnMM485zIR,9599,simonw,2022-01-26T01:56:48Z,2022-01-26T01:56:48Z,OWNER,"On https://readthedocs.org/dashboard/sqlite-utils/advanced/ I checked this box (previously unchecked): ![CleanShot 2022-01-25 at 17 56 22@2x](https://user-images.githubusercontent.com/9599/151090718-72e553a6-826a-43b6-98bd-f70b0cba8784.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114543475,Link to stable docs from older versions, https://github.com/simonw/sqlite-utils/issues/387#issuecomment-1021773056,https://api.github.com/repos/simonw/sqlite-utils/issues/387,1021773056,IC_kwDOCGYnMM485wUA,9599,simonw,2022-01-26T01:37:17Z,2022-01-26T01:37:17Z,OWNER,https://sqlite-utils.datasette.io/en/latest/python-api.html#getting-started,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1111293050,Python library docs should start with a self contained example, https://github.com/simonw/datasette/issues/1612#issuecomment-1021489826,https://api.github.com/repos/simonw/datasette/issues/1612,1021489826,IC_kwDOBm6k_c484rKi,9599,simonw,2022-01-25T18:34:21Z,2022-01-25T18:34:21Z,OWNER,"OK, that's live on https://latest.datasette.io/fixtures now - I really like it. Thanks for the suggestion!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114147905,Move canned queries closer to the SQL input area, https://github.com/simonw/datasette/issues/1612#issuecomment-1021477220,https://api.github.com/repos/simonw/datasette/issues/1612,1021477220,IC_kwDOBm6k_c484oFk,9599,simonw,2022-01-25T18:19:31Z,2022-01-25T18:19:31Z,OWNER,"Here's something I like: I also added a ""Tables"" `

` heading and bumped the tables themselves down from a `

` to a `

`: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114147905,Move canned queries closer to the SQL input area, https://github.com/simonw/datasette/issues/1612#issuecomment-1021472918,https://api.github.com/repos/simonw/datasette/issues/1612,1021472918,IC_kwDOBm6k_c484nCW,9599,simonw,2022-01-25T18:14:27Z,2022-01-25T18:15:54Z,OWNER,"They're currently shown at the very bottom of the page, under the list of tables and far away from the SQL query box: https://latest.datasette.io/fixtures I'm also questioning if ""Queries"" is the best header for this. Other options: - **Canned queries** (what the feature is called in the documentation, but I don't think it's a great user-facing term) - **Saved queries** - overlaps with a mechanism by which queries can be saved by the user using a plugin such as [datasette-saved-queries](https://github.com/simonw/datasette-saved-queries) - though that plugin does itself use the canned queries plugin hook so not completely unrelated - **Sample or Example queries** - I don't like these much because they're more than just examples - they are often the core functionality of the specific customized Datasette instance - **Prepared queries** - overlaps with terminology used in other databases, so not great either - **Pre-configured queries** - urgh, don't like that language, feels clumsy - **Query recipes** - bit out of left-field this one, only really makes sense for queries that include named parameters for specific use-cases Maybe ""Queries"" is right after all.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114147905,Move canned queries closer to the SQL input area, https://github.com/simonw/datasette/issues/1612#issuecomment-1021413700,https://api.github.com/repos/simonw/datasette/issues/1612,1021413700,IC_kwDOBm6k_c484YlE,9599,simonw,2022-01-25T17:07:29Z,2022-01-25T17:07:29Z,OWNER,"That's a much better place for them, I like this idea. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114147905,Move canned queries closer to the SQL input area, https://github.com/simonw/datasette/issues/1609#issuecomment-1020456608,https://api.github.com/repos/simonw/datasette/issues/1609,1020456608,IC_kwDOBm6k_c480u6g,9599,simonw,2022-01-24T19:20:09Z,2022-01-24T19:20:09Z,OWNER,Uvicorn have a release out now that would have fixed this issue if I hadn't shipped my own fix: https://github.com/encode/uvicorn/releases/tag/0.17.0.post1,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720,"Ensure ""pip install datasette"" still works with Python 3.6", https://github.com/simonw/datasette/issues/1605#issuecomment-1018766727,https://api.github.com/repos/simonw/datasette/issues/1605,1018766727,IC_kwDOBm6k_c48uSWH,9599,simonw,2022-01-21T18:41:21Z,2022-01-21T18:42:03Z,OWNER,"Yeah I think this all hinges on: - #1101 Also this comment about streaming full JSON arrays (not just newline-delimited) using [this trick](https://til.simonwillison.net/python/output-json-array-streaming): - https://github.com/simonw/datasette/issues/1356#issuecomment-1017016553 I'm about ready to figure these out, as with so much it's still a little bit blocked on the refactor stuff from: - #1518 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108671952,Scripted exports, https://github.com/simonw/datasette/issues/1609#issuecomment-1018104868,https://api.github.com/repos/simonw/datasette/issues/1609,1018104868,IC_kwDOBm6k_c48rwwk,9599,simonw,2022-01-21T02:24:13Z,2022-01-21T02:24:13Z,OWNER,Just shipped 0.60.1 with the fix - and tested that `pip install datasette` does indeed work correctly on Python 3.6.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720,"Ensure ""pip install datasette"" still works with Python 3.6", https://github.com/simonw/datasette/issues/1609#issuecomment-1018094767,https://api.github.com/repos/simonw/datasette/issues/1609,1018094767,IC_kwDOBm6k_c48ruSv,9599,simonw,2022-01-21T02:04:14Z,2022-01-21T02:04:14Z,OWNER,So I don't need to release 0.60.1 AND 0.60.2 after all - I can just release 0.60.1 with a bug fix that it no longer breaks installation for Python 3.6.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720,"Ensure ""pip install datasette"" still works with Python 3.6", https://github.com/simonw/datasette/issues/1609#issuecomment-1018092984,https://api.github.com/repos/simonw/datasette/issues/1609,1018092984,IC_kwDOBm6k_c48rt24,9599,simonw,2022-01-21T02:00:38Z,2022-01-21T02:00:38Z,OWNER,"Out of curiosity, I installed this latest `setup.py` file using both Python 3.6 and Python 3.10, ran `pip freeze` on both of them and created a Gist to compare the difference. The result is here: https://gist.github.com/simonw/2e7d5b1beba675ef9a5bcd310cadc372/revisions From that, it looks like the Python packages in my dependencies which have released new versions that don't work with Python 3.6 are: - https://pypi.org/project/janus/#history - https://pypi.org/project/Pint/#history - https://pypi.org/project/platformdirs/#history - https://pypi.org/project/uvicorn/#history (already discussed) Sure enough, for the first three of those browsing through their recent versions on PyPI confirms that they switched from e.g. ""Requires: Python >=3.6"" on https://pypi.org/project/janus/0.7.0/ to ""Requires: Python >=3.7"" on https://pypi.org/project/janus/1.0.0/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720,"Ensure ""pip install datasette"" still works with Python 3.6", https://github.com/simonw/datasette/issues/1609#issuecomment-1018091322,https://api.github.com/repos/simonw/datasette/issues/1609,1018091322,IC_kwDOBm6k_c48rtc6,9599,simonw,2022-01-21T01:56:42Z,2022-01-21T01:56:42Z,OWNER,"OK, the environment markers approach seems to work!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720,"Ensure ""pip install datasette"" still works with Python 3.6", https://github.com/simonw/datasette/issues/1609#issuecomment-1018086697,https://api.github.com/repos/simonw/datasette/issues/1609,1018086697,IC_kwDOBm6k_c48rsUp,9599,simonw,2022-01-21T01:46:43Z,2022-01-21T01:46:43Z,OWNER,https://github.com/simonw/datasette/runs/4890775227?check_suite_focus=true - the tests passed on Python 3.6 for this commit with the pinned dependencies: https://github.com/simonw/datasette/commit/41060e7e7cb838328c879de6a98ae794dc1886d0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720,"Ensure ""pip install datasette"" still works with Python 3.6", https://github.com/simonw/datasette/issues/1609#issuecomment-1018086273,https://api.github.com/repos/simonw/datasette/issues/1609,1018086273,IC_kwDOBm6k_c48rsOB,9599,simonw,2022-01-21T01:45:46Z,2022-01-21T01:45:46Z,OWNER,"This whole thing reminds me of my ongoing internal debate about version pinning: should the Datasette package released to PyPI pin to the exact versions of the dependencies that are known to work, or should it allow a range of dependencies so users can pick other versions of the dependencies to use in their environment? As I understand it, the general rule is to use exact pinning for applications but use ranges for libraries. Datasette is almost entirely an application... but it can also be used as a library - and in fact I'm hoping to encourage that usage more in the future, see: - #1398 I'd also like to release a packaged version of Datasette that doesn't require Uvicorn, for running on AWS Lambda and other function-as-a-service platforms. Those platforms have their own HTTP layer and hence don't need the Uvicorn dependency. Maybe the answer is to have a `datasette-core` package which provides the core of Datasette with unpinned dependencies and no Uvicorn, and then have the existing `datasette` package provide the Datasette CLI tool with Uvicorn and pinned dependencies?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720,"Ensure ""pip install datasette"" still works with Python 3.6", https://github.com/simonw/datasette/issues/1609#issuecomment-1018082792,https://api.github.com/repos/simonw/datasette/issues/1609,1018082792,IC_kwDOBm6k_c48rrXo,9599,simonw,2022-01-21T01:37:11Z,2022-01-21T01:37:11Z,OWNER,"Another option from https://twitter.com/samuel_hames/status/1484327636860293121 - environment markers, described in https://www.python.org/dev/peps/pep-0508/#environment-markers Found some examples of those in use using GitHub code search: https://cs.github.com/?scopeName=All+repos&scope=&q=%22%3Bpython_version%22+path%3Asetup.py - in particular https://github.com/xmendez/wfuzz/blob/1b695ee9a87d66a7d7bf6cae70d60a33fae51541/setup.py#L31-L38 ```python install_requires = [ 'pycurl', 'pyparsing<2.4.2;python_version<=""3.4""', 'pyparsing>=2.4*;python_version>=""3.5""', 'six', 'configparser;python_version<""3.5""', 'chardet', ] ``` So maybe I can ship 0.60.1 with loose dependencies _except_ for the `uvicorn` one on Python 3.6, using an environment marker. Here's my `setup.py` at the moment: https://github.com/simonw/datasette/blob/ffca55dfd7cc9b53522c2e5a2fa1ff67c9beadf2/setup.py#L44-L61 One other problem: there might be packages in that list right now which don't specify their 3.6 Python version but which will, at some point in the future, release a new version that doesn't work with 3.6 (like Uvicorn did) - in which case Python 3.6 installs would break in the future. I think what I'll do then is ship the `0.60.1` Python 3.6 version with strict upper limits on each version which are the current, tested-with-Datasette-on-Python3.6 versions.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720,"Ensure ""pip install datasette"" still works with Python 3.6", https://github.com/simonw/datasette/issues/1609#issuecomment-1018077009,https://api.github.com/repos/simonw/datasette/issues/1609,1018077009,IC_kwDOBm6k_c48rp9R,9599,simonw,2022-01-21T01:24:15Z,2022-01-21T01:24:43Z,OWNER,"Problem: if I ship this, it will be the most recent release of Datasette - but unlike other previous releases it has exactly pinned versions of all of the dependencies. Which is bad for people who run `pip install datasette` but want to not be stuck to those exact library versions. So maybe I ship this as 0.60.1, then ship a 0.60.2 release directly afterwards which unpins the dependencies again and requires Python 3.7?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720,"Ensure ""pip install datasette"" still works with Python 3.6", https://github.com/simonw/datasette/issues/1609#issuecomment-1018075357,https://api.github.com/repos/simonw/datasette/issues/1609,1018075357,IC_kwDOBm6k_c48rpjd,9599,simonw,2022-01-21T01:20:56Z,2022-01-21T01:20:56Z,OWNER,"I used the combo of `pyenv` and `pipenv` to run tests and figure out what the most recent versions of each dependency were that worked on Python 3.6. I also clicked around in the latest releases on pages such as https://pypi.org/project/aiofiles ``` cd /tmp git clone git@github.com:simonw/datasette cd /tmp/datasette pipenv shell --python 3.6.10 pip install -e '.[test]' pytest ``` I also used `pip freeze | grep black` to see which version was installed, since packages with `python_requires=` in them would automatically install the highest compatible version.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720,"Ensure ""pip install datasette"" still works with Python 3.6", https://github.com/simonw/datasette/issues/1609#issuecomment-1018064620,https://api.github.com/repos/simonw/datasette/issues/1609,1018064620,IC_kwDOBm6k_c48rm7s,9599,simonw,2022-01-21T01:00:12Z,2022-01-21T01:00:12Z,OWNER,"I think there are two possible solutions then: 1. Convince Uvicorn to publish one last `0.16.1` version which includes that `python_requires=` line, such that there's a version of Uvicorn on PyPI that Python 3.6 can still install. 2. Release a `0.60.1` version of Datasette which pins that Uvicorn version, and hence can be installed. I've made the request for 1) in Uvicorn Gitter here: https://gitter.im/encode/community?at=61ea044a6d9ba23328d0fa28 I'm going to investigate option 2) myself now.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720,"Ensure ""pip install datasette"" still works with Python 3.6", https://github.com/simonw/datasette/issues/1609#issuecomment-1018063681,https://api.github.com/repos/simonw/datasette/issues/1609,1018063681,IC_kwDOBm6k_c48rmtB,9599,simonw,2022-01-21T00:58:25Z,2022-01-21T00:58:32Z,OWNER,"On Twitter: https://twitter.com/simonw/status/1484317711672877065 Here's the problem: Uvicorn only added `python_requires` to their `setup.py` a few days ago, which means the releases they have out on PyPI at the moment don't specify the Python version they need, which is why this mechanism doesn't work as expected: - https://github.com/encode/uvicorn/pull/1328","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720,"Ensure ""pip install datasette"" still works with Python 3.6", https://github.com/simonw/datasette/issues/1608#issuecomment-1018017637,https://api.github.com/repos/simonw/datasette/issues/1608,1018017637,IC_kwDOBm6k_c48rbdl,9599,simonw,2022-01-20T23:27:59Z,2022-01-20T23:27:59Z,OWNER,"Got a couple of TILs out of this: - [Promoting the stable version of the documentation using rel=canonical](https://til.simonwillison.net/readthedocs/documentation-seo-canonical) - [Linking from /latest/ to /stable/ on Read The Docs](https://til.simonwillison.net/readthedocs/link-from-latest-to-stable)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154,Documentation should clarify /stable/ vs /latest/, https://github.com/simonw/datasette/issues/1608#issuecomment-1017998993,https://api.github.com/repos/simonw/datasette/issues/1608,1017998993,IC_kwDOBm6k_c48rW6R,9599,simonw,2022-01-20T22:56:00Z,2022-01-20T22:56:00Z,OWNER,"> https://sphinx-version-warning.readthedocs.io/ looks like it can show a banner for ""You are looking at v0.36 but you should be looking at 0.40"" but doesn't hand the case I need here which is ""you are looking at /latest/ but you should be looking at /stable/"". Correction! That tool DOES support that, as can be seen in their example configuration for their own documentation: https://github.com/humitos/sphinx-version-warning/blob/a82156c2ea08e5feab406514d0ccd9d48a345f48/docs/conf.py#L32-L38 ```python versionwarning_messages = { 'latest': 'This is a custom message only for version ""latest"" of this documentation.', } versionwarning_admonition_type = 'tip' versionwarning_banner_title = 'Tip' versionwarning_body_selector = 'div[itemprop=""articleBody""]' ```","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154,Documentation should clarify /stable/ vs /latest/, https://github.com/simonw/datasette/issues/1608#issuecomment-1017994925,https://api.github.com/repos/simonw/datasette/issues/1608,1017994925,IC_kwDOBm6k_c48rV6t,9599,simonw,2022-01-20T22:48:43Z,2022-01-20T22:54:02Z,OWNER,"https://sphinx-version-warning.readthedocs.io/ looks like it can show a banner for ""You are looking at v0.36 but you should be looking at 0.40"" but doesn't hand the case I need here which is ""you are looking at /latest/ but you should be looking at /stable/"". Just shipped my fix here: https://docs.datasette.io/en/latest/ ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154,Documentation should clarify /stable/ vs /latest/, https://github.com/simonw/datasette/issues/1608#issuecomment-1017988556,https://api.github.com/repos/simonw/datasette/issues/1608,1017988556,IC_kwDOBm6k_c48rUXM,9599,simonw,2022-01-20T22:37:51Z,2022-01-20T22:37:51Z,OWNER,"Here's a jQuery recipe that seems to do the right thing: ```javascript jQuery(function ($) { // If this is a /latest/ doc page, show banner linking to /stable/ if (!/\/latest\//.test(location.pathname)) { return; } var stableUrl = location.pathname.replace(""/latest/"", ""/stable/""); // Check it's not a 404 fetch(stableUrl, { method: ""HEAD"" }).then((response) => { if (response.status == 200) { var warning = $( `

Note

This documentation covers the development version of Datasette.

See this page for the current stable release.

` ); warning.find(""a"").attr(""href"", stableUrl); var body = $(""div.body""); if (!body.length) { body = $(""div.document""); } body.prepend(warning); } }); }); ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154,Documentation should clarify /stable/ vs /latest/, https://github.com/simonw/datasette/issues/1608#issuecomment-1017975322,https://api.github.com/repos/simonw/datasette/issues/1608,1017975322,IC_kwDOBm6k_c48rRIa,9599,simonw,2022-01-20T22:17:01Z,2022-01-20T22:27:07Z,OWNER,"Turns out that banner is something that ReadTheDocs implemented - I found it using GitHub code search, it's produced by this piece of JavaScript: https://github.com/readthedocs/readthedocs.org/blob/0852d7c10d725d954d3e9a93513171baa1116d9f/readthedocs/core/static-src/core/js/doc-embed/version-compare.js#L13-L21 ```javascript function init(data) { var rtd = rtddata.get(); /// Out of date message if (data.is_highest) { return; } var currentURL = window.location.pathname.replace(rtd['version'], data.slug); var warning = $( '
' + '

Note

' + '

' + 'You are not reading the most recent version of this documentation. ' + ' is the latest version available.' + '

' + '
'); warning .find('a') .attr('href', currentURL) .text(data.slug); var body = $(""div.body""); if (!body.length) { body = $(""div.document""); } body.prepend(warning); } ``` And here's where that module is called from the rest of their code: https://github.com/readthedocs/readthedocs.org/blob/bc3e147770e5740314a8e8c33fec5d111c850498/readthedocs/core/static-src/core/js/doc-embed/footer.js#L66-L86","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154,Documentation should clarify /stable/ vs /latest/, https://github.com/simonw/datasette/issues/1608#issuecomment-1017981599,https://api.github.com/repos/simonw/datasette/issues/1608,1017981599,IC_kwDOBm6k_c48rSqf,9599,simonw,2022-01-20T22:26:32Z,2022-01-20T22:26:32Z,OWNER,I'm tempted to imitate their JavaScript but check for `/latest/` in the URL and use that to append a similar message warning about this being the documentation for the in-development version.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154,Documentation should clarify /stable/ vs /latest/, https://github.com/simonw/datasette/issues/1608#issuecomment-1017971905,https://api.github.com/repos/simonw/datasette/issues/1608,1017971905,IC_kwDOBm6k_c48rQTB,9599,simonw,2022-01-20T22:11:40Z,2022-01-20T22:11:40Z,OWNER,"Huh, I had forgotten I already have a banner on older versions: ![D1A65C68-9A37-4FA2-80C4-534739A9D292](https://user-images.githubusercontent.com/9599/150430410-1e22e23f-ed27-4271-89ff-63467eb5f466.jpeg) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154,Documentation should clarify /stable/ vs /latest/, https://github.com/simonw/datasette/issues/1608#issuecomment-1017970132,https://api.github.com/repos/simonw/datasette/issues/1608,1017970132,IC_kwDOBm6k_c48rP3U,9599,simonw,2022-01-20T22:08:55Z,2022-01-20T22:08:55Z,OWNER,"Might want to consider SEO here too - I want people from search engines to land on `/stable/`, I wonder if I should noindex or `rel=canonical` the other documentation versions? Not sure what best practices for that is.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154,Documentation should clarify /stable/ vs /latest/, https://github.com/simonw/datasette/issues/1608#issuecomment-1017969452,https://api.github.com/repos/simonw/datasette/issues/1608,1017969452,IC_kwDOBm6k_c48rPss,9599,simonw,2022-01-20T22:07:55Z,2022-01-20T22:07:55Z,OWNER,"I think I want a banner at the top of the page making it obvious which version the documentation is talking about. This can be pretty low key for the current stable release, but should be visually more prominent for the `/latest/` branch and for older releases.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154,Documentation should clarify /stable/ vs /latest/, https://github.com/simonw/datasette/issues/1603#issuecomment-1017842366,https://api.github.com/repos/simonw/datasette/issues/1603,1017842366,IC_kwDOBm6k_c48qwq-,9599,simonw,2022-01-20T19:19:54Z,2022-01-20T19:19:54Z,OWNER,Wrote up a TIL: https://til.simonwillison.net/pixelmator/pixel-editing-favicon,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017808898,https://api.github.com/repos/simonw/datasette/issues/1603,1017808898,IC_kwDOBm6k_c48qogC,9599,simonw,2022-01-20T18:42:35Z,2022-01-20T18:42:35Z,OWNER,"Resized it down to 208 bytes with https://squoosh.app ![favicon](https://user-images.githubusercontent.com/9599/150401709-11a50492-a8c8-4eee-848c-c813f8cd3e4e.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017806497,https://api.github.com/repos/simonw/datasette/issues/1603,1017806497,IC_kwDOBm6k_c48qn6h,9599,simonw,2022-01-20T18:39:27Z,2022-01-20T18:39:57Z,OWNER,"Here's a comparison between my hand-edited version and the one I have now: ![CleanShot 2022-01-20 at 10 38 00@2x](https://user-images.githubusercontent.com/9599/150401244-7e78ee93-1973-4c95-8f91-01e51e0d5366.png) The new 32x32 image: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017800506,https://api.github.com/repos/simonw/datasette/issues/1603,1017800506,IC_kwDOBm6k_c48qmc6,9599,simonw,2022-01-20T18:31:18Z,2022-01-20T18:31:18Z,OWNER,"One last go at tidying this up. I decided to do a 32x32 pixel version in Pixelmator, using this trick to access a pixel brush: https://osxdaily.com/2016/11/17/enable-pixel-brush-pixelmator-mac/ Frustrating how the white boxes are all exactly four pixels high and ALMOST all four pixels wide, but one of them has to be three pixels wide to fit the space.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017149828,https://api.github.com/repos/simonw/datasette/issues/1603,1017149828,IC_kwDOBm6k_c48oHmE,9599,simonw,2022-01-20T06:07:31Z,2022-01-20T06:07:31Z,OWNER,"Now live on https://latest.datastte.io/ - I'm happy enough with this for the moment: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017143817,https://api.github.com/repos/simonw/datasette/issues/1603,1017143817,IC_kwDOBm6k_c48oGIJ,9599,simonw,2022-01-20T05:54:09Z,2022-01-20T05:54:09Z,OWNER,"Oops, I pushed the one I liked least out of the two options! Fixing now.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017143437,https://api.github.com/repos/simonw/datasette/issues/1603,1017143437,IC_kwDOBm6k_c48oGCN,9599,simonw,2022-01-20T05:53:17Z,2022-01-20T05:53:17Z,OWNER,New favicon now live on https://latest.datasette.io/ - see also https://latest.datasette.io/favicon.ico,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017142395,https://api.github.com/repos/simonw/datasette/issues/1603,1017142395,IC_kwDOBm6k_c48oFx7,9599,simonw,2022-01-20T05:50:55Z,2022-01-20T05:50:55Z,OWNER,The new `/favicon.ico` view: https://github.com/simonw/datasette/blob/0467723ee57c2cbc0f02daa47cef632dd4651df0/datasette/app.py#L182-L192,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017139321,https://api.github.com/repos/simonw/datasette/issues/1603,1017139321,IC_kwDOBm6k_c48oFB5,9599,simonw,2022-01-20T05:43:07Z,2022-01-20T05:46:42Z,OWNER,"My attempt with one less grid line looked awful when shrunk down to the browser size: File here: ![favicon](https://user-images.githubusercontent.com/9599/150280463-817a8065-8393-4c10-aec4-5a627cf46fae.png) I'm going with the first attempt for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017137117,https://api.github.com/repos/simonw/datasette/issues/1603,1017137117,IC_kwDOBm6k_c48oEfd,9599,simonw,2022-01-20T05:37:05Z,2022-01-20T05:37:34Z,OWNER,"I'm not crazy about the look - I think this version of the grid may have too many lines for this particular display size. I'm going to try reducing the number of lines in Figma to see if I like that better. https://www.figma.com/file/LKjceTFNtKm6wCbScDqm1Y/Datasette-Logo","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017136897,https://api.github.com/repos/simonw/datasette/issues/1603,1017136897,IC_kwDOBm6k_c48oEcB,9599,simonw,2022-01-20T05:36:29Z,2022-01-20T05:36:29Z,OWNER,"Here's what it looks like in Firefox, Chrome and Safari: ![CleanShot 2022-01-19 at 21 35 06@2x](https://user-images.githubusercontent.com/9599/150279832-6c233115-6540-4746-8cd1-dce25321ebbf.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017133357,https://api.github.com/repos/simonw/datasette/issues/1603,1017133357,IC_kwDOBm6k_c48oDkt,9599,simonw,2022-01-20T05:27:34Z,2022-01-20T05:27:34Z,OWNER,"I'm going to drop it in `datasette/static/favicon.png` and rewrite the `async def favicon()` function to return it, with caching headers.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017132927,https://api.github.com/repos/simonw/datasette/issues/1603,1017132927,IC_kwDOBm6k_c48oDd_,9599,simonw,2022-01-20T05:26:29Z,2022-01-20T05:26:29Z,OWNER,"Here's the 128x128 one - 11kb, I resized it using Preview: ![icon](https://user-images.githubusercontent.com/9599/150278798-48cc2da2-3640-414d-a440-20c9d93c09f4.png) Now running it through [Squoosh](https://squoosh.app/editor) using OxiPNG effort=3 colours=24 - brought it down to 1.36KB. ![favicon](https://user-images.githubusercontent.com/9599/150278906-b312940e-0cdb-43ef-a325-45d2b1c240de.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017131801,https://api.github.com/repos/simonw/datasette/issues/1603,1017131801,IC_kwDOBm6k_c48oDMZ,9599,simonw,2022-01-20T05:23:57Z,2022-01-20T05:23:57Z,OWNER,"https://adamj.eu/tech/2022/01/18/how-to-add-a-favicon-to-your-django-site/ suggests 64x64, I'm going with 128x128 just in case anyone invents a retina-retina screen.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017131209,https://api.github.com/repos/simonw/datasette/issues/1603,1017131209,IC_kwDOBm6k_c48oDDJ,9599,simonw,2022-01-20T05:22:40Z,2022-01-20T05:22:40Z,OWNER,This one is 101KB and 1536*1536 which is a bit big! https://github.com/simonw/datasette-app/blob/main/build/icon.png,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1600#issuecomment-1017124310,https://api.github.com/repos/simonw/datasette/issues/1600,1017124310,IC_kwDOBm6k_c48oBXW,9599,simonw,2022-01-20T05:06:09Z,2022-01-20T05:06:09Z,OWNER,Fixed: https://docs.datasette.io/en/latest/plugins.html#seeing-what-plugins-are-installed,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1104691662,plugins --all example should use cog, https://github.com/simonw/datasette/issues/1601#issuecomment-1017113831,https://api.github.com/repos/simonw/datasette/issues/1601,1017113831,IC_kwDOBm6k_c48n-zn,9599,simonw,2022-01-20T04:38:14Z,2022-01-20T04:38:14Z,OWNER,"I don't have solid tests in place for exercising SpatiaLite, but this change feels safe enough that I'm not going to add tests for it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1105916061,Add KNN and data_licenses to hidden tables list, https://github.com/simonw/datasette/issues/1577#issuecomment-1017112130,https://api.github.com/repos/simonw/datasette/issues/1577,1017112130,IC_kwDOBm6k_c48n-ZC,9599,simonw,2022-01-20T04:33:57Z,2022-01-20T04:33:57Z,OWNER,"Dropped support for Python 3.6: updated `setup.py`, changed all references in the documentation to say 3.7 is the minimum required version, got rid of the GitHub Actions tests against 3.6.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087913724,Drop support for Python 3.6, https://github.com/simonw/datasette/issues/1577#issuecomment-1017109401,https://api.github.com/repos/simonw/datasette/issues/1577,1017109401,IC_kwDOBm6k_c48n9uZ,9599,simonw,2022-01-20T04:27:34Z,2022-01-20T04:27:34Z,OWNER,"OK, now that Uvicorn has dropped 3.6 support - see #1606 - I think this decision is easy to make. I'm dropping 3.6.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087913724,Drop support for Python 3.6, https://github.com/simonw/datasette/issues/1606#issuecomment-1017109194,https://api.github.com/repos/simonw/datasette/issues/1606,1017109194,IC_kwDOBm6k_c48n9rK,9599,simonw,2022-01-20T04:27:07Z,2022-01-20T04:27:07Z,OWNER,"Relevant: - https://github.com/simonw/datasette/issues/1577","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108846067,Tests failing against Python 3.6, https://github.com/simonw/datasette/issues/1606#issuecomment-1017108960,https://api.github.com/repos/simonw/datasette/issues/1606,1017108960,IC_kwDOBm6k_c48n9ng,9599,simonw,2022-01-20T04:26:36Z,2022-01-20T04:26:36Z,OWNER,"https://pypistats.org/packages/datasette shows Python 3.6 is 0.24% of Datasette downloads. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108846067,Tests failing against Python 3.6, https://github.com/simonw/datasette/issues/1606#issuecomment-1017108291,https://api.github.com/repos/simonw/datasette/issues/1606,1017108291,IC_kwDOBm6k_c48n9dD,9599,simonw,2022-01-20T04:24:54Z,2022-01-20T04:24:54Z,OWNER,"https://github.com/simonw/latest-datasette-with-all-plugins/commit/1e12ffe70be791e3281b41810e837515314c1317 shows that 5 days ago Datasette upgraded from Uvicorn 0.16 to 0.17 Sure enough, in the changelog for 0.17: https://github.com/encode/uvicorn/blob/0.17.0/CHANGELOG.md > - Drop Python 3.6 support ([#1261](https://github.com/encode/uvicorn/pull/1261)) 06/01/22","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108846067,Tests failing against Python 3.6, https://github.com/simonw/datasette/issues/1601#issuecomment-1016639294,https://api.github.com/repos/simonw/datasette/issues/1601,1016639294,IC_kwDOBm6k_c48mK8-,9599,simonw,2022-01-19T16:26:44Z,2022-01-20T03:57:17Z,OWNER,"I need to add `sqlite_stat1` to the hidden tables too, see: - https://github.com/simonw/datasette/issues/1587 - https://github.com/simonw/sqlite-utils/issues/366","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1105916061,Add KNN and data_licenses to hidden tables list, https://github.com/simonw/datasette/issues/1601#issuecomment-1017095821,https://api.github.com/repos/simonw/datasette/issues/1601,1017095821,IC_kwDOBm6k_c48n6aN,9599,simonw,2022-01-20T03:56:14Z,2022-01-20T03:56:28Z,OWNER,"Oh interesting, I hadn't heard about KNN2. It looks like it was added to SpatiaLite on 21st June 2021 in https://www.gaia-gis.it/fossil/libspatialite/info/03786a62cdb4ab17 but the most recent release of SpatiaLite is 5.0.1 from 7th February 2021 - so it's not yet in a release.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1105916061,Add KNN and data_licenses to hidden tables list, https://github.com/simonw/datasette/issues/1356#issuecomment-1017016553,https://api.github.com/repos/simonw/datasette/issues/1356,1017016553,IC_kwDOBm6k_c48nnDp,9599,simonw,2022-01-20T01:06:37Z,2022-01-20T01:06:37Z,OWNER,"> A problem with this is that if you're using `--query` you likely want ALL of the results - at the moment the only Datasette output type that can stream everything is `.csv` and plugin formats can't handle full streams, see #1062 and #1177. I figured out a neat pattern for streaming JSON arrays in this TIL: https://til.simonwillison.net/python/output-json-array-streaming","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",910092577,"Research: syntactic sugar for using --get with SQL queries, maybe ""datasette query""", https://github.com/simonw/datasette/issues/1605#issuecomment-1016977725,https://api.github.com/repos/simonw/datasette/issues/1605,1016977725,IC_kwDOBm6k_c48ndk9,9599,simonw,2022-01-19T23:55:08Z,2022-01-19T23:55:08Z,OWNER,"Oh that's interesting. I was thinking about this from a slightly different angle recently - pondering what a static site generator built on top of Datasette might look like. Just a sketch at the moment, but I was imagining a YAML configuration file with a SQL query that returns a list of paths - then a tool that runs that query and uses the equivalent of `datasette --get` to create a static copy of each of those paths. I think these two ideas can probably be merged. I'd love to know more about how you are solving this right now!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108671952,Scripted exports, https://github.com/simonw/datasette/issues/1601#issuecomment-1016637722,https://api.github.com/repos/simonw/datasette/issues/1601,1016637722,IC_kwDOBm6k_c48mKka,9599,simonw,2022-01-19T16:25:07Z,2022-01-19T16:25:07Z,OWNER,Good idea - though I'm nervous about accidentally hiding a `data_licenses` table outside of the SpatiaLite case. I'll only hide that one if SpatiaLite is loaded.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1105916061,Add KNN and data_licenses to hidden tables list, https://github.com/simonw/datasette/issues/1604#issuecomment-1016636561,https://api.github.com/repos/simonw/datasette/issues/1604,1016636561,IC_kwDOBm6k_c48mKSR,9599,simonw,2022-01-19T16:23:54Z,2022-01-19T16:23:54Z,OWNER,"Potential design: datasette publish cloudrun data.db \ --service my-service \ --domain demo.datasette.io I think I'm OK with calling this `--domain` even when it is being used with a subdomain. This will require `datasette.io` to already have been verified with Google. Not sure how best to handle the DNS part - maybe print out instructions for the necessary CNAME?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108300685,Option to assign a domain/subdomain using `datasette publish cloudrun`, https://github.com/simonw/datasette/issues/1603#issuecomment-1016589519,https://api.github.com/repos/simonw/datasette/issues/1603,1016589519,IC_kwDOBm6k_c48l-zP,9599,simonw,2022-01-19T15:36:38Z,2022-01-19T15:36:38Z,OWNER,Also people can use a custom base template and link to a custom favicon if they want to.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1016589140,https://api.github.com/repos/simonw/datasette/issues/1603,1016589140,IC_kwDOBm6k_c48l-tU,9599,simonw,2022-01-19T15:36:16Z,2022-01-19T15:36:16Z,OWNER,For the moment then I will hard-code a new favicon and leave it to ASGI plugins if people want to define their own.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1016588326,https://api.github.com/repos/simonw/datasette/issues/1603,1016588326,IC_kwDOBm6k_c48l-gm,9599,simonw,2022-01-19T15:35:33Z,2022-01-19T15:35:33Z,OWNER,"I don't think abusing the template loader mechanism for this will work: Jinja provides an API for loading text templates, but I don't think it can be sensibly abused to open binary image files instead. Loaded code is here: https://github.com/pallets/jinja/blob/main/src/jinja2/loaders.py","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1016579661,https://api.github.com/repos/simonw/datasette/issues/1603,1016579661,IC_kwDOBm6k_c48l8ZN,9599,simonw,2022-01-19T15:27:05Z,2022-01-19T15:27:05Z,OWNER,I'd like this to be customizable. I'm going to load it from the template loading system such that a custom favicon for a specific instance can be dropped in using a file in `templates/`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1591#issuecomment-1013669543,https://api.github.com/repos/simonw/datasette/issues/1591,1013669543,IC_kwDOBm6k_c48a16n,9599,simonw,2022-01-15T11:56:59Z,2022-01-15T11:56:59Z,OWNER,"There's actually already a way to move regular Datasette `--setting` options to a `settings.json` file thanks to configuration directory mode: https://docs.datasette.io/en/stable/settings.html Maybe extending that mechanism to handle plugins would be a neat path forward here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1100015398,Maybe let plugins define custom serve options?, https://github.com/simonw/datasette/issues/1591#issuecomment-1013668967,https://api.github.com/repos/simonw/datasette/issues/1591,1013668967,IC_kwDOBm6k_c48a1xn,9599,simonw,2022-01-15T11:53:21Z,2022-01-15T11:53:21Z,OWNER,"The `datasette publish --plugin-secret name setting value` option already implements something a bit like this. https://docs.datasette.io/en/stable/plugins.html#secret-configuration-values It's a bit of a messy hack to compensate for metadata being visible. Maybe I could replace that mechanism with the proposed plugin configuration rethink from this issue. I still like the debug benefits of making plugin settings public - perhaps add a rule that if a plugin setting has a `secret:` prefix it gets redacted on a new `/-/plugin-settings` page.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1100015398,Maybe let plugins define custom serve options?, https://github.com/simonw/datasette/issues/1566#issuecomment-1012680228,https://api.github.com/repos/simonw/datasette/issues/1566,1012680228,IC_kwDOBm6k_c48XEYk,9599,simonw,2022-01-14T01:59:54Z,2022-01-14T01:59:54Z,OWNER,This is now shipped!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083669410,Release Datasette 0.60, https://github.com/simonw/datasette/issues/1595#issuecomment-1012664607,https://api.github.com/repos/simonw/datasette/issues/1595,1012664607,IC_kwDOBm6k_c48XAkf,9599,simonw,2022-01-14T01:22:58Z,2022-01-14T01:22:58Z,OWNER,"- Upgraded Pluggy dependency to 1.0. #1575 - Now using [Plausible](https://plausible.io/) analytics for the Datasette documentation. - The `db.execute_write()` internals method now defaults to blocking until the write operation has completed. Previously it defaulted to queuing the write and then continuing to run code while the write was in the queue. #1579 - `explain query plan` is now allowed with varying amounts of white space in the query. #1588 - New CLI reference page showing the output of `--help` for each of the `datasette` sub-commands. This lead to several small improvements to the help copy. #1594 - Fixed bug where columns with a underscore prefix could result in unnecessary hidden form fields. #1527","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102484126,Release notes for 0.60, https://github.com/simonw/datasette/issues/1595#issuecomment-1012626410,https://api.github.com/repos/simonw/datasette/issues/1595,1012626410,IC_kwDOBm6k_c48W3Pq,9599,simonw,2022-01-14T00:00:56Z,2022-01-14T01:17:47Z,OWNER,Commits since 0.60a1: https://github.com/simonw/datasette/compare/0.60a1...3664ddd40,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102484126,Release notes for 0.60, https://github.com/simonw/datasette/issues/1590#issuecomment-1012661522,https://api.github.com/repos/simonw/datasette/issues/1590,1012661522,IC_kwDOBm6k_c48W_0S,9599,simonw,2022-01-14T01:16:08Z,2022-01-14T01:16:34Z,OWNER,"OK, I'm going to recommend a workaround for this instead. Here's `asgi.py` updated to strip the prefix before passing the request on to Datasette: ```python import pathlib from asgi_cors import asgi_cors from channels.routing import URLRouter from django.urls import re_path from datasette.app import Datasette def rewrite_path(app, prefix_to_strip): async def rewrite_path_app(scope, receive, send): if ( scope[""type""] == ""http"" and ""path"" in scope and scope[""path""].startswith(prefix_to_strip) ): scope[""path""] = scope[""path""][len(prefix_to_strip) :] if ""raw_path"" in scope: scope[""raw_path""] = scope[""raw_path""][len(prefix_to_strip) :] await app(scope, receive, send) return rewrite_path_app datasette_ = Datasette( files=[""fixtures.db""], settings={""base_url"": ""/datasettes/"", ""plugins"": {}}, ) application = URLRouter( [ re_path( r""^datasettes/.*"", asgi_cors(rewrite_path(datasette_.app(), ""/datasettes""), allow_all=True), ), ] ) ``` This works on my laptop - please re-open the ticket if it doesn't work for you!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099723916,Table+query JSON and CSV links broken when using `base_url` setting, https://github.com/simonw/datasette/issues/1590#issuecomment-1012656790,https://api.github.com/repos/simonw/datasette/issues/1590,1012656790,IC_kwDOBm6k_c48W-qW,9599,simonw,2022-01-14T01:05:34Z,2022-01-14T01:05:34Z,OWNER,"I think this prefixed string mechanism is supposed to prevent the `base_url` prefix from being applied twice: https://github.com/simonw/datasette/blob/3664ddd400062123e99500d28b160c7944408c1a/datasette/url_builder.py#L9-L16 But with a bit of extra logging all of the inputs to that are NOT prefixed strings: ``` Urls.path called with: /datasettes/fixtures/compound_three_primary_keys?_sort=content (PrefixedUrlString = False) returning /datasettes/datasettes/fixtures/compound_three_primary_keys?_sort=content ``` So it looks like `urls.path(...)` is indeed the code responsible for doubling up that `/datasettes/` prefix.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099723916,Table+query JSON and CSV links broken when using `base_url` setting, https://github.com/simonw/datasette/issues/1590#issuecomment-1012653966,https://api.github.com/repos/simonw/datasette/issues/1590,1012653966,IC_kwDOBm6k_c48W9-O,9599,simonw,2022-01-14T00:59:07Z,2022-01-14T00:59:07Z,OWNER,"Since this is a special case bug for when using Datasette as a library I wonder if a good fix here would be to support something like this: ```python application = URLRouter([ re_path(r""^datasettes/.*"", asgi_cors(datasette_.app(remove_path_prefix=""datasettes/""), allow_all=True)), ]) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099723916,Table+query JSON and CSV links broken when using `base_url` setting, https://github.com/simonw/datasette/issues/1527#issuecomment-1012653109,https://api.github.com/repos/simonw/datasette/issues/1527,1012653109,IC_kwDOBm6k_c48W9w1,9599,simonw,2022-01-14T00:57:08Z,2022-01-14T00:57:08Z,OWNER,Bug is fixed on https://latest.datasette.io/fixtures/facetable?_sort=pk&_city_id__gt=1,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059555791,Columns starting with an underscore behave poorly in filters, https://github.com/simonw/datasette/issues/1598#issuecomment-1012643882,https://api.github.com/repos/simonw/datasette/issues/1598,1012643882,IC_kwDOBm6k_c48W7gq,9599,simonw,2022-01-14T00:34:49Z,2022-01-14T00:34:49Z,OWNER,There are four places in the documentation that use `.. literalinclude::` at the moment which I can replace - I can actually just link directly to the new https://docs.datasette.io/en/latest/cli-reference.html page instead of embedding the help directly.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102637351,Replace update-docs-help.py script with cog, https://github.com/simonw/datasette/issues/1527#issuecomment-1012635696,https://api.github.com/repos/simonw/datasette/issues/1527,1012635696,IC_kwDOBm6k_c48W5gw,9599,simonw,2022-01-14T00:19:10Z,2022-01-14T00:20:36Z,OWNER,"Oh! This is because `_city_id` has a leading underscore (for testing purposes). I think I need to filter out any keys that contain `__` in that case. What happens to columns that contain a `__`? They shouldn't be reflected in the hidden arguments either - this code is really only supposed to catch things like `_where` and `_m2m_through=` and `_col` - basically most of the list on https://docs.datasette.io/en/stable/json_api.html#special-table-arguments","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059555791,Columns starting with an underscore behave poorly in filters, https://github.com/simonw/datasette/issues/1527#issuecomment-1012634659,https://api.github.com/repos/simonw/datasette/issues/1527,1012634659,IC_kwDOBm6k_c48W5Qj,9599,simonw,2022-01-14T00:17:00Z,2022-01-14T00:18:11Z,OWNER,"That's because that page has this unnecessary hidden form field: ```html ``` That field is added by this bit in the template: https://github.com/simonw/datasette/blob/515f8d38ebae203efc15ca79a8b42848276b35e5/datasette/templates/table.html#L119-L122 Which is populated here: https://github.com/simonw/datasette/blob/ace86566b28280091b3844cf5fbecd20158e9004/datasette/views/table.py#L813-L821 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059555791,Columns starting with an underscore behave poorly in filters, https://github.com/simonw/datasette/issues/1597#issuecomment-1012629825,https://api.github.com/repos/simonw/datasette/issues/1597,1012629825,IC_kwDOBm6k_c48W4FB,9599,simonw,2022-01-14T00:07:36Z,2022-01-14T00:07:36Z,OWNER,"Fixed: ``` % datasette --help Usage: datasette [OPTIONS] COMMAND [ARGS]... Datasette is an open source multi-tool for exploring and publishing data About Datasette: https://datasette.io/ Full documentation: https://docs.datasette.io/ Options: --version Show the version and exit. --help Show this message and exit. Commands: serve* Serve up specified SQLite database files with a web UI inspect Generate JSON summary of provided database files ... % datasette inspect --help Usage: datasette inspect [OPTIONS] [FILES]... Generate JSON summary of provided database files This can then be passed to ""datasette --inspect-file"" to speed up count operations against immutable database files. Options: --inspect-file TEXT --load-extension TEXT Path to a SQLite extension to load --help Show this message and exit. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102612922,"""datasette inspect"" has no help summary", https://github.com/simonw/datasette/issues/1595#issuecomment-1012626243,https://api.github.com/repos/simonw/datasette/issues/1595,1012626243,IC_kwDOBm6k_c48W3ND,9599,simonw,2022-01-14T00:00:33Z,2022-01-14T00:00:33Z,OWNER,"Copying those in here: - New plugin hook: [filters_from_request(request, database, table, datasette)](https://docs.datasette.io/en/latest/plugin_hooks.html#plugin-hook-filters-from-request), which runs on the table page and can be used to support new custom query string parameters that modify the SQL query. ([#473](https://github.com/simonw/datasette/issues/473)) - The number of unique values in a facet is now always displayed. Previously it was only displayed if the user specified `?_facet_size=max`. ([#1556](https://github.com/simonw/datasette/issues/1556)) - Fixed bug where `?_facet_array=tags&_facet=tags` would only display one of the two selected facets. ([#625](https://github.com/simonw/datasette/issues/625)) - Facets of type `date` or `array` can now be configured in `metadata.json`, see [Facets in metadata.json](https://docs.datasette.io/en/latest/facets.html#facets-metadata). Thanks, David Larlet. ([#1552](https://github.com/simonw/datasette/issues/1552)) - New `?_nosuggest=1` parameter for table views, which disables facet suggestion. ([#1557](https://github.com/simonw/datasette/issues/1557)) - Label columns detected for foreign keys are now case-insensitive, so `Name` or `TITLE` will be detected in the same way as `name` or `title`. ([#1544](https://github.com/simonw/datasette/issues/1544)) - The query string variables exposed by `request.args` will now include blank strings for arguments such as `foo` in `?foo=&bar=1` rather than ignoring those parameters entirely. ([#1551](https://github.com/simonw/datasette/issues/1551)) - Database write connections now execute the [prepare_connection(conn, database, datasette)](https://docs.datasette.io/en/latest/plugin_hooks.html#plugin-hook-prepare-connection) plugin hook. ([#1564](https://github.com/simonw/datasette/issues/1564)) - The `Datasette()` constructor no longer requires the `files=` argument, and is now documented at [Datasette class](https://docs.datasette.io/en/latest/internals.html#internals-datasette). ([#1563](https://github.com/simonw/datasette/issues/1563)) - The tracing feature now traces write queries, not just read queries. ([#1568](https://github.com/simonw/datasette/issues/1568)) - Added two methods for writing to the database: [await db.execute_write_script(sql, block=False)](https://docs.datasette.io/en/latest/internals.html#database-execute-write-script) and [await db.execute_write_many(sql, params_seq, block=False)](https://docs.datasette.io/en/latest/internals.html#database-execute-write-many). ([#1570](https://github.com/simonw/datasette/issues/1570)) - Made several performance improvements to the database schema introspection code that runs when Datasette first starts up. ([#1555](https://github.com/simonw/datasette/issues/1555)) - Fixed bug where writable canned queries could not be used with custom templates. ([#1547](https://github.com/simonw/datasette/issues/1547))","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102484126,Release notes for 0.60, https://github.com/simonw/datasette/issues/1590#issuecomment-1012583091,https://api.github.com/repos/simonw/datasette/issues/1590,1012583091,IC_kwDOBm6k_c48Wsqz,9599,simonw,2022-01-13T22:41:15Z,2022-01-13T22:41:15Z,OWNER,"Seeing as this area of the code has produced so many bugs in the past, I think part of the fix may be to write comprehensive documentation about how routing works for the internals documentation. Doing so might help me figure this bug out!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099723916,Table+query JSON and CSV links broken when using `base_url` setting, https://github.com/simonw/datasette/issues/1595#issuecomment-1012575013,https://api.github.com/repos/simonw/datasette/issues/1595,1012575013,IC_kwDOBm6k_c48Wqsl,9599,simonw,2022-01-13T22:29:22Z,2022-01-13T22:29:22Z,OWNER,"Most of these are already written for these two alpha releases: - https://github.com/simonw/datasette/releases/tag/0.60a0 - https://github.com/simonw/datasette/releases/tag/0.60a1","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102484126,Release notes for 0.60, https://github.com/simonw/datasette/issues/1588#issuecomment-1012554208,https://api.github.com/repos/simonw/datasette/issues/1588,1012554208,IC_kwDOBm6k_c48Wlng,9599,simonw,2022-01-13T22:07:15Z,2022-01-13T22:07:15Z,OWNER,This works now: https://latest.datasette.io/fixtures?sql=explain+query+plan++select+*+from+facetable,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097101917,`explain query plan select` is too strict about whitespace, https://github.com/simonw/datasette/issues/650#issuecomment-1012552760,https://api.github.com/repos/simonw/datasette/issues/650,1012552760,IC_kwDOBm6k_c48WlQ4,9599,simonw,2022-01-13T22:04:56Z,2022-01-13T22:04:56Z,OWNER,"Challenge: explain the difference between view as in SQL view, and view as in the code that serves `TableView` / `DatabaseView` etc.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",534629631,Add a glossary to the documentation, https://github.com/simonw/datasette/issues/1336#issuecomment-1012546924,https://api.github.com/repos/simonw/datasette/issues/1336,1012546924,IC_kwDOBm6k_c48Wj1s,9599,simonw,2022-01-13T21:55:58Z,2022-01-13T21:55:58Z,OWNER,"See also: - #1412","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",895686039,Document turning on WAL for live served SQLite databases, https://github.com/simonw/datasette/issues/1466#issuecomment-1012546223,https://api.github.com/repos/simonw/datasette/issues/1466,1012546223,IC_kwDOBm6k_c48Wjqv,9599,simonw,2022-01-13T21:54:51Z,2022-01-13T21:54:51Z,OWNER,"Going with this for the copy: > [Datasette Desktop](https://datasette.io/desktop) is a packaged Mac application which bundles Datasette together with Python and allows you to install and run Datasette directly on your laptop. This is the best option for local installation if you are not comfortable using the command line.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",991467558,Add Datasette Desktop to installation documentation, https://github.com/simonw/datasette/issues/1594#issuecomment-1012536257,https://api.github.com/repos/simonw/datasette/issues/1594,1012536257,IC_kwDOBm6k_c48WhPB,9599,simonw,2022-01-13T21:38:48Z,2022-01-13T21:38:48Z,OWNER,https://docs.datasette.io/en/latest/cli-reference.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102359726,"Add a CLI reference page to the docs, inspired by sqlite-utils", https://github.com/simonw/datasette/issues/1594#issuecomment-1012535024,https://api.github.com/repos/simonw/datasette/issues/1594,1012535024,IC_kwDOBm6k_c48Wg7w,9599,simonw,2022-01-13T21:36:53Z,2022-01-13T21:36:53Z,OWNER,I went with a simpler pattern that `sqlite-utils` because Datasette has a lot less commands: https://github.com/simonw/datasette/blob/4b23f01f3e668c8f2a2f1a294be49f49b4073969/docs/cli-reference.rst#L9-L35,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102359726,"Add a CLI reference page to the docs, inspired by sqlite-utils", https://github.com/simonw/datasette/issues/1594#issuecomment-1012508787,https://api.github.com/repos/simonw/datasette/issues/1594,1012508787,IC_kwDOBm6k_c48Wahz,9599,simonw,2022-01-13T20:56:14Z,2022-01-13T20:56:34Z,OWNER,"The implementation can be _almost_ exactly the same as this: https://github.com/simonw/sqlite-utils/blame/74586d3cb26fa3cc3412721985ecdc1864c2a31d/docs/cli-reference.rst#L11-L76 I need to do something extra to ensure the output of `datasette --help-settings` is shown too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102359726,"Add a CLI reference page to the docs, inspired by sqlite-utils", https://github.com/simonw/datasette/issues/1591#issuecomment-1012506595,https://api.github.com/repos/simonw/datasette/issues/1591,1012506595,IC_kwDOBm6k_c48WZ_j,9599,simonw,2022-01-13T20:52:56Z,2022-01-13T20:52:56Z,OWNER,"You can already run `datasette --help-settings` to see detailed help on available settings. Maybe `datasette --help-plugin-settings` could do the same thing for plugin settings? Or I could even have available plugin settings show up as a list at the bottom of the `datasette --help-settings` output - which currently looks like this: ``` % datasette --help-settings Settings: default_page_size Default page size for the table view (default=100) max_returned_rows Maximum rows that can be returned from a table or custom query (default=1000) num_sql_threads Number of threads in the thread pool for executing SQLite queries (default=3) sql_time_limit_ms Time limit for a SQL query in milliseconds (default=1000) default_facet_size Number of values to return for requested facets (default=30) facet_time_limit_ms Time limit for calculating a requested facet (default=200) facet_suggest_time_limit_ms Time limit for calculating a suggested facet (default=50) hash_urls Include DB file contents hash in URLs, for far- future caching (default=False) allow_facet Allow users to specify columns to facet using ?_facet= parameter (default=True) allow_download Allow users to download the original SQLite database files (default=True) suggest_facets Calculate and display suggested facets (default=True) default_cache_ttl Default HTTP cache TTL (used in Cache-Control: max-age= header) (default=5) default_cache_ttl_hashed Default HTTP cache TTL for hashed URL pages (default=31536000) cache_size_kb SQLite cache size in KB (0 == use SQLite default) (default=0) allow_csv_stream Allow .csv?_stream=1 to download all rows (ignoring max_returned_rows) (default=True) max_csv_mb Maximum size allowed for CSV export in MB - set 0 to disable this limit (default=100) truncate_cells_html Truncate cells longer than this in HTML table view - set 0 to disable (default=2048) force_https_urls Force URLs in API output to always use https:// protocol (default=False) template_debug Allow display of template debug information with ?_context=1 (default=False) trace_debug Allow display of SQL trace debug information with ?_trace=1 (default=False) base_url Datasette URLs should use this base path (default=/) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1100015398,Maybe let plugins define custom serve options?, https://github.com/simonw/datasette/issues/1591#issuecomment-1012505706,https://api.github.com/repos/simonw/datasette/issues/1591,1012505706,IC_kwDOBm6k_c48WZxq,9599,simonw,2022-01-13T20:51:30Z,2022-01-13T20:51:30Z,OWNER,"Another option: if I make plugin settings a higher level concept in Datasette than they are at the moment, I could allow them to be set either using `--options` OR using the existing `metadata.yml/json` mechanism. https://docs.datasette.io/en/stable/plugins.html#plugin-configuration I want to make changes to that anyway, because I'm increasingly uncomfortable with plugin settings ending up in the ""metadata"" mechanism.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1100015398,Maybe let plugins define custom serve options?, https://github.com/simonw/datasette/issues/1591#issuecomment-1012504251,https://api.github.com/repos/simonw/datasette/issues/1591,1012504251,IC_kwDOBm6k_c48WZa7,9599,simonw,2022-01-13T20:49:19Z,2022-01-13T20:49:59Z,OWNER,"I try to stick pretty closely to what [Click](https://click.palletsprojects.com/en/8.0.x/) supports, and Click likes you to define options explicitly so that it can display them in the output of `--help`. But... that makes me think that actually showing these options in `--help` is likely a better idea. My `--plugin-setting` concept would have help that looks something like this: ``` % datasette serve --help ... --plugin-setting ... Setting for a specified plugin. ``` That's not great help! The alternative would be to allow plugins to register their extra options with the command - which would mean the help output could look like this instead: ``` % datasette serve --help ... --tiddlywiki-db Name of database to use for datasette-tiddlywiki ``` This feels like a good argument to me for plugins to explicitly register their settings. I'm not sure if I should enforce the `name-of-plugin-` prefix, or if I should allow plugins to define any setting they like. I'm still nervous about plugins over-riding existing or future default options to that command.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1100015398,Maybe let plugins define custom serve options?, https://github.com/simonw/sqlite-utils/issues/348#issuecomment-1011855133,https://api.github.com/repos/simonw/sqlite-utils/issues/348,1011855133,IC_kwDOCGYnMM48T68d,9599,simonw,2022-01-13T07:06:59Z,2022-01-13T07:06:59Z,OWNER,Wrote a lot more about this feature here: https://simonwillison.net/2022/Jan/12/how-i-build-a-feature/,"{""total_count"": 3, ""+1"": 3, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1067771698,Command for creating an empty database, https://github.com/simonw/datasette/issues/1592#issuecomment-1011185061,https://api.github.com/repos/simonw/datasette/issues/1592,1011185061,IC_kwDOBm6k_c48RXWl,9599,simonw,2022-01-12T15:50:41Z,2022-01-12T15:50:41Z,OWNER,Twitter: https://twitter.com/dracos/status/1481290103519592459,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1100499619,Row pages should show links to foreign keys, https://github.com/simonw/datasette/issues/1591#issuecomment-1010764036,https://api.github.com/repos/simonw/datasette/issues/1591,1010764036,IC_kwDOBm6k_c48PwkE,9599,simonw,2022-01-12T08:22:16Z,2022-01-12T08:22:32Z,OWNER,"The challenge here is avoiding clashes. What if a plugin adds an option that I later want to use for a new Datasette core feature? Or what if two plugins define the same option? Maybe the solution is to make them use namespaces defined by the plugin name. How about this: datasette t.db --plugin-setting datasette-tiddlywiki db t.db It's a bit verbose having an option that itself then takes THREE strings: plugin name, setting name, setting value - but it would work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1100015398,Maybe let plugins define custom serve options?, https://github.com/simonw/datasette/issues/1590#issuecomment-1010540923,https://api.github.com/repos/simonw/datasette/issues/1590,1010540923,IC_kwDOBm6k_c48O6F7,9599,simonw,2022-01-12T01:33:49Z,2022-01-12T01:33:49Z,OWNER,"Looking closer at the code quoted above, it doesn't modify `path` or `raw_path` at all - ALL it does is add the `route_path` to the scope.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099723916,Table+query JSON and CSV links broken when using `base_url` setting, https://github.com/simonw/datasette/issues/1590#issuecomment-1010538188,https://api.github.com/repos/simonw/datasette/issues/1590,1010538188,IC_kwDOBm6k_c48O5bM,9599,simonw,2022-01-12T01:28:41Z,2022-01-12T01:30:43Z,OWNER,"Oh wait! It looks like `route_path` is something I invented there. Yup, I added it in https://github.com/simonw/datasette/commit/a63412152518581c6a3d4e142b937e27dabdbfdb - commit message says: > - new `route_path` key in `request.scope` storing the path that was used for routing with the `base_url` prefix stripped So actually part of the mystery here is: why does the Fly hosted one NOT have that key?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099723916,Table+query JSON and CSV links broken when using `base_url` setting, https://github.com/simonw/datasette/issues/1590#issuecomment-1010538016,https://api.github.com/repos/simonw/datasette/issues/1590,1010538016,IC_kwDOBm6k_c48O5Yg,9599,simonw,2022-01-12T01:28:19Z,2022-01-12T01:28:19Z,OWNER,"The Daphne one has this key: `'route_path': '/-/asgi-scope',` Maybe Datasette's routing code needs to look out for that, if it's available, and use it to reconstruct the requested path? The code in question is here: https://github.com/simonw/datasette/blob/8c401ee0f054de2f568c3a8302c9223555146407/datasette/app.py#L1143-L1149","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099723916,Table+query JSON and CSV links broken when using `base_url` setting, https://github.com/simonw/datasette/issues/1590#issuecomment-1010537058,https://api.github.com/repos/simonw/datasette/issues/1590,1010537058,IC_kwDOBm6k_c48O5Ji,9599,simonw,2022-01-12T01:26:34Z,2022-01-12T01:26:34Z,OWNER,"I'm using the https://datasette.io/plugins/datasette-debug-asgi plugin to investigate. On my laptop using Daphne I get this: http://127.0.0.1:8032/datasettes/-/asgi-scope ``` {'actor': None, 'asgi': {'version': '3.0'}, 'client': ['127.0.0.1', 53767], 'csrftoken': ._asgi_csrf_decorator..app_wrapped_with_csrf..get_csrftoken at 0x1122aeef0>, 'headers': [(b'host', b'127.0.0.1:8032'), (b'user-agent', b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:95.0) Gecko' b'/20100101 Firefox/95.0'), (b'accept', b'text/html,application/xhtml+xml,application/xml;q=0.9,image/' b'avif,image/webp,*/*;q=0.8'), (b'accept-language', b'en-US,en;q=0.5'), (b'accept-encoding', b'gzip, deflate'), (b'dnt', b'1'), (b'connection', b'keep-alive'), (b'cookie', b'_ga=GA1.1.742283954.1628542653'), (b'upgrade-insecure-requests', b'1'), (b'sec-fetch-dest', b'document'), (b'sec-fetch-mode', b'navigate'), (b'sec-fetch-site', b'none'), (b'sec-fetch-user', b'?1')], 'http_version': '1.1', 'method': 'GET', 'path': '/datasettes/-/asgi-scope', 'path_remaining': '', 'query_string': b'', 'raw_path': b'/datasettes/-/asgi-scope', 'root_path': '', 'route_path': '/-/asgi-scope', 'scheme': 'http', 'server': ['127.0.0.1', 8032], 'type': 'http', 'url_route': {'kwargs': {}}} ``` On the demo running on Fly (which I just redeployed with that plugin) I get this: https://datasette-apache-proxy-demo.fly.dev/prefix/-/asgi-scope ``` {'actor': None, 'asgi': {'spec_version': '2.1', 'version': '3.0'}, 'client': ('86.109.12.167', 0), 'csrftoken': ._asgi_csrf_decorator..app_wrapped_with_csrf..get_csrftoken at 0x7f4c0413bca0>, 'headers': [(b'host', b'datasette-apache-proxy-demo.fly.dev'), (b'user-agent', b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:95.0) Gecko' b'/20100101 Firefox/95.0'), (b'accept', b'text/html,application/xhtml+xml,application/xml;q=0.9,image/' b'avif,image/webp,*/*;q=0.8'), (b'accept-language', b'en-US,en;q=0.5'), (b'accept-encoding', b'gzip, deflate, br'), (b'dnt', b'1'), (b'x-request-start', b't=1641950740651658'), (b'sec-fetch-dest', b'document'), (b'sec-fetch-mode', b'navigate'), (b'sec-fetch-site', b'none'), (b'sec-fetch-user', b'?1'), (b'fly-client-ip', b'24.5.172.176'), (b'x-forwarded-for', b'24.5.172.176, 213.188.193.173, 86.109.12.167'), (b'fly-forwarded-proto', b'https'), (b'x-forwarded-proto', b'https'), (b'fly-forwarded-ssl', b'on'), (b'x-forwarded-ssl', b'on'), (b'fly-forwarded-port', b'443'), (b'x-forwarded-port', b'443'), (b'fly-region', b'sjc'), (b'fly-request-id', b'01FS5Y805BX43HM94T8XW610KG'), (b'via', b'2 fly.io'), (b'fly-dispatch-start', b't=1641950740683198;instance=87f188a2'), (b'x-forwarded-host', b'datasette-apache-proxy-demo.fly.dev'), (b'x-forwarded-server', b'localhost'), (b'connection', b'Keep-Alive')], 'http_version': '1.1', 'method': 'GET', 'path': '/-/asgi-scope', 'query_string': b'', 'raw_path': b'/-/asgi-scope', 'root_path': '', 'scheme': 'https', 'server': ('127.0.0.1', 8001), 'type': 'http', 'url_route': {'kwargs': {}}} ``` The version that works as ` 'raw_path': b'/-/asgi-scope'` - the version that fails has `'raw_path': b'/datasettes/-/asgi-scope'`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099723916,Table+query JSON and CSV links broken when using `base_url` setting, https://github.com/simonw/datasette/issues/1590#issuecomment-1010533133,https://api.github.com/repos/simonw/datasette/issues/1590,1010533133,IC_kwDOBm6k_c48O4MN,9599,simonw,2022-01-12T01:19:19Z,2022-01-12T01:19:19Z,OWNER,"Thanks for the steps to reproduce - I have your bug running on my laptop now. I've been mostly testing this stuff using the hosted copy of Datasette here, which doesn't exhibit the bug: https://datasette-apache-proxy-demo.fly.dev/prefix/fixtures?sql=select+sqlite_version%28%29 Something interesting definitely going on here!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099723916,Table+query JSON and CSV links broken when using `base_url` setting, https://github.com/simonw/sqlite-utils/issues/381#issuecomment-1010462035,https://api.github.com/repos/simonw/sqlite-utils/issues/381,1010462035,IC_kwDOCGYnMM48Om1T,9599,simonw,2022-01-11T23:33:37Z,2022-01-11T23:33:37Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#returning-all-rows-in-a-table,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099584685,`sqlite-utils rows` options `--limit` and `--offset`, https://github.com/simonw/sqlite-utils/issues/382#issuecomment-1010461844,https://api.github.com/repos/simonw/sqlite-utils/issues/382,1010461844,IC_kwDOCGYnMM48OmyU,9599,simonw,2022-01-11T23:33:14Z,2022-01-11T23:33:14Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#returning-all-rows-in-a-table,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099585611,`--where` option for `sqlite-rows`, https://github.com/simonw/sqlite-utils/issues/381#issuecomment-1010441118,https://api.github.com/repos/simonw/sqlite-utils/issues/381,1010441118,IC_kwDOCGYnMM48Ohue,9599,simonw,2022-01-11T22:56:53Z,2022-01-11T22:57:09Z,OWNER,"`sqlite-utils search` has `--limit` already: https://sqlite-utils.datasette.io/en/latest/cli-reference.html#search ``` --limit INTEGER Number of rows to return - defaults to everything ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099584685,`sqlite-utils rows` options `--limit` and `--offset`, https://github.com/simonw/sqlite-utils/issues/383#issuecomment-1010440166,https://api.github.com/repos/simonw/sqlite-utils/issues/383,1010440166,IC_kwDOCGYnMM48Ohfm,9599,simonw,2022-01-11T22:55:05Z,2022-01-11T22:55:05Z,OWNER,Twitter thread about this: https://twitter.com/simonw/status/1481020195074293761,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099586786,Add documentation page with the output of `--help`, https://github.com/simonw/sqlite-utils/issues/383#issuecomment-1010387223,https://api.github.com/repos/simonw/sqlite-utils/issues/383,1010387223,IC_kwDOCGYnMM48OUkX,9599,simonw,2022-01-11T21:45:32Z,2022-01-11T21:45:32Z,OWNER,The new page of documentation: https://sqlite-utils.datasette.io/en/latest/cli-reference.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099586786,Add documentation page with the output of `--help`, https://github.com/simonw/sqlite-utils/issues/383#issuecomment-1010386802,https://api.github.com/repos/simonw/sqlite-utils/issues/383,1010386802,IC_kwDOCGYnMM48OUdy,9599,simonw,2022-01-11T21:44:53Z,2022-01-11T21:44:53Z,OWNER,Here's the `cog` code I used: https://github.com/simonw/sqlite-utils/blob/1d44b0cc2784c94aed1bcf350225cd86ee1aa7e5/docs/cli-reference.rst#L11-L76,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099586786,Add documentation page with the output of `--help`, https://github.com/simonw/sqlite-utils/issues/383#issuecomment-1010333511,https://api.github.com/repos/simonw/sqlite-utils/issues/383,1010333511,IC_kwDOCGYnMM48OHdH,9599,simonw,2022-01-11T20:27:08Z,2022-01-11T20:27:08Z,OWNER,"I'll call the new page ""CLI reference"", for consistency with the API reference page here: https://sqlite-utils.datasette.io/en/stable/reference.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099586786,Add documentation page with the output of `--help`, https://github.com/simonw/sqlite-utils/issues/380#issuecomment-1009544785,https://api.github.com/repos/simonw/sqlite-utils/issues/380,1009544785,IC_kwDOCGYnMM48LG5R,9599,simonw,2022-01-11T02:32:56Z,2022-01-11T02:32:56Z,OWNER,"CLI and Python library improvements to help run [ANALYZE](https://www.sqlite.org/lang_analyze.html) after creating indexes or inserting rows, to gain better performance from the SQLite query planner when it runs against indexes. Three new CLI commands: `create-database`, `analyze` and `bulk`. - New `sqlite-utils create-database` command for creating new empty database files. ([#348](https://github.com/simonw/sqlite-utils/issues/348)) - New Python methods for running `ANALYZE` against a database, table or index: `db.analyze()` and `table.analyze()`, see [Optimizing index usage with ANALYZE](https://sqlite-utils.datasette.io/en/stable/python-api.html#python-api-analyze). ([#366](https://github.com/simonw/sqlite-utils/issues/366)) - New [sqlite-utils analyze command](https://sqlite-utils.datasette.io/en/stable/cli.html#cli-analyze) for running `ANALYZE` using the CLI. ([#379](https://github.com/simonw/sqlite-utils/issues/379)) - The `create-index`, `insert` and `update` commands now have a new `--analyze` option for running `ANALYZE` after the command has completed. ([#379](https://github.com/simonw/sqlite-utils/issues/379)) - New [sqlite-utils bulk command](https://sqlite-utils.datasette.io/en/stable/cli.html#cli-bulk) which can import records in the same way as `sqlite-utils insert` (from JSON, CSV or TSV) and use them to bulk execute a parametrized SQL query. ([#375](https://github.com/simonw/sqlite-utils/issues/375)) - The CLI tool can now also be run using `python -m sqlite_utils`. ([#368](https://github.com/simonw/sqlite-utils/issues/368)) - Using `--fmt` now implies `--table`, so you don't need to pass both options. ([#374](https://github.com/simonw/sqlite-utils/issues/374)) - The `--convert` function applied to rows can now modify the row in place. ([#371](https://github.com/simonw/sqlite-utils/issues/371)) - The [insert-files command](https://sqlite-utils.datasette.io/en/stable/cli.html#cli-insert-files) supports two new columns: `stem` and `suffix`. ([#372](https://github.com/simonw/sqlite-utils/issues/372)) - The `--nl` import option now ignores blank lines in the input. ([#376](https://github.com/simonw/sqlite-utils/issues/376)) - Fixed bug where streaming input to the `insert` command with `--batch-size 1` would appear to only commit after several rows had been ingested, due to unnecessary input buffering. ([#364](https://github.com/simonw/sqlite-utils/issues/364))","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1098574572,Release notes for 3.21, https://github.com/simonw/sqlite-utils/issues/375#issuecomment-1009536276,https://api.github.com/repos/simonw/sqlite-utils/issues/375,1009536276,IC_kwDOCGYnMM48LE0U,9599,simonw,2022-01-11T02:12:58Z,2022-01-11T02:12:58Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#executing-sql-in-bulk,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097251014,`sqlite-utils bulk` command, https://github.com/simonw/sqlite-utils/pull/377#issuecomment-1009534817,https://api.github.com/repos/simonw/sqlite-utils/issues/377,1009534817,IC_kwDOCGYnMM48LEdh,9599,simonw,2022-01-11T02:09:38Z,2022-01-11T02:09:38Z,OWNER,"I tested this like so: ``` % wget 'https://raw.githubusercontent.com/wri/global-power-plant-database/master/output_database/global_power_plant_database.csv' % sqlite-utils create-database test.db % sqlite-utils create-table test.db power_plants url text owner text % sqlite-utils schema test.db CREATE TABLE [power_plants] ( [url] TEXT, [owner] TEXT ); % sqlite-utils bulk test.db 'insert into power_plants (url, owner) values (:url, :owner)' global_power_plant_database.csv --csv [------------------------------------] 0% [###################################-] 99% % sqlite-utils tables --counts test.db -t table count ------------ ------- power_plants 33643 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097477582,`sqlite-utils bulk` command, https://github.com/simonw/sqlite-utils/pull/377#issuecomment-1009532125,https://api.github.com/repos/simonw/sqlite-utils/issues/377,1009532125,IC_kwDOCGYnMM48LDzd,9599,simonw,2022-01-11T02:03:35Z,2022-01-11T02:03:35Z,OWNER,Documentation: https://github.com/simonw/sqlite-utils/blob/f4ea0d32c0543373eefaa9b9f3911eb07549eecb/docs/cli.rst#executing-sql-in-bulk,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097477582,`sqlite-utils bulk` command, https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1009521921,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1009521921,IC_kwDOCGYnMM48LBUB,9599,simonw,2022-01-11T01:37:53Z,2022-01-11T01:37:53Z,OWNER,"I decided to go with making this opt-in, mainly for consistency with the other places where I added this feature - see: - #379 - #366 You can now run the following: sqlite-utils create-index mydb.db mytable mycolumn --analyze And ``ANALYZE`` will be run on the index once it has been created.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index, https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009508865,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1009508865,IC_kwDOCGYnMM48K-IB,9599,simonw,2022-01-11T01:08:51Z,2022-01-11T01:08:51Z,OWNER,"The Python methods are all done now, next step is the CLI options. I'll do those in a separate issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE, https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009288898,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1009288898,IC_kwDOCGYnMM48KIbC,9599,simonw,2022-01-10T19:54:04Z,2022-01-10T19:54:04Z,OWNER,"Having browsed the API reference I think the methods that would benefit from an `analyze=True` parameter are: - `db.create_index` - `table.insert_all` - `table.upsert_all` - `table.delete_where`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE, https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009285627,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1009285627,IC_kwDOCGYnMM48KHn7,9599,simonw,2022-01-10T19:49:19Z,2022-01-10T19:51:25Z,OWNER,Documentation for those two new methods: https://sqlite-utils.datasette.io/en/latest/python-api.html#optimizing-index-usage-with-analyze,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE, https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009286373,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1009286373,IC_kwDOCGYnMM48KHzl,9599,simonw,2022-01-10T19:50:22Z,2022-01-10T19:50:22Z,OWNER,"With respect to #365, I'm now thinking that having the ability to say ""... and then run ANALYZE"" could be useful for a bunch of Python methods. For example: ```python db[""dogs""].insert_all(list_of_dogs, analyze=True) db[""dogs""].create_index([""name""], analyze=True) ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE, https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009273525,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1009273525,IC_kwDOCGYnMM48KEq1,9599,simonw,2022-01-10T19:32:39Z,2022-01-10T19:32:39Z,OWNER,"I'm going to implement the Python library methods based on the prototype: ```diff commit 650f97a08f29a688c530e5f6c9eedc9269ed7bdc Author: Simon Willison Date: Sat Jan 8 13:34:01 2022 -0800 Initial prototype of .analyze(), refs #366 diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py index dfc4723..1348b4a 100644 --- a/sqlite_utils/db.py +++ b/sqlite_utils/db.py @@ -923,6 +923,13 @@ class Database: ""Run a SQLite ``VACUUM`` against the database."" self.execute(""VACUUM;"") + def analyze(self, name=None): + ""Run ``ANALYZE`` against the entire database or a named table or index."" + sql = ""ANALYZE"" + if name is not None: + sql += "" [{}]"".format(name) + self.execute(sql) + class Queryable: def exists(self) -> bool: @@ -2902,6 +2909,10 @@ class Table(Queryable): ) return self + def analyze(self): + ""Run ANALYZE against this table"" + self.db.analyze(self.name) + def analyze_column( self, column: str, common_limit: int = 10, value_truncate=None, total_rows=None ) -> ""ColumnDetails"": ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE, https://github.com/simonw/sqlite-utils/pull/367#issuecomment-1009272446,https://api.github.com/repos/simonw/sqlite-utils/issues/367,1009272446,IC_kwDOCGYnMM48KEZ-,9599,simonw,2022-01-10T19:31:08Z,2022-01-10T19:31:08Z,OWNER,I'm going to implement this in a separate commit from this PR.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097041471,Initial prototype of .analyze() methods, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008557414,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008557414,IC_kwDOCGYnMM48HV1m,9599,simonw,2022-01-10T05:36:19Z,2022-01-10T05:36:19Z,OWNER,That did the trick.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/375#issuecomment-1008556706,https://api.github.com/repos/simonw/sqlite-utils/issues/375,1008556706,IC_kwDOCGYnMM48HVqi,9599,simonw,2022-01-10T05:33:41Z,2022-01-10T05:33:41Z,OWNER,"I tested the prototype like this: sqlite-utils blah.db 'create table blah (id integer primary key, name text)' echo 'id,name 1,Cleo 2,Chicken' > blah.csv sqlite-utils bulk blah.db 'insert into blah (id, name) values (:id, :name)' blah.csv --csv ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097251014,`sqlite-utils bulk` command, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008546573,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008546573,IC_kwDOCGYnMM48HTMN,9599,simonw,2022-01-10T05:05:15Z,2022-01-10T05:05:15Z,OWNER,"Bit nasty but it might work: ```python def try_until(expected): tries = 0 while True: rows = list(Database(db_path)[""rows""].rows) if rows == expected: return tries += 1 if tries > 10: assert False, ""Expected {}, got {}"".format(expected, rows) time.sleep(tries * 0.1) try_until([{""name"": ""Azi""}]) proc.stdin.write(b'{""name"": ""Suna""}\n') proc.stdin.flush() try_until([{""name"": ""Azi""}, {""name"": ""Suna""}]) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008545140,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008545140,IC_kwDOCGYnMM48HS10,9599,simonw,2022-01-10T05:01:34Z,2022-01-10T05:01:34Z,OWNER,"Urgh, tests are still failing intermittently - for example: ``` time.sleep(0.4) > assert list(Database(db_path)[""rows""].rows) == [{""name"": ""Azi""}] E AssertionError: assert [] == [{'name': 'Azi'}] E Right contains one more item: {'name': 'Azi'} E Full diff: E - [{'name': 'Azi'}] E + [] ``` I'm going to change this code to keep on trying up to 10 seconds - that should get the tests to pass faster on most machines.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008537194,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008537194,IC_kwDOCGYnMM48HQ5q,9599,simonw,2022-01-10T04:29:53Z,2022-01-10T04:31:29Z,OWNER,"After a bunch of debugging with `print()` statements it's clear that the problem isn't with when things are committed or the size of the batches - it's that the data sent to standard input is all being processed in one go, not a line at a time. I think that's because it is being buffered by this: https://github.com/simonw/sqlite-utils/blob/d2a79d200f9071a86027365fa2a576865b71064f/sqlite_utils/cli.py#L759-L770 The buffering is there so that we can sniff the first few bytes to detect if it's a CSV file - added in 99ff0a288c08ec2071139c6031eb880fa9c95310 for #230. So maybe for non-CSV inputs we should disable buffering?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008526736,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008526736,IC_kwDOCGYnMM48HOWQ,9599,simonw,2022-01-10T04:07:29Z,2022-01-10T04:07:29Z,OWNER,"I think this test is right: ```python def test_insert_streaming_batch_size_1(db_path): # https://github.com/simonw/sqlite-utils/issues/364 # Streaming with --batch-size 1 should commit on each record # Can't use CliRunner().invoke() here bacuse we need to # run assertions in between writing to process stdin proc = subprocess.Popen( [ sys.executable, ""-m"", ""sqlite_utils"", ""insert"", db_path, ""rows"", ""-"", ""--nl"", ""--batch-size"", ""1"", ], stdin=subprocess.PIPE, ) proc.stdin.write(b'{""name"": ""Azi""}') proc.stdin.flush() assert list(Database(db_path)[""rows""].rows) == [{""name"": ""Azi""}] proc.stdin.write(b'{""name"": ""Suna""}') proc.stdin.flush() assert list(Database(db_path)[""rows""].rows) == [{""name"": ""Azi""}, {""name"": ""Suna""}] proc.stdin.close() proc.wait() ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/348#issuecomment-1008383293,https://api.github.com/repos/simonw/sqlite-utils/issues/348,1008383293,IC_kwDOCGYnMM48GrU9,9599,simonw,2022-01-09T20:38:17Z,2022-01-09T20:38:17Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#creating-an-empty-database,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1067771698,Command for creating an empty database, https://github.com/simonw/sqlite-utils/issues/348#issuecomment-1008367607,https://api.github.com/repos/simonw/sqlite-utils/issues/348,1008367607,IC_kwDOCGYnMM48Gnf3,9599,simonw,2022-01-09T20:22:43Z,2022-01-09T20:22:43Z,OWNER,I'm not going to implement `--page-size` unless someone specifically requests it - I don't like having features that I've never needed to use myself.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1067771698,Command for creating an empty database, https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008364701,https://api.github.com/repos/simonw/sqlite-utils/issues/371,1008364701,IC_kwDOCGYnMM48Gmyd,9599,simonw,2022-01-09T20:04:35Z,2022-01-09T20:04:35Z,OWNER,"The previous code for highlighting errors in syntax (which was already a bit confused thanks to the added `return`, see https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991393684 - isn't compatible with this approach at all. I'm going to ditch it and just show a generic `Error: Could not compile code` message.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097128334,Support mutating row in `--convert` without returning it, https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008354207,https://api.github.com/repos/simonw/sqlite-utils/issues/371,1008354207,IC_kwDOCGYnMM48GkOf,9599,simonw,2022-01-09T18:54:54Z,2022-01-09T18:54:54Z,OWNER,"This seems to work: ```python def _compile_code(code, imports, variable=""value""): locals = {} globals = {""r"": recipes, ""recipes"": recipes} # If user defined a convert() function, return that try: exec(code, globals, locals) return locals[""convert""] except (AttributeError, SyntaxError, NameError, KeyError, TypeError): pass # Try compiling their code as a function instead body_variants = [code] # If single line and no 'return', try adding the return if ""\n"" not in code and not code.strip().startswith(""return ""): body_variants.insert(0, ""return {}"".format(code)) for variant in body_variants: new_code = [""def fn({}):"".format(variable)] for line in variant.split(""\n""): new_code.append("" {}"".format(line)) try: code_o = compile(""\n"".join(new_code), """", ""exec"") break except SyntaxError: # Try another variant, e.g. for 'return row[""column""] = 1' continue for import_ in imports: globals[import_.split(""."")[0]] = __import__(import_) exec(code_o, globals, locals) return locals[""fn""] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097128334,Support mutating row in `--convert` without returning it, https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008348032,https://api.github.com/repos/simonw/sqlite-utils/issues/371,1008348032,IC_kwDOCGYnMM48GiuA,9599,simonw,2022-01-09T18:14:02Z,2022-01-09T18:14:02Z,OWNER,Here's the code in question: https://github.com/simonw/sqlite-utils/blob/b8c134059e89f0fa040b84fb7d0bda25b9a52759/sqlite_utils/utils.py#L288-L299,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097128334,Support mutating row in `--convert` without returning it, https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008347768,https://api.github.com/repos/simonw/sqlite-utils/issues/371,1008347768,IC_kwDOCGYnMM48Gip4,9599,simonw,2022-01-09T18:12:30Z,2022-01-09T18:12:30Z,OWNER,"Tried this test: ```python result = CliRunner().invoke( cli.cli, [ ""insert"", db_path, ""rows"", ""-"", ""--convert"", 'row[""is_chicken""] = True', ], input='{""name"": ""Azi""}', ) ``` And got this error: > `E + where 1 = ', 2, 30, ' return row[""is_chicken""] = True\n'))>.exit_code` The code snippet compilation isn't currently compatible with this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097128334,Support mutating row in `--convert` without returning it, https://github.com/simonw/sqlite-utils/issues/374#issuecomment-1008346841,https://api.github.com/repos/simonw/sqlite-utils/issues/374,1008346841,IC_kwDOCGYnMM48GibZ,9599,simonw,2022-01-09T18:06:50Z,2022-01-09T18:06:50Z,OWNER,"In addition to a unit test I manually tested all of the above, e.g. ``` % sqlite-utils indexes global-power-plants.db sqlite_master --fmt rst ======= ============ ======= ===== ====== ====== ====== ===== table index_name seqno cid name desc coll key ======= ============ ======= ===== ====== ====== ====== ===== ======= ============ ======= ===== ====== ====== ====== ===== ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097135860,`--fmt` should imply `-t`, https://github.com/simonw/sqlite-utils/issues/374#issuecomment-1008346338,https://api.github.com/repos/simonw/sqlite-utils/issues/374,1008346338,IC_kwDOCGYnMM48GiTi,9599,simonw,2022-01-09T18:03:22Z,2022-01-09T18:03:22Z,OWNER,"Commands that support `--fmt` (via the `@output_options` decorator) are: - `tables` - `views` - `query` - `memory` - `search` - `rows` - `triggers` - `indexes` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097135860,`--fmt` should imply `-t`, https://github.com/simonw/sqlite-utils/issues/374#issuecomment-1008345267,https://api.github.com/repos/simonw/sqlite-utils/issues/374,1008345267,IC_kwDOCGYnMM48GiCz,9599,simonw,2022-01-09T17:56:37Z,2022-01-09T17:56:37Z,OWNER,"Better: ```python if fmt: table = True ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097135860,`--fmt` should imply `-t`, https://github.com/simonw/sqlite-utils/issues/373#issuecomment-1008344980,https://api.github.com/repos/simonw/sqlite-utils/issues/373,1008344980,IC_kwDOCGYnMM48Gh-U,9599,simonw,2022-01-09T17:54:53Z,2022-01-09T17:54:53Z,OWNER,Updated TIL: https://til.simonwillison.net/python/cog-to-update-help-in-readme#user-content-cog-for-restructuredtext,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097135732,List `--fmt` options in the docs , https://github.com/simonw/sqlite-utils/issues/373#issuecomment-1008344525,https://api.github.com/repos/simonw/sqlite-utils/issues/373,1008344525,IC_kwDOCGYnMM48Gh3N,9599,simonw,2022-01-09T17:52:22Z,2022-01-09T17:52:22Z,OWNER,Updated docs: https://sqlite-utils.datasette.io/en/latest/cli.html#table-formatted-output,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097135732,List `--fmt` options in the docs , https://github.com/simonw/sqlite-utils/issues/373#issuecomment-1008341078,https://api.github.com/repos/simonw/sqlite-utils/issues/373,1008341078,IC_kwDOCGYnMM48GhBW,9599,simonw,2022-01-09T17:31:12Z,2022-01-09T17:31:12Z,OWNER,"Found an example of using `cog` in a rST file here: https://github.com/nedbat/coveragepy/blob/f3238eea7e403d13a217b30579b1a1c2cbff62e3/doc/dbschema.rst#L21 ``` .. [[[cog from coverage.sqldata import SCHEMA_VERSION print("".. code::"") print() print(f"" SCHEMA_VERSION = {SCHEMA_VERSION}"") print() .. ]]] .. code:: SCHEMA_VERSION = 7 .. [[[end]]] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097135732,List `--fmt` options in the docs , https://github.com/simonw/sqlite-utils/issues/375#issuecomment-1008338186,https://api.github.com/repos/simonw/sqlite-utils/issues/375,1008338186,IC_kwDOCGYnMM48GgUK,9599,simonw,2022-01-09T17:13:33Z,2022-01-09T17:13:54Z,OWNER," cat blah.csv | sqlite-utils bulk blah.db - \ ""insert into blah (:foo, :bar)"" --csv ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097251014,`sqlite-utils bulk` command, https://github.com/simonw/sqlite-utils/issues/374#issuecomment-1008252732,https://api.github.com/repos/simonw/sqlite-utils/issues/374,1008252732,IC_kwDOCGYnMM48GLc8,9599,simonw,2022-01-09T08:25:30Z,2022-01-09T08:25:30Z,OWNER,Need to change `if table:` to `if table or fmt:` in a few places.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097135860,`--fmt` should imply `-t`, https://github.com/simonw/sqlite-utils/issues/372#issuecomment-1008247370,https://api.github.com/repos/simonw/sqlite-utils/issues/372,1008247370,IC_kwDOCGYnMM48GKJK,9599,simonw,2022-01-09T07:51:18Z,2022-01-09T07:51:18Z,OWNER,"Pathlib says the stem of that would be `dogs.and.cats.jpg` - best stick with that for consistency. https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.suffix It calls the last bit `suffix` - maybe I should use that instead of `ext`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097129710,Idea: `suffix` and `stem` file columns, https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008246366,https://api.github.com/repos/simonw/sqlite-utils/issues/371,1008246366,IC_kwDOCGYnMM48GJ5e,9599,simonw,2022-01-09T07:42:14Z,2022-01-09T07:42:14Z,OWNER,Also need to update relevant docs for that example.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097128334,Support mutating row in `--convert` without returning it, https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008246239,https://api.github.com/repos/simonw/sqlite-utils/issues/371,1008246239,IC_kwDOCGYnMM48GJ3f,9599,simonw,2022-01-09T07:41:24Z,2022-01-09T07:41:24Z,OWNER,"Might be a case of modifying this line: https://github.com/simonw/sqlite-utils/blob/e0c476bc380744680c8b7675c24fb0e9f5ec6dcd/sqlite_utils/cli.py#L828 To: ```python docs = (fn(doc) or doc for doc in docs) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097128334,Support mutating row in `--convert` without returning it, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008234293,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008234293,IC_kwDOCGYnMM48GG81,9599,simonw,2022-01-09T05:37:02Z,2022-01-09T05:37:02Z,OWNER,Calling `p.stdin.close()` and then `p.wait()` terminates the subprocess.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008233910,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008233910,IC_kwDOCGYnMM48GG22,9599,simonw,2022-01-09T05:32:53Z,2022-01-09T05:35:45Z,OWNER,"This is strange. The following: ```pycon >>> import subprocess >>> p = subprocess.Popen([""sqlite-utils"", ""insert"", ""/tmp/stream.db"", ""stream"", ""-"", ""--nl""], stdin=subprocess.PIPE) >>> p.stdin.write(b'\n'.join(b'{""id"": %s}' % str(i).encode(""utf-8"") for i in range(1000))) 11889 >>> # At this point /tmp/stream.db is still 0 bytes - but if I then run this: >>> p.stdin.close() >>> # /tmp/stream.db is now 20K and contains the written data ``` No wait, mystery solved - I can add `p.stdin.flush()` instead of `p.stdin.close()` and the file suddenly jumps up in size.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008232075,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008232075,IC_kwDOCGYnMM48GGaL,9599,simonw,2022-01-09T05:13:15Z,2022-01-09T05:13:56Z,OWNER,"I think the query that will help solve this is: `explain query plan select * from ny_times_us_counties where state = 1 and county = 2` In this case, the query planner needs to decide if it should use the index for the `state` column or the index for the `county` column. That's where the statistics come into play. In particular: | tbl | idx | stat | |----------------------|---------------------------------|---------------| | ny_times_us_counties | idx_ny_times_us_counties_date | 2092871 2915 | | ny_times_us_counties | idx_ny_times_us_counties_fips | 2092871 651 | | ny_times_us_counties | idx_ny_times_us_counties_county | 2092871 1085 | | ny_times_us_counties | idx_ny_times_us_counties_state | 2092871 37373 | Those numbers are explained by this comment in the SQLite C code: https://github.com/sqlite/sqlite/blob/5622c7f97106314719740098cf0854e7eaa81802/src/analyze.c#L41-L55 ``` ** There is normally one row per index, with the index identified by the ** name in the idx column. The tbl column is the name of the table to ** which the index belongs. In each such row, the stat column will be ** a string consisting of a list of integers. The first integer in this ** list is the number of rows in the index. (This is the same as the ** number of rows in the table, except for partial indices.) The second ** integer is the average number of rows in the index that have the same ** value in the first column of the index. ``` So that table is telling us that using a value in the `county` column will filter down to an average of 1,085 rows, whereas filtering on the `state` column will filter down to an average of 37,373 - so clearly the `county` index is the better index to use here! Just one catch: against both my` covid.db` and my `covid-analyzed.db` databases the `county` index is picked for both of them - so SQLite is somehow guessing that `county` is a better index even though it doesn't have statistics for that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes, https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008229839,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1008229839,IC_kwDOCGYnMM48GF3P,9599,simonw,2022-01-09T04:51:44Z,2022-01-09T04:51:44Z,OWNER,"Found one report on Stack Overflow from 9 years ago of someone seeing broken performance after running `ANALYZE`, hard to say that's a trend and not a single weird edge-case though! https://stackoverflow.com/q/12947214/6083","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index, https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008229341,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008229341,IC_kwDOCGYnMM48GFvd,9599,simonw,2022-01-09T04:45:38Z,2022-01-09T04:47:11Z,OWNER,"This is probably too fancy. I think maybe the way to do this is with `select * from [global-power-plants] where ""country_long"" = 'United Kingdom'` - then mess around with stats to see if I can get it to use the index or not based on them. Here's the explain for that: https://global-power-plants.datasettes.com/global-power-plants?sql=EXPLAIN+QUERY+PLAN+select+*+from+[global-power-plants]+where+%22country_long%22+%3D+%27United+Kingdom%27","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes, https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008227625,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008227625,IC_kwDOCGYnMM48GFUp,9599,simonw,2022-01-09T04:25:38Z,2022-01-09T04:25:38Z,OWNER,"```sql EXPLAIN QUERY PLAN select country_long, count(*) from [global-power-plants] group by country_long ``` https://global-power-plants.datasettes.com/global-power-plants?sql=EXPLAIN+QUERY+PLAN+select+country_long%2C+count%28*%29+from+%5Bglobal-power-plants%5D+group+by+country_long > SCAN TABLE global-power-plants USING COVERING INDEX ""global-power-plants_country_long""","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes, https://github.com/simonw/datasette/issues/1588#issuecomment-1008227436,https://api.github.com/repos/simonw/datasette/issues/1588,1008227436,IC_kwDOBm6k_c48GFRs,9599,simonw,2022-01-09T04:23:37Z,2022-01-09T04:25:04Z,OWNER,"Relevant code: https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/utils/__init__.py#L163-L170 https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/utils/__init__.py#L195-L204","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097101917,`explain query plan select` is too strict about whitespace, https://github.com/simonw/datasette/issues/1588#issuecomment-1008227491,https://api.github.com/repos/simonw/datasette/issues/1588,1008227491,IC_kwDOBm6k_c48GFSj,9599,simonw,2022-01-09T04:24:09Z,2022-01-09T04:24:09Z,OWNER,"I think this is the fix: ```python re.compile(r""^explain\s+query\s+plan\s+select\b""), ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097101917,`explain query plan select` is too strict about whitespace, https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008226862,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008226862,IC_kwDOCGYnMM48GFIu,9599,simonw,2022-01-09T04:17:55Z,2022-01-09T04:17:55Z,OWNER,"There are some clues as to what effect ANALYZE has in https://www.sqlite.org/optoverview.html Some quotes: > SQLite might use a skip-scan on an index if it knows that the first one or more columns contain many duplication values. If there are too few duplicates in the left-most columns of the index, then it would be faster to simply step ahead to the next value, and thus do a full table scan, than to do a binary search on an index to locate the next left-column value. > > The only way that SQLite can know that there are many duplicates in the left-most columns of an index is if the ANALYZE command has been run on the database. Without the results of ANALYZE, SQLite has to guess at the ""shape"" of the data in the table, and the default guess is that there are an average of 10 duplicates for every value in the left-most column of the index. Skip-scan only becomes profitable (it only gets to be faster than a full table scan) when the number of duplicates is about 18 or more. Hence, a skip-scan is never used on a database that has not been analyzed. And > Join reordering is automatic and usually works well enough that programmers do not have to think about it, especially if ANALYZE has been used to gather statistics about the available indexes, though occasionally some hints from the programmer are needed. And > The various sqlite_statN tables contain information on how selective the various indexes are. For example, the sqlite_stat1 table might indicate that an equality constraint on column x reduces the search space to 10 rows on average, whereas an equality constraint on column y reduces the search space to 3 rows on average. In that case, SQLite would prefer to use index ex2i2 since that index is more selective. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes, https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008226487,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008226487,IC_kwDOCGYnMM48GFC3,9599,simonw,2022-01-09T04:14:05Z,2022-01-09T04:14:05Z,OWNER,"Didn't manage to spot a meaningful difference with that database either: ``` analyze % python3 -m timeit '__import__(""sqlite3"").connect(""covid.db"").execute(""select fips, count(*) from [ny_times_us_counties] group by fips"").fetchall()' 2 loops, best of 5: 101 msec per loop analyze % python3 -m timeit '__import__(""sqlite3"").connect(""covid-analyzed.db"").execute(""select fips, count(*) from [ny_times_us_counties] group by fips"").fetchall()' 2 loops, best of 5: 103 msec per loop ``` Maybe `select fips, count(*) from [ny_times_us_counties] group by fips` isn't a good query for testing this?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes, https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008220270,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008220270,IC_kwDOCGYnMM48GDhu,9599,simonw,2022-01-09T03:12:38Z,2022-01-09T03:13:15Z,OWNER,"Basically no difference using this very basic benchmark: ``` analyze % python3 -m timeit '__import__(""sqlite3"").connect(""global-power-plants.db"").execute(""select country_long, count(*) from [global-power-plants] group by country_long"").fetchall()' 100 loops, best of 5: 2.39 msec per loop analyze % python3 -m timeit '__import__(""sqlite3"").connect(""global-power-plants-analyzed.db"").execute(""select country_long, count(*) from [global-power-plants] group by country_long"").fetchall()' 100 loops, best of 5: 2.38 msec per loop ``` I should try this against a much larger database. https://covid-19.datasettes.com/covid.db is 879MB.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes, https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008219844,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008219844,IC_kwDOCGYnMM48GDbE,9599,simonw,2022-01-09T03:08:09Z,2022-01-09T03:08:09Z,OWNER,"``` analyze % sqlite-utils global-power-plants-analyzed.db 'analyze' [{""rows_affected"": -1}] analyze % sqlite-utils tables global-power-plants-analyzed.db [{""table"": ""global-power-plants""}, {""table"": ""global-power-plants_fts""}, {""table"": ""global-power-plants_fts_data""}, {""table"": ""global-power-plants_fts_idx""}, {""table"": ""global-power-plants_fts_docsize""}, {""table"": ""global-power-plants_fts_config""}, {""table"": ""sqlite_stat1""}] analyze % sqlite-utils rows global-power-plants-analyzed.db sqlite_stat1 -t tbl idx stat ------------------------------- ---------------------------------- --------- global-power-plants_fts_config global-power-plants_fts_config 1 1 global-power-plants_fts_docsize 33643 global-power-plants_fts_idx global-power-plants_fts_idx 199 40 1 global-power-plants_fts_data 136 global-power-plants ""global-power-plants_owner"" 33643 4 global-power-plants ""global-power-plants_country_long"" 33643 202 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes, https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008219588,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008219588,IC_kwDOCGYnMM48GDXE,9599,simonw,2022-01-09T03:06:42Z,2022-01-09T03:06:42Z,OWNER,"``` analyze % sqlite-utils indexes global-power-plants.db -t table index_name seqno cid name desc coll key ------------------------------ ------------------------------------------------- ------- ----- ------------ ------ ------ ----- global-power-plants ""global-power-plants_owner"" 0 12 owner 0 BINARY 1 global-power-plants ""global-power-plants_country_long"" 0 1 country_long 0 BINARY 1 global-power-plants_fts_idx sqlite_autoindex_global-power-plants_fts_idx_1 0 0 segid 0 BINARY 1 global-power-plants_fts_idx sqlite_autoindex_global-power-plants_fts_idx_1 1 1 term 0 BINARY 1 global-power-plants_fts_config sqlite_autoindex_global-power-plants_fts_config_1 0 0 k 0 BINARY 1 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes, https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008219484,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008219484,IC_kwDOCGYnMM48GDVc,9599,simonw,2022-01-09T03:05:44Z,2022-01-09T03:05:44Z,OWNER,I'll start by running some experiments against the 11MB database file from https://global-power-plants.datasettes.com/global-power-plants.db,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes, https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008219191,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008219191,IC_kwDOCGYnMM48GDQ3,9599,simonw,2022-01-09T03:03:53Z,2022-01-09T03:03:53Z,OWNER,"Refs: - #366 - #365","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes, https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008163585,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1008163585,IC_kwDOCGYnMM48F1sB,9599,simonw,2022-01-08T22:14:39Z,2022-01-09T03:03:07Z,OWNER,"The reason I'm hesitating on this is that I've not actually used ANALYZE at all in nearly five years of messing around with SQLite! So I'm nervous that there are surprise downsides I haven't thought of. My hunch is that ANALYZE is only worth worrying about on much larger databases, in which case I'm OK supporting it as a thoroughly documented power-user feature rather than a default.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index, https://github.com/simonw/sqlite-utils/issues/368#issuecomment-1008216371,https://api.github.com/repos/simonw/sqlite-utils/issues/368,1008216371,IC_kwDOCGYnMM48GCkz,9599,simonw,2022-01-09T02:36:22Z,2022-01-09T02:36:22Z,OWNER,"In Python 3.6: https://docs.python.org/3.6/library/subprocess.html > This does not capture stdout or stderr by default. To do so, pass [`PIPE`](https://docs.python.org/3.6/library/subprocess.html#subprocess.PIPE ""subprocess.PIPE"") for the *stdout* and/or *stderr* arguments.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097087280,Offer `python -m sqlite_utils` as an alternative to `sqlite-utils`, https://github.com/simonw/sqlite-utils/issues/368#issuecomment-1008216271,https://api.github.com/repos/simonw/sqlite-utils/issues/368,1008216271,IC_kwDOCGYnMM48GCjP,9599,simonw,2022-01-09T02:35:09Z,2022-01-09T02:35:09Z,OWNER,"Test failure on Python 3.6: > `E TypeError: __init__() got an unexpected keyword argument 'capture_output'`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097087280,Offer `python -m sqlite_utils` as an alternative to `sqlite-utils`, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008216201,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008216201,IC_kwDOCGYnMM48GCiJ,9599,simonw,2022-01-09T02:34:12Z,2022-01-09T02:34:12Z,OWNER,"I can now write tests that look like this: https://github.com/simonw/sqlite-utils/blob/539f5ccd90371fa87f946018f8b77d55929e06db/tests/test_cli.py#L2024-L2030 Which means I can write a test that exercises this bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/368#issuecomment-1008215912,https://api.github.com/repos/simonw/sqlite-utils/issues/368,1008215912,IC_kwDOCGYnMM48GCdo,9599,simonw,2022-01-09T02:30:59Z,2022-01-09T02:30:59Z,OWNER,"Even better, inspired by `rich`, support `python -m sqlite_utils`. https://github.com/Textualize/rich/blob/master/rich/__main__.py","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097087280,Offer `python -m sqlite_utils` as an alternative to `sqlite-utils`, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008214998,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008214998,IC_kwDOCGYnMM48GCPW,9599,simonw,2022-01-09T02:23:20Z,2022-01-09T02:23:20Z,OWNER,"Possible way of running the test: add this to `sqlite_utils/cli.py`: ```python if __name__ == ""__main__"": cli() ``` Now the tool can be run using `python -m sqlite_utils.cli --help` Then in the test use `subprocess` to call `sys.executable` (the path to the current Python interpreter) and pass it `-m sqlite_utils.cli` to run the script!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008214406,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008214406,IC_kwDOCGYnMM48GCGG,9599,simonw,2022-01-09T02:18:21Z,2022-01-09T02:18:21Z,OWNER,"I'm having trouble figuring out the best way to write a unit test for this. Filed a relevant feature request for Click here: - https://github.com/pallets/click/issues/2171","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008163050,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1008163050,IC_kwDOCGYnMM48F1jq,9599,simonw,2022-01-08T22:10:51Z,2022-01-08T22:10:51Z,OWNER,"Is there a downside to having a `sqlite_stat1` table if it has wildly incorrect statistics in it? Imagine the following sequence of events: - User imports a few records, creating the table, using `sqlite-utils insert` - User runs `sqlite-utils create-index ...` which also creates and populates the `sqlite_stat1` table - User runs `insert` again to populate several million new records The user now has a database file with several million records and a statistics table that is wildly out of date, having been populated when they only had a few. Will this result in surprisingly bad query performance compared to it that statistics table did not exist at all? If so, I lean much harder towards `ANALYZE` as a strictly opt-in optimization, maybe with the `--analyze` option added to `sqlite-utils insert` top to help users opt in to updating their statistics after running big inserts.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index, https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1008158616,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1008158616,IC_kwDOCGYnMM48F0eY,9599,simonw,2022-01-08T21:35:32Z,2022-01-08T21:35:32Z,OWNER,"Built a prototype in a branch, see #367.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE, https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008158357,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1008158357,IC_kwDOCGYnMM48F0aV,9599,simonw,2022-01-08T21:33:07Z,2022-01-08T21:33:07Z,OWNER,"The one thing that worries me a little bit about doing this by default is that it adds a surprising new table to the database - it may be confusing to users if they run `create-index` and their database suddenly has a new `sqlite_stat1` table, see https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1008157132 Options here are: - Do it anyway. People can tolerate a surprise table appearing when they create an index. - Only run `ANALYZE` if the user says `sqlite-utils create-index ... --analyze` - Use the `--analyze` option, but also automatically run `ANALYZE` if they create an index and the database they are working with already has a `sqlite_stat1` table I'm currently leading towards that third option - @fgregg any thoughts?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index, https://github.com/simonw/datasette/issues/1587#issuecomment-1008157998,https://api.github.com/repos/simonw/datasette/issues/1587,1008157998,IC_kwDOBm6k_c48F0Uu,9599,simonw,2022-01-08T21:29:54Z,2022-01-08T21:29:54Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/00a2895cd2dc42c63846216b36b2dc9f41170129/datasette/database.py#L339-L354,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097040427,Add `sqlite_stat1`(-4) tables to hidden table list, https://github.com/simonw/datasette/issues/1587#issuecomment-1008157908,https://api.github.com/repos/simonw/datasette/issues/1587,1008157908,IC_kwDOBm6k_c48F0TU,9599,simonw,2022-01-08T21:29:06Z,2022-01-08T21:29:06Z,OWNER,"Depending on the SQLite version (and compile options) that ran `ANALYZE` these can be called: - `sqlite_stat1` - `sqlite_stat2` - `sqlite_stat3` - `sqlite_stat4`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097040427,Add `sqlite_stat1`(-4) tables to hidden table list, https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1008157132,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1008157132,IC_kwDOCGYnMM48F0HM,9599,simonw,2022-01-08T21:23:08Z,2022-01-08T21:25:05Z,OWNER,"Running `ANALYZE` creates a new visible table called `sqlite_stat1`: https://www.sqlite.org/fileformat.html#the_sqlite_stat1_table This should be added to the default list of hidden tables in Datasette. It looks something like this: | tbl | idx | stat | |---------------------------------|------------------------------------|-----------| | _counts | sqlite_autoindex__counts_1 | 5 1 | | global-power-plants_fts_config | global-power-plants_fts_config | 1 1 | | global-power-plants_fts_docsize | | 33643 | | global-power-plants_fts_idx | global-power-plants_fts_idx | 199 40 1 | | global-power-plants_fts_data | | 136 | | global-power-plants | ""global-power-plants_owner"" | 33643 4 | | global-power-plants | ""global-power-plants_country_long"" | 33643 202 | > In each such row, the sqlite_stat.stat column will be a string consisting of a list of integers followed by zero or more arguments. The first integer in this list is the approximate number of rows in the index. (The number of rows in the index is the same as the number of rows in the table, except for partial indexes.) The second integer is the approximate number of rows in the index that have the same value in the first column of the index. The third integer is the number number of rows in the index that have the same value for the first two columns. The N-th integer (for N>1) is the estimated average number of rows in the index which have the same value for the first N-1 columns. For a K-column index, there will be K+1 integers in the stat column. If the index is unique, then the last integer will be 1. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008155916,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008155916,IC_kwDOCGYnMM48Fz0M,9599,simonw,2022-01-08T21:16:46Z,2022-01-08T21:16:46Z,OWNER,"No, `chunks()` seems to work OK in the test I just added.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008154873,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008154873,IC_kwDOCGYnMM48Fzj5,9599,simonw,2022-01-08T21:11:55Z,2022-01-08T21:11:55Z,OWNER,"I'm suspicious that the `chunks()` utility function may not be working correctly: ```pycon In [10]: [list(d) for d in list(chunks('abc', 5))] Out[10]: [['a'], ['b'], ['c']] In [11]: [list(d) for d in list(chunks('abcdefghi', 5))] Out[11]: [['a'], ['b'], ['c'], ['d'], ['e'], ['f'], ['g'], ['h'], ['i']] In [12]: [list(d) for d in list(chunks('abcdefghi', 3))] Out[12]: [['a'], ['b'], ['c'], ['d'], ['e'], ['f'], ['g'], ['h'], ['i']] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008153586,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008153586,IC_kwDOCGYnMM48FzPy,9599,simonw,2022-01-08T21:06:15Z,2022-01-08T21:06:15Z,OWNER,"I added a print statement after `for query, params in queries_and_params` and confirmed that something in the code is waiting until 16 records are available to be inserted and then executing the inserts, even with `--batch-size 1`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008151884,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008151884,IC_kwDOCGYnMM48Fy1M,9599,simonw,2022-01-08T20:59:21Z,2022-01-08T20:59:21Z,OWNER,"(That Heroku example doesn't record the timestamp, which limits its usefulness)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008143248,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008143248,IC_kwDOCGYnMM48FwuQ,9599,simonw,2022-01-08T20:34:12Z,2022-01-08T20:34:12Z,OWNER,Built that tool: https://github.com/simonw/stream-delay and https://pypi.org/project/stream-delay/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008129841,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008129841,IC_kwDOCGYnMM48Ftcx,9599,simonw,2022-01-08T20:04:42Z,2022-01-08T20:04:42Z,OWNER,"It would be easier to test this if I had a utility for streaming out a file one line at a time. A few recipes for this in https://superuser.com/questions/526242/cat-file-to-terminal-at-particular-speed-of-lines-per-second - I'm going to build a quick `stream-delay` tool though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1007643254,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1007643254,IC_kwDOCGYnMM48D2p2,9599,simonw,2022-01-07T18:37:56Z,2022-01-07T18:37:56Z,OWNER,Or I could leave off `--no-analyze` and tell people that if they want to add an index without running analyze they can execute the `CREATE INDEX` themselves.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index, https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1007642831,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1007642831,IC_kwDOCGYnMM48D2jP,9599,simonw,2022-01-07T18:37:18Z,2022-01-07T18:37:18Z,OWNER,"After implementing #366 I can make it so `sqlite-utils create-index` automatically runs `db.analyze(index_name)` afterwards, maybe with a `--no-analyze` option in case anyone wants to opt out of that for specific performance reasons.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index, https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1007641634,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1007641634,IC_kwDOCGYnMM48D2Qi,9599,simonw,2022-01-07T18:35:35Z,2022-01-07T18:35:35Z,OWNER,"Since the existing CLI feature is this: $ sqlite-utils analyze-tables github.db tags I can add `sqlite-utils analyze` to reflect the Python library method.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE, https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1007639860,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1007639860,IC_kwDOCGYnMM48D100,9599,simonw,2022-01-07T18:32:59Z,2022-01-07T18:33:07Z,OWNER,"From the SQLite docs: > If no arguments are given, all attached databases are analyzed. If a schema name is given as the argument, then all tables and indices in that one database are analyzed. If the argument is a table name, then only that table and the indices associated with that table are analyzed. If the argument is an index name, then only that one index is analyzed. So I think this becomes two methods: - `db.analyze()` calls analyze on the whole database - `db.analyze(name_of_table_or_index)` for a specific named table or index - `table.analyze()` is a shortcut for `db.analyze(table.name)`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE, https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1007637963,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1007637963,IC_kwDOCGYnMM48D1XL,9599,simonw,2022-01-07T18:30:13Z,2022-01-07T18:30:13Z,OWNER,"Annoyingly I use the word ""analyze"" to mean something else in the CLI - for these features: - #207 - #320 there's only one method with a similar name in the Python library though and that's this one: https://github.com/simonw/sqlite-utils/blob/6e46b9913411682f3a3ec66f4d58886c1db8654b/sqlite_utils/db.py#L2904-L2906","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE, https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1007634999,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1007634999,IC_kwDOCGYnMM48D0o3,9599,simonw,2022-01-07T18:26:22Z,2022-01-07T18:26:22Z,OWNER,"I've not used the `ANALYZE` feature in SQLite at all before. Should probably add Python library methods for it. Annoyingly I use the word ""analyze"" to mean something else in the CLI - for these features: - #207 - #320","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index, https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1007633376,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1007633376,IC_kwDOCGYnMM48D0Pg,9599,simonw,2022-01-07T18:24:07Z,2022-01-07T18:24:07Z,OWNER,Relevant documentation: https://www.sqlite.org/lang_analyze.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index, https://github.com/simonw/sqlite-utils/issues/363#issuecomment-1006344080,https://api.github.com/repos/simonw/sqlite-utils/issues/363,1006344080,IC_kwDOCGYnMM47-5eQ,9599,simonw,2022-01-06T07:32:05Z,2022-01-06T07:32:05Z,OWNER,As part of this work I should add test coverage of this error message too: https://github.com/simonw/sqlite-utils/blob/413f8ed754e38d7b190de888c85fe8438336cb11/sqlite_utils/cli.py#L826,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094981339,Better error message if `--convert` code fails to return a dict, https://github.com/simonw/sqlite-utils/issues/363#issuecomment-1006343303,https://api.github.com/repos/simonw/sqlite-utils/issues/363,1006343303,IC_kwDOCGYnMM47-5SH,9599,simonw,2022-01-06T07:30:20Z,2022-01-06T07:30:20Z,OWNER,This check should run inside the `.insert_all()` method. It should raise a custom exception which the CLI code can then catch and turn into a click error.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094981339,Better error message if `--convert` code fails to return a dict, https://github.com/simonw/sqlite-utils/issues/356#issuecomment-1006318443,https://api.github.com/repos/simonw/sqlite-utils/issues/356,1006318443,IC_kwDOCGYnMM47-zNr,9599,simonw,2022-01-06T06:30:13Z,2022-01-06T06:30:13Z,OWNER,"Documentation: - https://sqlite-utils.datasette.io/en/latest/cli.html#inserting-unstructured-data-with-lines-and-text - https://sqlite-utils.datasette.io/en/latest/cli.html#applying-conversions-while-inserting-data","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option, https://github.com/simonw/sqlite-utils/issues/356#issuecomment-1006318007,https://api.github.com/repos/simonw/sqlite-utils/issues/356,1006318007,IC_kwDOCGYnMM47-zG3,9599,simonw,2022-01-06T06:28:53Z,2022-01-06T06:28:53Z,OWNER,Implemented in #361.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006315145,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006315145,IC_kwDOCGYnMM47-yaJ,9599,simonw,2022-01-06T06:20:51Z,2022-01-06T06:20:51Z,OWNER,This is all documented. I'm going to rebase-merge it to keep the individual commits.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006311742,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006311742,IC_kwDOCGYnMM47-xk-,9599,simonw,2022-01-06T06:12:19Z,2022-01-06T06:12:19Z,OWNER,"Got that working: ``` % echo 'This is cool' | sqlite-utils insert words.db words - --text --convert '({""word"": w} for w in text.split())' % sqlite-utils dump words.db BEGIN TRANSACTION; CREATE TABLE [words] ( [word] TEXT ); INSERT INTO ""words"" VALUES('This'); INSERT INTO ""words"" VALUES('is'); INSERT INTO ""words"" VALUES('cool'); COMMIT; ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006309834,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006309834,IC_kwDOCGYnMM47-xHK,9599,simonw,2022-01-06T06:08:01Z,2022-01-06T06:08:01Z,OWNER,"For `--text` the conversion function should be allowed to return an iterable instead of a dictionary, in which case it will be treated as the full list of records to be inserted.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006301546,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006301546,IC_kwDOCGYnMM47-vFq,9599,simonw,2022-01-06T05:44:47Z,2022-01-06T05:44:47Z,OWNER,Just need documentation for `--convert` now against the various different types of input.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006300280,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006300280,IC_kwDOCGYnMM47-ux4,9599,simonw,2022-01-06T05:40:45Z,2022-01-06T05:40:45Z,OWNER,"I'm going to rename `--all` to `--text`: > - Use `--text` to write the entire input to a column called ""text"" To avoid that clash with Python's `all()` function.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006299778,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006299778,IC_kwDOCGYnMM47-uqC,9599,simonw,2022-01-06T05:39:10Z,2022-01-06T05:39:10Z,OWNER,`all` is a bad variable name because it clashes with the Python `all()` built-in function.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006295276,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006295276,IC_kwDOCGYnMM47-tjs,9599,simonw,2022-01-06T05:26:11Z,2022-01-06T05:26:11Z,OWNER,"Here's the traceback if your `--convert` function doesn't return a dict right now: ``` % sqlite-utils insert /tmp/all.db blah /tmp/log.log --convert 'all.upper()' --all Traceback (most recent call last): File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/bin/sqlite-utils"", line 33, in sys.exit(load_entry_point('sqlite-utils', 'console_scripts', 'sqlite-utils')()) File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 1137, in __call__ return self.main(*args, **kwargs) File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 1062, in main rv = self.invoke(ctx) File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 1668, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 763, in invoke return __callback(*args, **kwargs) File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py"", line 949, in insert insert_upsert_implementation( File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py"", line 834, in insert_upsert_implementation db[table].insert_all( File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 2602, in insert_all first_record = next(records) File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 3044, in fix_square_braces for record in records: File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py"", line 831, in docs = (decode_base64_values(doc) for doc in docs) File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py"", line 86, in decode_base64_values to_fix = [ File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py"", line 89, in if isinstance(doc[k], dict) TypeError: string indices must be integers ``` I can live with that for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006294777,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006294777,IC_kwDOCGYnMM47-tb5,9599,simonw,2022-01-06T05:24:54Z,2022-01-06T05:24:54Z,OWNER,"> I added a custom error message for if the user's `--convert` code doesn't return a dict. That turned out to be a bad idea because it meant exhausting the iterator early for the check - before we got to the `.insert_all()` code that breaks the iterator up into chunks. I tried fixing that with `itertools.tee()` to run the generator twice but that's grossly memory-inefficient for large imports.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006288444,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006288444,IC_kwDOCGYnMM47-r48,9599,simonw,2022-01-06T05:07:10Z,2022-01-06T05:07:10Z,OWNER,"And here's a demo of `--convert` used with `--all` - I added a custom error message for if the user's `--convert` code doesn't return a dict. ``` % sqlite-utils insert /tmp/all.db blah /tmp/log.log --convert 'all.upper()' --all Error: Records returned by your --convert function must be dicts % sqlite-utils insert /tmp/all.db blah /tmp/log.log --convert '{""all"": all.upper()}' --all % sqlite-utils dump /tmp/all.db BEGIN TRANSACTION; CREATE TABLE [blah] ( [all] TEXT ); INSERT INTO ""blah"" VALUES('INFO: 127.0.0.1:60581 - ""GET / HTTP/1.1"" 200 OK INFO: 127.0.0.1:60581 - ""GET /FOO/-/STATIC/APP.CSS?CEAD5A HTTP/1.1"" 200 OK INFO: 127.0.0.1:60581 - ""GET /FAVICON.ICO HTTP/1.1"" 200 OK INFO: 127.0.0.1:60581 - ""GET /FOO/TIDDLYWIKI HTTP/1.1"" 200 OK INFO: 127.0.0.1:60581 - ""GET /FOO/-/STATIC/APP.CSS?CEAD5A HTTP/1.1"" 200 OK INFO: 127.0.0.1:60584 - ""GET /FOO/-/STATIC/SQL-FORMATTER-2.3.3.MIN.JS HTTP/1.1"" 200 OK INFO: 127.0.0.1:60586 - ""GET /FOO/-/STATIC/CODEMIRROR-5.57.0.MIN.JS HTTP/1.1"" 200 OK INFO: 127.0.0.1:60585 - ""GET /FOO/-/STATIC/CODEMIRROR-5.57.0.MIN.CSS HTTP/1.1"" 200 OK INFO: 127.0.0.1:60588 - ""GET /FOO/-/STATIC/CODEMIRROR-5.57.0-SQL.MIN.JS HTTP/1.1"" 200 OK INFO: 127.0.0.1:60587 - ""GET /FOO/-/STATIC/CM-RESIZE-1.0.1.MIN.JS HTTP/1.1"" 200 OK INFO: 127.0.0.1:60586 - ""GET /FOO/TIDDLYWIKI/TIDDLERS HTTP/1.1"" 200 OK INFO: 127.0.0.1:60586 - ""GET /FOO/-/STATIC/APP.CSS?CEAD5A HTTP/1.1"" 200 OK INFO: 127.0.0.1:60584 - ""GET /FOO/-/STATIC/TABLE.JS HTTP/1.1"" 200 OK '); COMMIT; ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006284673,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006284673,IC_kwDOCGYnMM47-q-B,9599,simonw,2022-01-06T04:55:52Z,2022-01-06T04:55:52Z,OWNER,"Test code that just worked for me: ``` sqlite-utils insert /tmp/blah.db blah /tmp/log.log --convert ' bits = line.split() return dict([(""b_{}"".format(i), bit) for i, bit in enumerate(bits)])' --lines ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006232013,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006232013,IC_kwDOCGYnMM47-eHN,9599,simonw,2022-01-06T02:21:35Z,2022-01-06T02:21:35Z,OWNER,"I'm having second thoughts about this bit: > Your Python code will be passed a ""row"" variable representing the imported row, and can return a modified row. > > If you are using `--lines` your code will be passed a ""line"" variable, and for `--all` an ""all"" variable. The code in question is this: https://github.com/simonw/sqlite-utils/blob/500a35ad4d91c8a6232134ce9406efec11bedff8/sqlite_utils/utils.py#L296-L303 Do I really want to add the complexity of supporting different variable names there? I think always using `value` might be better. Except... `value` made sense for the existing `sqlite-utils convert` command where you are running a conversion function against the value for the column in the current row - is it confusing if applied to lines or documents or `all`?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006230411,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006230411,IC_kwDOCGYnMM47-duL,9599,simonw,2022-01-06T02:17:35Z,2022-01-06T02:17:35Z,OWNER,"Documentation: https://github.com/simonw/sqlite-utils/blob/33223856ff7fe746b7b77750fbe5b218531d0545/docs/cli.rst#inserting-unstructured-data-with---lines-and---all - I went with a single section titled ""Inserting unstructured data with --lines and --all""","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006220129,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006220129,IC_kwDOCGYnMM47-bNh,9599,simonw,2022-01-06T01:52:26Z,2022-01-06T01:52:26Z,OWNER,I'm going to refactor all of the tests for `sqlite-utils insert` into a new `test_cli_insert.py` module.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006219848,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006219848,IC_kwDOCGYnMM47-bJI,9599,simonw,2022-01-06T01:51:36Z,2022-01-06T01:51:36Z,OWNER,"So far I've just implemented the new help: ``` % sqlite-utils insert --help Usage: sqlite-utils insert [OPTIONS] PATH TABLE FILE Insert records from FILE into a table, creating the table if it does not already exist. By default the input is expected to be a JSON array of objects. Or: - Use --nl for newline-delimited JSON objects - Use --csv or --tsv for comma-separated or tab-separated input - Use --lines to write each incoming line to a column called ""line"" - Use --all to write the entire input to a column called ""all"" You can also use --convert to pass a fragment of Python code that will be used to convert each input. Your Python code will be passed a ""row"" variable representing the imported row, and can return a modified row. If you are using --lines your code will be passed a ""line"" variable, and for --all an ""all"" variable. Options: --pk TEXT Columns to use as the primary key, e.g. id --flatten Flatten nested JSON objects, so {""a"": {""b"": 1}} becomes {""a_b"": 1} --nl Expect newline-delimited JSON -c, --csv Expect CSV input --tsv Expect TSV input --lines Treat each line as a single value called 'line' --all Treat input as a single value called 'all' --convert TEXT Python code to convert each item --import TEXT Python modules to import --delimiter TEXT Delimiter to use for CSV files --quotechar TEXT Quote character to use for CSV/TSV --sniff Detect delimiter and quote character --no-headers CSV file has no header row --batch-size INTEGER Commit every X records --alter Alter existing table to add any missing columns --not-null TEXT Columns that should be created as NOT NULL --default ... Default value that should be set for a column --encoding TEXT Character encoding for input, defaults to utf-8 -d, --detect-types Detect types for columns in CSV/TSV data --load-extension TEXT SQLite extensions to load --silent Do not show progress bar --ignore Ignore records if pk already exists --replace Replace records if pk already exists --truncate Truncate table before inserting records, if table already exists -h, --help Show this message and exit. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997496626,https://api.github.com/repos/simonw/sqlite-utils/issues/356,997496626,IC_kwDOCGYnMM47dJcy,9599,simonw,2021-12-20T00:38:15Z,2022-01-06T01:29:03Z,OWNER,"The implementation of this gets a tiny bit complicated. Ignoring `--convert`, the `--lines` option can internally produce `{""line"": ...}` records and the `--all` option can produce `{""all"": ...}` records. But... when `--convert` is used, what should the code run against? It could run against those already-converted records but that's a little bit strange, since you'd have to do this: sqlite-utils insert blah.db blah myfile.txt --all --convert '{""item"": s for s in value[""all""].split(""-"")}' Having to use `value[""all""]` there is unintuitive. It would be nicer to have a `all` variable to work against. But then for `--lines` should the local variable be called `line`? And how best to summarize these different names for local variables in the inline help for the feature?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option, https://github.com/simonw/sqlite-utils/issues/360#issuecomment-1006211113,https://api.github.com/repos/simonw/sqlite-utils/issues/360,1006211113,IC_kwDOCGYnMM47-ZAp,9599,simonw,2022-01-06T01:27:53Z,2022-01-06T01:27:53Z,OWNER,"It looks like you were using `sqlite-utils memory` - that works by loading the entire file into an in-memory database, so 170GB is very likely to run out of RAM. The line of code there exhibits another problem: it's reading the entire JSON file into a Python string, so it looks like it's going to run out of RAM even before it gets to the SQLite in-memory database section. To handle a file of this size you'd need to write it to a SQLite database on-disk first. The `sqlite-utils insert` command can do this, and it should be able to ""stream"" records in from a file without loading the entire thing into memory - but only for JSON-NL and CSV/TSV formats, not for JSON arrays. The code in question is here: https://github.com/simonw/sqlite-utils/blob/f3fd8613113d21d44238a6ec54b375f5aa72c4e0/sqlite_utils/cli.py#L738-L773 That's using Python generators for the CSV/TSV/JSON-NL variants... but it's doing this for regular JSON which requires reading the entire thing into memory: https://github.com/simonw/sqlite-utils/blob/f3fd8613113d21d44238a6ec54b375f5aa72c4e0/sqlite_utils/cli.py#L767 If you have the ability to control how your 170GB file is generated you may have more luck converting it to CSV or TSV or newline-delimited JSON, then using `sqlite-utils insert` to insert it into a database file. To be honest though I've never tested this tooling with anything nearly that big, so it's possible you'll still run into problems. If you do I'd love to hear about them! I would be tempted to tackle this size of job by writing a custom Python script, either using the `sqlite_utils` Python library or even calling `sqlite3` directly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1091819089,MemoryError, https://github.com/simonw/datasette/issues/1534#issuecomment-1005975080,https://api.github.com/repos/simonw/datasette/issues/1534,1005975080,IC_kwDOBm6k_c479fYo,9599,simonw,2022-01-05T18:29:06Z,2022-01-05T18:29:06Z,OWNER,"A really big downside to this is that it turns out many CDNs - apparently including Cloudflare - don't support the Vary header at all! More in this thread: https://twitter.com/simonw/status/1478470282931163137","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065432388,Maybe return JSON from HTML pages if `Accept: application/json` is sent, https://github.com/simonw/datasette/issues/1585#issuecomment-1003575286,https://api.github.com/repos/simonw/datasette/issues/1585,1003575286,IC_kwDOBm6k_c470Vf2,9599,simonw,2022-01-01T15:40:38Z,2022-01-01T15:40:38Z,OWNER,API tutorial: https://firebase.google.com/docs/hosting/api-deploy,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1091838742,Fire base caching for `publish cloudrun`, https://github.com/simonw/datasette/issues/1152#issuecomment-1001791592,https://api.github.com/repos/simonw/datasette/issues/1152,1001791592,IC_kwDOBm6k_c47tiBo,9599,simonw,2021-12-27T23:04:31Z,2021-12-27T23:04:31Z,OWNER,Another option: rethink permissions to always work in terms of where clauses users as part of a SQL query that returns the overall allowed set of databases or tables. This would require rethinking existing permissions but it might be worthwhile prior to 1.0.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770598024,Efficiently calculate list of databases/tables a user can view, https://github.com/simonw/datasette/issues/878#issuecomment-1001699559,https://api.github.com/repos/simonw/datasette/issues/878,1001699559,IC_kwDOBm6k_c47tLjn,9599,simonw,2021-12-27T18:53:04Z,2021-12-27T18:53:04Z,OWNER,"I'm going to see if I can come up with the simplest possible version of this pattern for the `/-/metadata` and `/-/metadata.json` page, then try it for the database query page, before tackling the much more complex table page.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/1576#issuecomment-1000935523,https://api.github.com/repos/simonw/datasette/issues/1576,1000935523,IC_kwDOBm6k_c47qRBj,9599,simonw,2021-12-24T21:33:05Z,2021-12-24T21:33:05Z,OWNER,"Another option would be to attempt to import `contextvars` and, if the import fails (for Python 3.6) continue using the current mechanism - then let Python 3.6 users know in the documentation that under Python 3.6 they will miss out on nested traces.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/1577#issuecomment-1000673444,https://api.github.com/repos/simonw/datasette/issues/1577,1000673444,IC_kwDOBm6k_c47pRCk,9599,simonw,2021-12-24T06:08:58Z,2021-12-24T06:08:58Z,OWNER,"https://pypistats.org/packages/datasette shows a breakdown of downloads by Python version: It looks like on a recent day I had 4,071 downloads from Python 3.7... and just 2 downloads from Python 3.6!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087913724,Drop support for Python 3.6, https://github.com/simonw/datasette/issues/1534#issuecomment-1000535904,https://api.github.com/repos/simonw/datasette/issues/1534,1000535904,IC_kwDOBm6k_c47ovdg,9599,simonw,2021-12-23T21:44:31Z,2021-12-23T21:44:31Z,OWNER,A big downside to this is that I would need to use `Vary: Accept` for when Datasette is running behind a cache such as Cloudflare - would that greatly reduce overall cache efficiency due to subtle variations in the accept headers sent by common browsers?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065432388,Maybe return JSON from HTML pages if `Accept: application/json` is sent, https://github.com/simonw/datasette/issues/1579#issuecomment-1000485719,https://api.github.com/repos/simonw/datasette/issues/1579,1000485719,IC_kwDOBm6k_c47ojNX,9599,simonw,2021-12-23T19:19:45Z,2021-12-23T19:19:45Z,OWNER,All of those removed `block=True` lines in 8c401ee0f054de2f568c3a8302c9223555146407 really help confirm to me that this was a good decision.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087931918,`.execute_write(... block=True)` should be the default behaviour, https://github.com/simonw/datasette/issues/1579#issuecomment-1000485505,https://api.github.com/repos/simonw/datasette/issues/1579,1000485505,IC_kwDOBm6k_c47ojKB,9599,simonw,2021-12-23T19:19:13Z,2021-12-23T19:19:13Z,OWNER,Updated docs for `execute_write_fn()`: https://github.com/simonw/datasette/blob/75153ea9b94d09ec3d61f7c6ebdf378e0c0c7a0b/docs/internals.rst#await-dbexecute_write_fnfn-blocktrue,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087931918,`.execute_write(... block=True)` should be the default behaviour, https://github.com/simonw/datasette/issues/1579#issuecomment-1000481686,https://api.github.com/repos/simonw/datasette/issues/1579,1000481686,IC_kwDOBm6k_c47oiOW,9599,simonw,2021-12-23T19:09:23Z,2021-12-23T19:09:23Z,OWNER,"Re-opening this because I missed updating some of the docs, and I also need to update Datasette's own code to not use `block=True` in a bunch of places.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087931918,`.execute_write(... block=True)` should be the default behaviour, https://github.com/simonw/datasette/issues/1579#issuecomment-1000479737,https://api.github.com/repos/simonw/datasette/issues/1579,1000479737,IC_kwDOBm6k_c47ohv5,9599,simonw,2021-12-23T19:04:23Z,2021-12-23T19:04:23Z,OWNER,Updated documentation: https://github.com/simonw/datasette/blob/00a2895cd2dc42c63846216b36b2dc9f41170129/docs/internals.rst#await-dbexecute_writesql-paramsnone-blocktrue,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087931918,`.execute_write(... block=True)` should be the default behaviour, https://github.com/simonw/datasette/issues/1579#issuecomment-1000477813,https://api.github.com/repos/simonw/datasette/issues/1579,1000477813,IC_kwDOBm6k_c47ohR1,9599,simonw,2021-12-23T18:59:41Z,2021-12-23T18:59:41Z,OWNER,"I'm going to go with `execute_write(..., block=False)` as the mechanism for fire-and-forget write queries.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087931918,`.execute_write(... block=True)` should be the default behaviour, https://github.com/simonw/datasette/issues/1579#issuecomment-1000477621,https://api.github.com/repos/simonw/datasette/issues/1579,1000477621,IC_kwDOBm6k_c47ohO1,9599,simonw,2021-12-23T18:59:12Z,2021-12-23T18:59:12Z,OWNER,"The easiest way to change this would be to default to `block=True` such that you need to pass `block=False` to the APIs to have them do fire-and-forget. An alternative would be to add new, separately named methods which do the fire-and-forget thing. If I hadn't recently added `execute_write_script` and `execute_write_many` in #1570 I'd be more into this idea, but I don't want to end up with eight methods - `execute_write`, `execute_write_queue`, `execute_write_many`, `execute_write_many_queue`, `execute_write_script`, `execute_write_scrript_queue`, `execute_write_fn`, `execute_write_fn_queue`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087931918,`.execute_write(... block=True)` should be the default behaviour, https://github.com/simonw/datasette/issues/1579#issuecomment-1000476413,https://api.github.com/repos/simonw/datasette/issues/1579,1000476413,IC_kwDOBm6k_c47og79,9599,simonw,2021-12-23T18:56:06Z,2021-12-23T18:56:06Z,OWNER,"This is technically a breaking change, but a GitHub code search at https://cs.github.com/?scopeName=All+repos&scope=&q=execute_write%20datasette%20-owner%3Asimonw shows only one repo not-owned-by-me using this, and they're using `block=True`: https://github.com/mfa/datasette-webhook-write/blob/e82440f372a2f2e3ed27d1bd34c9fa3a53b49b94/datasette_webhook_write/__init__.py#L88-L89","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087931918,`.execute_write(... block=True)` should be the default behaviour, https://github.com/simonw/datasette/issues/1578#issuecomment-1000471782,https://api.github.com/repos/simonw/datasette/issues/1578,1000471782,IC_kwDOBm6k_c47ofzm,9599,simonw,2021-12-23T18:44:01Z,2021-12-23T18:44:01Z,OWNER,"The example nginx config on https://docs.datasette.io/en/stable/deploying.html#nginx-proxy-configuration is currently: ``` daemon off; events { worker_connections 1024; } http { server { listen 80; location /my-datasette { proxy_pass http://127.0.0.1:8009/my-datasette; proxy_set_header Host $host; } } } ``` This looks to me like it might exhibit the bug. Need to confirm that and figure out an alternative.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087919372,Confirm if documented nginx proxy config works for row pages with escaped characters in their primary key, https://github.com/simonw/datasette/issues/1578#issuecomment-1000471371,https://api.github.com/repos/simonw/datasette/issues/1578,1000471371,IC_kwDOBm6k_c47oftL,9599,simonw,2021-12-23T18:42:50Z,2021-12-23T18:42:50Z,OWNER,"Confirmed, that fixed the bug for me on my server.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087919372,Confirm if documented nginx proxy config works for row pages with escaped characters in their primary key, https://github.com/simonw/datasette/issues/1578#issuecomment-1000470652,https://api.github.com/repos/simonw/datasette/issues/1578,1000470652,IC_kwDOBm6k_c47ofh8,9599,simonw,2021-12-23T18:40:46Z,2021-12-23T18:40:46Z,OWNER,"[This StackOverflow answer](https://serverfault.com/a/463932) suggests that the fix is to change this: proxy_pass http://127.0.0.1:8000/; To this: proxy_pass http://127.0.0.1:8000; Quoting the nginx documentation: http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass > A request URI is passed to the server as follows: > > - If the `proxy_pass` directive is specified with a URI, then when a request is passed to the server, the part of a [normalized](http://nginx.org/en/docs/http/ngx_http_core_module.html#location) request URI matching the location is replaced by a URI specified in the directive: > > location /name/ { > proxy_pass http://127.0.0.1/remote/; > } > > - If `proxy_pass` is specified without a URI, the request URI is passed to the server in the same form as sent by a client when the original request is processed, or the full normalized request URI is passed when processing the changed URI: > > location /some/path/ { > proxy_pass http://127.0.0.1; > }","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087919372,Confirm if documented nginx proxy config works for row pages with escaped characters in their primary key, https://github.com/simonw/datasette/issues/1578#issuecomment-1000469107,https://api.github.com/repos/simonw/datasette/issues/1578,1000469107,IC_kwDOBm6k_c47ofJz,9599,simonw,2021-12-23T18:36:38Z,2021-12-23T18:36:38Z,OWNER,"This problem doesn't occur on my `localhost` running Uvicorn directly - but I'm seeing it in my production environment that runs Datasette behind an nginx proxy: ``` location / { proxy_pass http://127.0.0.1:8000/; proxy_set_header Host $host; } ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087919372,Confirm if documented nginx proxy config works for row pages with escaped characters in their primary key, https://github.com/simonw/datasette/issues/1577#issuecomment-1000462309,https://api.github.com/repos/simonw/datasette/issues/1577,1000462309,IC_kwDOBm6k_c47odfl,9599,simonw,2021-12-23T18:20:46Z,2021-12-23T18:20:46Z,OWNER,There are a lot of improvements to `asyncio` in 3.7: https://docs.python.org/3/whatsnew/3.7.html#whatsnew37-asyncio,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087913724,Drop support for Python 3.6, https://github.com/simonw/datasette/issues/1577#issuecomment-1000461900,https://api.github.com/repos/simonw/datasette/issues/1577,1000461900,IC_kwDOBm6k_c47odZM,9599,simonw,2021-12-23T18:19:44Z,2021-12-23T18:19:44Z,OWNER,"The 3.7 feature I want to use today is [contextvars](https://docs.python.org/3/library/contextvars.html) - but I have a workaround for the moment, see https://github.com/simonw/datasette/issues/1576#issuecomment-999987418 So I'm going to hold off on dropping 3.6 for a little bit longer. I imagine I'll drop it before Datasette 1.0 though. Leaving this issue open to gather thoughts and feedback on this issue from Datasette users and potential users.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087913724,Drop support for Python 3.6, https://github.com/simonw/datasette/issues/1577#issuecomment-1000461275,https://api.github.com/repos/simonw/datasette/issues/1577,1000461275,IC_kwDOBm6k_c47odPb,9599,simonw,2021-12-23T18:18:11Z,2021-12-23T18:18:11Z,OWNER,"From the Twitter thread, there are still a decent amount of LTS Linux releases out there that are stuck on pre-3.7 Python. Though many of those are 3.5 and Datasette dropped support for 3.5 in November 2019: cf7776d36fbacefa874cbd6e5fcdc9fff7661203","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087913724,Drop support for Python 3.6, https://github.com/simonw/datasette/issues/1576#issuecomment-999990414,https://api.github.com/repos/simonw/datasette/issues/1576,999990414,IC_kwDOBm6k_c47mqSO,9599,simonw,2021-12-23T02:08:39Z,2021-12-23T18:16:35Z,OWNER,"It's tiny: I'm tempted to vendor it. https://github.com/Skyscanner/aiotask-context/blob/master/aiotask_context/__init__.py No, I'll add it as a pinned dependency, which I can then drop when I drop 3.6 support.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/1576#issuecomment-999987418,https://api.github.com/repos/simonw/datasette/issues/1576,999987418,IC_kwDOBm6k_c47mpja,9599,simonw,2021-12-23T01:59:58Z,2021-12-23T02:02:12Z,OWNER,"Another option: https://github.com/Skyscanner/aiotask-context - looks like it might be better as it's been updated for Python 3.7 in this commit https://github.com/Skyscanner/aiotask-context/commit/67108c91d2abb445655cc2af446fdb52ca7890c4 The Skyscanner one doesn't attempt to wrap any existing factories, but that's OK for my purposes since I don't need to handle arbitrary `asyncio` code written by other people.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/1576#issuecomment-999876666,https://api.github.com/repos/simonw/datasette/issues/1576,999876666,IC_kwDOBm6k_c47mOg6,9599,simonw,2021-12-22T20:59:22Z,2021-12-22T21:18:09Z,OWNER,"This article is relevant: [Context information storage for asyncio](https://blog.sqreen.com/asyncio/) - in particular the section https://blog.sqreen.com/asyncio/#context-inheritance-between-tasks which describes exactly the problem I have and their solution, which involves this trickery: ```python def request_task_factory(loop, coro): child_task = asyncio.tasks.Task(coro, loop=loop) parent_task = asyncio.Task.current_task(loop=loop) current_request = getattr(parent_task, 'current_request', None) setattr(child_task, 'current_request', current_request) return child_task loop = asyncio.get_event_loop() loop.set_task_factory(request_task_factory) ``` They released their solution as a library: https://pypi.org/project/aiocontext/ and https://github.com/sqreen/AioContext - but that company was acquired by Datadog back in April and doesn't seem to be actively maintaining their open source stuff any more: https://twitter.com/SqreenIO/status/1384906075506364417","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/1576#issuecomment-999878907,https://api.github.com/repos/simonw/datasette/issues/1576,999878907,IC_kwDOBm6k_c47mPD7,9599,simonw,2021-12-22T21:03:49Z,2021-12-22T21:10:46Z,OWNER,"`context_vars` can solve this but they were introduced in Python 3.7: https://www.python.org/dev/peps/pep-0567/ Python 3.6 support ends in a few days time, and it looks like Glitch has updated to 3.7 now - so maybe I can get away with Datasette needing 3.7 these days? Tweeted about that here: https://twitter.com/simonw/status/1473761478155010048","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/1576#issuecomment-999874886,https://api.github.com/repos/simonw/datasette/issues/1576,999874886,IC_kwDOBm6k_c47mOFG,9599,simonw,2021-12-22T20:55:42Z,2021-12-22T20:57:28Z,OWNER,"One way to solve this would be to introduce a `set_task_id()` method, which sets an ID which will be returned by `get_task_id()` instead of using `id(current_task(loop=loop))`. It would be really nice if I could solve this using `with` syntax somehow. Something like: ```python with trace_child_tasks(): ( suggested_facets, (facet_results, facets_timed_out), ) = await asyncio.gather( execute_suggested_facets(), execute_facets(), ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/1576#issuecomment-999874484,https://api.github.com/repos/simonw/datasette/issues/1576,999874484,IC_kwDOBm6k_c47mN-0,9599,simonw,2021-12-22T20:54:52Z,2021-12-22T20:54:52Z,OWNER,"Here's the full current relevant code from `tracer.py`: https://github.com/simonw/datasette/blob/ace86566b28280091b3844cf5fbecd20158e9004/datasette/tracer.py#L8-L64 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/1518#issuecomment-999870993,https://api.github.com/repos/simonw/datasette/issues/1518,999870993,IC_kwDOBm6k_c47mNIR,9599,simonw,2021-12-22T20:47:18Z,2021-12-22T20:50:24Z,OWNER,"The reason they aren't showing up in the traces is that traces are stored just for the currently executing `asyncio` task ID: https://github.com/simonw/datasette/blob/ace86566b28280091b3844cf5fbecd20158e9004/datasette/tracer.py#L13-L25 This is so traces for other incoming requests don't end up mixed together. But there's no current mechanism to track async tasks that are effectively ""child tasks"" of the current request, and hence should be tracked the same. https://stackoverflow.com/a/69349501/6083 suggests that you pass the task ID as an argument to the child tasks that are executed using `asyncio.gather()` to work around this kind of problem.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-999870282,https://api.github.com/repos/simonw/datasette/issues/1518,999870282,IC_kwDOBm6k_c47mM9K,9599,simonw,2021-12-22T20:45:56Z,2021-12-22T20:46:08Z,OWNER,"> New short-term goal: get facets and suggested facets to execute in parallel with the main query. Generate a trace graph that proves that is happening using `datasette-pretty-traces`. I wrote code to execute those in parallel using `asyncio.gather()` - which seems to work but causes the SQL run inside the parallel `async def` functions not to show up in the trace graph at all. ```diff diff --git a/datasette/views/table.py b/datasette/views/table.py index 9808fd2..ec9db64 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -1,3 +1,4 @@ +import asyncio import urllib import itertools import json @@ -615,44 +616,37 @@ class TableView(RowTableShared): if request.args.get(""_timelimit""): extra_args[""custom_time_limit""] = int(request.args.get(""_timelimit"")) - # Execute the main query! - results = await db.execute(sql, params, truncate=True, **extra_args) - - # Calculate the total count for this query - filtered_table_rows_count = None - if ( - not db.is_mutable - and self.ds.inspect_data - and count_sql == f""select count(*) from {table} "" - ): - # We can use a previously cached table row count - try: - filtered_table_rows_count = self.ds.inspect_data[database][""tables""][ - table - ][""count""] - except KeyError: - pass - - # Otherwise run a select count(*) ... - if count_sql and filtered_table_rows_count is None and not nocount: - try: - count_rows = list(await db.execute(count_sql, from_sql_params)) - filtered_table_rows_count = count_rows[0][0] - except QueryInterrupted: - pass - - # Faceting - if not self.ds.setting(""allow_facet"") and any( - arg.startswith(""_facet"") for arg in request.args - ): - raise BadRequest(""_facet= is not allowed"") + async def execute_count(): + # Calculate the total count for this query + filtered_table_rows_count = None + if ( + not db.is_mutable + and self.ds.inspect_data + and count_sql == f""select count(*) from {table} "" + ): + # We can use a previously cached table row count + try: + filtered_table_rows_count = self.ds.inspect_data[database][ + ""tables"" + ][table][""count""] + except KeyError: + pass + + if count_sql and filtered_table_rows_count is None and not nocount: + try: + count_rows = list(await db.execute(count_sql, from_sql_params)) + filtered_table_rows_count = count_rows[0][0] + except QueryInterrupted: + pass + + return filtered_table_rows_count + + filtered_table_rows_count = await execute_count() # pylint: disable=no-member facet_classes = list( itertools.chain.from_iterable(pm.hook.register_facet_classes()) ) - facet_results = {} - facets_timed_out = [] facet_instances = [] for klass in facet_classes: facet_instances.append( @@ -668,33 +662,58 @@ class TableView(RowTableShared): ) ) - if not nofacet: - for facet in facet_instances: - ( - instance_facet_results, - instance_facets_timed_out, - ) = await facet.facet_results() - for facet_info in instance_facet_results: - base_key = facet_info[""name""] - key = base_key - i = 1 - while key in facet_results: - i += 1 - key = f""{base_key}_{i}"" - facet_results[key] = facet_info - facets_timed_out.extend(instance_facets_timed_out) - - # Calculate suggested facets - suggested_facets = [] - if ( - self.ds.setting(""suggest_facets"") - and self.ds.setting(""allow_facet"") - and not _next - and not nofacet - and not nosuggest - ): - for facet in facet_instances: - suggested_facets.extend(await facet.suggest()) + async def execute_suggested_facets(): + # Calculate suggested facets + suggested_facets = [] + if ( + self.ds.setting(""suggest_facets"") + and self.ds.setting(""allow_facet"") + and not _next + and not nofacet + and not nosuggest + ): + for facet in facet_instances: + suggested_facets.extend(await facet.suggest()) + return suggested_facets + + async def execute_facets(): + facet_results = {} + facets_timed_out = [] + if not self.ds.setting(""allow_facet"") and any( + arg.startswith(""_facet"") for arg in request.args + ): + raise BadRequest(""_facet= is not allowed"") + + if not nofacet: + for facet in facet_instances: + ( + instance_facet_results, + instance_facets_timed_out, + ) = await facet.facet_results() + for facet_info in instance_facet_results: + base_key = facet_info[""name""] + key = base_key + i = 1 + while key in facet_results: + i += 1 + key = f""{base_key}_{i}"" + facet_results[key] = facet_info + facets_timed_out.extend(instance_facets_timed_out) + + return facet_results, facets_timed_out + + # Execute the main query, facets and facet suggestions in parallel: + ( + results, + suggested_facets, + (facet_results, facets_timed_out), + ) = await asyncio.gather( + db.execute(sql, params, truncate=True, **extra_args), + execute_suggested_facets(), + execute_facets(), + ) + + results = await db.execute(sql, params, truncate=True, **extra_args) # Figure out columns and rows for the query columns = [r[0] for r in results.description] ``` Here's the trace for `http://127.0.0.1:4422/fixtures/compound_three_primary_keys?_trace=1&_facet=pk1&_facet=pk2` with the missing facet and facet suggestion queries: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-999863269,https://api.github.com/repos/simonw/datasette/issues/1518,999863269,IC_kwDOBm6k_c47mLPl,9599,simonw,2021-12-22T20:35:41Z,2021-12-22T20:37:13Z,OWNER,"It looks like the count has to be executed before facets can be, because the facet_class constructor needs that total count figure: https://github.com/simonw/datasette/blob/6b1384b2f529134998fb507e63307609a5b7f5c0/datasette/views/table.py#L660-L671 It's used in facet suggestion logic here: https://github.com/simonw/datasette/blob/ace86566b28280091b3844cf5fbecd20158e9004/datasette/facets.py#L172-L178","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-999850191,https://api.github.com/repos/simonw/datasette/issues/1518,999850191,IC_kwDOBm6k_c47mIDP,9599,simonw,2021-12-22T20:29:38Z,2021-12-22T20:29:38Z,OWNER,New short-term goal: get facets and suggested facets to execute in parallel with the main query. Generate a trace graph that proves that is happening using `datasette-pretty-traces`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-999837569,https://api.github.com/repos/simonw/datasette/issues/1518,999837569,IC_kwDOBm6k_c47mE-B,9599,simonw,2021-12-22T20:15:45Z,2021-12-22T20:15:45Z,OWNER,"Also the whole `special_args` v.s. `request.args` thing is pretty confusing, I think that might be an older code pattern back from when I was using Sanic.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-999837220,https://api.github.com/repos/simonw/datasette/issues/1518,999837220,IC_kwDOBm6k_c47mE4k,9599,simonw,2021-12-22T20:15:04Z,2021-12-22T20:15:04Z,OWNER,"I think I can move this much higher up in the method, it's a bit confusing having it half way through: https://github.com/simonw/datasette/blob/6b1384b2f529134998fb507e63307609a5b7f5c0/datasette/views/table.py#L414-L436","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-999831967,https://api.github.com/repos/simonw/datasette/issues/1518,999831967,IC_kwDOBm6k_c47mDmf,9599,simonw,2021-12-22T20:04:47Z,2021-12-22T20:10:11Z,OWNER,"I think I might be able to clean up a lot of the stuff in here using the `render_cell` plugin hook: https://github.com/simonw/datasette/blob/6b1384b2f529134998fb507e63307609a5b7f5c0/datasette/views/table.py#L87-L89 The catch with that hook - https://docs.datasette.io/en/stable/plugin_hooks.html#render-cell-value-column-table-database-datasette - is that it gets called for every single cell. I don't want the overhead of looking up the foreign key relationships etc once for every value in a specific column. But maybe I could extend the hook to include a shared cache that gets used for all of the cells in a specific table? Something like this: ```python render_cell(value, column, table, database, datasette, cache) ``` `cache` is a dictionary - and the same dictionary is passed to every call to that hook while rendering a specific page. It's a bit of a gross hack though, and would it ever be useful for plugins outside of the default plugin in Datasette which does the foreign key stuff? If I can think of one other potential application for this `cache` then I might implement it. No, this optimization doesn't make sense: the most complex cell enrichment logic is the stuff that does a `select * from categories where id in (2, 5, 6)` query, using just the distinct set of IDs that are rendered on the current page. That's not going to fit in the `render_cell` hook no matter how hard I try to warp it into the right shape, because it needs full visibility of all of the results that are being rendered in order to collect those unique ID values.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/pull/1554#issuecomment-998354538,https://api.github.com/repos/simonw/datasette/issues/1554,998354538,IC_kwDOBm6k_c47ga5q,9599,simonw,2021-12-20T23:52:04Z,2021-12-20T23:52:04Z,OWNER,Abandoning this since it didn't work how I wanted.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079129258,TableView refactor, https://github.com/simonw/datasette/issues/1547#issuecomment-997514220,https://api.github.com/repos/simonw/datasette/issues/1547,997514220,IC_kwDOBm6k_c47dNvs,9599,simonw,2021-12-20T01:26:25Z,2021-12-20T01:26:25Z,OWNER,"OK, this should hopefully fix that for you: pip install https://github.com/simonw/datasette/archive/f36e010b3b69ada104b79d83c7685caf9359049e.zip","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1076388044,Writable canned queries fail to load custom templates, https://github.com/simonw/datasette/issues/1547#issuecomment-997513369,https://api.github.com/repos/simonw/datasette/issues/1547,997513369,IC_kwDOBm6k_c47dNiZ,9599,simonw,2021-12-20T01:24:43Z,2021-12-20T01:24:43Z,OWNER,"@wragge thanks, that's a bug! Working on that in #1575.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1076388044,Writable canned queries fail to load custom templates, https://github.com/simonw/datasette/issues/1575#issuecomment-997513177,https://api.github.com/repos/simonw/datasette/issues/1575,997513177,IC_kwDOBm6k_c47dNfZ,9599,simonw,2021-12-20T01:24:25Z,2021-12-20T01:24:25Z,OWNER,Looks like `specname` is new in Pluggy 1.0: https://github.com/pytest-dev/pluggy/blob/main/CHANGELOG.rst#pluggy-100-2021-08-25,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1084257842,__call__() got an unexpected keyword argument 'specname', https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997507074,https://api.github.com/repos/simonw/sqlite-utils/issues/356,997507074,IC_kwDOCGYnMM47dMAC,9599,simonw,2021-12-20T01:10:06Z,2021-12-20T01:16:11Z,OWNER,"Work-in-progress improved help: ``` Usage: sqlite-utils insert [OPTIONS] PATH TABLE FILE Insert records from FILE into a table, creating the table if it does not already exist. By default the input is expected to be a JSON array of objects. Or: - Use --nl for newline-delimited JSON objects - Use --csv or --tsv for comma-separated or tab-separated input - Use --lines to write each incoming line to a column called ""line"" - Use --all to write the entire input to a column called ""all"" You can also use --convert to pass a fragment of Python code that will be used to convert each input. Your Python code will be passed a ""row"" variable representing the imported row, and can return a modified row. If you are using --lines your code will be passed a ""line"" variable, and for --all an ""all"" variable. Options: --pk TEXT Columns to use as the primary key, e.g. id --flatten Flatten nested JSON objects, so {""a"": {""b"": 1}} becomes {""a_b"": 1} --nl Expect newline-delimited JSON -c, --csv Expect CSV input --tsv Expect TSV input --lines Treat each line as a single value called 'line' --all Treat input as a single value called 'all' --convert TEXT Python code to convert each item --import TEXT Python modules to import --delimiter TEXT Delimiter to use for CSV files --quotechar TEXT Quote character to use for CSV/TSV --sniff Detect delimiter and quote character --no-headers CSV file has no header row --batch-size INTEGER Commit every X records --alter Alter existing table to add any missing columns --not-null TEXT Columns that should be created as NOT NULL --default ... Default value that should be set for a column --encoding TEXT Character encoding for input, defaults to utf-8 -d, --detect-types Detect types for columns in CSV/TSV data --load-extension TEXT SQLite extensions to load --silent Do not show progress bar --ignore Ignore records if pk already exists --replace Replace records if pk already exists --truncate Truncate table before inserting records, if table already exists -h, --help Show this message and exit. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option, https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997508728,https://api.github.com/repos/simonw/sqlite-utils/issues/356,997508728,IC_kwDOCGYnMM47dMZ4,9599,simonw,2021-12-20T01:14:43Z,2021-12-20T01:14:43Z,OWNER,(This makes me want `--extract` from #352 even more.),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option, https://github.com/simonw/sqlite-utils/issues/163#issuecomment-997502242,https://api.github.com/repos/simonw/sqlite-utils/issues/163,997502242,IC_kwDOCGYnMM47dK0i,9599,simonw,2021-12-20T00:56:45Z,2021-12-20T00:56:52Z,OWNER,"> Maybe `sqlite-utils` should absorb all of the functionality from `sqlite-transform` - having two separate tools doesn't necessarily make sense. I implemented that in: - #251","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706001517,Idea: conversions= could take Python functions, https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997497262,https://api.github.com/repos/simonw/sqlite-utils/issues/356,997497262,IC_kwDOCGYnMM47dJmu,9599,simonw,2021-12-20T00:40:15Z,2021-12-20T00:40:15Z,OWNER,`--flatten` could do with a better description too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option, https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997496931,https://api.github.com/repos/simonw/sqlite-utils/issues/356,997496931,IC_kwDOCGYnMM47dJhj,9599,simonw,2021-12-20T00:39:14Z,2021-12-20T00:39:52Z,OWNER,"``` % sqlite-utils insert --help Usage: sqlite-utils insert [OPTIONS] PATH TABLE JSON_FILE Insert records from JSON file into a table, creating the table if it does not already exist. Input should be a JSON array of objects, unless --nl or --csv is used. Options: --pk TEXT Columns to use as the primary key, e.g. id --nl Expect newline-delimited JSON --flatten Flatten nested JSON objects -c, --csv Expect CSV --tsv Expect TSV --convert TEXT Python code to convert each item --import TEXT Python modules to import --delimiter TEXT Delimiter to use for CSV files --quotechar TEXT Quote character to use for CSV/TSV --sniff Detect delimiter and quote character --no-headers CSV file has no header row --batch-size INTEGER Commit every X records --alter Alter existing table to add any missing columns --not-null TEXT Columns that should be created as NOT NULL --default ... Default value that should be set for a column --encoding TEXT Character encoding for input, defaults to utf-8 -d, --detect-types Detect types for columns in CSV/TSV data --load-extension TEXT SQLite extensions to load --silent Do not show progress bar --ignore Ignore records if pk already exists --replace Replace records if pk already exists --truncate Truncate table before inserting records, if table already exists -h, --help Show this message and exit. ``` I can add a bunch of extra help at the top there to explain all of this stuff. That ""Input should be a JSON array of objects"" bit could be expanded to several paragraphs.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option, https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997492872,https://api.github.com/repos/simonw/sqlite-utils/issues/356,997492872,IC_kwDOCGYnMM47dIiI,9599,simonw,2021-12-20T00:23:31Z,2021-12-20T00:23:31Z,OWNER,"I think this should work on JSON, or CSV, or individual lines, or the entire content at once. So I'll require `--lines --convert ...` to import individual lines, or `--all --convert` to run the conversion against the entire input at once. What would `--lines` or `--all` do without `--convert`? Maybe insert records as `{""line"": ""line of text""}` or `{""all"": ""whole input}`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option, https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997486156,https://api.github.com/repos/simonw/sqlite-utils/issues/356,997486156,IC_kwDOCGYnMM47dG5M,9599,simonw,2021-12-19T23:51:02Z,2021-12-19T23:51:02Z,OWNER,This is going to need a `--import` multi option too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option, https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997485361,https://api.github.com/repos/simonw/sqlite-utils/issues/356,997485361,IC_kwDOCGYnMM47dGsx,9599,simonw,2021-12-19T23:45:30Z,2021-12-19T23:45:30Z,OWNER,Really interesting example input for this: https://blog.timac.org/2021/1219-state-of-swift-and-swiftui-ios15/iOS13.txt - see https://blog.timac.org/2021/1219-state-of-swift-and-swiftui-ios15/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option, https://github.com/simonw/datasette/issues/1565#issuecomment-997474022,https://api.github.com/repos/simonw/datasette/issues/1565,997474022,IC_kwDOBm6k_c47dD7m,9599,simonw,2021-12-19T22:36:49Z,2021-12-19T22:37:29Z,OWNER,"No way with a tagged template literal to pass an extra database name argument, so instead I need a method that returns a callable that can be used for the tagged template literal for a specific database - or the default database. This could work (bit weird looking though): ```javascript var rows = await datasette.query(""fixtures"")`select * from foo`; ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins, https://github.com/simonw/datasette/issues/1565#issuecomment-997473856,https://api.github.com/repos/simonw/datasette/issues/1565,997473856,IC_kwDOBm6k_c47dD5A,9599,simonw,2021-12-19T22:35:20Z,2021-12-19T22:35:20Z,OWNER,"Quick prototype of that tagged template `query` function: ```javascript function query(pieces, ...parameters) { var qs = new URLSearchParams(); var sql = pieces[0]; parameters.forEach((param, i) => { sql += `:p${i}${pieces[i + 1]}`; qs.append(`p${i}`, param); }); qs.append(""sql"", sql); return qs.toString(); } var id = 4; console.log(query`select * from ids where id > ${id}`); ``` Outputs: ``` p0=4&sql=select+*+from+ids+where+id+%3E+%3Ap0 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins, https://github.com/simonw/datasette/issues/1565#issuecomment-997472639,https://api.github.com/repos/simonw/datasette/issues/1565,997472639,IC_kwDOBm6k_c47dDl_,9599,simonw,2021-12-19T22:25:50Z,2021-12-19T22:25:50Z,OWNER,"Or... ```javascript rows = await datasette.query`select * from searchable where id > ${id}`; ``` And it knows how to turn that into a parameterized call using tagged template literals.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins, https://github.com/simonw/datasette/issues/1565#issuecomment-997472509,https://api.github.com/repos/simonw/datasette/issues/1565,997472509,IC_kwDOBm6k_c47dDj9,9599,simonw,2021-12-19T22:24:50Z,2021-12-19T22:24:50Z,OWNER,"... huh, it could even expose a JavaScript function that can be called to execute a SQL query. ```javascript datasette.query(""select * from blah"").then(...) ``` Maybe it takes an optional second argument that specifies the database - defaulting to the one for the current page.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins, https://github.com/simonw/datasette/issues/1565#issuecomment-997472370,https://api.github.com/repos/simonw/datasette/issues/1565,997472370,IC_kwDOBm6k_c47dDhy,9599,simonw,2021-12-19T22:23:36Z,2021-12-19T22:23:36Z,OWNER,This should also expose the JSON API endpoints used to execute SQL against this database.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins, https://github.com/simonw/datasette/issues/1518#issuecomment-997472214,https://api.github.com/repos/simonw/datasette/issues/1518,997472214,IC_kwDOBm6k_c47dDfW,9599,simonw,2021-12-19T22:22:08Z,2021-12-19T22:22:08Z,OWNER,"I sketched out a chained SQL builder pattern that might be useful for further tidying up this code - though with the new plugin hook I'm less excited about it than I was: ```python class TableQuery: def __init__(self, table, columns, pks, is_view=False, prev=None): self.table = table self.columns = columns self.pks = pks self.is_view = is_view self.prev = prev # These can be changed for different instances in the chain: self._where_clauses = None self._order_by = None self._page_size = None self._offset = None self._select_columns = None self.select_all_columns = '*' self.select_specified_columns = '*' @property def where_clauses(self): wheres = [] current = self while current: if current._where_clauses is not None: wheres.extend(current._where_clauses) current = current.prev return list(reversed(wheres)) def where(self, where): new_cls = TableQuery(self.table, self.columns, self.pks, self.is_view, self) new_cls._where_clauses = [where] return new_cls @classmethod async def introspect(cls, db, table): return cls( table, columns = await db.table_columns(table), pks = await db.primary_keys(table), is_view = bool(await db.get_view_definition(table)) ) @property def sql_from(self): return f""from {self.table}{self.sql_where}"" @property def sql_where(self): if not self.where_clauses: return """" else: return f"" where {' and '.join(self.where_clauses)}"" @property def sql_no_order_no_limit(self): return f""select {self.select_all_columns} from {self.table}{self.sql_where}"" @property def sql(self): return f""select {self.select_specified_columns} from {self.table} {self.sql_where}{self._order_by} limit {self._page_size}{self._offset}"" @property def sql_count(self): return f""select count(*) {self.sql_from}"" def __repr__(self): return f"""" ``` Usage: ```python from datasette.app import Datasette ds = Datasette(memory=True, files=[""/Users/simon/Dropbox/Development/datasette/fixtures.db""]) db = ds.get_database(""fixtures"") query = await TableQuery.introspect(db, ""facetable"") print(query.where(""foo = bar"").where(""baz = 1"").sql_count) # 'select count(*) from facetable where foo = bar and baz = 1' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1547#issuecomment-997471672,https://api.github.com/repos/simonw/datasette/issues/1547,997471672,IC_kwDOBm6k_c47dDW4,9599,simonw,2021-12-19T22:18:26Z,2021-12-19T22:18:26Z,OWNER,"I released this [in an alpha](https://github.com/simonw/datasette/releases/tag/0.60a1), so you can try out this fix using: pip install datasette==0.60a1","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1076388044,Writable canned queries fail to load custom templates, https://github.com/simonw/datasette/issues/1566#issuecomment-997470633,https://api.github.com/repos/simonw/datasette/issues/1566,997470633,IC_kwDOBm6k_c47dDGp,9599,simonw,2021-12-19T22:12:00Z,2021-12-19T22:12:00Z,OWNER,"Released another alpha, 0.60a1: https://github.com/simonw/datasette/releases/tag/0.60a1","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083669410,Release Datasette 0.60, https://github.com/simonw/datasette/issues/1545#issuecomment-997462604,https://api.github.com/repos/simonw/datasette/issues/1545,997462604,IC_kwDOBm6k_c47dBJM,9599,simonw,2021-12-19T21:17:08Z,2021-12-19T21:17:08Z,OWNER,"Here's the relevant code: https://github.com/simonw/datasette/blob/4094741c2881c2ada3f3f878b532fdaec7914953/datasette/app.py#L1204-L1219 It's using `route_path.split(""/"")` which should be OK because that's the incoming `request.path` path - which I would expect to use `/` even on Windows. Then it uses `os.path.join` which should do the right thing. I need to get myself a proper Windows development environment setup to investigate this one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1075893249,Custom pages don't work on windows, https://github.com/simonw/datasette/issues/1573#issuecomment-997462117,https://api.github.com/repos/simonw/datasette/issues/1573,997462117,IC_kwDOBm6k_c47dBBl,9599,simonw,2021-12-19T21:13:13Z,2021-12-19T21:13:13Z,OWNER,This might also be the impetus I need to bring the https://datasette.io/plugins/datasette-pretty-traces plugin into Datasette core itself.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1084185188,Make trace() a documented internal API, https://github.com/simonw/datasette/issues/1547#issuecomment-997460731,https://api.github.com/repos/simonw/datasette/issues/1547,997460731,IC_kwDOBm6k_c47dAr7,9599,simonw,2021-12-19T21:02:15Z,2021-12-19T21:02:15Z,OWNER,"Yes, this is a bug. It looks like the problem is with the `if write:` branch in this code here: https://github.com/simonw/datasette/blob/5fac26aa221a111d7633f2dd92014641f7c0ade9/datasette/views/database.py#L252-L327 Is missing this bit of code: https://github.com/simonw/datasette/blob/5fac26aa221a111d7633f2dd92014641f7c0ade9/datasette/views/database.py#L343-L347","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1076388044,Writable canned queries fail to load custom templates, https://github.com/simonw/datasette/issues/1570#issuecomment-997460061,https://api.github.com/repos/simonw/datasette/issues/1570,997460061,IC_kwDOBm6k_c47dAhd,9599,simonw,2021-12-19T20:56:54Z,2021-12-19T20:56:54Z,OWNER,Documentation: https://docs.datasette.io/en/latest/internals.html#await-db-execute-write-sql-params-none-block-false,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083921371,Separate db.execute_write() into three methods, https://github.com/simonw/datasette/issues/1555#issuecomment-997459958,https://api.github.com/repos/simonw/datasette/issues/1555,997459958,IC_kwDOBm6k_c47dAf2,9599,simonw,2021-12-19T20:55:59Z,2021-12-19T20:55:59Z,OWNER,"Closing this issue because I've optimized this a whole bunch, and it's definitely good enough for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997325189,https://api.github.com/repos/simonw/datasette/issues/1555,997325189,IC_kwDOBm6k_c47cfmF,9599,simonw,2021-12-19T03:55:01Z,2021-12-19T20:54:51Z,OWNER,"It's a bit annoying that the queries no longer show up in the trace at all now, thanks to running in `.execute_fn()`. I wonder if there's something smart I can do about that - maybe have `trace()` record that function with a traceback even though it doesn't have the executed SQL string? 5fac26aa221a111d7633f2dd92014641f7c0ade9 has the same problem.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997459637,https://api.github.com/repos/simonw/datasette/issues/1555,997459637,IC_kwDOBm6k_c47dAa1,9599,simonw,2021-12-19T20:53:46Z,2021-12-19T20:53:46Z,OWNER,Using #1571 showed me that the `DELETE FROM columns/foreign_keys/indexes WHERE database_name = ? and table_name = ?` queries were running way more times than I expected. I came up with a new optimization that just does `DELETE FROM columns/foreign_keys/indexes WHERE database_name = ?` instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1566#issuecomment-997457790,https://api.github.com/repos/simonw/datasette/issues/1566,997457790,IC_kwDOBm6k_c47c_9-,9599,simonw,2021-12-19T20:40:50Z,2021-12-19T20:40:57Z,OWNER,"Also release new version of `datasette-pretty-traces` with this feature: - https://github.com/simonw/datasette-pretty-traces/issues/7","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083669410,Release Datasette 0.60, https://github.com/simonw/datasette/issues/1555#issuecomment-997342494,https://api.github.com/repos/simonw/datasette/issues/1555,997342494,IC_kwDOBm6k_c47cj0e,9599,simonw,2021-12-19T07:22:04Z,2021-12-19T07:22:04Z,OWNER,"Another option would be to provide an abstraction that makes it easier to run a group of SQL queries in the same thread at the same time, and have them traced correctly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997324666,https://api.github.com/repos/simonw/datasette/issues/1555,997324666,IC_kwDOBm6k_c47cfd6,9599,simonw,2021-12-19T03:47:51Z,2021-12-19T03:48:09Z,OWNER,"Here's a hacked together prototype of running all of that stuff inside a single function passed to `.execute_fn()`: ```diff diff --git a/datasette/utils/internal_db.py b/datasette/utils/internal_db.py index 95055d8..58f9982 100644 --- a/datasette/utils/internal_db.py +++ b/datasette/utils/internal_db.py @@ -1,4 +1,5 @@ import textwrap +from datasette.utils import table_column_details async def init_internal_db(db): @@ -70,49 +71,70 @@ async def populate_schema_tables(internal_db, db): ""DELETE FROM tables WHERE database_name = ?"", [database_name], block=True ) tables = (await db.execute(""select * from sqlite_master WHERE type = 'table'"")).rows - tables_to_insert = [] - columns_to_delete = [] - columns_to_insert = [] - foreign_keys_to_delete = [] - foreign_keys_to_insert = [] - indexes_to_delete = [] - indexes_to_insert = [] - for table in tables: - table_name = table[""name""] - tables_to_insert.append( - (database_name, table_name, table[""rootpage""], table[""sql""]) - ) - columns_to_delete.append((database_name, table_name)) - columns = await db.table_column_details(table_name) - columns_to_insert.extend( - { - **{""database_name"": database_name, ""table_name"": table_name}, - **column._asdict(), - } - for column in columns - ) - foreign_keys_to_delete.append((database_name, table_name)) - foreign_keys = ( - await db.execute(f""PRAGMA foreign_key_list([{table_name}])"") - ).rows - foreign_keys_to_insert.extend( - { - **{""database_name"": database_name, ""table_name"": table_name}, - **dict(foreign_key), - } - for foreign_key in foreign_keys - ) - indexes_to_delete.append((database_name, table_name)) - indexes = (await db.execute(f""PRAGMA index_list([{table_name}])"")).rows - indexes_to_insert.extend( - { - **{""database_name"": database_name, ""table_name"": table_name}, - **dict(index), - } - for index in indexes + def collect_info(conn): + tables_to_insert = [] + columns_to_delete = [] + columns_to_insert = [] + foreign_keys_to_delete = [] + foreign_keys_to_insert = [] + indexes_to_delete = [] + indexes_to_insert = [] + + for table in tables: + table_name = table[""name""] + tables_to_insert.append( + (database_name, table_name, table[""rootpage""], table[""sql""]) + ) + columns_to_delete.append((database_name, table_name)) + columns = table_column_details(conn, table_name) + columns_to_insert.extend( + { + **{""database_name"": database_name, ""table_name"": table_name}, + **column._asdict(), + } + for column in columns + ) + foreign_keys_to_delete.append((database_name, table_name)) + foreign_keys = conn.execute( + f""PRAGMA foreign_key_list([{table_name}])"" + ).fetchall() + foreign_keys_to_insert.extend( + { + **{""database_name"": database_name, ""table_name"": table_name}, + **dict(foreign_key), + } + for foreign_key in foreign_keys + ) + indexes_to_delete.append((database_name, table_name)) + indexes = conn.execute(f""PRAGMA index_list([{table_name}])"").fetchall() + indexes_to_insert.extend( + { + **{""database_name"": database_name, ""table_name"": table_name}, + **dict(index), + } + for index in indexes + ) + return ( + tables_to_insert, + columns_to_delete, + columns_to_insert, + foreign_keys_to_delete, + foreign_keys_to_insert, + indexes_to_delete, + indexes_to_insert, ) + ( + tables_to_insert, + columns_to_delete, + columns_to_insert, + foreign_keys_to_delete, + foreign_keys_to_insert, + indexes_to_delete, + indexes_to_insert, + ) = await db.execute_fn(collect_info) + await internal_db.execute_write_many( """""" INSERT INTO tables (database_name, table_name, rootpage, sql) ``` First impressions: it looks like this helps **a lot** - as far as I can tell this is now taking around 21ms to get to the point at which all of those internal databases have been populated, where previously it took more than 180ms. ![CleanShot 2021-12-18 at 19 47 22@2x](https://user-images.githubusercontent.com/9599/146663192-bba098d5-e7bd-4e2e-b525-2270867888a0.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997324156,https://api.github.com/repos/simonw/datasette/issues/1555,997324156,IC_kwDOBm6k_c47cfV8,9599,simonw,2021-12-19T03:40:05Z,2021-12-19T03:40:05Z,OWNER,"Using the prototype of this: - https://github.com/simonw/datasette-pretty-traces/issues/5 I'm seeing about 180ms spent running all of these queries on startup! ![CleanShot 2021-12-18 at 19 38 37@2x](https://user-images.githubusercontent.com/9599/146663045-46bda669-90de-474f-8870-345182725dc1.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997321767,https://api.github.com/repos/simonw/datasette/issues/1555,997321767,IC_kwDOBm6k_c47cewn,9599,simonw,2021-12-19T03:10:58Z,2021-12-19T03:10:58Z,OWNER,"I wonder how much overhead there is switching between the `async` event loop main code and the thread that runs the SQL queries. Would there be a performance boost if I gathered all of the column/index information in a single function run on the thread using `db.execute_fn()` I wonder? It would eliminate a bunch of switching between threads. Would be great to understand how much of an impact that would have.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997321653,https://api.github.com/repos/simonw/datasette/issues/1555,997321653,IC_kwDOBm6k_c47ceu1,9599,simonw,2021-12-19T03:09:43Z,2021-12-19T03:09:43Z,OWNER,"On that same documentation page I just spotted this: > This feature is experimental and is subject to change. Further documentation will become available if and when the table-valued functions for PRAGMAs feature becomes officially supported. This makes me nervous to rely on pragma function optimizations in Datasette itself.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997321477,https://api.github.com/repos/simonw/datasette/issues/1555,997321477,IC_kwDOBm6k_c47cesF,9599,simonw,2021-12-19T03:07:33Z,2021-12-19T03:07:33Z,OWNER,"If I want to continue supporting SQLite prior to 3.16.0 (2017-01-02) I'll need this optimization to only kick in with versions that support table-valued PRAGMA functions, while keeping the old `PRAGMA foreign_key_list(table)` stuff working for those older versions. That's feasible, but it's a bit more work - and I need to make sure I have robust testing in place for SQLite 3.15.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997321327,https://api.github.com/repos/simonw/datasette/issues/1555,997321327,IC_kwDOBm6k_c47cepv,9599,simonw,2021-12-19T03:05:39Z,2021-12-19T03:05:44Z,OWNER,"This caught me out once before in: - https://github.com/simonw/datasette/issues/1276 Turns out Glitch was running SQLite 3.11.0 from 2016-02-15.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997321217,https://api.github.com/repos/simonw/datasette/issues/1555,997321217,IC_kwDOBm6k_c47ceoB,9599,simonw,2021-12-19T03:04:16Z,2021-12-19T03:04:16Z,OWNER,"One thing to watch out for though, from https://sqlite.org/pragma.html#pragfunc > The table-valued functions for PRAGMA feature was added in SQLite version 3.16.0 (2017-01-02). Prior versions of SQLite cannot use this feature. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997321115,https://api.github.com/repos/simonw/datasette/issues/1555,997321115,IC_kwDOBm6k_c47cemb,9599,simonw,2021-12-19T03:03:12Z,2021-12-19T03:03:12Z,OWNER,"Table columns is a bit harder, because `table_xinfo` is only in SQLite 3.26.0 or higher: https://github.com/simonw/datasette/blob/d637ed46762fdbbd8e32b86f258cd9a53c1cfdc7/datasette/utils/__init__.py#L565-L581 So if that function is available: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++table_xinfo.*%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_table_xinfo%28sqlite_master.name%29+AS+table_xinfo%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27 ```sql SELECT sqlite_master.name, table_xinfo.* FROM sqlite_master, pragma_table_xinfo(sqlite_master.name) AS table_xinfo WHERE sqlite_master.type = 'table' ``` And otherwise, using `table_info`: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++table_info.*%2C%0D%0A++0+as+hidden%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_table_info%28sqlite_master.name%29+AS+table_info%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27 ```sql SELECT sqlite_master.name, table_info.*, 0 as hidden FROM sqlite_master, pragma_table_info(sqlite_master.name) AS table_info WHERE sqlite_master.type = 'table' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997320824,https://api.github.com/repos/simonw/datasette/issues/1555,997320824,IC_kwDOBm6k_c47ceh4,9599,simonw,2021-12-19T02:59:57Z,2021-12-19T03:00:44Z,OWNER,"To list all indexes: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++index_list.*%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_index_list%28sqlite_master.name%29+AS+index_list%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27 ```sql SELECT sqlite_master.name, index_list.* FROM sqlite_master, pragma_index_list(sqlite_master.name) AS index_list WHERE sqlite_master.type = 'table' ``` Foreign keys: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++foreign_key_list.*%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_foreign_key_list%28sqlite_master.name%29+AS+foreign_key_list%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27 ```sql SELECT sqlite_master.name, foreign_key_list.* FROM sqlite_master, pragma_foreign_key_list(sqlite_master.name) AS foreign_key_list WHERE sqlite_master.type = 'table' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1566#issuecomment-997272328,https://api.github.com/repos/simonw/datasette/issues/1566,997272328,IC_kwDOBm6k_c47cSsI,9599,simonw,2021-12-18T19:18:01Z,2021-12-18T19:18:01Z,OWNER,"Added some useful new documented internal methods in: - #1570","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083669410,Release Datasette 0.60, https://github.com/simonw/datasette/issues/1555#issuecomment-997272223,https://api.github.com/repos/simonw/datasette/issues/1555,997272223,IC_kwDOBm6k_c47cSqf,9599,simonw,2021-12-18T19:17:13Z,2021-12-18T19:17:13Z,OWNER,That's a good optimization. Still need to deal with the huge flurry of `PRAGMA` queries though before I can consider this done.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1570#issuecomment-997267583,https://api.github.com/repos/simonw/datasette/issues/1570,997267583,IC_kwDOBm6k_c47cRh_,9599,simonw,2021-12-18T18:46:05Z,2021-12-18T18:46:12Z,OWNER,This will replace the work done in #1569.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083921371,Separate db.execute_write() into three methods, https://github.com/simonw/datasette/issues/1555#issuecomment-997267416,https://api.github.com/repos/simonw/datasette/issues/1555,997267416,IC_kwDOBm6k_c47cRfY,9599,simonw,2021-12-18T18:44:53Z,2021-12-18T18:45:28Z,OWNER,"Rather than adding a `executemany=True` parameter, I'm now thinking a better design might be to have three methods: - `db.execute_write(sql, params=None, block=False)` - `db.execute_writescript(sql, block=False)` - `db.execute_writemany(sql, params_seq, block=False)`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1569#issuecomment-997266687,https://api.github.com/repos/simonw/datasette/issues/1569,997266687,IC_kwDOBm6k_c47cRT_,9599,simonw,2021-12-18T18:41:40Z,2021-12-18T18:41:40Z,OWNER,Updated documentation: https://docs.datasette.io/en/latest/internals.html#await-db-execute-write-sql-params-none-executescript-false-block-false,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083895395,"db.execute_write(..., executescript=True) parameter", https://github.com/simonw/datasette/issues/1555#issuecomment-997266100,https://api.github.com/repos/simonw/datasette/issues/1555,997266100,IC_kwDOBm6k_c47cRK0,9599,simonw,2021-12-18T18:40:02Z,2021-12-18T18:40:02Z,OWNER,The implementation of `cursor.executemany()` looks very efficient - it turns into a call to this C function with `multiple` set to `1`: https://github.com/python/cpython/blob/e002bbc6cce637171fb2b1391ffeca8643a13843/Modules/_sqlite/cursor.c#L468-L469,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997262475,https://api.github.com/repos/simonw/datasette/issues/1555,997262475,IC_kwDOBm6k_c47cQSL,9599,simonw,2021-12-18T18:34:18Z,2021-12-18T18:34:18Z,OWNER," Using `executescript=True` that call now takes 1.89ms to create all of those tables.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1569#issuecomment-997249563,https://api.github.com/repos/simonw/datasette/issues/1569,997249563,IC_kwDOBm6k_c47cNIb,9599,simonw,2021-12-18T18:21:23Z,2021-12-18T18:21:23Z,OWNER,"Goal here is to gain the ability to use `conn.executescript()` and still have it show up in the tracer. https://docs.python.org/3/library/sqlite3.html#sqlite3.Cursor.executescript","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083895395,"db.execute_write(..., executescript=True) parameter", https://github.com/simonw/datasette/issues/1555#issuecomment-997248364,https://api.github.com/repos/simonw/datasette/issues/1555,997248364,IC_kwDOBm6k_c47cM1s,9599,simonw,2021-12-18T18:20:10Z,2021-12-18T18:20:10Z,OWNER,"Idea: teach `execute_write` to accept an optional `executescript=True` parameter, like this: ```diff diff --git a/datasette/database.py b/datasette/database.py index 468e936..1a424f5 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -94,10 +94,14 @@ class Database: f""file:{self.path}{qs}"", uri=True, check_same_thread=False ) - async def execute_write(self, sql, params=None, block=False): + async def execute_write(self, sql, params=None, executescript=False, block=False): + assert not executescript and params, ""Cannot use params with executescript=True"" def _inner(conn): with conn: - return conn.execute(sql, params or []) + if executescript: + return conn.executescript(sql) + else: + return conn.execute(sql, params or []) with trace(""sql"", database=self.name, sql=sql.strip(), params=params): results = await self.execute_write_fn(_inner, block=block) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997245301,https://api.github.com/repos/simonw/datasette/issues/1555,997245301,IC_kwDOBm6k_c47cMF1,9599,simonw,2021-12-18T18:17:04Z,2021-12-18T18:17:04Z,OWNER,"One downside of `conn.executescript()` is that it won't be picked up by the tracing mechanism - in fact nothing that uses `await db.execute_write_fn(fn, block=True)` or `await db.execute_fn(fn, block=True)` gets picked up by tracing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997241969,https://api.github.com/repos/simonw/datasette/issues/1555,997241969,IC_kwDOBm6k_c47cLRx,9599,simonw,2021-12-18T18:13:04Z,2021-12-18T18:13:04Z,OWNER,Also: running all of those `CREATE TABLE IF NOT EXISTS` in a single call to `conn.executescript()` rather than as separate queries may speed things up too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997241645,https://api.github.com/repos/simonw/datasette/issues/1555,997241645,IC_kwDOBm6k_c47cLMt,9599,simonw,2021-12-18T18:12:26Z,2021-12-18T18:12:26Z,OWNER,"A simpler optimization would be just to turn all of those column and index reads into a single efficient UNION query against each database, then figure out the most efficient pattern to send them all as writes in one go as opposed to calling `.execute_write()` in a loop.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1566#issuecomment-997235388,https://api.github.com/repos/simonw/datasette/issues/1566,997235388,IC_kwDOBm6k_c47cJq8,9599,simonw,2021-12-18T17:32:07Z,2021-12-18T17:32:07Z,OWNER,I can release a new version of `datasette-leaflet-freedraw` as soon as this is out.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083669410,Release Datasette 0.60, https://github.com/simonw/datasette/issues/1555#issuecomment-997235086,https://api.github.com/repos/simonw/datasette/issues/1555,997235086,IC_kwDOBm6k_c47cJmO,9599,simonw,2021-12-18T17:30:13Z,2021-12-18T17:30:13Z,OWNER,"Now that trace sees write queries (#1568) it's clear that there is a whole lot more DB activity then I had realized: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997234858,https://api.github.com/repos/simonw/datasette/issues/1555,997234858,IC_kwDOBm6k_c47cJiq,9599,simonw,2021-12-18T17:28:44Z,2021-12-18T17:28:44Z,OWNER,"Maybe it would be worth exploring attaching each DB in turn to the _internal connection in order to perform these queries faster. I'm a bit worried about leaks though: the internal database isn't meant to be visible, even temporarily attaching another DB to it could cause SQL queries against that DB to be able to access the internal data. So maybe instead the _internal connection gets to connect to the other DBs? There's a maximum of ten there I think, which is good for most but not all cases. But the cases with the most connected databases will see the worst performance!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1568#issuecomment-997153253,https://api.github.com/repos/simonw/datasette/issues/1568,997153253,IC_kwDOBm6k_c47b1nl,9599,simonw,2021-12-18T06:20:23Z,2021-12-18T06:20:23Z,OWNER,Now running at https://latest-with-plugins.datasette.io/github/commits?_trace=1,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083726550,Trace should show queries on the write connection too, https://github.com/simonw/datasette/issues/1568#issuecomment-997128950,https://api.github.com/repos/simonw/datasette/issues/1568,997128950,IC_kwDOBm6k_c47bvr2,9599,simonw,2021-12-18T02:38:01Z,2021-12-18T02:38:01Z,OWNER,"Prototype: ```diff diff --git a/datasette/database.py b/datasette/database.py index 0a0c104..468e936 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -99,7 +99,9 @@ class Database: with conn: return conn.execute(sql, params or []) - return await self.execute_write_fn(_inner, block=block) + with trace(""sql"", database=self.name, sql=sql.strip(), params=params): + results = await self.execute_write_fn(_inner, block=block) + return results async def execute_write_fn(self, fn, block=False): task_id = uuid.uuid5(uuid.NAMESPACE_DNS, ""datasette.io"") ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083726550,Trace should show queries on the write connection too, https://github.com/simonw/datasette/issues/1555#issuecomment-997128508,https://api.github.com/repos/simonw/datasette/issues/1555,997128508,IC_kwDOBm6k_c47bvk8,9599,simonw,2021-12-18T02:33:57Z,2021-12-18T02:33:57Z,OWNER,"Here's why - `trace` only applies to read, not write SQL operations: https://github.com/simonw/datasette/blob/7c8f8aa209e4ba7bf83976f8495d67c28fbfca24/datasette/database.py#L209-L211","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997128368,https://api.github.com/repos/simonw/datasette/issues/1555,997128368,IC_kwDOBm6k_c47bviw,9599,simonw,2021-12-18T02:32:43Z,2021-12-18T02:32:43Z,OWNER,I wonder why the `INSERT INTO` queries don't show up in that `?trace=1` view?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997128251,https://api.github.com/repos/simonw/datasette/issues/1555,997128251,IC_kwDOBm6k_c47bvg7,9599,simonw,2021-12-18T02:31:51Z,2021-12-18T02:31:51Z,OWNER,"I was thinking it might even be possible to convert this into a `insert into tables select from ...` query: https://github.com/simonw/datasette/blob/c00f29affcafce8314366852ba1a0f5a7dd25690/datasette/utils/internal_db.py#L102-L112 But the `SELECT` runs against a separate database from the `INSERT INTO`, so I would have to setup a cross-database connection for this which feels a little too complicated.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997128080,https://api.github.com/repos/simonw/datasette/issues/1555,997128080,IC_kwDOBm6k_c47bveQ,9599,simonw,2021-12-18T02:30:19Z,2021-12-18T02:30:19Z,OWNER,"I think all of these queries happen in one place - in the `populate_schema_tables()` function - so optimizing them might be localized to just that area of the code, which would be nice: https://github.com/simonw/datasette/blob/c00f29affcafce8314366852ba1a0f5a7dd25690/datasette/utils/internal_db.py#L97-L183","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1561#issuecomment-997127784,https://api.github.com/repos/simonw/datasette/issues/1561,997127784,IC_kwDOBm6k_c47bvZo,9599,simonw,2021-12-18T02:27:56Z,2021-12-18T02:27:56Z,OWNER,"Oh that's an interesting solution, combining the hashes of all of the individual databases. I'm actually not a big fan of `hashed_url` mode - I implemented it right at the start of the project because it felt like a clever hack, and then ended up making it not-the-default a few years ago: - #418 - #419 - #421 I've since not found myself wanting to use it at all for any of my projects - which makes me nervous, because it means there's a pretty complex feature that I'm not using at all, so it's only really protected by the existing unit tests for it. What I'd really like to do is figure out how to have hashed URL mode work entirely as a plugin - then I could extract it from Datasette core entirely (which would simplify a bunch of stuff) but people who find the optimization useful would be able to access it. I'm not sure that the existing plugin hooks are robust enough to do that yet though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082765654,"add hash id to ""_memory"" url if hashed url mode is turned on and crossdb is also turned on", https://github.com/simonw/datasette/issues/1563#issuecomment-997127084,https://api.github.com/repos/simonw/datasette/issues/1563,997127084,IC_kwDOBm6k_c47bvOs,9599,simonw,2021-12-18T02:22:30Z,2021-12-18T02:22:30Z,OWNER,Docs here: https://docs.datasette.io/en/latest/internals.html#datasette-class,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083573206,Datasette(... files=) should not be a required argument, https://github.com/simonw/datasette/issues/1563#issuecomment-997125191,https://api.github.com/repos/simonw/datasette/issues/1563,997125191,IC_kwDOBm6k_c47buxH,9599,simonw,2021-12-18T02:10:20Z,2021-12-18T02:10:20Z,OWNER,I should document the usage of this constructor in https://docs.datasette.io/en/stable/internals.html#datasette-class,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083573206,Datasette(... files=) should not be a required argument, https://github.com/simonw/datasette/issues/1546#issuecomment-997124280,https://api.github.com/repos/simonw/datasette/issues/1546,997124280,IC_kwDOBm6k_c47bui4,9599,simonw,2021-12-18T02:05:16Z,2021-12-18T02:05:16Z,OWNER,"Sure - there are actually several levels to this. The code that creates connections to the database is this: https://github.com/simonw/datasette/blob/83bacfa9452babe7bd66e3579e23af988d00f6ac/datasette/database.py#L72-L95 For files on disk, it does this: ```python # For read-only connections conn = sqlite3.connect( ""file:my.db?mode=ro"", uri=True, check_same_thread=False) # For connections that should be treated as immutable: conn = sqlite3.connect( ""file:my.db?immutable=1"", uri=True, check_same_thread=False) ``` For in-memory databases it runs this after the connection has been created: ```python conn.execute(""PRAGMA query_only=1"") ``` SQLite `PRAGMA` queries are treated as dangerous: someone could run `PRAGMA query_only=0` to turn that previous option off for example. So this function runs against any incoming SQL to verify that it looks like a `SELECT ...` and doesn't have anything like that in it. https://github.com/simonw/datasette/blob/83bacfa9452babe7bd66e3579e23af988d00f6ac/datasette/utils/__init__.py#L195-L204 You can see the tests for that here: https://github.com/simonw/datasette/blob/b1fed48a95516ae84c0f020582303ab50ab817e2/tests/test_utils.py#L136-L170","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1076057610,validating the sql, https://github.com/simonw/datasette/issues/1564#issuecomment-997122938,https://api.github.com/repos/simonw/datasette/issues/1564,997122938,IC_kwDOBm6k_c47buN6,9599,simonw,2021-12-18T01:55:25Z,2021-12-18T01:55:46Z,OWNER,"Made this change while working on this issue: - #1567 I'm going to write a test for this that uses that `sleep()` SQL function from c35b84a2aabe2f14aeacf6cda4110ae1e94d6059.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083581011,_prepare_connection not called on write connections, https://github.com/simonw/datasette/issues/1565#issuecomment-997121215,https://api.github.com/repos/simonw/datasette/issues/1565,997121215,IC_kwDOBm6k_c47bty_,9599,simonw,2021-12-18T01:45:44Z,2021-12-18T01:45:44Z,OWNER,I want to get this into Datasette 0.60 - #1566 - it's a small change that can unlock a lot of potential.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins, https://github.com/simonw/datasette/issues/621#issuecomment-997120723,https://api.github.com/repos/simonw/datasette/issues/621,997120723,IC_kwDOBm6k_c47btrT,9599,simonw,2021-12-18T01:42:33Z,2021-12-18T01:42:33Z,OWNER,I refactored this code out into the `filters.py` module in aa7f0037a46eb76ae6fe9bf2a1f616c58738ecdf,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520681725,Syntax for ?_through= that works as a form field, https://github.com/simonw/datasette/issues/617#issuecomment-552253893,https://api.github.com/repos/simonw/datasette/issues/617,552253893,MDEyOklzc3VlQ29tbWVudDU1MjI1Mzg5Mw==,9599,simonw,2019-11-11T00:46:42Z,2021-12-18T01:41:47Z,OWNER,"As noted in https://github.com/simonw/datasette/issues/621#issuecomment-552253208 a common pattern in this method is blocks of code that append new items to the `where_clauses`, `params` and `extra_human_descriptions` arrays. This is a useful refactoring opportunity. Code that fits this pattern: * The code that builds based on the filters: `where_clauses, params = filters.build_where_clauses(table)` and `human_description_en = filters.human_description_en(extra=extra_human_descriptions)` * Code that handles `?_where=`: `where_clauses.extend(request.args[""_where""])` - though note that this also appends to a `extra_wheres_for_ui` array which nothing else uses * The `_through=` code, see #621 for details * The code that deals with `?_search=` FTS The keyset pagination code modifies `where_clauses` and `params` too, but I don't think it's quite going to work with the same abstraction that would cover the above examples. [UPDATE December 2021 - this comment became the basis for a new `filters_from_request` plugin hook, see also #473]","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",519613116,Refactor TableView.data() method, https://github.com/simonw/datasette/issues/1518#issuecomment-981153060,https://api.github.com/repos/simonw/datasette/issues/1518,981153060,IC_kwDOBm6k_c46ezUk,9599,simonw,2021-11-28T21:13:09Z,2021-12-17T23:37:08Z,OWNER,"Two new requirements inspired by work on the `datasette-table` (and `datasette-notebook`) projects: - #1533 - #1534","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-997082845,https://api.github.com/repos/simonw/datasette/issues/1518,997082845,IC_kwDOBm6k_c47bkbd,9599,simonw,2021-12-17T23:10:09Z,2021-12-17T23:10:17Z,OWNER,These changes so far are now in the 0.60a0 alpha: https://github.com/simonw/datasette/releases/tag/0.60a0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/pull/1559#issuecomment-997082676,https://api.github.com/repos/simonw/datasette/issues/1559,997082676,IC_kwDOBm6k_c47bkY0,9599,simonw,2021-12-17T23:09:41Z,2021-12-17T23:09:41Z,OWNER,This is now available to try out in Datasette 0.60a0: https://github.com/simonw/datasette/releases/tag/0.60a0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082743068,"filters_from_request plugin hook, now used in TableView", https://github.com/simonw/datasette/pull/1562#issuecomment-997082189,https://api.github.com/repos/simonw/datasette/issues/1562,997082189,IC_kwDOBm6k_c47bkRN,9599,simonw,2021-12-17T23:08:14Z,2021-12-17T23:08:14Z,OWNER,"Oh that makes sense: In Python 3.6 this happens: ``` Collecting janus<1.1,>=0.6.2 Using cached janus-0.7.0-py3-none-any.whl (6.9 kB) ``` While in Python 3.7 or higher this happens: ``` Collecting janus<1.1,>=0.6.2 Downloading janus-1.0.0-py3-none-any.whl (6.9 kB) ``` So this is safe to apply because `pip` is smart enough to pick the version of Janus that works for that Python version.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083246400,"Update janus requirement from <0.8,>=0.6.2 to >=0.6.2,<1.1", https://github.com/simonw/datasette/pull/1562#issuecomment-997081673,https://api.github.com/repos/simonw/datasette/issues/1562,997081673,IC_kwDOBm6k_c47bkJJ,9599,simonw,2021-12-17T23:06:38Z,2021-12-17T23:06:38Z,OWNER,"From this diff between `0.7.0` and `1.0`: https://github.com/aio-libs/janus/compare/v0.7.0...v1.0.0 It looks like the only change relevant to compatibility is `loop = asyncio.get_running_loop()` directly instead of falling back to `asyncio.get_event_loop()` if `get_running_loop` isn't available.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083246400,"Update janus requirement from <0.8,>=0.6.2 to >=0.6.2,<1.1", https://github.com/simonw/datasette/pull/1562#issuecomment-997080352,https://api.github.com/repos/simonw/datasette/issues/1562,997080352,IC_kwDOBm6k_c47bj0g,9599,simonw,2021-12-17T23:03:08Z,2021-12-17T23:03:08Z,OWNER,"They say they've dropped 3.6 support, but Datasette's tests against 3.6 are still passing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083246400,"Update janus requirement from <0.8,>=0.6.2 to >=0.6.2,<1.1", https://github.com/simonw/datasette/issues/1566#issuecomment-997078812,https://api.github.com/repos/simonw/datasette/issues/1566,997078812,IC_kwDOBm6k_c47bjcc,9599,simonw,2021-12-17T22:58:55Z,2021-12-17T22:58:55Z,OWNER,The release notes for the 0.60a0 alpha will be useful here: https://github.com/simonw/datasette/releases/tag/0.60a0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083669410,Release Datasette 0.60, https://github.com/simonw/datasette/issues/1565#issuecomment-997077410,https://api.github.com/repos/simonw/datasette/issues/1565,997077410,IC_kwDOBm6k_c47bjGi,9599,simonw,2021-12-17T22:54:45Z,2021-12-17T22:54:45Z,OWNER,"The table page should expose the query both with and without the `limit` clause. The above gave me back: ```sql select id, ACCESS_TYP, UNIT_ID, UNIT_NAME, SUID_NMA, AGNCY_ID, AGNCY_NAME, AGNCY_LEV, AGNCY_TYP, AGNCY_WEB, LAYER, MNG_AG_ID, MNG_AGENCY, MNG_AG_LEV, MNG_AG_TYP, PARK_URL, COUNTY, ACRES, LABEL_NAME, YR_EST, DES_TP, GAP_STS, geometry from CPAD_2020a_Units where ""AGNCY_LEV"" = :p0 order by id limit 101 ``` But I actually wanted to run a `fetch()` against a version of that without the `order by id limit 101` bit (I wanted to figure out the `Extent()` of the `geometry` column) - so I need something like `datasette.table_sql_no_order_no_limit`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins, https://github.com/simonw/datasette/issues/1565#issuecomment-997069128,https://api.github.com/repos/simonw/datasette/issues/1565,997069128,IC_kwDOBm6k_c47bhFI,9599,simonw,2021-12-17T22:31:18Z,2021-12-17T22:31:18Z,OWNER,This should aim to be as consistent as possible with the various arguments to hooks on https://docs.datasette.io/en/stable/plugin_hooks.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins, https://github.com/simonw/datasette/pull/1559#issuecomment-996961196,https://api.github.com/repos/simonw/datasette/issues/1559,996961196,IC_kwDOBm6k_c47bGus,9599,simonw,2021-12-17T19:00:53Z,2021-12-17T19:00:53Z,OWNER,"I'm going to merge this to `main` now. I can continue the refactoring there, but having it in `main` means I can put out an alpha release with the new hook which will unblock me from running tests against it in this repo: https://github.com/simonw/datasette-leaflet-freedraw/pull/8","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082743068,"filters_from_request plugin hook, now used in TableView", https://github.com/simonw/datasette/pull/1559#issuecomment-996959325,https://api.github.com/repos/simonw/datasette/issues/1559,996959325,IC_kwDOBm6k_c47bGRd,9599,simonw,2021-12-17T18:59:54Z,2021-12-17T18:59:54Z,OWNER,I've convinced myself that this plugin hook design is good through this `datasette-leaflet-freedraw` prototype: https://github.com/simonw/datasette-leaflet-freedraw/blob/e8a16a0fe90656b8d655c02881d23a2b9833281d/datasette_leaflet_freedraw/__init__.py,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082743068,"filters_from_request plugin hook, now used in TableView", https://github.com/simonw/datasette/issues/473#issuecomment-996958442,https://api.github.com/repos/simonw/datasette/issues/473,996958442,IC_kwDOBm6k_c47bGDq,9599,simonw,2021-12-17T18:59:27Z,2021-12-17T18:59:27Z,OWNER,I'm happy with how the prototype that used this plugin in `datasette-leaflet-freedraw` turned out: https://github.com/simonw/datasette-leaflet-freedraw/blob/e8a16a0fe90656b8d655c02881d23a2b9833281d/datasette_leaflet_freedraw/__init__.py,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",445850934,Plugin hook: filters_from_request, https://github.com/simonw/datasette/issues/473#issuecomment-996345233,https://api.github.com/repos/simonw/datasette/issues/473,996345233,IC_kwDOBm6k_c47YwWR,9599,simonw,2021-12-17T01:20:31Z,2021-12-17T18:13:01Z,OWNER,I could use this hook to add table filtering on a map to the existing `datasette-leaflet-freedraw` plugin.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",445850934,Plugin hook: filters_from_request, https://github.com/simonw/datasette/pull/1559#issuecomment-996895423,https://api.github.com/repos/simonw/datasette/issues/1559,996895423,IC_kwDOBm6k_c47a2q_,9599,simonw,2021-12-17T17:28:44Z,2021-12-17T17:28:44Z,OWNER,"Before I land this I'm going to build one prototype plugin against it to confirm that the new hook is useful in its current shape. I'll add support for filtering a table by drawing on a map to https://datasette.io/plugins/datasette-leaflet-freedraw","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082743068,"filters_from_request plugin hook, now used in TableView", https://github.com/simonw/datasette/pull/1204#issuecomment-996488925,https://api.github.com/repos/simonw/datasette/issues/1204,996488925,IC_kwDOBm6k_c47ZTbd,9599,simonw,2021-12-17T07:10:48Z,2021-12-17T07:10:48Z,OWNER,I think this is missing the `_macro.html` template file but I have that in my Dropbox.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",793002853,WIP: Plugin includes, https://github.com/simonw/datasette/issues/473#issuecomment-996484551,https://api.github.com/repos/simonw/datasette/issues/473,996484551,IC_kwDOBm6k_c47ZSXH,9599,simonw,2021-12-17T07:02:21Z,2021-12-17T07:04:23Z,OWNER,"The one slightly weird thing about this hook is how it adds `extra_context` without an obvious way for plugins to add extra HTML to the templates based on that context. Maybe I need the proposed mechanism from - #1191 Which has an in-progress PR: - #1204","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",445850934,Plugin hook: filters_from_request, https://github.com/simonw/datasette/issues/1191#issuecomment-761104933,https://api.github.com/repos/simonw/datasette/issues/1191,761104933,MDEyOklzc3VlQ29tbWVudDc2MTEwNDkzMw==,9599,simonw,2021-01-15T18:21:26Z,2021-12-17T07:03:02Z,OWNER,"Also related: #857 (comprehensive documentation of variables available to templates) - since then the plugin hook could be fed the full template context and use that to do its thing. Or maybe the plugin hooks gets to return the name of a template that should be `{% include %}` into the page at that point? But the plugin may want to add extra context that is available to that template include.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",787098345,Ability for plugins to collaborate when adding extra HTML to blocks in default templates, https://github.com/simonw/datasette/pull/1559#issuecomment-996286808,https://api.github.com/repos/simonw/datasette/issues/1559,996286808,IC_kwDOBm6k_c47YiFY,9599,simonw,2021-12-17T00:01:43Z,2021-12-17T00:01:43Z,OWNER,"This already has tests and documentation, and I've used it to refactor out the logic for `?_where=` and `?_search=` and `?_through=`. Do I like this enough to land it on `main`? Also, I think I can still use it to refactor out the `Filters` code that implements `?col=x` and `?col__lt=5` and suchlike.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082743068,"filters_from_request plugin hook, now used in TableView", https://github.com/simonw/datasette/issues/473#issuecomment-996286199,https://api.github.com/repos/simonw/datasette/issues/473,996286199,IC_kwDOBm6k_c47Yh73,9599,simonw,2021-12-17T00:00:22Z,2021-12-17T00:00:22Z,OWNER,Documentation for that hook in the PR branch: https://github.com/simonw/datasette/blob/54e9b3972f277431a001e685f78e5dd6403a6d8d/docs/plugin_hooks.rst#filters_from_requestrequest-database-table-datasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",445850934,Plugin hook: filters_from_request, https://github.com/simonw/datasette/issues/1518#issuecomment-996286104,https://api.github.com/repos/simonw/datasette/issues/1518,996286104,IC_kwDOBm6k_c47Yh6Y,9599,simonw,2021-12-17T00:00:07Z,2021-12-17T00:00:07Z,OWNER,Documentation of the new hook in the PR: https://github.com/simonw/datasette/blob/54e9b3972f277431a001e685f78e5dd6403a6d8d/docs/plugin_hooks.rst#filters_from_requestrequest-database-table-datasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/473#issuecomment-996275108,https://api.github.com/repos/simonw/datasette/issues/473,996275108,IC_kwDOBm6k_c47YfOk,9599,simonw,2021-12-16T23:32:22Z,2021-12-16T23:32:30Z,OWNER,This filter design can only influence the `where` component of the SQL clause - it's not able to modify the `SELECT` columns or adjust the `ORDER BY` or `OFFSET LIMIT` parts. I think that's OK.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",445850934,Plugin hook: filters_from_request, https://github.com/simonw/datasette/issues/1518#issuecomment-996272906,https://api.github.com/repos/simonw/datasette/issues/1518,996272906,IC_kwDOBm6k_c47YesK,9599,simonw,2021-12-16T23:27:42Z,2021-12-16T23:27:42Z,OWNER,Got a TIL out of this: https://til.simonwillison.net/pluggy/multiple-hooks-same-file,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/473#issuecomment-996267817,https://api.github.com/repos/simonw/datasette/issues/473,996267817,IC_kwDOBm6k_c47Ydcp,9599,simonw,2021-12-16T23:17:52Z,2021-12-16T23:19:00Z,OWNER,"I revisited this idea in #1518 and came up with a slightly different name and design for the hook: ```python @hookspec def filters_from_request(request, database, table, datasette): """""" Return FilterArguments( where_clauses=[str, str, str], params={}, human_descriptions=[str, str, str], extra_context={} ) based on the request"""""" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",445850934,Plugin hook: filters_from_request, https://github.com/simonw/datasette/issues/1518#issuecomment-996264617,https://api.github.com/repos/simonw/datasette/issues/1518,996264617,IC_kwDOBm6k_c47Ycqp,9599,simonw,2021-12-16T23:11:12Z,2021-12-16T23:11:12Z,OWNER,I managed to extract both `_search=` and `_where=` out using a prototype of that hook. I wonder if it could extract the complex code for `?_next` too?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-996250585,https://api.github.com/repos/simonw/datasette/issues/1518,996250585,IC_kwDOBm6k_c47YZPZ,9599,simonw,2021-12-16T22:43:37Z,2021-12-16T22:45:07Z,OWNER,"Ran into a problem prototyping that hook up for handling `?_where=` - that feature also adds a little bit of extra template context in order to show the interface for removing wheres - the `extra_wheres_for_ui` variable: https://github.com/simonw/datasette/blob/0663d5525cc41e9260ac7d1f6386d3a6eb5ad2a9/datasette/views/table.py#L457-L463 Maybe change to this? ```python class FilterArguments(NamedTuple): where_clauses: List[str] params: Dict[str, Union[str, int, float]] human_descriptions: List[str] extra_context: Dict[str, Any] ``` That might be necessary for `_search` too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-996248713,https://api.github.com/repos/simonw/datasette/issues/1518,996248713,IC_kwDOBm6k_c47YYyJ,9599,simonw,2021-12-16T22:39:47Z,2021-12-16T22:39:47Z,OWNER,"The hook could return a named tuple like this one: ```python from typing import NamedTuple, List, Optional, Union, Dict class FilterArguments(NamedTuple): where_clauses: List[str] params: Dict[str, Union[str, int, float]] human_descriptions: List[str] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-996240802,https://api.github.com/repos/simonw/datasette/issues/1518,996240802,IC_kwDOBm6k_c47YW2i,9599,simonw,2021-12-16T22:25:00Z,2021-12-16T22:36:04Z,OWNER,"I think that plugin hook would get given the `request` object (and `datasette` and the name of the database and table) and returns a list of SQL fragments, a dictionary of lookup arguments and a list of human-description fragments - or an awaitable. `filters_from_request(request, database, table, datasette)` perhaps? (Similar in name to `actor_from_request`). ```python @hookspec def filters_from_request(request, database, table, datasette): """"""Return (where_clauses, params_dict, human_descriptions) based on the request"""""" ``` Turns out that's pretty much exactly what I implemented in 5116c4ec8aed5091e1f75415424b80f613518dc6 for #473: ```python @hookspec def table_filter(): ""Custom filtering of the current table based on the request"" ``` ```python TableFilter = namedtuple(""TableFilter"", ( ""human_description_extras"", ""where_clauses"", ""params"") ) ``` ```python # filter_arguments plugin hook support for awaitable_fn in pm.hook.table_filter(): extras = await awaitable_fn( view=self, name=name, table=table, request=request ) human_description_extras.extend(extras.human_description_extras) where_clauses.extend(extras.where_clauses) params.update(extras.params) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/sqlite-utils/issues/358#issuecomment-996232461,https://api.github.com/repos/simonw/sqlite-utils/issues/358,996232461,IC_kwDOCGYnMM47YU0N,9599,simonw,2021-12-16T22:10:39Z,2021-12-16T22:10:39Z,OWNER,"This goes beyond the `transform()` method - the curious methods that create new SQL tables could benefit from the ability to add `CHECK` constraints too. I haven't used these myself, do you have any `CREATE TABLE` examples that use them that you can share?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082651698,Support for CHECK constraints, https://github.com/simonw/datasette/issues/1518#issuecomment-996227713,https://api.github.com/repos/simonw/datasette/issues/1518,996227713,IC_kwDOBm6k_c47YTqB,9599,simonw,2021-12-16T22:02:35Z,2021-12-16T22:03:55Z,OWNER,"Is there an opportunity to refactor things using a new plugin hook here? Maybe the `register_filters` hook from #473, where the hook becomes responsible for building where clauses (and human descriptions of them) based on the incoming query string. That version dealt with `Filter` classes, but those might be a bit too low-level for this. `?_spatial_within=GEOJSON` was an interesting idea attached to that issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-996225889,https://api.github.com/repos/simonw/datasette/issues/1518,996225889,IC_kwDOBm6k_c47YTNh,9599,simonw,2021-12-16T21:59:32Z,2021-12-16T22:00:42Z,OWNER,I added a ton of comments to the `data()` method which really helps get a better feel for how this all works: https://github.com/simonw/datasette/blob/0663d5525cc41e9260ac7d1f6386d3a6eb5ad2a9/datasette/views/table.py#L322,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-996225235,https://api.github.com/repos/simonw/datasette/issues/1518,996225235,IC_kwDOBm6k_c47YTDT,9599,simonw,2021-12-16T21:58:24Z,2021-12-16T21:58:41Z,OWNER,"A fundamental operation of this view is to construct the SQL query and accompanying human description based on the incoming query string parameters. The human description is the bit at the top of https://latest.datasette.io/fixtures/searchable?_search=dog&_sort=pk&_facet=text2&text2=sara+weasel that says: > 1 row where search matches ""dog"" and text2 = ""sara weasel"" sorted by pk (Also used in the page ``). The code actually gathers three things: - Fragments of the `where` clause, for example ` ""text2"" = :p0` - Parameters, e.g. `{""p0"": ""sara weasel""}` - Human description components, e.g. `text2 = ""sara weasel""` Some operations such as `?_where=` don't currently provide an extra human description component. `_where=` also doesn't populate a parameter, but maybe it could? Would be neat if in the future `?_where=foo+=+:bar` worked and added a `bar` input field to the screen, as seen with custom queries.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-996219117,https://api.github.com/repos/simonw/datasette/issues/1518,996219117,IC_kwDOBm6k_c47YRjt,9599,simonw,2021-12-16T21:47:51Z,2021-12-16T21:49:24Z,OWNER,"Should facets really not be displayed on pages past page one (where `?_next=` is set)? That made sense to me at the time, but I'm now having second thoughts about it. I guess it's a useful performance tweak for when crawlers keep hitting the `?_next=` link. Actually it looks like facets DO display on subsequent pages, e.g. on https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_next=200 - but facet suggestions do not, thanks to this code: https://github.com/simonw/datasette/blob/2c07327d23d9c5cf939ada9ba4091c1b8b2ba42d/datasette/views/table.py#L777-L785 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1558#issuecomment-996204369,https://api.github.com/repos/simonw/datasette/issues/1558,996204369,IC_kwDOBm6k_c47YN9R,9599,simonw,2021-12-16T21:23:25Z,2021-12-16T21:23:25Z,OWNER,"Related: Following the fix for #625 I noticed that `facets_timed_out` gives you just the column name, but doesn't let you know which particular type of facet (`date` or `array` for example) suffered the timeout: https://github.com/simonw/datasette/blob/0d4145d0f4d8b2a7edc1ba4aac1be56cd536a10a/datasette/facets.py#L269-L270 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082584499,Redesign `facet_results` JSON structure prior to Datasette 1.0, https://github.com/simonw/sqlite-utils/issues/357#issuecomment-996179930,https://api.github.com/repos/simonw/sqlite-utils/issues/357,996179930,IC_kwDOCGYnMM47YH_a,9599,simonw,2021-12-16T20:43:19Z,2021-12-16T20:43:19Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079422215,pytest-runner is not required, https://github.com/simonw/datasette/issues/625#issuecomment-996170510,https://api.github.com/repos/simonw/datasette/issues/625,996170510,IC_kwDOBm6k_c47YFsO,9599,simonw,2021-12-16T20:27:41Z,2021-12-16T20:27:41Z,OWNER,"And here's the new JSON: https://latest.datasette.io/fixtures/facetable.json?_facet=created&_facet_date=created&_facet=tags&_facet_array=tags&_nosuggest=1 ``` { ""database"": ""fixtures"", ""table"": ""facetable"", ""is_view"": false, ""human_description_en"": """", ... ""facet_results"": { ""created"": { ""name"": ""created"", ""type"": ""column"", ... }, ""tags"": { ""name"": ""tags"", ""type"": ""column"", ... }, ""created_2"": { ""name"": ""created"", ""type"": ""date"", ... }, ""tags_2"": { ""name"": ""tags"", ""type"": ""array"", ... } } } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/625#issuecomment-996165659,https://api.github.com/repos/simonw/datasette/issues/625,996165659,IC_kwDOBm6k_c47YEgb,9599,simonw,2021-12-16T20:19:53Z,2021-12-16T20:19:53Z,OWNER,Demo of the fix: https://latest.datasette.io/fixtures/facetable?_facet=created&_facet_date=created&_facet=tags&_facet_array=tags#facet-tags,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/625#issuecomment-996161380,https://api.github.com/repos/simonw/datasette/issues/625,996161380,IC_kwDOBm6k_c47YDdk,9599,simonw,2021-12-16T20:13:05Z,2021-12-16T20:13:05Z,OWNER,I updated the example code in the facet plugin hook documentation: https://github.com/simonw/datasette/blob/95d0dd7a1cf6be6b7da41e1404184217eb93f64a/docs/plugin_hooks.rst#register_facet_classes,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/625#issuecomment-996152213,https://api.github.com/repos/simonw/datasette/issues/625,996152213,IC_kwDOBm6k_c47YBOV,9599,simonw,2021-12-16T19:59:46Z,2021-12-16T20:00:05Z,OWNER,"Since no-one is using that plugin hook I'm going to alter its contract slightly. I'll still keep the existing JSON format working though (until 1.0), since it's much more likely that people are using that JSON somewhere.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/830#issuecomment-996151246,https://api.github.com/repos/simonw/datasette/issues/830,996151246,IC_kwDOBm6k_c47YA_O,9599,simonw,2021-12-16T19:58:22Z,2021-12-16T19:58:22Z,OWNER,"As of today, 16 December 2021, I'm still not seeing any evidence that anyone is using this hook (yet) according to GitHub code search: https://cs.github.com/?scopeName=All+repos&scope=&q=register_facet_classes%20-repo%3Asimonw%2Fdatasette","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636511683,Redesign register_facet_classes plugin hook, https://github.com/simonw/datasette/issues/625#issuecomment-996150904,https://api.github.com/repos/simonw/datasette/issues/625,996150904,IC_kwDOBm6k_c47YA54,9599,simonw,2021-12-16T19:57:52Z,2021-12-16T19:57:52Z,OWNER,Good news - GitHub's new code search doesn't show ANYONE using that plugin hook - not surprising since it has that documentation warning plus it's just not a very clearly usable hook: https://cs.github.com/?scopeName=All+repos&scope=&q=register_facet_classes%20-repo%3Asimonw%2Fdatasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/625#issuecomment-996149720,https://api.github.com/repos/simonw/datasette/issues/625,996149720,IC_kwDOBm6k_c47YAnY,9599,simonw,2021-12-16T19:56:14Z,2021-12-16T19:56:14Z,OWNER,"This bad design is even covered in the plugin hooks documentation: https://docs.datasette.io/en/0.59.4/plugin_hooks.html#register-facet-classes It does at least have the following warning: > **Warning** > > The design of this plugin hook is unstable and may change. See [issue 830](https://github.com/simonw/datasette/issues/830).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/625#issuecomment-996146762,https://api.github.com/repos/simonw/datasette/issues/625,996146762,IC_kwDOBm6k_c47X_5K,9599,simonw,2021-12-16T19:51:44Z,2021-12-16T19:51:44Z,OWNER,"Here's where `facet_results` is built up: https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/views/table.py#L752-L758 So the decision to key things based on column name is actually embedded deep in the existing facet classes here: https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/facets.py#L224-L226 https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/facets.py#L395-L397 https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/facets.py#L510-L512","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/1558#issuecomment-996134716,https://api.github.com/repos/simonw/datasette/issues/1558,996134716,IC_kwDOBm6k_c47X888,9599,simonw,2021-12-16T19:46:21Z,2021-12-16T19:46:21Z,OWNER,"The flaw in the current design is illustrated by this example: ``` ""facet_results"": { ""tags"": { ""name"": ""tags"", ""type"": ""array"", ""results"": [...], ""hideable"": false, ""toggle_url"": ""/fixtures/facetable.json?_facet=tags&_trace=1&_nosuggest=1"", ""truncated"": false }, ""created"": { ""name"": ""created"", ""type"": ""date"", ""results"": [...] ``` This was the cause of the bug in #625 - the each of those objects is keyed by the name of the column, which left no room for faceting the same column once by date and once by column value.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082584499,Redesign `facet_results` JSON structure prior to Datasette 1.0, https://github.com/simonw/datasette/issues/625#issuecomment-996130862,https://api.github.com/repos/simonw/datasette/issues/625,996130862,IC_kwDOBm6k_c47X8Au,9599,simonw,2021-12-16T19:44:48Z,2021-12-16T19:44:48Z,OWNER,"Decision: as an initial fix I'm going to de-duplicate those keys by using `tags__array` etc - with a `_2` on the end if that key is already used. I'll open a separate issue to redesign this better for Datasette 1.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/625#issuecomment-996121736,https://api.github.com/repos/simonw/datasette/issues/625,996121736,IC_kwDOBm6k_c47X5yI,9599,simonw,2021-12-16T19:37:08Z,2021-12-16T19:37:08Z,OWNER,"Really `facet_results` here should be an array of objects, not an object that maps poorly designed string keys to those objects.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/625#issuecomment-996119954,https://api.github.com/repos/simonw/datasette/issues/625,996119954,IC_kwDOBm6k_c47X5WS,9599,simonw,2021-12-16T19:36:01Z,2021-12-16T19:36:11Z,OWNER,"Datasette's own HTML rendering code doesn't actually use the keys in `facet_results` - it instead loops through `sorted_facet_results` which is defined like this: https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/views/table.py#L937-L941 And used like this: https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/templates/table.html#L154-L156","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/625#issuecomment-996118401,https://api.github.com/repos/simonw/datasette/issues/625,996118401,IC_kwDOBm6k_c47X4-B,9599,simonw,2021-12-16T19:34:28Z,2021-12-16T19:34:55Z,OWNER,"The big question here is do I break any existing clients of the `""facet_results""` JSON API? It's still pre-1.0 so I could break them, but I've also built my own code against this in the past so it's likely other people have too. If I don't break them, I will instead need to come up with a naming convention for those keys - something like `""tags__array""` for example. As well as a way to ensure that a column called `tags__array` doesn't end up conflicting with the `tags__array` key!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/1557#issuecomment-996115949,https://api.github.com/repos/simonw/datasette/issues/1557,996115949,IC_kwDOBm6k_c47X4Xt,9599,simonw,2021-12-16T19:30:55Z,2021-12-16T19:30:55Z,OWNER,"Demo: compare https://latest.datasette.io/fixtures/facetable?_facet=_city_id&_nosuggest=1 to https://latest.datasette.io/fixtures/facetable?_facet=_city_id Documentation: bottom of https://docs.datasette.io/en/latest/json_api.html#special-table-arguments","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082564912,`?_nosuggest=1` parameter for disabling facet suggestions on table view, https://github.com/simonw/datasette/issues/1556#issuecomment-996104214,https://api.github.com/repos/simonw/datasette/issues/1556,996104214,IC_kwDOBm6k_c47X1gW,9599,simonw,2021-12-16T19:15:00Z,2021-12-16T19:15:28Z,OWNER,"Demo: https://latest.datasette.io/fixtures/facetable?_facet=planet_int&_facet=_city_id&_facet=created#facet-created <img width=""756"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/146434231-a7e1800b-eeef-4e5b-83e2-3b466c84b51b.png""> ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1081318247,"Show count of facet values always, not just for `?_facet_size=max`", https://github.com/simonw/datasette/issues/1553#issuecomment-996103956,https://api.github.com/repos/simonw/datasette/issues/1553,996103956,IC_kwDOBm6k_c47X1cU,9599,simonw,2021-12-16T19:14:38Z,2021-12-16T19:14:38Z,OWNER,This is a really interesting idea - kind of similar to how many APIs include custom HTTP headers informing of rate-limits.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1079111498,if csv export is truncated in non streaming mode set informative response header, https://github.com/simonw/datasette/issues/625#issuecomment-996100774,https://api.github.com/repos/simonw/datasette/issues/625,996100774,IC_kwDOBm6k_c47X0qm,9599,simonw,2021-12-16T19:10:01Z,2021-12-16T19:10:48Z,OWNER,"I think the problem here may be in the design of the JSON returned by facets. It looks like this: ``` ""facet_results"": { ""tags"": { ""name"": ""tags"", ""type"": ""array"", ""results"": [...], ""hideable"": false, ""toggle_url"": ""/fixtures/facetable.json?_facet=tags&_trace=1&_nosuggest=1"", ""truncated"": false }, ""created"": { ""name"": ""created"", ""type"": ""date"", ""results"": [...] ``` The problem then is that the `tags` key is over-ridden by the second facet with a different type against the same column name! https://latest-with-plugins.datasette.io/fixtures/facetable?_trace=1&_facet=created&_facet_date=created&_facet_array=tags&_facet=tags confirms that the SQL queries for those facets are being executed - but the final JSON doesn't show them on https://latest-with-plugins.datasette.io/fixtures/facetable.json?_trace=1&_facet=created&_facet_date=created&_facet_array=tags&_facet=tags They're not available in the template context either: https://latest-with-plugins.datasette.io/fixtures/facetable?_facet=created&_facet_date=created&_facet_array=tags&_facet=tags&_context=1","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/625#issuecomment-996093884,https://api.github.com/repos/simonw/datasette/issues/625,996093884,IC_kwDOBm6k_c47Xy-8,9599,simonw,2021-12-16T19:00:28Z,2021-12-16T19:00:28Z,OWNER,Implementing #1552 has made a fix for this bug even more important.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/1552#issuecomment-996084899,https://api.github.com/repos/simonw/datasette/issues/1552,996084899,IC_kwDOBm6k_c47Xwyj,9599,simonw,2021-12-16T18:48:14Z,2021-12-16T18:48:14Z,OWNER,Updated documentation: https://github.com/simonw/datasette/blob/20a2ed6bec367d2f6759be4a879364a72780b59d/docs/facets.rst#facets-in-metadatajson,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`), https://github.com/simonw/datasette/issues/1552#issuecomment-996077053,https://api.github.com/repos/simonw/datasette/issues/1552,996077053,IC_kwDOBm6k_c47Xu39,9599,simonw,2021-12-16T18:36:41Z,2021-12-16T18:36:41Z,OWNER,"... actually no, I WILL document this, because not documenting this is what got us to this point in the first place!","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 1, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`), https://github.com/simonw/datasette/issues/1552#issuecomment-996076373,https://api.github.com/repos/simonw/datasette/issues/1552,996076373,IC_kwDOBm6k_c47XutV,9599,simonw,2021-12-16T18:35:40Z,2021-12-16T18:35:40Z,OWNER,"I'm going to ship your fix now, but I'm not going to add this to the documentation yet because I hope to improve the design prior to Datasette 1.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`), https://github.com/simonw/datasette/issues/1552#issuecomment-996046304,https://api.github.com/repos/simonw/datasette/issues/1552,996046304,IC_kwDOBm6k_c47XnXg,9599,simonw,2021-12-16T17:53:40Z,2021-12-16T18:16:12Z,OWNER,"I'm also not convinced that this configuration syntax is right. It's a bit weird having a `""facets""` list that can either by column-name-strings or `{""type-of-facet"": ""column-name""}` objects. Maybe there's a better design for this? Part of the problem here is that facets were designed to accept optional extra configuration - partly to support `m2m` facets in #495 - but I haven't actually shipped any facets that use that ability. Facet by delimiter would be a good one to exercise that ability: - #510","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`), https://github.com/simonw/datasette/issues/1552#issuecomment-996045776,https://api.github.com/repos/simonw/datasette/issues/1552,996045776,IC_kwDOBm6k_c47XnPQ,9599,simonw,2021-12-16T17:52:54Z,2021-12-16T17:52:54Z,OWNER,"I tried that fix you suggested and now this `metadata.json` does the right thing: ```json { ""databases"": { ""fixtures"": { ""tables"": { ""facetable"": { ""facets"": [ { ""array"": ""tags"" } ] } } } } } ``` It does further highlight the bug in #625 though - since then if you try to add `?_facet=tags` to facet by tags treating them NOT as an array your request to do so is ignored.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`), https://github.com/simonw/datasette/issues/1552#issuecomment-996034408,https://api.github.com/repos/simonw/datasette/issues/1552,996034408,IC_kwDOBm6k_c47Xkdo,9599,simonw,2021-12-16T17:37:37Z,2021-12-16T17:37:37Z,OWNER,"I think you're right! I had completely forgotten that piece of code. This just turned into a bug fix and a documentation update. Thanks for the research!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`), https://github.com/simonw/datasette/issues/262#issuecomment-995034911,https://api.github.com/repos/simonw/datasette/issues/262,995034911,IC_kwDOBm6k_c47Twcf,9599,simonw,2021-12-15T18:03:46Z,2021-12-15T18:03:56Z,OWNER,"This is relevant to the big refactor in: - #1518","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,Add ?_extra= mechanism for requesting extra properties in JSON, https://github.com/simonw/datasette/issues/1552#issuecomment-995034143,https://api.github.com/repos/simonw/datasette/issues/1552,995034143,IC_kwDOBm6k_c47TwQf,9599,simonw,2021-12-15T18:02:53Z,2021-12-15T18:02:53Z,OWNER,"This is definitely a missing feature. The ""different types of facet"" stuff feels incomplete to me generally - this is one issue, but this one as well: - #625","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`), https://github.com/simonw/datasette/issues/1423#issuecomment-995023410,https://api.github.com/repos/simonw/datasette/issues/1423,995023410,IC_kwDOBm6k_c47Ttoy,9599,simonw,2021-12-15T17:48:40Z,2021-12-15T17:48:40Z,OWNER,You've caused me to rethink this feature - I no longer think there's value in only showing these numbers if `?_facet_size=max` as opposed to all of the time. New issue coming up.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 1, ""eyes"": 0}",962391325,Show count of facet values if ?_facet_size=max, https://github.com/simonw/datasette/issues/1542#issuecomment-995022217,https://api.github.com/repos/simonw/datasette/issues/1542,995022217,IC_kwDOBm6k_c47TtWJ,9599,simonw,2021-12-15T17:47:07Z,2021-12-15T17:47:07Z,OWNER,"This does make sense to me. I've been hoping to significantly improve the way JavaScript plugins work - there are some notes on that here: - #983 Encouraging plugins such as `datasette-cluster-map` to emit events that can then be listened to by other plugins is a really interesting idea that I hadn't considered.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1072106103,feature request: order and dependency of plugins (that use js), https://github.com/simonw/datasette/issues/1518#issuecomment-994085710,https://api.github.com/repos/simonw/datasette/issues/1518,994085710,IC_kwDOBm6k_c47QItO,9599,simonw,2021-12-14T22:03:16Z,2021-12-14T22:04:28Z,OWNER,"There are actually four forms of SQL query used by the table page: - `from_sql` - just the `from table_name where ...` - `sql_no_order_no_limit` - used for faceting, `""select {select_all_columns} from {table_name} {where}""` - `sql` - the above but with order and limit clauses: `""select {select_specified_columns} from {table_name} {where}{order_by} limit {page_size}{offset}""` - `count_sql` used for the count, built out of `from_sql`: `""select count(*) {from_sql}""` I'm tempted to encapsulate those in a `Query` class.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-994042389,https://api.github.com/repos/simonw/datasette/issues/1518,994042389,IC_kwDOBm6k_c47P-IV,9599,simonw,2021-12-14T21:35:53Z,2021-12-14T21:35:53Z,OWNER,"Maybe a better way to approach this would be to focus on the JSON side of things - try to get a basic JSON version with `?_extra=` support working, then eventually build that up to the point where it can power the HTML version.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/621#issuecomment-994005634,https://api.github.com/repos/simonw/datasette/issues/621,994005634,IC_kwDOBm6k_c47P1KC,9599,simonw,2021-12-14T21:02:50Z,2021-12-14T21:02:50Z,OWNER,"This would also mean that an extra text input box could be easily shown on the page. https://latest-with-plugins.datasette.io/fixtures/roadside_attractions?_through={""table"":""roadside_attraction_characteristics"",""column"":""characteristic_id"",""value"":""1""} but with the annotated box added (and made to look good): <img width=""737"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/146078853-d66e34e0-73c9-4c35-9414-5aff3bf825ee.png""> ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520681725,Syntax for ?_through= that works as a form field, https://github.com/simonw/datasette/issues/621#issuecomment-993958242,https://api.github.com/repos/simonw/datasette/issues/621,993958242,IC_kwDOBm6k_c47Ppli,9599,simonw,2021-12-14T20:33:25Z,2021-12-14T20:33:56Z,OWNER,"Alternative idea: since current syntax is: `?_through={""table"":""roadside_attraction_characteristics"",""column"":""characteristic_id"",""value"":""1""}` The form-encoding-friendly syntax could be: `?_through.{""table"":""roadside_attraction_characteristics"",""column"":""characteristic_id""}=1` Which is more consistent than the array proposal: `?_through.[""roadside_attraction_characteristics"",""characteristic_id""]=1`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520681725,Syntax for ?_through= that works as a form field, https://github.com/simonw/datasette/issues/621#issuecomment-993813210,https://api.github.com/repos/simonw/datasette/issues/621,993813210,IC_kwDOBm6k_c47PGLa,9599,simonw,2021-12-14T17:30:13Z,2021-12-14T20:23:57Z,OWNER,"Might be able to create a web form that's unambiguous using: `https://latest.datasette.io/fixtures/roadside_attractions?_through.[""roadside_attraction_characteristics"",""characteristic_id""]=1` So: ```html <input type=""text"" name=""_through.["roadside_attraction_characteristics","characteristic_id"]' value=""1""> ``` I'm pretty confident this is allowed by the HTML specification. This works: ```html <form action=""https://httpbin.org/get""> <input type=""text"" name='_through.["roadside_attraction_characteristics","characteristic_id"]' value=""1""> <input type=""submit""> </form> ``` ASGI parsing seems to work too: https://latest-with-plugins.datasette.io/-/asgi-scope?_through.[%22roadside_attraction_characteristics%22%2C%22characteristic_id%22]=1","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520681725,Syntax for ?_through= that works as a form field, https://github.com/simonw/datasette/issues/1518#issuecomment-993794247,https://api.github.com/repos/simonw/datasette/issues/1518,993794247,IC_kwDOBm6k_c47PBjH,9599,simonw,2021-12-14T17:09:40Z,2021-12-14T17:09:40Z,OWNER,- `table_actions` should be an extra.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-993000787,https://api.github.com/repos/simonw/datasette/issues/1518,993000787,IC_kwDOBm6k_c47L_1T,9599,simonw,2021-12-13T23:19:20Z,2021-12-14T17:06:05Z,OWNER,"Useful old comment here: https://github.com/simonw/datasette/issues/617#issuecomment-552253893 > As noted in [#621 (comment)](https://github.com/simonw/datasette/issues/621#issuecomment-552253208) a common pattern in this method is blocks of code that append new items to the `where_clauses`, `params` and `extra_human_descriptions` arrays. This is a useful refactoring opportunity. > > Code that fits this pattern: > > * The code that builds based on the filters: `where_clauses, params = filters.build_where_clauses(table)` and `human_description_en = filters.human_description_en(extra=extra_human_descriptions)` > * Code that handles `?_where=`: `where_clauses.extend(request.args[""_where""])` - though note that this also appends to a `extra_wheres_for_ui` array which nothing else uses > * The `_through=` code, see [Syntax for ?_through= that works as a form field #621](https://github.com/simonw/datasette/issues/621) for details > * The code that deals with `?_search=` FTS > > The keyset pagination code modifies `where_clauses` and `params` too, but I don't think it's quite going to work with the same abstraction that would cover the above examples. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/pull/1554#issuecomment-993006521,https://api.github.com/repos/simonw/datasette/issues/1554,993006521,IC_kwDOBm6k_c47MBO5,9599,simonw,2021-12-13T23:28:47Z,2021-12-13T23:28:47Z,OWNER,"That's frustrating: you can only attach comments to lines that were changed in the PR or are within about 3-4 lines of them: ![comments](https://user-images.githubusercontent.com/9599/145905357-5d8873f5-99c9-4b46-b4d5-35d38f5cb686.gif) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079129258,TableView refactor, https://github.com/simonw/datasette/issues/1518#issuecomment-992833868,https://api.github.com/repos/simonw/datasette/issues/1518,992833868,IC_kwDOBm6k_c47LXFM,9599,simonw,2021-12-13T19:59:17Z,2021-12-13T19:59:17Z,OWNER,"Built a new plugin to help with this work by improving the display of `?_trace=1` output: https://datasette.io/plugins/datasette-pretty-traces ![image](https://user-images.githubusercontent.com/9599/145879751-36621f43-ba68-4ccd-b14b-379ed8f2111a.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-991978789,https://api.github.com/repos/simonw/datasette/issues/1518,991978789,IC_kwDOBm6k_c47IGUl,9599,simonw,2021-12-12T22:04:19Z,2021-12-12T22:04:19Z,OWNER,Idea: in JSON output include a `warnings` block listing any _ parameters that were not recognized.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1551#issuecomment-991960719,https://api.github.com/repos/simonw/datasette/issues/1551,991960719,IC_kwDOBm6k_c47IB6P,9599,simonw,2021-12-12T19:58:17Z,2021-12-12T19:58:17Z,OWNER,"Here's an example of the difference that causes: ```pycon >>> import urllib.parse >>> urllib.parse.parse_qs(""foo=bar"") {'foo': ['bar']} >>> urllib.parse.parse_qs(""foo=bar&baz="") {'foo': ['bar']} >>> urllib.parse.parse_qs(""foo=bar&baz="", keep_blank_values=True) {'foo': ['bar'], 'baz': ['']} ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077893013,`keep_blank_values=True` when parsing `request.args`, https://github.com/simonw/datasette/issues/1551#issuecomment-991960416,https://api.github.com/repos/simonw/datasette/issues/1551,991960416,IC_kwDOBm6k_c47IB1g,9599,simonw,2021-12-12T19:56:12Z,2021-12-12T19:56:12Z,OWNER,"Python documentation for `parse_qs`: https://docs.python.org/3/library/urllib.parse.html#urllib.parse.parse_qs > The optional argument *keep_blank_values* is a flag indicating whether blank values in percent-encoded queries should be treated as blank strings. A true value indicates that blanks should be retained as blank strings. The default false value indicates that blank values are to be ignored and treated as if they were not included.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077893013,`keep_blank_values=True` when parsing `request.args`, https://github.com/simonw/datasette/issues/1551#issuecomment-991960179,https://api.github.com/repos/simonw/datasette/issues/1551,991960179,IC_kwDOBm6k_c47IBxz,9599,simonw,2021-12-12T19:54:45Z,2021-12-12T19:54:45Z,OWNER,This is technically a backwards-incompatible for any plugins that use `request.args` - but it's unlikely to break anything. At any rate this needs to happen before Datasette 1.0!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077893013,`keep_blank_values=True` when parsing `request.args`,