{"html_url": "https://github.com/simonw/datasette/pull/1835#issuecomment-1270586897", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1835", "id": 1270586897, "node_id": "IC_kwDOBm6k_c5Lu54R", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-10-06T19:34:00Z", "updated_at": "2022-10-06T19:34:00Z", "author_association": "OWNER", "body": "Wow, great catch! The whole point of inspect data was to avoid this kind of expensive operation on startup so this makes total sense - I had no idea Datasette was still trying to hash a giant file every time the server started.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1400121355, "label": "use inspect data for hash and file size"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1480#issuecomment-1269275153", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1480", "id": 1269275153, "node_id": "IC_kwDOBm6k_c5Lp5oR", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-10-06T03:54:33Z", "updated_at": "2022-10-06T03:54:33Z", "author_association": "OWNER", "body": "I've been having success using Fly recently for a project which I thought would be too large for Cloud Run. I wrote about that here:\r\n\r\n- https://simonwillison.net/2022/Sep/5/laion-aesthetics-weeknotes/", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1015646369, "label": "Exceeding Cloud Run memory limits when deploying a 4.8G database"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1832#issuecomment-1267925830", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1832", "id": 1267925830, "node_id": "IC_kwDOBm6k_c5LkwNG", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-10-05T04:31:57Z", "updated_at": "2022-10-05T04:31:57Z", "author_association": "OWNER", "body": "Turns out this already works - `__bool__` falls back on `__len__`: https://docs.python.org/3/reference/datamodel.html#object.__bool__\r\n\r\n> When this method is not defined, [`__len__()`](https://docs.python.org/3/reference/datamodel.html#object.__len__ \"object.__len__\") is called, if it is defined, and the object is considered true if its result is nonzero.\r\n\r\nI'll add a test to demonstrate this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1397193691, "label": "__bool__ method on Results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1832#issuecomment-1267918117", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1832", "id": 1267918117, "node_id": "IC_kwDOBm6k_c5LkuUl", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-10-05T04:19:52Z", "updated_at": "2022-10-05T04:19:52Z", "author_association": "OWNER", "body": "Code can go here:\r\n\r\nhttps://github.com/simonw/datasette/blob/b6ba117b7978b58b40e3c3c2b723b92c3010ed53/datasette/database.py#L511-L515\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1397193691, "label": "__bool__ method on Results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1829#issuecomment-1267709546", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1829", "id": 1267709546, "node_id": "IC_kwDOBm6k_c5Lj7Zq", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-10-04T23:19:24Z", "updated_at": "2022-10-04T23:21:07Z", "author_association": "OWNER", "body": "There's also a `check_visibility()` helper which I'm not using in these particular cases but which may be relevant. It's called like this:\r\n\r\nhttps://github.com/simonw/datasette/blob/4218c9cd742b79b1e3cb80878e42b7e39d16ded2/datasette/views/database.py#L65-L77\r\n\r\nAnd is defined here: https://github.com/simonw/datasette/blob/4218c9cd742b79b1e3cb80878e42b7e39d16ded2/datasette/app.py#L694-L710\r\n\r\nIt's actually documented as a public method here: https://docs.datasette.io/en/stable/internals.html#await-check-visibility-actor-action-resource-none\r\n\r\n> This convenience method can be used to answer the question \"should this item be considered private, in that it is visible to me but it is not visible to anonymous users?\"\r\n>\r\n> It returns a tuple of two booleans, `(visible, private)`. `visible` indicates if the actor can see this resource. `private` will be `True` if an anonymous user would not be able to view the resource.\r\n\r\nNote that this documented method cannot actually do the right thing - because it's not being given the multiple permissions that need to be checked in order to completely answer the question.\r\n\r\nSo I probably need to redesign that method a bit.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1396948693, "label": "Table/database that is private due to inherited permissions does not show padlock"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1829#issuecomment-1267708232", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1829", "id": 1267708232, "node_id": "IC_kwDOBm6k_c5Lj7FI", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-10-04T23:17:36Z", "updated_at": "2022-10-04T23:17:36Z", "author_association": "OWNER", "body": "Here's the relevant code from the table page:\r\n\r\nhttps://github.com/simonw/datasette/blob/4218c9cd742b79b1e3cb80878e42b7e39d16ded2/datasette/views/table.py#L215-L227\r\n\r\nNote how `ensure_permissions()` there takes the table, database and instance into account... but the `private` assignment  (used to decide if the padlock should display or not) only considers the `view-table` check.\r\n\r\nHere's the same code for the database page:\r\n\r\nhttps://github.com/simonw/datasette/blob/4218c9cd742b79b1e3cb80878e42b7e39d16ded2/datasette/views/database.py#L139-L141\r\n\r\nAnd for canned query pages:\r\n\r\nhttps://github.com/simonw/datasette/blob/4218c9cd742b79b1e3cb80878e42b7e39d16ded2/datasette/views/database.py#L228-L240\r\n\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1396948693, "label": "Table/database that is private due to inherited permissions does not show padlock"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/485#issuecomment-1264769569", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/485", "id": 1264769569, "node_id": "IC_kwDOBm6k_c5LYtoh", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-10-03T00:04:42Z", "updated_at": "2022-10-03T00:04:42Z", "author_association": "OWNER", "body": "I love these tips - tools that can compile a simple machine learning model to a SQL query! Would be pretty cool if I could bundle a model in Datasette itself as a big in-memory SQLite SQL query:\r\n\r\n- https://github.com/Chryzanthemum/xgb2sql\r\n- https://github.com/konstantint/SKompiler", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 447469253, "label": "Improvements to table label detection "}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1805#issuecomment-1264753894", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1805", "id": 1264753894, "node_id": "IC_kwDOBm6k_c5LYpzm", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-10-02T23:02:54Z", "updated_at": "2022-10-02T23:02:54Z", "author_association": "OWNER", "body": "I'm tempted to add `word-wrap: anywhere` only to links that are know to be longer than a certain threshold.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1363552780, "label": "truncate_cells_html does not work for links?"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1805#issuecomment-1264753725", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1805", "id": 1264753725, "node_id": "IC_kwDOBm6k_c5LYpw9", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-10-02T23:02:17Z", "updated_at": "2022-10-02T23:02:17Z", "author_association": "OWNER", "body": "After reverting `word--wrap anywhere` https://latest.datasette.io/_memory?sql=select+%27https%3A%2F%2Fexample.com%2Faaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.jpg%27+as+truncated now looks like this, which isn't as good:\r\n\r\n<img width=\"1640\" alt=\"image\" src=\"https://user-images.githubusercontent.com/9599/193480193-050bd5b4-720c-41e6-be71-1c1bb9c4befd.png\">\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1363552780, "label": "truncate_cells_html does not work for links?"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1828#issuecomment-1264753439", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1828", "id": 1264753439, "node_id": "IC_kwDOBm6k_c5LYpsf", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-10-02T23:01:17Z", "updated_at": "2022-10-02T23:01:17Z", "author_association": "OWNER", "body": "That change deployed and https://github-to-sqlite.dogsheep.net/github/commits now looks like this:\r\n\r\n<img width=\"1388\" alt=\"image\" src=\"https://user-images.githubusercontent.com/9599/193480158-de81ac0a-5cb2-4d53-a75c-025c78f293ee.png\">\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1393903845, "label": "word-wrap: anywhere resulting in weird display"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1828#issuecomment-1264738081", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1828", "id": 1264738081, "node_id": "IC_kwDOBm6k_c5LYl8h", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-10-02T21:34:37Z", "updated_at": "2022-10-02T21:34:37Z", "author_association": "OWNER", "body": "I'm running a build of that demo instance here (takes ~30m) https://github.com/dogsheep/github-to-sqlite/actions/runs/3170164705", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1393903845, "label": "word-wrap: anywhere resulting in weird display"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/485#issuecomment-1264737290", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/485", "id": 1264737290, "node_id": "IC_kwDOBm6k_c5LYlwK", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-10-02T21:29:59Z", "updated_at": "2022-10-02T21:29:59Z", "author_association": "OWNER", "body": "To clarify: the feature this issue is talking about relates to the way Datasette automatically displays foreign key relationships, for example on this page: https://github-to-sqlite.dogsheep.net/github/commits\r\n\r\n<img width=\"1233\" alt=\"image\" src=\"https://user-images.githubusercontent.com/9599/193476985-d41148cf-2b2f-49b9-b717-e92145afab31.png\">\r\n\r\nEach of those columns is a foreign key to another table. The link text that is displayed there comes from the \"label column\" that has either been configured or automatically detected for that other table.\r\n\r\nI wonder if this could be handled with a tiny machine learning model that's trained to help pick the best label column?\r\n\r\nInputs to that model could include:\r\n\r\n- The names of the columns\r\n- The number of unique values in each column\r\n- The type of each column (or maybe only `TEXT` columns should be considered)\r\n- How many `null` values there are\r\n- Is the column marked as unique?\r\n- What's the average (or median or some other statistic) string length of values in each column?\r\n\r\nOutput would be the most likely label column, or some indicator that no likely candidates had been found.\r\n\r\nMy hunch is that this would be better solved using a few extra heuristics rather than by training a model, but it does feel like an interesting opportunity to experiment with a tiny ML model.\r\n\r\nAsked for tips about this on Twitter: https://twitter.com/simonw/status/1576680930680262658\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 447469253, "label": "Improvements to table label detection "}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1805#issuecomment-1264736537", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1805", "id": 1264736537, "node_id": "IC_kwDOBm6k_c5LYlkZ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-10-02T21:25:37Z", "updated_at": "2022-10-02T21:25:37Z", "author_association": "OWNER", "body": "`word-wrap: anywhere` had some nasty side-effects, removing that:\r\n\r\n- #1828", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1363552780, "label": "truncate_cells_html does not work for links?"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1262920929", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/297", "id": 1262920929, "node_id": "IC_kwDOCGYnMM5LRqTh", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-29T23:06:44Z", "updated_at": "2022-09-29T23:06:44Z", "author_association": "OWNER", "body": "Currently the only other use of `-t` is for this:\r\n```\r\n  -t, --table            Output as a formatted table\r\n```\r\nSo I think it's OK to use it to mean something slightly different for this command, since `sqlite-utils insert` doesn't do any output of data in any format.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 944846776, "label": "Option for importing CSV data using the SQLite .import mechanism"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1262918833", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/297", "id": 1262918833, "node_id": "IC_kwDOCGYnMM5LRpyx", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-29T23:02:52Z", "updated_at": "2022-09-29T23:02:52Z", "author_association": "OWNER", "body": "The other nice thing about having this as a separate command is that I can implement a tiny subset of the overall `sqlite-utils insert` features at first, and then add additional features in subsequent releases.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 944846776, "label": "Option for importing CSV data using the SQLite .import mechanism"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1262917059", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/297", "id": 1262917059, "node_id": "IC_kwDOCGYnMM5LRpXD", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-29T22:59:28Z", "updated_at": "2022-09-29T22:59:28Z", "author_association": "OWNER", "body": "I quite like `sqlite-utils fast-csv` - I think it's clear enough what it does, and running `--help` can clarify if needed.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 944846776, "label": "Option for importing CSV data using the SQLite .import mechanism"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1262915322", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/297", "id": 1262915322, "node_id": "IC_kwDOCGYnMM5LRo76", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-29T22:57:31Z", "updated_at": "2022-09-29T22:57:42Z", "author_association": "OWNER", "body": "Maybe `sqlite-utils fast-csv` is right? Not entirely clear that's an insert though as opposed to a faster version of in-memory querying in the style of `sqlite-utils memory`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 944846776, "label": "Option for importing CSV data using the SQLite .import mechanism"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1262914416", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/297", "id": 1262914416, "node_id": "IC_kwDOCGYnMM5LRotw", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-29T22:56:53Z", "updated_at": "2022-09-29T22:56:53Z", "author_association": "OWNER", "body": "Potential names/designs:\r\n\r\n- `sqlite-utils fast data.db rows rows.csv`\r\n- `sqlite-utils insert-fast data.db rows rows.csv`\r\n- `sqlite-utils fast-csv data.db rows rows.csv`\r\n\r\nOr more interestingly... what if it could accept multiple CSV files to create multiple tables?\r\n\r\n- `sqlite-utils fast data.db rows.csv other.csv`\r\n\r\nWould still need to support creating tables with different names though. Maybe like this:\r\n\r\n- `sqlite-utils fast data.db -t mytable rows.csv -t othertable other.csv`\r\n\r\nI seem to be leaning towards `fast` as the command name, but as a standalone command name it's a bit meaningless - how do we know that's about CSV import and not about fast querying or similar?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 944846776, "label": "Option for importing CSV data using the SQLite .import mechanism"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1262913145", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/297", "id": 1262913145, "node_id": "IC_kwDOCGYnMM5LRoZ5", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-29T22:54:13Z", "updated_at": "2022-09-29T22:54:13Z", "author_association": "OWNER", "body": "After reviewing `sqlite-utils insert --help` I'm confident that MOST of these options wouldn't make sense for a \"fast\" moder that just supports CSV and works by piping directly to the `sqlite3` binary:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/d792dad1cf5f16525da81b1e162fb71d469995f3/docs/cli-reference.rst#L251-L279\r\n\r\nI'm going to implement a separate command instead.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 944846776, "label": "Option for importing CSV data using the SQLite .import mechanism"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/1825#issuecomment-1260368537", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1825", "id": 1260368537, "node_id": "IC_kwDOBm6k_c5LH7KZ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-28T04:21:18Z", "updated_at": "2022-09-28T04:21:18Z", "author_association": "OWNER", "body": "This is great, thank you very much!\r\n\r\nhttps://datasette--1825.org.readthedocs.build/en/1825/deploying.html#running-datasette-using-openrc", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1388227245, "label": "Add documentation for serving via OpenRC"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1826#issuecomment-1260357878", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1826", "id": 1260357878, "node_id": "IC_kwDOBm6k_c5LH4j2", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-28T04:05:45Z", "updated_at": "2022-09-28T04:05:45Z", "author_association": "OWNER", "body": "Though now I notice that the copy right there needs to be updated to reflect the new `row` parameter to `render_cell`!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1388631785, "label": "render_cell documentation example doesn't match the method signature"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1826#issuecomment-1260357583", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1826", "id": 1260357583, "node_id": "IC_kwDOBm6k_c5LH4fP", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-28T04:05:16Z", "updated_at": "2022-09-28T04:05:16Z", "author_association": "OWNER", "body": "This is deliberate. The Datasette plugin system allows you to specify only a subset of the parameters for a hook - in this example, only the `value` is needed so the others can be omitted.\r\n\r\nThere's a note about this at the very top of that documentation page: https://docs.datasette.io/en/stable/plugin_hooks.html#plugin-hooks\r\n\r\n> When you implement a plugin hook you can accept any or all of the parameters that are documented as being passed to that hook.\r\n> \r\n> For example, you can implement the `render_cell` plugin hook like this even though the full documented hook signature is `render_cell(value, column, table, database, datasette)`:\r\n> ```python\r\n> @hookimpl\r\n> def render_cell(value, column):\r\n>     if column == \"stars\":\r\n>         return \"*\" * int(value)\r\n> ```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1388631785, "label": "render_cell documentation example doesn't match the method signature"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/526#issuecomment-1260355224", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/526", "id": 1260355224, "node_id": "IC_kwDOBm6k_c5LH36Y", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-28T04:01:25Z", "updated_at": "2022-09-28T04:01:25Z", "author_association": "OWNER", "body": "The ultimate protection against those memory bombs is to support more streaming output formats. Related issues:\r\n\r\n- #1177 \r\n- #1062", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 459882902, "label": "Stream all results for arbitrary SQL and canned queries"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/526#issuecomment-1259693536", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/526", "id": 1259693536, "node_id": "IC_kwDOBm6k_c5LFWXg", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-27T15:42:55Z", "updated_at": "2022-09-27T15:42:55Z", "author_association": "OWNER", "body": "It's interesting to note WHY the time limit works against this so well.\r\n\r\nThe time limit as-implemented looks like this:\r\n\r\nhttps://github.com/simonw/datasette/blob/5f9f567acbc58c9fcd88af440e68034510fb5d2b/datasette/utils/__init__.py#L181-L201\r\n\r\nThe key here is `conn.set_progress_handler(handler, n)` - which specifies that the handler function should be called every `n` SQLite operations.\r\n\r\nThe handler function then checks to see if too much time has transpired and conditionally cancels the query.\r\n\r\nThis also doubles up as a \"maximum number of operations\" guard, which is what's happening when you attempt to fetch an infinite number of rows from an infinite table.\r\n\r\nThat limit code could even be extended to say \"exit the query after either 5s or 50,000,000 operations\".\r\n\r\nI don't think that's necessary though.\r\n\r\nTo be honest I'm having trouble with the idea of dropping `max_returned_rows` mainly because what Datasette does (allow arbitrary untrusted SQL queries) is dangerous, so I've designed in multiple redundant defence-in-depth mechanisms right from the start.", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 1, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 459882902, "label": "Stream all results for arbitrary SQL and canned queries"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/526#issuecomment-1258906440", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/526", "id": 1258906440, "node_id": "IC_kwDOBm6k_c5LCWNI", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-27T03:04:37Z", "updated_at": "2022-09-27T03:04:37Z", "author_association": "OWNER", "body": "It would be really neat if we could explore this idea in a plugin, but I don't think Datasette has plugin hooks in the right place for that at the moment.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 459882902, "label": "Stream all results for arbitrary SQL and canned queries"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/526#issuecomment-1258905781", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/526", "id": 1258905781, "node_id": "IC_kwDOBm6k_c5LCWC1", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-27T03:03:35Z", "updated_at": "2022-09-27T03:03:47Z", "author_association": "OWNER", "body": "Yes good point, the time limit does already protect against that. I've been contemplating a permissioned-users-only relaxation of that time limit too, and I got that idea mixed up with this one in my head.\r\n\r\nOn that basis maybe this feature would be safe after all? Would need to do some testing, but it may be that the existing time limit provides enough protection here already.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 459882902, "label": "Stream all results for arbitrary SQL and canned queries"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/526#issuecomment-1258864140", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/526", "id": 1258864140, "node_id": "IC_kwDOBm6k_c5LCL4M", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-27T01:55:32Z", "updated_at": "2022-09-27T01:55:32Z", "author_association": "OWNER", "body": "That recursive query is a great example of the kind of thing having a maximum row limit protects against.\r\n\r\nImagine if Datasette CSVs did allow unlimited retrievals. Someone could hit the CSV endpoint for that recursive query and tie up Datasette's SQL connection effectively forever.\r\n\r\nEven if this feature becomes a permission-guarded thing we still need to take that case into account.\r\n\r\nAt the very least it would be good if the query could be cancelled if the client disconnects - so if someone accidentally starts an infinite query they can cancel the request and free up the server resources.\r\n\r\nIt might be a good idea to implement a page that shows \"currently running\" queries and allows users with the right permission to terminate them from that page.\r\n\r\nAnother option: a \"limit of last resource\" - either a very high row limit (10,000,000 perhaps) or even a time limit, saying that all queries will be cancelled if they take longer than thirty minutes or similar.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 459882902, "label": "Stream all results for arbitrary SQL and canned queries"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/526#issuecomment-1258860845", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/526", "id": 1258860845, "node_id": "IC_kwDOBm6k_c5LCLEt", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-27T01:48:31Z", "updated_at": "2022-09-27T01:50:01Z", "author_association": "OWNER", "body": "The protection is supposed to be from this line:\r\n```python\r\nrows = cursor.fetchmany(max_returned_rows + 1) \r\n```\r\nBy capping the call to `.fetchman()` at `max_returned_rows + 1` (the `+ 1` is to allow detection of whether or not there is a next page) I'm ensuring that Datasette never attempts to iterate over a huge result set.\r\n\r\nSQLite and the `sqlite3` library seem to handle this correctly. Here's an example:\r\n\r\n```pycon\r\n>>> import sqlite3\r\n>>> conn = sqlite3.connect(\":memory:\")\r\n>>> cursor = conn.execute(\"\"\"\r\n... with recursive counter(x) as (\r\n...   select 0\r\n...     union\r\n...   select x + 1 from counter\r\n... )\r\n... select * from counter\"\"\")\r\n>>> cursor.fetchmany(10)\r\n[(0,), (1,), (2,), (3,), (4,), (5,), (6,), (7,), (8,), (9,), (10,)]\r\n```\r\n`counter` there is an infinitely long table ([see TIL](https://til.simonwillison.net/sqlite/simple-recursive-cte)) - but we can retrieve the first 10 results without going into an infinite loop.\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 459882902, "label": "Stream all results for arbitrary SQL and canned queries"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/526#issuecomment-1258846992", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/526", "id": 1258846992, "node_id": "IC_kwDOBm6k_c5LCHsQ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-27T01:21:41Z", "updated_at": "2022-09-27T01:21:41Z", "author_association": "OWNER", "body": "My main concern here is that public Datasette instances could easily have all of their available database connections consumed by long-running queries - either accidentally or deliberately.\r\n\r\nI do totally understand the need for this feature though. I think it can absolutely make sense provided it's protected by authentication and permissions.\r\n\r\nMaybe even limit the number of concurrent downloads at once such that there's always at least one database connection free for other requests.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 459882902, "label": "Stream all results for arbitrary SQL and canned queries"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/1823#issuecomment-1258828705", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1823", "id": 1258828705, "node_id": "IC_kwDOBm6k_c5LCDOh", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-27T00:45:46Z", "updated_at": "2022-09-27T00:45:46Z", "author_association": "OWNER", "body": "Also need to do a bit more of an audit to see if there is anywhere else that this style should be applied.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1386917344, "label": "Keyword-only arguments for a bunch of internal methods"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/1823#issuecomment-1258828509", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1823", "id": 1258828509, "node_id": "IC_kwDOBm6k_c5LCDLd", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-27T00:45:26Z", "updated_at": "2022-09-27T00:45:26Z", "author_association": "OWNER", "body": "I should update the documentation to reflect this change.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1386917344, "label": "Keyword-only arguments for a bunch of internal methods"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1822#issuecomment-1258827688", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1822", "id": 1258827688, "node_id": "IC_kwDOBm6k_c5LCC-o", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-27T00:44:04Z", "updated_at": "2022-09-27T00:44:04Z", "author_association": "OWNER", "body": "I'll do this in a PR.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1386854246, "label": "Switch to keyword-only arguments for a bunch of internal methods"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1817#issuecomment-1258818028", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1817", "id": 1258818028, "node_id": "IC_kwDOBm6k_c5LCAns", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-27T00:27:53Z", "updated_at": "2022-09-27T00:27:53Z", "author_association": "OWNER", "body": "Made a start on this:\r\n```diff\r\ndiff --git a/datasette/hookspecs.py b/datasette/hookspecs.py\r\nindex 34e19664..fe0971e5 100644\r\n--- a/datasette/hookspecs.py\r\n+++ b/datasette/hookspecs.py\r\n@@ -31,25 +31,29 @@ def prepare_jinja2_environment(env, datasette):\r\n \r\n \r\n @hookspec\r\n-def extra_css_urls(template, database, table, columns, view_name, request, datasette):\r\n+def extra_css_urls(\r\n+    template, database, table, columns, sql, params, view_name, request, datasette\r\n+):\r\n     \"\"\"Extra CSS URLs added by this plugin\"\"\"\r\n \r\n \r\n @hookspec\r\n-def extra_js_urls(template, database, table, columns, view_name, request, datasette):\r\n+def extra_js_urls(\r\n+    template, database, table, columns, sql, params, view_name, request, datasette\r\n+):\r\n     \"\"\"Extra JavaScript URLs added by this plugin\"\"\"\r\n \r\n \r\n @hookspec\r\n def extra_body_script(\r\n-    template, database, table, columns, view_name, request, datasette\r\n+    template, database, table, columns, sql, params, view_name, request, datasette\r\n ):\r\n     \"\"\"Extra JavaScript code to be included in <script> at bottom of body\"\"\"\r\n \r\n \r\n @hookspec\r\n def extra_template_vars(\r\n-    template, database, table, columns, view_name, request, datasette\r\n+    template, database, table, columns, sql, params, view_name, request, datasette\r\n ):\r\n     \"\"\"Extra template variables to be made available to the template - can return dict or callable or awaitable\"\"\"\r\n```\r\n```diff\r\ndiff --git a/datasette/app.py b/datasette/app.py\r\nindex 03d1dacc..2f3a46fe 100644\r\n--- a/datasette/app.py\r\n+++ b/datasette/app.py\r\n@@ -1036,7 +1036,9 @@ class Datasette:\r\n \r\n         return await template.render_async(template_context)\r\n \r\n-    async def _asset_urls(self, key, template, context, request, view_name):\r\n+    async def _asset_urls(\r\n+        self, key, template, context, request, view_name, sql, params\r\n+    ):\r\n         # Flatten list-of-lists from plugins:\r\n         seen_urls = set()\r\n         collected = []\r\n@@ -1045,6 +1047,8 @@ class Datasette:\r\n             database=context.get(\"database\"),\r\n             table=context.get(\"table\"),\r\n             columns=context.get(\"columns\"),\r\n+            sql=sql,\r\n+            params=params,\r\n             view_name=view_name,\r\n             request=request,\r\n             datasette=self,\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1384273985, "label": "Expose `sql` and `params` arguments to various plugin hooks"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1822#issuecomment-1258760299", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1822", "id": 1258760299, "node_id": "IC_kwDOBm6k_c5LByhr", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-26T23:25:12Z", "updated_at": "2022-09-26T23:25:55Z", "author_association": "OWNER", "body": "A start:\r\n```diff\r\ndiff --git a/datasette/utils/asgi.py b/datasette/utils/asgi.py\r\nindex 8a2fa060..41ade961 100644\r\n--- a/datasette/utils/asgi.py\r\n+++ b/datasette/utils/asgi.py\r\n@@ -118,7 +118,7 @@ class Request:\r\n         return dict(parse_qsl(body.decode(\"utf-8\"), keep_blank_values=True))\r\n \r\n     @classmethod\r\n-    def fake(cls, path_with_query_string, method=\"GET\", scheme=\"http\", url_vars=None):\r\n+    def fake(cls, path_with_query_string, *, method=\"GET\", scheme=\"http\", url_vars=None):\r\n         \"\"\"Useful for constructing Request objects for tests\"\"\"\r\n         path, _, query_string = path_with_query_string.partition(\"?\")\r\n         scope = {\r\n@@ -204,7 +204,7 @@ class AsgiWriter:\r\n         )\r\n \r\n \r\n-async def asgi_send_json(send, info, status=200, headers=None):\r\n+async def asgi_send_json(send, info, *, status=200, headers=None):\r\n     headers = headers or {}\r\n     await asgi_send(\r\n         send,\r\n@@ -215,7 +215,7 @@ async def asgi_send_json(send, info, status=200, headers=None):\r\n     )\r\n \r\n \r\n-async def asgi_send_html(send, html, status=200, headers=None):\r\n+async def asgi_send_html(send, html, *, status=200, headers=None):\r\n     headers = headers or {}\r\n     await asgi_send(\r\n         send,\r\n@@ -226,7 +226,7 @@ async def asgi_send_html(send, html, status=200, headers=None):\r\n     )\r\n \r\n \r\n-async def asgi_send_redirect(send, location, status=302):\r\n+async def asgi_send_redirect(send, location, *, status=302):\r\n     await asgi_send(\r\n         send,\r\n         \"\",\r\n@@ -236,12 +236,12 @@ async def asgi_send_redirect(send, location, status=302):\r\n     )\r\n \r\n \r\n-async def asgi_send(send, content, status, headers=None, content_type=\"text/plain\"):\r\n+async def asgi_send(send, content, status, *, headers=None, content_type=\"text/plain\"):\r\n     await asgi_start(send, status, headers, content_type)\r\n     await send({\"type\": \"http.response.body\", \"body\": content.encode(\"utf-8\")})\r\n \r\n \r\n-async def asgi_start(send, status, headers=None, content_type=\"text/plain\"):\r\n+async def asgi_start(send, status, *, headers=None, content_type=\"text/plain\"):\r\n     headers = headers or {}\r\n     # Remove any existing content-type header\r\n     headers = {k: v for k, v in headers.items() if k.lower() != \"content-type\"}\r\n@@ -259,7 +259,7 @@ async def asgi_start(send, status, headers=None, content_type=\"text/plain\"):\r\n \r\n \r\n async def asgi_send_file(\r\n-    send, filepath, filename=None, content_type=None, chunk_size=4096, headers=None\r\n+    send, filepath, filename=None, *, content_type=None, chunk_size=4096, headers=None\r\n ):\r\n     headers = headers or {}\r\n     if filename:\r\n@@ -284,7 +284,7 @@ async def asgi_send_file(\r\n             )\r\n \r\n \r\n-def asgi_static(root_path, chunk_size=4096, headers=None, content_type=None):\r\n+def asgi_static(root_path, *, chunk_size=4096, headers=None, content_type=None):\r\n     root_path = Path(root_path)\r\n \r\n     async def inner_static(request, send):\r\n@@ -313,7 +313,7 @@ def asgi_static(root_path, chunk_size=4096, headers=None, content_type=None):\r\n \r\n \r\n class Response:\r\n-    def __init__(self, body=None, status=200, headers=None, content_type=\"text/plain\"):\r\n+    def __init__(self, body=None, *, status=200, headers=None, content_type=\"text/plain\"):\r\n         self.body = body\r\n         self.status = status\r\n         self.headers = headers or {}\r\n@@ -346,6 +346,7 @@ class Response:\r\n         self,\r\n         key,\r\n         value=\"\",\r\n+        *,\r\n         max_age=None,\r\n         expires=None,\r\n         path=\"/\",\r\n@@ -374,7 +375,7 @@ class Response:\r\n         self._set_cookie_headers.append(cookie.output(header=\"\").strip())\r\n \r\n     @classmethod\r\n-    def html(cls, body, status=200, headers=None):\r\n+    def html(cls, body, *, status=200, headers=None):\r\n         return cls(\r\n             body,\r\n             status=status,\r\n@@ -383,7 +384,7 @@ class Response:\r\n         )\r\n \r\n     @classmethod\r\n-    def text(cls, body, status=200, headers=None):\r\n+    def text(cls, body, *, status=200, headers=None):\r\n         return cls(\r\n             str(body),\r\n             status=status,\r\n@@ -392,7 +393,7 @@ class Response:\r\n         )\r\n \r\n     @classmethod\r\n-    def json(cls, body, status=200, headers=None, default=None):\r\n+    def json(cls, body, *, status=200, headers=None, default=None):\r\n         return cls(\r\n             json.dumps(body, default=default),\r\n             status=status,\r\n@@ -401,7 +402,7 @@ class Response:\r\n         )\r\n \r\n     @classmethod\r\n-    def redirect(cls, path, status=302, headers=None):\r\n+    def redirect(cls, path, *, status=302, headers=None):\r\n         headers = headers or {}\r\n         headers[\"Location\"] = path\r\n         return cls(\"\", status=status, headers=headers)\r\n@@ -412,6 +413,7 @@ class AsgiFileDownload:\r\n         self,\r\n         filepath,\r\n         filename=None,\r\n+        *,\r\n         content_type=\"application/octet-stream\",\r\n         headers=None,\r\n     ):\r\n```\r\n```diff\r\ndiff --git a/datasette/app.py b/datasette/app.py\r\nindex 03d1dacc..4d4e5584 100644\r\n--- a/datasette/app.py\r\n+++ b/datasette/app.py\r\n@@ -190,6 +190,7 @@ class Datasette:\r\n     def __init__(\r\n         self,\r\n         files=None,\r\n+        *,\r\n         immutables=None,\r\n         cache_headers=True,\r\n         cors=False,\r\n ```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1386854246, "label": "Switch to keyword-only arguments for a bunch of internal methods"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1822#issuecomment-1258757544", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1822", "id": 1258757544, "node_id": "IC_kwDOBm6k_c5LBx2o", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-26T23:21:23Z", "updated_at": "2022-09-26T23:21:23Z", "author_association": "OWNER", "body": "Everything on https://docs.datasette.io/en/stable/internals.html that uses keyword arguments should do this I think.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1386854246, "label": "Switch to keyword-only arguments for a bunch of internal methods"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1817#issuecomment-1258756231", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1817", "id": 1258756231, "node_id": "IC_kwDOBm6k_c5LBxiH", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-26T23:19:34Z", "updated_at": "2022-09-26T23:19:34Z", "author_association": "OWNER", "body": "This is a good idea - it's something I should do before Datasette 1.0.\r\n\r\nI was a tiny bit worried about compatibility (Datasette is 3.7+) but it looks like they have been in Python since 3.0!", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 1, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1384273985, "label": "Expose `sql` and `params` arguments to various plugin hooks"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1819#issuecomment-1258754105", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1819", "id": 1258754105, "node_id": "IC_kwDOBm6k_c5LBxA5", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-26T23:16:15Z", "updated_at": "2022-09-26T23:16:15Z", "author_association": "OWNER", "body": "Demo: https://latest.datasette.io/_memory?sql=with+recursive+counter(x)+as+(%0D%0A++select+0%0D%0A++++union%0D%0A++select+x+%2B+1+from+counter%0D%0A)%2C%0D%0Ablah+as+(select+*+from+counter+limit+5000000)%0D%0Aselect+count(*)+from+blah", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1385026210, "label": "Preserve query on timeout"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1819#issuecomment-1258746600", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1819", "id": 1258746600, "node_id": "IC_kwDOBm6k_c5LBvLo", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-26T23:05:40Z", "updated_at": "2022-09-26T23:05:40Z", "author_association": "OWNER", "body": "Implementing it like this, so at least you can copy and paste the SQL query back out again:\r\n\r\n<img width=\"796\" alt=\"image\" src=\"https://user-images.githubusercontent.com/9599/192395953-48512c94-10e0-4cf8-8ae5-b9e65e3d7b0f.png\">\r\n\r\nI'm not doing a full textarea because this error can be raised in multiple places, including on the table page itself. It's not just an error associated with the manual query page.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1385026210, "label": "Preserve query on timeout"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1819#issuecomment-1258738435", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1819", "id": 1258738435, "node_id": "IC_kwDOBm6k_c5LBtMD", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-26T22:52:19Z", "updated_at": "2022-09-26T22:52:19Z", "author_association": "OWNER", "body": "This is a good idea.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1385026210, "label": "Preserve query on timeout"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1818#issuecomment-1258735747", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1818", "id": 1258735747, "node_id": "IC_kwDOBm6k_c5LBsiD", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-26T22:47:59Z", "updated_at": "2022-09-26T22:47:59Z", "author_association": "OWNER", "body": "Another option here is to tie into a feature I built in `sqlite-utils` with this problem in mind but never introduced on the Datasette side of things: https://sqlite-utils.datasette.io/en/stable/python-api.html#cached-table-counts-using-triggers", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1384549993, "label": "Setting to turn off table row counts entirely"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1818#issuecomment-1258735283", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1818", "id": 1258735283, "node_id": "IC_kwDOBm6k_c5LBsaz", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-26T22:47:19Z", "updated_at": "2022-09-26T22:47:19Z", "author_association": "OWNER", "body": "That's a really interesting idea: for a lot of databases (those made out of straight imports from CSV) `max(rowid)` would indeed reflect the size of the table, but would be a MUCH faster operation than attempting a `count(*)`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1384549993, "label": "Setting to turn off table row counts entirely"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1258697384", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/491", "id": 1258697384, "node_id": "IC_kwDOCGYnMM5LBjKo", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-26T22:12:45Z", "updated_at": "2022-09-26T22:12:45Z", "author_association": "OWNER", "body": "That feels like a slightly different command to me - maybe `sqlite-utils backup data.db data-backup.db`? It doesn't have any of the mechanics for merging tables together. Could be a useful feature separately though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1383646615, "label": "Ability to merge databases and tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1821#issuecomment-1258692555", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1821", "id": 1258692555, "node_id": "IC_kwDOBm6k_c5LBh_L", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-26T22:06:39Z", "updated_at": "2022-09-26T22:06:39Z", "author_association": "OWNER", "body": "- https://github.com/simonw/datasette/actions/runs/3131344150\r\n- https://github.com/simonw/datasette/releases/tag/0.63a0\r\n- https://pypi.org/project/datasette/0.63a0/", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1386734383, "label": "Release Datasette 0.63a0"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/494#issuecomment-1258521333", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/494", "id": 1258521333, "node_id": "IC_kwDOCGYnMM5LA4L1", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-26T19:32:36Z", "updated_at": "2022-09-26T19:32:36Z", "author_association": "OWNER", "body": "Tweeted about it too: https://twitter.com/simonw/status/1574481628507668480", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1386593843, "label": "Document how to use Just"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/494#issuecomment-1258516872", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/494", "id": 1258516872, "node_id": "IC_kwDOCGYnMM5LA3GI", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-26T19:28:36Z", "updated_at": "2022-09-26T19:28:36Z", "author_association": "OWNER", "body": "New documentation: https://sqlite-utils.datasette.io/en/latest/contributing.html#using-just-and-pipenv", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1386593843, "label": "Document how to use Just"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/483#issuecomment-1258479462", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/483", "id": 1258479462, "node_id": "IC_kwDOCGYnMM5LAt9m", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-26T19:04:29Z", "updated_at": "2022-09-26T19:04:43Z", "author_association": "OWNER", "body": "Documentation:\r\n\r\n- https://sqlite-utils.datasette.io/en/latest/cli.html#cli-install\r\n- https://sqlite-utils.datasette.io/en/latest/cli.html#cli-uninstall\r\n- https://sqlite-utils.datasette.io/en/latest/cli-reference.html#install\r\n- https://sqlite-utils.datasette.io/en/latest/cli-reference.html#uninstall\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1363765916, "label": "`sqlite-utils install` command"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/493#issuecomment-1258476455", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/493", "id": 1258476455, "node_id": "IC_kwDOCGYnMM5LAtOn", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-26T19:01:49Z", "updated_at": "2022-09-26T19:01:49Z", "author_association": "OWNER", "body": "I tried the tips in https://stackoverflow.com/questions/15258831/how-to-handle-two-dashes-in-rest (not the settings change though, because I might want smart quotes elsewhere) and they didn't work.\r\n\r\nMaybe I should disable smart quotes entirely?\r\n\r\nI feel like there should be an escaping trick that works here though. I tried `insert -\\\\-convert` but it didn't help.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1386562662, "label": "Tiny typographical error in install/uninstall docs"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/483#issuecomment-1258451968", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/483", "id": 1258451968, "node_id": "IC_kwDOCGYnMM5LAnQA", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-26T18:37:54Z", "updated_at": "2022-09-26T18:40:41Z", "author_association": "OWNER", "body": "The implementation of this can be an almost exact copy of Datasette's, which was added in this commit: https://github.com/simonw/datasette/commit/01fe5b740171bfaea3752fc5754431dac53777e3\r\n\r\nCurrent code for that is here: https://github.com/simonw/datasette/blob/0.62/datasette/cli.py#L319-L340 - which is improved to use the `from runpy import run_module` function.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1363765916, "label": "`sqlite-utils install` command"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1258450447", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/491", "id": 1258450447, "node_id": "IC_kwDOCGYnMM5LAm4P", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-26T18:36:23Z", "updated_at": "2022-09-26T18:36:23Z", "author_association": "OWNER", "body": "This is also the kind of feature that would need to express itself in both the Python library and the CLI utility.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1383646615, "label": "Ability to merge databases and tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1258449887", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/491", "id": 1258449887, "node_id": "IC_kwDOCGYnMM5LAmvf", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-26T18:35:50Z", "updated_at": "2022-09-26T18:35:50Z", "author_association": "OWNER", "body": "This is a really interesting idea.\r\n\r\nI'm nervous about needing to set the rules for how duplicate tables should be merged though. This feels like a complex topic - one where there isn't necessarily an obviously \"correct\" way of doing it, but where different problems that people are solving might need different merging approaches.\r\n\r\nLikewise, merging isn't just a database-to-database thing at that point - I could see a need for merging two tables using similar rules to those used for merging two databases.\r\n\r\nSo I think I'd want to have some good concrete use-cases in mind before trying to design how something like this should work. Will leave this thread open for people to drop those in!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1383646615, "label": "Ability to merge databases and tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/492#issuecomment-1258446128", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/492", "id": 1258446128, "node_id": "IC_kwDOCGYnMM5LAl0w", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-26T18:32:14Z", "updated_at": "2022-09-26T18:33:19Z", "author_association": "OWNER", "body": "This idea would make more sense if there was a good mechanism to say \"run the conversion script held in this file\" as opposed to passing it as an option. This would also make having to remember bash escaping rules ([see tip](https://til.simonwillison.net/zsh/argument-heredoc)) much easier!\r\n\r\n`shot-scraper` has that for `--javascript`, using the `--input` option: https://shot-scraper.datasette.io/en/stable/javascript.html#shot-scraper-javascript-help\r\n\r\nMaybe `--convert-script` would work here? Or `--convert-file`? It should accept `-` for stdin too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1386530156, "label": "Idea: ability to pass extra variables to `--convert` scripts"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/490#issuecomment-1258437060", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/490", "id": 1258437060, "node_id": "IC_kwDOCGYnMM5LAjnE", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-26T18:24:44Z", "updated_at": "2022-09-26T18:24:44Z", "author_association": "OWNER", "body": "Just saw your great write-up on this: https://jeqo.github.io/notes/2022-09-24-ingest-logs-sqlite/", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 1, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1382457780, "label": "Ability to insert multi-line files"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1817#issuecomment-1256662785", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1817", "id": 1256662785, "node_id": "IC_kwDOBm6k_c5K5ycB", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-23T20:53:21Z", "updated_at": "2022-09-23T20:53:21Z", "author_association": "OWNER", "body": "Maybe the signature for that method should be:\r\n\r\n```python\r\nasync def render_template( \r\n     self, templates, context=None, plugin_context=None, request=None, view_name=None \r\n ): \r\n```\r\nWhere `plugin_context` is a special dictionary of values that can be passed through to plugin hooks that accept them - so `database`, `table`, `columns`, `sql` and `params`.\r\n\r\nThose would then be passed when specific views call `render_template()` - which they currently do via calling `BaseView.render(...)`, but actually the views that are used for tables and queries don't even call that directly due to the weird designed used with `DataView` subclasses that implement a `.data()` method.\r\n\r\nSo yet another change that's blocked on fixing that long-running weird piece of technical debt:\r\n- https://github.com/simonw/datasette/issues/1518\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1384273985, "label": "Expose `sql` and `params` arguments to various plugin hooks"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1817#issuecomment-1256659788", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1817", "id": 1256659788, "node_id": "IC_kwDOBm6k_c5K5xtM", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-23T20:49:22Z", "updated_at": "2022-09-23T20:49:22Z", "author_association": "OWNER", "body": "Implementation challenge: all four of those hooks are called inside the `datasette.render_template()` method, which has this signature:\r\n\r\nhttps://github.com/simonw/datasette/blob/cb1e093fd361b758120aefc1a444df02462389a3/datasette/app.py#L945-L947\r\n\r\nSo I would have to pull the `sql` and `params` variables out of the `context` since they are not being passed to that method. OR I could teach that method to take those as optional arguments.\r\n\r\nMight be an opportunity to clean up this hack:\r\n\r\nhttps://github.com/simonw/datasette/blob/cb1e093fd361b758120aefc1a444df02462389a3/datasette/app.py#L959-L964\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1384273985, "label": "Expose `sql` and `params` arguments to various plugin hooks"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1817#issuecomment-1256652548", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1817", "id": 1256652548, "node_id": "IC_kwDOBm6k_c5K5v8E", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-23T20:41:32Z", "updated_at": "2022-09-23T20:41:32Z", "author_association": "OWNER", "body": "Which plugin hooks should take `sql` and `params`?\r\n\r\n- [extra_template_vars(template, database, table, columns, view_name, request, datasette)](https://docs.datasette.io/en/0.62/plugin_hooks.html#extra-template-vars-template-database-table-columns-view-name-request-datasette)\r\n- [extra_css_urls(template, database, table, columns, view_name, request, datasette)](https://docs.datasette.io/en/0.62/plugin_hooks.html#extra-css-urls-template-database-table-columns-view-name-request-datasette)\r\n- [extra_js_urls(template, database, table, columns, view_name, request, datasette)](https://docs.datasette.io/en/0.62/plugin_hooks.html#extra-js-urls-template-database-table-columns-view-name-request-datasette)\r\n- [extra_body_script(template, database, table, columns, view_name, request, datasette)](https://docs.datasette.io/en/0.62/plugin_hooks.html#extra-body-script-template-database-table-columns-view-name-request-datasette)\r\n\r\nAnd maybe these:\r\n\r\n- [render_cell(row, value, column, table, database, datasette)](https://docs.datasette.io/en/0.62/plugin_hooks.html#render-cell-row-value-column-table-database-datasette)\r\n- [table_actions(datasette, actor, database, table, request)](https://docs.datasette.io/en/0.62/plugin_hooks.html#table-actions-datasette-actor-database-table-request)\r\n\r\nI'll start by implementing the first set, then I'll think further about those \"maybes\".", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1384273985, "label": "Expose `sql` and `params` arguments to various plugin hooks"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1817#issuecomment-1256650449", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1817", "id": 1256650449, "node_id": "IC_kwDOBm6k_c5K5vbR", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-23T20:38:53Z", "updated_at": "2022-09-23T20:38:53Z", "author_association": "OWNER", "body": "I've wanted something like this in the past too. I think the thing to do here might be to add `sql` and `params` arguments to a bunch of the plugin hooks, such that they can see the main query that is being used on the page that they are helping to render.\r\n\r\nWhile I'm working on this: https://docs.datasette.io/en/0.62/plugin_hooks.html#register-output-renderer-datasette output renderer functions take `sql` but do not currently take `params` - they should also take `params`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1384273985, "label": "Expose `sql` and `params` arguments to various plugin hooks"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/490#issuecomment-1256428818", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/490", "id": 1256428818, "node_id": "IC_kwDOCGYnMM5K45US", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-23T16:37:58Z", "updated_at": "2022-09-23T16:38:35Z", "author_association": "OWNER", "body": "It should be possible to achieve this with the `--text` option: https://sqlite-utils.datasette.io/en/stable/cli.html?highlight=text#convert-with-text\r\n\r\nGiven an example like this in `multiline.log`:\r\n```\r\n2022-03-01T12:04:52: Here is a log message\r\n  that spans multiple lines\r\n2022-03-01T12:04:52: This is a single line\r\n2022-03-01T12:04:52: Here is another message\r\n  that spans multiple lines\r\n```\r\nYou should be able to run something like this:\r\n```\r\nsqlite-utils insert /tmp/log.db log multiline.log --text --convert \"\r\nimport re\r\n\r\nr = re.compile(r'^(?P<datetime>\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}):(?P<log>.*)', re.MULTILINE)\r\n\r\ndef convert(text):\r\n    return [m.groupdict() for m in r.finditer(text)]\r\n\"\r\n```\r\nAfter running this I get:\r\n```\r\nsqlite-utils rows /tmp/log.db log\r\n[{\"datetime\": \"2022-03-01T12:04:52\", \"log\": \" Here is a log message\"},\r\n {\"datetime\": \"2022-03-01T12:04:52\", \"log\": \" This is a single line\"},\r\n {\"datetime\": \"2022-03-01T12:04:52\", \"log\": \" Here is another message\"}]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1382457780, "label": "Ability to insert multi-line files"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/358#issuecomment-1255341690", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/358", "id": 1255341690, "node_id": "IC_kwDOCGYnMM5K0v56", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-22T17:35:23Z", "updated_at": "2022-09-22T17:35:23Z", "author_association": "OWNER", "body": "Make me think also that `sqlite-utils create-table` should have an option to dump out the SQL without actually creating the table.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1082651698, "label": "Support for CHECK constraints"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/358#issuecomment-1255340974", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/358", "id": 1255340974, "node_id": "IC_kwDOCGYnMM5K0vuu", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-22T17:34:45Z", "updated_at": "2022-09-22T17:34:45Z", "author_association": "OWNER", "body": "A few other recipes off the top of my head:\r\n\r\n- `title:maxlength:20` - set a max length, `length(title) <= 20`\r\n- `created:date` - check for `yyyy-mm-dd` date, `select :date == date(:date) is not null` ([demo](https://latest.datasette.io/_memory?sql=select+%3Adate+%3D%3D+date%28%3Adate%29+is+not+null&date=2022-01-01))\r\n- `age:positiveint` - check `age` is a positive integer, `printf('%', age) = age and age > 0` (untested)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1082651698, "label": "Support for CHECK constraints"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/358#issuecomment-1255333969", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/358", "id": 1255333969, "node_id": "IC_kwDOCGYnMM5K0uBR", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-22T17:29:09Z", "updated_at": "2022-09-22T17:29:09Z", "author_association": "OWNER", "body": "Quick demo of a check constraint for JSON validation:\r\n```\r\nsqlite> create table test (id integer primary key, tags text, check (json(tags) is not null));\r\nsqlite> sqlite> insert into test (tags ('[\"one\", \"two\"]');\r\nsqlite> insert into test (tags) values ('[\"one\", \"two\"');\r\nError: stepping, malformed JSON (1)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1082651698, "label": "Support for CHECK constraints"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/358#issuecomment-1255332217", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/358", "id": 1255332217, "node_id": "IC_kwDOCGYnMM5K0tl5", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-22T17:27:34Z", "updated_at": "2022-09-22T17:27:34Z", "author_association": "OWNER", "body": "I've been thinking about this more recently.\r\n\r\nI think the first place to explore these will be in the `create-table` command (and underlying APIs).\r\n\r\nRelevant docs: https://www.sqlite.org/lang_createtable.html#check_constraints\r\n\r\n> A CHECK constraint may be attached to a column definition or specified as a table constraint. In practice it makes no difference. Each time a new row is inserted into the table or an existing row is updated, the expression associated with each CHECK constraint is evaluated and cast to a NUMERIC value in the same way as a [CAST expression](https://www.sqlite.org/lang_expr.html#castexpr). If the result is zero (integer value 0 or real value 0.0), then a constraint violation has occurred. If the CHECK expression evaluates to NULL, or any other non-zero value, it is not a constraint violation. The expression of a CHECK constraint may not contain a subquery.\r\n\r\nSomething like this:\r\n\r\n    sqlite-utils create-table data.db entries id integer title text tags text --pk id --check tags:json\r\n\r\nWhere `--check tags:json` uses a pre-baked recipe for using the SQLite JSON function to check that the content is valid JSON and reject it otherwise.\r\n\r\nThen can bundle a bunch of other pre-baked recipes, but also support the following:\r\n\r\n    --check 'x > 3' --check 'length(phone) >= 10'\r\n\r\nThe besign reason for the `column:recipe` format here is to reuse `--check` for both pre-defined recipes that affect a single column AND for freeform expressions that get added to the end of the table.\r\n\r\nDetecting `column name:recipe` with a regex feels safe to me.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1082651698, "label": "Support for CHECK constraints"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/488#issuecomment-1254033981", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/488", "id": 1254033981, "node_id": "IC_kwDOCGYnMM5Kvwo9", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-21T17:49:32Z", "updated_at": "2022-09-21T17:50:10Z", "author_association": "OWNER", "body": "It looks like SQLite has a `SELECT NULLIF(value, '')` function which returns `null` if that value is equal to `''`.\r\n\r\nWe need to only apply that function to columns that we know to be of type integer or float though - text columns containing empty strings should not be rewritten to null.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1373224657, "label": "`sqlite-utils transform` should set empty strings to null when converting text columns to integer/float"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/488#issuecomment-1254032378", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/488", "id": 1254032378, "node_id": "IC_kwDOCGYnMM5KvwP6", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-21T17:47:54Z", "updated_at": "2022-09-21T17:47:54Z", "author_association": "OWNER", "body": "New tests should go in: https://github.com/simonw/sqlite-utils/blob/main/tests/test_transform.py\r\n\r\nI think the implementation fix needs to go near here: https://github.com/simonw/sqlite-utils/blob/0b315d3fa83c1584eaeec32f24912898621e437a/sqlite_utils/db.py#L1770-L1775\r\n\r\nThe trick is going to be teaching that generated SQL to know which columns are `integer` or `float` and to convert `\"\"` to `null` as part of that operation. ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1373224657, "label": "`sqlite-utils transform` should set empty strings to null when converting text columns to integer/float"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/488#issuecomment-1254029808", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/488", "id": 1254029808, "node_id": "IC_kwDOCGYnMM5Kvvnw", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-21T17:45:20Z", "updated_at": "2022-09-21T17:45:41Z", "author_association": "OWNER", "body": "No, I'm going to say that this is a bug - it's WEIRD having a `integer` or `float` column containing an empty string. I'm OK changing that - I very much doubt anyone is relying on this functionality.\r\n\r\nSo no need for a new option here - just fixing the bug is sensible.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1373224657, "label": "`sqlite-utils transform` should set empty strings to null when converting text columns to integer/float"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1816#issuecomment-1251724180", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1816", "id": 1251724180, "node_id": "IC_kwDOBm6k_c5Km8uU", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-20T01:13:05Z", "updated_at": "2022-09-20T01:13:05Z", "author_association": "OWNER", "body": "Oops, that has a bug:\r\n\r\n```\r\nError: Invalid setting '{key}' in settings.json\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1378640768, "label": "Validate settings.json on startup in configuration directory mode"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1816#issuecomment-1251682970", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1816", "id": 1251682970, "node_id": "IC_kwDOBm6k_c5Kmyqa", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-19T23:44:54Z", "updated_at": "2022-09-19T23:44:54Z", "author_association": "OWNER", "body": "I was going to add type validation too, but that's actually a bit tricky because the logic for that currently lives in Click option parsing here: https://github.com/simonw/datasette/blob/ddc999ad1296e8c69cffede3e367dda059b8adad/datasette/cli.py#L71-L88", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1378640768, "label": "Validate settings.json on startup in configuration directory mode"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1814#issuecomment-1251677554", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1814", "id": 1251677554, "node_id": "IC_kwDOBm6k_c5KmxVy", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-19T23:35:06Z", "updated_at": "2022-09-19T23:35:06Z", "author_association": "OWNER", "body": "It might have been useful for Datasette to show an error when started against a `settings.json` file that contains an invalid setting though.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1378495690, "label": "Static files not served"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1814#issuecomment-1251677220", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1814", "id": 1251677220, "node_id": "IC_kwDOBm6k_c5KmxQk", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-19T23:34:30Z", "updated_at": "2022-09-19T23:34:30Z", "author_association": "OWNER", "body": "The `settings.json` file can only be used with settings that are set using `--setting name value` - the full list of those is here: https://docs.datasette.io/en/stable/settings.html\r\n\r\nThe `--static` option works differently. In configuration directory mode you can skip it entirely and instead have a `/static/` folder - so your directory structure would look like this:\r\n\r\n```\r\nbibliography/static/styles.css\r\n```\r\nAnd then when you run `datasette bibliography/` the following URL will work:\r\n\r\n    http://127.0.0.1:8001/static/styles.css\r\n", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 1, \"eyes\": 0}", "issue": {"value": 1378495690, "label": "Static files not served"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1809#issuecomment-1249990033", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1809", "id": 1249990033, "node_id": "IC_kwDOBm6k_c5KgVWR", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-17T03:39:05Z", "updated_at": "2022-09-17T03:39:05Z", "author_association": "OWNER", "body": "New docs section on the need to call `await ds.invoke_startup()`: https://github.com/simonw/datasette/blob/ddc999ad1296e8c69cffede3e367dda059b8adad/docs/testing_plugins.rst#setting-up-a-datasette-test-instance", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1373595927, "label": "`prepare_jinja2_environment()` hook should take `datasette` argument"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1809#issuecomment-1249987643", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1809", "id": 1249987643, "node_id": "IC_kwDOBm6k_c5KgUw7", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-17T03:19:24Z", "updated_at": "2022-09-17T03:19:24Z", "author_association": "OWNER", "body": "In looking at the documentation on [writing tests](https://docs.datasette.io/en/latest/testing_plugins.html), there are a lot of examples like this:\r\n```python\r\n    def test_that_opens_the_debugger_or_errors():\r\n        ds = Datasette([db_path], pdb=True)\r\n        response = await ds.client.get(\"/\")\r\n```\r\nI really don't like having to tell people to add `await ds.invoke_startup()` to every test that might look like this.\r\n\r\nSince it's safe to call that function multiple times, I'm going to have `ds.client.get()` and friends call it for you too - so if you forget in a plugin test it won't matter.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1373595927, "label": "`prepare_jinja2_environment()` hook should take `datasette` argument"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1809#issuecomment-1249986079", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1809", "id": 1249986079, "node_id": "IC_kwDOBm6k_c5KgUYf", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-17T03:07:24Z", "updated_at": "2022-09-17T03:07:24Z", "author_association": "OWNER", "body": "Datasette's own tests started to break because calls to the `TestClient` were performed without awaiting that method. I fixed that by adding this to `_request()` inside that class:\r\n\r\n```python\r\n    async def _request(\r\n        self,\r\n        path,\r\n        follow_redirects=True,\r\n        redirect_count=0,\r\n        method=\"GET\",\r\n        cookies=None,\r\n        headers=None,\r\n        post_body=None,\r\n        content_type=None,\r\n        if_none_match=None,\r\n    ):\r\n        if not self.ds._startup_invoked:\r\n            await self.ds.invoke_startup()\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1373595927, "label": "`prepare_jinja2_environment()` hook should take `datasette` argument"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1809#issuecomment-1249985971", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1809", "id": 1249985971, "node_id": "IC_kwDOBm6k_c5KgUWz", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-17T03:06:32Z", "updated_at": "2022-09-17T03:06:32Z", "author_association": "OWNER", "body": "This is likely going to cause some tests in plugins to break, but I'm OK with that - I'll fix them as I find them once this release is out.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1373595927, "label": "`prepare_jinja2_environment()` hook should take `datasette` argument"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1809#issuecomment-1249985741", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1809", "id": 1249985741, "node_id": "IC_kwDOBm6k_c5KgUTN", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-17T03:04:51Z", "updated_at": "2022-09-17T03:04:51Z", "author_association": "OWNER", "body": "I'm going to throw an error in `ds.render_template()` if you haven't previously called `await ds.invoke_startup()`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1373595927, "label": "`prepare_jinja2_environment()` hook should take `datasette` argument"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/1812#issuecomment-1249746777", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1812", "id": 1249746777, "node_id": "IC_kwDOBm6k_c5KfZ9Z", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-16T19:50:45Z", "updated_at": "2022-09-16T19:50:45Z", "author_association": "OWNER", "body": "Main difference I can see:\r\n\r\n![CleanShot 2022-09-16 at 12 49 47@2x](https://user-images.githubusercontent.com/9599/190719563-a7b1bcc7-bfdc-4759-95c1-e19bcd0217c3.png)\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1375930971, "label": "Bump furo from 2022.6.21 to 2022.9.15"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/1812#issuecomment-1249745637", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1812", "id": 1249745637, "node_id": "IC_kwDOBm6k_c5KfZrl", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-16T19:49:12Z", "updated_at": "2022-09-16T19:49:12Z", "author_association": "OWNER", "body": "Preview looks good.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1375930971, "label": "Bump furo from 2022.6.21 to 2022.9.15"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/489#issuecomment-1248621072", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/489", "id": 1248621072, "node_id": "IC_kwDOCGYnMM5KbHIQ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-15T20:56:09Z", "updated_at": "2022-09-15T20:56:09Z", "author_association": "OWNER", "body": "Prototype so far:\r\n```diff\r\ndiff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py\r\nindex 767b170..d96c507 100644\r\n--- a/sqlite_utils/cli.py\r\n+++ b/sqlite_utils/cli.py\r\n@@ -1762,6 +1762,17 @@ def query(\r\n     is_flag=True,\r\n     help=\"Analyze resulting tables and output results\",\r\n )\r\n+@click.option(\"--key\", help=\"read data from this key of the root object\")\r\n+@click.option(\r\n+    \"--auto-key\",\r\n+    is_flag=True,\r\n+    help=\"Find a key in the root object that is a list of objects\",\r\n+)\r\n+@click.option(\r\n+    \"--analyze\",\r\n+    is_flag=True,\r\n+    help=\"Analyze resulting tables and output results\",\r\n+)\r\n @load_extension_option\r\n def memory(\r\n     paths,\r\n@@ -1784,6 +1795,8 @@ def memory(\r\n     schema,\r\n     dump,\r\n     save,\r\n+    key,\r\n+    auto_key,\r\n     analyze,\r\n     load_extension,\r\n ):\r\n@@ -1838,7 +1851,9 @@ def memory(\r\n                 csv_table = stem\r\n             stem_counts[stem] = stem_counts.get(stem, 1) + 1\r\n             csv_fp = csv_path.open(\"rb\")\r\n-        rows, format_used = rows_from_file(csv_fp, format=format, encoding=encoding)\r\n+        rows, format_used = rows_from_file(\r\n+            csv_fp, format=format, encoding=encoding, key=key, auto_key=auto_key\r\n+        )\r\n         tracker = None\r\n         if format_used in (Format.CSV, Format.TSV) and not no_detect_types:\r\n             tracker = TypeTracker()\r\ndiff --git a/sqlite_utils/utils.py b/sqlite_utils/utils.py\r\nindex 8754554..2e69c26 100644\r\n--- a/sqlite_utils/utils.py\r\n+++ b/sqlite_utils/utils.py\r\n@@ -231,6 +231,8 @@ def rows_from_file(\r\n     encoding: Optional[str] = None,\r\n     ignore_extras: Optional[bool] = False,\r\n     extras_key: Optional[str] = None,\r\n+    key: Optional[str] = None,\r\n+    auto_key: Optional[bool] = False,\r\n ) -> Tuple[Iterable[dict], Format]:\r\n     \"\"\"\r\n     Load a sequence of dictionaries from a file-like object containing one of four different formats.\r\n@@ -271,13 +273,31 @@ def rows_from_file(\r\n     :param encoding: the character encoding to use when reading CSV/TSV data\r\n     :param ignore_extras: ignore any extra fields on rows\r\n     :param extras_key: put any extra fields in a list with this key\r\n+    :param key: read data from this key of the root object\r\n+    :param auto_key: find a key in the root object that is a list of objects\r\n     \"\"\"\r\n     if ignore_extras and extras_key:\r\n         raise ValueError(\"Cannot use ignore_extras= and extras_key= together\")\r\n+    if key and auto_key:\r\n+        raise ValueError(\"Cannot use key= and auto_key= together\")\r\n     if format == Format.JSON:\r\n         decoded = json.load(fp)\r\n         if isinstance(decoded, dict):\r\n-            decoded = [decoded]\r\n+            if auto_key:\r\n+                list_keys = [\r\n+                    k\r\n+                    for k in decoded\r\n+                    if isinstance(decoded[k], list)\r\n+                    and decoded[k]\r\n+                    and all(isinstance(o, dict) for o in decoded[k])\r\n+                ]\r\n+                if len(list_keys) == 1:\r\n+                    decoded = decoded[list_keys[0]]\r\n+            elif key:\r\n+                # Raises KeyError, I think that's OK\r\n+                decoded = decoded[key]\r\n+            if not isinstance(decoded, list):\r\n+                decoded = [decoded]\r\n         if not isinstance(decoded, list):\r\n             raise RowsFromFileBadJSON(\"JSON must be a list or a dictionary\")\r\n         return decoded, Format.JSON\r\n@@ -305,7 +325,9 @@ def rows_from_file(\r\n         first_bytes = buffered.peek(2048).strip()\r\n         if first_bytes.startswith(b\"[\") or first_bytes.startswith(b\"{\"):\r\n             # TODO: Detect newline-JSON\r\n-            return rows_from_file(buffered, format=Format.JSON)\r\n+            return rows_from_file(\r\n+                buffered, format=Format.JSON, key=key, auto_key=auto_key\r\n+            )\r\n         else:\r\n             dialect = csv.Sniffer().sniff(\r\n                 first_bytes.decode(encoding or \"utf-8-sig\", \"ignore\")\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1374939463, "label": "Ability to load JSON records held in a file with a single top level key that is a list of objects"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248591268", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/486", "id": 1248591268, "node_id": "IC_kwDOCGYnMM5Ka_2k", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-15T20:36:02Z", "updated_at": "2022-09-15T20:40:03Z", "author_association": "OWNER", "body": "I had a big CSV file lying around, I converted it to other formats like this:\r\n\r\n    sqlite-utils insert /tmp/t.db t /tmp/en.openfoodfacts.org.products.csv --csv\r\n    sqlite-utils rows /tmp/t.db t --nl > /tmp/big.nl\r\n    sqlite-utils rows /tmp/t.db t > /tmp/big.json\r\n\r\nThen tested the progress bar like this:\r\n\r\n    sqlite-utils insert /tmp/t2.db t /tmp/big.nl --nl\r\n\r\nOutput:\r\n\r\n```\r\nsqlite-utils insert /tmp/t2.db t /tmp/big.nl --nl\r\n  [------------------------------------]    0%\r\n  [#######-----------------------------]   20%  00:00:20\r\n```\r\nWith `--silent` it is silent.\r\n\r\nAnd for regular JSON:\r\n\r\n```\r\nsqlite-utils insert /tmp/t3.db t /tmp/big.json \r\n  [####################################]  100%\r\n```\r\nThis is actually not doing the right thing. The problem is that `sqlite-utils` doesn't include a streaming JSON parser, so it instead reads that entire JSON file into memory first (exhausting the progress bar to 100% instantly) and then does the rest of the work in-memory while the bar sticks at 100%.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1366512990, "label": "progressbar for inserts/upserts of all fileformats, closes #485"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/485#issuecomment-1248597643", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/485", "id": 1248597643, "node_id": "IC_kwDOCGYnMM5KbBaL", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-15T20:39:39Z", "updated_at": "2022-09-15T20:39:52Z", "author_association": "OWNER", "body": "A note from PR #486: https://github.com/simonw/sqlite-utils/issues/486#issuecomment-1248591268_\r\n\r\n> ```\r\n> sqlite-utils insert /tmp/t3.db t /tmp/big.json \r\n>   [####################################]  100%\r\n> ```\r\n> This is actually not doing the right thing. The problem is that `sqlite-utils` doesn't include a streaming JSON parser, so it instead reads that entire JSON file into memory first (exhausting the progress bar to 100% instantly) and then does the rest of the work in-memory while the bar sticks at 100%.\r\n\r\nI decided to land this anyway. If a streaming JSON parser is added later it will start to work.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1366423176, "label": "Progressbar not shown when inserting/upserting jsonlines file"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248593835", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/486", "id": 1248593835, "node_id": "IC_kwDOCGYnMM5KbAer", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-15T20:37:14Z", "updated_at": "2022-09-15T20:37:14Z", "author_association": "OWNER", "body": "I'm going to land this anyway. The lack of a streaming JSON parser is a separate issue, I don't think it should block landing this improvement.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1366512990, "label": "progressbar for inserts/upserts of all fileformats, closes #485"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248582147", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/486", "id": 1248582147, "node_id": "IC_kwDOCGYnMM5Ka9oD", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-15T20:29:17Z", "updated_at": "2022-09-15T20:29:17Z", "author_association": "OWNER", "body": "This looks good to me. I need to run some manual tests before merging (it's a good sign that the automated tests pass though).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1366512990, "label": "progressbar for inserts/upserts of all fileformats, closes #485"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248568775", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/486", "id": 1248568775, "node_id": "IC_kwDOCGYnMM5Ka6XH", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-15T20:16:14Z", "updated_at": "2022-09-15T20:16:14Z", "author_association": "OWNER", "body": "https://github.com/actions/setup-python/blob/main/docs/advanced-usage.md#using-the-python-version-input says can set the full version:\r\n\r\n```\r\n- uses: actions/setup-python@v4\r\n  with:\r\n    python-version: \"3.10.6\" \r\n```\r\nI'll try that.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1366512990, "label": "progressbar for inserts/upserts of all fileformats, closes #485"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248567323", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/486", "id": 1248567323, "node_id": "IC_kwDOCGYnMM5Ka6Ab", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-15T20:14:45Z", "updated_at": "2022-09-15T20:14:45Z", "author_association": "OWNER", "body": "There's a fix for `mypy` that has landed but isn't out in a release yet:\r\n- https://github.com/python/mypy/issues/13385\r\n\r\nFor the moment looks like pinning to Python 3.10.6 could help. Need to figure out how to do that in GitHub Actions though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1366512990, "label": "progressbar for inserts/upserts of all fileformats, closes #485"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248565396", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/486", "id": 1248565396, "node_id": "IC_kwDOCGYnMM5Ka5iU", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-15T20:12:50Z", "updated_at": "2022-09-15T20:12:50Z", "author_association": "OWNER", "body": "Annoying `mypy` test failure:\r\n\r\n```\r\n/Users/runner/hostedtoolcache/Python/3.10.7/x64/lib/python3.10/site-packages/numpy/__init__.pyi:636:\r\nerror: Positional-only parameters are only supported in Python 3.8 and greater\r\n```\r\nLooks like this:\r\n- https://github.com/python/mypy/issues/13627", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1366512990, "label": "progressbar for inserts/upserts of all fileformats, closes #485"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/489#issuecomment-1248522618", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/489", "id": 1248522618, "node_id": "IC_kwDOCGYnMM5KavF6", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-15T19:29:20Z", "updated_at": "2022-09-15T19:29:20Z", "author_association": "OWNER", "body": "I think refactoring `sqlite-utils insert` to use `rows_from_file` needs to happen as part of this work.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1374939463, "label": "Ability to load JSON records held in a file with a single top level key that is a list of objects"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/489#issuecomment-1248512739", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/489", "id": 1248512739, "node_id": "IC_kwDOCGYnMM5Kasrj", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-15T19:18:24Z", "updated_at": "2022-09-15T19:21:01Z", "author_association": "OWNER", "body": "Why doesn't `sqlite-utils insert` use the `rows_from_file` function I wonder?\r\n\r\nhttps://github.com/simonw/sqlite-utils/issues/279#issuecomment-864207841 says:\r\n\r\n> I can refactor `sqlite-utils insert` to use this new code too.\r\n\r\nMaybe I forgot to do that?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1374939463, "label": "Ability to load JSON records held in a file with a single top level key that is a list of objects"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/489#issuecomment-1248501824", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/489", "id": 1248501824, "node_id": "IC_kwDOCGYnMM5KaqBA", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-15T19:10:48Z", "updated_at": "2022-09-15T19:10:48Z", "author_association": "OWNER", "body": "This feels pretty good:\r\n```\r\n% sqlite-utils memory ~/Downloads/CVR_Export_20220908084311/*.json --schema --auto-key\r\nCREATE TABLE [BallotTypeContestManifest] (\r\n   [BallotTypeId] INTEGER,\r\n   [ContestId] INTEGER\r\n);\r\nCREATE VIEW t1 AS select * from [BallotTypeContestManifest];\r\nCREATE VIEW t AS select * from [BallotTypeContestManifest];\r\nCREATE TABLE [BallotTypeManifest] (\r\n   [Description] TEXT,\r\n   [Id] INTEGER,\r\n   [ExternalId] TEXT\r\n);\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1374939463, "label": "Ability to load JSON records held in a file with a single top level key that is a list of objects"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/489#issuecomment-1248484094", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/489", "id": 1248484094, "node_id": "IC_kwDOCGYnMM5Kalr-", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-15T18:56:31Z", "updated_at": "2022-09-15T18:56:31Z", "author_association": "OWNER", "body": "Actually I quite like `--key X` - it could work for single nested objects too. You could insert a single record like this:\r\n\r\n```json\r\n{\r\n  \"record\" {\r\n    \"id\": 1\r\n  }\r\n}\r\n```\r\n```\r\nsqlite-utils insert db.db records record.json --key record\r\n``` ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1374939463, "label": "Ability to load JSON records held in a file with a single top level key that is a list of objects"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/489#issuecomment-1248481303", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/489", "id": 1248481303, "node_id": "IC_kwDOCGYnMM5KalAX", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-15T18:54:30Z", "updated_at": "2022-09-15T18:55:14Z", "author_association": "OWNER", "body": "Maybe this would make more sense as a mechanism where you can say \"Use the data in the key called X\" - but there's a special option for \"figure out that key automatically\".\r\n\r\nThe syntax then could be:\r\n\r\n`--list-key List`\r\n\r\nOr for automatic detection:\r\n\r\n`--list-key-auto`\r\n\r\nCould also go with `--key List` and `--key-auto` - but would that be as obvious as `--list-key`?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1374939463, "label": "Ability to load JSON records held in a file with a single top level key that is a list of objects"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/489#issuecomment-1248479485", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/489", "id": 1248479485, "node_id": "IC_kwDOCGYnMM5Kakj9", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-15T18:52:52Z", "updated_at": "2022-09-15T18:53:45Z", "author_association": "OWNER", "body": "The most similar option I have at the moment is probably `--flatten`. What would good names for this option be?\r\n\r\n- `--auto-list`\r\n- `--auto-key`\r\n- `--inner-key`\r\n- `--auto-json`\r\n- `--find-list`\r\n- `--find-key`\r\n\r\nThose are all bad.\r\n\r\nAnother option: introduce a new explicit format for it. Right now the explicit formats you can use are:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/d9b9e075f07a20f1137cd2e34ed5d3f1a3db4ad8/docs/cli-reference.rst#L153-L158\r\n\r\nSo I could add a `:autojson` format.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1374939463, "label": "Ability to load JSON records held in a file with a single top level key that is a list of objects"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/489#issuecomment-1248475718", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/489", "id": 1248475718, "node_id": "IC_kwDOCGYnMM5KajpG", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-15T18:49:05Z", "updated_at": "2022-09-15T18:49:53Z", "author_association": "OWNER", "body": "Here's how I used my prototype to build [that Gist](https://gist.github.com/simonw/0e6901974a14ab7d56c2746a04d72c8c):\r\n\r\n    sqlite-utils memory ~/Downloads/CVR_Export_20220908084311/*.json --schema > database.sql\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1374939463, "label": "Ability to load JSON records held in a file with a single top level key that is a list of objects"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/489#issuecomment-1248474806", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/489", "id": 1248474806, "node_id": "IC_kwDOCGYnMM5Kaja2", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-15T18:48:09Z", "updated_at": "2022-09-15T18:48:09Z", "author_association": "OWNER", "body": "Built a prototype of this that works really well:\r\n```diff\r\n diff --git a/sqlite_utils/utils.py b/sqlite_utils/utils.py\r\nindex c0b7bf1..f9a482c 100644\r\n--- a/sqlite_utils/utils.py\r\n+++ b/sqlite_utils/utils.py\r\n@@ -272,7 +272,19 @@ def rows_from_file(\r\n     if format == Format.JSON:\r\n         decoded = json.load(fp)\r\n         if isinstance(decoded, dict):\r\n-            decoded = [decoded]\r\n+            # TODO: Solve for if this isn't what people want\r\n+            # Does it have just one key that is a list of dicts?\r\n+            list_keys = [\r\n+                k\r\n+                for k in decoded\r\n+                if isinstance(decoded[k], list)\r\n+                and decoded[k]\r\n+                and all(isinstance(o, dict) for o in decoded[k])\r\n+            ]\r\n+            if len(list_keys) == 1:\r\n+                decoded = decoded[list_keys[0]]\r\n+            else:\r\n+                decoded = [decoded]\r\n         if not isinstance(decoded, list):\r\n             raise RowsFromFileBadJSON(\"JSON must be a list or a dictionary\")\r\n         return decoded, Format.JSON\r\n```\r\nI used that to build this: https://gist.github.com/simonw/0e6901974a14ab7d56c2746a04d72c8c\r\n\r\nOne problem though: right now, if you do this `sqlite-utils` treats it as a single object and adds a `tags` column with JSON in it:\r\n```\r\necho '{\"title\": \"Hi\", \"tags\": [{\"t\": \"one\"}]}` | sqlite-utils insert db.db t -\r\n```\r\nIf I implement this new mechanism the above line would behave differently - which would be a backwards incompatible change.\r\n\r\nSo I probably need some kind of opt-in mechanism for this. And I need a good name for it.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1374939463, "label": "Ability to load JSON records held in a file with a single top level key that is a list of objects"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1810#issuecomment-1248290151", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1810", "id": 1248290151, "node_id": "IC_kwDOBm6k_c5KZ2Vn", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-15T15:51:04Z", "updated_at": "2022-09-15T15:51:25Z", "author_association": "OWNER", "body": "I could prototype this idea as a `datasette-featured-tables` plugin that delivers its own custom `index.html` template.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1374626873, "label": "Featured table(s) on the homepage"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1810#issuecomment-1248289857", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1810", "id": 1248289857, "node_id": "IC_kwDOBm6k_c5KZ2RB", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-15T15:50:46Z", "updated_at": "2022-09-15T15:50:46Z", "author_association": "OWNER", "body": "Idea: allow the user to specify one or more featured tables. Each table is then shown as a summary on the homepage - with the total number of rows and the first 5 rows. If the table has search configured there's a search box too.\r\n\r\nIf the instance has only one database with only one table (excluding hidden tables) it gets featured automatically perhaps (maybe with a way to opt-out of that if you want to).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1374626873, "label": "Featured table(s) on the homepage"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1810#issuecomment-1248187089", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1810", "id": 1248187089, "node_id": "IC_kwDOBm6k_c5KZdLR", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-15T14:31:36Z", "updated_at": "2022-09-15T14:31:36Z", "author_association": "OWNER", "body": "Twitter conversation that inspired this issue: https://twitter.com/psychemedia/status/1570410108785684481", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1374626873, "label": "Featured table(s) on the homepage"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1809#issuecomment-1247317941", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1809", "id": 1247317941, "node_id": "IC_kwDOBm6k_c5KWI-1", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-14T21:24:43Z", "updated_at": "2022-09-14T21:24:43Z", "author_association": "OWNER", "body": "It looks like Datasette Lite does NOT invoke that method, which is likely a bug: https://github.com/simonw/datasette-lite/blob/e7ccaf621b3cdf613ebaf544304d387f2af32edf/webworker.js#L103-L110", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1373595927, "label": "`prepare_jinja2_environment()` hook should take `datasette` argument"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1809#issuecomment-1247316715", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1809", "id": 1247316715, "node_id": "IC_kwDOBm6k_c5KWIrr", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-14T21:23:10Z", "updated_at": "2022-09-14T21:23:10Z", "author_association": "OWNER", "body": "It might be good to have Datasette LOUDLY fail if you attempt to use it without calling `await ds.invoke_startup()`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1373595927, "label": "`prepare_jinja2_environment()` hook should take `datasette` argument"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1809#issuecomment-1247316097", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1809", "id": 1247316097, "node_id": "IC_kwDOBm6k_c5KWIiB", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-14T21:22:24Z", "updated_at": "2022-09-14T21:22:24Z", "author_association": "OWNER", "body": "It looks like this is the only place that calls `invoke_startup()`: https://github.com/simonw/datasette/blob/1d64c9a8dac45b9a3452acf8e76dfadea2b0bc49/datasette/cli.py#L590-L591\r\n\r\n`datasette-publish-vercel` is the one deployment mechanism that skips running Uvicorn, and it calls that method separately here: https://github.com/simonw/datasette-publish-vercel/blob/1559d979b4e3b1f2f83c51c3c0c10192ff9a6d0c/datasette_publish_vercel/__init__.py#L42-L52\r\n\r\n```python\r\nds = Datasette(\r\n    [],\r\n    {database_files},\r\n    static_mounts=static_mounts,\r\n    metadata=metadata{extras},\r\n    secret=secret,\r\n    cors=True,\r\n    settings={settings}{crossdb}\r\n)\r\nasyncio.run(ds.invoke_startup())\r\napp = ds.app()\r\n```\r\nSo preparing the Jinja environment inside that function would work fine.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1373595927, "label": "`prepare_jinja2_environment()` hook should take `datasette` argument"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1809#issuecomment-1247314352", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1809", "id": 1247314352, "node_id": "IC_kwDOBm6k_c5KWIGw", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-14T21:20:12Z", "updated_at": "2022-09-14T21:20:12Z", "author_association": "OWNER", "body": "The reason to support `await_me_maybe` is in case a hook wants to execute a SQL query as part of configuring the Jinja template loader.\r\n\r\nThat's exactly what `datasette-edit-templates` needs to do, though it's currently achieving that in a `startup()` hook instead: https://github.com/simonw/datasette-edit-templates/blob/087f6a6cabc20020f2b0524f11aa3a7836320848/datasette_edit_templates/__init__.py#L32-L48\r\n\r\n```python\r\n@hookimpl\r\ndef startup(datasette):\r\n    datasette._edit_templates = {}\r\n\r\n    async def inner():\r\n        db = get_database(datasette)\r\n        # Does the table exist?\r\n        if not await db.table_exists(TABLE):\r\n            for sql in CREATE_TABLE:\r\n                await db.execute_write(sql, block=True)\r\n        else:\r\n            # Load all templates from that table\r\n            rows = await db.execute(\"select template, body FROM {}\".format(TABLE))\r\n            for name, content in rows:\r\n                datasette._edit_templates[name] = content\r\n\r\n    return inner\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1373595927, "label": "`prepare_jinja2_environment()` hook should take `datasette` argument"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1809#issuecomment-1247313134", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1809", "id": 1247313134, "node_id": "IC_kwDOBm6k_c5KWHzu", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-14T21:18:46Z", "updated_at": "2022-09-14T21:18:46Z", "author_association": "OWNER", "body": "`await_me_maybe` might be tricky though, because the only place that hook is executed is here:\r\n\r\nhttps://github.com/simonw/datasette/blob/8430c3bc7dd22b173c1a8c6cd7180e3b31240cd1/datasette/app.py#L348\r\n\r\nWhich is inside the `Datasette.__init__` method.\r\n\r\nTo implement an `await` on that would need to move it somewhere else - probably here: https://github.com/simonw/datasette/blob/8430c3bc7dd22b173c1a8c6cd7180e3b31240cd1/datasette/app.py#L391-L393\r\n\r\nBut I'm not 100% confident that is always executed at the right time to ensure the Jinja environment is properly configured? I think it is, but would need to make sure.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1373595927, "label": "`prepare_jinja2_environment()` hook should take `datasette` argument"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1809#issuecomment-1247311275", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1809", "id": 1247311275, "node_id": "IC_kwDOBm6k_c5KWHWr", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-14T21:16:32Z", "updated_at": "2022-09-14T21:16:32Z", "author_association": "OWNER", "body": "It should also implement the `await_me_maybe` pattern so you can return an `async` function from it.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1373595927, "label": "`prepare_jinja2_environment()` hook should take `datasette` argument"}, "performed_via_github_app": null}