{"id": 2001006157, "node_id": "PR_kwDOCGYnMM5f2OZC", "number": 604, "title": "Add more STRICT table support", "user": {"value": 16437338, "label": "tkhattra"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2023-11-19T19:38:53Z", "updated_at": "2023-12-08T05:17:20Z", "closed_at": "2023-12-08T05:05:27Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/604", "body": "- https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982014776\r\n\r\nMake `table.transform()` preserve STRICT mode.\r\n\r\n\r\n----\r\n:books: Documentation preview :books:: https://sqlite-utils--604.org.readthedocs.build/en/604/\r\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/604/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 959137143, "node_id": "MDU6SXNzdWU5NTkxMzcxNDM=", "number": 1415, "title": "feature request: document minimum permissions for service account for cloudrun", "user": {"value": 536941, "label": "fgregg"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-08-03T13:48:43Z", "updated_at": "2023-11-05T16:46:59Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "Thanks again for such a powerful project.\r\n\r\nFor deploying to cloudrun from github actions, I'd like to create a service account with minimal permissions.\r\n\r\nIt would be great to document what those minimum permission that need to be set in the IAM.\r\n\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1415/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1901768721, "node_id": "PR_kwDOBm6k_c5anSg5", "number": 2191, "title": "Move `permissions`, `allow` blocks, canned queries and more out of `metadata.yaml` and into `datasette.yaml`", "user": {"value": 15178711, "label": "asg017"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2023-09-18T21:21:16Z", "updated_at": "2023-10-12T16:16:38Z", "closed_at": "2023-10-12T16:16:38Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/datasette/pulls/2191", "body": "The PR moves the following fields from `metadata.yaml` to `datasette.yaml`:\r\n\r\n```\r\npermissions\r\nallow\r\nallow_sql\r\nqueries\r\nextra_css_urls\r\nextra_js_urls\r\n```\r\n\r\nThis is a significant breaking change that users will need to upgrade their `metadata.yaml` files for. But the format/locations are similar to the previous version, so it shouldn't be too difficult to upgrade.\r\n\r\nOne note: I'm still working on the Configuration docs, specifically the \"reference\" section. Though it's pretty small, the rest of read to review", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2191/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1891212159, "node_id": "PR_kwDOBm6k_c5aD33C", "number": 2183, "title": "`datasette.yaml` plugin support", "user": {"value": 15178711, "label": "asg017"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2023-09-11T20:26:04Z", "updated_at": "2023-09-13T21:06:25Z", "closed_at": "2023-09-13T21:06:25Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/datasette/pulls/2183", "body": "Part of #2093\r\n\r\nIn #2149 , we ported over `\"settings.json\"` into the new `datasette.yaml` config file, with a top-level `\"settings\"` key. This PR ports over plugin configuration into top-level `\"plugins\"` key, as well as nested database/table plugin config.\r\n\r\nFrom now on, no plugin-related configuration is allowed in `metadata.yaml`, and must be in `datasette.yaml` in this new format. This is a pretty significant breaking change. Thankfully, you should be able to copy-paste your legacy plugin key/values into the new `datasette.yaml` format.\r\n\r\nAn example of what `datasette.yaml` would look like with this new plugin config:\r\n\r\n```yaml\r\n\r\nplugins:\r\n datasette-my-plugin:\r\n config_key: value\r\n\r\ndatabases:\r\n fixtures:\r\n plugins: \r\n datasette-my-plugin:\r\n config_key: fixtures-db-value\r\n tables:\r\n students:\r\n plugins:\r\n datasette-my-plugin:\r\n config_key: fixtures-students-table-value\r\n\r\n```\r\n\r\nAs an additional benefit, this now works with the new `-s` flag:\r\n\r\n```bash\r\ndatasette --memory -s 'plugins.datasette-my-plugin.config_key' new_value\r\n```\r\n\r\n\r\nMarked as a \"Draft\" right now until I add better documentation. We also should have a plan for the next alpha release to document and publicize this change, especially for plugin authors (since their docs will have to change to say `datasette.yaml` instead of `metadata.yaml`\r\n\r\n\r\n----\n:books: Documentation preview :books:: https://datasette--2183.org.readthedocs.build/en/2183/\n\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2183/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 336464733, "node_id": "MDU6SXNzdWUzMzY0NjQ3MzM=", "number": 328, "title": "Installation instructions, including how to use the docker image", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2018-06-28T03:59:33Z", "updated_at": "2023-09-05T14:10:39Z", "closed_at": "2018-06-28T04:02:10Z", "author_association": "OWNER", "pull_request": null, "body": "", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/328/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1870672704, "node_id": "PR_kwDOBm6k_c5Y-7Em", "number": 2162, "title": "Add new `--internal internal.db` option, deprecate legacy `_internal` database", "user": {"value": 15178711, "label": "asg017"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2023-08-29T00:05:07Z", "updated_at": "2023-08-29T03:24:23Z", "closed_at": "2023-08-29T03:24:23Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/datasette/pulls/2162", "body": "refs #2157 \r\n\r\nThis PR adds a new `--internal` option to datasette serve. If provided, it is the path to a persistent internal database that Datasette core and Datasette plugins can use to store data, as discussed in the proposal issue. \r\n\r\nThis PR also removes and deprecates the previous in-memory `_internal` database. Those tables now appear in the `internal` database, with `core_` prefixes (ex `tables` in `_internal` is now `core_tables` in `internal`).\r\n\r\n\r\n## A note on the new `core_` tables\r\nHowever, one important notes about those new `core_` tables: If a `--internal` DB is passed in, that means those `core_` tables will persist across multiple Datasette instances. This wasn't the case before, since `_internal` was always an in-memory database created from scratch.\r\n\r\nI tried to put those `core_` tables as `TEMP` tables - after all, there's always one 1 `internal` DB connection at a time, so I figured it would work. But, since we use the `Database()` wrapper for the internal DB, it has two separate connections: a default read-only connection and a write connection that is created when a write operation occurs. Which meant the `TEMP` tables would be created by the write connection, but not available in the read-only connection. \r\n\r\nSo I had a brillant idea: Attach an in-memory named database with `cache=shared`, and create those tables there! \r\n\r\n```sql\r\nATTACH DATABASE 'file:datasette_internal_core?mode=memory&cache=shared' AS core;\r\n```\r\n\r\nWe'd run this on both the read-only connection and the write-only connection. That way, those tables would stay in memory, they'd communicate with the `cache=shared` feature, and we'd be good to go.\r\n\r\n\r\nHowever, I couldn't find an easy way to run a `ATTACH DATABASE` command on the read-only query. \r\n\r\nUsing `Database()` as a wrapper for the internal DB is pretty limiting - it's meant for Datasette \"data\" databases, where we want multiple readers and possibly 1 write connection at a time. But the internal database doesn't really require that kind of support - I think we could get away with a single read/write connection, but it seemed like too big of a rabbithole to go through now. \r\n\r\n\r\n\r\n----\r\n:books: Documentation preview :books:: https://datasette--2162.org.readthedocs.build/en/2162/\r\n\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2162/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1865649347, "node_id": "I_kwDOBm6k_c5vM4zD", "number": 2156, "title": "datasette -s/--setting option for setting nested configuration options", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2023-08-24T18:09:27Z", "updated_at": "2023-08-28T19:33:05Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "> I've been thinking about what it might look like to allow command-line arguments to be used to define _any_ of the configuration options in `datasette.yml`, as alternative and more convenient syntax.\r\n>\r\n> Here's what I've come up with:\r\n> ```\r\n> datasette \\\r\n> -s settings.sql_time_limit_ms 1000 \\\r\n> -s plugins.datasette-auth-tokens.manage_tokens true \\\r\n> -s plugins.datasette-auth-tokens.manage_tokens_database tokens \\\r\n> mydatabase.db tokens.db\r\n> ```\r\n> Which would be equivalent to `datasette.yml` containing this:\r\n> ```yaml\r\n> plugins:\r\n> datasette-auth-tokens:\r\n> manage_tokens: true\r\n> manage_tokens_database: tokens\r\n> settings:\r\n> sql_time_limit_ms: 1000\r\n> ```\r\nMore details in https://github.com/simonw/datasette/issues/2143#issuecomment-1690792514\r\n ", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2156/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1863810783, "node_id": "I_kwDOBm6k_c5vF37f", "number": 2150, "title": "form label { width: 15% } is a bad default", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2023-08-23T18:22:27Z", "updated_at": "2023-08-23T18:37:18Z", "closed_at": "2023-08-23T18:35:48Z", "author_association": "OWNER", "pull_request": null, "body": "See:\r\n- https://github.com/simonw/datasette-configure-fts/issues/14\r\n- https://github.com/simonw/datasette-auth-tokens/issues/12", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2150/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1843710170, "node_id": "I_kwDOBm6k_c5t5Mja", "number": 2136, "title": "Query view shouldn't return `columns`", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 9700784, "label": "Datasette 1.0a3"}, "comments": 4, "created_at": "2023-08-09T17:23:57Z", "updated_at": "2023-08-09T19:03:04Z", "closed_at": "2023-08-09T19:03:04Z", "author_association": "OWNER", "pull_request": null, "body": "I just noticed that https://latest.datasette.io/fixtures/roadside_attraction_characteristics.json?_labels=on&_size=1 returns:\r\n```json\r\n{\r\n \"ok\": true,\r\n \"next\": \"1\",\r\n \"rows\": [\r\n {\r\n \"rowid\": 1,\r\n \"attraction_id\": {\r\n \"value\": 1,\r\n \"label\": \"The Mystery Spot\"\r\n },\r\n \"characteristic_id\": {\r\n \"value\": 2,\r\n \"label\": \"Paranormal\"\r\n }\r\n }\r\n ],\r\n \"truncated\": false\r\n}\r\n```\r\nBut https://latest.datasette.io/fixtures.json?sql=select+rowid%2C+attraction_id%2C+characteristic_id+from+roadside_attraction_characteristics+order+by+rowid+limit+1 returns:\r\n```json\r\n{\r\n \"rows\": [\r\n {\r\n \"rowid\": 1,\r\n \"attraction_id\": 1,\r\n \"characteristic_id\": 2\r\n }\r\n ],\r\n \"columns\": [\r\n \"rowid\",\r\n \"attraction_id\",\r\n \"characteristic_id\"\r\n ],\r\n \"ok\": true,\r\n \"truncated\": false\r\n}\r\n```\r\nThe `columns` key in the query response is inconsistent with the table response.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2136/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1822937426, "node_id": "I_kwDOBm6k_c5sp9FS", "number": 2111, "title": "Implement new /content.json?sql=...", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 9700784, "label": "Datasette 1.0a3"}, "comments": 4, "created_at": "2023-07-26T18:22:39Z", "updated_at": "2023-08-08T02:00:37Z", "closed_at": "2023-08-08T02:00:22Z", "author_association": "OWNER", "pull_request": null, "body": "This will be the base that the remaining work builds on top of. Refs:\r\n- #2109 ", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2111/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1816917522, "node_id": "PR_kwDOCGYnMM5WJ6Jm", "number": 573, "title": "feat: Implement a prepare_connection plugin hook", "user": {"value": 15178711, "label": "asg017"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2023-07-22T22:48:44Z", "updated_at": "2023-07-22T22:59:09Z", "closed_at": "2023-07-22T22:59:09Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/573", "body": "Just like the [Datasette prepare_connection hook](https://docs.datasette.io/en/stable/plugin_hooks.html#prepare-connection-conn-database-datasette), this PR adds a similar hook for the `sqlite-utils` plugin system. \r\n\r\nThe sole argument is `conn`, since I don't believe a `database` or `datasette` argument would be relevant here. \r\n\r\nI want to do this so I can release `sqlite-utils` plugins for my [SQLite extensions](https://github.com/asg017/sqlite-ecosystem), similar to the Datasette plugins I've release for them. \r\n\r\nAn example plugin: https://gist.github.com/asg017/d7cdf0d56e2be87efda28cebee27fa3c\r\n\r\n```bash\r\n$ sqlite-utils install https://gist.github.com/asg017/d7cdf0d56e2be87efda28cebee27fa3c/archive/5f5ad549a40860787629c69ca120a08c32519e99.zip\r\n\r\n$ sqlite-utils memory 'select hello(\"alex\") as response'\r\n[{\"response\": \"Hello, alex!\"}]\r\n```\r\nRefs:\r\n- #574 \r\n\r\n\r\n----\r\n:books: Documentation preview :books:: https://sqlite-utils--573.org.readthedocs.build/en/573/\r\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/573/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1355148385, "node_id": "I_kwDOBm6k_c5Qxexh", "number": 1796, "title": "Research an upgrade to CodeMirror 6", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-08-30T04:27:46Z", "updated_at": "2023-07-03T04:58:21Z", "closed_at": "2023-07-03T04:58:21Z", "author_association": "OWNER", "pull_request": null, "body": "There are still a bunch of bugs in CodeMirror 5 that affect various mobile browsers - see Datasette Discord report here: https://discord.com/channels/823971286308356157/823971286941302908/1013878624992108645\r\n\r\nhttps://user-images.githubusercontent.com/9599/187349269-7b7c0c8c-3894-4810-82f0-de7c1eb940b3.mp4\r\n\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1796/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1575131737, "node_id": "I_kwDOCGYnMM5d4ppZ", "number": 525, "title": "Repeated calls to `Table.convert()` fail", "user": {"value": 167893, "label": "mcarpenter"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2023-02-07T22:40:47Z", "updated_at": "2023-05-08T21:59:41Z", "closed_at": "2023-05-08T21:54:02Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "## Summary\r\nWhen using the API, repeated calls to `Table.convert()` do not work correctly since all conversions quietly use the callable (function, lambda) from the first call to `convert()` only. Subsequent invocations with different callables use the callable from the first invocation only.\r\n\r\n## Example\r\n```python\r\nfrom sqlite_utils import Database\r\n\r\ndb = Database(memory=True)\r\ntable = db['table']\r\ncol = 'x'\r\ntable.insert_all([{col: 1}])\r\nprint(table.get(1))\r\n\r\ntable.convert(col, lambda x: x*2)\r\nprint(table.get(1))\r\n\r\ndef zeroize(x):\r\n return 0\r\n#zeroize = lambda x: 0\r\n#zeroize.__name__ = 'zeroize'\r\ntable.convert(col, zeroize)\r\nprint(table.get(1))\r\n```\r\n\r\nOutput:\r\n```\r\n{'x': 1}\r\n{'x': 2}\r\n{'x': 4}\r\n```\r\nExpected:\r\n```\r\n{'x': 1}\r\n{'x': 2}\r\n{'x': 0}\r\n```\r\n\r\n## Explanation\r\nThis is some relevant [documentation](https://github.com/simonw/sqlite-utils/blob/1491b66dd7439dd87cd5cd4c4684f46eb3c5751b/docs/python-api.rst#registering-custom-sql-functions:~:text=By%20default%20registering%20a%20function%20with%20the%20same%20name%20and%20number%20of%20arguments%20will%20have%20no%20effect).\r\n\r\n * `Table.convert()` takes a `Callable` to perform data conversion on a column\r\n * The `Callable` is passed to `Database.register_function()`\r\n * `Database.register_function()` uses the callable's `__name__` attribute for registration\r\n * (Aside: all lambdas have a `__name__` of ``: I thought this was the problem, and it was close, but not quite)\r\n * However `convert()` first wraps the callable by local function [`convert_value()`](https://github.com/simonw/sqlite-utils/blob/fc221f9b62ed8624b1d2098e564f525c84497969/sqlite_utils/db.py#L2661)\r\n * Consequently `register_function()` sees name `convert_value` for all invocations from `convert()`\r\n * `register_function()` silently ignores registrations using the same name, retaining only the first such registration\r\n\r\nThere's a mismatch between the comments and the code: https://github.com/simonw/sqlite-utils/blob/fc221f9b62ed8624b1d2098e564f525c84497969/sqlite_utils/db.py#L404\r\n\r\nbut actually the existing function is returned/used instead (as the \"registering custom sql functions\" doc I linked above says too). Seems like this can be rectified to match the comment?\r\n\r\n## Suggested fix\r\nI think there are four things:\r\n1. The call to `register_function()` from `convert()`should have an explicit `name=` parameter (to continue using `convert_value()` and the progress bar).\r\n2. For functions, this name can be the real function name. (I understand the sqlite api needs a name, and it's nice if those are recognizable names where possible). For lambdas would `'lambda-{uuid}'` or similar be acceptable? \r\n3. `register_function()` really should throw an error on repeated attempts to register a duplicate (function, arity)-pair.\r\n4. A test? I haven't looked at the test framework here but seems this should be testable.\r\n\r\n## See also \r\n- #458 ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/525/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1699184583, "node_id": "I_kwDOCGYnMM5lR3_H", "number": 540, "title": "sphinx.builders.linkcheck build error", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2023-05-07T18:37:09Z", "updated_at": "2023-05-08T04:56:13Z", "closed_at": "2023-05-07T18:42:36Z", "author_association": "OWNER", "pull_request": null, "body": "https://readthedocs.org/projects/sqlite-utils/builds/20512693/\r\n```\r\n Running Sphinx v6.2.1\r\n\r\nTraceback (most recent call last):\r\n File \"/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/registry.py\", line 442, in load_extension\r\n mod = import_module(extname)\r\n File \"/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/importlib/__init__.py\", line 127, in import_module\r\n return _bootstrap._gcd_import(name[level:], package, level)\r\n File \"\", line 1014, in _gcd_import\r\n File \"\", line 991, in _find_and_load\r\n File \"\", line 975, in _find_and_load_unlocked\r\n File \"\", line 671, in _load_unlocked\r\n File \"\", line 783, in exec_module\r\n File \"\", line 219, in _call_with_frames_removed\r\n File \"/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/builders/linkcheck.py\", line 20, in \r\n from requests import Response\r\n File \"/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/requests/__init__.py\", line 43, in \r\n import urllib3\r\n File \"/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/urllib3/__init__.py\", line 38, in \r\n raise ImportError(\r\nImportError: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2n 7 Dec 2017. See: https://github.com/urllib3/urllib3/issues/2168\r\n\r\nThe above exception was the direct cause of the following exception:\r\n\r\nTraceback (most recent call last):\r\n File \"/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/cmd/build.py\", line 280, in build_main\r\n app = Sphinx(args.sourcedir, args.confdir, args.outputdir,\r\n File \"/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/application.py\", line 225, in __init__\r\n self.setup_extension(extension)\r\n File \"/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/application.py\", line 404, in setup_extension\r\n self.registry.load_extension(self, extname)\r\n File \"/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/registry.py\", line 445, in load_extension\r\n raise ExtensionError(__('Could not import extension %s') % extname,\r\nsphinx.errors.ExtensionError: Could not import extension sphinx.builders.linkcheck (exception: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2n 7 Dec 2017. See: https://github.com/urllib3/urllib3/issues/2168)\r\n\r\nExtension error:\r\nCould not import extension sphinx.builders.linkcheck (exception: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2n 7 Dec 2017. See: https://github.com/urllib3/urllib3/issues/2168)\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/540/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1699174055, "node_id": "I_kwDOCGYnMM5lR1an", "number": 539, "title": "`--raw-lines` option, like `--raw` for multiple lines", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2023-05-07T18:07:46Z", "updated_at": "2023-05-07T18:43:24Z", "closed_at": "2023-05-07T18:26:18Z", "author_association": "OWNER", "pull_request": null, "body": "I wanted to output newline-separated output of the first column of every row in the results - like `--row` but for more than one line.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/539/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1393202060, "node_id": "I_kwDOCGYnMM5TCpOM", "number": 496, "title": "devrel/python api: Pylance type hinting", "user": {"value": 7908073, "label": "chapmanjacobd"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-10-01T03:03:34Z", "updated_at": "2023-05-03T05:53:27Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "Pylance is generally pretty good at figuring out stuff but `sqlite-utils` has some quirks which make type hinting kinda useless. Maybe you don't care but I thought I would bring it to your attention.\r\n\r\nFor example:\r\n\r\n```\r\ndb[\"subs\"].insert_all(subs, pk=\"index\")\r\n```\r\n\r\n```\r\nCannot access member \"insert_all\" for type \"View\"\r\n Member \"insert_all\" is unknown\r\n```\r\n\r\n`insert_all` and all the other methods show up as a type issues because the program can't know whether something is a View or a Table. Fair enough. But that basically throws all type checking out the window.\r\n\r\n`pk=\"index\"` also shows up as a type issue:\r\n\r\n```\r\nArgument of type \"Literal['index']\" cannot be assigned to parameter \"pk\" of type \"Default\" in function \"insert_all\"\r\n \"Literal['index']\" is incompatible with \"Default\"\r\n```\r\n\r\nI think this is because DEFAULT is an empty class? \r\n\r\nmaybe a few small changes could be made to make the library more type-friendly\r\n\r\nThe interim solution is of course to turn off type hints completely for the line\r\n```\r\ndb[\"subs\"].insert_all(subs, pk=\"index\") # type: ignore\r\n```\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/496/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1686033652, "node_id": "I_kwDOBm6k_c5kftT0", "number": 2065, "title": "Datasette cannot be installed with Rye", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2023-04-27T03:35:42Z", "updated_at": "2023-04-27T05:09:36Z", "closed_at": "2023-04-27T05:09:36Z", "author_association": "OWNER", "pull_request": null, "body": "https://github.com/mitsuhiko/rye\r\n\r\nI tried this:\r\n\r\n rye install datasette\r\n\r\nBut now:\r\n\r\n```\r\n% ~/.rye/shims/datasette\r\nTraceback (most recent call last):\r\n File \"/Users/simon/.rye/shims/datasette\", line 5, in \r\n from datasette.cli import cli\r\n File \"/Users/simon/.rye/tools/datasette/lib/python3.11/site-packages/datasette/cli.py\", line 17, in \r\n from .app import (\r\n File \"/Users/simon/.rye/tools/datasette/lib/python3.11/site-packages/datasette/app.py\", line 14, in \r\n import pkg_resources\r\nModuleNotFoundError: No module named 'pkg_resources'\r\n```\r\nI think that's because `setuptools` is not included in Rye.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2065/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1373210675, "node_id": "I_kwDODD6af85R2Ygz", "number": 13, "title": "fails before generating views. ERR: table sqlite_master may not be modified", "user": {"value": 116795, "label": "pax"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-09-14T15:41:50Z", "updated_at": "2023-04-11T03:46:17Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "generates checkins.db but seems to fail before generating views \r\n\r\nnote: it worked on an Ubuntu WSL but fails on macOS 12.5.1\r\n\r\nlater edit: I suspect this is a problem with my local set-up, `dogsheep-beta index` also throws the same error\r\n\r\nfull error:\r\n\r\n Importing 2591 checkins [###################################-] 98% 00:00:00\r\n Traceback (most recent call last):\r\n File \"/Users/pax/devbox/envAll/bin/swarm-to-sqlite\", line 8, in \r\n sys.exit(cli())\r\n File \"/Users/pax/devbox/envAll/lib/python3.8/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/Users/pax/devbox/envAll/lib/python3.8/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/Users/pax/devbox/envAll/lib/python3.8/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/Users/pax/devbox/envAll/lib/python3.8/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/Users/pax/devbox/envAll/lib/python3.8/site-packages/swarm_to_sqlite/cli.py\", line 77, in cli\r\n ensure_foreign_keys(db)\r\n File \"/Users/pax/devbox/envAll/lib/python3.8/site-packages/swarm_to_sqlite/utils.py\", line 145, in ensure_foreign_keys\r\n db[fk.table].add_foreign_key(fk.column, fk.other_table, fk.other_column)\r\n File \"/Users/pax/devbox/envAll/lib/python3.8/site-packages/sqlite_utils/db.py\", line 2123, in add_foreign_key\r\n self.db.add_foreign_keys([(self.name, column, other_table, other_column)])\r\n File \"/Users/pax/devbox/envAll/lib/python3.8/site-packages/sqlite_utils/db.py\", line 1086, in add_foreign_keys\r\n cursor.execute(\r\n sqlite3.OperationalError: table sqlite_master may not be modified", "repo": {"value": 205429375, "label": "swarm-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/13/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1620164673, "node_id": "PR_kwDOCGYnMM5L08O8", "number": 531, "title": "Add paths for homebrew on Apple silicon", "user": {"value": 25778, "label": "eyeseast"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2023-03-11T22:27:52Z", "updated_at": "2023-04-09T01:49:44Z", "closed_at": "2023-04-09T01:49:43Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/531", "body": "This also passes in the extension path when specified in GIS methods. Wherever we know an extension path, we use `db.init_spatialite(find_spatialite() or load_extension)`.\r\n\r\n\r\n----\n:books: Documentation preview :books:: https://sqlite-utils--531.org.readthedocs.build/en/531/\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/531/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1646734246, "node_id": "I_kwDOBm6k_c5iJyum", "number": 2049, "title": "Custom SQL queries should use new JSON ?_extra= format", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 8755003, "label": "Datasette 1.0a-next"}, "comments": 4, "created_at": "2023-03-30T00:42:53Z", "updated_at": "2023-04-05T23:29:27Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Related:\r\n- #262\r\n\r\nI've made the change to the table view, now I need the new format to work for arbitrary SQL queries too.\r\n\r\nNote that this incorporates both arbitrary SQL queries and canned queries.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2049/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1590183272, "node_id": "I_kwDOBm6k_c5eyEVo", "number": 2027, "title": "How to redirect from \"/\" to a specific db/table", "user": {"value": 1350673, "label": "dmick"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2023-02-18T03:14:01Z", "updated_at": "2023-03-08T04:42:22Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Using nginx to redirect public IP to the local uvicorn server as 'normal'. I can't figure out how to redirect such that '/' results in accessing the one db/table I want to serve; redirecting / to /db/table breaks some of the CSS; fooling with base_url doesn't seem to help. Can someone explain this, if it's possible?", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2027/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1515815014, "node_id": "I_kwDOBm6k_c5aWYBm", "number": 1973, "title": "render_cell plugin hook's row object is not a sqlite.Row", "user": {"value": 193185, "label": "cldellow"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2023-01-01T20:27:46Z", "updated_at": "2023-01-29T00:40:31Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "From https://docs.datasette.io/en/stable/plugin_hooks.html#render-cell-row-value-column-table-database-datasette:\r\n\r\n> row - sqlite.Row\r\n> The SQLite row object that the value being rendered is part of\r\n\r\nThis appears to actually be a [CustomRow](https://github.com/simonw/datasette/blob/f0fadc28ddb9f82e5cc1ecaa51e8a342eb6dc528/datasette/utils/__init__.py#L773-L789), but I think that's unrelated to my issue.\r\n\r\nI have a table:\r\n\r\n```sql\r\nCREATE TABLE IF NOT EXISTS \"dss_job_stats\"(\r\n job_id integer not null references dss_job(id) on delete cascade,\r\n host text not null,\r\n // other columns elided as irrelevant\r\n primary key (job_id, host)\r\n);\r\n```\r\n\r\nOn datasette 0.63.2, the `render_cell` hook receives a `row` value that looks like:\r\n\r\n```\r\nCustomRow([('job_id', {'value': 2, 'label': '2'}), ('host', 'cldellow.com')])\r\n```\r\n\r\nI expected the `job_id` value to be `2`, but it's actually `{'value': 2, 'label': '2'}`.\r\n\r\nI can work around this, but was wondering if this was intended behaviour?", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1973/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1553615704, "node_id": "I_kwDOBm6k_c5cmktY", "number": 2001, "title": "Datasette is not compatible with SQLite's strict quoting compilation option", "user": {"value": 406380, "label": "gwk"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2023-01-23T19:10:07Z", "updated_at": "2023-01-25T04:59:58Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I have linked Python3.11 on macOS against recent SQLite that was compiled using `-DSQLITE_DQS=0`. This option disables interpretation of double-quoted identifiers as string literals, described in the SQLite docs as a \"MySQL 3.x misfeature\". See https://www.sqlite.org/quirks.html#dblquote for background.\r\n\r\nDatasette uses the double-quote syntax in a number of key places, and is thus completely broken in this environment.\r\n\r\nMy experience was to `pip install datasette`, then run `datasette serve -I my-data.db`. When I visit `http://127.0.0.1:8001` I get a 500 response.\r\n\r\nThe error: `sqlite3.OperationalError: no such column: geometry_columns`\r\n\r\nThe responsible SQL: `'select 1 from sqlite_master where tbl_name = \"geometry_columns\"'`\r\n\r\nI then installed datasette from GitHub master in development mode and changed the offending SQL to use correct quotes: `\"select 1 from sqlite_master where tbl_name = 'geometry_columns'\"`.\r\n\r\nWith this change, I get a little further, but have the same problem with the first table name in my database (in my case, \"Meta\"):\r\n```\r\nOperationalError: no such column: Meta\r\nTraceback (most recent call last):\r\n File \"/Users/gwk/external/datasette/datasette/app.py\", line 1522, in route_path\r\n response = await view(request, send)\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/Users/gwk/external/datasette/datasette/views/base.py\", line 151, in view\r\n return await self.dispatch_request(request)\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/Users/gwk/external/datasette/datasette/views/base.py\", line 105, in dispatch_request\r\n response = await handler(request)\r\n ^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/Users/gwk/external/datasette/datasette/views/index.py\", line 70, in get\r\n \"fts_table\": await db.fts_table(table),\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/Users/gwk/external/datasette/datasette/database.py\", line 363, in fts_table\r\n return await self.execute_fn(lambda conn: detect_fts(conn, table))\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/Users/gwk/external/datasette/datasette/database.py\", line 213, in execute_fn\r\n return await asyncio.get_event_loop().run_in_executor(\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/usr/local/py/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/thread.py\", line 58, in run\r\n result = self.fn(*self.args, **self.kwargs)\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/Users/gwk/external/datasette/datasette/database.py\", line 211, in in_thread\r\n return fn(conn)\r\n ^^^^^^^^\r\n File \"/Users/gwk/external/datasette/datasette/database.py\", line 363, in \r\n return await self.execute_fn(lambda conn: detect_fts(conn, table))\r\n ^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/Users/gwk/external/datasette/datasette/utils/__init__.py\", line 588, in detect_fts\r\n rows = conn.execute(detect_fts_sql(table)).fetchall()\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\nsqlite3.OperationalError: no such column: Meta\r\nINFO: 127.0.0.1:50258 - \"GET / HTTP/1.1\" 500 Internal Server Error\r\n```\r\n\r\nI will try to continue playing with this, but I also hope that the datasette developers will enable this mode in a test environment as I am unlikely to be able to exercise all of the SQL in the codebase, or make a pull request very soon.\r\n\r\nNote that the DQS setting compile-time option can be overridden at runtime with calls to the C API:\r\n```\r\nsqlite3_db_config(db, SQLITE_DBCONFIG_DQS_DDL, 0, (void*)0);\r\nsqlite3_db_config(db, SQLITE_DBCONFIG_DQS_DML, 0, (void*)0);\r\n```\r\n\r\nAs far as I can tell, `sqlite3_db_config` is not exposed in Python, but perhaps we could figure out how to invoke it using `ctypes`.\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2001/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1529707837, "node_id": "I_kwDOBm6k_c5bLX09", "number": 1988, "title": "Reconsider pattern where plugins could break existing template context", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 4, "created_at": "2023-01-11T21:13:43Z", "updated_at": "2023-01-11T21:25:05Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "> I hadn't run into an issue with plugins like `datasette-template-sql` interfering with the existing context for other features before! Definitely not a good thing.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/datasette-write/issues/6#issuecomment-1379490596_\r\n ", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1988/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1529452371, "node_id": "I_kwDOBm6k_c5bKZdT", "number": 1987, "title": "installpython3.com is now a spam website", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2023-01-11T17:55:12Z", "updated_at": "2023-01-11T18:29:26Z", "closed_at": "2023-01-11T18:29:25Z", "author_association": "OWNER", "pull_request": null, "body": "Need to stop linking to it from the docs.\r\n\r\nI'll link to https://www.python.org/about/gettingstarted/ instead.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1987/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 806849424, "node_id": "MDU6SXNzdWU4MDY4NDk0MjQ=", "number": 1221, "title": "Support SSL/TLS directly", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-02-12T00:18:29Z", "updated_at": "2022-12-18T02:39:04Z", "closed_at": "2021-02-12T00:52:18Z", "author_association": "OWNER", "pull_request": null, "body": "This should be pretty easy because Uvicorn supports them already. Need a good mechanism for testing it - https://pypi.org/project/trustme/ looks ideal.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1221/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1384549993, "node_id": "I_kwDOBm6k_c5Sho5p", "number": 1818, "title": "Setting to turn off table row counts entirely", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-09-24T06:39:22Z", "updated_at": "2022-12-11T02:03:09Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "There are situations - such as loading SQLite files remotely using HTTP range headers - where counting all of the rows in a table should be avoided entirely.\r\n\r\n> > Also, this chunked inefficiency means that I have to hack the URL to not load tables of a database as it seems to try to load the whole database when I click on a database.\r\n>\r\n> I bet that's because Datasette tries to show a count of all of the rows in each table when it shows the list on that page, which triggers a full table scan.\r\n>\r\n> Would be great to have a setting that turns that feature off, which could then be exposed as a query string option for Datasette Lite.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/datasette-lite/issues/49#issuecomment-1256880715_\r\n ", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1818/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1198822563, "node_id": "I_kwDOBm6k_c5HdJSj", "number": 1706, "title": "[feature] immutable mode for a directory, not just individual sqlite file", "user": {"value": 9020979, "label": "hydrosquall"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-04-10T00:50:57Z", "updated_at": "2022-12-09T19:11:40Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "## Motivation\r\n\r\n- I have a directory of sqlite databases\r\n- I'd like to use immutable mode when opening them for better performance [docs](https://docs.datasette.io/en/0.54/performance.html#immutable-mode)\r\n- Currently using this flag throws the following error\r\n\r\n IsADirectoryError: [Errno 21] Is a directory: '/name-of-directory'\r\n\r\n## Proposal\r\n\r\nImmutable flag works for both single files and directories\r\n\r\n datasette -i /folder-of-sqlite-files", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1706/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1473659191, "node_id": "I_kwDOBm6k_c5X1kE3", "number": 1929, "title": "Incorrect link from the API explorer to the JSON API documentation", "user": {"value": 3556, "label": "davidbgk"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-12-03T02:08:58Z", "updated_at": "2022-12-06T19:36:23Z", "closed_at": "2022-12-06T19:34:20Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "I installed `datasette==1.0a1`.\r\n\r\nWhen I go to http://127.0.0.1:8001/-/api I have a link: `Use this tool to try out the [Datasette API](https://docs.datasette.io/en/1.0a1/json_api.html).` but that documentation page does not exist.\r\n\r\nI'm not sure where it has to be fixed, should it link to the stable page https://docs.datasette.io/en/stable/json_api.html , the latest one https://docs.datasette.io/en/latest/json_api.html#the-json-write-api or would it be more appropriated to deploy documentation for the `1.0a1` version?", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1929/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1479914599, "node_id": "I_kwDOCGYnMM5YNbRn", "number": 516, "title": "Feature request: output number of ignored/replaced rows for insert command", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-12-06T18:59:21Z", "updated_at": "2022-12-06T19:08:14Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "https://hachyderm.io/@briandorsey/109468185742876820\r\n\r\n> I'm fiddling with piping json to `insert -ignore` I'd love to see the count of records inserted & ignored, but didn't see a way to do that in the help/docs.\r\n>\r\n> Example: `xh \"https://hachyderm.io/api/v1/timelines/tag/rust?max_id=109443380308326328\" | sqlite-utils insert aoc.db aoc - --pk=id --ignore`", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/516/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1470509936, "node_id": "I_kwDOBm6k_c5XpjNw", "number": 1924, "title": "Docs for replace:true and ignore:true options for insert API", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7867486, "label": "Datasette 1.0a1"}, "comments": 4, "created_at": "2022-12-01T01:33:25Z", "updated_at": "2022-12-01T18:15:15Z", "closed_at": "2022-12-01T02:08:02Z", "author_association": "OWNER", "pull_request": null, "body": "Equivalent to https://sqlite-utils.datasette.io/en/stable/cli.html#insert-replacing-data", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1924/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1450303205, "node_id": "I_kwDOBm6k_c5Wcd7l", "number": 1891, "title": "1.0a0 release notes", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 8658075, "label": "Datasette 1.0a0"}, "comments": 4, "created_at": "2022-11-15T19:58:20Z", "updated_at": "2022-11-29T19:23:41Z", "closed_at": "2022-11-29T19:23:41Z", "author_association": "OWNER", "pull_request": null, "body": "This release will mainly help preview the new Datasette write API:\r\n- #1850", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1891/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1425029275, "node_id": "I_kwDOBm6k_c5U8Dib", "number": 1864, "title": "Delete a single record from an existing table", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 8658075, "label": "Datasette 1.0a0"}, "comments": 4, "created_at": "2022-10-27T04:53:22Z", "updated_at": "2022-11-29T18:54:04Z", "closed_at": "2022-11-29T18:54:04Z", "author_association": "OWNER", "pull_request": null, "body": "API design:\r\n```\r\nPOST /db/table/row-pks/-/delete\r\nOr...\r\nDELETE /db/table/row-pks/-/delete\r\n```\r\nI'm just going to do `POST` for the moment, like I did here:\r\n- #1874\r\n\r\nPermission: `delete-row`\r\n\r\nStill needed:\r\n\r\n- [ ] Tests for rowid tables\r\n- [ ] Tests for compound primary keys", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1864/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1456012874, "node_id": "I_kwDOBm6k_c5WyP5K", "number": 1905, "title": "`publish heroku` failing due to old Python version", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-11-19T00:01:45Z", "updated_at": "2022-11-19T01:12:05Z", "closed_at": "2022-11-19T00:52:29Z", "author_association": "OWNER", "pull_request": null, "body": "Reported on Discord: https://discord.com/channels/823971286308356157/823971286941302908/1042814317118115901\r\n\r\n```\r\n-----> Building on the Heroku-22 stack\r\n-----> Determining which buildpack to use for this app\r\n-----> Python app detected\r\n-----> Using Python version specified in runtime.txt\r\n ! Requested runtime 'python-3.8.10' is not available for this stack (heroku-22).\r\n ! For supported versions, see: https://devcenter.heroku.com/articles/python-support\r\n ! Push rejected, failed to compile Python app.\r\n\r\n ! Push failed\r\n \u25b8 Build failed\r\n```\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1905/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1452364777, "node_id": "I_kwDOBm6k_c5WkVPp", "number": 1896, "title": "Extract logic for resolving a URL to a database / table / row", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 8658075, "label": "Datasette 1.0a0"}, "comments": 4, "created_at": "2022-11-16T22:25:20Z", "updated_at": "2022-11-18T22:57:47Z", "closed_at": "2022-11-18T22:56:55Z", "author_association": "OWNER", "pull_request": null, "body": "> In trying to write this I realize that there's a lot of duplicated code with delete row, specifically around resolving the incoming URL into a row (or a database or a table).\r\n>\r\n> Since this is so common, I think it's worth extracting the logic out first.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1863#issuecomment-1317755263_\r\n ", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1896/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1452495049, "node_id": "I_kwDOBm6k_c5Wk1DJ", "number": 1899, "title": "Clicking within the CodeMirror area below the SQL (i.e. when there's only a single line) doesn't cause the editor to get focused ", "user": {"value": 95570, "label": "bgrins"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-11-17T00:29:52Z", "updated_at": "2022-11-18T07:28:28Z", "closed_at": "2022-11-18T07:20:53Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "After the upgrade to 6 (#1893) I noticed this. I think it's because we're doing overflow:hidden to accomplish the CSS resizer.\r\n\r\nWhen there's a single line of SQL there's a gap below that line where clicking doesn't do anything. It should focus at the end of the line.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1899/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1433576351, "node_id": "I_kwDOBm6k_c5VcqOf", "number": 1880, "title": "Datasette with many and large databases > Memory use", "user": {"value": 525934, "label": "amitkoth"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-11-02T18:10:27Z", "updated_at": "2022-11-16T17:50:29Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "> Datasette maintains an in-memory SQLite database with details of the the databases, tables and columns for all of the attached databases.\r\n\r\nThe above is from the docs ^. There's two problems here - the number of datasette \"instances\" in a single server/VM and the size of the database itself. We want the **opposite** of in-memory, including what happens on SQLlite - documented in https://www.sqlite.org/inmemorydb.html\r\n\r\nFrom the context in https://github.com/simonw/datasette/issues/1150 - does it mean datasette is memory-bound to the size of the dataset - which might be a deal-breaker for many large-scale use cases?\r\n\r\nIn an extreme case - let's say a single server had 100 SQLlite databases, which would enable 100 \"instances\" of datasette to run, one per client (e.g. in a SaaS multi-tenant environment). How could we achieve all these goals:\r\n\r\n1. Allow any _one_ of these 100 databases to grow to say 2Tb in size \r\n2. Have one datasette instance, which connects to 1 of the 100 instances, based on incoming credentials/tenant ID\r\n3. Minimize memory use entirely - both by datasette and SQLlite, such that almost all operations are executed in real-time on-disk with little to no memory consumption per-tenant, or per-database.\r\n\r\nAny ideas appreciated - we're looking to use this in a SaaS type of setting - many instances, single server.\r\n\r\n@simonw great work on datasette, in general! Possibly related to https://github.com/simonw/datasette/issues/1480 but we don't want use any kind of serverless infra - this is a long-running VM/server.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1880/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1429030341, "node_id": "I_kwDOBm6k_c5VLUXF", "number": 1874, "title": "API to drop a table", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 8658075, "label": "Datasette 1.0a0"}, "comments": 4, "created_at": "2022-10-30T21:55:11Z", "updated_at": "2022-11-15T19:59:53Z", "closed_at": "2022-11-14T05:45:06Z", "author_association": "OWNER", "pull_request": null, "body": "`POST /db/table/-/drop`\r\n\r\nRequire `drop-table` permission.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1874/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1423364990, "node_id": "I_kwDOBm6k_c5U1tN-", "number": 1858, "title": "`max_signed_tokens_ttl` setting for a maximum duration on API tokens", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 8658075, "label": "Datasette 1.0a0"}, "comments": 4, "created_at": "2022-10-26T03:05:53Z", "updated_at": "2022-11-15T19:58:52Z", "closed_at": "2022-10-27T03:15:05Z", "author_association": "OWNER", "pull_request": null, "body": "It's currently possible to use `/-/create-token` to create a token that lasts forever.\r\n\r\nSome administrators may wish to have a maximum expiry instead. I should support that with a setting.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1858/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 473083260, "node_id": "MDU6SXNzdWU0NzMwODMyNjA=", "number": 50, "title": "\"Too many SQL variables\" on large inserts", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2019-07-25T21:43:31Z", "updated_at": "2022-11-04T14:38:36Z", "closed_at": "2019-07-28T11:59:33Z", "author_association": "OWNER", "pull_request": null, "body": "Reported here: https://github.com/dogsheep/healthkit-to-sqlite/issues/9\r\n\r\nIt looks like there's a default limit of 999 variables - we need to be smart about that, maybe dynamically lower the batch size based on the number of columns.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/50/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 802513359, "node_id": "MDU6SXNzdWU4MDI1MTMzNTk=", "number": 1217, "title": "Possible to deploy as a python app (for Rstudio connect server)?", "user": {"value": 6165713, "label": "plpxsk"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-02-05T22:21:24Z", "updated_at": "2022-11-04T11:37:52Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Is it possible to deploy a `datasette` application as a python\u00a0web app?\r\n\r\nIn my enterprise, I have option to deploy python apps via [Rstudio Connect](https://github.com/rstudio/rsconnect-python), and I would like to publish a `datasette` dashboard for sharing.\r\n\r\nI welcome any pointers to converting `datasette serve` into a python app that can be run as something like `python datasette.py --my_data.db`", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1217/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1342430983, "node_id": "I_kwDOBm6k_c5QA98H", "number": 1786, "title": "Adjust height of textarea for no JS case", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-08-18T01:15:15Z", "updated_at": "2022-10-27T21:50:12Z", "closed_at": "2022-08-18T16:06:09Z", "author_association": "OWNER", "pull_request": null, "body": "Datasette Lite: https://lite.datasette.io/?sql=https://gist.githubusercontent.com/simonw/1f8a91123ccefd8844187225b1832d7a/raw/5069075b86aa79358fbab3d4482d1d269077d632/recipes.sql#/data?sql=select+id%2C+name%2C+ingredients%2C+%28%0A++select+json_group_array%28value%29+from+json_each%28ingredients%29%0A++where+value+in+%28select+value+from+json_each%28%3Ap0%29%29%0A%29+as+matching_ingredients%0Afrom+recipes%0Awhere+json_array_length%28matching_ingredients%29+%3E+0%0Aorder+by+json_array_length%28matching_ingredients%29+desc&p0=%5B%22sugar%22%2C+%22cheese%22%5D\r\n\r\n![46F8101E-8CE3-4F61-B200-F865E6B5DBCC](https://user-images.githubusercontent.com/9599/185270723-f55513b0-b561-434d-9d7c-4fe5be9756e0.jpeg)\r\n\r\n\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1786/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1413641049, "node_id": "I_kwDOCGYnMM5UQnNZ", "number": 501, "title": "Tests failing due to updated tabulate library", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-10-18T18:07:52Z", "updated_at": "2022-10-18T18:23:40Z", "closed_at": "2022-10-18T18:23:40Z", "author_association": "OWNER", "pull_request": null, "body": "Failure here: https://github.com/simonw/sqlite-utils/actions/runs/3275786702/jobs/5391063221\r\n\r\nI figured out the problem:\r\n\r\n```diff\r\ndiff --git a/docs/cli-reference.rst b/docs/cli-reference.rst\r\nindex b88e38a..82b4b6c 100644\r\n--- a/docs/cli-reference.rst\r\n+++ b/docs/cli-reference.rst\r\n@@ -112,11 +112,15 @@ See :ref:`cli_query`.\r\n --tsv Output TSV\r\n --no-headers Omit CSV headers\r\n -t, --table Output as a formatted table\r\n- --fmt TEXT Table format - one of fancy_grid, fancy_outline,\r\n- github, grid, html, jira, latex, latex_booktabs,\r\n- latex_longtable, latex_raw, mediawiki, moinmoin,\r\n- orgtbl, pipe, plain, presto, pretty, psql, rst,\r\n- simple, textile, tsv, unsafehtml, youtrack\r\n+ --fmt TEXT Table format - one of asciidoc, double_grid,\r\n+ double_outline, fancy_grid, fancy_outline, github,\r\n+ grid, heavy_grid, heavy_outline, html, jira,\r\n+ latex, latex_booktabs, latex_longtable, latex_raw,\r\n+ mediawiki, mixed_grid, mixed_outline, moinmoin,\r\n+ orgtbl, outline, pipe, plain, presto, pretty,\r\n+ psql, rounded_grid, rounded_outline, rst, simple,\r\n+ simple_grid, simple_outline, textile, tsv,\r\n+ unsafehtml, youtrack\r\n --json-cols Detect JSON cols and output them as JSON, not\r\n escaped strings\r\n -r, --raw Raw output, first column of first row\r\n@@ -176,11 +180,15 @@ See :ref:`cli_memory`.\r\n --tsv Output TSV\r\n --no-headers Omit CSV headers\r\n -t, --table Output as a formatted table\r\n- --fmt TEXT Table format - one of fancy_grid, fancy_outline,\r\n- github, grid, html, jira, latex, latex_booktabs,\r\n- latex_longtable, latex_raw, mediawiki, moinmoin,\r\n- orgtbl, pipe, plain, presto, pretty, psql, rst,\r\n- simple, textile, tsv, unsafehtml, youtrack\r\n+ --fmt TEXT Table format - one of asciidoc, double_grid,\r\n+ double_outline, fancy_grid, fancy_outline, github,\r\n+ grid, heavy_grid, heavy_outline, html, jira,\r\n+ latex, latex_booktabs, latex_longtable, latex_raw,\r\n+ mediawiki, mixed_grid, mixed_outline, moinmoin,\r\n+ orgtbl, outline, pipe, plain, presto, pretty,\r\n+ psql, rounded_grid, rounded_outline, rst, simple,\r\n+ simple_grid, simple_outline, textile, tsv,\r\n+ unsafehtml, youtrack\r\n --json-cols Detect JSON cols and output them as JSON, not\r\n escaped strings\r\n -r, --raw Raw output, first column of first row\r\n@@ -401,11 +409,14 @@ See :ref:`cli_search`.\r\n --tsv Output TSV\r\n --no-headers Omit CSV headers\r\n -t, --table Output as a formatted table\r\n- --fmt TEXT Table format - one of fancy_grid, fancy_outline,\r\n- github, grid, html, jira, latex, latex_booktabs,\r\n- latex_longtable, latex_raw, mediawiki, moinmoin,\r\n- orgtbl, pipe, plain, presto, pretty, psql, rst, simple,\r\n- textile, tsv, unsafehtml, youtrack\r\n+ --fmt TEXT Table format - one of asciidoc, double_grid,\r\n+ double_outline, fancy_grid, fancy_outline, github,\r\n+ grid, heavy_grid, heavy_outline, html, jira, latex,\r\n+ latex_booktabs, latex_longtable, latex_raw, mediawiki,\r\n+ mixed_grid, mixed_outline, moinmoin, orgtbl, outline,\r\n+ pipe, plain, presto, pretty, psql, rounded_grid,\r\n+ rounded_outline, rst, simple, simple_grid,\r\n+ simple_outline, textile, tsv, unsafehtml, youtrack\r\n --json-cols Detect JSON cols and output them as JSON, not escaped\r\n strings\r\n --load-extension TEXT Path to SQLite extension, with optional :entrypoint\r\n@@ -651,11 +662,14 @@ See :ref:`cli_tables`.\r\n --tsv Output TSV\r\n --no-headers Omit CSV headers\r\n -t, --table Output as a formatted table\r\n- --fmt TEXT Table format - one of fancy_grid, fancy_outline,\r\n- github, grid, html, jira, latex, latex_booktabs,\r\n- latex_longtable, latex_raw, mediawiki, moinmoin,\r\n- orgtbl, pipe, plain, presto, pretty, psql, rst, simple,\r\n- textile, tsv, unsafehtml, youtrack\r\n+ --fmt TEXT Table format - one of asciidoc, double_grid,\r\n+ double_outline, fancy_grid, fancy_outline, github,\r\n+ grid, heavy_grid, heavy_outline, html, jira, latex,\r\n+ latex_booktabs, latex_longtable, latex_raw, mediawiki,\r\n+ mixed_grid, mixed_outline, moinmoin, orgtbl, outline,\r\n+ pipe, plain, presto, pretty, psql, rounded_grid,\r\n+ rounded_outline, rst, simple, simple_grid,\r\n+ simple_outline, textile, tsv, unsafehtml, youtrack\r\n --json-cols Detect JSON cols and output them as JSON, not escaped\r\n strings\r\n --columns Include list of columns for each table\r\n@@ -689,11 +703,14 @@ See :ref:`cli_views`.\r\n --tsv Output TSV\r\n --no-headers Omit CSV headers\r\n -t, --table Output as a formatted table\r\n- --fmt TEXT Table format - one of fancy_grid, fancy_outline,\r\n- github, grid, html, jira, latex, latex_booktabs,\r\n- latex_longtable, latex_raw, mediawiki, moinmoin,\r\n- orgtbl, pipe, plain, presto, pretty, psql, rst, simple,\r\n- textile, tsv, unsafehtml, youtrack\r\n+ --fmt TEXT Table format - one of asciidoc, double_grid,\r\n+ double_outline, fancy_grid, fancy_outline, github,\r\n+ grid, heavy_grid, heavy_outline, html, jira, latex,\r\n+ latex_booktabs, latex_longtable, latex_raw, mediawiki,\r\n+ mixed_grid, mixed_outline, moinmoin, orgtbl, outline,\r\n+ pipe, plain, presto, pretty, psql, rounded_grid,\r\n+ rounded_outline, rst, simple, simple_grid,\r\n+ simple_outline, textile, tsv, unsafehtml, youtrack\r\n --json-cols Detect JSON cols and output them as JSON, not escaped\r\n strings\r\n --columns Include list of columns for each view\r\n@@ -732,11 +749,15 @@ See :ref:`cli_rows`.\r\n --tsv Output TSV\r\n --no-headers Omit CSV headers\r\n -t, --table Output as a formatted table\r\n- --fmt TEXT Table format - one of fancy_grid, fancy_outline,\r\n- github, grid, html, jira, latex, latex_booktabs,\r\n- latex_longtable, latex_raw, mediawiki, moinmoin,\r\n- orgtbl, pipe, plain, presto, pretty, psql, rst,\r\n- simple, textile, tsv, unsafehtml, youtrack\r\n+ --fmt TEXT Table format - one of asciidoc, double_grid,\r\n+ double_outline, fancy_grid, fancy_outline, github,\r\n+ grid, heavy_grid, heavy_outline, html, jira,\r\n+ latex, latex_booktabs, latex_longtable, latex_raw,\r\n+ mediawiki, mixed_grid, mixed_outline, moinmoin,\r\n+ orgtbl, outline, pipe, plain, presto, pretty,\r\n+ psql, rounded_grid, rounded_outline, rst, simple,\r\n+ simple_grid, simple_outline, textile, tsv,\r\n+ unsafehtml, youtrack\r\n --json-cols Detect JSON cols and output them as JSON, not\r\n escaped strings\r\n --load-extension TEXT Path to SQLite extension, with optional\r\n@@ -768,11 +789,14 @@ See :ref:`cli_triggers`.\r\n --tsv Output TSV\r\n --no-headers Omit CSV headers\r\n -t, --table Output as a formatted table\r\n- --fmt TEXT Table format - one of fancy_grid, fancy_outline,\r\n- github, grid, html, jira, latex, latex_booktabs,\r\n- latex_longtable, latex_raw, mediawiki, moinmoin,\r\n- orgtbl, pipe, plain, presto, pretty, psql, rst, simple,\r\n- textile, tsv, unsafehtml, youtrack\r\n+ --fmt TEXT Table format - one of asciidoc, double_grid,\r\n+ double_outline, fancy_grid, fancy_outline, github,\r\n+ grid, heavy_grid, heavy_outline, html, jira, latex,\r\n+ latex_booktabs, latex_longtable, latex_raw, mediawiki,\r\n+ mixed_grid, mixed_outline, moinmoin, orgtbl, outline,\r\n+ pipe, plain, presto, pretty, psql, rounded_grid,\r\n+ rounded_outline, rst, simple, simple_grid,\r\n+ simple_outline, textile, tsv, unsafehtml, youtrack\r\n --json-cols Detect JSON cols and output them as JSON, not escaped\r\n strings\r\n --load-extension TEXT Path to SQLite extension, with optional :entrypoint\r\n@@ -804,11 +828,14 @@ See :ref:`cli_indexes`.\r\n --tsv Output TSV\r\n --no-headers Omit CSV headers\r\n -t, --table Output as a formatted table\r\n- --fmt TEXT Table format - one of fancy_grid, fancy_outline,\r\n- github, grid, html, jira, latex, latex_booktabs,\r\n- latex_longtable, latex_raw, mediawiki, moinmoin,\r\n- orgtbl, pipe, plain, presto, pretty, psql, rst, simple,\r\n- textile, tsv, unsafehtml, youtrack\r\n+ --fmt TEXT Table format - one of asciidoc, double_grid,\r\n+ double_outline, fancy_grid, fancy_outline, github,\r\n+ grid, heavy_grid, heavy_outline, html, jira, latex,\r\n+ latex_booktabs, latex_longtable, latex_raw, mediawiki,\r\n+ mixed_grid, mixed_outline, moinmoin, orgtbl, outline,\r\n+ pipe, plain, presto, pretty, psql, rounded_grid,\r\n+ rounded_outline, rst, simple, simple_grid,\r\n+ simple_outline, textile, tsv, unsafehtml, youtrack\r\n --json-cols Detect JSON cols and output them as JSON, not escaped\r\n strings\r\n --load-extension TEXT Path to SQLite extension, with optional :entrypoint\r\ndiff --git a/docs/cli.rst b/docs/cli.rst\r\nindex 8bc4176..1d67e88 100644\r\n--- a/docs/cli.rst\r\n+++ b/docs/cli.rst\r\n@@ -187,10 +187,15 @@ Available ``--fmt`` options are:\r\n cog.out(\"\\n\" + \"\\n\".join('- ``{}``'.format(t) for t in tabulate.tabulate_formats) + \"\\n\\n\")\r\n .. ]]]\r\n \r\n+- ``asciidoc``\r\n+- ``double_grid``\r\n+- ``double_outline``\r\n - ``fancy_grid``\r\n - ``fancy_outline``\r\n - ``github``\r\n - ``grid``\r\n+- ``heavy_grid``\r\n+- ``heavy_outline``\r\n - ``html``\r\n - ``jira``\r\n - ``latex``\r\n@@ -198,15 +203,22 @@ Available ``--fmt`` options are:\r\n - ``latex_longtable``\r\n - ``latex_raw``\r\n - ``mediawiki``\r\n+- ``mixed_grid``\r\n+- ``mixed_outline``\r\n - ``moinmoin``\r\n - ``orgtbl``\r\n+- ``outline``\r\n - ``pipe``\r\n - ``plain``\r\n - ``presto``\r\n - ``pretty``\r\n - ``psql``\r\n+- ``rounded_grid``\r\n+- ``rounded_outline``\r\n - ``rst``\r\n - ``simple``\r\n+- ``simple_grid``\r\n+- ``simple_outline``\r\n - ``textile``\r\n - ``tsv``\r\n - ``unsafehtml``\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/501/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1413610718, "node_id": "I_kwDOCGYnMM5UQfze", "number": 500, "title": "Turn --flatten into a documented utility function", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-10-18T17:43:36Z", "updated_at": "2022-10-18T18:02:10Z", "closed_at": "2022-10-18T18:00:40Z", "author_association": "OWNER", "pull_request": null, "body": "The `--flatten` implementation isn't currently available to Python code - people have to roll their own implementation. Feedback from a conversation at DjangoCon.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/500/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 703246031, "node_id": "MDU6SXNzdWU3MDMyNDYwMzE=", "number": 51, "title": "github-to-sqlite should handle rate limits better", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-09-17T04:01:50Z", "updated_at": "2022-10-14T16:34:07Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "From #50 - right now it will crash with an error of it hits the rate limit. Since the rate limit information (including reset time) is available in the headers it could automatically sleep and try again instead.", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/51/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 377155320, "node_id": "MDU6SXNzdWUzNzcxNTUzMjA=", "number": 370, "title": "Integration with JupyterLab", "user": {"value": 82988, "label": "psychemedia"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2018-11-04T13:57:13Z", "updated_at": "2022-09-29T08:17:47Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "I just watched a demo video for the [JupyterLab Chart Editor](https://www.crowdcast.io/e/introducing-JupyterLab-Chart-Editor/) which wraps the plotly chart editor app in a JupyterLab panel and lets you open a plotly chart JSON file in that editor. Essentially, it pops an HTML app into a panel in JupyterLab, and I think registers the app as a file viewer for a particular file type. (I'm not completely taken by it, tbh, because it means you can do irreproducible things to the chart definition file, but that's another issue).\r\n\r\nJupyterLab extensions can also open files from a dialogue as the iframe/html previewer shows: https://github.com/timkpaine/jupyterlab_iframe.\r\n\r\nThis made me wonder about what `datasette` integration with JupyterLab might do.\r\n\r\nFor example, by right-clicking on a CSV file (for which there is already a CSV table view) in the file browser, offer a *View / Run as datasette* file viewer option that will:\r\n\r\n- run the CSV file through `csvs-to-sqlite`;\r\n- launch the `datasette` server and display the `datasette` view in a JupyterLab panel.\r\n\r\n(? Create a new SQLite db for each CSV file and launch each datasette view on a new port? Or have a JupyterLab (session?) SQLite db that stores all `datasette` viewed CSVs and runs on a single port?) \r\n\r\nAs a freebie, the `datasette` API would allow you to run efficient SQL queries against the file eg using using `pandas.read_sql()` queries in a notebook in the same space.\r\n\r\nRelated:\r\n\r\n- [JupyterLab extensions docs](https://jupyterlab.readthedocs.io/en/stable/user/extensions.html)\r\n- a [cookiecutter for wrting JupyterLab extensions using Javascript](https://github.com/jupyterlab/extension-cookiecutter-js)\r\n- a [cookiecutter for writing JupyterLab extensions using Typescript](https://github.com/jupyterlab/extension-cookiecutter-ts)\r\n- tutorial: [Let\u2019s Make an xkcd JupyterLab Extension](https://jupyterlab.readthedocs.io/en/stable/developer/xkcd_extension_tutorial.html)", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/370/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1382457780, "node_id": "I_kwDOCGYnMM5SZqG0", "number": 490, "title": "Ability to insert multi-line files", "user": {"value": 6180701, "label": "jeqo"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-09-22T13:29:22Z", "updated_at": "2022-09-26T18:24:44Z", "closed_at": "2022-09-23T16:37:58Z", "author_association": "NONE", "pull_request": null, "body": "I was looking into how to parse application log files that contain multiline text (e.g. Java stack traces) into sqlite. \r\nI can see that at the moment `--lines` helps, but falls short when processing multi-line texts.\r\n\r\nI wonder if this functionality would be useful for sqlite-utils. A similar approach to Elastic logstash/filebeat can be adopted: https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html \r\n\r\nPotential changes:\r\n\r\n- add a `--multiline` option\r\n- additional properties for\r\n - multiline-pattern (regex expression)\r\n - multiline-negate: true/false\r\n - multiline-what: previous or next\r\n\r\nOr if this is achievable in a different way, please share. Thanks!", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/490/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1374626873, "node_id": "I_kwDOBm6k_c5R7yQ5", "number": 1810, "title": "Featured table(s) on the homepage", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-09-15T14:30:49Z", "updated_at": "2022-09-15T15:51:25Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Many Datasette instances mainly exist to serve a single table - for example:\r\n\r\n- https://global-power-plants.datasettes.com/global-power-plants/global-power-plants\r\n- https://laion-aesthetic.datasette.io/laion-aesthetic-6pls/images\r\n\r\nIt would be neat if the / homepage of those instances could be configured to highlight that specific table.\r\n\r\nOr maybe more than one?", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1810/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1359557737, "node_id": "I_kwDOBm6k_c5RCTRp", "number": 1798, "title": "Parts of YAML file do not work when db name is \"off\"", "user": {"value": 562352, "label": "CharlesNepote"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-09-01T22:10:57Z", "updated_at": "2022-09-02T00:02:53Z", "closed_at": "2022-09-01T23:56:33Z", "author_association": "NONE", "pull_request": null, "body": "I guess this issue is not very important and probably rare.\r\n\r\nTo reproduce:\r\n* create and populate a db named `off.db`\r\n* in the yaml file, add any kind of information below `databases:\\n off:`\r\n* the data are not taken into account (because \"off\" is interpreted as \"false\")\r\n\r\nYAML file:\r\n```yaml\r\ntitle: Some title\r\ndescription_html: |-\r\n

This is an experiment.

\r\ndatabases:\r\n off:\r\n tables:\r\n products_from_owners:\r\n title: products_from_owners*\r\n description_html: |-\r\n

Description

\r\n```\r\n\r\nThe result for http://xxxx.xxx/-/metadata gives:\r\n```json\r\n{\r\n \"title\": \"Some title\",\r\n \"description_html\": \"

This is an experiment.

\",\r\n \"databases\": {\r\n \"false\": {\r\n \"tables\": {\r\n \"products_from_owners\": {\r\n \"title\": \"products_from_owners*\",\r\n \"description_html\": \"

Description

\"\r\n }\r\n }\r\n }\r\n }\r\n}\r\n```\r\n=> see the `\"false\"` instead of `\"off\"`.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1798/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1355433619, "node_id": "PR_kwDOCGYnMM4-B7Mc", "number": 480, "title": "search_sql add include_rank option", "user": {"value": 7908073, "label": "chapmanjacobd"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-08-30T09:10:29Z", "updated_at": "2022-08-31T03:40:35Z", "closed_at": "2022-08-31T03:40:35Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/480", "body": "I haven't tested this yet but wanted to get a heads-up whether this kind of change would be useful or if I should just duplicate the function and tweak it within my code\r\n\r\n\r\n----\n:books: Documentation preview :books:: https://sqlite-utils--480.org.readthedocs.build/en/480/\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/480/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 855476501, "node_id": "MDU6SXNzdWU4NTU0NzY1MDE=", "number": 1298, "title": "improve table horizontal scroll experience", "user": {"value": 192568, "label": "mroswell"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-04-12T01:55:16Z", "updated_at": "2022-08-30T21:11:49Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "Wide tables aren't a huge problem if you know to click and drag right. But it's not at all obvious to do that. (it also tends to blue-select any content as it's dragging.) Depending on column widths, public users might entirely miss all the columns to the right. \r\n\r\nThere is a scrollbar at the bottom of the table, but I'm displaying ALL my records because it's the only way for datasette-vega to make accurate charts. So that bottom scrollbar is likely to be missed. I wonder if some sort of javascript-y mouseover to an arrow might help, similar to those seen in image carousels. Ah: here's a perfect example:\r\n\r\n1. Visit http://google.com\r\n2. Search for: animals endangered\r\n3. Note the 'g-right-button' (in the code) that looks like a right-facing caret in a circle. \r\n4. Click on that and the carousel scrolls right (and 'g-left-button' appears on the left).\r\n\r\nMight be tricky to do that on a table, rather than a one-row carousel, but it's worth experimenting with.\r\n\r\nAnother option is just to put the scrollbars at the top of the table, too. \r\n\r\nMeantime, I'm trying to build a button like the \"View/hide all columns on https://salaries.news.baltimoresun.com/salaries-be494cf/2019+Maryland+state+salaries\r\nMight be nice to have that available by default, with settings in the metadata showing which are on by default.\r\n\r\n(I saw some other closed issues related to horizontal scrolling, and admit I don't entirely understand them. For instance, the animated gif at https://github.com/simonw/datasette/issues/998#issuecomment-714117534 confuses me. )\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1298/reactions\", \"total_count\": 4, \"+1\": 4, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1326087800, "node_id": "PR_kwDOCGYnMM48hI-_", "number": 460, "title": "Cross-link CLI to Python docs", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-08-02T16:18:28Z", "updated_at": "2022-08-18T21:58:10Z", "closed_at": "2022-08-18T21:58:07Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/460", "body": "Work in progress, partly to test the ReadTheDocs preview link action.\r\n\r\nRefs:\r\n- #426\r\n\r\n\r\n----\n:books: Documentation preview :books:: https://readthedocs-preview--460.org.readthedocs.build/en/460/\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/460/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1318907685, "node_id": "I_kwDOBm6k_c5OnO8l", "number": 1773, "title": "500 error if sorted by a column not in the ?_col= list", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 8303187, "label": "Datasette 0.62"}, "comments": 4, "created_at": "2022-07-27T01:20:27Z", "updated_at": "2022-08-14T16:06:25Z", "closed_at": "2022-08-14T15:44:05Z", "author_association": "OWNER", "pull_request": null, "body": "For example: https://latest.datasette.io/fixtures/sortable?_sort_desc=sortable&_col=sortable_with_nulls\r\n\r\nThat's `?_sort_desc=sortable&_col=sortable_with_nulls`\r\n\r\n\"image\"\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1773/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 779156520, "node_id": "MDU6SXNzdWU3NzkxNTY1MjA=", "number": 1175, "title": "Use structlog for logging", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-01-05T15:11:36Z", "updated_at": "2022-07-26T12:52:10Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "To solve #241 JSON logging.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1175/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1279863844, "node_id": "I_kwDOCGYnMM5MSSwk", "number": 449, "title": "Utilities for duplicating tables and creating a table with the results of a query", "user": {"value": 1690072, "label": "davidleejy"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-06-22T09:41:43Z", "updated_at": "2022-07-15T21:46:13Z", "closed_at": "2022-07-15T21:21:36Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "is there a duplicate table functionality? Otherwise, I'd be happy to submit a PR.\r\n\r\nIn sqlite3 it would look like:\r\n\r\n```python\r\nimport sqlite3 as sl\r\n\r\ncon = sl.connect('prompt-tune.db')\r\n\r\ndef db_duplicate_table(table_name, table_name_new, con=con):\r\n # Duplicates table `table_name` to a new table `table_name_new`.\r\n try:\r\n cur = con.cursor()\r\n cur.execute(f\"\"\"CREATE TABLE {table_name_new} AS SELECT * FROM {table_name}\"\"\")\r\n except Exception as e:\r\n print(e)\r\n finally:\r\n cur.close()\r\n\r\ndb_duplicate_table('orig_table', 'new_table')\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/449/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 727848625, "node_id": "MDU6SXNzdWU3Mjc4NDg2MjU=", "number": 12, "title": "Some workout columns should be float, not text", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-10-23T02:47:02Z", "updated_at": "2022-06-23T04:35:02Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "Columns `duration`, `totalDistance` and `totalEnergyBurned` should be converted to float.\r\n\r\nhttps://github.com/dogsheep/healthkit-to-sqlite/blob/71e36e1cf034b96de2a8e6652265d782d3fdf63b/healthkit_to_sqlite/utils.py#L50-L57", "repo": {"value": 197882382, "label": "healthkit-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/12/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1257724585, "node_id": "I_kwDOCGYnMM5K91qp", "number": 441, "title": "Combining `rows_where()` and `search()` to limit which rows are searched", "user": {"value": 1448859, "label": "betatim"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-06-02T06:01:55Z", "updated_at": "2022-06-14T21:57:57Z", "closed_at": "2022-06-14T21:54:38Z", "author_association": "NONE", "pull_request": null, "body": "What is the right way to limit a full text search query to some rows of a table?\r\n\r\nFor example, I have a table that contains the following columns: `title`, `content`, `owner` (each row represents a document). The `owner` column is a username. It feels right to store all documents in one table, instead of having one table per owner. In particular because I'd like to full text search all documents, only documents owned by one user and documents owned by a set of users.\r\n\r\nI tried to combine `.rows_where(\"owner = ?\", \"1234\")` and `.search()` from the `Table` class but I don't think that is meant to work. I discovered `.search_sql()` as a way to generate the FTS SQL statement. By hand I can edit it to add a `AND [original].[owner] = :owner` to the `where` clause. This seems to do what I want.\r\n\r\nMy two questions:\r\n1. is adding a `AND ...` to the `where` clause actually the right thing to do or should I be doing something else (my SQL skills are low)?\r\n2. is there a built-in to sqlite-utils way to achieve this?\r\n\r\nRight now I am thinking I will make my own version of `search_sql()` that generates a query that contains an additional `owner = :owner` for my particular use-case.\r\n\r\nBonus question: is this generally useful/something to add to sqlite-utils or too niche?", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/441/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1237586379, "node_id": "I_kwDOBm6k_c5JxBHL", "number": 1742, "title": "?_trace=1 fails with datasette-geojson for some reason", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-05-16T19:06:05Z", "updated_at": "2022-05-16T19:42:13Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "view-source:https://calands.datasettes.com/calands/CPAD_2020a_SuperUnits.geojson?_sort=id&id__exact=4&_labels=on&_trace=1 is showing me a blank page.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1742/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1223459734, "node_id": "I_kwDOBm6k_c5I7IOW", "number": 1737, "title": "Automated test for Pyodide compatibility", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-05-02T23:24:25Z", "updated_at": "2022-05-02T23:40:50Z", "closed_at": "2022-05-02T23:40:50Z", "author_association": "OWNER", "pull_request": null, "body": "Refs:\r\n- #1733\r\n\r\nNeed something in the test suite such that if Datasette breaks against Pyodide in the future we hear about it.\r\n\r\nI'm thinking this is an opportunity to use [shot-scraper javascript](https://github.com/simonw/shot-scraper#scraping-pages-using-javascript).", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1737/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1065432388, "node_id": "I_kwDOBm6k_c4_gTVE", "number": 1534, "title": "Maybe return JSON from HTML pages if `Accept: application/json` is sent", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-11-28T20:48:09Z", "updated_at": "2022-04-27T21:59:34Z", "closed_at": "2022-02-02T23:39:33Z", "author_association": "OWNER", "pull_request": null, "body": "Relates to #1533 - and to the work I've been doing on the https://github.com/simonw/datasette-table Web Component.\r\n\r\nIt would be useful to support users pasting in a URL to a Datasette table or query without first having to add the `.json` extension themselves - since then other systems could hit that URL with `Accept: application/json` to get back the JSON representation without first needing to read the `Link: ` header from #1533 to figure out what the URL to that JSON is.\r\n\r\n(There is weird logic deep in Datasette that says that you add `.json` to the path UNLESS the table name itself ends with `.json`, in which case you add `?_format=json` - this is super-confusing).\r\n\r\n[Update: I removed that confusing feature here: [https://simonwillison.net/2022/Mar/19/weeknotes/](https://simonwillison.net/2022/Mar/19/weeknotes/)]", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1534/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 340396247, "node_id": "MDU6SXNzdWUzNDAzOTYyNDc=", "number": 339, "title": "Expose SANIC_RESPONSE_TIMEOUT config option in a sensible way", "user": {"value": 12617395, "label": "bsilverm"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2018-07-11T20:38:06Z", "updated_at": "2022-03-21T22:22:40Z", "closed_at": "2022-03-21T22:22:34Z", "author_association": "NONE", "pull_request": null, "body": "Is it possible to configure the sql_time_limit_ms beyond 60 seconds? It seems queries are still timing out at 60 seconds when sql_time_limit_ms is set to 180000. We have a very large data set and often encounter timeouts when testing new queries from the datasette UI. We are optimizing our database as much as we can, but still may require more than 60 seconds for complex queries.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/339/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1065429936, "node_id": "I_kwDOBm6k_c4_gSuw", "number": 1532, "title": "Use datasette-table Web Component to guide the design of the JSON API for 1.0", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 4, "created_at": "2021-11-28T20:37:18Z", "updated_at": "2022-03-16T20:13:34Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "I realized that one of the reasons I'm having trouble committing to nailing down the JSON API for 1.0 is that I don't use it much myself - I use the `?_shape=array` one quite often, but I don't have any projects that are using the default, more fully-featured API.\r\n\r\nAs an experiment I built a Web Component for embedding Datasette tables on pages - https://github.com/simonw/datasette-table - and I think it's actually going to be a really useful tool for helping me dog food the v1.0 API design.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1532/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1131295060, "node_id": "I_kwDOBm6k_c5DbjFU", "number": 1634, "title": "Update Dockerfile generated by `datasette publish`", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 4, "created_at": "2022-02-11T00:07:26Z", "updated_at": "2022-03-11T17:38:08Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "The generated `Dockerfile` currently looks something like this:\r\n```Dockerfile\r\nFROM python:3.8\r\nCOPY . /app\r\nWORKDIR /app\r\n\r\nENV DATASETTE_SECRET 'edab49cbc5d5f6f33238f54852037e3fee710821960b73edd2ce743454182ae2'\r\nRUN pip install -U datasette datasette-auth-passwords datasette-tiddlywiki datasette-graphql\r\nRUN datasette inspect fixtures.db other.db --inspect-file inspect-data.json\r\nENV PORT 8080\r\nEXPOSE 8080\r\nCMD datasette serve --host 0.0.0.0 -i fixtures.db -i other.db --cors --inspect-file inspect-data.json --metadata metadata.json --create --port $PORT /data/*.db\r\n```\r\nThis is still on Python 3.8, and it generates a pretty large image compared to the `Dockerfile` used for https://hub.docker.com/datasetteproject/datasette - https://github.com/simonw/datasette/blob/0.60.2/Dockerfile\r\n\r\nHere's the code that generates it: https://github.com/simonw/datasette/blob/7d24fd405f3c60e4c852c5d746c91aa2ba23cf5b/datasette/utils/__init__.py#L389-L400", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1634/reactions\", \"total_count\": 2, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 2, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1154399841, "node_id": "I_kwDOBm6k_c5Ezr5h", "number": 1645, "title": "Sensible `cache-control` headers for static assets, including those served by plugins", "user": {"value": 697092, "label": "curiousleo"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 4, "created_at": "2022-02-28T18:12:03Z", "updated_at": "2022-03-08T02:59:29Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "## What I'm seeing\r\n\r\nWith `default_cache_ttl = 86400`, I see the following:\r\n\r\nA table view returns `Cache-control: max-age=86400`:\r\n\r\n![Screenshot_20220228_190000](https://user-images.githubusercontent.com/697092/156034352-4d64683e-39c8-49af-81df-0217a5957bbd.png)\r\n\r\nA static asset returns no `Cache-control` header:\r\n\r\n![Screenshot_20220228_185933](https://user-images.githubusercontent.com/697092/156034363-d0b03cc2-5889-4ed2-b601-8c1846b8469a.png)\r\n\r\n## What I expected to see\r\n\r\nI expected the static asset to return a `Cache-control` header indicating that this response can be cached.\r\n\r\n## Why this matters\r\n\r\nI'm productionising a Datasette deployment right now and was looking into putting it behind a Varnish instance. I was surprised to see requests for static assets being served from Datasette rather than Varnish, this is what led me to look more closely at the response headers.\r\n\r\nWhile Datasette serves those static assets pretty quickly, I don't see why Datasette should serve them. By their nature, static assets like images and JS files are very cacheable, so it should be easy to serve them from a cache like Varnish.\r\n\r\n(Note that Varnish can easily be configured to override this header, enabling caching for static assets. But it would be better if this override was not necessary.)\r\n\r\n## Discussion\r\n\r\nIt seems clear to me that serving static assets without a `Cache-control` header is not ideal.\r\n\r\nI see two options here:\r\n\r\nA. Static assets use the same logic as table / SQL views to set the `Cache-control` header based on `default_cache_ttl`.\r\nB. An additional setting for static assets is introduced (`default_static_cache_ttl`, say).", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1645/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1063388037, "node_id": "I_kwDOCGYnMM4_YgOF", "number": 343, "title": "Provide function to generate hash_id from specified columns", "user": {"value": 82988, "label": "psychemedia"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-11-25T10:12:12Z", "updated_at": "2022-03-02T04:25:25Z", "closed_at": "2022-03-02T04:25:25Z", "author_association": "NONE", "pull_request": null, "body": "Hi\r\n\r\nI note that you define `_hash()` to create a `hash_id` from non-id column values in a table [here](https://github.com/simonw/sqlite-utils/blob/8f386a0d300d1b1c76132bb75972b755049fb742/sqlite_utils/db.py#L2996).\r\n\r\nIt would be useful to be able to call a complementary function to generate a corresponding `_id` from a subset of specified columns when adding items to another table, eg to support the creation of foreign keys.\r\n\r\nOr is there a better pattern for doing that?", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/343/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 677272618, "node_id": "MDU6SXNzdWU2NzcyNzI2MTg=", "number": 928, "title": "Test failures caused by failed attempts to mock pip", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-08-11T23:53:18Z", "updated_at": "2022-02-23T16:19:47Z", "closed_at": "2020-08-12T00:07:49Z", "author_association": "OWNER", "pull_request": null, "body": "Errors like this one:\r\n\r\nhttps://github.com/simonw/datasette/pull/927/checks?check_run_id=973559696\r\n\r\n```\r\n2020-08-11T23:36:39.8801334Z =================================== FAILURES ===================================\r\n2020-08-11T23:36:39.8802411Z _________________________________ test_install _________________________________\r\n2020-08-11T23:36:39.8803242Z \r\n2020-08-11T23:36:39.8804935Z thing = \r\n2020-08-11T23:36:39.8806663Z comp = 'main', import_path = 'pip._internal.cli.main'\r\n2020-08-11T23:36:39.8807696Z \r\n2020-08-11T23:36:39.8808728Z def _dot_lookup(thing, comp, import_path):\r\n2020-08-11T23:36:39.8810573Z try:\r\n2020-08-11T23:36:39.8812262Z > return getattr(thing, comp)\r\n2020-08-11T23:36:39.8817136Z E AttributeError: module 'pip._internal.cli' has no attribute 'main'\r\n2020-08-11T23:36:39.8843043Z \r\n2020-08-11T23:36:39.8855951Z /opt/hostedtoolcache/Python/3.8.5/x64/lib/python3.8/unittest/mock.py:1215: AttributeError\r\n2020-08-11T23:36:39.8873372Z \r\n2020-08-11T23:36:39.8877803Z During handling of the above exception, another exception occurred:\r\n2020-08-11T23:36:39.8906532Z \r\n2020-08-11T23:36:39.8925767Z def get_src_prefix():\r\n2020-08-11T23:36:39.8928277Z # type: () -> str\r\n2020-08-11T23:36:39.8930068Z if running_under_virtualenv():\r\n2020-08-11T23:36:39.8949721Z src_prefix = os.path.join(sys.prefix, 'src')\r\n2020-08-11T23:36:39.8951813Z else:\r\n2020-08-11T23:36:39.8969014Z # FIXME: keep src in cwd for now (it is not a temporary folder)\r\n2020-08-11T23:36:39.9012110Z try:\r\n2020-08-11T23:36:39.9013489Z > src_prefix = os.path.join(os.getcwd(), 'src')\r\n2020-08-11T23:36:39.9014538Z E FileNotFoundError: [Errno 2] No such file or directory\r\n2020-08-11T23:36:39.9016122Z \r\n2020-08-11T23:36:39.9017617Z /opt/hostedtoolcache/Python/3.8.5/x64/lib/python3.8/site-packages/pip/_internal/locations.py:50: FileNotFoundError\r\n2020-08-11T23:36:39.9018802Z \r\n2020-08-11T23:36:39.9020070Z During handling of the above exception, another exception occurred:\r\n2020-08-11T23:36:39.9020930Z \r\n2020-08-11T23:36:39.9022275Z args = (), keywargs = {}\r\n2020-08-11T23:36:39.9023183Z \r\n2020-08-11T23:36:39.9024077Z @wraps(func)\r\n2020-08-11T23:36:39.9024984Z def patched(*args, **keywargs):\r\n2020-08-11T23:36:39.9028770Z > with self.decoration_helper(patched,\r\n2020-08-11T23:36:39.9031861Z args,\r\n2020-08-11T23:36:39.9038358Z keywargs) as (newargs, newkeywargs):\r\n2020-08-11T23:36:39.9039654Z \r\n2020-08-11T23:36:39.9040566Z /opt/hostedtoolcache/Python/3.8.5/x64/lib/python3.8/unittest/mock.py:1322: \r\n2020-08-11T23:36:39.9041492Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \r\n```", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/928/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 335200136, "node_id": "MDU6SXNzdWUzMzUyMDAxMzY=", "number": 327, "title": "Explore if SquashFS can be used to shrink size of packaged Docker containers", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2018-06-24T18:15:16Z", "updated_at": "2022-02-17T23:37:24Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Inspired by this article: https://cldellow.com/2018/06/22/sqlite-parquet-vtable.html#sqlite-database-indexed--squashed\r\n\r\nhttps://en.wikipedia.org/wiki/SquashFS is \"a compressed read-only file system for Linux\" - which means it could be a really nice fit for Datasette and its read-only SQLite databases.\r\n\r\nIt would be interesting to explore a Dockerfile recipe that used SquashFS to compress the SQLite database file that was bundled up by `datasette package` and friends.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/327/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1126692066, "node_id": "I_kwDOCGYnMM5DJ_Ti", "number": 403, "title": "Document how to add a primary key to a rowid table using `sqlite-utils transform --pk`", "user": {"value": 536941, "label": "fgregg"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-02-08T01:39:40Z", "updated_at": "2022-02-09T04:22:43Z", "closed_at": "2022-02-08T19:33:59Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "*Original title: Add option for adding a new, serial, primary key*\r\n\r\nsometimes we have tables that don't have primary keys, but ought to have them. we *can* use rowid for that, but it would often be nicer to have an explicit primary key. using the current value of rowid would be fine.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/403/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1072792507, "node_id": "I_kwDOCGYnMM4_8YO7", "number": 352, "title": "`sqlite-utils insert --extract colname`", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-12-07T00:55:44Z", "updated_at": "2022-02-03T22:59:36Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Is there a reason I've not added `--extract` as an option for `sqlite-utils insert` next? There's a `extracts=` option for the various `table.insert()` etc methods - last line in this code block:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/213a0ff177f23a35f3b235386366ff132eb879f1/sqlite_utils/db.py#L2483-L2495", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/352/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1094981339, "node_id": "I_kwDOCGYnMM5BRBbb", "number": 363, "title": "Better error message if `--convert` code fails to return a dict", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-01-06T05:26:28Z", "updated_at": "2022-02-03T22:52:30Z", "closed_at": "2022-02-03T22:51:30Z", "author_association": "OWNER", "pull_request": null, "body": "Here's the traceback if your `--convert` function doesn't return a dict right now:\r\n```\r\n% sqlite-utils insert /tmp/all.db blah /tmp/log.log --convert 'all.upper()' --all \r\n\r\nTraceback (most recent call last):\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/bin/sqlite-utils\", line 33, in \r\n sys.exit(load_entry_point('sqlite-utils', 'console_scripts', 'sqlite-utils')())\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 1137, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 1062, in main\r\n rv = self.invoke(ctx)\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 1668, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 1404, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 763, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py\", line 949, in insert\r\n insert_upsert_implementation(\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py\", line 834, in insert_upsert_implementation\r\n db[table].insert_all(\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py\", line 2602, in insert_all\r\n first_record = next(records)\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py\", line 3044, in fix_square_braces\r\n for record in records:\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py\", line 831, in \r\n docs = (decode_base64_values(doc) for doc in docs)\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py\", line 86, in decode_base64_values\r\n to_fix = [\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py\", line 89, in \r\n if isinstance(doc[k], dict)\r\nTypeError: string indices must be integers\r\n```\r\nIt would be nicer if that returned a more useful error message.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/361#issuecomment-1006295276_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/363/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1065431383, "node_id": "I_kwDOBm6k_c4_gTFX", "number": 1533, "title": "Add `Link: rel=\"alternate\"` header pointing to JSON for a table/query", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 4, "created_at": "2021-11-28T20:43:25Z", "updated_at": "2022-02-02T07:56:51Z", "closed_at": "2022-02-02T07:49:33Z", "author_association": "OWNER", "pull_request": null, "body": "Originally explored in https://github.com/simonw/datasette-notebook/issues/2#issuecomment-980789406 - I wanted an efficient way to scan a list of URLs and figure out which if any of those corresponded to Datasette tables, canned queries or SQL output that could be represented as a table on a page.\r\n\r\nIt looks like a neat way to do that is with ` Link:` header like this:\r\n\r\n`Link: http://127.0.0.1:8058/fixtures/compound_three_primary_keys.json; rel=\"alternate\"; type=\"application/datasette+json\"`\r\n\r\nI can put a ` Could add support for `--batch-size` as seen in `insert`/`upsert` too - causing it to break the list up into batches and commit for each one.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/391#issuecomment-1021876055_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/392/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 752966476, "node_id": "MDU6SXNzdWU3NTI5NjY0NzY=", "number": 1114, "title": "--load-extension=spatialite not working with datasetteproject/datasette docker image", "user": {"value": 2182, "label": "danp"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-11-29T17:35:20Z", "updated_at": "2022-01-20T21:29:42Z", "closed_at": "2020-11-29T17:37:45Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "https://github.com/simonw/datasette/commit/6aa5886379dd9017215904fb28567b80018902f9 added the `--load-extension=spatialite` shortcut looking for the extension in these places:\r\n\r\nhttps://github.com/simonw/datasette/blob/12877d7a48e2aa28bb5e780f929a218f7265d849/datasette/utils/__init__.py#L56-L60\r\n\r\nHowever, in the datasetteproject/datasette docker image the file is at `/usr/local/lib/mod_spatialite.so`.\r\n\r\nThis results in the example command [here](https://docs.datasette.io/en/stable/installation.html#loading-spatialite) failing:\r\n\r\n```\r\n% docker run --rm -p 8001:8001 -v `pwd`:/mnt datasetteproject/datasette datasette -p 8001 -h 0.0.0.0 /mnt/data.db --load-extension=spatialite\r\nError: Could not find SpatiaLite extension\r\n```\r\n\r\nBut it does work when given an explicit path:\r\n\r\n```\r\n% docker run --rm -p 8001:8001 -v `pwd`:/mnt datasetteproject/datasette datasette -p 8001 -h 0.0.0.0 /mnt/data.db --load-extension=/usr/local/lib/mod_spatialite.so\r\nINFO: Started server process [1]\r\nINFO: Waiting for application startup.\r\nINFO: Application startup complete.\r\nINFO: Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit)\r\n...\r\n```\r\n\r\nPerhaps `SPATIALITE_PATHS` should include `/usr/local/lib/mod_spatialite.so`?", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1114/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 838382890, "node_id": "MDU6SXNzdWU4MzgzODI4OTA=", "number": 1273, "title": "Refresh SpatiaLite documentation", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-03-23T06:05:55Z", "updated_at": "2022-01-20T21:28:50Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "https://docs.datasette.io/en/0.55/spatialite.html was written before I had tools like [geojson-to-sqlite](https://datasette.io/tools/geojson-to-sqlite) and [shapefile-to-sqlite](https://datasette.io/tools/shapefile-to-sqlite).", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1273/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1102484126, "node_id": "I_kwDOBm6k_c5BtpKe", "number": 1595, "title": "Release notes for 0.60", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7571612, "label": "Datasette 0.60"}, "comments": 4, "created_at": "2022-01-13T22:23:14Z", "updated_at": "2022-01-14T01:37:39Z", "closed_at": "2022-01-14T01:37:39Z", "author_association": "OWNER", "pull_request": null, "body": null, "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1595/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1099586786, "node_id": "I_kwDOCGYnMM5Bilzi", "number": 383, "title": "Add documentation page with the output of `--help`", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-01-11T20:25:58Z", "updated_at": "2022-01-11T22:55:05Z", "closed_at": "2022-01-11T21:44:05Z", "author_association": "OWNER", "pull_request": null, "body": "Can be maintained using `cog` from #373. Similar in purpose to the API reference page, but this is for the CLI.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/383/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1097135860, "node_id": "I_kwDOCGYnMM5BZPb0", "number": 374, "title": "`--fmt` should imply `-t`", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 4, "created_at": "2022-01-09T08:23:07Z", "updated_at": "2022-01-10T19:27:26Z", "closed_at": "2022-01-09T18:07:59Z", "author_association": "OWNER", "pull_request": null, "body": "Not sure why I didn't implement this.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/374/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1087919372, "node_id": "I_kwDOBm6k_c5A2FUM", "number": 1578, "title": "Confirm if documented nginx proxy config works for row pages with escaped characters in their primary key", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-12-23T18:27:59Z", "updated_at": "2021-12-24T21:33:19Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Found this while working on https://github.com/simonw/datasette-tiddlywiki\r\n\r\n\"image\"\r\n\r\nThen clicking on `/tiddlywiki/tiddlers/%24%3A%2FDefaultTiddlers` returns a 404.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1578/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 781262510, "node_id": "MDU6SXNzdWU3ODEyNjI1MTA=", "number": 1181, "title": "Certain database names results in 404: \"Database not found: None\"", "user": {"value": 1470389, "label": "jieter"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 6346396, "label": "Datasette 0.54"}, "comments": 4, "created_at": "2021-01-07T12:01:16Z", "updated_at": "2021-12-21T18:25:15Z", "closed_at": "2021-01-25T05:13:19Z", "author_association": "NONE", "pull_request": null, "body": "I have a file named `test-database (1).sqlite`. When requesting the home route `/`, I see datasette is able to read it correctly:\r\n\r\n\"Screenshot\r\n\r\nHowever, if I click any of the links, datasette replies with: `Error 404 Database not found: None`\r\n\r\nIt seems the hash is crucial, as renaming the file to `database (1).sqlite` makes the error go away.\r\n\r\nThis lines checks for a single dash:\r\nhttps://github.com/simonw/datasette/blob/97fb10c17dd007a275ab743742e93e932335ad67/datasette/views/base.py#L184\r\n\r\n```\r\n$ datasette test-database\\ \\(1\\).sqlite \r\nINFO: Started server process [68314]\r\nINFO: Waiting for application startup.\r\nINFO: Application startup complete.\r\nINFO: Uvicorn running on http://127.0.0.1:8001 (Press CTRL+C to quit)\r\nINFO: 127.0.0.1:54043 - \"GET /favicon.ico HTTP/1.1\" 200 OK\r\nINFO: 127.0.0.1:54043 - \"GET / HTTP/1.1\" 200 OK\r\n...\r\nINFO: 127.0.0.1:54044 - \"GET /favicon.ico HTTP/1.1\" 200 OK\r\nINFO: 127.0.0.1:54044 - \"GET /test-database (1) HTTP/1.1\" 404 Not Found\r\n\r\n```\r\nVersion:\r\n```\r\n$ datasette --version\r\ndatasette, version 0.53\r\n```\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1181/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 706001517, "node_id": "MDU6SXNzdWU3MDYwMDE1MTc=", "number": 163, "title": "Idea: conversions= could take Python functions", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-09-22T00:37:12Z", "updated_at": "2021-12-20T00:56:52Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Right now you use `conversions=` like this:\r\n\r\n```python\r\ndb[\"example\"].insert({\r\n \"name\": \"The Bigfoot Discovery Museum\"\r\n}, conversions={\"name\": \"upper(?)\"})\r\n```\r\nHow about if you could optionally provide a Python function (or a lambda) like this?\r\n```python\r\ndb[\"example\"].insert({\r\n \"name\": \"The Bigfoot Discovery Museum\"\r\n}, conversions={\"name\": lambda s: s.upper()})\r\n```\r\nThis would work by creating a random name for that function, registering it (similar to #162), executing the SQL and then un-registering the custom function at the end.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/163/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1083246400, "node_id": "PR_kwDOBm6k_c4wAMK8", "number": 1562, "title": "Update janus requirement from <0.8,>=0.6.2 to >=0.6.2,<1.1", "user": {"value": 49699333, "label": "dependabot[bot]"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-12-17T13:11:10Z", "updated_at": "2021-12-17T23:08:29Z", "closed_at": "2021-12-17T23:08:28Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/datasette/pulls/1562", "body": "Updates the requirements on [janus](https://github.com/aio-libs/janus) to permit the latest version.\n
\nRelease notes\n

Sourced from janus's releases.

\n
\n

janus 1.0.0 release

\n
    \n
  • Dropped Python 3.6 support
    • \n
  • Janus is marked as stable, no API changes was made for years
    • \n
\n
\n
\n
\nChangelog\n

Sourced from janus's changelog.

\n
\n

1.0.0 (2021-12-17)

\n
    \n
  • Drop Python 3.6 support
    • \n
\n

0.7.0 (2021-11-24)

\n
    \n
  • Add SyncQueue and AsyncQueue Protocols to provide type hints for sync and async queues #374
    • \n
\n

0.6.2 (2021-10-24)

\n
    \n
  • Fix Python 3.10 compatibility #358
    • \n
\n

0.6.1 (2020-10-26)

\n
    \n
  • \n

    Raise RuntimeError on queue.join() after queue closing. #295

    \n
    • \n
  • \n

    Replace timeout type from Optional[int] to Optional[float] #267

    \n
    • \n
\n

0.6.0 (2020-10-10)

\n
    \n
  • \n

    Drop Python 3.5, the minimal supported version is Python 3.6

    \n
    • \n
  • \n

    Support Python 3.9

    \n
    • \n
  • \n

    Refomat with black

    \n
    • \n
\n

0.5.0 (2020-04-23)

\n
    \n
  • Remove explicit loop arguments and forbid creating queues outside event loops #246
    • \n
\n

0.4.0 (2018-07-28)

\n
    \n
  • \n

    Add py.typed macro #89

    \n
    • \n
  • \n

    Drop python 3.4 support and fix minimal version python3.5.3 #88

    \n
    • \n
  • \n

    Add property with that indicates if queue is closed #86

    \n
    • \n
\n

0.3.2 (2018-07-06)

\n
    \n
  • Fixed python 3.7 support #97
    • \n
\n\n
\n

... (truncated)

\n
\n
\nCommits\n\n
\n
\n\n\nDependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.\n\n[//]: # (dependabot-automerge-start)\n[//]: # (dependabot-automerge-end)\n\n---\n\n
\nDependabot commands and options\n
\n\nYou can trigger Dependabot actions by commenting on this PR:\n- `@dependabot rebase` will rebase this PR\n- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it\n- `@dependabot merge` will merge this PR after your CI passes on it\n- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it\n- `@dependabot cancel merge` will cancel a previously requested merge and block automerging\n- `@dependabot reopen` will reopen this PR if it is closed\n- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually\n- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)\n- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)\n- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)\n\n\n
", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1562/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1077322009, "node_id": "I_kwDOCGYnMM5ANqEZ", "number": 355, "title": "Allow users to pass a full convert() function definition", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-12-10T23:59:58Z", "updated_at": "2021-12-11T00:51:15Z", "closed_at": "2021-12-11T00:49:31Z", "author_association": "OWNER", "pull_request": null, "body": "> I think the fix for this is to change the rules about what code is accepted in both the `-` mode and the literal code string mode: you can pass in a Python expression, OR a fragment that gets turned into a function, OR code that implements its own `def convert(value)` function. So this would work too:\r\n> ```sh\r\n> sqlite-utils convert my.db mytable col1 '\r\n> def convert(value):\r\n> return value.upper()\r\n> '\r\n> ```\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991381679_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/355/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 863884805, "node_id": "MDU6SXNzdWU4NjM4ODQ4MDU=", "number": 1304, "title": "Document how to send multiple values for \"Named parameters\" ", "user": {"value": 9308268, "label": "rayvoelker"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-04-21T13:19:06Z", "updated_at": "2021-12-08T03:23:14Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "https://docs.datasette.io/en/stable/sql_queries.html#named-parameters\r\n\r\nI thought that I had seen an example of how to do this example below, but I can't seem to find it\r\n\r\n```sql\r\nselect\r\n *\r\nfrom\r\n bib\r\nwhere\r\n bib.bib_record_num in (1008088,1008092)\r\n```\r\n\r\n```sql\r\nselect\r\n *\r\nfrom\r\n bib\r\nwhere\r\n bib.bib_record_num in (:bib_record_numbers)\r\n```\r\n![image](https://user-images.githubusercontent.com/9308268/115558839-2333a480-a281-11eb-85e6-ce3bada79140.png)\r\n\r\nhttps://ilsweb.cincinnatilibrary.org/collection-analysis/current_collection-204d100?sql=select%0D%0A++*%0D%0Afrom%0D%0A++bib%0D%0Awhere%0D%0A++bib.bib_record_num+in+%28%3Abib_record_numbers%29&bib_record_numbers=1008088%2C1008092\r\n\r\nOr, maybe this isn't a fully supported feature.\r\n\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1304/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1059219106, "node_id": "I_kwDOBm6k_c4_Imai", "number": 1524, "title": "Improve Apache proxy documentation, link to demo", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-11-20T20:03:14Z", "updated_at": "2021-11-20T23:34:03Z", "closed_at": "2021-11-20T23:34:03Z", "author_association": "OWNER", "pull_request": null, "body": "> The latest demo is now live at https://datasette-apache-proxy-demo.fly.dev/prefix/fixtures/sortable?_facet=pk2\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1519#issuecomment-974697824_\r\n\r\nI'm going to put out 0.59.3 bugfix release with this, but I'd like to first improve the documentation on https://docs.datasette.io/en/stable/deploying.html#apache-proxy-configuration to highlight the new demo.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1524/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 459590021, "node_id": "MDU6SXNzdWU0NTk1OTAwMjE=", "number": 519, "title": "Decide what goes into Datasette 1.0", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 4, "created_at": "2019-06-23T15:47:41Z", "updated_at": "2021-11-15T23:26:11Z", "closed_at": "2021-11-15T23:26:11Z", "author_association": "OWNER", "pull_request": null, "body": "Datasette ASGI #272 is a big part of it... but 1.0 will generally be an indicator that Datasette is a stable platform for developers to write plugins and custom templates against. So lots to think about.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/519/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1053122092, "node_id": "I_kwDOCGYnMM4-xV4s", "number": 339, "title": "`table.lookup()` option to populate additional columns when creating a record", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-11-15T01:41:17Z", "updated_at": "2021-11-15T02:02:34Z", "closed_at": "2021-11-15T02:02:00Z", "author_association": "OWNER", "pull_request": null, "body": "> For the commits table I feel like I want a version of `table.lookup()` that can be passed additional columns to populate only if the record does not exist yet.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/git-history/issues/12#issuecomment-967455017_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/339/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 845794436, "node_id": "MDU6SXNzdWU4NDU3OTQ0MzY=", "number": 1284, "title": "Feature or Documentation Request: Individual table as home page template", "user": {"value": 192568, "label": "mroswell"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-03-31T03:56:17Z", "updated_at": "2021-11-04T03:15:01Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "It would be great to have a sample showing how to move a single database that has a single table, to the index page. I'm trying it now, and find there is a real depth of Datasette and Python understanding that's required to be successful. \r\n\r\nI've got all the basic jinja concepts down... variables, template control structures, template inheritance, template overrides, css, html, the --template-dir and --static arguments, etc. \r\n\r\nBut copying the table.html file to index.html doesn't work. There are undocumented functions and filters... I can figure some of them out (yay, url_builder.py and utils/__init__.py!) but it's a slog better handled by a much stronger Python developer. \r\n\r\nOne sample would make a world of difference. The ideal form of this documentation would be a diff between the default table.html and how that would look if essentially moved to index.html. The use case is for everyone who wants to create a public-facing website to explore a single table at the root directory. (Maybe a second bit of documentation for people who have a single database with multiple tables.)\r\n\r\n(Hmm... might be cool to have a setting for that, where it happens automagically! If only one table, then home page is at the table level. if only one database, then home page is at the database level.... as an option.)\r\n\r\nI suppose I could ignore this, and somehow do this in the DNS settings once I hook up Vercel to a domain name, maybe.. and remove the breadcrumbs in table.html... but for now, a documentation request in the form of a diff... for viewing a single table (or a single database) at the root.\r\n\r\n(Actually, there's probably room for a whole expanded section on templates. Noticed some nice table metadata in one of the datasette examples, for instance... Hmm... maybe a whole library of solutions in one place... maybe a documentation hackathon! If that's of interest, of course it's a separate issue. )\r\n\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1284/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 761915790, "node_id": "MDU6SXNzdWU3NjE5MTU3OTA=", "number": 206, "title": "sqlite-utils should suggest --csv if JSON parsing fails", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-12-11T05:17:56Z", "updated_at": "2021-10-30T15:52:17Z", "closed_at": "2021-01-03T18:42:22Z", "author_association": "OWNER", "pull_request": null, "body": "```\r\n~ % gsutil cat gs://ossf-criticality-score/python_top_200.csv | sqlite-utils insert /tmp/crit.db crit -\r\n...\r\n File \"/usr/local/Cellar/python@3.9/3.9.0_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/decoder.py\", line 337, in decode\r\n obj, end = self.raw_decode(s, idx=_w(s, 0).end())\r\n File \"/usr/local/Cellar/python@3.9/3.9.0_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/decoder.py\", line 355, in raw_decode\r\n raise JSONDecodeError(\"Expecting value\", s, err.value) from None\r\njson.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)\r\n```\r\nA nicer error message here would be one that says the JSON is invalid but suggests that maybe you could try `--csv`.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/206/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 995098231, "node_id": "MDU6SXNzdWU5OTUwOTgyMzE=", "number": 1470, "title": "?_sort=rowid with _next= returns error", "user": {"value": 19851673, "label": "eigenfoo"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-09-13T16:36:15Z", "updated_at": "2021-10-18T19:30:15Z", "closed_at": "2021-10-10T01:15:03Z", "author_association": "NONE", "pull_request": null, "body": "For example:\r\n\r\n- Go to https://cryptics.eigenfoo.xyz/clues/clues?_next=100 (this is the second page of results in a Datasette site)\r\n- Search anything using the FTS search bar. For example, searching for `hello` will take you to https://cryptics.eigenfoo.xyz/clues/clues?_search=hello&_sort=rowid&_next=100\r\n- A `500 Error: list index out of range` is raised.\r\n\r\nThis is because the search URL includes the `&_next=100` UTM parameter, carried over from where the FTS search was run. However, there isn't a second page in the search results, so a `list index out of range` error is raised. You can confirm that removing this UTM parameter from the URL returns the appropriate search results.\r\n\r\nThe FTS search request should strip any `_next` UTM parameter.\r\n\r\n---\r\n\r\n```bash\r\ndatasette, version 0.58.1\r\nsqlite-utils, version 3.17\r\n```", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1470/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 990844088, "node_id": "MDU6SXNzdWU5OTA4NDQwODg=", "number": 325, "title": "sqlite-utils memory can't deal with multiple files with the same name", "user": {"value": 144773, "label": "karlb"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-09-08T08:14:42Z", "updated_at": "2021-09-22T20:52:56Z", "closed_at": "2021-09-22T20:45:45Z", "author_association": "NONE", "pull_request": null, "body": "When I use multiple files with the same name, e.g. in `sqlite-utils memory a/bug.csv b/bug.csv`, sqlite-utils creates invalid views.\r\n```\r\nTraceback (most recent call last):\r\n File \"/home/karl/.local/bin/sqlite-utils\", line 8, in \r\n sys.exit(cli())\r\n File \"/home/karl/.local/pipx/venvs/sqlite-utils/lib/python3.9/site-packages/click/core.py\", line 1137, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/home/karl/.local/pipx/venvs/sqlite-utils/lib/python3.9/site-packages/click/core.py\", line 1062, in main\r\n rv = self.invoke(ctx)\r\n File \"/home/karl/.local/pipx/venvs/sqlite-utils/lib/python3.9/site-packages/click/core.py\", line 1668, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/home/karl/.local/pipx/venvs/sqlite-utils/lib/python3.9/site-packages/click/core.py\", line 1404, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/home/karl/.local/pipx/venvs/sqlite-utils/lib/python3.9/site-packages/click/core.py\", line 763, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/home/karl/.local/pipx/venvs/sqlite-utils/lib/python3.9/site-packages/sqlite_utils/cli.py\", line 1299, in memory\r\n db[csv_table].transform(types=tracker.types)\r\n File \"/home/karl/.local/pipx/venvs/sqlite-utils/lib/python3.9/site-packages/sqlite_utils/db.py\", line 1287, in transform\r\n self.db.execute(sql)\r\n File \"/home/karl/.local/pipx/venvs/sqlite-utils/lib/python3.9/site-packages/sqlite_utils/db.py\", line 421, in execute\r\n return self.conn.execute(sql)\r\nsqlite3.OperationalError: error in view t1: no such table: main.bug\r\n```\r\n\r\nThis can be reproduced with\r\n```sh\r\n#!/bin/bash\r\nmkdir foo\r\nmkdir bar\r\necho -e 'col1,col2\\nval1,val2' > foo/bug.csv\r\necho -e 'col3,col4\\nval3,val4' > bar/bug.csv\r\nsqlite-utils memory */bug.csv 'SELECT 1'\r\n```\r\n\r\nIdeally, the tables would get unique names by including the next path segment until the names are unique. But just making the numbered t* aliases work would be good enough.\r\n\r\nThis problem can of course be worked around by renaming the files, but it would be nice if this case was handled more gracefully.\r\n\r\nThanks a lot for this great tool!", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/325/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 978743426, "node_id": "MDU6SXNzdWU5Nzg3NDM0MjY=", "number": 13, "title": "xml.etree.ElementTree.ParseError: not well-formed (invalid token)", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-08-25T05:48:21Z", "updated_at": "2021-08-26T18:45:13Z", "closed_at": "2021-08-26T18:45:13Z", "author_association": "MEMBER", "pull_request": null, "body": "Got this error today:\r\n```\r\n(evernote-to-sqlite) /tmp % evernote-to-sqlite enex evernote.db simonwillison\\'s\\ notebook.enex \r\nImporting from ENEX [######------------------------------] 17%\r\nTraceback (most recent call last):\r\n File \"/Users/simon/.local/bin/evernote-to-sqlite\", line 8, in \r\n sys.exit(cli())\r\n File \"/Users/simon/.local/pipx/venvs/evernote-to-sqlite/lib/python3.9/site-packages/click/core.py\", line 1137, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/Users/simon/.local/pipx/venvs/evernote-to-sqlite/lib/python3.9/site-packages/click/core.py\", line 1062, in main\r\n rv = self.invoke(ctx)\r\n File \"/Users/simon/.local/pipx/venvs/evernote-to-sqlite/lib/python3.9/site-packages/click/core.py\", line 1668, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/Users/simon/.local/pipx/venvs/evernote-to-sqlite/lib/python3.9/site-packages/click/core.py\", line 1404, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/Users/simon/.local/pipx/venvs/evernote-to-sqlite/lib/python3.9/site-packages/click/core.py\", line 763, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/Users/simon/.local/pipx/venvs/evernote-to-sqlite/lib/python3.9/site-packages/evernote_to_sqlite/cli.py\", line 31, in enex\r\n save_note(db, note)\r\n File \"/Users/simon/.local/pipx/venvs/evernote-to-sqlite/lib/python3.9/site-packages/evernote_to_sqlite/utils.py\", line 36, in save_note\r\n content = ET.tostring(ET.fromstring(content_xml)).decode(\"utf-8\")\r\n File \"/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/xml/etree/ElementTree.py\", line 1347, in XML\r\n parser.feed(text)\r\nxml.etree.ElementTree.ParseError: not well-formed (invalid token): line 2, column 132\r\n```", "repo": {"value": 303218369, "label": "evernote-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 831751367, "node_id": "MDU6SXNzdWU4MzE3NTEzNjc=", "number": 246, "title": "Escaping FTS search strings", "user": {"value": 16001974, "label": "DeNeutoy"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-03-15T12:15:09Z", "updated_at": "2021-08-18T18:57:13Z", "closed_at": "2021-08-18T18:43:12Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "\r\nThanks for the excellent library, it's very nice to use!\r\n\r\nI've been building some in memory search functionality for a data annotation tool i'm making, and I got tripped up a little bit with escaping the full text search queries. First I tried using `db.quote(q)`, which doesn't work, because sqlite FTS has it's own (separate)[ query syntax](https://www2.sqlite.org/fts5.html#full_text_query_syntax). You can see this happening here also:\r\n\r\nhttp://search-24ways.herokuapp.com/24ways-f8f455f/articles?_search=acces%2A\r\n\r\nI got around this by aggressively escaping quotes inside the query string like this:\r\n\r\n```python\r\n quoted = q.replace('\"', '\"\"')\r\n quoted = f'\"{quoted}\"'\r\n print(quoted)\r\n results = db[\"data\"].search(quoted, columns=[\"id\"])\r\n return [x[\"id\"] for x in results]\r\n\r\n```\r\n\r\nThis works in the sense it doesn't crash, but it also removes access to the search query syntax. Given the well specified definition, it might be possible for sqlite-utils to provide a `db.quote_query(q)` which would intelligently escape a query whilst leaving the syntax intact. This would be very nice!\r\n\r\n\r\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/246/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 268469569, "node_id": "MDU6SXNzdWUyNjg0Njk1Njk=", "number": 39, "title": "Protect against malicious SQL that causes damage even though our DB is immutable", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 2857392, "label": "Ship first public release"}, "comments": 4, "created_at": "2017-10-25T16:44:27Z", "updated_at": "2021-08-17T23:52:07Z", "closed_at": "2017-11-05T02:53:47Z", "author_association": "OWNER", "pull_request": null, "body": "I\u2019m currently operating under the assumption that it\u2019s safe to allow arbitrary SQL statements because we are dealing with an immutable database. But this might not be the case - there are some pretty weird SQLite language extensions (ATTACH, PRAGMA etc) and I\u2019m not certain they cannot be used to break things in a way that would affect future requests to the API.\r\n\r\nSolution: provide a \u201csafe mode\u201d option which disables the ?sql= mechanism. This still leaves the URL filter lookups, so I need to make sure that those are \u201csafe\u201d.\r\n\r\nIn the future I may also implement a whitelist option where datasets can be configured to only allow specific filters against specific columns.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/39/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 965102534, "node_id": "MDU6SXNzdWU5NjUxMDI1MzQ=", "number": 311, "title": "Add reference documentation generated from docstrings", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-08-10T16:04:00Z", "updated_at": "2021-08-11T12:03:50Z", "closed_at": "2021-08-11T12:03:50Z", "author_association": "OWNER", "pull_request": null, "body": "Using https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html\r\n\r\nI'm not a big fan of this kind of documentation because it so often comes in place of narrative documentation - but the library has great narrative documentation now, so the reference documentation can link to it in places.\r\n\r\nThis will also encourage me to add good docstrings everywhere, useful for IDEs and suchlike.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/311/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 963527045, "node_id": "MDU6SXNzdWU5NjM1MjcwNDU=", "number": 1424, "title": "Document exceptions that can be raised by db.execute() and friends", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-08-08T22:23:25Z", "updated_at": "2021-08-08T22:27:31Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Not currently covered here: https://docs.datasette.io/en/stable/internals.html#await-db-execute-sql", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1424/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 961367843, "node_id": "MDU6SXNzdWU5NjEzNjc4NDM=", "number": 1422, "title": "Ability to default to hiding the SQL for a canned query", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-08-05T02:51:39Z", "updated_at": "2021-08-07T05:32:29Z", "closed_at": "2021-08-07T05:32:29Z", "author_association": "OWNER", "pull_request": null, "body": "I'm working on a project with some HUGE (400+ lines of SQL) canned queries right now.\r\n\r\nAny time you land on the canned query page you have to scroll down a long distance to get to the results!\r\n\r\nWould be useful to be able to default to https://latest.datasette.io/fixtures/magic_parameters?_hide_sql=1 without needing the parameter.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1422/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 952179830, "node_id": "MDU6SXNzdWU5NTIxNzk4MzA=", "number": 2, "title": "Command for fetching Hacker News threads from the search API", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-07-25T02:00:45Z", "updated_at": "2021-07-25T03:12:57Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "I want to be able to fetch every item for a domain, e.g. https://news.ycombinator.com/from?site=simonwillison.net", "repo": {"value": 248903544, "label": "hacker-news-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/2/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 855446829, "node_id": "MDExOlB1bGxSZXF1ZXN0NjEzMTc4OTY4", "number": 1296, "title": "Dockerfile: use Ubuntu 20.10 as base", "user": {"value": 82332573, "label": "tmcl-it"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-04-12T00:23:32Z", "updated_at": "2021-07-20T08:52:13Z", "closed_at": null, "author_association": "FIRST_TIME_CONTRIBUTOR", "pull_request": "simonw/datasette/pulls/1296", "body": "This PR changes the main Dockerfile to use ubuntu:20.10 as base image instead of python:3.9.2-slim-buster (itself based on debian:buster-slim).\r\n\r\nThe Dockerfile is essentially the one from https://github.com/simonw/datasette/issues/1249#issuecomment-803698983 with some additional cleanups to slim it down.\r\n\r\nThis fixes a couple of issues:\r\n1. The SQLite version in Debian Buster (2.6.0) doesn't support generated columns\r\n2. Installing SpatiaLite from the Debian sid repositories has the side effect of also installing updates to libc and libstdc++ from sid.\r\n\r\nAs a bonus, the Docker image becomes smaller:\r\n\r\n\r\n```\r\n$ docker image ls\r\nREPOSITORY TAG IMAGE ID CREATED SIZE\r\ndatasette 0.56-ubuntu f7aca255140a 5 hours ago 212MB\r\ndatasetteproject/datasette 0.56 efb3b282f390 13 days ago 258MB\r\n```\r\n\r\n### Reproduction of the first issue\r\n\r\n```\r\n$ curl -O https://latest.datasette.io/fixtures.db\r\n % Total % Received % Xferd Average Speed Time Time Time Current\r\n Dload Upload Total Spent Left Speed\r\n100 260k 0 260k 0 0 489k 0 --:--:-- --:--:-- --:--:-- 489k\r\n\r\n$ docker run -v `pwd`:/mnt datasetteproject/datasette:0.56 datasette /mnt/fixtures.db\r\nTraceback (most recent call last):\r\n File \"/usr/local/bin/datasette\", line 8, in \r\n sys.exit(cli())\r\n File \"/usr/local/lib/python3.9/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/usr/local/lib/python3.9/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/usr/local/lib/python3.9/site-packages/click/core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/usr/local/lib/python3.9/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/usr/local/lib/python3.9/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/usr/local/lib/python3.9/site-packages/datasette/cli.py\", line 544, in serve\r\n asyncio.get_event_loop().run_until_complete(check_databases(ds))\r\n File \"/usr/local/lib/python3.9/asyncio/base_events.py\", line 642, in run_until_complete\r\n return future.result()\r\n File \"/usr/local/lib/python3.9/site-packages/datasette/cli.py\", line 584, in check_databases\r\n await database.execute_fn(check_connection)\r\n File \"/usr/local/lib/python3.9/site-packages/datasette/database.py\", line 155, in execute_fn\r\n return await asyncio.get_event_loop().run_in_executor(\r\n File \"/usr/local/lib/python3.9/concurrent/futures/thread.py\", line 52, in run\r\n result = self.fn(*self.args, **self.kwargs)\r\n File \"/usr/local/lib/python3.9/site-packages/datasette/database.py\", line 153, in in_thread\r\n return fn(conn)\r\n File \"/usr/local/lib/python3.9/site-packages/datasette/utils/__init__.py\", line 892, in check_connection\r\n for r in conn.execute(\r\nsqlite3.DatabaseError: malformed database schema (generated_columns) - near \"AS\": syntax error\r\n```\r\n\r\nHere is the SQLite version:\r\n\r\n```\r\n$ docker run -v `pwd`:/mnt -it datasetteproject/datasette:0.56 /bin/bash\r\nroot@d9220d3b95dd:/# python3\r\nPython 3.9.2 (default, Mar 27 2021, 02:50:26) \r\n[GCC 8.3.0] on linux\r\nType \"help\", \"copyright\", \"credits\" or \"license\" for more information.\r\n>>> import sqlite3\r\n>>> sqlite3.version\r\n'2.6.0'\r\n```\r\n\r\n### Reproduction of the second issue\r\n\r\n```\r\n$ docker build . -t datasette --build-arg VERSION=0.55\r\n[...snip...]\r\nThe following packages will be upgraded:\r\n libc-bin libc6 libstdc++6\r\n[...snip...]\r\nUnpacking libc6:amd64 (2.31-11) over (2.28-10) ...\r\n[...snip...]\r\nUnpacking libstdc++6:amd64 (10.2.1-6) over (8.3.0-6) ...\r\n[...snip...]\r\n```\r\n\r\nBoth libc and libstdc++ are backwards compatible, so the image still works, but it will result in a combination of libraries and Python versions that exists only in the Datasette image, so it's likely untested. In addition, since Debian sid is an always-changing rolling-release, the versions of libc, libstdc++, Spatialite, and their dependencies change frequently, so the library versions in the Datasette image will depend on the day when it was built.\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1296/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 944870799, "node_id": "MDU6SXNzdWU5NDQ4NzA3OTk=", "number": 1394, "title": "Big performance boost on faceting: skip the inner order by", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-07-14T23:32:29Z", "updated_at": "2021-07-16T02:23:32Z", "closed_at": "2021-07-15T00:05:50Z", "author_association": "OWNER", "pull_request": null, "body": "I just noticed something that could make for a huge performance improvement in faceting.\r\n\r\nThe default query used by Datasette when faceting looks like this:\r\n```sql\r\nselect\r\n country_long,\r\n count(*)\r\nfrom (\r\n select * from [global-power-plants] order by rowid\r\n)\r\nwhere\r\n country_long is not null\r\ngroup by\r\n country_long\r\norder by\r\n count(*) desc\r\n```\r\nHere it takes 53ms: https://global-power-plants.datasettes.com/global-power-plants?sql=select%0D%0A++country_long%2C%0D%0A++count%28*%29%0D%0Afrom+%28%0D%0A++select+*+from+%5Bglobal-power-plants%5D+order+by+rowid%0D%0A%29%0D%0Awhere%0D%0A++country_long+is+not+null%0D%0Agroup+by%0D%0A++country_long%0D%0Aorder+by%0D%0A++count%28*%29+desc\r\n\r\nNote that there's a `order by rowid` in there which isn't necessary - the order on that inner query doesn't matter since we're grouping and counting.\r\n\r\nI had assumed SQLite would optimize this away - but it turns out it doesn't! Consider this version of the query, with that pointless order by removed:\r\n```\r\nselect\r\n country_long,\r\n count(*)\r\nfrom (\r\n select * from [global-power-plants]\r\n)\r\nwhere\r\n country_long is not null\r\ngroup by\r\n country_long\r\norder by\r\n count(*) desc\r\n```\r\nhttps://global-power-plants.datasettes.com/global-power-plants?sql=select%0D%0A++country_long%2C%0D%0A++count%28*%29%0D%0Afrom+%28%0D%0A++select+*+from+%5Bglobal-power-plants%5D%0D%0A%29%0D%0Awhere%0D%0A++country_long+is+not+null%0D%0Agroup+by%0D%0A++country_long%0D%0Aorder+by%0D%0A++count%28*%29+desc runs in 7.2ms!\r\n\r\nI tried this optimization on a table with 2.5m rows in it - without the optimization it took 5 seconds, with the optimization it took 450ms. So this is a very significant improvement!", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1394/reactions\", \"total_count\": 2, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 1, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 466996584, "node_id": "MDExOlB1bGxSZXF1ZXN0Mjk2NzM1MzIw", "number": 557, "title": "Get tests running on Windows using Travis CI", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2019-07-11T16:36:57Z", "updated_at": "2021-07-10T23:39:48Z", "closed_at": "2021-07-10T23:39:48Z", "author_association": "OWNER", "pull_request": "simonw/datasette/pulls/557", "body": "Refs #511", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/557/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null}