{"id": 1901483874, "node_id": "PR_kwDOBm6k_c5amULw", "number": 2190, "title": "Raise an exception if a \"plugins\" block exists in metadata.json", "user": {"value": 15178711, "label": "asg017"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2023-09-18T18:08:56Z", "updated_at": "2023-10-12T16:20:51Z", "closed_at": "2023-10-12T16:20:51Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/datasette/pulls/2190", "body": "refs #2183 #2093\r\n\r\nFrom [this comment](https://github.com/simonw/datasette/pull/2183#issuecomment-1714699724) in #2183: If a `\"plugins\"` block appears in `metadata.json`, it means that a user hasn't migrated over their plugin configuration from `metadata.json` to `datasette.yaml`, which is a breaking change in Datasette 1.0. \r\n\r\nThis PR will ensure that an error is raised whenever that happens.\r\n\r\n<!-- readthedocs-preview datasette start -->\r\n----\n:books: Documentation preview :books:: https://datasette--2190.org.readthedocs.build/en/2190/\n\r\n<!-- readthedocs-preview datasette end -->", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2190/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null}
{"id": 1901768721, "node_id": "PR_kwDOBm6k_c5anSg5", "number": 2191, "title": "Move `permissions`,  `allow` blocks, canned queries and more out of `metadata.yaml` and into `datasette.yaml`", "user": {"value": 15178711, "label": "asg017"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2023-09-18T21:21:16Z", "updated_at": "2023-10-12T16:16:38Z", "closed_at": "2023-10-12T16:16:38Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/datasette/pulls/2191", "body": "The PR moves the following fields from `metadata.yaml` to `datasette.yaml`:\r\n\r\n```\r\npermissions\r\nallow\r\nallow_sql\r\nqueries\r\nextra_css_urls\r\nextra_js_urls\r\n```\r\n\r\nThis is a significant breaking change that users will need to upgrade their `metadata.yaml` files for. But the format/locations are similar to the previous version, so it shouldn't be too difficult to upgrade.\r\n\r\nOne note: I'm still working on the Configuration docs, specifically the \"reference\" section. Though it's pretty small, the rest of read to review", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2191/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null}
{"id": 1891212159, "node_id": "PR_kwDOBm6k_c5aD33C", "number": 2183, "title": "`datasette.yaml` plugin support", "user": {"value": 15178711, "label": "asg017"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2023-09-11T20:26:04Z", "updated_at": "2023-09-13T21:06:25Z", "closed_at": "2023-09-13T21:06:25Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/datasette/pulls/2183", "body": "Part of #2093\r\n\r\nIn #2149 , we ported over `\"settings.json\"` into the new `datasette.yaml` config file, with a top-level `\"settings\"` key. This PR ports over plugin configuration into top-level `\"plugins\"` key, as well as nested database/table plugin config.\r\n\r\nFrom now on, no plugin-related configuration is allowed in `metadata.yaml`, and must be in `datasette.yaml` in this new format. This is a pretty significant breaking change. Thankfully, you should be able to copy-paste your legacy plugin key/values into the new `datasette.yaml` format.\r\n\r\nAn example of what `datasette.yaml` would look like with this new plugin config:\r\n\r\n```yaml\r\n\r\nplugins:\r\n  datasette-my-plugin:\r\n    config_key: value\r\n\r\ndatabases:\r\n  fixtures:\r\n      plugins: \r\n        datasette-my-plugin:\r\n          config_key: fixtures-db-value\r\n    tables:\r\n      students:\r\n        plugins:\r\n          datasette-my-plugin:\r\n            config_key: fixtures-students-table-value\r\n\r\n```\r\n\r\nAs an additional benefit, this now works with the new `-s` flag:\r\n\r\n```bash\r\ndatasette --memory -s 'plugins.datasette-my-plugin.config_key' new_value\r\n```\r\n\r\n\r\nMarked as a \"Draft\" right now until I add better documentation. We also should have a plan for the next alpha release to document and publicize this change, especially for plugin authors (since their docs will have to change to say `datasette.yaml` instead of `metadata.yaml`\r\n\r\n<!-- readthedocs-preview datasette start -->\r\n----\n:books: Documentation preview :books:: https://datasette--2183.org.readthedocs.build/en/2183/\n\r\n<!-- readthedocs-preview datasette end -->", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2183/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null}
{"id": 1884330740, "node_id": "PR_kwDOBm6k_c5ZszDF", "number": 2174, "title": "Use $DATASETTE_INTERNAL in absence of --internal", "user": {"value": 15178711, "label": "asg017"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2023-09-06T16:07:15Z", "updated_at": "2023-09-08T00:46:13Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": "simonw/datasette/pulls/2174", "body": "#refs 2157, specifically [this comment](https://github.com/simonw/datasette/issues/2157#issuecomment-1700291967)\r\n\r\nPassing in `--internal my_internal.db` over and over again can get repetitive. \r\n\r\nThis PR adds a new configurable env variable `DATASETTE_INTERNAL_DB_PATH`. If it's defined, then it takes place as the path to the internal database. Users can still overwrite this behavior by passing in their own `--internal internal.db` flag.\r\n\r\nIn draft mode for now, needs tests and documentation. \r\n\r\nSide note: Maybe we can have a sections in the docs that lists all the \"configuration environment variables\" that Datasette respects? I did a quick grep and found:\r\n\r\n- `DATASETTE_LOAD_PLUGINS`\r\n- `DATASETTE_SECRETS`\r\n\r\n\r\n<!-- readthedocs-preview datasette start -->\r\n----\n:books: Documentation preview :books:: https://datasette--2174.org.readthedocs.build/en/2174/\n\r\n<!-- readthedocs-preview datasette end -->", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2174/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null}
{"id": 1870672704, "node_id": "PR_kwDOBm6k_c5Y-7Em", "number": 2162, "title": "Add new `--internal internal.db` option, deprecate legacy `_internal` database", "user": {"value": 15178711, "label": "asg017"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2023-08-29T00:05:07Z", "updated_at": "2023-08-29T03:24:23Z", "closed_at": "2023-08-29T03:24:23Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/datasette/pulls/2162", "body": "refs #2157 \r\n\r\nThis PR adds a new `--internal` option to datasette serve. If provided, it is the path to a persistent internal database that Datasette core and Datasette plugins can use to store data, as discussed in the proposal issue. \r\n\r\nThis PR also removes and deprecates the previous in-memory `_internal` database. Those tables now appear in the `internal` database, with `core_` prefixes (ex `tables` in `_internal` is now `core_tables` in `internal`).\r\n\r\n\r\n## A note on the new `core_` tables\r\nHowever, one important notes about those new `core_` tables: If a `--internal` DB is passed in, that means those `core_` tables will persist across multiple Datasette instances. This wasn't the case before, since `_internal` was always an in-memory database created from scratch.\r\n\r\nI tried to put those `core_` tables as `TEMP` tables - after all, there's always one 1 `internal` DB connection at a time, so I figured it would work. But, since we use the `Database()` wrapper for the internal DB, it has two separate connections: a default read-only connection and a write connection that is created when a write operation occurs. Which meant the `TEMP` tables would be created by the write connection, but not available in the read-only connection. \r\n\r\nSo I had a brillant idea: Attach an in-memory named database with `cache=shared`, and create those tables there! \r\n\r\n```sql\r\nATTACH DATABASE 'file:datasette_internal_core?mode=memory&cache=shared' AS core;\r\n```\r\n\r\nWe'd run this on both the read-only connection and the write-only connection. That way, those tables would stay in memory, they'd communicate with the `cache=shared` feature, and we'd be good to go.\r\n\r\n\r\nHowever, I couldn't find an easy way to run a `ATTACH DATABASE` command on the read-only query. \r\n\r\nUsing `Database()` as a wrapper for the internal DB is pretty limiting - it's meant for Datasette \"data\" databases, where we want multiple readers and possibly 1 write connection at a time. But the internal database doesn't really require that kind of support - I think we could get away with a single read/write connection, but it seemed like too big of a rabbithole to go through now. \r\n\r\n\r\n<!-- readthedocs-preview datasette start -->\r\n----\r\n:books: Documentation preview :books:: https://datasette--2162.org.readthedocs.build/en/2162/\r\n\r\n<!-- readthedocs-preview datasette end -->", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2162/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null}
{"id": 1864112887, "node_id": "PR_kwDOBm6k_c5Yo7bk", "number": 2151, "title": "Test Datasette on multiple SQLite versions", "user": {"value": 15178711, "label": "asg017"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-08-23T22:42:51Z", "updated_at": "2023-08-23T22:58:13Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": "simonw/datasette/pulls/2151", "body": "still testing, hope it works!\r\n\r\n<!-- readthedocs-preview datasette start -->\r\n----\n:books: Documentation preview :books:: https://datasette--2151.org.readthedocs.build/en/2151/\n\r\n<!-- readthedocs-preview datasette end -->", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2151/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 1, "state_reason": null}
{"id": 1861812208, "node_id": "PR_kwDOBm6k_c5YhH-W", "number": 2149, "title": "Start a new `datasette.yaml` configuration file, with settings support", "user": {"value": 15178711, "label": "asg017"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2023-08-22T16:24:16Z", "updated_at": "2023-08-23T01:26:11Z", "closed_at": "2023-08-23T01:26:11Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/datasette/pulls/2149", "body": "refs #2093 #2143 \r\n\r\nThis is the first step to implementing the new `datasette.yaml`/`datasette.json` configuration file. \r\n\r\n- The old `--config` argument is now back, and is the path to a `datasette.yaml` file. Acts like the `--metadata` flag.\r\n- The old `settings.json` behavior has been removed.\r\n- The `\"settings\"` key inside `datasette.yaml` defines the same `--settings` flags\r\n- Values passed in `--settings` will over-write values in `datasette.yaml`\r\n\r\nDocs for the Config file is pretty light, not much to add until we add more config to the file.\r\n\r\n<!-- readthedocs-preview datasette start -->\r\n----\n:books: Documentation preview :books:: https://datasette--2149.org.readthedocs.build/en/2149/\n\r\n<!-- readthedocs-preview datasette end -->", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2149/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null}
{"id": 1816917522, "node_id": "PR_kwDOCGYnMM5WJ6Jm", "number": 573, "title": "feat: Implement a prepare_connection plugin hook", "user": {"value": 15178711, "label": "asg017"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2023-07-22T22:48:44Z", "updated_at": "2023-07-22T22:59:09Z", "closed_at": "2023-07-22T22:59:09Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/573", "body": "Just like the [Datasette prepare_connection hook](https://docs.datasette.io/en/stable/plugin_hooks.html#prepare-connection-conn-database-datasette), this PR adds a similar hook for the `sqlite-utils` plugin system. \r\n\r\nThe sole argument is `conn`, since I don't believe a `database` or `datasette` argument would be relevant here. \r\n\r\nI want to do this so I can release `sqlite-utils` plugins for my [SQLite extensions](https://github.com/asg017/sqlite-ecosystem), similar to the Datasette plugins I've release for them. \r\n\r\nAn example plugin: https://gist.github.com/asg017/d7cdf0d56e2be87efda28cebee27fa3c\r\n\r\n```bash\r\n$ sqlite-utils install https://gist.github.com/asg017/d7cdf0d56e2be87efda28cebee27fa3c/archive/5f5ad549a40860787629c69ca120a08c32519e99.zip\r\n\r\n$ sqlite-utils memory 'select hello(\"alex\") as response'\r\n[{\"response\": \"Hello, alex!\"}]\r\n```\r\nRefs:\r\n- #574 \r\n\r\n<!-- readthedocs-preview sqlite-utils start -->\r\n----\r\n:books: Documentation preview :books:: https://sqlite-utils--573.org.readthedocs.build/en/573/\r\n\r\n<!-- readthedocs-preview sqlite-utils end -->", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/573/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null}
{"id": 1344823170, "node_id": "PR_kwDOBm6k_c49e3_k", "number": 1789, "title": "Add new entrypoint option to `--load-extension`", "user": {"value": 15178711, "label": "asg017"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2022-08-19T19:27:47Z", "updated_at": "2022-08-23T18:42:52Z", "closed_at": "2022-08-23T18:34:30Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/datasette/pulls/1789", "body": "Closes #1784 \r\n\r\nThe `--load-extension` flag can now accept an optional \"entrypoint\" value, to specify which entrypoint SQLite should load from the given extension. \r\n\r\n```bash\r\n# would load default entrypoint like before\r\ndatasette data.db --load-extension ext\r\n\r\n# loads the extensions with the \"sqlite3_foo_init\" entrpoint\r\ndatasette data.db --load-extension ext:sqlite3_foo_init\r\n\r\n# loads the extensions with the \"sqlite3_bar_init\" entrpoint\r\ndatasette data.db --load-extension ext:sqlite3_bar_init\r\n```\r\n\r\nFor testing, I added a small SQLite extension in C at `tests/ext.c`. If compiled, then pytest will run the unit tests in `test_load_extensions.py`to verify that Datasette loads in extensions correctly (and loads the correct entrypoints). Compiling the extension requires a C compiler, I compiled it on my Mac with:\r\n\r\n```\r\ngcc ext.c -I path/to/sqlite -fPIC -shared -o ext.dylib\r\n```\r\n\r\nWhere `path/to/sqlite` is a directory that contains the SQLite amalgamation header files.\r\n\r\nRe documentation: I added a bit to the help text for `--load-extension` (which I believe should auto-add to documentation?), and the existing extension documentation is spatialite specific. Let me know if a new extensions documentation page would be helpful!", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1789/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null}