{"id": 1907765514, "node_id": "I_kwDOBm6k_c5xtjEK", "number": 2195, "title": "`datasette publish` needs support for the new config/metadata split", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2023-09-21T21:08:12Z", "updated_at": "2023-09-21T22:57:48Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "> ... which raises the challenge that `datasette publish` doesn't yet know what to do with a config file!\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/datasette/issues/2194#issuecomment-1730259871_\r\n ", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2195/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1859415334, "node_id": "PR_kwDOBm6k_c5YY5Ea", "number": 2148, "title": "Bump sphinx, furo, blacken-docs dependencies", "user": {"value": 49699333, "label": "dependabot[bot]"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2023-08-21T13:48:11Z", "updated_at": "2023-08-29T00:15:31Z", "closed_at": "2023-08-29T00:15:27Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/datasette/pulls/2148", "body": "Bumps the python-packages group with 3 updates: [sphinx](https://github.com/sphinx-doc/sphinx), [furo](https://github.com/pradyunsg/furo) and [blacken-docs](https://github.com/asottile/blacken-docs).\n\nUpdates `sphinx` from 7.1.2 to 7.2.2\n
\nRelease notes\n

Sourced from sphinx's releases.

\n
\n

Sphinx 7.2.2

\n

Changelog: https://www.sphinx-doc.org/en/master/changes.html

\n

Sphinx 7.2.1

\n

Changelog: https://www.sphinx-doc.org/en/master/changes.html

\n

Sphinx 7.2.0

\n

Changelog: https://www.sphinx-doc.org/en/master/changes.html

\n
\n
\n
\nChangelog\n

Sourced from sphinx's changelog.

\n
\n

Release 7.2.2 (released Aug 17, 2023)

\n

Bugs fixed

\n\n

Release 7.2.1 (released Aug 17, 2023)

\n

Bugs fixed

\n\n

Release 7.2.0 (released Aug 17, 2023)

\n

Dependencies

\n\n

Deprecated

\n\n\n
\n

... (truncated)

\n
\n
\nCommits\n\n
\n
\n\nUpdates `furo` from 2023.7.26 to 2023.8.19\n
\nChangelog\n

Sourced from furo's changelog.

\n
\n

Changelog

\n\n

2023.08.19 -- Xenolithic Xanadu

\n\n

2023.08.17 -- Wonderous White

\n\n

2023.07.26 -- Vigilant Volt

\n\n

2023.05.20 -- Unassuming Ultramarine

\n\n

2023.03.27 -- Tasty Tangerine

\n\n

2023.03.23 -- Sassy Saffron

\n\n\n
\n

... (truncated)

\n
\n
\nCommits\n\n
\n
\n\nUpdates `blacken-docs` from 1.15.0 to 1.16.0\n
\nChangelog\n

Sourced from blacken-docs's changelog.

\n
\n

1.16.0 (2023-08-16)

\n\n
\n
\n
\nCommits\n\n
\n
\n\n\nDependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.\n\n[//]: # (dependabot-automerge-start)\n[//]: # (dependabot-automerge-end)\n\n---\n\n
\nDependabot commands and options\n
\n\nYou can trigger Dependabot actions by commenting on this PR:\n- `@dependabot rebase` will rebase this PR\n- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it\n- `@dependabot merge` will merge this PR after your CI passes on it\n- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it\n- `@dependabot cancel merge` will cancel a previously requested merge and block automerging\n- `@dependabot reopen` will reopen this PR if it is closed\n- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually\n- `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency\n- `@dependabot ignore major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself)\n- `@dependabot ignore minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself)\n- `@dependabot ignore dependency` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself)\n- `@dependabot unignore dependency` will remove all of the ignore conditions of the specified dependency\n- `@dependabot unignore ` will remove the ignore condition of the specified dependency and ignore conditions\n\n\n
\r\n\r\n\r\n----\n:books: Documentation preview :books:: https://datasette--2148.org.readthedocs.build/en/2148/\n\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2148/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1817289521, "node_id": "I_kwDOCGYnMM5sUaMx", "number": 577, "title": "Get `add_foreign_keys()` to work without modifying `sqlite_master`", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2023-07-23T20:40:18Z", "updated_at": "2023-08-18T17:43:11Z", "closed_at": "2023-08-18T00:48:10Z", "author_association": "OWNER", "pull_request": null, "body": "https://github.com/simonw/sqlite-utils/blob/13ebcc575d2547c45e8d31288b71a3242c16b886/sqlite_utils/db.py#L1165-L1174\r\n\r\nThis is the only place in the code that attempts to modify `sqlite_master` directly, which fails on some Python installations.\r\n\r\nCould this use the `.transform()` trick instead?\r\n\r\nOr automatically switch to that trick if it hits an error?", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/577/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1841501975, "node_id": "I_kwDOBm6k_c5twxcX", "number": 2133, "title": "[feature request]`datasette install plugins.json` options", "user": {"value": 54462, "label": "HaveF"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2023-08-08T15:06:50Z", "updated_at": "2023-08-10T00:31:24Z", "closed_at": "2023-08-09T22:04:46Z", "author_association": "NONE", "pull_request": null, "body": "Hi, simon \u2764\ufe0f \r\n\r\n`datasette plugins --all > plugins.json` could generate all plugins info. On another machine, it would be great to install all plugins just by `datasette install plugins.json`", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2133/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 774332247, "node_id": "MDExOlB1bGxSZXF1ZXN0NTQ1MjY0NDM2", "number": 1159, "title": "Improve the display of facets information", "user": {"value": 552629, "label": "lovasoa"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 9, "created_at": "2020-12-24T11:01:47Z", "updated_at": "2023-07-31T18:57:59Z", "closed_at": null, "author_association": "FIRST_TIME_CONTRIBUTOR", "pull_request": "simonw/datasette/pulls/1159", "body": "This PR changes the display of facets to hopefully make them more readable.\r\n\r\nBefore | After\r\n---|---\r\n![image](https://user-images.githubusercontent.com/552629/103084609-b1ec2980-45df-11eb-85bc-68ab8df3e8d9.png) | ![image](https://user-images.githubusercontent.com/552629/103085220-620e6200-45e1-11eb-8189-5dd5d3e2569e.png)\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1159/reactions\", \"total_count\": 4, \"+1\": 4, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1801394744, "node_id": "I_kwDOCGYnMM5rXxo4", "number": 567, "title": "Plugin system", "user": {"value": 15178711, "label": "asg017"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2023-07-12T17:02:14Z", "updated_at": "2023-07-22T22:59:37Z", "closed_at": "2023-07-22T22:59:36Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "I'd like there to be a plugin system for sqlite-utils, similar to the datasette/llm plugins. I'd like to make plugins that would do things like:\r\n\r\n- Register SQLite extensions for more SQL functions + virtual tables\r\n- Register new subcommands\r\n- Different input file formats for `sqlite-utils memory`\r\n- Different output file formats (in addition to `--csv` `--tsv` `--nl` etc.\r\n\r\nA few real-world use-cases of plugins I'd like to see in sqlite-utils:\r\n\r\n- Register many of my sqlite extensions in sqlite-utils (`sqlite-http`, `sqlite-lines`, `sqlite-regex`, etc.)\r\n- New subcommands to work with `sqlite-vss` vector tables\r\n- Input/ouput Parquet/Avro/Arrow IPC files with `sqlite-arrow`", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/567/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1695428235, "node_id": "I_kwDOCGYnMM5lDi6L", "number": 538, "title": "`table.upsert_all` fails to write rows when `not_null` is present", "user": {"value": 1231935, "label": "xavdid"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2023-05-04T07:30:38Z", "updated_at": "2023-05-08T20:06:35Z", "closed_at": "2023-05-08T19:27:02Z", "author_association": "NONE", "pull_request": null, "body": "I found an odd bug today, where calls to `table.upsert_all` don't write rows if you include the `not_null` kwarg.\r\n\r\n## Repro Example\r\n\r\n```py\r\nfrom sqlite_utils import Database\r\n\r\ndb = Database(\"upsert-test.db\")\r\n\r\ndb[\"comments\"].upsert_all(\r\n [{\"id\": 1, \"name\": \"david\"}],\r\n pk=\"id\",\r\n not_null=[\"name\"],\r\n)\r\n\r\nassert list(db[\"comments\"].rows) # err!\r\n```\r\n\r\nThe schema is correctly created:\r\n\r\n```sql\r\nCREATE TABLE [comments] (\r\n [id] INTEGER PRIMARY KEY,\r\n [name] TEXT NOT NULL\r\n)\r\n```\r\n\r\nBut no rows are created. Removing either the `not_null` kwargs works as expected, as does an `insert_all` call.\r\n\r\n## Version Info\r\n\r\n- Python: `3.11.0`\r\n- sqlite-utils: `3.30`\r\n- sqlite: `3.39.5 2022-10-14`", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/538/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1663399821, "node_id": "I_kwDOBm6k_c5jJXeN", "number": 2058, "title": "500 \"attempt to write a readonly database\" error caused by \"PRAGMA schema_version\"", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2023-04-11T23:57:50Z", "updated_at": "2023-04-13T16:35:21Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "I've not been able to replicate this myself yet, but I've seen log files from a user affected by it.\r\n\r\n```\r\nFile \"/usr/local/lib/python3.11/site-packages/datasette/views/base.py\", line 89, in dispatch_request\r\nawait self.ds.refresh_schemas()\r\nFile \"/usr/local/lib/python3.11/site-packages/datasette/app.py\", line 371, in refresh_schemas\r\nawait self._refresh_schemas()\r\nFile \"/usr/local/lib/python3.11/site-packages/datasette/app.py\", line 386, in _refresh_schemas\r\nschema_version = (await db.execute(\"PRAGMA schema_version\")).first()[0]\r\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\nFile \"/usr/local/lib/python3.11/site-packages/datasette/database.py\", line 267, in execute\r\nresults = await self.execute_fn(sql_operation_in_thread)\r\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\nFile \"/usr/local/lib/python3.11/site-packages/datasette/database.py\", line 213, in execute_fn\r\nreturn await asyncio.get_event_loop().run_in_executor(\r\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\nFile \"/usr/local/lib/python3.11/concurrent/futures/thread.py\", line 58, in run\r\nresult = self.fn(*self.args, **self.kwargs)\r\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\nFile \"/usr/local/lib/python3.11/site-packages/datasette/database.py\", line 211, in in_thread\r\nreturn fn(conn)\r\n^^^^^^^^\r\nFile \"/usr/local/lib/python3.11/site-packages/datasette/database.py\", line 237, in sql_operation_in_thread\r\ncursor.execute(sql, params if params is not None else {})\r\nsqlite3.OperationalError: attempt to write a readonly database\r\n```\r\nThat's running the official Datasette Docker image on https://fly.io/ - it's causing 500 errors on every page of their site.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2058/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 702386948, "node_id": "MDU6SXNzdWU3MDIzODY5NDg=", "number": 159, "title": ".delete_where() does not auto-commit (unlike .insert() or .upsert())", "user": {"value": 11712349, "label": "spdkils"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2020-09-16T01:55:52Z", "updated_at": "2023-04-01T17:21:05Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "When you use the delete_where() function on a table, it never commits....\r\n\r\nIs that intentional?", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/159/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1323346408, "node_id": "I_kwDOBm6k_c5O4Kno", "number": 1775, "title": "i18n support", "user": {"value": 428820, "label": "johnfelipe"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2022-07-31T02:51:04Z", "updated_at": "2023-02-10T18:04:40Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I want contribute for translate UI to es, de, de and it if you share strings", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1775/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1499081664, "node_id": "I_kwDOBm6k_c5ZWivA", "number": 1959, "title": "Refactor test suite to use mostly `async def` tests", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2022-12-15T21:02:54Z", "updated_at": "2022-12-17T21:49:37Z", "closed_at": "2022-12-17T21:49:36Z", "author_association": "OWNER", "pull_request": null, "body": "I got blocked working on this issue due to weird and hard-to-debug test suite problems:\r\n\r\n- #1955 \r\n\r\nThe test suite has needed a major upgrade for several years now. It has a LOT of `def test_...` synchronous functions that could be upgraded to `async def` for better performance and less test complexity - I've used the new `async def` pattern in plugins and new tests for a couple of years now.\r\n\r\nHopefully I can get more of the tests to use in-memory named databases too, ideally so I can fix this consistent problem:\r\n\r\n- #1843", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1959/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1434094365, "node_id": "I_kwDOBm6k_c5Veosd", "number": 1881, "title": "Tool for simulating permission checks against actors", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2022-11-03T04:43:20Z", "updated_at": "2022-12-09T01:38:21Z", "closed_at": "2022-11-04T00:13:05Z", "author_association": "OWNER", "pull_request": null, "body": "In working on this issue:\r\n- #1855\r\n\r\nI realized that if I'm going to make actors more complicated (the proposed `_r` key for additional restricted permissions) I really need an interactive tool for simulating these checks, similar to the https://latest.datasette.io/-/allow-debug tool.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1881/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1468603401, "node_id": "I_kwDOBm6k_c5XiRwJ", "number": 1913, "title": "Release Datasette 1.0a0", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 8658075, "label": "Datasette 1.0a0"}, "comments": 9, "created_at": "2022-11-29T19:41:42Z", "updated_at": "2022-11-29T20:10:35Z", "closed_at": "2022-11-29T20:10:35Z", "author_association": "OWNER", "pull_request": null, "body": "I attempted the release just now - https://github.com/simonw/datasette/releases/tag/1.0a0 - and got an unexpected test failure:\r\n\r\nhttps://github.com/simonw/datasette/actions/runs/3577355358/attempts/1\r\n\r\n```\r\n> assert delete_response.status_code == 200\r\nE assert 404 == 200\r\nE + where 404 = .status_code\r\n\r\n/home/runner/work/datasette/datasette/tests/test_api_write.py:396: AssertionError\r\n=========================== short test summary info ============================\r\nFAILED tests/test_api_write.py::test_delete_row[compound_pk_table-row_for_create2-pks2-article,k] - assert 404 == 200\r\n + where 404 = .status_code\r\n```\r\nI hit \"retry\" on that test but I expect it to fail again.\r\n\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1913/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1452457263, "node_id": "I_kwDOBm6k_c5Wkr0v", "number": 1897, "title": "Serve schema JSON to the SQL editor to enable autocomplete", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2022-11-16T23:47:45Z", "updated_at": "2022-11-18T05:33:20Z", "closed_at": "2022-11-18T02:54:43Z", "author_association": "OWNER", "pull_request": null, "body": "See:\r\n- https://github.com/simonw/datasette/issues/1893#issuecomment-1317831555\r\n ", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1897/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1015646369, "node_id": "I_kwDOBm6k_c48iYih", "number": 1480, "title": "Exceeding Cloud Run memory limits when deploying a 4.8G database", "user": {"value": 110420, "label": "ghing"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2021-10-04T21:20:24Z", "updated_at": "2022-10-07T04:39:10Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "When I try to deploy a 4.8G SQLite database to Google Cloud Run, I get this error message:\r\n\r\n> Memory limit of 8192M exceeded with 8826M used. Consider increasing the memory limit, see https://cloud.google.com/run/docs/configuring/memory-limits\r\n\r\nUnfortunately, the maximum amount of memory that can be allocated to an instance is 8192M.\r\n\r\nNaively profiling the memory usage of running Datasette with this database locally on my MacBook shows the following memory usage (using Activity Monitor) when I just start up Datasette locally:\r\n\r\n- Real Memory Size: 70.6 MB\r\n- Virtual Memory Size: 4.51 GB\r\n- Shared Memory Size: 2.5 MB\r\n- Private Memory Size: 57.4 MB\r\n\r\nI'm trying to understand if there's a query or other operation that gets run during container deployment that causes memory use to be so large and if this can be avoided somehow.\r\n\r\nThis is somewhat related to #1082, but on a different platform, so I decided to open a new issue.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1480/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1374939463, "node_id": "I_kwDOCGYnMM5R8-lH", "number": 489, "title": "Ability to load JSON records held in a file with a single top level key that is a list of objects", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2022-09-15T18:46:03Z", "updated_at": "2022-09-15T20:56:10Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "It's very common for JSON to look like this:\r\n```json\r\n{\r\n \"Version\": \"5.5.52.6\",\r\n \"List\": [\r\n {\r\n \"Description\": \"Nonpartisan\",\r\n \"Id\": 1,\r\n \"ExternalId\": \"\"\r\n },\r\n {\r\n \"Description\": \"Undeclared\",\r\n \"Id\": 2,\r\n \"ExternalId\": \"\"\r\n }\r\n ]\r\n}\r\n```\r\nThis example taken from the records downloaded from https://www.elections.alaska.gov/election-results/e/\r\n\r\nRight now you can't import this into `sqlite-utils` - you need to run it through `jq .List` first.\r\n\r\nBut since this is so common, it would be neat if `sqlite-utils` could have a rule of thumb that says \"if it's an object, but it has a single key that is is a list of objects, use that instead\".", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/489/reactions\", \"total_count\": 2, \"+1\": 2, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 507454958, "node_id": "MDU6SXNzdWU1MDc0NTQ5NTg=", "number": 596, "title": "Handle really wide tables better", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2019-10-15T20:05:46Z", "updated_at": "2022-09-07T00:58:41Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "If a table has hundreds of columns the Datasette UI starts getting unwieldy.\r\n\r\nAddressing this would be neat. One option would be to only select the first 30 columns by default and provide a UI for selecting more.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/596/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1344823170, "node_id": "PR_kwDOBm6k_c49e3_k", "number": 1789, "title": "Add new entrypoint option to `--load-extension`", "user": {"value": 15178711, "label": "asg017"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2022-08-19T19:27:47Z", "updated_at": "2022-08-23T18:42:52Z", "closed_at": "2022-08-23T18:34:30Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/datasette/pulls/1789", "body": "Closes #1784 \r\n\r\nThe `--load-extension` flag can now accept an optional \"entrypoint\" value, to specify which entrypoint SQLite should load from the given extension. \r\n\r\n```bash\r\n# would load default entrypoint like before\r\ndatasette data.db --load-extension ext\r\n\r\n# loads the extensions with the \"sqlite3_foo_init\" entrpoint\r\ndatasette data.db --load-extension ext:sqlite3_foo_init\r\n\r\n# loads the extensions with the \"sqlite3_bar_init\" entrpoint\r\ndatasette data.db --load-extension ext:sqlite3_bar_init\r\n```\r\n\r\nFor testing, I added a small SQLite extension in C at `tests/ext.c`. If compiled, then pytest will run the unit tests in `test_load_extensions.py`to verify that Datasette loads in extensions correctly (and loads the correct entrypoints). Compiling the extension requires a C compiler, I compiled it on my Mac with:\r\n\r\n```\r\ngcc ext.c -I path/to/sqlite -fPIC -shared -o ext.dylib\r\n```\r\n\r\nWhere `path/to/sqlite` is a directory that contains the SQLite amalgamation header files.\r\n\r\nRe documentation: I added a bit to the help text for `--load-extension` (which I believe should auto-add to documentation?), and the existing extension documentation is spatialite specific. Let me know if a new extensions documentation page would be helpful!", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1789/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1292060682, "node_id": "I_kwDOCGYnMM5NA0gK", "number": 450, "title": "Add --ignore option to more commands", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2022-07-02T13:52:02Z", "updated_at": "2022-07-15T22:39:09Z", "closed_at": "2022-07-15T22:37:45Z", "author_association": "OWNER", "pull_request": null, "body": "As seen in https://sqlite-utils.datasette.io/en/stable/cli-reference.html#add-foreign-key\r\n\r\nCould make this TIL trick unnecessary: https://til.simonwillison.net/bash/ignore-errors", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/450/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1294641696, "node_id": "I_kwDOBm6k_c5NKqog", "number": 1767, "title": "Ability to set a custom favicon", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2022-07-05T18:41:12Z", "updated_at": "2022-07-05T18:56:43Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "If you're running a website on Datasette, like https://www.niche-museums.com/ or https://til.simonwillison.net/ - you should have the ability to easily specify a custom favicon.\r\n\r\nCurrently the `/favicon.ico` view is hard-coded to do this: https://github.com/simonw/datasette/blob/9f1eb0d4eac483b953392157bd9fd6cc4df37de7/datasette/app.py#L179-L188", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1767/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 607223136, "node_id": "MDU6SXNzdWU2MDcyMjMxMzY=", "number": 741, "title": "Replace \"datasette publish --extra-options\" with \"--setting\"", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 9, "created_at": "2020-04-27T04:29:04Z", "updated_at": "2022-05-12T19:21:16Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "See https://github.com/simonw/datasette-publish-now/issues/9#issuecomment-618155764 - the `--extra-options` mechanism is in practice just used to set `--config` options in data that you publish, but that means you end up with pretty messy looking commands:\r\n\r\n datasette publish my.db --extra-options=\"--config default_page_size:50 --config sql_time_limit_ms:3500\"\r\n\r\nA neater design would be to support `--config` as an option for `datasette publish` directly:\r\n\r\n datasette publish my.db --config default_page_size:50 --config sql_time_limit_ms:3500\r\n\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/741/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1223234932, "node_id": "I_kwDOBm6k_c5I6RV0", "number": 1733, "title": "Get Datasette compatible with Pyodide", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2022-05-02T19:01:58Z", "updated_at": "2022-05-04T15:14:01Z", "closed_at": "2022-05-02T20:15:27Z", "author_association": "OWNER", "pull_request": null, "body": "I've already got this working as a prototype. Here are the changes I had to make:\r\n\r\n- Replace the two dependencies that don't publish pure Python wheels to PyPI: `click-default-group` and `python-baseconv`\r\n- Get Datasette to work without threading - which it turns out is exclusively used for database connections\r\n- Make the `uvicorn` dependency optional (only needed when Datasette runs in the CLI)\r\n\r\nTODO:\r\n\r\n- [x] Switch to `click-default-group-wheel`\r\n- [x] https://github.com/simonw/datasette/issues/1734\r\n- [x] Work around `uvicorn` import error\r\n- [x] https://github.com/simonw/datasette/issues/1735\r\n- [x] #1737\r\n\r\nGoal is to be able to do the following directly in https://pyodide.org/en/stable/console.html\r\n\r\n```python\r\nimport micropip\r\nawait micropip.install(\"datasette\")\r\nfrom datasette.app import Datasette\r\nds = Datasette()\r\nawait ds.client.get(\"/.json\")\r\n```", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1733/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 531755959, "node_id": "MDU6SXNzdWU1MzE3NTU5NTk=", "number": 647, "title": "Move hashed URL mode out to a plugin", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 9, "created_at": "2019-12-03T06:29:03Z", "updated_at": "2022-03-19T11:56:05Z", "closed_at": "2022-03-15T23:13:06Z", "author_association": "OWNER", "pull_request": null, "body": "They used to be the default until #418. Since making them optional I haven't felt the need to use them even once.\r\n\r\nThat suggests to me that they should be removed. I think their effect could be entirely handled by an ASGI wrapping plugin.\r\n\r\nhttps://datasette.readthedocs.io/en/0.32/performance.html#hashed-url-mode", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/647/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1124237013, "node_id": "I_kwDOCGYnMM5DAn7V", "number": 398, "title": "Add SpatiaLite helpers to CLI", "user": {"value": 25778, "label": "eyeseast"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2022-02-04T14:01:28Z", "updated_at": "2022-02-16T01:02:29Z", "closed_at": "2022-02-16T00:58:07Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "Now that #385 is merged, add CLI versions of those methods.\r\n\r\n```sh\r\n# init spatialite\r\nsqlite-utils init-spatialite database.db\r\n\r\n# or maybe/also\r\nsqlite-utils create database.db --enable-wal --spatialite\r\n\r\n# add geometry columns\r\n# needs a database, table, geometry column name, type, with optional SRID and not-null\r\n# this needs to create a table if it doesn't already exist\r\nsqlite-utils add-geometry-column database.db table-name geometry --srid 4326 --not-null\r\n\r\n# spatial index an existing table/column\r\nsqlite-utils create-spatial-index database.db table-name geometry\r\n```\r\n\r\nShould be mostly straightforward. The one thing worth highlighting in docs is that geometry columns can only be added to existing tables. Trying to add a geometry column to a table that doesn't exist yet might mean you have a schema like `{\"rowid\": int, \"geometry\": bytes}`. Might be worth nudging people to explicitly create a table first, then add geometry columns.\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/398/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1078702875, "node_id": "I_kwDOBm6k_c5AS7Mb", "number": 1552, "title": "Allow to set `facets_array` in metadata (like current `facets`)", "user": {"value": 3556, "label": "davidbgk"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7571612, "label": "Datasette 0.60"}, "comments": 9, "created_at": "2021-12-13T16:00:44Z", "updated_at": "2022-01-13T22:26:15Z", "closed_at": "2021-12-16T18:47:48Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "For now, you can set a `facets` value (array) in your metadata file but I couldn't find a way to set a `facets_array` in order to provide default facets for arrays (like tags). My use-case is to access to [that kind of view](https://latest.datasette.io/fixtures/facetable?_facet_array=tags) by default without URL's parameters as with other default facets.\r\n\r\n_I'm new to datasette, and I'm willing to help with a PR if that is not already implemented and I missed it!_", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1552/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 776634318, "node_id": "MDU6SXNzdWU3NzY2MzQzMTg=", "number": 1164, "title": "Mechanism for minifying JavaScript that ships with Datasette", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2020-12-30T20:59:06Z", "updated_at": "2022-01-13T22:21:29Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "> If I'm going to minify it I'll need to figure out a build step in Datasette itself so that I can easily work on that minified version.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/datasette/issues/983#issuecomment-752748496_", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1164/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 776635426, "node_id": "MDU6SXNzdWU3NzY2MzU0MjY=", "number": 1165, "title": "Mechanism for executing JavaScript unit tests", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2020-12-30T21:02:34Z", "updated_at": "2022-01-13T22:21:29Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "> I'm going to need to add JavaScript unit tests for this new plugin system.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/datasette/issues/983#issuecomment-752757289_", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1165/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 519613116, "node_id": "MDU6SXNzdWU1MTk2MTMxMTY=", "number": 617, "title": "Refactor TableView.data() method", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2019-11-08T01:55:41Z", "updated_at": "2021-12-18T01:41:47Z", "closed_at": "2021-12-11T19:17:11Z", "author_association": "OWNER", "pull_request": null, "body": "This is by far the most complex piece of Datasette - the `TableView.data()` method is over 500 lines long and is increasingly getting in the way of cleanly implementing new features (e.g. #615 and #613).\r\n\r\nNeed to break it up into smaller, cleaner pieces.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/617/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 962391325, "node_id": "MDU6SXNzdWU5NjIzOTEzMjU=", "number": 1423, "title": "Show count of facet values if ?_facet_size=max", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2021-08-06T04:42:20Z", "updated_at": "2021-12-15T17:48:40Z", "closed_at": "2021-08-16T18:56:43Z", "author_association": "OWNER", "pull_request": null, "body": "I sometimes want to get a count of the values in a facet - if it's a facet of US states for example I want to know if all 50 are represented.\r\n\r\nIdea: if `?_facet_size=max` is present, add a count to the facet heading. So on:\r\n\r\nhttps://latest.datasette.io/fixtures/compound_three_primary_keys?_facet=content&_facet_size=max&_facet=pk1&_facet=pk2#facet-pk2\r\n\r\nIt could have something like this:\r\n\r\n\"fixtures__compound_three_primary_keys__1_001_rows_and_NY_629_755_\u2022_Untitled-1_\u2014_datasette\"\r\n\r\nNote that the first column shows >1000 - because in that case we've truncated the facet calculation since the maximum allowed returned rows is 1000.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1423/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1066603133, "node_id": "PR_kwDOCGYnMM4vKAzW", "number": 347, "title": "Test against pysqlite3 running SQLite 3.37", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2021-11-29T23:17:57Z", "updated_at": "2021-12-11T01:02:19Z", "closed_at": null, "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/347", "body": "Refs #346 and #344.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/347/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 964322136, "node_id": "MDU6SXNzdWU5NjQzMjIxMzY=", "number": 1426, "title": "Manage /robots.txt in Datasette core, block robots by default", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2021-08-09T19:56:56Z", "updated_at": "2021-12-04T07:11:29Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "See accompanying Twitter thread: https://twitter.com/simonw/status/1424820203603431439\r\n\r\n> Datasette currently has a plugin for configuring robots.txt, but I'm beginning to think it should be part of core and crawlers should be blocked by default - having people explicitly opt-in to having their sites crawled and indexed feels a lot safer https://datasette.io/plugins/datasette-block-robots\r\n\r\nI have a lot of Datasettes deployed now, and tailing logs shows that they are being *hammered* by search engine crawlers even though many of them are not interesting enough to warrant indexing.\r\n\r\nI'm starting to think blocking crawlers would actually be a better default for most people, provided it was well documented and easy to understand how to allow them.\r\n\r\nDefault-deny is usually a better policy than default-allow!", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1426/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 944903881, "node_id": "MDU6SXNzdWU5NDQ5MDM4ODE=", "number": 1396, "title": "\"invalid reference format\" publishing Docker image", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2021-07-15T01:02:07Z", "updated_at": "2021-10-19T08:10:26Z", "closed_at": "2021-07-15T19:47:25Z", "author_association": "OWNER", "pull_request": null, "body": "Error ocurred at the end of the publish flow for Datasette 0.58: https://github.com/simonw/datasette/runs/3072216421\r\n```\r\nRemoving intermediate container cf32b9440907\r\n ---> dfd6985b2afc\r\nSuccessfully built dfd6985b2afc\r\nSuccessfully tagged ***/datasette:0.58\r\ninvalid reference format\r\nError: Process completed with exit code 1.\r\n```", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1396/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 275125561, "node_id": "MDU6SXNzdWUyNzUxMjU1NjE=", "number": 123, "title": "Datasette serve should accept paths/URLs to CSVs and other file formats", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2017-11-19T02:05:48Z", "updated_at": "2021-07-19T00:04:32Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "This would remove the csvs-to-sqlite step which I end up using for almost everything.\r\n\r\nI'm hesitant to introduce pandas as a required dependency though since it require compiling numpy. Could build it so this option is only available if you have pandas installed.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/123/reactions\", \"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 1, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 906385991, "node_id": "MDU6SXNzdWU5MDYzODU5OTE=", "number": 1349, "title": "CSV ?_stream=on redundantly calculates facets for every page", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2021-05-29T06:11:23Z", "updated_at": "2021-06-17T18:12:32Z", "closed_at": "2021-06-01T15:52:53Z", "author_association": "OWNER", "pull_request": null, "body": "I'm trying to figure out why a full CSV export from https://covid-19.datasettes.com/covid/ny_times_us_counties runs unbearably slowly.\r\n\r\nIt's because the streaming endpoint works by scrolling through every page, and it turns out every page calculates facets and suggested facets!", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1349/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 763361458, "node_id": "MDU6SXNzdWU3NjMzNjE0NTg=", "number": 1142, "title": "\"Stream all rows\" is not at all obvious", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2020-12-12T06:24:57Z", "updated_at": "2021-06-17T18:12:31Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Got a question about how to download all rows - the current option isn't at all clear.\r\n\r\n\"loans__ppp_loans__9_511_rows_where_where_search_matches__tech__sorted_by_rowid\"\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1142/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 813899472, "node_id": "MDU6SXNzdWU4MTM4OTk0NzI=", "number": 1238, "title": "Custom pages don't work with base_url setting", "user": {"value": 79913, "label": "tsibley"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2021-02-22T21:58:58Z", "updated_at": "2021-06-05T18:59:55Z", "closed_at": "2021-06-05T18:59:55Z", "author_association": "NONE", "pull_request": null, "body": "It seems that custom pages aren't routing properly when the `base_url` setting is used.\r\n\r\nTo reproduce, with Datasette 0.55.\r\n\r\nCreate a `templates/pages/custom.html` with some text.\r\n```\r\nmkdir -p templates/pages/\r\necho \"Hello, world!\" > templates/pages/custom.html\r\n```\r\n\r\nStart Datasette.\r\n\r\n```\r\ndatasette --template-dir templates/\r\n```\r\n\r\nVisit http://localhost:8001/custom and see \"Hello, world!\".\r\n\r\nStart Datasette with a `base_url`.\r\n\r\n```\r\ndatasette --template-dir templates/ --setting base_url /prefix/\r\n```\r\n\r\nVisit http://localhost:8001/prefix/custom and see a \"Database not found: custom\" 404.\r\n\r\nNote that like all routes, http://localhost:8001/custom still works when run with `base_url`.\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1238/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 812228314, "node_id": "MDU6SXNzdWU4MTIyMjgzMTQ=", "number": 1236, "title": "Ability to increase size of the SQL editor window", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2021-02-19T18:09:27Z", "updated_at": "2021-05-18T03:28:25Z", "closed_at": "2021-02-22T21:05:21Z", "author_association": "OWNER", "pull_request": null, "body": "", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1236/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 496415321, "node_id": "MDU6SXNzdWU0OTY0MTUzMjE=", "number": 1, "title": "Figure out some interesting example SQL queries", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2019-09-20T15:28:07Z", "updated_at": "2021-05-03T03:46:23Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "My knowledge of genetics has left me short here. I'd love to be able to provide some interesting example SELECT queries - maybe one that spots if you are [likely to have red hair?](https://www.snpedia.com/index.php/Rs1805007)", "repo": {"value": 209590345, "label": "genome-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 780267857, "node_id": "MDU6SXNzdWU3ODAyNjc4NTc=", "number": 1178, "title": "Use force_https_urls on when deploying with Cloud Run", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 6346396, "label": "Datasette 0.54"}, "comments": 9, "created_at": "2021-01-06T08:20:55Z", "updated_at": "2021-01-24T21:21:05Z", "closed_at": "2021-01-06T18:24:47Z", "author_association": "OWNER", "pull_request": null, "body": "_Original title: datasette.absolute_url() should return https:// not http:// on Cloud Run_\r\n\r\nhttps://latest-with-plugins.datasette.io/github/issue_comments.Notebook?_labels=on currently provides `http://` links, which break in Observable since it won't load `http://` content.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1178/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 777535402, "node_id": "MDU6SXNzdWU3Nzc1MzU0MDI=", "number": 215, "title": "Use _counts to speed up counts", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2021-01-02T22:30:17Z", "updated_at": "2021-01-03T20:19:40Z", "closed_at": "2021-01-03T20:19:40Z", "author_association": "OWNER", "pull_request": null, "body": "Utility mechanism for taking advantage of the new `_counts` table from #212 would be nice.\r\n\r\nThese can trigger automatically if the `_counts` table exists, but since `sqlite-utils` needs to work against any existing database there should be a way of opting out of this optimization.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/215/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 771216293, "node_id": "MDU6SXNzdWU3NzEyMTYyOTM=", "number": 1155, "title": "Better internal database_name for _internal database", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2020-12-18T22:47:27Z", "updated_at": "2020-12-22T20:04:35Z", "closed_at": "2020-12-22T20:04:35Z", "author_association": "OWNER", "pull_request": null, "body": "https://latest.datasette.io/login-as-root then https://latest.datasette.io/_internal\r\n\r\n\"_schemas__columns__132_rows\"\r\n\r\nThat `database_name` is ugly. It should just be `_internal`.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1155/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 756439516, "node_id": "MDU6SXNzdWU3NTY0Mzk1MTY=", "number": 1124, "title": "Datasette on Amazon Linux on ARM returns 404 for static assets", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2020-12-03T18:20:37Z", "updated_at": "2020-12-03T21:42:02Z", "closed_at": "2020-12-03T21:10:54Z", "author_association": "OWNER", "pull_request": null, "body": "Very weird bug this one. Steps to reproduce:\r\n```\r\n# I started a amzn2-ami-hvm-2.0.20201126.0-arm64-gp2 t4g.micro instance\r\nec2 % ssh -i simonw-ec2.pem ec2-user@ec2-18-219-238-192.us-east-2.compute.amazonaws.com\r\n[ec2-user@ip-172-31-30-7 ~]$ sudo yum install python3\r\nLoaded plugins: extras_suggestions, langpacks, priorities, update-motd\r\n...\r\nIs this ok [y/d/N]: y\r\nDownloading packages:\r\n(1/4): python3-3.7.9-1.amzn2.0.1.aarch64.rpm | 72 kB 00:00:00 \r\n(2/4): python3-pip-9.0.3-1.amzn2.0.2.noarch.rpm | 1.9 MB 00:00:00 \r\n(3/4): python3-setuptools-38.4.0-3.amzn2.0.6.noarch.rpm | 617 kB 00:00:00 \r\n(4/4): python3-libs-3.7.9-1.amzn2.0.1.aarch64.rpm | 9.1 MB 00:00:00 \r\n-----------------------------------------------------------------------------------------------------------------------------------------------------------\r\nTotal 68 MB/s | 12 MB 00:00:00 \r\nRunning transaction check\r\nRunning transaction test\r\nTransaction test succeeded\r\nRunning transaction\r\n Installing : python3-pip-9.0.3-1.amzn2.0.2.noarch 1/4 \r\n Installing : python3-3.7.9-1.amzn2.0.1.aarch64 2/4 \r\n Installing : python3-setuptools-38.4.0-3.amzn2.0.6.noarch 3/4 \r\n Installing : python3-libs-3.7.9-1.amzn2.0.1.aarch64 4/4 \r\n Verifying : python3-setuptools-38.4.0-3.amzn2.0.6.noarch 1/4 \r\n Verifying : python3-pip-9.0.3-1.amzn2.0.2.noarch 2/4 \r\n Verifying : python3-3.7.9-1.amzn2.0.1.aarch64 3/4 \r\n Verifying : python3-libs-3.7.9-1.amzn2.0.1.aarch64 4/4 \r\n\r\nInstalled:\r\n python3.aarch64 0:3.7.9-1.amzn2.0.1 \r\n\r\nDependency Installed:\r\n python3-libs.aarch64 0:3.7.9-1.amzn2.0.1 python3-pip.noarch 0:9.0.3-1.amzn2.0.2 python3-setuptools.noarch 0:38.4.0-3.amzn2.0.6 \r\n\r\nComplete!\r\n[ec2-user@ip-172-31-30-7 ~]$ python3 -m pip install --user pipx\r\nCollecting pipx\r\n Downloading https://files.pythonhosted.org/packages/42/8a/8447ec14562c5d97afbee54ec390864718bccce9dfb0506c8c77852489d3/pipx-0.15.6.0-py3-none-any.whl (43kB)\r\n 100% |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 51kB 3.8MB/s \r\nCollecting packaging>=20.0 (from pipx)\r\n Downloading https://files.pythonhosted.org/packages/28/87/8edcf555adaf60d053ead881bc056079e29319b643ca710339ce84413136/packaging-20.7-py2.py3-none-any.whl\r\nCollecting argcomplete<2.0,>=1.9.4 (from pipx)\r\n Downloading https://files.pythonhosted.org/packages/e3/d0/ee7fc6ceac8957196def9bfa3b955d02163058defd3edd51ef7b1ff2769e/argcomplete-1.12.2-py2.py3-none-any.whl\r\nCollecting userpath>=1.4.1 (from pipx)\r\n Downloading https://files.pythonhosted.org/packages/fb/f1/faca8a3cc86bd2223aaf72edd222bc31d6ae2eea5feaf17a144634a92e6d/userpath-1.4.1-py2.py3-none-any.whl\r\nCollecting pyparsing>=2.0.2 (from packaging>=20.0->pipx)\r\n Downloading https://files.pythonhosted.org/packages/8a/bb/488841f56197b13700afd5658fc279a2025a39e22449b7cf29864669b15d/pyparsing-2.4.7-py2.py3-none-any.whl (67kB)\r\n 100% |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 71kB 8.8MB/s \r\nCollecting importlib-metadata<4,>=0.23; python_version == \"3.7\" (from argcomplete<2.0,>=1.9.4->pipx)\r\n Downloading https://files.pythonhosted.org/packages/99/c7/4ccf2baa455613aa9e61372365aba8594ff2806c82189c31a6c65e7c679e/importlib_metadata-3.1.1-py3-none-any.whl\r\nCollecting click (from userpath>=1.4.1->pipx)\r\n Downloading https://files.pythonhosted.org/packages/d2/3d/fa76db83bf75c4f8d338c2fd15c8d33fdd7ad23a9b5e57eb6c5de26b430e/click-7.1.2-py2.py3-none-any.whl (82kB)\r\n 100% |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 92kB 10.5MB/s \r\nCollecting distro; platform_system == \"Linux\" (from userpath>=1.4.1->pipx)\r\n Downloading https://files.pythonhosted.org/packages/25/b7/b3c4270a11414cb22c6352ebc7a83aaa3712043be29daa05018fd5a5c956/distro-1.5.0-py2.py3-none-any.whl\r\nCollecting zipp>=0.5 (from importlib-metadata<4,>=0.23; python_version == \"3.7\"->argcomplete<2.0,>=1.9.4->pipx)\r\n Downloading https://files.pythonhosted.org/packages/41/ad/6a4f1a124b325618a7fb758b885b68ff7b058eec47d9220a12ab38d90b1f/zipp-3.4.0-py3-none-any.whl\r\nInstalling collected packages: pyparsing, packaging, zipp, importlib-metadata, argcomplete, click, distro, userpath, pipx\r\nSuccessfully installed argcomplete-1.12.2 click-7.1.2 distro-1.5.0 importlib-metadata-3.1.1 packaging-20.7 pipx-0.15.6.0 pyparsing-2.4.7 userpath-1.4.1 zipp-3.4.0\r\n[ec2-user@ip-172-31-30-7 ~]$ python3 -m pipx ensurepath\r\n/home/ec2-user/.local/bin is already in PATH.\r\n/home/ec2-user/.local/bin is already in PATH.\r\n\r\nAll pipx binary directories have been added to PATH. If you are sure\r\nyou want to proceed, try again with the '--force' flag.\r\n\r\nOtherwise pipx is ready to go! \u2728 \ud83c\udf1f \u2728\r\n[ec2-user@ip-172-31-30-7 ~]$ pipx install datasette\r\n installed package datasette 0.52.2, Python 3.7.9\r\n These apps are now globally available\r\n - datasette\r\ndone! \u2728 \ud83c\udf1f \u2728\r\n[ec2-user@ip-172-31-30-7 ~]$ datasette --version\r\ndatasette, version 0.52.2\r\n[ec2-user@ip-172-31-30-7 ~]$ datasette --get /-/static/app.css\r\n404\r\n```", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1124/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 753668177, "node_id": "MDU6SXNzdWU3NTM2NjgxNzc=", "number": 1116, "title": "GENERATED column support", "user": {"value": 2789593, "label": "nattaylor"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2020-11-30T17:33:47Z", "updated_at": "2020-11-30T21:29:59Z", "closed_at": "2020-11-30T21:29:59Z", "author_association": "NONE", "pull_request": null, "body": "I think this is a feature request... perhaps I should just try to contribute it myself, but thought I'd check in case support is planned already.\r\n\r\nFor a table with the following schema, datasette 0.51.1 doesn't pick up the GENERATED columns and the column list only contains `(rowid, body)` If I edit the SQL and select the generated columns, it will happily show them.\r\n\r\nAt first glance it appears that [`def table_column_details(conn, table):`](https://github.com/simonw/datasette/blob/4777362bf2692bc72b221ec47c3e6216151d1b89/datasette/utils/__init__.py#L575) would have to be refactored to use a different methodology to get the columns, since `PRAGMA table_info(deeds);` returns just `0|body|TEXT|0||0` so maybe it wouldn't be worth it.\r\n\r\n```\r\nCREATE TABLE deeds (\r\n body TEXT,\r\n id INT GENERATED ALWAYS AS (json_extract(body, '$.id')) STORED,\r\n consideration INT GENERATED ALWAYS AS (json_extract(body, '$.consideration')) STORED\r\n);\r\n```", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1116/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 735532751, "node_id": "MDU6SXNzdWU3MzU1MzI3NTE=", "number": 192, "title": "sqlite-utils search command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 6079500, "label": "3.0"}, "comments": 9, "created_at": "2020-11-03T18:07:59Z", "updated_at": "2020-11-08T17:07:01Z", "closed_at": "2020-11-08T17:07:01Z", "author_association": "OWNER", "pull_request": null, "body": "A command that knows how to run a search against a FTS enabled table and return results ranked by relevance.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/192/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 722674708, "node_id": "MDU6SXNzdWU3MjI2NzQ3MDg=", "number": 1024, "title": "Figure out how to run an environment that exercises the base_url proxy setting", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 6026070, "label": "0.51"}, "comments": 9, "created_at": "2020-10-15T21:03:39Z", "updated_at": "2020-10-23T19:44:06Z", "closed_at": "2020-10-15T22:34:04Z", "author_association": "OWNER", "pull_request": null, "body": "Refs #1023.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1024/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 707427200, "node_id": "MDU6SXNzdWU3MDc0MjcyMDA=", "number": 172, "title": "Improve performance of extract operations", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2020-09-23T14:40:50Z", "updated_at": "2020-09-24T15:43:57Z", "closed_at": "2020-09-24T15:43:57Z", "author_association": "OWNER", "pull_request": null, "body": "This command took about 12 minutes (against a 150MB file with 680,000 rows in it):\r\n```\r\nsqlite-utils extract salaries.db salaries \\\r\n 'Organization Group Code' 'Organization Group' \\\r\n --table 'organization_groups' \\\r\n --fk-column 'organization_group_id' \\\r\n --rename 'Organization Group Code' code \\\r\n --rename 'Organization Group' name\r\n```\r\nI'm pretty confident we can do better than that.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/172/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 648421105, "node_id": "MDU6SXNzdWU2NDg0MjExMDU=", "number": 877, "title": "Consider dropping explicit CSRF protection entirely?", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2020-06-30T19:00:55Z", "updated_at": "2020-09-15T20:42:05Z", "closed_at": "2020-09-15T20:42:04Z", "author_association": "OWNER", "pull_request": null, "body": "https://scotthelme.co.uk/csrf-is-dead/ from Feb 2017 has background here. The `SameSite=lax` cookie property effectively eliminates CSRF in modern browsers. https://caniuse.com/#search=SameSite shows 92.13% global support for it.\r\n\r\nDatasette already uses `SameSite=lax` when it sets cookies by default: https://github.com/simonw/datasette/blob/af350ba4571b8e3f9708c40f2ddb48fea7ac1084/datasette/utils/asgi.py#L327-L341\r\n\r\nA few options then. I could ditch CSRF protection entirely. I could make it optional - turn it off by default, but let users who care about that remaining 7.87% of global users opt back into it.\r\n\r\nOne catch: login CSRF: I don't see how `SameSite=lax` protects against that attack.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/877/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 651844316, "node_id": "MDExOlB1bGxSZXF1ZXN0NDQ1MDIzMzI2", "number": 118, "title": "Add insert --truncate option", "user": {"value": 79913, "label": "tsibley"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2020-07-06T21:58:40Z", "updated_at": "2020-07-08T17:26:21Z", "closed_at": "2020-07-08T17:26:21Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/118", "body": "\r\n\r\nDeletes all rows in the table (if it exists) before inserting new rows.\r\nSQLite doesn't implement a TRUNCATE TABLE statement but does optimize an\r\nunqualified DELETE FROM.\r\n\r\nThis can be handy if you want to refresh the entire contents of a table\r\nbut a) don't have a PK (so can't use --replace), b) don't want the table\r\nto disappear (even briefly) for other connections, and c) have to handle\r\nrecords that used to exist being deleted.\r\n\r\nIdeally the replacement of rows would appear instantaneous to other\r\nconnections by putting the DELETE + INSERT in a transaction, but this is\r\nvery difficult without breaking other code as the current transaction\r\nhandling is inconsistent and non-systematic. There exists the\r\npossibility for the DELETE to succeed but the INSERT to fail, leaving an\r\nempty table. This is not much worse, however, than the current\r\npossibility of one chunked INSERT succeeding and being committed while\r\nthe next chunked INSERT fails, leaving a partially complete operation.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/118/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 640917326, "node_id": "MDU6SXNzdWU2NDA5MTczMjY=", "number": 852, "title": "canned_queries() plugin hook", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5533512, "label": "Datasette 0.45"}, "comments": 9, "created_at": "2020-06-18T05:24:35Z", "updated_at": "2020-06-20T03:08:40Z", "closed_at": "2020-06-20T03:08:40Z", "author_association": "OWNER", "pull_request": null, "body": "Canned queries are currently baked into `metadata.json` which is read once on startup.\r\n\r\nAllowing users to interactively create new canned queries - even if just through a plugin - would make a lot of sense.\r\n\r\nIs this a new plugin hook or some other mechanism? Lots to think about here.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/852/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 638104520, "node_id": "MDU6SXNzdWU2MzgxMDQ1MjA=", "number": 841, "title": "Research feasibility of 100% test coverage", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2020-06-13T06:07:01Z", "updated_at": "2020-06-13T21:38:46Z", "closed_at": "2020-06-13T21:38:46Z", "author_association": "OWNER", "pull_request": null, "body": "Inspired by https://twitter.com/mikeal/status/1271473021593636866\r\n> Almost every library I\u2019ve written in the last 2 years has had 100% coverage and that\u2019s probably not going to change in the future. It\u2019s not that hard to start at 100% and hold onto it and the workflow it enables is so much nicer.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/841/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 634917088, "node_id": "MDU6SXNzdWU2MzQ5MTcwODg=", "number": 818, "title": "Example permissions plugin", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5512395, "label": "Datasette 0.44"}, "comments": 9, "created_at": "2020-06-08T20:35:56Z", "updated_at": "2020-06-11T05:40:07Z", "closed_at": "2020-06-11T05:40:07Z", "author_association": "OWNER", "pull_request": null, "body": "To show how they work. Also useful to confirm how they interact with the default permissions.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/818/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 629595228, "node_id": "MDExOlB1bGxSZXF1ZXN0NDI2ODkxNDcx", "number": 796, "title": "New WIP writable canned queries", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 9, "created_at": "2020-06-03T00:08:00Z", "updated_at": "2020-06-03T15:16:52Z", "closed_at": "2020-06-03T15:16:50Z", "author_association": "OWNER", "pull_request": "simonw/datasette/pulls/796", "body": "Refs #698. Replaces #703\r\n\r\nStill todo:\r\n\r\n- [x] Unit tests\r\n- ~~Figure out `.json` mode~~\r\n- [x] Flash message solution\r\n- ~~CSRF protection~~\r\n- [x] Better error message display on errors\r\n- [x] Documentation\r\n- ~~Maybe widgets?~~ I'll do these later", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/796/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 605110015, "node_id": "MDU6SXNzdWU2MDUxMTAwMTU=", "number": 731, "title": "Option to automatically configure based on directory layout", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2020-04-22T22:17:47Z", "updated_at": "2020-04-27T16:32:44Z", "closed_at": "2020-04-27T16:30:26Z", "author_association": "OWNER", "pull_request": null, "body": "My Datasette projects increasingly take on the following structure:\r\n\r\n- `metadata.json` with the metadata\r\n- One or more `something.db` database files\r\n- A `templates/` folder with some custom templates\r\n- A `plugins/` folder with some custom plugins\r\n\r\nThen I have to run Datasette like this:\r\n\r\n datasette *.db -m metadata.json --template-dir=templates --plugins-dir=plugins\r\n\r\nIt would be really interesting if Datasette had a special mode where you could point it at a directory with the above layout and it would automatically configure itself based on the contents.\r\n\r\nMaybe even allow `datasette serve` to detect if it was passed a single argument that's a directory, not a file, and kick in to \"directory layout configuration mode\" in that case:\r\n\r\n datasette .\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/731/reactions\", \"total_count\": 2, \"+1\": 2, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 521275281, "node_id": "MDU6SXNzdWU1MjEyNzUyODE=", "number": 13, "title": "Set up a live demo Datasette instance", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5225818, "label": "1.0"}, "comments": 9, "created_at": "2019-11-12T01:27:02Z", "updated_at": "2020-03-24T00:03:26Z", "closed_at": "2020-03-24T00:03:25Z", "author_association": "MEMBER", "pull_request": null, "body": "I deployed https://github-to-sqlite-releases-j7hipcg4aq-uc.a.run.app/ by running this:\r\n```\r\n#!/bin/bash\r\n# Fetch repos for simonw and dogsheep\r\ngithub-to-sqlite repos github.db simonw dogsheep -a auth.json\r\n\r\n# Fetch releases for the repos tagged 'datasette-io'\r\nsqlite-utils github.db \"\r\nselect full_name from repos where rowid in (\r\n select repos.rowid from repos, json_each(repos.topics) j\r\n where j.value = 'datasette-io'\r\n)\" --csv --no-headers | while read repo;\r\n do github-to-sqlite releases \\\r\n github.db $(echo $repo | tr -d '\\r') \\\r\n -a auth.json;\r\n sleep 2;\r\n done;\r\n```\r\nAnd then deploying using this:\r\n```\r\n$ datasette publish cloudrun github.db \\\r\n --title \"github-to-sqlite releases demo\" \\\r\n --about_url=\"https://github.com/simonw/github-to-sqlite\" \\\r\n --about='github-to-sqlite' \\\r\n --install=datasette-render-markdown \\\r\n --install=datasette-json-html \\\r\n --service=github-to-sqlite-releases\r\n```\r\nThis should happen automatically for every release. I can run it once a day in Circle CI to keep the demo database up-to-date.", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 568091133, "node_id": "MDU6SXNzdWU1NjgwOTExMzM=", "number": 676, "title": "?_searchmode=raw option for running FTS searches without escaping characters", "user": {"value": 58088336, "label": "tunguyenatwork"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2020-02-20T06:56:57Z", "updated_at": "2020-02-25T05:57:24Z", "closed_at": "2020-02-25T05:56:04Z", "author_association": "NONE", "pull_request": null, "body": "After the version 0.34. I am not able to use the wildchar in the _search option( or the full text search). It will not return any result unless I specify the whole word for text search. \r\n\r\nIf I use 'match :search || \"*\" ' in the sql statement then it will work as expected.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/676/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 506300941, "node_id": "MDExOlB1bGxSZXF1ZXN0MzI3NTQxMDQ2", "number": 595, "title": "bump uvicorn to 0.9.0 to be Python-3.8 friendly", "user": {"value": 4312421, "label": "stonebig"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2019-10-13T10:00:04Z", "updated_at": "2019-11-12T04:46:48Z", "closed_at": "2019-11-12T04:46:48Z", "author_association": "NONE", "pull_request": "simonw/datasette/pulls/595", "body": "as uvicorn-0.9 is needed to get websockets-8.0.2, which is needed to have Python-3.8 compatibility", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/595/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 267759136, "node_id": "MDU6SXNzdWUyNjc3NTkxMzY=", "number": 20, "title": "Config file with support for defining canned queries", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": {"value": 9599, "label": "simonw"}, "milestone": {"value": 2949431, "label": "Custom templates edition"}, "comments": 9, "created_at": "2017-10-23T17:53:06Z", "updated_at": "2017-12-05T19:05:35Z", "closed_at": "2017-12-05T17:44:09Z", "author_association": "OWNER", "pull_request": null, "body": "Probably using YAML because then we get support for multiline strings:\r\n\r\n bats:\r\n db: bats.sqlite3\r\n name: \"Bat sightings\"\r\n queries:\r\n specific_row: |\r\n select * from Bats\r\n where a = 1;\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/20/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}