{"html_url": "https://github.com/simonw/datasette/issues/2189#issuecomment-1724083324", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2189", "id": 1724083324, "node_id": "IC_kwDOBm6k_c5mw2x8", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-18T17:46:21Z", "updated_at": "2023-09-18T17:46:21Z", "author_association": "OWNER", "body": "Sometimes it takes a few clicks for the bug to occur, but it does seem to always be within the in-memory database.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1901416155, "label": "Server hang on parallel execution of queries to named in-memory databases"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2189#issuecomment-1724084199", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2189", "id": 1724084199, "node_id": "IC_kwDOBm6k_c5mw2_n", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-18T17:47:01Z", "updated_at": "2023-09-18T17:47:01Z", "author_association": "OWNER", "body": "I managed to trigger it by loading `http://127.0.0.1:8045/airtable_refs/airtable_refs` - which worked - and then hitting refresh on that page a bunch of times until it hung.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1901416155, "label": "Server hang on parallel execution of queries to named in-memory databases"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2189#issuecomment-1724089666", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2189", "id": 1724089666, "node_id": "IC_kwDOBm6k_c5mw4VC", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-18T17:49:24Z", "updated_at": "2023-09-18T17:49:24Z", "author_association": "OWNER", "body": "I switched that particular implementation to using an on-disk database instead of an in-memory database and could no longer recreate the bug.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1901416155, "label": "Server hang on parallel execution of queries to named in-memory databases"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2189#issuecomment-1724157182", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2189", "id": 1724157182, "node_id": "IC_kwDOBm6k_c5mxIz-", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-18T18:30:30Z", "updated_at": "2023-09-18T18:30:30Z", "author_association": "OWNER", "body": "OK, I can trigger the bug like this:\r\n\r\n```bash\r\ndatasette pottery2.db -p 8045 --get /airtable_refs/airtable_refs\r\n```\r\nCan I write a bash script that fails (and terminates the process) if it takes longer than X seconds?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1901416155, "label": "Server hang on parallel execution of queries to named in-memory databases"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2189#issuecomment-1724159882", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2189", "id": 1724159882, "node_id": "IC_kwDOBm6k_c5mxJeK", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-18T18:32:29Z", "updated_at": "2023-09-18T18:32:29Z", "author_association": "OWNER", "body": "This worked, including on macOS even though GPT-4 thought `timeout` would not work there: https://chat.openai.com/share/cc4628e9-5240-4f35-b640-16a9c178b315\r\n```bash\r\n#!/bin/bash\r\n\r\n# Run the command with a timeout of 5 seconds\r\ntimeout 5s datasette pottery2.db -p 8045 --get /airtable_refs/airtable_refs\r\n\r\n# Check the exit code from timeout\r\nif [ $? -eq 124 ]; then\r\n echo \"Error: Command timed out after 5 seconds.\"\r\n exit 1\r\nfi\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1901416155, "label": "Server hang on parallel execution of queries to named in-memory databases"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2189#issuecomment-1724257290", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2189", "id": 1724257290, "node_id": "IC_kwDOBm6k_c5mxhQK", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-18T19:39:27Z", "updated_at": "2023-09-18T19:44:26Z", "author_association": "OWNER", "body": "I'm now trying this test script:\r\n```bash\r\n#!/bin/bash\r\n\r\nport=8064\r\n# Start datasette server in the background and get its PID\r\ndatasette pottery2.db -p $port &\r\nserver_pid=$!\r\n\r\n# Wait for a moment to ensure the server has time to start up\r\nsleep 2\r\n\r\n# Initialize counters and parameters\r\nretry_count=0\r\nmax_retries=3\r\nsuccess_count=0\r\npath=\"/airtable_refs/airtable_refs\"\r\n\r\n# Function to run curl with a timeout\r\nfunction test_curl {\r\n # Run the curl command with a timeout of 3 seconds\r\n timeout 3s curl -s \"http://localhost:${port}${path}\" > /dev/null\r\n if [ $? -eq 0 ]; then\r\n # Curl was successful\r\n ((success_count++))\r\n fi\r\n}\r\n\r\n# Try three parallel curl requests\r\nwhile [[ $retry_count -lt $max_retries ]]; do\r\n # Reset the success counter\r\n success_count=0\r\n\r\n # Run the curls in parallel\r\n echo \" Running curls\"\r\n test_curl\r\n test_curl\r\n test_curl # & test_curl & test_curl &\r\n\r\n # Wait for all curls to finish\r\n #wait\r\n\r\n # Check the success count\r\n if [[ $success_count -eq 3 ]]; then\r\n # All curls succeeded, break out of the loop\r\n echo \" All curl succeeded\"\r\n break\r\n fi\r\n\r\n ((retry_count++))\r\ndone\r\n\r\n# Kill the datasette server\r\necho \"Killing datasette server with PID $server_pid\"\r\nkill -9 $server_pid\r\nsleep 2\r\n\r\n# Print result\r\nif [[ $success_count -eq 3 ]]; then\r\n echo \"All three curls succeeded.\"\r\n exit 0\r\nelse\r\n echo \"Error: Not all curls succeeded after $retry_count attempts.\"\r\n exit 1\r\nfi\r\n```\r\nI run it like this:\r\n```bash\r\ngit bisect reset\r\ngit bisect start\r\ngit bisect good 0.59.4\r\ngit bisect bad 1.0a6\r\ngit bisect run ../airtable-export/testit.sh\r\n```\r\nBut... it's not having the desired result, I think because the bug is intermittent so each time I run it the bisect spits out a different commit as the one that is to blame.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1901416155, "label": "Server hang on parallel execution of queries to named in-memory databases"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2189#issuecomment-1724258279", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2189", "id": 1724258279, "node_id": "IC_kwDOBm6k_c5mxhfn", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-18T19:40:13Z", "updated_at": "2023-09-18T19:40:13Z", "author_association": "OWNER", "body": "Output while it is running looks like this:\r\n```\r\nrunning '../airtable-export/testit.sh'\r\nINFO: Started server process [75649]\r\nINFO: Waiting for application startup.\r\nINFO: Application startup complete.\r\nINFO: Uvicorn running on http://127.0.0.1:8064 (Press CTRL+C to quit)\r\n Running curls\r\n Running curls\r\n Running curls\r\nKilling datasette server with PID 75649\r\n../airtable-export/testit.sh: line 54: 75649 Killed: 9 datasette pottery2.db -p $port\r\nError: Not all curls succeeded after 3 attempts.\r\nBisecting: 155 revisions left to test after this (roughly 7 steps)\r\n[247e460e08bf823142f7b84058fe44e43626787f] Update beautifulsoup4 requirement (#1703)\r\nrunning '../airtable-export/testit.sh'\r\nINFO: Started server process [75722]\r\nINFO: Waiting for application startup.\r\nINFO: Application startup complete.\r\nINFO: Uvicorn running on http://127.0.0.1:8064 (Press CTRL+C to quit)\r\n Running curls\r\n Running curls\r\n Running curls\r\nKilling datasette server with PID 75722\r\n../airtable-export/testit.sh: line 54: 75722 Killed: 9 datasette pottery2.db -p $port\r\nError: Not all curls succeeded after 3 attempts.\r\nBisecting: 77 revisions left to test after this (roughly 6 steps)\r\n[3ef47a0896c7e63404a34e465b7160c80eaa571d] Link rel=alternate header for tables and rows\r\nrunning '../airtable-export/testit.sh'\r\nINFO: Started server process [75818]\r\nINFO: Waiting for application startup.\r\nINFO: Application startup complete.\r\nINFO: Uvicorn running on http://127.0.0.1:8064 (Press CTRL+C to quit)\r\n Running curls\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1901416155, "label": "Server hang on parallel execution of queries to named in-memory databases"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2189#issuecomment-1724259229", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2189", "id": 1724259229, "node_id": "IC_kwDOBm6k_c5mxhud", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-18T19:40:56Z", "updated_at": "2023-09-18T19:40:56Z", "author_association": "OWNER", "body": "I tried it with a path of `/` and everything passed - so it's definitely the path of `/airtable_refs/airtable_refs` (an in-memory database created by an experimental branch of https://github.com/simonw/airtable-export) that triggers the problem.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1901416155, "label": "Server hang on parallel execution of queries to named in-memory databases"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2189#issuecomment-1724263390", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2189", "id": 1724263390, "node_id": "IC_kwDOBm6k_c5mxive", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-18T19:44:03Z", "updated_at": "2023-09-18T19:44:03Z", "author_association": "OWNER", "body": "I knocked it down to 1 retry just to see what happened.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1901416155, "label": "Server hang on parallel execution of queries to named in-memory databases"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2189#issuecomment-1724276917", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2189", "id": 1724276917, "node_id": "IC_kwDOBm6k_c5mxmC1", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-18T19:54:23Z", "updated_at": "2023-09-18T19:54:23Z", "author_association": "OWNER", "body": "Turned out I wasn't running the `datasette` from the current directory, so it was not testing what I intended.\r\n\r\nFIxed that with `pip install -e .` in the `datasette/` directory.\r\n\r\nNow I'm seeing some passes, which look like this:\r\n```\r\nrunning '../airtable-export/testit.sh'\r\nINFO: Started server process [77810]\r\nINFO: Waiting for application startup.\r\nINFO: Application startup complete.\r\nINFO: Uvicorn running on http://127.0.0.1:8064 (Press CTRL+C to quit)\r\n Running curls\r\nINFO: 127.0.0.1:59439 - \"GET /airtable_refs/airtable_refs HTTP/1.1\" 200 OK\r\nINFO: 127.0.0.1:59440 - \"GET /airtable_refs/airtable_refs HTTP/1.1\" 200 OK\r\nINFO: 127.0.0.1:59441 - \"GET /airtable_refs/airtable_refs HTTP/1.1\" 200 OK\r\n All curl succeeded\r\nKilling datasette server with PID 77810\r\n../airtable-export/testit.sh: line 54: 77810 Killed: 9 datasette pottery2.db -p $port\r\nAll three curls succeeded.\r\nBisecting: 4 revisions left to test after this (roughly 2 steps)\r\n[7463b051cf8d7f856df5eba9f7aa944183ebabe5] Cosmetic tweaks after blacken-docs, refs #1718\r\nrunning '../airtable-export/testit.sh'\r\nINFO: Started server process [77826]\r\nINFO: Waiting for application startup.\r\nINFO: Application startup complete.\r\nINFO: Uvicorn running on http://127.0.0.1:8064 (Press CTRL+C to quit)\r\n Running curls\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1901416155, "label": "Server hang on parallel execution of queries to named in-memory databases"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2189#issuecomment-1724278386", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2189", "id": 1724278386, "node_id": "IC_kwDOBm6k_c5mxmZy", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-18T19:55:32Z", "updated_at": "2023-09-18T19:55:32Z", "author_association": "OWNER", "body": "OK it looks like it found it!\r\n\r\n```\r\n942411ef946e9a34a2094944d3423cddad27efd3 is the first bad commit\r\ncommit \r\n\r\nAuthor: Simon Willison \r\nDate: Tue Apr 26 15:48:56 2022 -0700\r\n\r\n Execute some TableView queries in parallel\r\n \r\n Use ?_noparallel=1 to opt out (undocumented, useful for benchmark comparisons)\r\n \r\n Refs #1723, #1715\r\n\r\n datasette/views/table.py | 93 ++++++++++++++++++++++++++++++++++--------------\r\n 1 file changed, 67 insertions(+), 26 deletions(-)\r\nbisect found first bad commit\r\n```\r\nhttps://github.com/simonw/datasette/commit/942411ef946e9a34a2094944d3423cddad27efd3 does look like the cause of this problem.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1901416155, "label": "Server hang on parallel execution of queries to named in-memory databases"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2189#issuecomment-1724281824", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2189", "id": 1724281824, "node_id": "IC_kwDOBm6k_c5mxnPg", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-18T19:58:06Z", "updated_at": "2023-09-18T19:58:06Z", "author_association": "OWNER", "body": "I also confirmed that `http://127.0.0.1:8064/airtable_refs/airtable_refs?_noparallel=1` does not trigger the bug but `http://127.0.0.1:8064/airtable_refs/airtable_refs` does.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1901416155, "label": "Server hang on parallel execution of queries to named in-memory databases"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2189#issuecomment-1724298817", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2189", "id": 1724298817, "node_id": "IC_kwDOBm6k_c5mxrZB", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-18T20:11:26Z", "updated_at": "2023-09-18T20:11:26Z", "author_association": "OWNER", "body": "Now that I've confirmed that parallel query execution of the kind introduced in https://github.com/simonw/datasette/commit/942411ef946e9a34a2094944d3423cddad27efd3 can cause hangs (presumably some kind of locking issue) against in-memory databases, some options:\r\n\r\n1. Disable parallel execution entirely and rip out related code.\r\n2. Disable parallel execution entirely by leaving that code but having it always behave as if `_noparallel=1`\r\n3. Continue digging and try and find some way to avoid this problem\r\n\r\nThe parallel execution work is something I was playing with last year in the hope of speeding up Datasette pages like the table page which need to execute a bunch of queries - one for each facet, plus one for each column to see if it should be suggested as a facet.\r\n\r\nI wrote about this at the time here: https://simonwillison.net/2022/May/6/weeknotes/\r\n\r\nMy hope was that despite Python's GIL this optimization would still help, because the SQLite C module releases the GIL once it gets to SQLite.\r\n\r\nBut... that didn't hold up. It looked like enough work was happening in Python land with the GIL that the optimization didn't improve things.\r\n\r\nRunning the `nogil` fork of Python DID improve things though! I left the code in partly on the hope that the `nogil` fork would be accepted into Python core.\r\n\r\n... which it now has! But it will still be a year or two before it fully lands: https://discuss.python.org/t/a-steering-council-notice-about-pep-703-making-the-global-interpreter-lock-optional-in-cpython/30474\r\n\r\nSo I'm not particularly concerned about dropping the parallel execution. If I do drop it though do I leave the potentially complex code in that relates to it?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1901416155, "label": "Server hang on parallel execution of queries to named in-memory databases"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2189#issuecomment-1724305169", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2189", "id": 1724305169, "node_id": "IC_kwDOBm6k_c5mxs8R", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-18T20:16:22Z", "updated_at": "2023-09-18T20:16:36Z", "author_association": "OWNER", "body": "Looking again at this code:\r\n\r\nhttps://github.com/simonw/datasette/blob/6ed7908580fa2ba9297c3225d85c56f8b08b9937/datasette/database.py#L87-L117\r\n\r\n`check_same_thread=False` really stands out here.\r\n\r\nPython docs at https://docs.python.org/3/library/sqlite3.html\r\n\r\n> **check_same_thread** ([*bool*](https://docs.python.org/3/library/functions.html#bool \"bool\")) -- If `True` (default), [`ProgrammingError`](https://docs.python.org/3/library/sqlite3.html#sqlite3.ProgrammingError \"sqlite3.ProgrammingError\") will be raised if the database connection is used by a thread other than the one that created it. If `False`, the connection may be accessed in multiple threads; write operations may need to be serialized by the user to avoid data corruption. See [`threadsafety`](https://docs.python.org/3/library/sqlite3.html#sqlite3.threadsafety \"sqlite3.threadsafety\") for more information.\r\n\r\nI think I'm playing with fire by allowing multiple threads to access the same connection without doing my own serialization of those requests.\r\n\r\nI _do_ do that using the write connection - and in this particular case the bug isn't coming from write queries, it's coming from read queries - but perhaps SQLite has issues with threading for reads, too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1901416155, "label": "Server hang on parallel execution of queries to named in-memory databases"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2189#issuecomment-1724315591", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2189", "id": 1724315591, "node_id": "IC_kwDOBm6k_c5mxvfH", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-18T20:24:30Z", "updated_at": "2023-09-18T20:24:30Z", "author_association": "OWNER", "body": "[Using SQLite In Multi-Threaded Applications](https://www.sqlite.org/threadsafe.html)\r\n\r\nThat indicates that there's a SQLite option for \"Serialized\" mode where it's safe to access anything SQLite provides from multiple threads, but as far as I can tell Python doesn't give you an option to turn that mode on or off for a connection - you can read `sqlite3.threadsafet`y to see if that mode was compiled in or not, but not actually change it.\r\n\r\nOn my Mac `sqlite3.threadsafety` returns 1 which means https://docs.python.org/3/library/sqlite3.html#sqlite3.threadsafety \"Multi-thread: In this mode, SQLite can be safely used by multiple threads provided that no single database connection is used simultaneously in two or more threads.\" - it would need to return 3 for that serialized mode.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1901416155, "label": "Server hang on parallel execution of queries to named in-memory databases"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2189#issuecomment-1724317367", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2189", "id": 1724317367, "node_id": "IC_kwDOBm6k_c5mxv63", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-18T20:25:44Z", "updated_at": "2023-09-18T20:25:44Z", "author_association": "OWNER", "body": "My current hunch is that SQLite gets unhappy if multiple threads access the same underlying C object - which sometimes happens with in-memory connections and Datasette presumably because they are faster than file-backed databases.\r\n\r\nI'm going to remove the `asyncio.gather()` code from the table view. I'll ship a 0.x release with that fix too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1901416155, "label": "Server hang on parallel execution of queries to named in-memory databases"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2189#issuecomment-1724325068", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2189", "id": 1724325068, "node_id": "IC_kwDOBm6k_c5mxxzM", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-18T20:29:41Z", "updated_at": "2023-09-18T20:29:41Z", "author_association": "OWNER", "body": "The one other thing affected by this change is this documentation, which suggests a not-actually-safe pattern: https://github.com/simonw/datasette/blob/6ed7908580fa2ba9297c3225d85c56f8b08b9937/docs/internals.rst#L1292-L1321", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1901416155, "label": "Server hang on parallel execution of queries to named in-memory databases"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2189#issuecomment-1726749355", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2189", "id": 1726749355, "node_id": "IC_kwDOBm6k_c5m7Bqr", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-20T01:28:16Z", "updated_at": "2023-09-20T01:28:16Z", "author_association": "OWNER", "body": "Added a note to that example in the documentation: https://github.com/simonw/datasette/blob/4e6a34179eaedec44c1263275d7592fd83d7e2ac/docs/internals.rst?plain=1#L1320", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1901416155, "label": "Server hang on parallel execution of queries to named in-memory databases"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2189#issuecomment-1728192688", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2189", "id": 1728192688, "node_id": "IC_kwDOBm6k_c5nAiCw", "user": {"value": 173848, "label": "yozlet"}, "created_at": "2023-09-20T17:53:31Z", "updated_at": "2023-09-20T17:53:31Z", "author_association": "NONE", "body": "`/me munches popcorn at a furious rate, utterly entralled`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1901416155, "label": "Server hang on parallel execution of queries to named in-memory databases"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2189#issuecomment-1730162283", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2189", "id": 1730162283, "node_id": "IC_kwDOBm6k_c5nIC5r", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-21T19:19:47Z", "updated_at": "2023-09-21T19:19:47Z", "author_association": "OWNER", "body": "I'm going to release this in `1.0a7`, and I'll backport it to a `0.64.4` release too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1901416155, "label": "Server hang on parallel execution of queries to named in-memory databases"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2189#issuecomment-1730231404", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2189", "id": 1730231404, "node_id": "IC_kwDOBm6k_c5nITxs", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-21T20:10:28Z", "updated_at": "2023-09-21T20:10:28Z", "author_association": "OWNER", "body": "Release 0.64.4: https://docs.datasette.io/en/stable/changelog.html#v0-64-4", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1901416155, "label": "Server hang on parallel execution of queries to named in-memory databases"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2189#issuecomment-1730232308", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2189", "id": 1730232308, "node_id": "IC_kwDOBm6k_c5nIT_0", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-21T20:11:16Z", "updated_at": "2023-09-21T20:11:16Z", "author_association": "OWNER", "body": "We're planning a breaking change in `1.0a7`:\r\n- #2191 \r\n\r\nSince that's a breaking change I'm going to ship 1.0a7 right now with this fix, then ship that breaking change as `1.0a8` instead.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1901416155, "label": "Server hang on parallel execution of queries to named in-memory databases"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2189#issuecomment-1730388418", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2189", "id": 1730388418, "node_id": "IC_kwDOBm6k_c5nI6HC", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-21T22:26:19Z", "updated_at": "2023-09-21T22:26:19Z", "author_association": "OWNER", "body": "1.0a7 is out with this fix as well now: https://docs.datasette.io/en/1.0a7/changelog.html#a7-2023-09-21", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1901416155, "label": "Server hang on parallel execution of queries to named in-memory databases"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2188#issuecomment-1722662413", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2188", "id": 1722662413, "node_id": "IC_kwDOBm6k_c5mrb4N", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-18T02:01:34Z", "updated_at": "2023-09-18T02:01:34Z", "author_association": "OWNER", "body": "I'm not interested in these in Datasette core itself, but I think they have a ton of potential for plugins.\r\n\r\nI wonder what the best way to handle that would be?\r\n\r\nRight now it's possible to write a plugin that [adds extra routes](https://docs.datasette.io/en/stable/plugin_hooks.html#register-routes-datasette), so someone could build a `/dbname/-/prql?query=xxx` endpoint.\r\n\r\nIf this could return JSON, they could add JavaScript to the `/dbname` page that provided a UI for kicking off one of those queries.\r\n\r\nSomething that could make that more ergonomic might be the plugin hook that allows plugins to add extra HTML to different core database pages - e.g. adding a \"Query this database using PRQL\" button or link or even a full form at the top of that database page. That's this issue here:\r\n\r\n- #1191", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1900026059, "label": "Plugin Hooks for \"compile to SQL\" languages"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2188#issuecomment-1722848454", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2188", "id": 1722848454, "node_id": "IC_kwDOBm6k_c5msJTG", "user": {"value": 15178711, "label": "asg017"}, "created_at": "2023-09-18T06:58:53Z", "updated_at": "2023-09-18T06:58:53Z", "author_association": "CONTRIBUTOR", "body": "Thinking about this more, here a list of things I imagine a \"compile-to-sql\" plugin would want to do:\r\n\r\n1. Attach itself to the SQL code editor (switch from SQL -> PRQL/Logica, additional syntax highlighting)\r\n2. Add \"Query using PRQL\" buttons in various parts of Datasette's UI, like `/dbname` page\r\n3. Use `$LANGUAGE=` instead of `sql=` in the JSON API and the SQL results pages\r\n4. Have their own dedicated code editor page\r\n\r\n\r\n1) and 2) would be difficult to do with current plugin hooks, unless we add the concept of \"slots\" and get the JS plugin support in. 3) could maybe be done with the [`asgi_wrapper(datasette)`](https://docs.datasette.io/en/stable/plugin_hooks.html#asgi-wrapper-datasette) hook? And 4) ca n be done easily with the `register_routes()` hooks. \r\n\r\nSo it really only sounds like extending the SQL editor will be the hard part. In #2094 I want to add JavaScript plugin hooks for extending the SQL editor, which may work here. \r\n\r\nIf I get the time/motivation, I might try out a `datasette-prql` extension, just because I like playing with it. It'd be really cool if I can get the `asgi_wrapper()` hook to work right there...", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1900026059, "label": "Plugin Hooks for \"compile to SQL\" languages"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2186#issuecomment-1721740872", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2186", "id": 1721740872, "node_id": "IC_kwDOBm6k_c5mn65I", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-15T19:26:51Z", "updated_at": "2023-09-15T19:27:19Z", "author_association": "OWNER", "body": "Here's where it's called at the moment: https://github.com/simonw/datasette/blob/16f0b6d8222d06682a31b904d0a402c391ae1c1c/datasette/views/base.py#L297-L313\r\n\r\nAnd the docs: https://github.com/simonw/datasette/blob/1.0a6/docs/plugin_hooks.rst#register-output-renderer-datasette\r\n\r\nI'm tempted to add a `get_count` argument which, when awaited, returns the full count. Then plugins could do this:\r\n\r\n```python\r\nasync def render_notebook(datasette, request, get_count, rows):\r\n count = await get_count()\r\n # ...\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1898927976, "label": "Mechanism for register_output_renderer hooks to access full count"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2186#issuecomment-1721742055", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2186", "id": 1721742055, "node_id": "IC_kwDOBm6k_c5mn7Ln", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-15T19:27:59Z", "updated_at": "2023-09-15T19:27:59Z", "author_association": "OWNER", "body": "This feels like it might be quite a nice pattern generally - providing optional arguments to plugins and views that can be `await get_x()` called to run an extra calculation.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1898927976, "label": "Mechanism for register_output_renderer hooks to access full count"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2185#issuecomment-1719468727", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2185", "id": 1719468727, "node_id": "IC_kwDOBm6k_c5mfQK3", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2023-09-14T13:36:07Z", "updated_at": "2023-09-19T13:42:35Z", "author_association": "NONE", "body": "## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2185?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\nPatch coverage has no change and project coverage change: **`-0.04%`** :warning:\n> Comparison is base [(`6ed7908`)](https://app.codecov.io/gh/simonw/datasette/commit/6ed7908580fa2ba9297c3225d85c56f8b08b9937?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69% compared to head [(`fe5f881`)](https://app.codecov.io/gh/simonw/datasette/pull/2185?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.66%.\n\n
Additional details and impacted files\n\n\n```diff\n@@ Coverage Diff @@\n## main #2185 +/- ##\n==========================================\n- Coverage 92.69% 92.66% -0.04% \n==========================================\n Files 40 40 \n Lines 6039 6039 \n==========================================\n- Hits 5598 5596 -2 \n- Misses 441 443 +2 \n```\n\n\n[see 1 file with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2185/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n\n\n
\n\n[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2185?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). \n:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1896578249, "label": "Bump the python-packages group with 3 updates"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/594#issuecomment-1714919806", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/594", "id": 1714919806, "node_id": "IC_kwDOCGYnMM5mN5l-", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-12T03:49:41Z", "updated_at": "2023-09-12T03:49:41Z", "author_association": "OWNER", "body": "Digging in a bit more:\r\n```pycon\r\n>>> pprint(list(db.query('PRAGMA foreign_key_list(courses)')))\r\n[{'from': 'campus_name',\r\n 'id': 0,\r\n 'match': 'NONE',\r\n 'on_delete': 'NO ACTION',\r\n 'on_update': 'NO ACTION',\r\n 'seq': 0,\r\n 'table': 'departments',\r\n 'to': 'campus_name'},\r\n {'from': 'dept_code',\r\n 'id': 0,\r\n 'match': 'NONE',\r\n 'on_delete': 'NO ACTION',\r\n 'on_update': 'NO ACTION',\r\n 'seq': 1,\r\n 'table': 'departments',\r\n 'to': 'dept_code'}]\r\n```\r\nI think the way you tell it's a compound foreign key is that both of those have the same `id` value - of `0` - but they then have two different `seq` values of `0` and `1`.\r\n\r\nRight now I ignore those columns entirely: https://github.com/simonw/sqlite-utils/blob/622c3a5a7dd53a09c029e2af40c2643fe7579340/sqlite_utils/db.py#L1523-L1540", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1891614971, "label": "Represent compound foreign keys in table.foreign_keys output"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/594#issuecomment-1714920708", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/594", "id": 1714920708, "node_id": "IC_kwDOCGYnMM5mN50E", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-12T03:51:13Z", "updated_at": "2023-09-12T03:51:13Z", "author_association": "OWNER", "body": "Changing this without breaking backwards compatibility (and forcing a 4.0 release) will be tricky, because `ForeignKey()` is a `namedtuple`:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/622c3a5a7dd53a09c029e2af40c2643fe7579340/sqlite_utils/db.py#L148-L150\r\n\r\nI could swap it out for a `dataclass` and add those extra columns, but I need to make sure that code like this still works:\r\n```python\r\nfor table, column, other_table, other_column in table.foreign_keys:\r\n # ...\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1891614971, "label": "Represent compound foreign keys in table.foreign_keys output"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2183#issuecomment-1714544153", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2183", "id": 1714544153, "node_id": "IC_kwDOBm6k_c5mMd4Z", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2023-09-11T20:37:52Z", "updated_at": "2023-09-13T20:58:51Z", "author_association": "NONE", "body": "## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2183?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\nPatch coverage: **`95.00%`** and project coverage change: **`-0.04%`** :warning:\n> Comparison is base [(`a4c96d0`)](https://app.codecov.io/gh/simonw/datasette/commit/a4c96d01b27ce7cd06662a024da3547132a7c412?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69% compared to head [(`659dcbd`)](https://app.codecov.io/gh/simonw/datasette/pull/2183?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.66%.\n\n> :exclamation: Current head 659dcbd differs from pull request most recent head acca338. Consider uploading reports for the commit acca338 to get more accurate results\n\n
Additional details and impacted files\n\n\n```diff\n@@ Coverage Diff @@\n## main #2183 +/- ##\n==========================================\n- Coverage 92.69% 92.66% -0.04% \n==========================================\n Files 40 40 \n Lines 6025 6039 +14 \n==========================================\n+ Hits 5585 5596 +11 \n- Misses 440 443 +3 \n```\n\n\n| [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2183?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage \u0394 | |\n|---|---|---|\n| [datasette/app.py](https://app.codecov.io/gh/simonw/datasette/pull/2183?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.19% <95.00%> (-0.24%)` | :arrow_down: |\n\n\n
\n\n[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2183?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). \n:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1891212159, "label": "`datasette.yaml` plugin support"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2183#issuecomment-1714699724", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2183", "id": 1714699724, "node_id": "IC_kwDOBm6k_c5mND3M", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-11T23:01:36Z", "updated_at": "2023-09-11T23:02:30Z", "author_association": "OWNER", "body": "On thinking about this further, I'm fine releasing it as another alpha provided it causes Datasette to error loudly with an explanatory message if you attempt to load `-m metadata.json` at a metadata file that includes `\"plugins\"` configuration.\r\n\r\nSomething like this:\r\n```bash\r\ndatasette -m metadata.json\r\n```\r\nOutputs:\r\n\r\n> Datasette no longer accepts plugin configuration in --metadata. Move your `\"plugins\"` configuration blocks to a separate file - we suggest calling that `datasette.yaml` - and start Datasette with `datasette -c datasette.yaml`.\r\n\r\nFor added usability points, let's have it suggest `datasette.json` if they used `metadata.json` and `datasette.yaml` if they tried to use `metadata.yaml`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1891212159, "label": "`datasette.yaml` plugin support"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2183#issuecomment-1716801971", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2183", "id": 1716801971, "node_id": "IC_kwDOBm6k_c5mVFGz", "user": {"value": 15178711, "label": "asg017"}, "created_at": "2023-09-13T01:34:01Z", "updated_at": "2023-09-13T01:34:01Z", "author_association": "CONTRIBUTOR", "body": "@simonw docs are finished, this is ready for review!\r\n\r\nOne thing: I added \"Configuration\" as a top-level item in the documentation site, at the very bottom. Not sure if this is the best, maybe it can be named \"datasette.yaml Configuration\" or something similar?\r\n\r\nMostly because \"Configuration\" by itself can mean many things, but adding \"datasette.yaml\" would make it pretty clear it's about that specific file, and is easier to scan. I'd also be fine with using \"datasette.yaml\" instead of \"datasette.json\", since writing in YAML is much more forgiving (and advanced users will know JSON is also supported)\r\n\r\nAlso, maybe this is a chance to consolidate the docs a bit? I think \"Settings\", \"Configuration\", \"Metadata\", and \"Authentication and permissions\" should possibly be under the same section. Maybe even consolidate the different Plugin pages that exist?\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1891212159, "label": "`datasette.yaml` plugin support"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2183#issuecomment-1718316733", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2183", "id": 1718316733, "node_id": "IC_kwDOBm6k_c5ma269", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-13T21:05:36Z", "updated_at": "2023-09-13T21:05:36Z", "author_association": "OWNER", "body": "I'm going to land this and make any further documentation tweaks on `main`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1891212159, "label": "`datasette.yaml` plugin support"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2182#issuecomment-1719451803", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2182", "id": 1719451803, "node_id": "IC_kwDOBm6k_c5mfMCb", "user": {"value": 49699333, "label": "dependabot[bot]"}, "created_at": "2023-09-14T13:27:26Z", "updated_at": "2023-09-14T13:27:26Z", "author_association": "CONTRIBUTOR", "body": "Looks like these dependencies are updatable in another way, so this is no longer needed.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1890593563, "label": "Bump the python-packages group with 2 updates"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2181#issuecomment-1710969448", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2181", "id": 1710969448, "node_id": "IC_kwDOBm6k_c5l-1Jo", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-08T01:53:27Z", "updated_at": "2023-09-08T01:53:27Z", "author_association": "OWNER", "body": "Documentation preview:\r\n- https://datasette--2181.org.readthedocs.build/en/2181/internals.html#await-actors-from-ids-actor-ids\r\n- https://datasette--2181.org.readthedocs.build/en/2181/plugin_hooks.html#plugin-hook-actors-from-ids", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1886812002, "label": "actors_from_ids plugin hook and datasette.actors_from_ids() method"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2181#issuecomment-1710972324", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2181", "id": 1710972324, "node_id": "IC_kwDOBm6k_c5l-12k", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2023-09-08T01:58:44Z", "updated_at": "2023-09-08T03:43:43Z", "author_association": "NONE", "body": "## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2181?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\nPatch coverage: **`100.00%`** and no project coverage change.\n> Comparison is base [(`c263704`)](https://app.codecov.io/gh/simonw/datasette/commit/c26370485a4fd4bf130da051be9163d92c57f24f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68% compared to head [(`e1c5a9d`)](https://app.codecov.io/gh/simonw/datasette/pull/2181?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69%.\n\n
Additional details and impacted files\n\n\n```diff\n@@ Coverage Diff @@\n## main #2181 +/- ##\n=======================================\n Coverage 92.68% 92.69% \n=======================================\n Files 40 40 \n Lines 6017 6025 +8 \n=======================================\n+ Hits 5577 5585 +8 \n Misses 440 440 \n```\n\n\n| [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2181?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage \u0394 | |\n|---|---|---|\n| [datasette/app.py](https://app.codecov.io/gh/simonw/datasette/pull/2181?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.43% <100.00%> (+0.03%)` | :arrow_up: |\n| [datasette/hookspecs.py](https://app.codecov.io/gh/simonw/datasette/pull/2181?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2hvb2tzcGVjcy5weQ==) | `100.00% <100.00%> (\u00f8)` | |\n\n\n
\n\n[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2181?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). \n\n:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1886812002, "label": "actors_from_ids plugin hook and datasette.actors_from_ids() method"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2181#issuecomment-1711054840", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2181", "id": 1711054840, "node_id": "IC_kwDOBm6k_c5l_J_4", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-08T04:23:47Z", "updated_at": "2023-09-08T04:23:47Z", "author_association": "OWNER", "body": "I've implemented this hook once now in:\r\n- https://github.com/datasette/datasette-remote-actors\r\n\r\nAnd built and tested a debug tool for it in:\r\n- https://github.com/datasette/datasette-debug-actors-from-ids\r\n\r\nI'm now confident in the design of this plugin hook, I'm going to land it.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1886812002, "label": "actors_from_ids plugin hook and datasette.actors_from_ids() method"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2180#issuecomment-1710947637", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2180", "id": 1710947637, "node_id": "IC_kwDOBm6k_c5l-v01", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-08T01:17:28Z", "updated_at": "2023-09-08T01:17:28Z", "author_association": "OWNER", "body": "I think this is both a plugin hook and a `await datasette.actors_from_ids(actor_ids)` internal API function that calls it.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1886791100, "label": "Plugin hook: `actors_from_ids()`"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2180#issuecomment-1710969339", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2180", "id": 1710969339, "node_id": "IC_kwDOBm6k_c5l-1H7", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-08T01:53:16Z", "updated_at": "2023-09-08T01:53:16Z", "author_association": "OWNER", "body": "Documentation preview:\r\n- https://datasette--2181.org.readthedocs.build/en/2181/internals.html#await-actors-from-ids-actor-ids\r\n- https://datasette--2181.org.readthedocs.build/en/2181/plugin_hooks.html#plugin-hook-actors-from-ids", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1886791100, "label": "Plugin hook: `actors_from_ids()`"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2180#issuecomment-1711028355", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2180", "id": 1711028355, "node_id": "IC_kwDOBm6k_c5l_DiD", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-08T03:36:02Z", "updated_at": "2023-09-08T03:36:02Z", "author_association": "OWNER", "body": "I shipped the first version of \r\n\r\n- https://github.com/datasette/datasette-remote-actors/issues/1\r\n\r\nWhen I land this plugin in a Datasette release I need to update that repo to depend on the new alpha.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1886791100, "label": "Plugin hook: `actors_from_ids()`"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2180#issuecomment-1711054624", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2180", "id": 1711054624, "node_id": "IC_kwDOBm6k_c5l_J8g", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-08T04:23:23Z", "updated_at": "2023-09-08T04:23:23Z", "author_association": "OWNER", "body": "I've implemented this hook once now in:\r\n- https://github.com/datasette/datasette-remote-actors\r\n\r\nAnd built and tested a debug tool for it in:\r\n- https://github.com/datasette/datasette-debug-actors-from-ids\r\n\r\nI'm now confident in the design of this plugin hook, I'm going to land it.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1886791100, "label": "Plugin hook: `actors_from_ids()`"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2180#issuecomment-1711057080", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2180", "id": 1711057080, "node_id": "IC_kwDOBm6k_c5l_Ki4", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-08T04:28:03Z", "updated_at": "2023-09-08T04:28:03Z", "author_association": "OWNER", "body": "Landed:\r\n\r\n- https://docs.datasette.io/en/latest/plugin_hooks.html#actors-from-ids-datasette-actor-ids\r\n- https://docs.datasette.io/en/latest/internals.html#await-actors-from-ids-actor-ids", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1886791100, "label": "Plugin hook: `actors_from_ids()`"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2180#issuecomment-1712895084", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2180", "id": 1712895084, "node_id": "IC_kwDOBm6k_c5mGLRs", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-10T17:44:14Z", "updated_at": "2023-09-10T17:44:14Z", "author_association": "OWNER", "body": "Used by `datasette-short-links` here: https://github.com/datasette/datasette-short-links/blob/468c3e25dbe06a8dcba8edda59bc16a18e126a51/datasette_short_links/__init__.py#L108-L115", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1886791100, "label": "Plugin hook: `actors_from_ids()`"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/593#issuecomment-1710939868", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/593", "id": 1710939868, "node_id": "IC_kwDOCGYnMM5l-t7c", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2023-09-08T01:03:40Z", "updated_at": "2023-09-09T00:44:52Z", "author_association": "NONE", "body": "## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/593?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\nPatch coverage: **`100.00%`** and no project coverage change.\n> Comparison is base [(`5d123f0`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/5d123f031fc4fadc98f508e0ef6b7b6671e86155?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.77% compared to head [(`b86374f`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/593?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.77%.\n\n
Additional details and impacted files\n\n\n```diff\n@@ Coverage Diff @@\n## main #593 +/- ##\n=======================================\n Coverage 95.77% 95.77% \n=======================================\n Files 8 8 \n Lines 2837 2840 +3 \n=======================================\n+ Hits 2717 2720 +3 \n Misses 120 120 \n```\n\n\n| [Files Changed](https://app.codecov.io/gh/simonw/sqlite-utils/pull/593?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage \u0394 | |\n|---|---|---|\n| [sqlite\\_utils/db.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/593?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.22% <100.00%> (+<0.01%)` | :arrow_up: |\n\n\n
\n\n[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/593?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). \n:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1886783150, "label": ".transform() now preserves rowid values, refs #592"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1710930934", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/592", "id": 1710930934, "node_id": "IC_kwDOCGYnMM5l-rv2", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-08T00:47:57Z", "updated_at": "2023-09-08T00:47:57Z", "author_association": "OWNER", "body": "That's odd, I wrote a test for this just now and it passes already:\r\n```python\r\ndef test_transform_preserves_rowids(fresh_db):\r\n # Create a rowid table\r\n fresh_db[\"places\"].insert_all(\r\n (\r\n {\"name\": \"Paris\", \"country\": \"France\"},\r\n {\"name\": \"London\", \"country\": \"UK\"},\r\n {\"name\": \"New York\", \"country\": \"USA\"},\r\n ),\r\n )\r\n assert fresh_db[\"places\"].use_rowid\r\n previous_rows = list(\r\n tuple(row) for row in fresh_db.execute(\"select rowid, name from places\")\r\n )\r\n # Transform it\r\n fresh_db[\"places\"].transform(column_order=(\"country\", \"name\"))\r\n # Should be the same\r\n next_rows = list(\r\n tuple(row) for row in fresh_db.execute(\"select rowid, name from places\")\r\n )\r\n assert previous_rows == next_rows\r\n```\r\nSo maybe I'm wrong about the cause of that bug?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1886771493, "label": "`table.transform()` should preserve `rowid` values"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1710931605", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/592", "id": 1710931605, "node_id": "IC_kwDOCGYnMM5l-r6V", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-08T00:49:02Z", "updated_at": "2023-09-08T00:49:02Z", "author_association": "OWNER", "body": "I tried bumping that up to 10,000 rows instead of just 3 but the test still passed.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1886771493, "label": "`table.transform()` should preserve `rowid` values"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1710933716", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/592", "id": 1710933716, "node_id": "IC_kwDOCGYnMM5l-sbU", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-08T00:52:42Z", "updated_at": "2023-09-08T00:52:42Z", "author_association": "OWNER", "body": "I just noticed that the table where I encountered this bug wasn't actually a `rowid` table after all - it had an `id` column that was a text primary key.\r\n\r\nThe reason the `rowid` was important is that's how the FTS mechanism in Datasette relates FTS entries to their rows.\r\n\r\nBut I tried this test and it passed, too:\r\n```python\r\ndef test_transform_preserves_rowids(fresh_db):\r\n fresh_db[\"places\"].insert_all(\r\n [\r\n {\"id\": \"1\", \"name\": \"Paris\", \"country\": \"France\"},\r\n {\"id\": \"2\", \"name\": \"London\", \"country\": \"UK\"},\r\n {\"id\": \"3\", \"name\": \"New York\", \"country\": \"USA\"},\r\n ],\r\n pk=\"id\",\r\n )\r\n previous_rows = list(\r\n tuple(row) for row in fresh_db.execute(\"select rowid, id, name from places\")\r\n )\r\n # Transform it\r\n fresh_db[\"places\"].transform(column_order=(\"country\", \"name\"))\r\n # Should be the same\r\n next_rows = list(\r\n tuple(row) for row in fresh_db.execute(\"select rowid, id, name from places\")\r\n )\r\n assert previous_rows == next_rows\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1886771493, "label": "`table.transform()` should preserve `rowid` values"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1710934448", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/592", "id": 1710934448, "node_id": "IC_kwDOCGYnMM5l-smw", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-08T00:54:03Z", "updated_at": "2023-09-08T00:54:03Z", "author_association": "OWNER", "body": "Oh! Maybe the row ID preservation here is a coincidence because the tables are created from scratch and count 1, 2, 3.\r\n\r\nIf I delete a row from the table and then insert some more - breaking the `rowid` sequence - it might show the bug.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1886771493, "label": "`table.transform()` should preserve `rowid` values"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1710935270", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/592", "id": 1710935270, "node_id": "IC_kwDOCGYnMM5l-szm", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-08T00:55:30Z", "updated_at": "2023-09-08T00:55:30Z", "author_association": "OWNER", "body": "Yes! That recreated the bug:\r\n```\r\n> assert previous_rows == next_rows\r\nE AssertionError: assert equals failed\r\nE [ [ \r\nE (1, '1', 'Paris'), (1, '1', 'Paris'), \r\nE (3, '3', 'New York'), (2, '3', 'New York'), \r\nE (4, '4', 'London'), (3, '4', 'London'), \r\nE ] ...\r\nE \r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1886771493, "label": "`table.transform()` should preserve `rowid` values"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1712895580", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/592", "id": 1712895580, "node_id": "IC_kwDOCGYnMM5mGLZc", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-10T17:46:41Z", "updated_at": "2023-09-10T17:46:41Z", "author_association": "OWNER", "body": "In working on this I learned that `rowid` values in SQLite are way less stable than I had thought - in particular, they are often entirely rewritten on a `VACUUM`:\r\n\r\nhttps://www.sqlite.org/lang_vacuum.html#how_vacuum_works\r\n\r\n> The VACUUM command may change the [ROWIDs](https://www.sqlite.org/lang_createtable.html#rowid) of entries in any tables that do not have an explicit [INTEGER PRIMARY KEY](https://www.sqlite.org/lang_createtable.html#rowid).\r\n\r\nSo this fix wasn't as valuable as I thought. I need to move away from ever assuming that a `rowid` is a useful foreign key for anything.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1886771493, "label": "`table.transform()` should preserve `rowid` values"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2178#issuecomment-1710565268", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2178", "id": 1710565268, "node_id": "IC_kwDOBm6k_c5l9SeU", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-07T17:58:04Z", "updated_at": "2023-09-07T17:59:06Z", "author_association": "OWNER", "body": "Relevant code: https://github.com/simonw/datasette/blob/fbcb103c0cb6668018ace539a01a6a1f156e8d6a/datasette/views/table.py#L1132-L1149\r\n\r\nWhich calls this undocumented method:\r\n\r\nhttps://github.com/simonw/datasette/blob/fbcb103c0cb6668018ace539a01a6a1f156e8d6a/datasette/app.py#L938-L973", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1886350562, "label": "Don't show foreign key links to tables the user cannot access"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2178#issuecomment-1710567329", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2178", "id": 1710567329, "node_id": "IC_kwDOBm6k_c5l9S-h", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-07T17:59:59Z", "updated_at": "2023-09-07T17:59:59Z", "author_association": "OWNER", "body": "Should I put the permission check in that undocumented `datasette.expand_foreign_keys()` method? I think so - it should accept `request.actor` as one of its arguments.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1886350562, "label": "Don't show foreign key links to tables the user cannot access"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2178#issuecomment-1710871095", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2178", "id": 1710871095, "node_id": "IC_kwDOBm6k_c5l-dI3", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-07T23:07:16Z", "updated_at": "2023-09-07T23:07:16Z", "author_association": "OWNER", "body": "I ran this:\r\n\r\n datasette content.db -p 8043 -m fk-auth.yml --root\r\n\r\nAgainst this YAML:\r\n```yaml\r\ndatabases:\r\n content:\r\n tables:\r\n users:\r\n allow:\r\n id: root\r\n```\r\nAnd it worked as it should - here's a screenshot of an anonymous user and a root user viewing the same page:\r\n\r\n![CleanShot 2023-09-07 at 16 05 34@2x](https://github.com/simonw/datasette/assets/9599/3e91da08-107c-421c-8a00-aa650b960c58)\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1886350562, "label": "Don't show foreign key links to tables the user cannot access"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2178#issuecomment-1710878391", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2178", "id": 1710878391, "node_id": "IC_kwDOBm6k_c5l-e63", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-07T23:19:05Z", "updated_at": "2023-09-07T23:19:05Z", "author_association": "OWNER", "body": "This fix didn't work on Datasette Cloud. I used `/-/permissions` to debug it and saw this:\r\n\r\n![image](https://github.com/simonw/datasette/assets/9599/61d2bc5f-1f96-41ea-8658-91dfbcb6610c)\r\n\r\nOnly checking `view-table` is not enough: for my instance on Datasette Cloud the view permission check that should have failed was for the database or instance.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1886350562, "label": "Don't show foreign key links to tables the user cannot access"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2178#issuecomment-1710879239", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2178", "id": 1710879239, "node_id": "IC_kwDOBm6k_c5l-fIH", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-07T23:20:32Z", "updated_at": "2023-09-07T23:20:32Z", "author_association": "OWNER", "body": "To test that locally, use this YAML instead:\r\n```yaml\r\ndatabases:\r\n content:\r\n allow:\r\n id: root\r\n tables:\r\n releases:\r\n allow: true\r\n```\r\nAnd:\r\n```yaml\r\nallow:\r\n id: root\r\ndatabases:\r\n content:\r\n tables:\r\n releases:\r\n allow: true", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1886350562, "label": "Don't show foreign key links to tables the user cannot access"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2177#issuecomment-1708777964", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2177", "id": 1708777964, "node_id": "IC_kwDOBm6k_c5l2eHs", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-06T17:04:30Z", "updated_at": "2023-09-06T17:04:30Z", "author_association": "OWNER", "body": "Here's the thinking: setting the`DATASETTE_INTERNAL` environment variable for your entire machine will cause any time you run `datasette x.db` to result in a process that shares the same `internal.db` database as other instances.\r\n\r\nYou might run more than one instance at once (I often have 4 or 5 going). This would currently break, because they would over-write each other's catalog tables:\r\n\r\nhttps://github.com/simonw/datasette/blob/e4abae3fd7a828625d00c35c316852ffbaa5ef2f/datasette/utils/internal_db.py#L5-L62\r\n\r\nThe breaking wouldn't be obvious because the catalog tables aren't used by any features yet, but it's still bad.\r\n\r\nThis convinced us that actually we should move those `catalog_` tables OUT of `internal.db`. The `_internal` database will be reserved for plugins that want to use it for caching, storing progress, etc.\r\n\r\nI think we move them to an in-memory `_catalog` database which is excluded from `ds.databases` (like `_internal` is ) but can be accessed using `datasette.get_catalog_database()` - similar to `datasette.get_internal_database()`.\r\n\r\nSo each instance of Datasette gets its own truly private `_catalog`, which is in-memory and so gets cleared at the end of each process.\r\n\r\nAn interesting thing that came up about a shared `_internal` database is that it provides opportunities for things like a notes plugin which allows you to attach notes to any row in any table in a database... where those notes become available to multiple Datasette instances that you might launch on the same laptop.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1884408624, "label": "Move schema tables from _internal to _catalog"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/591#issuecomment-1708693020", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/591", "id": 1708693020, "node_id": "IC_kwDOCGYnMM5l2JYc", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2023-09-06T16:14:03Z", "updated_at": "2023-11-04T00:54:25Z", "author_association": "NONE", "body": "## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/591?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\nAll modified and coverable lines are covered by tests :white_check_mark:\n> Comparison is base [(`347fdc8`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/347fdc865e91b8d3410f49a5c9d5b499fbb594c1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.74% compared to head [(`1f14df1`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/591?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.74%.\n\n
Additional details and impacted files\n\n\n```diff\n@@ Coverage Diff @@\n## main #591 +/- ##\n=======================================\n Coverage 95.74% 95.74% \n=======================================\n Files 8 8 \n Lines 2842 2842 \n=======================================\n Hits 2721 2721 \n Misses 121 121 \n```\n\n\n\n\n
\n\n[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/591?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). \n:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1884335789, "label": "Test against Python 3.12 preview"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/591#issuecomment-1708695907", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/591", "id": 1708695907, "node_id": "IC_kwDOCGYnMM5l2KFj", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-06T16:15:59Z", "updated_at": "2023-09-06T16:19:14Z", "author_association": "OWNER", "body": "The test failure was while installing `numpy`, relating to importing `distutils` - maybe relevant:\r\n- https://github.com/pypa/setuptools/issues/3661\r\n\r\n```\r\n25h Installing build dependencies: started\r\n Installing build dependencies: finished with status 'done'\r\n Getting requirements to build wheel: started\r\n Getting requirements to build wheel: finished with status 'done'\r\nERROR: Exception:\r\nTraceback (most recent call last):\r\n...\r\n File \"/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_internal/utils/misc.py\", line 697, in get_requires_for_build_wheel\r\n return super().get_requires_for_build_wheel(config_settings=cs)\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_impl.py\", line 166, in get_requires_for_build_wheel\r\n return self._call_hook('get_requires_for_build_wheel', {\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_impl.py\", line 321, in _call_hook\r\n raise BackendUnavailable(data.get('traceback', ''))\r\npip._vendor.pyproject_hooks._impl.BackendUnavailable: Traceback (most recent call last):\r\n File \"/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py\", line 77, in _build_backend\r\n obj = import_module(mod_path)\r\n ^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/importlib/__init__.py\", line 90, in import_module\r\n return _bootstrap._gcd_import(name[level:], package, level)\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"\", line 1381, in _gcd_import\r\n File \"\", line 1354, in _find_and_load\r\n File \"\", line 1304, in _find_and_load_unlocked\r\n File \"\", line 488, in _call_with_frames_removed\r\n File \"\", line 1381, in _gcd_import\r\n File \"\", line 1354, in _find_and_load\r\n File \"\", line 1325, in _find_and_load_unlocked\r\n File \"\", line 929, in _load_unlocked\r\n File \"\", line 994, in exec_module\r\n File \"\", line 488, in _call_with_frames_removed\r\n File \"/tmp/pip-build-env-x9nyg3kd/overlay/lib/python3.12/site-packages/setuptools/__init__.py\", line 10, in \r\n import distutils.core\r\nModuleNotFoundError: No module named 'distutils'\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1884335789, "label": "Test against Python 3.12 preview"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/591#issuecomment-1793278279", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/591", "id": 1793278279, "node_id": "IC_kwDOCGYnMM5q40FH", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-11-04T00:58:03Z", "updated_at": "2023-11-04T00:58:03Z", "author_association": "OWNER", "body": "I'm going to abandon this PR and ship the 3.12 testing change directly to `main`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1884335789, "label": "Test against Python 3.12 preview"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2174#issuecomment-1708699926", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2174", "id": 1708699926, "node_id": "IC_kwDOBm6k_c5l2LEW", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2023-09-06T16:18:13Z", "updated_at": "2023-09-08T00:46:13Z", "author_association": "NONE", "body": "## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2174?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\nPatch and project coverage have no change.\n> Comparison is base [(`05707aa`)](https://app.codecov.io/gh/simonw/datasette/commit/05707aa16b5c6c39fbe48b3176b85a8ffe493938?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68% compared to head [(`d75b519`)](https://app.codecov.io/gh/simonw/datasette/pull/2174?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68%.\n> Report is 7 commits behind head on main.\n\n
Additional details and impacted files\n\n\n```diff\n@@ Coverage Diff @@\n## main #2174 +/- ##\n=======================================\n Coverage 92.68% 92.68% \n=======================================\n Files 40 40 \n Lines 6012 6017 +5 \n=======================================\n+ Hits 5572 5577 +5 \n Misses 440 440 \n```\n\n\n| [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage \u0394 | |\n|---|---|---|\n| [datasette/cli.py](https://app.codecov.io/gh/simonw/datasette/pull/2174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `82.53% <\u00f8> (\u00f8)` | |\n\n... and [1 file with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2174/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n\n\n
\n\n[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2174?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). \n\n:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1884330740, "label": "Use $DATASETTE_INTERNAL in absence of --internal"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2174#issuecomment-1708727204", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2174", "id": 1708727204, "node_id": "IC_kwDOBm6k_c5l2Ruk", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-06T16:35:39Z", "updated_at": "2023-09-07T23:45:42Z", "author_association": "OWNER", "body": "We can use this here: https://click.palletsprojects.com/en/8.1.x/options/#values-from-environment-variables\r\n```python\r\n@click.option(..., envvar=\"DATASETTE_INTERNAL\")\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1884330740, "label": "Use $DATASETTE_INTERNAL in absence of --internal"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2174#issuecomment-1708728926", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2174", "id": 1708728926, "node_id": "IC_kwDOBm6k_c5l2SJe", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-06T16:36:56Z", "updated_at": "2023-09-06T16:36:56Z", "author_association": "OWNER", "body": "`DATASETTE_INTERNAL` would be more consistent with `DATASETTE_SECRET`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1884330740, "label": "Use $DATASETTE_INTERNAL in absence of --internal"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2173#issuecomment-1707565495", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2173", "id": 1707565495, "node_id": "IC_kwDOBm6k_c5lx2G3", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-06T02:33:56Z", "updated_at": "2023-09-06T02:33:56Z", "author_association": "OWNER", "body": "Running tests to see if Pyodide works - that was the reason I switched to my `click-default-group-wheel` package originally.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1883055640, "label": "click-default-group>=1.2.3"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2173#issuecomment-1707570044", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2173", "id": 1707570044, "node_id": "IC_kwDOBm6k_c5lx3N8", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2023-09-06T02:39:59Z", "updated_at": "2023-09-06T02:49:01Z", "author_association": "NONE", "body": "## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2173?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\nPatch and project coverage have no change.\n> Comparison is base [(`fd083e3`)](https://app.codecov.io/gh/simonw/datasette/commit/fd083e37ec53e7e625111168d324a572344a3b19?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68% compared to head [(`a481ebd`)](https://app.codecov.io/gh/simonw/datasette/pull/2173?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68%.\n> Report is 1 commits behind head on main.\n\n
Additional details and impacted files\n\n\n```diff\n@@ Coverage Diff @@\n## main #2173 +/- ##\n=======================================\n Coverage 92.68% 92.68% \n=======================================\n Files 40 40 \n Lines 6012 6012 \n=======================================\n Hits 5572 5572 \n Misses 440 440 \n```\n\n\n\n\n
\n\n[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2173?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). \n:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1883055640, "label": "click-default-group>=1.2.3"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2173#issuecomment-1707570378", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2173", "id": 1707570378, "node_id": "IC_kwDOBm6k_c5lx3TK", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-06T02:40:31Z", "updated_at": "2023-09-06T02:40:31Z", "author_association": "OWNER", "body": "Surprising error:\r\n```\r\n blacken-docs -l 60 docs/*.rst\r\n shell: /usr/bin/bash -e {0}\r\n env:\r\n pythonLocation: /opt/hostedtoolcache/Python/3.11.4/x64\r\n PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.11.4/x64/lib/pkgconfig\r\n Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.4/x64\r\n Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.4/x64\r\n Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.4/x64\r\n LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.11.4/x64/lib\r\ndocs/writing_plugins.rst:365: code block parse error Cannot parse: 8:0: \r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1883055640, "label": "click-default-group>=1.2.3"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/590#issuecomment-1704387161", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/590", "id": 1704387161, "node_id": "IC_kwDOCGYnMM5lluJZ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-03T19:50:36Z", "updated_at": "2023-09-03T19:50:36Z", "author_association": "OWNER", "body": "Maybe just populate `db.memory: bool` and `db.memory_name: Optional[str]` for this, then document them.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1879214365, "label": "Ability to tell if a Database is an in-memory one"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/589#issuecomment-1704383901", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/589", "id": 1704383901, "node_id": "IC_kwDOCGYnMM5lltWd", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-03T19:34:05Z", "updated_at": "2023-09-03T19:34:05Z", "author_association": "OWNER", "body": "For that particular case I realized I'd quite like to have a mechanism for applying functions for a block of code and then de-registering them at the end - a context manager.\r\n\r\nI played with this idea a bit:\r\n\r\n```python\r\nwith db.register_functions(md5, md5_random):\r\n db.query(...)\r\n```\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1879209560, "label": "Mechanism for de-registering registered SQL functions"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/589#issuecomment-1704384111", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/589", "id": 1704384111, "node_id": "IC_kwDOCGYnMM5lltZv", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-03T19:35:03Z", "updated_at": "2023-09-03T19:35:03Z", "author_association": "OWNER", "body": "Normally in Python/`sqlite3` you de-register a function by passing `None` to it.\r\n\r\nYou can't do that with `db.register_function()` at the moment because a `fn` of `None` does something else:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/1260bdc7bfe31c36c272572c6389125f8de6ef71/sqlite_utils/db.py#L461-L464", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1879209560, "label": "Mechanism for de-registering registered SQL functions"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/589#issuecomment-1704384393", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/589", "id": 1704384393, "node_id": "IC_kwDOCGYnMM5llteJ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-03T19:36:34Z", "updated_at": "2023-09-03T19:36:34Z", "author_association": "OWNER", "body": "Here's a prototype: https://github.com/simonw/sqlite-utils/commit/62f673835c4a66f87cf6f949eaff43c8b014619b\r\n\r\nStill needs tests and documentation (and some more thought to make sure it's doing the right thing).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1879209560, "label": "Mechanism for de-registering registered SQL functions"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2168#issuecomment-1701823609", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2168", "id": 1701823609, "node_id": "IC_kwDOBm6k_c5lb8R5", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-08-31T21:43:06Z", "updated_at": "2023-08-31T21:44:13Z", "author_association": "OWNER", "body": "Not sure what to call this. Maybe `app_wrapper()`?\r\n\r\nOr perhaps it's simpler than that, something like this:\r\n\r\n```python\r\n@hookspec\r\ndef process_response(datasette, request, response):\r\n \"\"\"Last chance to modify the response before it is returned to the client\"\"\"\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1876353656, "label": "Consider a request/response wrapping hook slightly higher level than asgi_wrapper()"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2168#issuecomment-1701826521", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2168", "id": 1701826521, "node_id": "IC_kwDOBm6k_c5lb8_Z", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-08-31T21:46:13Z", "updated_at": "2023-08-31T21:46:13Z", "author_association": "OWNER", "body": "This could even be a pair of hooks - `process_request()` and `process_response()`.\r\n\r\nOr could take a leaf from Django, which redesigned middleware to use this pattern instead:\r\n\r\n```python\r\ndef simple_middleware(get_response):\r\n # One-time configuration and initialization.\r\n def middleware(request):\r\n # Code to be executed for each request before\r\n # the view (and later middleware) are called.\r\n response = get_response(request)\r\n # Code to be executed for each request/response after\r\n # the view is called.\r\n return response\r\n return middleware\r\n```\r\nOr even borrow an idea from `pytest` where fixtures can `yield` in the middle, like this:\r\n```python\r\n@pytest.fixture\r\ndef sending_user(mail_admin):\r\n user = mail_admin.create_user()\r\n yield user\r\n mail_admin.delete_user(user)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1876353656, "label": "Consider a request/response wrapping hook slightly higher level than asgi_wrapper()"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2168#issuecomment-1701828197", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2168", "id": 1701828197, "node_id": "IC_kwDOBm6k_c5lb9Zl", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-08-31T21:48:00Z", "updated_at": "2023-08-31T21:48:57Z", "author_association": "OWNER", "body": "A pattern like this could be interesting:\r\n```python\r\nasync def my_middleware(datasette, request, get_response):\r\n # Mess with request here if neccessary\r\n response = await get_response(request)\r\n # mess with response\r\n return response\r\n```\r\nThe Django pattern is more complicated but does have that mechanism for running one-time configuration prior to defining the `middleware()` function, which is neat.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1876353656, "label": "Consider a request/response wrapping hook slightly higher level than asgi_wrapper()"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2168#issuecomment-1701830241", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2168", "id": 1701830241, "node_id": "IC_kwDOBm6k_c5lb95h", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-08-31T21:50:18Z", "updated_at": "2023-08-31T21:50:18Z", "author_association": "OWNER", "body": "The hook could be called `register_middleware()` and could work like `register_routes()` and `register_commands()`:\r\n\r\n```python\r\n@hookspec\r\ndef register_middleware(datasette):\r\n \"\"\"Register middleware: returns a list of async def middleware functions\"\"\"\r\n\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1876353656, "label": "Consider a request/response wrapping hook slightly higher level than asgi_wrapper()"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2168#issuecomment-1701831013", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2168", "id": 1701831013, "node_id": "IC_kwDOBm6k_c5lb-Fl", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-08-31T21:51:12Z", "updated_at": "2023-08-31T21:52:15Z", "author_association": "OWNER", "body": "Need to make sure the design of this takes streaming responses into account. Those could be pretty tricky here.\r\n\r\nI nice thing about `asgi_wrapper()` is that it handles those already.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1876353656, "label": "Consider a request/response wrapping hook slightly higher level than asgi_wrapper()"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2168#issuecomment-1712897194", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2168", "id": 1712897194, "node_id": "IC_kwDOBm6k_c5mGLyq", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-10T17:54:07Z", "updated_at": "2023-09-10T17:54:07Z", "author_association": "OWNER", "body": "This looks relevant:\r\n\r\nhttps://pluggy.readthedocs.io/en/stable/#wrappers\r\n\r\n> A *hookimpl* can be marked with the `\"wrapper\"` option, which indicates that the function will be called to *wrap* (or surround) all other normal *hookimpl* calls. A *hook wrapper* can thus execute some code ahead and after the execution of all corresponding non-wrappper *hookimpls*.\r\n\r\nThis could be the perfect mechanism for implementing this hook, although I still need to figure out how it interacts with streaming.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1876353656, "label": "Consider a request/response wrapping hook slightly higher level than asgi_wrapper()"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2166#issuecomment-1701045404", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2166", "id": 1701045404, "node_id": "IC_kwDOBm6k_c5lY-Sc", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2023-08-31T13:31:15Z", "updated_at": "2023-09-06T13:25:17Z", "author_association": "NONE", "body": "## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2166?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\nPatch and project coverage have no change.\n> Comparison is base [(`05707aa`)](https://app.codecov.io/gh/simonw/datasette/commit/05707aa16b5c6c39fbe48b3176b85a8ffe493938?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68% compared to head [(`7c9df6e`)](https://app.codecov.io/gh/simonw/datasette/pull/2166?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68%.\n\n
Additional details and impacted files\n\n\n```diff\n@@ Coverage Diff @@\n## main #2166 +/- ##\n=======================================\n Coverage 92.68% 92.68% \n=======================================\n Files 40 40 \n Lines 6012 6012 \n=======================================\n Hits 5572 5572 \n Misses 440 440 \n```\n\n\n\n\n
\n\n[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2166?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). \n:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1875519316, "label": "Bump the python-packages group with 1 update"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2165#issuecomment-1699802028", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2165", "id": 1699802028, "node_id": "IC_kwDOBm6k_c5lUOus", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-08-30T20:35:02Z", "updated_at": "2023-08-30T20:35:02Z", "author_association": "OWNER", "body": "Testing this is going to be a bit of a pain.\r\n\r\nI think I'll add a whole separate test block to CI which installs a couple of plugins and then exercises this feature using `datasette plugins`.\r\n\r\nI'll use `datasette-init` and `datasette-json-html` just because they are small and simple.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1874327336, "label": "DATASETTE_LOAD_PLUGINS environment variable for loading specific plugins"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2165#issuecomment-1699809688", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2165", "id": 1699809688, "node_id": "IC_kwDOBm6k_c5lUQmY", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-08-30T20:41:06Z", "updated_at": "2023-08-30T20:41:06Z", "author_association": "OWNER", "body": "Slight weirdness: I noticed that the output from the `datasette plugins` command looks like this for plugins loaded with the new environment variable:\r\n```json\r\n {\r\n \"name\": \"datasette_pretty_json\",\r\n \"static\": false,\r\n \"templates\": false,\r\n \"version\": null,\r\n \"hooks\": [\r\n \"render_cell\"\r\n ]\r\n },\r\n```\r\nThat should ideally be `datasette-pretty-json`, not `datasette_pretty_json`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1874327336, "label": "DATASETTE_LOAD_PLUGINS environment variable for loading specific plugins"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2165#issuecomment-1699811810", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2165", "id": 1699811810, "node_id": "IC_kwDOBm6k_c5lURHi", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-08-30T20:42:41Z", "updated_at": "2023-08-30T20:42:41Z", "author_association": "OWNER", "body": "The `load_setuptools_entrypoints()` function in Pluggy [does this](https://github.com/pytest-dev/pluggy/blob/0b41c9766508a46ae666cf281684df3164b3e2a9/src/pluggy/_manager.py#L376):\r\n\r\n```python\r\n for ep in dist.entry_points:\r\n if (\r\n ep.group != group\r\n or (name is not None and ep.name != name)\r\n # already registered\r\n or self.get_plugin(ep.name)\r\n or self.is_blocked(ep.name)\r\n ):\r\n continue\r\n plugin = ep.load()\r\n self.register(plugin, name=ep.name)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1874327336, "label": "DATASETTE_LOAD_PLUGINS environment variable for loading specific plugins"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2165#issuecomment-1699812599", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2165", "id": 1699812599, "node_id": "IC_kwDOBm6k_c5lURT3", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2023-08-30T20:43:19Z", "updated_at": "2023-08-30T22:05:05Z", "author_association": "NONE", "body": "## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2165?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\nPatch coverage: **`26.66%`** and project coverage change: **`-0.17%`** :warning:\n> Comparison is base [(`30b28c8`)](https://app.codecov.io/gh/simonw/datasette/commit/30b28c8367a9c6870386ea10a202705b40862457?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.84% compared to head [(`5eddf34`)](https://app.codecov.io/gh/simonw/datasette/pull/2165?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68%.\n\n> :exclamation: Current head 5eddf34 differs from pull request most recent head 6321c9c. Consider uploading reports for the commit 6321c9c to get more accurate results\n\n
Additional details and impacted files\n\n\n```diff\n@@ Coverage Diff @@\n## main #2165 +/- ##\n==========================================\n- Coverage 92.84% 92.68% -0.17% \n==========================================\n Files 40 40 \n Lines 5998 6012 +14 \n==========================================\n+ Hits 5569 5572 +3 \n- Misses 429 440 +11 \n```\n\n\n| [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2165?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage \u0394 | |\n|---|---|---|\n| [datasette/plugins.py](https://app.codecov.io/gh/simonw/datasette/pull/2165?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3BsdWdpbnMucHk=) | `64.58% <26.66%> (-17.77%)` | :arrow_down: |\n\n\n
\n\n[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2165?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). \n:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1874327336, "label": "DATASETTE_LOAD_PLUGINS environment variable for loading specific plugins"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2165#issuecomment-1699884314", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2165", "id": 1699884314, "node_id": "IC_kwDOBm6k_c5lUi0a", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-08-30T21:38:29Z", "updated_at": "2023-08-30T21:39:51Z", "author_association": "OWNER", "body": "Here's the reason for that name disparity:\r\n\r\nhttps://github.com/simonw/datasette/blob/30b28c8367a9c6870386ea10a202705b40862457/datasette/plugins.py#L54-L65\r\n\r\nNote how the `distinfo.project_name` name is used when available. That seems to work for regularly installed plugins but not for plugins loaded via `DATASETTE_LOAD_PLUGINS`.\r\n\r\nAnd that's looking things up in `plugin_to_distinfo` which is populated here:\r\n\r\nhttps://github.com/simonw/datasette/blob/30b28c8367a9c6870386ea10a202705b40862457/datasette/plugins.py#L37", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1874327336, "label": "DATASETTE_LOAD_PLUGINS environment variable for loading specific plugins"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2165#issuecomment-1699910555", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2165", "id": 1699910555, "node_id": "IC_kwDOBm6k_c5lUpOb", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-08-30T22:05:14Z", "updated_at": "2023-08-30T22:05:14Z", "author_association": "OWNER", "body": "Documentation preview: https://github.com/simonw/datasette/blob/6321c9c055a640ed6ea98e231dc5813dcde1f773/docs/plugins.rst#controlling-which-plugins-are-loaded", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1874327336, "label": "DATASETTE_LOAD_PLUGINS environment variable for loading specific plugins"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2164#issuecomment-1699728102", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2164", "id": 1699728102, "node_id": "IC_kwDOBm6k_c5lT8rm", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-08-30T19:35:32Z", "updated_at": "2023-08-30T19:35:47Z", "author_association": "OWNER", "body": "Figured this out so far with the help of GPT-4: https://chat.openai.com/share/0e785865-621b-4fb3-ba05-7449e57c8496\r\n\r\nNow this works:\r\n```bash\r\nDATASETTE_LOAD_PLUGINS=datasette-write-ui datasette plugins\r\n```\r\n```json\r\n[\r\n {\r\n \"name\": \"datasette_write_ui\",\r\n \"static\": true,\r\n \"templates\": true,\r\n \"version\": null,\r\n \"hooks\": [\r\n \"extra_template_vars\",\r\n \"register_routes\"\r\n ]\r\n }\r\n]\r\n```\r\nOr multiple plugins:\r\n```bash\r\nDATASETTE_LOAD_PLUGINS=datasette-write-ui,datasette-pretty-json datasette plugins\r\n```\r\nOutputs:\r\n```json\r\n[\r\n {\r\n \"name\": \"datasette_pretty_json\",\r\n \"static\": false,\r\n \"templates\": false,\r\n \"version\": null,\r\n \"hooks\": [\r\n \"render_cell\"\r\n ]\r\n },\r\n {\r\n \"name\": \"datasette_write_ui\",\r\n \"static\": true,\r\n \"templates\": true,\r\n \"version\": null,\r\n \"hooks\": [\r\n \"extra_template_vars\",\r\n \"register_routes\"\r\n ]\r\n }\r\n]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1874255116, "label": "Ability to only load a specific list of plugins"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2163#issuecomment-1697818917", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2163", "id": 1697818917, "node_id": "IC_kwDOBm6k_c5lMqkl", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-08-29T16:56:20Z", "updated_at": "2023-08-29T16:56:20Z", "author_association": "OWNER", "body": "https://github.com/simonw/datasette/blob/50da908213a0fc405ecd7a40090dfea7a2e7395c/datasette/utils/internal_db.py#L8-L62", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1872043170, "label": "Rename core_X to catalog_X in the internals"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2162#issuecomment-1696594855", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2162", "id": 1696594855, "node_id": "IC_kwDOBm6k_c5lH_un", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2023-08-29T00:19:29Z", "updated_at": "2023-08-29T03:22:21Z", "author_association": "NONE", "body": "## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\nPatch coverage: **`100.00%`** and project coverage change: **`-0.03%`** :warning:\n> Comparison is base [(`2e28258`)](https://app.codecov.io/gh/simonw/datasette/commit/2e2825869fc2655b5fcadc743f6f9dec7a49bc65?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.82% compared to head [(`73489ca`)](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.79%.\n> Report is 1 commits behind head on main.\n\n
Additional details and impacted files\n\n\n```diff\n@@ Coverage Diff @@\n## main #2162 +/- ##\n==========================================\n- Coverage 92.82% 92.79% -0.03% \n==========================================\n Files 40 40 \n Lines 5948 5953 +5 \n==========================================\n+ Hits 5521 5524 +3 \n- Misses 427 429 +2 \n```\n\n\n| [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage \u0394 | |\n|---|---|---|\n| [datasette/default\\_permissions.py](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RlZmF1bHRfcGVybWlzc2lvbnMucHk=) | `96.87% <\u00f8> (-0.04%)` | :arrow_down: |\n| [datasette/app.py](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.32% <100.00%> (+0.03%)` | :arrow_up: |\n| [datasette/cli.py](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `82.53% <100.00%> (-0.06%)` | :arrow_down: |\n| [datasette/database.py](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RhdGFiYXNlLnB5) | `94.72% <100.00%> (+0.04%)` | :arrow_up: |\n| [datasette/utils/internal\\_db.py](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL2ludGVybmFsX2RiLnB5) | `100.00% <100.00%> (\u00f8)` | |\n| [datasette/views/database.py](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `95.45% <100.00%> (\u00f8)` | |\n| [datasette/views/special.py](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3NwZWNpYWwucHk=) | `94.06% <100.00%> (-0.85%)` | :arrow_down: |\n\n\n
\n\n[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). \n:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1870672704, "label": "Add new `--internal internal.db` option, deprecate legacy `_internal` database"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2162#issuecomment-1696707458", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2162", "id": 1696707458, "node_id": "IC_kwDOBm6k_c5lIbOC", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-08-29T03:17:54Z", "updated_at": "2023-08-29T03:17:54Z", "author_association": "OWNER", "body": "Documentation preview: https://datasette--2162.org.readthedocs.build/en/2162/internals.html#datasette-s-internal-database", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1870672704, "label": "Add new `--internal internal.db` option, deprecate legacy `_internal` database"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2162#issuecomment-1696709110", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2162", "id": 1696709110, "node_id": "IC_kwDOBm6k_c5lIbn2", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-08-29T03:20:40Z", "updated_at": "2023-08-29T03:22:47Z", "author_association": "OWNER", "body": "> However, one important notes about those new `core_` tables: If a `--internal` DB is passed in, that means those `core_` tables will persist across multiple Datasette instances. This wasn't the case before, since `_internal` was always an in-memory database created from scratch.\r\n\r\nI'm completely happy for the `core_*` tables (or `datasette_*` or some other name) to live in the persisted-to-disk `internal.db` database, even though they're effectively meant to be an in-memory cache.\r\n\r\nI don't think it causes any harm, and it could even be quite useful to have them visible on disk - other applications could read the `internal.db` database while Datasette itself is running, should they have some weird reason to want to do that!\r\n\r\nHaving those tables stick around in `internal.db` after Datasette shuts down could be useful for other debugging activities as well.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1870672704, "label": "Add new `--internal internal.db` option, deprecate legacy `_internal` database"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2162#issuecomment-1696710911", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2162", "id": 1696710911, "node_id": "IC_kwDOBm6k_c5lIcD_", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-08-29T03:23:44Z", "updated_at": "2023-08-29T03:23:44Z", "author_association": "OWNER", "body": "I'm going to merge this so we can see how it feels.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1870672704, "label": "Add new `--internal internal.db` option, deprecate legacy `_internal` database"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2161#issuecomment-1696267473", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2161", "id": 1696267473, "node_id": "IC_kwDOBm6k_c5lGvzR", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2023-08-28T19:38:28Z", "updated_at": "2023-08-28T19:39:33Z", "author_association": "NONE", "body": "## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2161?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\nPatch coverage: **`75.00%`** and project coverage change: **`+0.05%`** :tada:\n> Comparison is base [(`527cec6`)](https://app.codecov.io/gh/simonw/datasette/commit/527cec66b0403e689c8fb71fc8b381a1d7a46516?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.76% compared to head [(`a5cbf80`)](https://app.codecov.io/gh/simonw/datasette/pull/2161?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.81%.\n\n
Additional details and impacted files\n\n\n```diff\n@@ Coverage Diff @@\n## main #2161 +/- ##\n==========================================\n+ Coverage 92.76% 92.81% +0.05% \n==========================================\n Files 40 40 \n Lines 5943 5944 +1 \n==========================================\n+ Hits 5513 5517 +4 \n+ Misses 430 427 -3 \n```\n\n\n| [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2161?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage \u0394 | |\n|---|---|---|\n| [datasette/cli.py](https://app.codecov.io/gh/simonw/datasette/pull/2161?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `82.37% <75.00%> (+1.02%)` | :arrow_up: |\n\n\n
\n\n[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2161?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). \n:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1870345352, "label": "-s/--setting x y gets merged into datasette.yml, refs #2143, #2156"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2160#issuecomment-1695754277", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2160", "id": 1695754277, "node_id": "IC_kwDOBm6k_c5lEygl", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2023-08-28T14:00:17Z", "updated_at": "2023-08-29T00:32:52Z", "author_association": "NONE", "body": "## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2160?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\nPatch and project coverage have no change.\n> Comparison is base [(`2e28258`)](https://app.codecov.io/gh/simonw/datasette/commit/2e2825869fc2655b5fcadc743f6f9dec7a49bc65?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.82% compared to head [(`c81b024`)](https://app.codecov.io/gh/simonw/datasette/pull/2160?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.82%.\n\n> :exclamation: Current head c81b024 differs from pull request most recent head 3529c9c. Consider uploading reports for the commit 3529c9c to get more accurate results\n\n
Additional details and impacted files\n\n\n```diff\n@@ Coverage Diff @@\n## main #2160 +/- ##\n=======================================\n Coverage 92.82% 92.82% \n=======================================\n Files 40 40 \n Lines 5948 5948 \n=======================================\n Hits 5521 5521 \n Misses 427 427 \n```\n\n\n\n\n
\n\n[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2160?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). \n:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1869807874, "label": "Bump sphinx, furo, blacken-docs dependencies"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2160#issuecomment-1696586213", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2160", "id": 1696586213, "node_id": "IC_kwDOBm6k_c5lH9nl", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-08-29T00:07:46Z", "updated_at": "2023-08-29T00:07:46Z", "author_association": "OWNER", "body": "I figured out why this was failing in:\r\n- https://github.com/simonw/datasette/pull/2148\r\n\r\nIt's because Sphinx dropped support for Python 3.8.\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1869807874, "label": "Bump sphinx, furo, blacken-docs dependencies"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2160#issuecomment-1696586767", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2160", "id": 1696586767, "node_id": "IC_kwDOBm6k_c5lH9wP", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-08-29T00:08:26Z", "updated_at": "2023-08-29T00:08:35Z", "author_association": "OWNER", "body": "https://github.com/simonw/datasette/issues/2148#issuecomment-1689177556\r\n\r\n> Simplest possible solution is to only run the `pip install .[docs]` bit under Python 3.9+, ditto for the docs tests. I think I'll try that.\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1869807874, "label": "Bump sphinx, furo, blacken-docs dependencies"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2160#issuecomment-1696592763", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2160", "id": 1696592763, "node_id": "IC_kwDOBm6k_c5lH_N7", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-08-29T00:16:38Z", "updated_at": "2023-08-29T00:16:38Z", "author_association": "OWNER", "body": "Since this bumps Sphinx I'm manually reviewing the ReadTheDocs preview a bit, looks good to me: https://datasette--2160.org.readthedocs.build/en/2160/", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1869807874, "label": "Bump sphinx, furo, blacken-docs dependencies"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2160#issuecomment-1696595326", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2160", "id": 1696595326, "node_id": "IC_kwDOBm6k_c5lH_1-", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-08-29T00:20:08Z", "updated_at": "2023-08-29T00:20:08Z", "author_association": "OWNER", "body": "Cog failed!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1869807874, "label": "Bump sphinx, furo, blacken-docs dependencies"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/588#issuecomment-1694823972", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/588", "id": 1694823972, "node_id": "IC_kwDOCGYnMM5lBPYk", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-08-28T00:41:54Z", "updated_at": "2023-08-28T00:41:54Z", "author_association": "OWNER", "body": "Tips on typing `**kwargs`: https://adamj.eu/tech/2021/05/11/python-type-hints-args-and-kwargs/", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1868713944, "label": "`table.get(column=value)` option for retrieving things not by their primary key"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2157#issuecomment-1692465334", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2157", "id": 1692465334, "node_id": "IC_kwDOBm6k_c5k4Pi2", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-08-24T21:54:09Z", "updated_at": "2023-08-24T21:54:09Z", "author_association": "OWNER", "body": "We discussed this in-person this morning and these notes reflect what we talked about perfectly.\r\n\r\nI've had so many bugs with plugins that I've written myself that have forgotten to special-case the `_internal` database when looping through `datasette.databases.keys()` - removing it from there entirely would help a lot.\r\n\r\nJust one tiny disagreement: for `datasette-comments` I think having it store things in `_internal` could be an option, but in most cases I expect users to chose NOT to do that - because being able to join against those tables for more advanced queries is going to be super useful.\r\n\r\nShow me all rows in `foia_requests` with at least one associated comment in `datasette_comments.comments` kind of tihng.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1865869205, "label": "Proposal: Make the `_internal` database persistent, customizable, and hidden"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2157#issuecomment-1692465763", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2157", "id": 1692465763, "node_id": "IC_kwDOBm6k_c5k4Ppj", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-08-24T21:54:29Z", "updated_at": "2023-08-24T21:54:29Z", "author_association": "OWNER", "body": "But yes, I'm a big +1 on this whole plan.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1865869205, "label": "Proposal: Make the `_internal` database persistent, customizable, and hidden"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2157#issuecomment-1700291967", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2157", "id": 1700291967, "node_id": "IC_kwDOBm6k_c5lWGV_", "user": {"value": 15178711, "label": "asg017"}, "created_at": "2023-08-31T02:45:56Z", "updated_at": "2023-08-31T02:45:56Z", "author_association": "CONTRIBUTOR", "body": "@simonw what do you think about adding a `DATASETTE_INTERNAL_DB_PATH` env variable, where when defined, is the default location of the internal DB? This means when the `--internal` flag is NOT provided, Datasette would check to see if `DATASETTE_INTERNAL_DB_PATH` exists, and if so, uses that as the internal database (and would fallback to an ephemeral memory database)\r\n\r\nMy rationale: some plugins may require, or strongly encourage, a persistent internal database (`datasette-comments`, `datasette-bookmarks`, `datasette-link-shortener`, etc.). However, for users that have a global installation of Datasette (say from `brew install` or a global `pip install`), it would be annoying having to specify `--internal` every time. So instead, they can just add `export DATASETTE_INTERNAL_DB_PATH=\"/path/to/internal.db\"` to their bashrc/zshrc/whereever to not have to worry about `--internal`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1865869205, "label": "Proposal: Make the `_internal` database persistent, customizable, and hidden"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2156#issuecomment-1692186522", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2156", "id": 1692186522, "node_id": "IC_kwDOBm6k_c5k3Lea", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-08-24T18:10:04Z", "updated_at": "2023-08-24T18:10:04Z", "author_association": "OWNER", "body": "I have an implementation in https://github.com/simonw/datasette/issues/2143#issuecomment-1690792514 too - I'm going to land that as a PR.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1865649347, "label": "datasette -s/--setting option for setting nested configuration options"}, "performed_via_github_app": null}