{"html_url": "https://github.com/simonw/sqlite-utils/pull/604#issuecomment-1843975536", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/604", "id": 1843975536, "node_id": "IC_kwDOCGYnMM5t6NVw", "user": {"value": 16437338, "label": "tkhattra"}, "created_at": "2023-12-07T01:17:05Z", "updated_at": "2023-12-07T01:17:05Z", "author_association": "CONTRIBUTOR", "body": "Apologies - I pushed a fix that addresses the mypy failures.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 2001006157, "label": "Add more STRICT table support"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2213#issuecomment-1843072926", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2213", "id": 1843072926, "node_id": "IC_kwDOBm6k_c5t2w-e", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2023-12-06T15:05:44Z", "updated_at": "2023-12-06T15:05:44Z", "author_association": "CONTRIBUTOR", "body": "it probably does not make sense to gzip large sqlite database files on the fly. it can take many seconds to gzip a large file and you either have to have this big thing in memory, or write it to disk, which some deployment environments will not like.\r\n\r\ni wonder if it would make sense to gzip the databases as part of the datasette publish process. it would be very cool to statically serve those as if they dynamically zipped (i.e. serve the filename example.db, not example.db.zip, and rely on the browser to expand).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 2028698018, "label": "feature request: gzip compression of database downloads"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/344#issuecomment-1815825863", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/344", "id": 1815825863, "node_id": "IC_kwDOCGYnMM5sO03H", "user": {"value": 16437338, "label": "tkhattra"}, "created_at": "2023-11-17T06:44:49Z", "updated_at": "2023-11-17T06:44:49Z", "author_association": "CONTRIBUTOR", "body": "hello Simon,\r\n\r\nI've added more STRICT table support per https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982014776 in changeset https://github.com/simonw/sqlite-utils/commit/e4b9b582cdb4e48430865f8739f341bc8017c1e4.\r\nIt also fixes table.transform() to preserve STRICT mode.\r\nPlease pull if you deem appropriate. Thanks!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066474200, "label": "Support STRICT tables"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2209#issuecomment-1812750369", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2209", "id": 1812750369, "node_id": "IC_kwDOBm6k_c5sDGAh", "user": {"value": 198537, "label": "rgieseke"}, "created_at": "2023-11-15T15:29:37Z", "updated_at": "2023-11-15T15:29:37Z", "author_association": "CONTRIBUTOR", "body": "Looks like tests are passing now but there is an issue with yaml loading and/or cog.\r\n\r\nhttps://github.com/simonw/datasette/actions/runs/6879299298/job/18710911166?pr=2209", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1994861266, "label": "Fix query for suggested facets with column named value"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2209#issuecomment-1812623778", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2209", "id": 1812623778, "node_id": "IC_kwDOBm6k_c5sCnGi", "user": {"value": 198537, "label": "rgieseke"}, "created_at": "2023-11-15T14:22:42Z", "updated_at": "2023-11-15T15:24:09Z", "author_association": "CONTRIBUTOR", "body": "Whoops, looks like I forgot to check for other places where the 'facetable' table is used in the tests.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1994861266, "label": "Fix query for suggested facets with column named value"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2208#issuecomment-1812617851", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2208", "id": 1812617851, "node_id": "IC_kwDOBm6k_c5sClp7", "user": {"value": 198537, "label": "rgieseke"}, "created_at": "2023-11-15T14:18:58Z", "updated_at": "2023-11-15T14:18:58Z", "author_association": "CONTRIBUTOR", "body": "Without aliases:\r\n\r\n![image](https://github.com/simonw/datasette/assets/198537/d9703d3b-9733-4e87-9954-4fc60a07784a)\r\n\r\nThe proposed fix in #2209 also works when the 'value' column is actually facetable (just added another value in the 'value' column).\r\n\r\n![image](https://github.com/simonw/datasette/assets/198537/a37a0a1a-c36a-4c78-bdce-01b582637cc6)\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1994857251, "label": "No suggested facets when a column named 'value' is included"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2202#issuecomment-1801876943", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2202", "id": 1801876943, "node_id": "IC_kwDOBm6k_c5rZnXP", "user": {"value": 49699333, "label": "dependabot[bot]"}, "created_at": "2023-11-08T13:19:00Z", "updated_at": "2023-11-08T13:19:00Z", "author_association": "CONTRIBUTOR", "body": "Superseded by #2206.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1959278971, "label": "Bump the python-packages group with 1 update"}, "performed_via_github_app": "{\"id\": 29110, \"slug\": \"dependabot\", \"node_id\": \"MDM6QXBwMjkxMTA=\", \"owner\": {\"login\": \"github\", \"id\": 9919, \"node_id\": \"MDEyOk9yZ2FuaXphdGlvbjk5MTk=\", \"avatar_url\": \"https://avatars.githubusercontent.com/u/9919?v=4\", \"gravatar_id\": \"\", \"url\": \"https://api.github.com/users/github\", \"html_url\": \"https://github.com/github\", \"followers_url\": \"https://api.github.com/users/github/followers\", \"following_url\": \"https://api.github.com/users/github/following{/other_user}\", \"gists_url\": \"https://api.github.com/users/github/gists{/gist_id}\", \"starred_url\": \"https://api.github.com/users/github/starred{/owner}{/repo}\", \"subscriptions_url\": \"https://api.github.com/users/github/subscriptions\", \"organizations_url\": \"https://api.github.com/users/github/orgs\", \"repos_url\": \"https://api.github.com/users/github/repos\", \"events_url\": \"https://api.github.com/users/github/events{/privacy}\", \"received_events_url\": \"https://api.github.com/users/github/received_events\", \"type\": \"Organization\", \"site_admin\": false}, \"name\": \"Dependabot\", \"description\": \"\", \"external_url\": \"https://dependabot-api.githubapp.com\", \"html_url\": \"https://github.com/apps/dependabot\", \"created_at\": \"2019-04-16T22:34:25Z\", \"updated_at\": \"2023-10-12T13:35:09Z\", \"permissions\": {\"checks\": \"write\", \"contents\": \"write\", \"issues\": \"write\", \"members\": \"read\", \"metadata\": \"read\", \"pull_requests\": \"write\", \"statuses\": \"read\", \"vulnerability_alerts\": \"read\", \"workflows\": \"write\"}, \"events\": [\"check_suite\", \"issues\", \"issue_comment\", \"label\", \"pull_request\", \"pull_request_review\", \"pull_request_review_comment\", \"repository\"]}"} {"html_url": "https://github.com/simonw/datasette/pull/2200#issuecomment-1777228352", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2200", "id": 1777228352, "node_id": "IC_kwDOBm6k_c5p7lpA", "user": {"value": 49699333, "label": "dependabot[bot]"}, "created_at": "2023-10-24T13:40:25Z", "updated_at": "2023-10-24T13:40:25Z", "author_association": "CONTRIBUTOR", "body": "Superseded by #2202.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1949756141, "label": "Bump the python-packages group with 1 update"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1655#issuecomment-1767219901", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1655", "id": 1767219901, "node_id": "IC_kwDOBm6k_c5pVaK9", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2023-10-17T21:29:03Z", "updated_at": "2023-10-17T21:29:03Z", "author_association": "CONTRIBUTOR", "body": "@yejiyang why don\u2019t you move this discussion to my fork to spare simon\u2019s notifications ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1163369515, "label": "query result page is using 400mb of browser memory 40x size of html page and 400x size of csv data"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1655#issuecomment-1766994810", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1655", "id": 1766994810, "node_id": "IC_kwDOBm6k_c5pUjN6", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2023-10-17T19:01:59Z", "updated_at": "2023-10-17T19:01:59Z", "author_association": "CONTRIBUTOR", "body": "hi @yejiyang, have your tried using my fork of datasette: https://github.com/fgregg/datasette/tree/no_limit_csv_publish\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1163369515, "label": "query result page is using 400mb of browser memory 40x size of html page and 400x size of csv data"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2199#issuecomment-1760401731", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2199", "id": 1760401731, "node_id": "IC_kwDOBm6k_c5o7ZlD", "user": {"value": 15178711, "label": "asg017"}, "created_at": "2023-10-12T21:41:42Z", "updated_at": "2023-10-12T21:41:42Z", "author_association": "CONTRIBUTOR", "body": "I dig it - I was thinking an Observable notebook where you paste your `metadata.json`/`metadata.yaml` and it would generate the new metadata + datasette.yaml files, but an extensible `datasette upgrade` plugin would be nice for future plugins.\r\n\r\nOne thing to think about: If someone has comments in their original `metadata.yaml`, could we preserve them in the new files? tbh maybe not too important bc if people cared that much they could just copy + paste, and it might be too distracting \r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1940346034, "label": "Detailed upgrade instructions for metadata.yaml -> datasette.yaml"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/433#issuecomment-1747231893", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/433", "id": 1747231893, "node_id": "IC_kwDOCGYnMM5oJKSV", "user": {"value": 62745, "label": "spookylukey"}, "created_at": "2023-10-04T16:15:09Z", "updated_at": "2023-10-04T16:28:21Z", "author_association": "CONTRIBUTOR", "body": "I confirm the bug, as above, and that @jonafato 's patch fixes it for me. However, it's not the right fix. The problem is that ProgressBar is being used in the wrong way. This also results in two lines being printed instead of one, like this:\r\n\r\n```\r\n [#######-----------------------------] 20%\r\n [####################################] 100%% \r\n```\r\nThe bug is reproducible for me in any terminal, including Gnome Terminal and Guake, and VSCode. With VSCode I can use this launch.json to reproduce it:\r\n\r\n```json\r\n\r\n{\r\n \"version\": \"0.2.0\",\r\n \"configurations\": [\r\n {\r\n \"name\": \"Python: Module\",\r\n \"type\": \"python\",\r\n \"request\": \"launch\",\r\n \"module\": \"sqlite_utils\",\r\n \"justMyCode\": false,\r\n \"args\": [\"insert\", \"test.db\", \"test\", \"--csv\", \"tests/sniff/example1.csv\"]\r\n }\r\n ]\r\n}\r\n```\r\n\r\n[edit - deleted my analysis of why the current code is wrong, which was confused and confusing]", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1239034903, "label": "CLI eats my cursor"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2155#issuecomment-1737363182", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2155", "id": 1737363182, "node_id": "IC_kwDOBm6k_c5njg7u", "user": {"value": 418191, "label": "jaywgraves"}, "created_at": "2023-09-27T13:05:41Z", "updated_at": "2023-09-27T13:05:41Z", "author_association": "CONTRIBUTOR", "body": "I'm hitting the #2123 issue and I just patched my local version with this and it seems to work fine.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1865572575, "label": "Fix hupper.start_reloader entry point"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2190#issuecomment-1729961503", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2190", "id": 1729961503, "node_id": "IC_kwDOBm6k_c5nHR4f", "user": {"value": 15178711, "label": "asg017"}, "created_at": "2023-09-21T16:56:57Z", "updated_at": "2023-09-21T16:56:57Z", "author_association": "CONTRIBUTOR", "body": "TODO: add similar checks for permissions/allow/canned queries", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1901483874, "label": "Raise an exception if a \"plugins\" block exists in metadata.json"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2188#issuecomment-1722848454", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2188", "id": 1722848454, "node_id": "IC_kwDOBm6k_c5msJTG", "user": {"value": 15178711, "label": "asg017"}, "created_at": "2023-09-18T06:58:53Z", "updated_at": "2023-09-18T06:58:53Z", "author_association": "CONTRIBUTOR", "body": "Thinking about this more, here a list of things I imagine a \"compile-to-sql\" plugin would want to do:\r\n\r\n1. Attach itself to the SQL code editor (switch from SQL -> PRQL/Logica, additional syntax highlighting)\r\n2. Add \"Query using PRQL\" buttons in various parts of Datasette's UI, like `/dbname` page\r\n3. Use `$LANGUAGE=` instead of `sql=` in the JSON API and the SQL results pages\r\n4. Have their own dedicated code editor page\r\n\r\n\r\n1) and 2) would be difficult to do with current plugin hooks, unless we add the concept of \"slots\" and get the JS plugin support in. 3) could maybe be done with the [`asgi_wrapper(datasette)`](https://docs.datasette.io/en/stable/plugin_hooks.html#asgi-wrapper-datasette) hook? And 4) ca n be done easily with the `register_routes()` hooks. \r\n\r\nSo it really only sounds like extending the SQL editor will be the hard part. In #2094 I want to add JavaScript plugin hooks for extending the SQL editor, which may work here. \r\n\r\nIf I get the time/motivation, I might try out a `datasette-prql` extension, just because I like playing with it. It'd be really cool if I can get the `asgi_wrapper()` hook to work right there...", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1900026059, "label": "Plugin Hooks for \"compile to SQL\" languages"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1191#issuecomment-1722845490", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1191", "id": 1722845490, "node_id": "IC_kwDOBm6k_c5msIky", "user": {"value": 15178711, "label": "asg017"}, "created_at": "2023-09-18T06:55:52Z", "updated_at": "2023-09-18T06:55:52Z", "author_association": "CONTRIBUTOR", "body": "One note here: this feature could be called \"slots\", similar to [Layout Slots](https://vitepress.dev/guide/extending-default-theme#layout-slots) in Vitepress.\r\n\r\nIn Vitepress, you can add custom components/widget/gadgets into determined named \"slots\", like so:\r\n\r\n```\r\ndoc-top\r\ndoc-bottom\r\ndoc-footer-before\r\ndoc-before\r\ndoc-after\r\n...\r\n```\r\n\r\nWould be great to do in both Python and Javascript, with the upcoming JavaScript API #2052. In `datasette-write-ui`, all we do is add a few \"Insert row\" and \"edit this row\" buttons and that required completely capturing the `table.html` template, which isn't great for other plugins. But having \"slots\" like `table-footer-before` or `table-row-id` or something would be great to work with.\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 787098345, "label": "Ability for plugins to collaborate when adding extra HTML to blocks in default templates"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2182#issuecomment-1719451803", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2182", "id": 1719451803, "node_id": "IC_kwDOBm6k_c5mfMCb", "user": {"value": 49699333, "label": "dependabot[bot]"}, "created_at": "2023-09-14T13:27:26Z", "updated_at": "2023-09-14T13:27:26Z", "author_association": "CONTRIBUTOR", "body": "Looks like these dependencies are updatable in another way, so this is no longer needed.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1890593563, "label": "Bump the python-packages group with 2 updates"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2183#issuecomment-1716801971", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2183", "id": 1716801971, "node_id": "IC_kwDOBm6k_c5mVFGz", "user": {"value": 15178711, "label": "asg017"}, "created_at": "2023-09-13T01:34:01Z", "updated_at": "2023-09-13T01:34:01Z", "author_association": "CONTRIBUTOR", "body": "@simonw docs are finished, this is ready for review!\r\n\r\nOne thing: I added \"Configuration\" as a top-level item in the documentation site, at the very bottom. Not sure if this is the best, maybe it can be named \"datasette.yaml Configuration\" or something similar?\r\n\r\nMostly because \"Configuration\" by itself can mean many things, but adding \"datasette.yaml\" would make it pretty clear it's about that specific file, and is easier to scan. I'd also be fine with using \"datasette.yaml\" instead of \"datasette.json\", since writing in YAML is much more forgiving (and advanced users will know JSON is also supported)\r\n\r\nAlso, maybe this is a chance to consolidate the docs a bit? I think \"Settings\", \"Configuration\", \"Metadata\", and \"Authentication and permissions\" should possibly be under the same section. Maybe even consolidate the different Plugin pages that exist?\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1891212159, "label": "`datasette.yaml` plugin support"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2157#issuecomment-1700291967", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2157", "id": 1700291967, "node_id": "IC_kwDOBm6k_c5lWGV_", "user": {"value": 15178711, "label": "asg017"}, "created_at": "2023-08-31T02:45:56Z", "updated_at": "2023-08-31T02:45:56Z", "author_association": "CONTRIBUTOR", "body": "@simonw what do you think about adding a `DATASETTE_INTERNAL_DB_PATH` env variable, where when defined, is the default location of the internal DB? This means when the `--internal` flag is NOT provided, Datasette would check to see if `DATASETTE_INTERNAL_DB_PATH` exists, and if so, uses that as the internal database (and would fallback to an ephemeral memory database)\r\n\r\nMy rationale: some plugins may require, or strongly encourage, a persistent internal database (`datasette-comments`, `datasette-bookmarks`, `datasette-link-shortener`, etc.). However, for users that have a global installation of Datasette (say from `brew install` or a global `pip install`), it would be annoying having to specify `--internal` every time. So instead, they can just add `export DATASETTE_INTERNAL_DB_PATH=\"/path/to/internal.db\"` to their bashrc/zshrc/whereever to not have to worry about `--internal`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1865869205, "label": "Proposal: Make the `_internal` database persistent, customizable, and hidden"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2148#issuecomment-1696591957", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2148", "id": 1696591957, "node_id": "IC_kwDOBm6k_c5lH_BV", "user": {"value": 49699333, "label": "dependabot[bot]"}, "created_at": "2023-08-29T00:15:29Z", "updated_at": "2023-08-29T00:15:29Z", "author_association": "CONTRIBUTOR", "body": "This pull request was built based on a group rule. Closing it will not ignore any of these versions in future pull requests.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1859415334, "label": "Bump sphinx, furo, blacken-docs dependencies"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2152#issuecomment-1695736691", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2152", "id": 1695736691, "node_id": "IC_kwDOBm6k_c5lEuNz", "user": {"value": 49699333, "label": "dependabot[bot]"}, "created_at": "2023-08-28T13:49:35Z", "updated_at": "2023-08-28T13:49:35Z", "author_association": "CONTRIBUTOR", "body": "Superseded by #2160.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1865174661, "label": "Bump the python-packages group with 3 updates"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2148#issuecomment-1689198413", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2148", "id": 1689198413, "node_id": "IC_kwDOBm6k_c5krx9N", "user": {"value": 49699333, "label": "dependabot[bot]"}, "created_at": "2023-08-23T02:57:55Z", "updated_at": "2023-08-23T02:57:55Z", "author_association": "CONTRIBUTOR", "body": "Looks like this PR has been edited by someone other than Dependabot. That means Dependabot can't rebase it - sorry!\n\nIf you're happy for Dependabot to recreate it from scratch, overwriting any edits, you can request `@dependabot recreate`.\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1859415334, "label": "Bump sphinx, furo, blacken-docs dependencies"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2093#issuecomment-1688532012", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2093", "id": 1688532012, "node_id": "IC_kwDOBm6k_c5kpPQs", "user": {"value": 15178711, "label": "asg017"}, "created_at": "2023-08-22T16:21:40Z", "updated_at": "2023-08-22T16:21:40Z", "author_association": "CONTRIBUTOR", "body": "OK Here's the gameplan for this, which is closely tied to #2143 :\r\n\r\n- We will add a new `datasette.json`/`datasette.yaml` configuration file to datasette, which combines settings/plugin config/permissions/canned queries into a new file format\r\n- Metadata will NOT be a part of this file\r\n- TOML support is not planned, but maybe we can create a separate issue for support TOML with JSON/YAML\r\n- The `settings.json` file will be deprecated, and the `--config` arg will be brought back.\r\n- Command line arguments can still be used to overwrite values (ex `--setting` will overwrite settings in `datasette.yaml`\r\n\r\nThe format of `datasette.json` will follow what Simon listed here: https://github.com/simonw/datasette/issues/2143#issuecomment-1684484426\r\n\r\nHere's the current implementation plan:\r\n\r\n1. Add a new `--config` flag and port over `\"settings\"` into a new datasette.json config file, remove settings.json\r\n2. Add top-level plugin config support to `datasette.json`\r\n3. Figure out database/table structure of config `datasette.json`\r\n4. Port over database/table level plugin config support `datasette.json`\r\n5. Port over permissions/auth settings to `datasette.json`\r\n6. Deprecate non-metadata values in `metadata.json`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1781530343, "label": "Proposal: Combine settings, metadata, static, etc. into a single `datasette.yaml` File"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2145#issuecomment-1686745094", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2145", "id": 1686745094, "node_id": "IC_kwDOBm6k_c5kibAG", "user": {"value": 15178711, "label": "asg017"}, "created_at": "2023-08-21T17:30:01Z", "updated_at": "2023-08-21T17:30:01Z", "author_association": "CONTRIBUTOR", "body": "Another point: The new Datasette write API should refuse to insert a row with a NULL primary key. That will likely decrease the likelihood someone find themselves with NULLs in their primary keys, at least with Datasette users. Especially buggy code that uses the write API, like our `datasette-write-ui` bug that led to this issue.\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1857234285, "label": "If a row has a primary key of `null` various things break"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2144#issuecomment-1686366557", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2144", "id": 1686366557, "node_id": "IC_kwDOBm6k_c5kg-ld", "user": {"value": 49699333, "label": "dependabot[bot]"}, "created_at": "2023-08-21T13:48:15Z", "updated_at": "2023-08-21T13:48:15Z", "author_association": "CONTRIBUTOR", "body": "Superseded by #2148.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1856760386, "label": "Bump the python-packages group with 3 updates"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2143#issuecomment-1684496274", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2143", "id": 1684496274, "node_id": "IC_kwDOBm6k_c5kZ1-S", "user": {"value": 15178711, "label": "asg017"}, "created_at": "2023-08-18T22:30:45Z", "updated_at": "2023-08-18T22:30:45Z", "author_association": "CONTRIBUTOR", "body": "> That said, I do really like a bias towards settings that can be changed at runtime\r\n\r\nDoes this include things like `--settings` values or plugin config? I can totally see being able to update metadata without restarting, but not sure if that would work well with `--setting`, plugin config, or auth/permissions stuff. \r\n\r\nWell it could work with `--setting` and auth/permissions, with a lot of core changes. But changing plugin config on the fly could be challenging, for plugin authors. ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1855885427, "label": "De-tangling Metadata before Datasette 1.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2143#issuecomment-1684205563", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2143", "id": 1684205563, "node_id": "IC_kwDOBm6k_c5kYu_7", "user": {"value": 15178711, "label": "asg017"}, "created_at": "2023-08-18T17:12:54Z", "updated_at": "2023-08-18T17:12:54Z", "author_association": "CONTRIBUTOR", "body": "Another option would be, instead of flat `datasette.json`/`datasette.yaml` files, we could instead use a Python file, like `datasette_config.py`. That way one could dynamically generate config (ex dev vs prod, auto-discover credentials, etc.). Kinda like Django settings.\r\n\r\nThough I imagine Python imports might make this complex to do, and json/yaml is already supported and pretty easy to write\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1855885427, "label": "De-tangling Metadata before Datasette 1.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2143#issuecomment-1684202932", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2143", "id": 1684202932, "node_id": "IC_kwDOBm6k_c5kYuW0", "user": {"value": 15178711, "label": "asg017"}, "created_at": "2023-08-18T17:10:21Z", "updated_at": "2023-08-18T17:10:21Z", "author_association": "CONTRIBUTOR", "body": "I agree with all your points!\r\n\r\nI think the best solution would be having a `datasette.json` config file, where you \"configure\" your datasette instances, with settings, permissions/auth, plugin configuration, and table settings (sortable column, label columns, etc.). Which #2093 would do.\r\n\r\nThen optionally, you have a `metadata.json`, or use `datasette_metadata`, or some other plugin to define metadata (ex the future [sqlite-docs](https://github.com/asg017/sqlite-docs) plugin).\r\n\r\nEverything in `datasette.json` could also be overwritten by CLI flags, like `--setting key value`, `--plugin xxxx key value`.\r\n\r\nWe could even completely remove `settings.json` in favor or just `datasette.json`. Mostly because I think the less files the better, especially if they have generic names like `settings.json` or `config.json`. ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1855885427, "label": "De-tangling Metadata before Datasette 1.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2142#issuecomment-1683950031", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2142", "id": 1683950031, "node_id": "IC_kwDOBm6k_c5kXwnP", "user": {"value": 49699333, "label": "dependabot[bot]"}, "created_at": "2023-08-18T13:49:24Z", "updated_at": "2023-08-18T13:49:24Z", "author_association": "CONTRIBUTOR", "body": "Looks like these dependencies are updatable in another way, so this is no longer needed.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1854970601, "label": "Bump the python-packages group with 2 updates"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2141#issuecomment-1682256251", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2141", "id": 1682256251, "node_id": "IC_kwDOBm6k_c5kRTF7", "user": {"value": 49699333, "label": "dependabot[bot]"}, "created_at": "2023-08-17T13:07:43Z", "updated_at": "2023-08-17T13:07:43Z", "author_association": "CONTRIBUTOR", "body": "Looks like blacken-docs is updatable in another way, so this is no longer needed.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1853289039, "label": "Bump the python-packages group with 1 update"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2125#issuecomment-1668187546", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2125", "id": 1668187546, "node_id": "IC_kwDOBm6k_c5jboWa", "user": {"value": 49699333, "label": "dependabot[bot]"}, "created_at": "2023-08-07T16:20:26Z", "updated_at": "2023-08-07T16:20:26Z", "author_association": "CONTRIBUTOR", "body": "Looks like sphinx is up-to-date now, so this is no longer needed.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1833193570, "label": "Bump sphinx from 6.1.3 to 7.1.2"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2121#issuecomment-1668186872", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2121", "id": 1668186872, "node_id": "IC_kwDOBm6k_c5jboL4", "user": {"value": 49699333, "label": "dependabot[bot]"}, "created_at": "2023-08-07T16:20:19Z", "updated_at": "2023-08-07T16:20:19Z", "author_association": "CONTRIBUTOR", "body": "Looks like furo is up-to-date now, so this is no longer needed.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1824399610, "label": "Bump furo from 2023.3.27 to 2023.7.26"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2098#issuecomment-1668186815", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2098", "id": 1668186815, "node_id": "IC_kwDOBm6k_c5jboK_", "user": {"value": 49699333, "label": "dependabot[bot]"}, "created_at": "2023-08-07T16:20:18Z", "updated_at": "2023-08-07T16:20:18Z", "author_association": "CONTRIBUTOR", "body": "Looks like blacken-docs is up-to-date now, so this is no longer needed.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1796830110, "label": "Bump blacken-docs from 1.14.0 to 1.15.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/578#issuecomment-1668113177", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/578", "id": 1668113177, "node_id": "IC_kwDOCGYnMM5jbWMZ", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2023-08-07T15:41:49Z", "updated_at": "2023-08-07T15:41:49Z", "author_association": "CONTRIBUTOR", "body": "I wonder if this should be two hooks: input and output. The current `--csv` (and `--tsv`) options apply to both. Haven't looked at how it's implemented. Or maybe it's one hook that returns a format for reading and for writing.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1818838294, "label": "Plugin hook for adding new output formats"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2124#issuecomment-1662215579", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2124", "id": 1662215579, "node_id": "IC_kwDOBm6k_c5jE2Wb", "user": {"value": 49699333, "label": "dependabot[bot]"}, "created_at": "2023-08-02T13:28:43Z", "updated_at": "2023-08-02T13:28:43Z", "author_association": "CONTRIBUTOR", "body": "Superseded by #2125.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1826424151, "label": "Bump sphinx from 6.1.3 to 7.1.1"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2107#issuecomment-1655678215", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2107", "id": 1655678215, "node_id": "IC_kwDOBm6k_c5ir6UH", "user": {"value": 49699333, "label": "dependabot[bot]"}, "created_at": "2023-07-28T13:23:16Z", "updated_at": "2023-07-28T13:23:16Z", "author_association": "CONTRIBUTOR", "body": "Superseded by #2124.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1820346348, "label": "Bump sphinx from 6.1.3 to 7.1.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2077#issuecomment-1653652665", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2077", "id": 1653652665, "node_id": "IC_kwDOBm6k_c5ikLy5", "user": {"value": 49699333, "label": "dependabot[bot]"}, "created_at": "2023-07-27T13:40:52Z", "updated_at": "2023-07-27T13:40:52Z", "author_association": "CONTRIBUTOR", "body": "Superseded by #2121.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1719759468, "label": "Bump furo from 2023.3.27 to 2023.5.20"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2075#issuecomment-1649849249", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2075", "id": 1649849249, "node_id": "IC_kwDOBm6k_c5iVrOh", "user": {"value": 49699333, "label": "dependabot[bot]"}, "created_at": "2023-07-25T13:28:35Z", "updated_at": "2023-07-25T13:28:35Z", "author_association": "CONTRIBUTOR", "body": "Superseded by #2107.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1710164693, "label": "Bump sphinx from 6.1.3 to 7.0.1"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/578#issuecomment-1648339661", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/578", "id": 1648339661, "node_id": "IC_kwDOCGYnMM5iP6rN", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2023-07-24T17:44:30Z", "updated_at": "2023-07-24T17:44:30Z", "author_association": "CONTRIBUTOR", "body": "> A related feature would be support for plugins to add new ways of ingesting data - currently sqlite-utils insert works against JSON, newline-JSON, CSV and TSV.\r\n\r\nThis is my goal, to have one plugin that handles input and output symmetrically. I'd like to be able to do something like this:\r\n\r\n```sh\r\nsqlite-utils insert data.db table file.geojson --format geojson\r\n# ... explore and manipulate in Datasette\r\nsqlite-utils query data.db ... --format geojson > output.geojson\r\n```\r\n\r\nThis would work especially well with [datasette-query-files](https://github.com/eyeseast/datasette-query-files), since I already have the queries I need saved in standalone SQL files.\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1818838294, "label": "Plugin hook for adding new output formats"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646656283", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/567", "id": 1646656283, "node_id": "IC_kwDOCGYnMM5iJfsb", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2023-07-22T19:32:24Z", "updated_at": "2023-07-22T19:32:24Z", "author_association": "CONTRIBUTOR", "body": "Cool. I might try to add a geojson plugin that handles both input and output. That would help me out a lot. ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1801394744, "label": "Plugin system"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1642808866", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/567", "id": 1642808866, "node_id": "IC_kwDOCGYnMM5h60Yi", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2023-07-19T21:54:27Z", "updated_at": "2023-07-19T21:54:27Z", "author_association": "CONTRIBUTOR", "body": "Would this possibly make a bunch of `x-to-sqlite` tools obsolete? Or nudge some to become plugins?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1801394744, "label": "Plugin system"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2104#issuecomment-1641082395", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2104", "id": 1641082395, "node_id": "IC_kwDOBm6k_c5h0O4b", "user": {"value": 15178711, "label": "asg017"}, "created_at": "2023-07-18T22:41:37Z", "updated_at": "2023-07-18T22:41:37Z", "author_association": "CONTRIBUTOR", "body": "For filtering virtual table's \"shadow tables\" (ex the FTS5 _content and most the spatialite tables), you can use `pragma_table_list` (first appeared in SQLite 3.37 (2021-11-27), which has a `type` column that calls out `type=\"shadow\"` tables https://www.sqlite.org/pragma.html#pragma_table_list", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 1}", "issue": {"value": 1808215339, "label": "Tables starting with an underscore should be treated as hidden"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1638910473", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/567", "id": 1638910473, "node_id": "IC_kwDOCGYnMM5hr8oJ", "user": {"value": 15178711, "label": "asg017"}, "created_at": "2023-07-17T21:27:41Z", "updated_at": "2023-07-17T21:27:41Z", "author_association": "CONTRIBUTOR", "body": "Another use-case: I want to make a `sqlite-utils` plugin that'll help me insert data into Datasette. \r\n\r\n```bash\r\nsqlite-utils insert-datasette \\\r\n --token $DATASETTE_API_KEY \\\r\n https://latest.datasette.io/fixtures/my-table \\\r\n 'select ...'\r\n```\r\n\r\n\r\nThis could also be a datasette plugin (ex `datasette upload-data ...`, but you can also think of `sqlite-utils` plugins that upload to S3, a postgres table, other DBMS's, etc.)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1801394744, "label": "Plugin system"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2052#issuecomment-1630776144", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2052", "id": 1630776144, "node_id": "IC_kwDOBm6k_c5hM6tQ", "user": {"value": 9020979, "label": "hydrosquall"}, "created_at": "2023-07-11T12:54:03Z", "updated_at": "2023-07-11T12:54:03Z", "author_association": "CONTRIBUTOR", "body": "Thanks for the review and the code pointers @simonw - I've made the suggested edits, fixed the renamed variable, and confirmed that the panels still render on the `table` and `database` views. ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1651082214, "label": "feat: Javascript Plugin API (Custom panels, column menu items with JS actions)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2087#issuecomment-1616853644", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2087", "id": 1616853644, "node_id": "IC_kwDOBm6k_c5gXzqM", "user": {"value": 15178711, "label": "asg017"}, "created_at": "2023-07-02T22:00:48Z", "updated_at": "2023-07-02T22:00:48Z", "author_association": "CONTRIBUTOR", "body": "I just saw in the docs that Dasette auto-detects `settings.json`:\r\n\r\n> settings.json - settings that would normally be passed using --setting - here they should be stored as a JSON object of key/value pairs\r\n> [*Source*](https://docs.datasette.io/en/stable/settings.html#:~:text=settings.json%20%2D%20settings%20that%20would%20normally%20be%20passed%20using%20%2D%2Dsetting%20%2D%20here%20they%20should%20be%20stored%20as%20a%20JSON%20object%20of%20key/value%20pairs)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1765870617, "label": "`--settings settings.json` option"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2093#issuecomment-1616286848", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2093", "id": 1616286848, "node_id": "IC_kwDOBm6k_c5gVpSA", "user": {"value": 15178711, "label": "asg017"}, "created_at": "2023-07-02T02:17:46Z", "updated_at": "2023-07-02T02:17:46Z", "author_association": "CONTRIBUTOR", "body": "Storing metadata in the database won't be required. I imagine there'll be many different ways to store metadata, including any possible `datasette_metadata` or sqlite-docs, or the older metadata.json way. \r\n\r\nThe next question will be how precedence should work - i'd imagine metadata.json > plugins > datasette_metadata > sqlite-docs", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1781530343, "label": "Proposal: Combine settings, metadata, static, etc. into a single `datasette.yaml` File"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2052#issuecomment-1616095810", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2052", "id": 1616095810, "node_id": "IC_kwDOBm6k_c5gU6pC", "user": {"value": 15178711, "label": "asg017"}, "created_at": "2023-07-01T20:31:31Z", "updated_at": "2023-07-01T20:31:31Z", "author_association": "CONTRIBUTOR", "body": "> Just curious, is there a query that can be used to compile this programmatically, or did you identify these through memory?\r\n\r\nI just did a github search for `user:simonw \"def extra_js_urls(\"` ! Though I'm sure other plugins made by people other than Simon also exist out there https://github.com/search?q=user%3Asimonw+%22def+extra_js_urls%28%22&type=code", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1651082214, "label": "feat: Javascript Plugin API (Custom panels, column menu items with JS actions)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2052#issuecomment-1615997736", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2052", "id": 1615997736, "node_id": "IC_kwDOBm6k_c5gUiso", "user": {"value": 9020979, "label": "hydrosquall"}, "created_at": "2023-07-01T16:55:24Z", "updated_at": "2023-07-01T16:55:24Z", "author_association": "CONTRIBUTOR", "body": "> Ok @hydrosquall a couple things before this PR should be good to go:\r\n\r\nThank you @asg017 ! I've pushed both suggested changes onto this branch.\r\n\r\n> Not sure how difficult it'll be to inject it server-side\r\n\r\nIf we are OK with having a build system, it would free me up to do do many things! We could make datasette-manager.js a server-side rendered file as a \"template\" instead of having it as a static JS file, but I'm not sure it's worth the extra jump in complexity / loss of syntax highlighting in the JS file.\r\n\r\nIn the short-term, I could see an intermediary solution where a unit test in the preferred language was able to read both `version.py` and `datasette-manager.js`, and make sure that the strings versions are in sync. (This assumes that we want the manager and datasette's versions to be synced, and not decoupled). Since the version is not changing very often, a \"manual sync\" might be good enough. \r\n\r\n> In terms of how to integrate this into Datasette, a few options I can see working:\r\n\r\nThis sounds good to me. I'm not sure how to add a settings flag, but will be interested to see the PR that adds support for it.\r\n\r\n> I'm also curious to see how \"plugins for a plugin' would work\r\n\r\nI'm comfortable to wait until we have a realistic usecase for this. In the short term, I think we could give plugins a way to grant access to a \"public API of other plugins\", and also ask to be notified when plugins with other names have loaded, but don't picture the datasette manager getting more involved than that. \r\n\r\n> here's a list of Simon's Datasette plugins that use \"extra_js_urls()\"\r\n\r\nNeat, thanks for compiling this list! Just curious, is there a query that can be used to compile this programmatically, or did you identify these through memory?\r\n\r\n> I want to make a javascript plugin on top of the code-mirror editor to make a few things nicer (function auto-complete, table/column descriptions, etc.)\r\n\r\nI look forward to trying this out \ud83d\udc4d \r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1651082214, "label": "feat: Javascript Plugin API (Custom panels, column menu items with JS actions)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2093#issuecomment-1613896210", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2093", "id": 1613896210, "node_id": "IC_kwDOBm6k_c5gMhoS", "user": {"value": 15178711, "label": "asg017"}, "created_at": "2023-06-29T22:53:33Z", "updated_at": "2023-06-29T22:53:33Z", "author_association": "CONTRIBUTOR", "body": "Maybe we can have a separate issue for revamping `metadata.json`? A `datasette_metadata` table or the `sqlite-docs` extension seem like two reasonable additions that we can work through. Storing metadata inside a SQLite database makes sense, but I don't think storing `datasette.*` style config (ex ports, settings, etc.) inside a SQLite DB makes sense, since it's very environment-dependent", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1781530343, "label": "Proposal: Combine settings, metadata, static, etc. into a single `datasette.yaml` File"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2093#issuecomment-1613895188", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2093", "id": 1613895188, "node_id": "IC_kwDOBm6k_c5gMhYU", "user": {"value": 15178711, "label": "asg017"}, "created_at": "2023-06-29T22:51:53Z", "updated_at": "2023-06-29T22:51:53Z", "author_association": "CONTRIBUTOR", "body": "I agree with not liking `metadata.json` stuff in a `datasette.*` config file. Editing description of a table/column in a file like `datasette.*` seems odd to me. \r\n\r\nThough since plugin configuration currently lives in `metadata.json`, I think it should be removed from there and placed in `datasette.*`, at least for top-level config like `datasette-auth-github`'s config. Keeping `metadata.json` strictly for documentation/licensing/column units makes sense to me, but anything plugin related should be in some config file, like `datasette.*`.\r\n\r\nAnd ya, supporting both `datasette.*` and CLI flags makes a lot of sense to me. Any `--setting` flag should override anything in `datasette.*` for easier debugging, with possibly a warning message so people don't get confused. Same with `--port` and a port defined in `datasette.*`", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1781530343, "label": "Proposal: Combine settings, metadata, static, etc. into a single `datasette.yaml` File"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2052#issuecomment-1613778296", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2052", "id": 1613778296, "node_id": "IC_kwDOBm6k_c5gME14", "user": {"value": 15178711, "label": "asg017"}, "created_at": "2023-06-29T20:36:09Z", "updated_at": "2023-06-29T20:36:09Z", "author_association": "CONTRIBUTOR", "body": "Ok @hydrosquall a couple things before this PR should be good to go:\r\n\r\n- Can we move `datasette/static/table-example-plugins.js` into `demos/plugins/static`?\r\n- For `datasetteManager.VERSION`, can we fill that in or just comment it out for now? Not sure how difficult it'll be to inject it server-side. I imagine we could also have a small build process with esbuild/rollup that just injects a version string into `manager.js` directly, so we don't have to worry about server-rendering (but that can be a future PR)\r\n\r\nIn terms of how to integrate this into Datasette, a few options I can see working:\r\n\r\n- Push this as-is and figure it out before the next release\r\n- Hide this feature behind a settings flag (`--setting unstable-js-plugins on`) and use that setting to hide/show `` in `base.html`\r\n\r\nI'll let @simonw decide which one to work with. I kindof like the idea of having an \"unstable\" opt-in process to enable JS plugins, to give us time to try it out with a wide variety of plugins until we feel its ready.\r\n\r\nI'm also curious to see how \"plugins for a plugin' would work, like #1542. For example, if the leaflet plugin showed default markers, but also included its own hook for other plugins to add more markers/styling. I'm imagine that the individual plugin would re-create their own plugin system compared to this, since handling \"plugins of plugins\" at the top with Datasette seems really convoluted. \r\n\r\nAlso for posterity, here's a list of Simon's Datasette plugins that use \"extra_js_urls()\", which probably means they can be ported/re-written to use this new plugin system:\r\n\r\n- [`datasette-vega`](https://github.com/simonw/datasette-vega/blob/00de059ab1ef77394ba9f9547abfacf966c479c4/datasette_vega/__init__.py#L25)\r\n- [`datasette-cluster-map`](https://github.com/simonw/datasette-cluster-map/blob/795d25ad9ff6cba0307191f44fecc8f8070bef5c/datasette_cluster_map/__init__.py#L14)\r\n- [`datasette-leaflet-geojson`](https://github.com/simonw/datasette-leaflet-geojson/blob/64713aa497750400b9ac2c12e8bb6ffab8eb77f3/datasette_leaflet_geojson/__init__.py#L47)\r\n- [`datasette-pretty-traces`](https://github.com/simonw/datasette-pretty-traces/blob/5219d65eca3d7d7a73bb9d3120df42fe046a1315/datasette_pretty_traces/__init__.py#L5)\r\n- [`datasette-youtube-embed`](https://github.com/simonw/datasette-youtube-embed/blob/4b4a0d7e58ebe15f47e9baf68beb9908c1d899da/datasette_youtube_embed/__init__.py#L55)\r\n- [`datasette-leaflet-freedraw`](https://github.com/simonw/datasette-leaflet-freedraw/blob/8f28c2c2080ec9d29f18386cc6a2573a1c8fbde7/datasette_leaflet_freedraw/__init__.py#L66)\r\n- [`datasette-hovercards`](https://github.com/simonw/datasette-hovercards/blob/9439ba46b7140fb03223faff0d21aeba5615a287/datasette_hovercards/__init__.py#L5)\r\n- [`datasette-mp3-audio`](https://github.com/simonw/datasette-mp3-audio/blob/4402168792f452a46ab7b488e40ec49cd4b12185/datasette_mp3_audio/__init__.py#L6)\r\n- [`datasette-geojson-map`](https://github.com/simonw/datasette-geojson-map/blob/32af5f1fd1a07278bbf8071fbb20a61e0f613246/datasette_geojson_map/__init__.py#L30)", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 1}", "issue": {"value": 1651082214, "label": "feat: Javascript Plugin API (Custom panels, column menu items with JS actions)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2052#issuecomment-1606352600", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2052", "id": 1606352600, "node_id": "IC_kwDOBm6k_c5fvv7Y", "user": {"value": 15178711, "label": "asg017"}, "created_at": "2023-06-26T00:17:04Z", "updated_at": "2023-06-26T00:17:04Z", "author_association": "CONTRIBUTOR", "body": ":wave: would love to see this get merged soon! I want to make a javascript plugin on top of the code-mirror editor to make a few things nicer (function auto-complete, table/column descriptions, etc.), and this would help out a bunch", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1651082214, "label": "feat: Javascript Plugin API (Custom panels, column menu items with JS actions)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/529#issuecomment-1592110694", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/529", "id": 1592110694, "node_id": "IC_kwDOCGYnMM5e5a5m", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2023-06-14T23:11:47Z", "updated_at": "2023-06-14T23:12:12Z", "author_association": "CONTRIBUTOR", "body": "sorry i was wrong. `sqlite-utils --raw-lines` works correctly\r\n\r\n```\r\nsqlite-utils --raw-lines :memory: \"SELECT * FROM (VALUES ('test'), ('line2'))\" | cat -A\r\ntest$\r\nline2$\r\n\r\nsqlite-utils --csv --no-headers :memory: \"SELECT * FROM (VALUES ('test'), ('line2'))\" | cat -A\r\ntest$\r\nline2$\r\n```\r\n\r\nI think this was fixed somewhat recently", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1581090327, "label": "Microsoft line endings"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1264218914", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/491", "id": 1264218914, "node_id": "IC_kwDOCGYnMM5LWnMi", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2022-10-01T03:18:36Z", "updated_at": "2023-06-14T22:14:24Z", "author_association": "CONTRIBUTOR", "body": "> some good concrete use-cases in mind\r\n\r\nI actually found myself wanting something like this the past couple days. The use-case was databases with slightly different schema but same table names.\r\n\r\nhere is a full script:\r\n\r\n```\r\nimport argparse\r\nfrom pathlib import Path\r\n\r\nfrom sqlite_utils import Database\r\n\r\n\r\ndef connect(args, conn=None, **kwargs) -> Database:\r\n db = Database(conn or args.database, **kwargs)\r\n with db.conn:\r\n db.conn.execute(\"PRAGMA main.cache_size = 8000\")\r\n return db\r\n\r\n\r\ndef parse_args() -> argparse.Namespace:\r\n parser = argparse.ArgumentParser()\r\n parser.add_argument(\"database\")\r\n parser.add_argument(\"dbs_folder\")\r\n parser.add_argument(\"--db\", \"-db\", help=argparse.SUPPRESS)\r\n parser.add_argument(\"--verbose\", \"-v\", action=\"count\", default=0)\r\n args = parser.parse_args()\r\n\r\n if args.db:\r\n args.database = args.db\r\n Path(args.database).touch()\r\n args.db = connect(args)\r\n\r\n return args\r\n\r\n\r\ndef merge_db(args, source_db):\r\n source_db = str(Path(source_db).resolve())\r\n\r\n s_db = connect(argparse.Namespace(database=source_db, verbose = args.verbose))\r\n for table in s_db.table_names():\r\n data = s_db[table].rows\r\n args.db[table].insert_all(data, alter=True, replace=True)\r\n\r\n args.db.conn.commit()\r\n\r\n\r\ndef merge_directory():\r\n args = parse_args()\r\n source_dbs = list(Path(args.dbs_folder).glob('*.db'))\r\n for s_db in source_dbs:\r\n merge_db(args, s_db)\r\n\r\n\r\nif __name__ == '__main__':\r\n merge_directory()\r\n```\r\n\r\nedit: I've made some improvements to this and put it on PyPI:\r\n\r\n```\r\n$ pip install xklb\r\n$ lb merge-db -h\r\nusage: library merge-dbs DEST_DB SOURCE_DB ... [--only-target-columns] [--only-new-rows] [--upsert] [--pk PK ...] [--table TABLE ...]\r\n\r\n Merge-DBs will insert new rows from source dbs to target db, table by table. If primary key(s) are provided,\r\n and there is an existing row with the same PK, the default action is to delete the existing row and insert the new row\r\n replacing all existing fields.\r\n\r\n Upsert mode will update matching PK rows such that if a source row has a NULL field and\r\n the destination row has a value then the value will be preserved instead of changed to the source row's NULL value.\r\n\r\n Ignore mode (--only-new-rows) will insert only rows which don't already exist in the destination db\r\n\r\n Test first by using temp databases as the destination db.\r\n Try out different modes / flags until you are satisfied with the behavior of the program\r\n\r\n library merge-dbs --pk path (mktemp --suffix .db) tv.db movies.db\r\n\r\n Merge database data and tables\r\n\r\n library merge-dbs --upsert --pk path video.db tv.db movies.db\r\n library merge-dbs --only-target-columns --only-new-rows --table media,playlists --pk path audio-fts.db audio.db\r\n\r\n library merge-dbs --pk id --only-tables subreddits reddit/81_New_Music.db audio.db\r\n library merge-dbs --only-new-rows --pk subreddit,path --only-tables reddit_posts reddit/81_New_Music.db audio.db -v\r\n\r\npositional arguments:\r\n database\r\n source_dbs\r\n```\r\n\r\nAlso if you want to dedupe a table based on a \"business key\" which isn't explicitly your primary key(s) you can run this:\r\n\r\n```\r\n$ lb dedupe-db -h\r\nusage: library dedupe-dbs DATABASE TABLE --bk BUSINESS_KEYS [--pk PRIMARY_KEYS] [--only-columns COLUMNS]\r\n\r\n Dedupe your database (not to be confused with the dedupe subcommand)\r\n\r\n It should not need to be said but *backup* your database before trying this tool!\r\n\r\n Dedupe-DB will help remove duplicate rows based on non-primary-key business keys\r\n\r\n library dedupe-db ./video.db media --bk path\r\n\r\n If --primary-keys is not provided table metadata primary keys will be used\r\n If --only-columns is not provided all non-primary and non-business key columns will be upserted\r\n\r\npositional arguments:\r\n database\r\n table\r\n\r\noptions:\r\n -h, --help show this help message and exit\r\n --skip-0\r\n --only-columns ONLY_COLUMNS\r\n Comma separated column names to upsert\r\n --primary-keys PRIMARY_KEYS, --pk PRIMARY_KEYS\r\n Comma separated primary keys\r\n --business-keys BUSINESS_KEYS, --bk BUSINESS_KEYS\r\n Comma separated business keys\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1383646615, "label": "Ability to merge databases and tables"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/535#issuecomment-1592052320", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/535", "id": 1592052320, "node_id": "IC_kwDOCGYnMM5e5Mpg", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2023-06-14T22:05:28Z", "updated_at": "2023-06-14T22:05:28Z", "author_association": "CONTRIBUTOR", "body": "piping to `jq` is good enough usually", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1655860104, "label": "rows: --transpose or psql extended view-like functionality"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/555#issuecomment-1592047502", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/555", "id": 1592047502, "node_id": "IC_kwDOCGYnMM5e5LeO", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2023-06-14T22:00:10Z", "updated_at": "2023-06-14T22:01:57Z", "author_association": "CONTRIBUTOR", "body": "You may want to try doing a performance comparison between this and just selecting all the ids with few constraints and then doing the filtering within python.\r\n\r\nThat might seem like a lazy-programmer, inefficient way but queries with large resultsets are a different profile than what databases like SQLITE are designed for. That is not to say that SQLITE is slow or that python is always faster but when you start reading >20% of an index there is an equilibrium that is reached. Especially when adding in writing extra temp tables and stuff to memory/disk. And especially given the `NOT IN` style of query...\r\n\r\nYou may also try chunking like this:\r\n\r\n```py\r\ndef chunks(lst, n) -> Generator:\r\n for i in range(0, len(lst), n):\r\n yield lst[i : i + n]\r\n\r\nSQLITE_PARAM_LIMIT = 32765\r\n\r\ndata = []\r\nchunked = chunks(video_ids, consts.SQLITE_PARAM_LIMIT)\r\nfor ids in chunked:\r\n data.expand(\r\n list(\r\n db.query(\r\n f\"\"\"SELECT * from videos\r\n WHERE id in (\"\"\"\r\n + \",\".join([\"?\"] * len(ids))\r\n + \")\",\r\n (*ids,),\r\n )\r\n )\r\n )\r\n```\r\n\r\nbut that actually won't work with your `NOT IN` requirements. You need to query the full resultset to check any row.\r\n\r\nSince you are doing stuff with files/videos in SQLITE you might be interested in my side project: https://github.com/chapmanjacobd/library", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1733198948, "label": "Filter table by a large bunch of ids"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/557#issuecomment-1590531892", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/557", "id": 1590531892, "node_id": "IC_kwDOCGYnMM5ezZc0", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2023-06-14T06:09:21Z", "updated_at": "2023-06-14T06:09:21Z", "author_association": "CONTRIBUTOR", "body": "I put together a [simple script](https://github.com/chapmanjacobd/library/blob/42129c5ebe15f9d74653c0f5ca4ed0c991d383e0/xklb/scripts/dedupe_db.py) to upsert and remove duplicate rows based on business keys. If anyone has similar problems with above this might help\r\n\r\n```\r\nCREATE TABLE my_table (\r\n id INTEGER PRIMARY KEY,\r\n column1 TEXT,\r\n column2 TEXT,\r\n column3 TEXT\r\n);\r\n\r\nINSERT INTO my_table (column1, column2, column3)\r\nVALUES\r\n ('Value 1', 'Duplicate 1', 'Duplicate A'),\r\n ('Value 2', 'Duplicate 2', 'Duplicate B'),\r\n ('Value 3', 'Duplicate 2', 'Duplicate C'),\r\n ('Value 4', 'Duplicate 3', 'Duplicate D'),\r\n ('Value 5', 'Duplicate 3', 'Duplicate E'),\r\n ('Value 6', 'Duplicate 3', 'Duplicate F');\r\n```\r\n\r\n```\r\nlibrary dedupe-db test.db my_table --bk column2\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1740150327, "label": "Aliased ROWID option for tables created from alter=True commands"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2052#issuecomment-1585149909", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2052", "id": 1585149909, "node_id": "IC_kwDOBm6k_c5ee3fV", "user": {"value": 9020979, "label": "hydrosquall"}, "created_at": "2023-06-09T21:35:00Z", "updated_at": "2023-06-09T21:35:00Z", "author_association": "CONTRIBUTOR", "body": "Thanks @cldellow for the thoughtful comments! These are all things that I'll keep in mind as we figure out how/if this API is actually used by plugin authors once it's actually out in the world.\r\n\r\n> Yes, this would work - but it requires me to continue to communicate the column names out of band (in order to fetch the facet data per-column before registering my plugin), vs being able to re-use them from the plugin implementation.\r\n\r\nAh, I understand now! Thanks for explaining. ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1651082214, "label": "feat: Javascript Plugin API (Custom panels, column menu items with JS actions)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/557#issuecomment-1577355134", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/557", "id": 1577355134, "node_id": "IC_kwDOCGYnMM5eBId-", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2023-06-05T19:26:26Z", "updated_at": "2023-06-05T19:26:26Z", "author_association": "CONTRIBUTOR", "body": "this isn't really actionable... I'm just being a whiny baby. I have tasted the milk of being able to use `upsert_all`, `insert_all`, etc without having to write DDL to create tables. The meat of the issue is that SQLITE doesn't make rowid stable between vacuums so it is not possible to take shortcuts", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1740150327, "label": "Aliased ROWID option for tables created from alter=True commands"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/556#issuecomment-1575310378", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/556", "id": 1575310378, "node_id": "IC_kwDOCGYnMM5d5VQq", "user": {"value": 601708, "label": "mcint"}, "created_at": "2023-06-04T01:21:15Z", "updated_at": "2023-06-04T01:21:15Z", "author_association": "CONTRIBUTOR", "body": "I've resolved my use, with the line-buffered output and while read loop for line buffered input, but I leave this here so the incremental saving or line-buffered use-case can be explicitly handled or rejected (or deferred).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1740026046, "label": "Support storing incrementally piped values"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2052#issuecomment-1548617257", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2052", "id": 1548617257, "node_id": "IC_kwDOBm6k_c5cTgYp", "user": {"value": 193185, "label": "cldellow"}, "created_at": "2023-05-15T21:32:20Z", "updated_at": "2023-05-15T21:32:20Z", "author_association": "CONTRIBUTOR", "body": "> Were you picturing that the whole plugin config object could be returned as a promise, or that the individual hooks (like makeColumnActions or makeAboveTablePanelConfigs supported returning a promise of arrays instead only returning plain arrays?\r\n\r\nThe latter - that you could return a promise of arrays, so it parallels the [\"await me maybe\" pattern in Datasette](https://simonwillison.net/2020/Sep/2/await-me-maybe/), where you can return either a value, a callable or an awaitable.\r\n\r\n> I have a hunch that what you're describing might be achievable without adding Promises to the API with something\r\n\r\nOops, I did a poor job explaining. Yes, this would work - but it requires me to continue to communicate the column names out of band (in order to fetch the facet data per-column before registering my plugin), vs being able to re-use them from the plugin implementation.\r\n\r\nThis isn't that big of a deal - it'd be a nice ergonomic improvement, but nowhere near as a big of an improvement as having an officially sanctioned way to add stuff to the column menus in the first place.\r\n\r\nThis could also be layered on in a future commit without breaking v1 users, too, so it's not at all urgent.\r\n\r\n> especially if those lines are encapsulated by a function we provide (maybe something that's available on the window provided by Datasette as an inline script tag\r\n\r\nAh, this is maybe the the key point. Since it's all hosted inside Datasette, Datasette can provide some arbitrary sugar to make it easier to work with.\r\n\r\nMy experience with async scripts in JS is that people sometimes don't understand the race conditions inherent to them. If they copy/paste from a tutorial, it does just work. But then they'll delete half the code, and by chance it still works on their machine/Datasette templates, and now someone's headed for an annoying debugging session -- maybe them, maybe someone else who tries to re-use their plugin.\r\n\r\nAgain, a fairly minor thing, though.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1651082214, "label": "feat: Javascript Plugin API (Custom panels, column menu items with JS actions)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2068#issuecomment-1547911570", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2068", "id": 1547911570, "node_id": "IC_kwDOBm6k_c5cQ0GS", "user": {"value": 49699333, "label": "dependabot[bot]"}, "created_at": "2023-05-15T13:59:35Z", "updated_at": "2023-05-15T13:59:35Z", "author_association": "CONTRIBUTOR", "body": "Superseded by #2075.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1690842199, "label": "Bump sphinx from 6.1.3 to 7.0.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2052#issuecomment-1546362374", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2052", "id": 1546362374, "node_id": "IC_kwDOBm6k_c5cK54G", "user": {"value": 9020979, "label": "hydrosquall"}, "created_at": "2023-05-12T22:09:03Z", "updated_at": "2023-05-12T22:09:03Z", "author_association": "CONTRIBUTOR", "body": "Hey @cldellow , thanks for the thoughtful feedback and describing the \"lazy facets\" feature! \r\n\r\nIt sounds like the [postTask](https://developer.mozilla.org/en-US/docs/Web/API/Scheduler/postTask) API might be relevant for the types of network request scheduling you have in mind. \r\n\r\nAddressing your points inline below:\r\n\r\n> It might also be nice if the plugins could return Promises.\r\n\r\nWere you picturing that the whole plugin config object could be returned as a promise, or that the individual hooks (like `makeColumnActions` or `makeAboveTablePanelConfigs` supported returning a promise of arrays instead only returning plain arrays?\r\n\r\nI think what you're describing can be achievable, but I want to make sure I do so in a way that addresses your need / keeps the complexity of the plugin core system at a level this is approachable . \r\n\r\nI have a hunch that what you're describing might be achievable without adding Promises to the API with something like\r\n\r\n```\r\nfetch('/api/with-custom-facets').then(myFacets => {\r\n // reusing the go() idiom\r\n go(manager, myFacets);\r\n})\r\n```\r\n\r\nbut I'd like to confirm if that's the case before investigating adding support.\r\n\r\n> bulletproof plugin registration code that is robust against the order in which the script tags load\r\n\r\nYes, I think what you wrote looks right to me! While it looks a little bit verbose compared to the second example, I'm hoping we can mitigate the cost of that during this API incubation phase by making it an easy-to-copy paste code snippet.\r\n\r\nI haven't heard of the GA queing pattern before, thanks for the example. I won't have time to implement of proof of concept in the next few weeks, but I took some time to think through the pros/cons to decide whether we may want to add this in a future release:\r\n\r\nI can see that this approach brings advantages\r\n\r\n- Plugin developers don't need to know the name of the datasette initialization event to start their plugin\r\n- Pushing a function to an array probably is easier (definitely more concise) than adding a document event listener\r\n- One less event listener sitting in memory\r\n\r\nIt also has some minor costs\r\n\r\n- A malicious plugin could choose to (or accidentally) mess with the order of the queue if multiple scripts are lined up\r\n- Some risk in encouraging people to mutate global state\r\n- (not a cost, more a moot point): changing this API may not make a meaningful difference if we're discussing whether people enter 2 vs 5 lines of code, especially if those lines are encapsulated by a function we provide (maybe something that's available on the `window` provided by Datasette as an inline script tag). \r\n\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1651082214, "label": "feat: Javascript Plugin API (Custom panels, column menu items with JS actions)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/527#issuecomment-1540900733", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/527", "id": 1540900733, "node_id": "IC_kwDOCGYnMM5b2Ed9", "user": {"value": 167893, "label": "mcarpenter"}, "created_at": "2023-05-09T21:15:05Z", "updated_at": "2023-05-09T21:15:05Z", "author_association": "CONTRIBUTOR", "body": "Sorry, I completely missed your first comment whilst on Easter break.\r\n\r\nThis looks like a good practical compromise before v4. Thanks!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1578790070, "label": "`Table.convert()` skips falsey values"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2052#issuecomment-1530822437", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2052", "id": 1530822437, "node_id": "IC_kwDOBm6k_c5bPn8l", "user": {"value": 193185, "label": "cldellow"}, "created_at": "2023-05-02T03:35:30Z", "updated_at": "2023-05-02T16:02:38Z", "author_association": "CONTRIBUTOR", "body": "Also, just checking - is this how I'd write bulletproof plugin registration code that is robust against the order in which the script tags load (eg if both my code and the Datasette code are loaded via a `