html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app
https://github.com/simonw/sqlite-utils/pull/604#issuecomment-1843975536,https://api.github.com/repos/simonw/sqlite-utils/issues/604,1843975536,IC_kwDOCGYnMM5t6NVw,16437338,2023-12-07T01:17:05Z,2023-12-07T01:17:05Z,CONTRIBUTOR,Apologies - I pushed a fix that addresses the mypy failures.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",2001006157,
https://github.com/simonw/datasette/issues/2213#issuecomment-1843072926,https://api.github.com/repos/simonw/datasette/issues/2213,1843072926,IC_kwDOBm6k_c5t2w-e,536941,2023-12-06T15:05:44Z,2023-12-06T15:05:44Z,CONTRIBUTOR,"it probably does not make sense to gzip large sqlite database files on the fly. it can take many seconds to gzip a large file and you either have to have this big thing in memory, or write it to disk, which some deployment environments will not like.
i wonder if it would make sense to gzip the databases as part of the datasette publish process. it would be very cool to statically serve those as if they dynamically zipped (i.e. serve the filename example.db, not example.db.zip, and rely on the browser to expand).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",2028698018,
https://github.com/simonw/sqlite-utils/issues/344#issuecomment-1815825863,https://api.github.com/repos/simonw/sqlite-utils/issues/344,1815825863,IC_kwDOCGYnMM5sO03H,16437338,2023-11-17T06:44:49Z,2023-11-17T06:44:49Z,CONTRIBUTOR,"hello Simon,
I've added more STRICT table support per https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982014776 in changeset https://github.com/simonw/sqlite-utils/commit/e4b9b582cdb4e48430865f8739f341bc8017c1e4.
It also fixes table.transform() to preserve STRICT mode.
Please pull if you deem appropriate. Thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066474200,
https://github.com/simonw/datasette/pull/2209#issuecomment-1812750369,https://api.github.com/repos/simonw/datasette/issues/2209,1812750369,IC_kwDOBm6k_c5sDGAh,198537,2023-11-15T15:29:37Z,2023-11-15T15:29:37Z,CONTRIBUTOR,"Looks like tests are passing now but there is an issue with yaml loading and/or cog.
https://github.com/simonw/datasette/actions/runs/6879299298/job/18710911166?pr=2209","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1994861266,
https://github.com/simonw/datasette/pull/2209#issuecomment-1812623778,https://api.github.com/repos/simonw/datasette/issues/2209,1812623778,IC_kwDOBm6k_c5sCnGi,198537,2023-11-15T14:22:42Z,2023-11-15T15:24:09Z,CONTRIBUTOR,"Whoops, looks like I forgot to check for other places where the 'facetable' table is used in the tests.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1994861266,
https://github.com/simonw/datasette/issues/2208#issuecomment-1812617851,https://api.github.com/repos/simonw/datasette/issues/2208,1812617851,IC_kwDOBm6k_c5sClp7,198537,2023-11-15T14:18:58Z,2023-11-15T14:18:58Z,CONTRIBUTOR,"Without aliases:
The proposed fix in #2209 also works when the 'value' column is actually facetable (just added another value in the 'value' column).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1994857251,
https://github.com/simonw/datasette/pull/2202#issuecomment-1801876943,https://api.github.com/repos/simonw/datasette/issues/2202,1801876943,IC_kwDOBm6k_c5rZnXP,49699333,2023-11-08T13:19:00Z,2023-11-08T13:19:00Z,CONTRIBUTOR,Superseded by #2206.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1959278971,"{""id"": 29110, ""slug"": ""dependabot"", ""node_id"": ""MDM6QXBwMjkxMTA="", ""owner"": {""login"": ""github"", ""id"": 9919, ""node_id"": ""MDEyOk9yZ2FuaXphdGlvbjk5MTk="", ""avatar_url"": ""https://avatars.githubusercontent.com/u/9919?v=4"", ""gravatar_id"": """", ""url"": ""https://api.github.com/users/github"", ""html_url"": ""https://github.com/github"", ""followers_url"": ""https://api.github.com/users/github/followers"", ""following_url"": ""https://api.github.com/users/github/following{/other_user}"", ""gists_url"": ""https://api.github.com/users/github/gists{/gist_id}"", ""starred_url"": ""https://api.github.com/users/github/starred{/owner}{/repo}"", ""subscriptions_url"": ""https://api.github.com/users/github/subscriptions"", ""organizations_url"": ""https://api.github.com/users/github/orgs"", ""repos_url"": ""https://api.github.com/users/github/repos"", ""events_url"": ""https://api.github.com/users/github/events{/privacy}"", ""received_events_url"": ""https://api.github.com/users/github/received_events"", ""type"": ""Organization"", ""site_admin"": false}, ""name"": ""Dependabot"", ""description"": """", ""external_url"": ""https://dependabot-api.githubapp.com"", ""html_url"": ""https://github.com/apps/dependabot"", ""created_at"": ""2019-04-16T22:34:25Z"", ""updated_at"": ""2023-10-12T13:35:09Z"", ""permissions"": {""checks"": ""write"", ""contents"": ""write"", ""issues"": ""write"", ""members"": ""read"", ""metadata"": ""read"", ""pull_requests"": ""write"", ""statuses"": ""read"", ""vulnerability_alerts"": ""read"", ""workflows"": ""write""}, ""events"": [""check_suite"", ""issues"", ""issue_comment"", ""label"", ""pull_request"", ""pull_request_review"", ""pull_request_review_comment"", ""repository""]}"
https://github.com/simonw/datasette/pull/2200#issuecomment-1777228352,https://api.github.com/repos/simonw/datasette/issues/2200,1777228352,IC_kwDOBm6k_c5p7lpA,49699333,2023-10-24T13:40:25Z,2023-10-24T13:40:25Z,CONTRIBUTOR,Superseded by #2202.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1949756141,
https://github.com/simonw/datasette/issues/1655#issuecomment-1767219901,https://api.github.com/repos/simonw/datasette/issues/1655,1767219901,IC_kwDOBm6k_c5pVaK9,536941,2023-10-17T21:29:03Z,2023-10-17T21:29:03Z,CONTRIBUTOR,@yejiyang why don’t you move this discussion to my fork to spare simon’s notifications ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1163369515,
https://github.com/simonw/datasette/issues/1655#issuecomment-1766994810,https://api.github.com/repos/simonw/datasette/issues/1655,1766994810,IC_kwDOBm6k_c5pUjN6,536941,2023-10-17T19:01:59Z,2023-10-17T19:01:59Z,CONTRIBUTOR,"hi @yejiyang, have your tried using my fork of datasette: https://github.com/fgregg/datasette/tree/no_limit_csv_publish
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1163369515,
https://github.com/simonw/datasette/issues/2199#issuecomment-1760401731,https://api.github.com/repos/simonw/datasette/issues/2199,1760401731,IC_kwDOBm6k_c5o7ZlD,15178711,2023-10-12T21:41:42Z,2023-10-12T21:41:42Z,CONTRIBUTOR,"I dig it - I was thinking an Observable notebook where you paste your `metadata.json`/`metadata.yaml` and it would generate the new metadata + datasette.yaml files, but an extensible `datasette upgrade` plugin would be nice for future plugins.
One thing to think about: If someone has comments in their original `metadata.yaml`, could we preserve them in the new files? tbh maybe not too important bc if people cared that much they could just copy + paste, and it might be too distracting
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1940346034,
https://github.com/simonw/sqlite-utils/issues/433#issuecomment-1747231893,https://api.github.com/repos/simonw/sqlite-utils/issues/433,1747231893,IC_kwDOCGYnMM5oJKSV,62745,2023-10-04T16:15:09Z,2023-10-04T16:28:21Z,CONTRIBUTOR,"I confirm the bug, as above, and that @jonafato 's patch fixes it for me. However, it's not the right fix. The problem is that ProgressBar is being used in the wrong way. This also results in two lines being printed instead of one, like this:
```
[#######-----------------------------] 20%
[####################################] 100%%
```
The bug is reproducible for me in any terminal, including Gnome Terminal and Guake, and VSCode. With VSCode I can use this launch.json to reproduce it:
```json
{
""version"": ""0.2.0"",
""configurations"": [
{
""name"": ""Python: Module"",
""type"": ""python"",
""request"": ""launch"",
""module"": ""sqlite_utils"",
""justMyCode"": false,
""args"": [""insert"", ""test.db"", ""test"", ""--csv"", ""tests/sniff/example1.csv""]
}
]
}
```
[edit - deleted my analysis of why the current code is wrong, which was confused and confusing]","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1239034903,
https://github.com/simonw/datasette/pull/2155#issuecomment-1737363182,https://api.github.com/repos/simonw/datasette/issues/2155,1737363182,IC_kwDOBm6k_c5njg7u,418191,2023-09-27T13:05:41Z,2023-09-27T13:05:41Z,CONTRIBUTOR,I'm hitting the #2123 issue and I just patched my local version with this and it seems to work fine.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865572575,
https://github.com/simonw/datasette/pull/2190#issuecomment-1729961503,https://api.github.com/repos/simonw/datasette/issues/2190,1729961503,IC_kwDOBm6k_c5nHR4f,15178711,2023-09-21T16:56:57Z,2023-09-21T16:56:57Z,CONTRIBUTOR,TODO: add similar checks for permissions/allow/canned queries,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901483874,
https://github.com/simonw/datasette/issues/2188#issuecomment-1722848454,https://api.github.com/repos/simonw/datasette/issues/2188,1722848454,IC_kwDOBm6k_c5msJTG,15178711,2023-09-18T06:58:53Z,2023-09-18T06:58:53Z,CONTRIBUTOR,"Thinking about this more, here a list of things I imagine a ""compile-to-sql"" plugin would want to do:
1. Attach itself to the SQL code editor (switch from SQL -> PRQL/Logica, additional syntax highlighting)
2. Add ""Query using PRQL"" buttons in various parts of Datasette's UI, like `/dbname` page
3. Use `$LANGUAGE=` instead of `sql=` in the JSON API and the SQL results pages
4. Have their own dedicated code editor page
1) and 2) would be difficult to do with current plugin hooks, unless we add the concept of ""slots"" and get the JS plugin support in. 3) could maybe be done with the [`asgi_wrapper(datasette)`](https://docs.datasette.io/en/stable/plugin_hooks.html#asgi-wrapper-datasette) hook? And 4) ca n be done easily with the `register_routes()` hooks.
So it really only sounds like extending the SQL editor will be the hard part. In #2094 I want to add JavaScript plugin hooks for extending the SQL editor, which may work here.
If I get the time/motivation, I might try out a `datasette-prql` extension, just because I like playing with it. It'd be really cool if I can get the `asgi_wrapper()` hook to work right there...","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1900026059,
https://github.com/simonw/datasette/issues/1191#issuecomment-1722845490,https://api.github.com/repos/simonw/datasette/issues/1191,1722845490,IC_kwDOBm6k_c5msIky,15178711,2023-09-18T06:55:52Z,2023-09-18T06:55:52Z,CONTRIBUTOR,"One note here: this feature could be called ""slots"", similar to [Layout Slots](https://vitepress.dev/guide/extending-default-theme#layout-slots) in Vitepress.
In Vitepress, you can add custom components/widget/gadgets into determined named ""slots"", like so:
```
doc-top
doc-bottom
doc-footer-before
doc-before
doc-after
...
```
Would be great to do in both Python and Javascript, with the upcoming JavaScript API #2052. In `datasette-write-ui`, all we do is add a few ""Insert row"" and ""edit this row"" buttons and that required completely capturing the `table.html` template, which isn't great for other plugins. But having ""slots"" like `table-footer-before` or `table-row-id` or something would be great to work with.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",787098345,
https://github.com/simonw/datasette/pull/2182#issuecomment-1719451803,https://api.github.com/repos/simonw/datasette/issues/2182,1719451803,IC_kwDOBm6k_c5mfMCb,49699333,2023-09-14T13:27:26Z,2023-09-14T13:27:26Z,CONTRIBUTOR,"Looks like these dependencies are updatable in another way, so this is no longer needed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1890593563,
https://github.com/simonw/datasette/pull/2183#issuecomment-1716801971,https://api.github.com/repos/simonw/datasette/issues/2183,1716801971,IC_kwDOBm6k_c5mVFGz,15178711,2023-09-13T01:34:01Z,2023-09-13T01:34:01Z,CONTRIBUTOR,"@simonw docs are finished, this is ready for review!
One thing: I added ""Configuration"" as a top-level item in the documentation site, at the very bottom. Not sure if this is the best, maybe it can be named ""datasette.yaml Configuration"" or something similar?
Mostly because ""Configuration"" by itself can mean many things, but adding ""datasette.yaml"" would make it pretty clear it's about that specific file, and is easier to scan. I'd also be fine with using ""datasette.yaml"" instead of ""datasette.json"", since writing in YAML is much more forgiving (and advanced users will know JSON is also supported)
Also, maybe this is a chance to consolidate the docs a bit? I think ""Settings"", ""Configuration"", ""Metadata"", and ""Authentication and permissions"" should possibly be under the same section. Maybe even consolidate the different Plugin pages that exist?
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1891212159,
https://github.com/simonw/datasette/issues/2157#issuecomment-1700291967,https://api.github.com/repos/simonw/datasette/issues/2157,1700291967,IC_kwDOBm6k_c5lWGV_,15178711,2023-08-31T02:45:56Z,2023-08-31T02:45:56Z,CONTRIBUTOR,"@simonw what do you think about adding a `DATASETTE_INTERNAL_DB_PATH` env variable, where when defined, is the default location of the internal DB? This means when the `--internal` flag is NOT provided, Datasette would check to see if `DATASETTE_INTERNAL_DB_PATH` exists, and if so, uses that as the internal database (and would fallback to an ephemeral memory database)
My rationale: some plugins may require, or strongly encourage, a persistent internal database (`datasette-comments`, `datasette-bookmarks`, `datasette-link-shortener`, etc.). However, for users that have a global installation of Datasette (say from `brew install` or a global `pip install`), it would be annoying having to specify `--internal` every time. So instead, they can just add `export DATASETTE_INTERNAL_DB_PATH=""/path/to/internal.db""` to their bashrc/zshrc/whereever to not have to worry about `--internal`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865869205,
https://github.com/simonw/datasette/pull/2148#issuecomment-1696591957,https://api.github.com/repos/simonw/datasette/issues/2148,1696591957,IC_kwDOBm6k_c5lH_BV,49699333,2023-08-29T00:15:29Z,2023-08-29T00:15:29Z,CONTRIBUTOR,This pull request was built based on a group rule. Closing it will not ignore any of these versions in future pull requests.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1859415334,
https://github.com/simonw/datasette/pull/2152#issuecomment-1695736691,https://api.github.com/repos/simonw/datasette/issues/2152,1695736691,IC_kwDOBm6k_c5lEuNz,49699333,2023-08-28T13:49:35Z,2023-08-28T13:49:35Z,CONTRIBUTOR,Superseded by #2160.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865174661,
https://github.com/simonw/datasette/pull/2148#issuecomment-1689198413,https://api.github.com/repos/simonw/datasette/issues/2148,1689198413,IC_kwDOBm6k_c5krx9N,49699333,2023-08-23T02:57:55Z,2023-08-23T02:57:55Z,CONTRIBUTOR,"Looks like this PR has been edited by someone other than Dependabot. That means Dependabot can't rebase it - sorry!
If you're happy for Dependabot to recreate it from scratch, overwriting any edits, you can request `@dependabot recreate`.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1859415334,
https://github.com/simonw/datasette/issues/2093#issuecomment-1688532012,https://api.github.com/repos/simonw/datasette/issues/2093,1688532012,IC_kwDOBm6k_c5kpPQs,15178711,2023-08-22T16:21:40Z,2023-08-22T16:21:40Z,CONTRIBUTOR,"OK Here's the gameplan for this, which is closely tied to #2143 :
- We will add a new `datasette.json`/`datasette.yaml` configuration file to datasette, which combines settings/plugin config/permissions/canned queries into a new file format
- Metadata will NOT be a part of this file
- TOML support is not planned, but maybe we can create a separate issue for support TOML with JSON/YAML
- The `settings.json` file will be deprecated, and the `--config` arg will be brought back.
- Command line arguments can still be used to overwrite values (ex `--setting` will overwrite settings in `datasette.yaml`
The format of `datasette.json` will follow what Simon listed here: https://github.com/simonw/datasette/issues/2143#issuecomment-1684484426
Here's the current implementation plan:
1. Add a new `--config` flag and port over `""settings""` into a new datasette.json config file, remove settings.json
2. Add top-level plugin config support to `datasette.json`
3. Figure out database/table structure of config `datasette.json`
4. Port over database/table level plugin config support `datasette.json`
5. Port over permissions/auth settings to `datasette.json`
6. Deprecate non-metadata values in `metadata.json`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781530343,
https://github.com/simonw/datasette/issues/2145#issuecomment-1686745094,https://api.github.com/repos/simonw/datasette/issues/2145,1686745094,IC_kwDOBm6k_c5kibAG,15178711,2023-08-21T17:30:01Z,2023-08-21T17:30:01Z,CONTRIBUTOR,"Another point: The new Datasette write API should refuse to insert a row with a NULL primary key. That will likely decrease the likelihood someone find themselves with NULLs in their primary keys, at least with Datasette users. Especially buggy code that uses the write API, like our `datasette-write-ui` bug that led to this issue.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,
https://github.com/simonw/datasette/pull/2144#issuecomment-1686366557,https://api.github.com/repos/simonw/datasette/issues/2144,1686366557,IC_kwDOBm6k_c5kg-ld,49699333,2023-08-21T13:48:15Z,2023-08-21T13:48:15Z,CONTRIBUTOR,Superseded by #2148.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1856760386,
https://github.com/simonw/datasette/issues/2143#issuecomment-1684496274,https://api.github.com/repos/simonw/datasette/issues/2143,1684496274,IC_kwDOBm6k_c5kZ1-S,15178711,2023-08-18T22:30:45Z,2023-08-18T22:30:45Z,CONTRIBUTOR,"> That said, I do really like a bias towards settings that can be changed at runtime
Does this include things like `--settings` values or plugin config? I can totally see being able to update metadata without restarting, but not sure if that would work well with `--setting`, plugin config, or auth/permissions stuff.
Well it could work with `--setting` and auth/permissions, with a lot of core changes. But changing plugin config on the fly could be challenging, for plugin authors. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,
https://github.com/simonw/datasette/issues/2143#issuecomment-1684205563,https://api.github.com/repos/simonw/datasette/issues/2143,1684205563,IC_kwDOBm6k_c5kYu_7,15178711,2023-08-18T17:12:54Z,2023-08-18T17:12:54Z,CONTRIBUTOR,"Another option would be, instead of flat `datasette.json`/`datasette.yaml` files, we could instead use a Python file, like `datasette_config.py`. That way one could dynamically generate config (ex dev vs prod, auto-discover credentials, etc.). Kinda like Django settings.
Though I imagine Python imports might make this complex to do, and json/yaml is already supported and pretty easy to write
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,
https://github.com/simonw/datasette/issues/2143#issuecomment-1684202932,https://api.github.com/repos/simonw/datasette/issues/2143,1684202932,IC_kwDOBm6k_c5kYuW0,15178711,2023-08-18T17:10:21Z,2023-08-18T17:10:21Z,CONTRIBUTOR,"I agree with all your points!
I think the best solution would be having a `datasette.json` config file, where you ""configure"" your datasette instances, with settings, permissions/auth, plugin configuration, and table settings (sortable column, label columns, etc.). Which #2093 would do.
Then optionally, you have a `metadata.json`, or use `datasette_metadata`, or some other plugin to define metadata (ex the future [sqlite-docs](https://github.com/asg017/sqlite-docs) plugin).
Everything in `datasette.json` could also be overwritten by CLI flags, like `--setting key value`, `--plugin xxxx key value`.
We could even completely remove `settings.json` in favor or just `datasette.json`. Mostly because I think the less files the better, especially if they have generic names like `settings.json` or `config.json`. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,
https://github.com/simonw/datasette/pull/2142#issuecomment-1683950031,https://api.github.com/repos/simonw/datasette/issues/2142,1683950031,IC_kwDOBm6k_c5kXwnP,49699333,2023-08-18T13:49:24Z,2023-08-18T13:49:24Z,CONTRIBUTOR,"Looks like these dependencies are updatable in another way, so this is no longer needed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1854970601,
https://github.com/simonw/datasette/pull/2141#issuecomment-1682256251,https://api.github.com/repos/simonw/datasette/issues/2141,1682256251,IC_kwDOBm6k_c5kRTF7,49699333,2023-08-17T13:07:43Z,2023-08-17T13:07:43Z,CONTRIBUTOR,"Looks like blacken-docs is updatable in another way, so this is no longer needed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1853289039,
https://github.com/simonw/datasette/pull/2125#issuecomment-1668187546,https://api.github.com/repos/simonw/datasette/issues/2125,1668187546,IC_kwDOBm6k_c5jboWa,49699333,2023-08-07T16:20:26Z,2023-08-07T16:20:26Z,CONTRIBUTOR,"Looks like sphinx is up-to-date now, so this is no longer needed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1833193570,
https://github.com/simonw/datasette/pull/2121#issuecomment-1668186872,https://api.github.com/repos/simonw/datasette/issues/2121,1668186872,IC_kwDOBm6k_c5jboL4,49699333,2023-08-07T16:20:19Z,2023-08-07T16:20:19Z,CONTRIBUTOR,"Looks like furo is up-to-date now, so this is no longer needed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1824399610,
https://github.com/simonw/datasette/pull/2098#issuecomment-1668186815,https://api.github.com/repos/simonw/datasette/issues/2098,1668186815,IC_kwDOBm6k_c5jboK_,49699333,2023-08-07T16:20:18Z,2023-08-07T16:20:18Z,CONTRIBUTOR,"Looks like blacken-docs is up-to-date now, so this is no longer needed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1796830110,
https://github.com/simonw/sqlite-utils/issues/578#issuecomment-1668113177,https://api.github.com/repos/simonw/sqlite-utils/issues/578,1668113177,IC_kwDOCGYnMM5jbWMZ,25778,2023-08-07T15:41:49Z,2023-08-07T15:41:49Z,CONTRIBUTOR,I wonder if this should be two hooks: input and output. The current `--csv` (and `--tsv`) options apply to both. Haven't looked at how it's implemented. Or maybe it's one hook that returns a format for reading and for writing.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1818838294,
https://github.com/simonw/datasette/pull/2124#issuecomment-1662215579,https://api.github.com/repos/simonw/datasette/issues/2124,1662215579,IC_kwDOBm6k_c5jE2Wb,49699333,2023-08-02T13:28:43Z,2023-08-02T13:28:43Z,CONTRIBUTOR,Superseded by #2125.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1826424151,
https://github.com/simonw/datasette/pull/2107#issuecomment-1655678215,https://api.github.com/repos/simonw/datasette/issues/2107,1655678215,IC_kwDOBm6k_c5ir6UH,49699333,2023-07-28T13:23:16Z,2023-07-28T13:23:16Z,CONTRIBUTOR,Superseded by #2124.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1820346348,
https://github.com/simonw/datasette/pull/2077#issuecomment-1653652665,https://api.github.com/repos/simonw/datasette/issues/2077,1653652665,IC_kwDOBm6k_c5ikLy5,49699333,2023-07-27T13:40:52Z,2023-07-27T13:40:52Z,CONTRIBUTOR,Superseded by #2121.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1719759468,
https://github.com/simonw/datasette/pull/2075#issuecomment-1649849249,https://api.github.com/repos/simonw/datasette/issues/2075,1649849249,IC_kwDOBm6k_c5iVrOh,49699333,2023-07-25T13:28:35Z,2023-07-25T13:28:35Z,CONTRIBUTOR,Superseded by #2107.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1710164693,
https://github.com/simonw/sqlite-utils/issues/578#issuecomment-1648339661,https://api.github.com/repos/simonw/sqlite-utils/issues/578,1648339661,IC_kwDOCGYnMM5iP6rN,25778,2023-07-24T17:44:30Z,2023-07-24T17:44:30Z,CONTRIBUTOR,"> A related feature would be support for plugins to add new ways of ingesting data - currently sqlite-utils insert works against JSON, newline-JSON, CSV and TSV.
This is my goal, to have one plugin that handles input and output symmetrically. I'd like to be able to do something like this:
```sh
sqlite-utils insert data.db table file.geojson --format geojson
# ... explore and manipulate in Datasette
sqlite-utils query data.db ... --format geojson > output.geojson
```
This would work especially well with [datasette-query-files](https://github.com/eyeseast/datasette-query-files), since I already have the queries I need saved in standalone SQL files.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1818838294,
https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646656283,https://api.github.com/repos/simonw/sqlite-utils/issues/567,1646656283,IC_kwDOCGYnMM5iJfsb,25778,2023-07-22T19:32:24Z,2023-07-22T19:32:24Z,CONTRIBUTOR,Cool. I might try to add a geojson plugin that handles both input and output. That would help me out a lot. ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1801394744,
https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1642808866,https://api.github.com/repos/simonw/sqlite-utils/issues/567,1642808866,IC_kwDOCGYnMM5h60Yi,25778,2023-07-19T21:54:27Z,2023-07-19T21:54:27Z,CONTRIBUTOR,Would this possibly make a bunch of `x-to-sqlite` tools obsolete? Or nudge some to become plugins?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1801394744,
https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1638910473,https://api.github.com/repos/simonw/sqlite-utils/issues/567,1638910473,IC_kwDOCGYnMM5hr8oJ,15178711,2023-07-17T21:27:41Z,2023-07-17T21:27:41Z,CONTRIBUTOR,"Another use-case: I want to make a `sqlite-utils` plugin that'll help me insert data into Datasette.
```bash
sqlite-utils insert-datasette \
--token $DATASETTE_API_KEY \
https://latest.datasette.io/fixtures/my-table \
'select ...'
```
This could also be a datasette plugin (ex `datasette upload-data ...`, but you can also think of `sqlite-utils` plugins that upload to S3, a postgres table, other DBMS's, etc.)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1801394744,
https://github.com/simonw/datasette/pull/2052#issuecomment-1630776144,https://api.github.com/repos/simonw/datasette/issues/2052,1630776144,IC_kwDOBm6k_c5hM6tQ,9020979,2023-07-11T12:54:03Z,2023-07-11T12:54:03Z,CONTRIBUTOR,"Thanks for the review and the code pointers @simonw - I've made the suggested edits, fixed the renamed variable, and confirmed that the panels still render on the `table` and `database` views. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,
https://github.com/simonw/datasette/issues/2087#issuecomment-1616853644,https://api.github.com/repos/simonw/datasette/issues/2087,1616853644,IC_kwDOBm6k_c5gXzqM,15178711,2023-07-02T22:00:48Z,2023-07-02T22:00:48Z,CONTRIBUTOR,"I just saw in the docs that Dasette auto-detects `settings.json`:
> settings.json - settings that would normally be passed using --setting - here they should be stored as a JSON object of key/value pairs
> [*Source*](https://docs.datasette.io/en/stable/settings.html#:~:text=settings.json%20%2D%20settings%20that%20would%20normally%20be%20passed%20using%20%2D%2Dsetting%20%2D%20here%20they%20should%20be%20stored%20as%20a%20JSON%20object%20of%20key/value%20pairs)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1765870617,
https://github.com/simonw/datasette/issues/2093#issuecomment-1616286848,https://api.github.com/repos/simonw/datasette/issues/2093,1616286848,IC_kwDOBm6k_c5gVpSA,15178711,2023-07-02T02:17:46Z,2023-07-02T02:17:46Z,CONTRIBUTOR,"Storing metadata in the database won't be required. I imagine there'll be many different ways to store metadata, including any possible `datasette_metadata` or sqlite-docs, or the older metadata.json way.
The next question will be how precedence should work - i'd imagine metadata.json > plugins > datasette_metadata > sqlite-docs","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781530343,
https://github.com/simonw/datasette/pull/2052#issuecomment-1615997736,https://api.github.com/repos/simonw/datasette/issues/2052,1615997736,IC_kwDOBm6k_c5gUiso,9020979,2023-07-01T16:55:24Z,2023-07-01T16:55:24Z,CONTRIBUTOR,"> Ok @hydrosquall a couple things before this PR should be good to go:
Thank you @asg017 ! I've pushed both suggested changes onto this branch.
> Not sure how difficult it'll be to inject it server-side
If we are OK with having a build system, it would free me up to do do many things! We could make datasette-manager.js a server-side rendered file as a ""template"" instead of having it as a static JS file, but I'm not sure it's worth the extra jump in complexity / loss of syntax highlighting in the JS file.
In the short-term, I could see an intermediary solution where a unit test in the preferred language was able to read both `version.py` and `datasette-manager.js`, and make sure that the strings versions are in sync. (This assumes that we want the manager and datasette's versions to be synced, and not decoupled). Since the version is not changing very often, a ""manual sync"" might be good enough.
> In terms of how to integrate this into Datasette, a few options I can see working:
This sounds good to me. I'm not sure how to add a settings flag, but will be interested to see the PR that adds support for it.
> I'm also curious to see how ""plugins for a plugin' would work
I'm comfortable to wait until we have a realistic usecase for this. In the short term, I think we could give plugins a way to grant access to a ""public API of other plugins"", and also ask to be notified when plugins with other names have loaded, but don't picture the datasette manager getting more involved than that.
> here's a list of Simon's Datasette plugins that use ""extra_js_urls()""
Neat, thanks for compiling this list! Just curious, is there a query that can be used to compile this programmatically, or did you identify these through memory?
> I want to make a javascript plugin on top of the code-mirror editor to make a few things nicer (function auto-complete, table/column descriptions, etc.)
I look forward to trying this out 👍
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,
https://github.com/simonw/datasette/issues/2093#issuecomment-1613896210,https://api.github.com/repos/simonw/datasette/issues/2093,1613896210,IC_kwDOBm6k_c5gMhoS,15178711,2023-06-29T22:53:33Z,2023-06-29T22:53:33Z,CONTRIBUTOR,"Maybe we can have a separate issue for revamping `metadata.json`? A `datasette_metadata` table or the `sqlite-docs` extension seem like two reasonable additions that we can work through. Storing metadata inside a SQLite database makes sense, but I don't think storing `datasette.*` style config (ex ports, settings, etc.) inside a SQLite DB makes sense, since it's very environment-dependent","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781530343,
https://github.com/simonw/datasette/pull/2052#issuecomment-1606352600,https://api.github.com/repos/simonw/datasette/issues/2052,1606352600,IC_kwDOBm6k_c5fvv7Y,15178711,2023-06-26T00:17:04Z,2023-06-26T00:17:04Z,CONTRIBUTOR,":wave: would love to see this get merged soon! I want to make a javascript plugin on top of the code-mirror editor to make a few things nicer (function auto-complete, table/column descriptions, etc.), and this would help out a bunch","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,
https://github.com/simonw/sqlite-utils/issues/529#issuecomment-1592110694,https://api.github.com/repos/simonw/sqlite-utils/issues/529,1592110694,IC_kwDOCGYnMM5e5a5m,7908073,2023-06-14T23:11:47Z,2023-06-14T23:12:12Z,CONTRIBUTOR,"sorry i was wrong. `sqlite-utils --raw-lines` works correctly
```
sqlite-utils --raw-lines :memory: ""SELECT * FROM (VALUES ('test'), ('line2'))"" | cat -A
test$
line2$
sqlite-utils --csv --no-headers :memory: ""SELECT * FROM (VALUES ('test'), ('line2'))"" | cat -A
test$
line2$
```
I think this was fixed somewhat recently","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1581090327,
https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1264218914,https://api.github.com/repos/simonw/sqlite-utils/issues/491,1264218914,IC_kwDOCGYnMM5LWnMi,7908073,2022-10-01T03:18:36Z,2023-06-14T22:14:24Z,CONTRIBUTOR,"> some good concrete use-cases in mind
I actually found myself wanting something like this the past couple days. The use-case was databases with slightly different schema but same table names.
here is a full script:
```
import argparse
from pathlib import Path
from sqlite_utils import Database
def connect(args, conn=None, **kwargs) -> Database:
db = Database(conn or args.database, **kwargs)
with db.conn:
db.conn.execute(""PRAGMA main.cache_size = 8000"")
return db
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser()
parser.add_argument(""database"")
parser.add_argument(""dbs_folder"")
parser.add_argument(""--db"", ""-db"", help=argparse.SUPPRESS)
parser.add_argument(""--verbose"", ""-v"", action=""count"", default=0)
args = parser.parse_args()
if args.db:
args.database = args.db
Path(args.database).touch()
args.db = connect(args)
return args
def merge_db(args, source_db):
source_db = str(Path(source_db).resolve())
s_db = connect(argparse.Namespace(database=source_db, verbose = args.verbose))
for table in s_db.table_names():
data = s_db[table].rows
args.db[table].insert_all(data, alter=True, replace=True)
args.db.conn.commit()
def merge_directory():
args = parse_args()
source_dbs = list(Path(args.dbs_folder).glob('*.db'))
for s_db in source_dbs:
merge_db(args, s_db)
if __name__ == '__main__':
merge_directory()
```
edit: I've made some improvements to this and put it on PyPI:
```
$ pip install xklb
$ lb merge-db -h
usage: library merge-dbs DEST_DB SOURCE_DB ... [--only-target-columns] [--only-new-rows] [--upsert] [--pk PK ...] [--table TABLE ...]
Merge-DBs will insert new rows from source dbs to target db, table by table. If primary key(s) are provided,
and there is an existing row with the same PK, the default action is to delete the existing row and insert the new row
replacing all existing fields.
Upsert mode will update matching PK rows such that if a source row has a NULL field and
the destination row has a value then the value will be preserved instead of changed to the source row's NULL value.
Ignore mode (--only-new-rows) will insert only rows which don't already exist in the destination db
Test first by using temp databases as the destination db.
Try out different modes / flags until you are satisfied with the behavior of the program
library merge-dbs --pk path (mktemp --suffix .db) tv.db movies.db
Merge database data and tables
library merge-dbs --upsert --pk path video.db tv.db movies.db
library merge-dbs --only-target-columns --only-new-rows --table media,playlists --pk path audio-fts.db audio.db
library merge-dbs --pk id --only-tables subreddits reddit/81_New_Music.db audio.db
library merge-dbs --only-new-rows --pk subreddit,path --only-tables reddit_posts reddit/81_New_Music.db audio.db -v
positional arguments:
database
source_dbs
```
Also if you want to dedupe a table based on a ""business key"" which isn't explicitly your primary key(s) you can run this:
```
$ lb dedupe-db -h
usage: library dedupe-dbs DATABASE TABLE --bk BUSINESS_KEYS [--pk PRIMARY_KEYS] [--only-columns COLUMNS]
Dedupe your database (not to be confused with the dedupe subcommand)
It should not need to be said but *backup* your database before trying this tool!
Dedupe-DB will help remove duplicate rows based on non-primary-key business keys
library dedupe-db ./video.db media --bk path
If --primary-keys is not provided table metadata primary keys will be used
If --only-columns is not provided all non-primary and non-business key columns will be upserted
positional arguments:
database
table
options:
-h, --help show this help message and exit
--skip-0
--only-columns ONLY_COLUMNS
Comma separated column names to upsert
--primary-keys PRIMARY_KEYS, --pk PRIMARY_KEYS
Comma separated primary keys
--business-keys BUSINESS_KEYS, --bk BUSINESS_KEYS
Comma separated business keys
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1383646615,
https://github.com/simonw/sqlite-utils/issues/535#issuecomment-1592052320,https://api.github.com/repos/simonw/sqlite-utils/issues/535,1592052320,IC_kwDOCGYnMM5e5Mpg,7908073,2023-06-14T22:05:28Z,2023-06-14T22:05:28Z,CONTRIBUTOR,piping to `jq` is good enough usually,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1655860104,
https://github.com/simonw/sqlite-utils/issues/555#issuecomment-1592047502,https://api.github.com/repos/simonw/sqlite-utils/issues/555,1592047502,IC_kwDOCGYnMM5e5LeO,7908073,2023-06-14T22:00:10Z,2023-06-14T22:01:57Z,CONTRIBUTOR,"You may want to try doing a performance comparison between this and just selecting all the ids with few constraints and then doing the filtering within python.
That might seem like a lazy-programmer, inefficient way but queries with large resultsets are a different profile than what databases like SQLITE are designed for. That is not to say that SQLITE is slow or that python is always faster but when you start reading >20% of an index there is an equilibrium that is reached. Especially when adding in writing extra temp tables and stuff to memory/disk. And especially given the `NOT IN` style of query...
You may also try chunking like this:
```py
def chunks(lst, n) -> Generator:
for i in range(0, len(lst), n):
yield lst[i : i + n]
SQLITE_PARAM_LIMIT = 32765
data = []
chunked = chunks(video_ids, consts.SQLITE_PARAM_LIMIT)
for ids in chunked:
data.expand(
list(
db.query(
f""""""SELECT * from videos
WHERE id in (""""""
+ "","".join([""?""] * len(ids))
+ "")"",
(*ids,),
)
)
)
```
but that actually won't work with your `NOT IN` requirements. You need to query the full resultset to check any row.
Since you are doing stuff with files/videos in SQLITE you might be interested in my side project: https://github.com/chapmanjacobd/library","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1733198948,
https://github.com/simonw/sqlite-utils/issues/557#issuecomment-1590531892,https://api.github.com/repos/simonw/sqlite-utils/issues/557,1590531892,IC_kwDOCGYnMM5ezZc0,7908073,2023-06-14T06:09:21Z,2023-06-14T06:09:21Z,CONTRIBUTOR,"I put together a [simple script](https://github.com/chapmanjacobd/library/blob/42129c5ebe15f9d74653c0f5ca4ed0c991d383e0/xklb/scripts/dedupe_db.py) to upsert and remove duplicate rows based on business keys. If anyone has similar problems with above this might help
```
CREATE TABLE my_table (
id INTEGER PRIMARY KEY,
column1 TEXT,
column2 TEXT,
column3 TEXT
);
INSERT INTO my_table (column1, column2, column3)
VALUES
('Value 1', 'Duplicate 1', 'Duplicate A'),
('Value 2', 'Duplicate 2', 'Duplicate B'),
('Value 3', 'Duplicate 2', 'Duplicate C'),
('Value 4', 'Duplicate 3', 'Duplicate D'),
('Value 5', 'Duplicate 3', 'Duplicate E'),
('Value 6', 'Duplicate 3', 'Duplicate F');
```
```
library dedupe-db test.db my_table --bk column2
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1740150327,
https://github.com/simonw/datasette/pull/2052#issuecomment-1585149909,https://api.github.com/repos/simonw/datasette/issues/2052,1585149909,IC_kwDOBm6k_c5ee3fV,9020979,2023-06-09T21:35:00Z,2023-06-09T21:35:00Z,CONTRIBUTOR,"Thanks @cldellow for the thoughtful comments! These are all things that I'll keep in mind as we figure out how/if this API is actually used by plugin authors once it's actually out in the world.
> Yes, this would work - but it requires me to continue to communicate the column names out of band (in order to fetch the facet data per-column before registering my plugin), vs being able to re-use them from the plugin implementation.
Ah, I understand now! Thanks for explaining. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,
https://github.com/simonw/sqlite-utils/issues/557#issuecomment-1577355134,https://api.github.com/repos/simonw/sqlite-utils/issues/557,1577355134,IC_kwDOCGYnMM5eBId-,7908073,2023-06-05T19:26:26Z,2023-06-05T19:26:26Z,CONTRIBUTOR,"this isn't really actionable... I'm just being a whiny baby. I have tasted the milk of being able to use `upsert_all`, `insert_all`, etc without having to write DDL to create tables. The meat of the issue is that SQLITE doesn't make rowid stable between vacuums so it is not possible to take shortcuts","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1740150327,
https://github.com/simonw/sqlite-utils/issues/556#issuecomment-1575310378,https://api.github.com/repos/simonw/sqlite-utils/issues/556,1575310378,IC_kwDOCGYnMM5d5VQq,601708,2023-06-04T01:21:15Z,2023-06-04T01:21:15Z,CONTRIBUTOR,"I've resolved my use, with the line-buffered output and while read loop for line buffered input, but I leave this here so the incremental saving or line-buffered use-case can be explicitly handled or rejected (or deferred).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1740026046,
https://github.com/simonw/datasette/pull/2068#issuecomment-1547911570,https://api.github.com/repos/simonw/datasette/issues/2068,1547911570,IC_kwDOBm6k_c5cQ0GS,49699333,2023-05-15T13:59:35Z,2023-05-15T13:59:35Z,CONTRIBUTOR,Superseded by #2075.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1690842199,
https://github.com/simonw/datasette/pull/2052#issuecomment-1546362374,https://api.github.com/repos/simonw/datasette/issues/2052,1546362374,IC_kwDOBm6k_c5cK54G,9020979,2023-05-12T22:09:03Z,2023-05-12T22:09:03Z,CONTRIBUTOR,"Hey @cldellow , thanks for the thoughtful feedback and describing the ""lazy facets"" feature!
It sounds like the [postTask](https://developer.mozilla.org/en-US/docs/Web/API/Scheduler/postTask) API might be relevant for the types of network request scheduling you have in mind.
Addressing your points inline below:
> It might also be nice if the plugins could return Promises.
Were you picturing that the whole plugin config object could be returned as a promise, or that the individual hooks (like `makeColumnActions` or `makeAboveTablePanelConfigs` supported returning a promise of arrays instead only returning plain arrays?
I think what you're describing can be achievable, but I want to make sure I do so in a way that addresses your need / keeps the complexity of the plugin core system at a level this is approachable .
I have a hunch that what you're describing might be achievable without adding Promises to the API with something like
```
fetch('/api/with-custom-facets').then(myFacets => {
// reusing the go() idiom
go(manager, myFacets);
})
```
but I'd like to confirm if that's the case before investigating adding support.
> bulletproof plugin registration code that is robust against the order in which the script tags load
Yes, I think what you wrote looks right to me! While it looks a little bit verbose compared to the second example, I'm hoping we can mitigate the cost of that during this API incubation phase by making it an easy-to-copy paste code snippet.
I haven't heard of the GA queing pattern before, thanks for the example. I won't have time to implement of proof of concept in the next few weeks, but I took some time to think through the pros/cons to decide whether we may want to add this in a future release:
I can see that this approach brings advantages
- Plugin developers don't need to know the name of the datasette initialization event to start their plugin
- Pushing a function to an array probably is easier (definitely more concise) than adding a document event listener
- One less event listener sitting in memory
It also has some minor costs
- A malicious plugin could choose to (or accidentally) mess with the order of the queue if multiple scripts are lined up
- Some risk in encouraging people to mutate global state
- (not a cost, more a moot point): changing this API may not make a meaningful difference if we're discussing whether people enter 2 vs 5 lines of code, especially if those lines are encapsulated by a function we provide (maybe something that's available on the `window` provided by Datasette as an inline script tag).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,
https://github.com/simonw/sqlite-utils/issues/527#issuecomment-1540900733,https://api.github.com/repos/simonw/sqlite-utils/issues/527,1540900733,IC_kwDOCGYnMM5b2Ed9,167893,2023-05-09T21:15:05Z,2023-05-09T21:15:05Z,CONTRIBUTOR,"Sorry, I completely missed your first comment whilst on Easter break.
This looks like a good practical compromise before v4. Thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1578790070,
https://github.com/simonw/datasette/pull/2052#issuecomment-1530817667,https://api.github.com/repos/simonw/datasette/issues/2052,1530817667,IC_kwDOBm6k_c5bPmyD,193185,2023-05-02T03:24:53Z,2023-05-02T03:24:53Z,CONTRIBUTOR,"Thanks for putting this together! I've been slammed with work/personal stuff so haven't been able to actually prototype anything with this. :(
tl;dr: I think this would be useful immediately as is. It might also be nice if the plugins could return `Promise`s.
The long version: I read the design notes and example plugin. I think I'd be able to use this in [datasette-ui-extras](https://github.com/cldellow/datasette-ui-extras) for my lazy-facets feature.
The lazy-facets feature tries to provide a snappier user experience. It does this by altering how suggested facets work.
First, at page render time:
(A) it lies to Datasette and claims that no columns support facets, this avoids the lengthy delays/timeouts that can happen if the dataset is large.
(B) there's a python plugin that implements the [extra_body_script](https://docs.datasette.io/en/stable/plugin_hooks.html#extra-body-script-template-database-table-columns-view-name-request-datasette) hook, to write out the list of column names for future use by JavaScript
Second, at page load time: there is some JavaScript that:
(C) makes AJAX requests to suggest facets for each column - it makes 1 request per column, using the data from (B)
(D) wires up the column menus to add Facet-by-this options for each facet
With the currently proposed plugin scheme, I think (D) could be moved into the plugin. I'd do the ajax requests, then register the plugin.
If the plugin scheme also supported promises, I think (B) and (C) could also be moved into the plugin.
Does that make sense? Sorry for the wall of text!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,
https://github.com/simonw/datasette/pull/2064#issuecomment-1529737426,https://api.github.com/repos/simonw/datasette/issues/2064,1529737426,IC_kwDOBm6k_c5bLfDS,49699333,2023-05-01T13:58:50Z,2023-05-01T13:58:50Z,CONTRIBUTOR,Superseded by #2068.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1683229834,
https://github.com/simonw/datasette/pull/2063#issuecomment-1521837780,https://api.github.com/repos/simonw/datasette/issues/2063,1521837780,IC_kwDOBm6k_c5atWbU,49699333,2023-04-25T13:57:52Z,2023-04-25T13:57:52Z,CONTRIBUTOR,Superseded by #2064.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1681339696,
https://github.com/simonw/datasette/pull/2052#issuecomment-1510423051,https://api.github.com/repos/simonw/datasette/issues/2052,1510423051,IC_kwDOBm6k_c5aBzoL,9020979,2023-04-16T16:12:14Z,2023-04-20T05:14:39Z,CONTRIBUTOR,"# Javascript Plugin Docs (alpha)
## Motivation
The Datasette JS Plugin API allows developers to add interactive features to the UI, without having to modify the Python source code.
## Setup
No external/NPM dependencies are needed.
Plugin behavior is coordinated by the Datasette `manager`. Every page has 1 `manager`.
There are 2 ways to add your plugin to the `manager`.
1. Read `window.__DATASETTE__` if the manager was already loaded.
```js
const manager = window.__DATASETTE__;
```
2. Wait for the `datasette_init` event to fire if your code was loaded before the manager is ready.
```js
document.addEventListener(""datasette_init"", function (evt) {
const { detail: manager } = evt;
// register plugin here
});
```
3. Add plugin to the manager by calling `manager.registerPlugin` in a JS file. Each plugin will supply 1 or more hooks with
- unique name (`YOUR_PLUGIN_NAME`)
- a numeric version (starting at `0.1`),
- configuration value, the details vary by hook. (In this example, `getColumnActions` takes a function)
```js
manager.registerPlugin(""YOUR_PLUGIN_NAME"", {
version: 0.1,
makeColumnActions: (columnMeta) => {
return [
{
label: ""Copy name to clipboard"",
// evt = native click event
onClick: (evt) => copyToClipboard(columnMeta.column),
}
];
},
});
```
There are 2 plugin hooks available to `manager.registerPlugin`:
- `makeColumnActions` - Add items to the cog menu for headers on datasette table pages
- `makeAboveTablePanelConfigs` - Add items to ""tabbed"" panel above the `` on pages that use the Datasette table template.
While there are additional properties on the `manager`, but it's not advised to depend on them directly as the shape is subject to change.
4. To make your JS file available as a Datasette plugin from the Python side, you can add a python file resembling [this](https://github.com/simonw/datasette/pull/2052/files#diff-c5ecf3d22075a60d04a4e95da2e15c612cf1bc84e38d777b67ba60dbd156e293) to your plugins directory. Note that you could host your JS file anywhere, it doesn't have to be served from the Datasette statics folder.
I welcome ideas for more hooks, or feedback on the current design!
## Examples
See the [example plugins file](https://github.com/simonw/datasette/blob/2d92b9328022d86505261bcdac419b6ed9cb2236/datasette/static/table-example-plugins.js) for additional examples.
## Hooks API Guide
### `makeAboveTablePanelConfigs`
Provide a function with a list of panel objects. Each panel object should contain
1. A unique string `id`
2. A string `label` for the tab
3. A `render` function. The first argument is reference to an HTML [Element](https://developer.mozilla.org/en-US/docs/Web/API/Element).
Example:
```js
manager.registerPlugin(""panel-plugin-graphs"", {
version: 0.1,
makeAboveTablePanelConfigs: () => {
return [
{
id: 'first-panel',
label: ""My new panel"",
render: node => {
const description = document.createElement('p');
description.innerText = 'Hello world';
node.appendChild(description);
}
}
];
},
});
```
### `makeColumnActions`
Provide a function that returns a list of action objects. Each action object has
1. A string `label` for the menu dropdown label
2. An onClick `render` function.
Example:
```js
manager.registerPlugin(""column-name-plugin"", {
version: 0.1,
getColumnActions: (columnMeta) => {
// Info about selected column.
const { columnName, columnNotNull, columnType, isPk } = columnMeta;
return [
{
label: ""Copy name to clipboard"",
onClick: (evt) => copyToClipboard(column),
}
];
},
});
```
The getColumnActions callback has access to an object with metadata about the clicked column. These fields include:
- columnName: string (name of the column)
- columnNotNull: boolean
- columnType: sqlite datatype enum (text, number, etc)
- isPk: Whether this is the primary key: boolean
You can use this column metadata to customize the action config objects (for example, handling different summaries for text vs number columns).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,
https://github.com/simonw/datasette/pull/2052#issuecomment-1515694393,https://api.github.com/repos/simonw/datasette/issues/2052,1515694393,IC_kwDOBm6k_c5aV6k5,9020979,2023-04-20T04:25:55Z,2023-04-20T04:25:55Z,CONTRIBUTOR,"Thanks for the thoughtful review and generous examples @asg017 ! I'll make the changes you suggested soon. Bonus thoughts inlined below.
> comments
These were very much appreciated, it's important to a plugin system that details like this feel right! I'll address them in batch later in the week.
> I know TypeScript can be a little controversial
FWIW I am in favor of doing Typescript - I just wanted to keep the initial set of files in this PR as simple as possible to review. Really appreciate you scaffolding this initial set of types + I think it would be a welcome addition to maintain a set of types.d.ts files.
I'm entertaining the idea of writing the actual source code in Typescript as long as the compiled output is readable b/c it can be tricky to keep the types and plain JS files in sync. Curious if you have encountered projects that are good at preventing drift.
> Maybe they should have more ""action-y"" names
This is a great observation. I'm inclined towards something like `make*` or `build*` since to me `add*` make me think the thing the method is attached to is being mutated, but I agree that any of these may be clearer than the current `get*` setup. I'll go through and update these.
> Maybe we can make it easier to do pure-js datasette plugins?
I really like this idea! It'll be easier to get contributors if they don't have to touch the python side at _all_.
> And then do the PERMITTED_VIEWS filtering in JS rather than Python.
One cost of doing this is that pages that won't use the JS would still have to load the unused code (given that I'm not sending up anything complex like lazy loading). But hopefully the manager core size is close to negligible, and it won't be a big deal. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,
https://github.com/simonw/datasette/pull/2052#issuecomment-1510423215,https://api.github.com/repos/simonw/datasette/issues/2052,1510423215,IC_kwDOBm6k_c5aBzqv,9020979,2023-04-16T16:12:59Z,2023-04-16T16:12:59Z,CONTRIBUTOR,"## Research notes
- I stuck to the ""minimal dependencies"" ethos of datasette (no React, Typescript, JS linting, etc).
- Main threads on JS plugin development
- Main: sketch of pluggy-inspired system: https://github.com/simonw/datasette/issues/983
- Main: provide locations in Datasette HTML that are designed for multiple plugins to safely cooperate with each other (starting with the panel, but eventually could extend to ""search boxes""): https://github.com/simonw/datasette/issues/1191
- Main: HTML hooks for JS plugin authors: https://github.com/simonw/datasette/issues/987
- Prior threads on JS plugins in Datasette for future design directions
- Idea: pass useful strings to JS plugins: https://github.com/simonw/datasette/issues/1565
- Idea: help with plugin dependency loading: https://github.com/simonw/datasette/issues/1542 . (IMO - the plugin providing the dependency can emit an event once it's done. Other plugins can listen for it, or ask the manager to inform them when the dependency is available).
- Idea: help plugins to manage state in shareable URLs (plugins shouldn't have to interact with the URL directly, should have some basic insulation from clobbering each others' keys): https://github.com/simonw/datasette/issues/1144
- Articles on plugins reviewed
- https://css-tricks.com/designing-a-javascript-plugin-system/
- Plugin/Extension systems reviewed (mostly JS).
- Yarn: https://yarnpkg.com/advanced/plugin-tutorial
- Tappable https://github.com/webpack/tapable (used by Auto, webpack)
- Pluggy: https://pluggy.readthedocs.io/en/stable/
- VSCode: https://code.visualstudio.com/api/get-started/your-first-extension
- Chrome: https://developer.chrome.com/docs/extensions/reference/
- Figma/Figjam Widget: https://www.figma.com/widget-docs/
- Datadog Apps: [Programming Model](https://github.com/DataDog/apps/blob/master/docs/en/programming-model.md)
- Storybook: https://storybook.js.org/docs/react/addons/addons-api","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,
https://github.com/simonw/datasette/pull/2052#issuecomment-1509461324,https://api.github.com/repos/simonw/datasette/issues/2052,1509461324,IC_kwDOBm6k_c5Z-I1M,9020979,2023-04-15T01:57:06Z,2023-04-15T01:57:06Z,CONTRIBUTOR,Notes from 1:1 - it _is_ possible to pass in URL params into a ObservableHQ notebook: https://observablehq.com/@bherbertlc/pass-values-as-url-parameters,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,
https://github.com/simonw/sqlite-utils/pull/531#issuecomment-1501017004,https://api.github.com/repos/simonw/sqlite-utils/issues/531,1501017004,IC_kwDOCGYnMM5Zd7Os,25778,2023-04-09T01:49:43Z,2023-04-09T01:49:43Z,CONTRIBUTOR,I'm going to close this in favor of #536. Will try a cleaner approach to custom paths once that one is merge.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1620164673,
https://github.com/simonw/datasette/pull/2014#issuecomment-1487999503,https://api.github.com/repos/simonw/datasette/issues/2014,1487999503,IC_kwDOBm6k_c5YsRIP,49699333,2023-03-29T06:09:11Z,2023-03-29T06:09:11Z,CONTRIBUTOR,Superseded by #2047.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1566081801,
https://github.com/simonw/datasette/pull/2043#issuecomment-1486944644,https://api.github.com/repos/simonw/datasette/issues/2043,1486944644,IC_kwDOBm6k_c5YoPmE,49699333,2023-03-28T13:58:20Z,2023-03-28T13:58:20Z,CONTRIBUTOR,Superseded by #2046.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1639446870,
https://github.com/simonw/sqlite-utils/pull/531#issuecomment-1465315726,https://api.github.com/repos/simonw/sqlite-utils/issues/531,1465315726,IC_kwDOCGYnMM5XVvGO,25778,2023-03-12T22:21:56Z,2023-03-12T22:21:56Z,CONTRIBUTOR,"Exactly, that's what I was running into. On my M2 MacBook, SpatiaLite ends up in what is -- for the moment -- a non-standard location, so even when I passed in the location with `--load-extension`, I still hit an error on `create-spatial-index`.
What I learned doing this originally is that SQLite needs to load the extension for each connection, even if all the SpatiaLite stuff is already in the database. So that's why `init_spatialite()` gets called again.
Here's the code where I hit the error: https://github.com/eyeseast/boston-parcels/blob/main/Makefile#L30 It works using this branch.
I'm not attached to this solution if you can think of something better. And I'm not sure, TBH, my test would actually catch what I'm after here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1620164673,
https://github.com/simonw/datasette/issues/2033#issuecomment-1457172180,https://api.github.com/repos/simonw/datasette/issues/2033,1457172180,IC_kwDOBm6k_c5W2q7U,25778,2023-03-06T22:54:52Z,2023-03-06T22:54:52Z,CONTRIBUTOR,This would be a nice feature to have with `datasette publish` too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1612296210,
https://github.com/simonw/sqlite-utils/issues/525#issuecomment-1435318713,https://api.github.com/repos/simonw/sqlite-utils/issues/525,1435318713,IC_kwDOCGYnMM5VjTm5,167893,2023-02-17T21:55:01Z,2023-02-17T21:55:01Z,CONTRIBUTOR,"Meanwhile, a cheap workaround is to invalidate the registered function cache:
``` python
table.convert(...)
db._registered_functions = set()
table.convert(...)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1575131737,
https://github.com/simonw/sqlite-utils/issues/525#issuecomment-1423387341,https://api.github.com/repos/simonw/sqlite-utils/issues/525,1423387341,IC_kwDOCGYnMM5U1yrN,167893,2023-02-08T23:48:52Z,2023-02-09T00:17:30Z,CONTRIBUTOR,PR below,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1575131737,
https://github.com/simonw/sqlite-utils/issues/520#issuecomment-1421571810,https://api.github.com/repos/simonw/sqlite-utils/issues/520,1421571810,IC_kwDOCGYnMM5Uu3bi,167893,2023-02-07T22:43:09Z,2023-02-07T22:43:09Z,CONTRIBUTOR,"Hey, isn't this essentially the same issue as #448 ?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1516644980,
https://github.com/simonw/datasette/pull/564#issuecomment-1420941334,https://api.github.com/repos/simonw/datasette/issues/564,1420941334,IC_kwDOBm6k_c5UsdgW,82988,2023-02-07T15:14:10Z,2023-02-07T15:14:10Z,CONTRIBUTOR,"Is this feature covered by any more recent updates to `datasette`, or via any plugins that you're aware of?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",473288428,
https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1419357290,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1419357290,IC_kwDOCGYnMM5Umaxq,25778,2023-02-06T16:21:44Z,2023-02-06T16:21:44Z,CONTRIBUTOR,SQLite doesn't have a native `DATETIME` type. It stores dates internally as strings and then has [functions](https://www.sqlite.org/lang_datefunc.html) to work with date-like strings. Yes it's weird.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,
https://github.com/simonw/datasette/issues/1696#issuecomment-1407767434,https://api.github.com/repos/simonw/datasette/issues/1696,1407767434,IC_kwDOBm6k_c5T6NOK,193185,2023-01-29T20:56:20Z,2023-01-29T20:56:20Z,CONTRIBUTOR,"I did some horrible things in https://github.com/cldellow/datasette-ui-extras/issues/2 to enable this in my plugin -- example here: https://dux-demo.fly.dev/cooking/posts?_facet=owner_user_id&owner_user_id=67
The implementation relies on two things:
- a `filters_from_request` hook that adds a good human description (unfortunately, without the benefit of the CSS styling you mention)
- doing something evil to hijack the `exact` and `not` operators in the `Filters` class. We can't leave them as is, or we'll get 2 human descriptions -- the built-in Datasette one and the one from my plugin. We can't remove them, or the filters UI will stop supporting the `=` and `!=` operators
This got me thinking: it'd be neat if the list of operators that the filters UI supported wasn't a closed set.
A motivating example: adding a geospatial `NEAR` operator. Ideally it'd take two arguments - a target point and a radius, so you could express a filter like `find me all rows whose lat/lng are within 10km of 43.4516° N, 80.4925° W`. (Optionally, the UI could be enhanced if the geonames database was loaded and queried, so a user could say `find me all rows whose lat/lng are within 10km of Kitchener, ON`, and the city gets translated to a lat/lng for them)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1186696202,
https://github.com/simonw/datasette/pull/2008#issuecomment-1407716963,https://api.github.com/repos/simonw/datasette/issues/2008,1407716963,IC_kwDOBm6k_c5T6A5j,193185,2023-01-29T17:04:03Z,2023-01-29T17:04:03Z,CONTRIBUTOR,"Performance tests - I think most places don't have them as a formal gate enforced by CI. TypeScript and scalac seem to have tests that run to capture timings. The timings are included by a bot as a comment or build check, and also stored in a database so you can graph changes over time to spot regressions. Probably overkill for Datasette!
Window functions - oh, good point. Looks like Ubuntu shipped JSON1 support as far back as sqlite 3.11. I'll let this PR linger until there's a way to run against different SQLite versions. For now, I'm shipping this with `datasette-ui-extras`, since I think it's OK for a plugin to enforce a higher minimum requirement.
Tests - there actually did end up being test changes to capture the undercount bug of the current implementation, so the current implementation would fail against the new tests.
Perhaps a non-window function version could be written that uses `random()` instead of `row_number() over ()` in order to get a unique key. It's technically not unique, but in practice, I imagine it'll work well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1560982210,
https://github.com/simonw/datasette/pull/2008#issuecomment-1407561308,https://api.github.com/repos/simonw/datasette/issues/2008,1407561308,IC_kwDOBm6k_c5T5a5c,193185,2023-01-29T04:50:50Z,2023-01-29T04:50:50Z,CONTRIBUTOR,I pushed a revised version which ends up being faster -- the example which currently takes 4 seconds now runs in 500ms.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1560982210,
https://github.com/simonw/datasette/pull/2008#issuecomment-1407558284,https://api.github.com/repos/simonw/datasette/issues/2008,1407558284,IC_kwDOBm6k_c5T5aKM,193185,2023-01-29T04:23:58Z,2023-01-29T04:24:27Z,CONTRIBUTOR,"Ack, this PR is broken. I see now that the `inner.*` is necessary for ensuring the correct count in the face of rows having duplicate values in views.
That fixes the overcounting, but I think can undercount when the rows have the same data, eg a view like:
```sql
SELECT '[""bar""]' tags UNION ALL SELECT '[""bar""]'
```
will produce a count of `{""bar"": 1 }`, when it should be `{""bar"": 2}`. In fact, this could apply in tables without primary keys, too.
If `inner` came from a base table that had a primary key or a rowid, we could use those column(s) to solve that case.
I guess a general solution would be to compute a window function so we have a distinct ID for each row. Will fiddle to see if I can get that working.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1560982210,
https://github.com/simonw/datasette/issues/1973#issuecomment-1407523547,https://api.github.com/repos/simonw/datasette/issues/1973,1407523547,IC_kwDOBm6k_c5T5Rrb,193185,2023-01-29T00:40:31Z,2023-01-29T00:40:31Z,CONTRIBUTOR,"A +1 for switching to `CustomRow`: I think you currently only get a `CustomRow` if the result set had a column that was an fkey ([this code](https://github.com/simonw/datasette/blob/3c352b7132ef09b829abb69a0da0ad00be5edef9/datasette/views/table.py#L667-L682))
Otherwise you get vanilla `sqlite3.Row`s, which will fail if you try to access `.columns` or lookup the cell by name, which surprised me recently","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1515815014,
https://github.com/simonw/datasette/pull/2008#issuecomment-1407470429,https://api.github.com/repos/simonw/datasette/issues/2008,1407470429,IC_kwDOBm6k_c5T5Etd,193185,2023-01-28T19:34:29Z,2023-01-28T19:34:29Z,CONTRIBUTOR,"I don't know how/if you do automated tests for performance, so I haven't changed any of the tests.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1560982210,
https://github.com/simonw/sqlite-utils/issues/523#issuecomment-1407264466,https://api.github.com/repos/simonw/sqlite-utils/issues/523,1407264466,IC_kwDOCGYnMM5T4SbS,536941,2023-01-28T02:41:14Z,2023-01-28T02:41:14Z,CONTRIBUTOR,"I also often then run another little script to cast all empty strings to null, but i save that for another issue if this gets accepted.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1560651350,
https://github.com/simonw/datasette/pull/2003#issuecomment-1404065571,https://api.github.com/repos/simonw/datasette/issues/2003,1404065571,IC_kwDOBm6k_c5TsFcj,536941,2023-01-25T18:44:42Z,2023-01-25T18:44:42Z,CONTRIBUTOR,see this related discussion to a change in API in sqlite-utils https://github.com/simonw/sqlite-utils/pull/203#issuecomment-753567932,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1555701851,
https://github.com/simonw/datasette/issues/2001#issuecomment-1403078134,https://api.github.com/repos/simonw/datasette/issues/2001,1403078134,IC_kwDOBm6k_c5ToUX2,193185,2023-01-25T04:20:43Z,2023-01-25T04:22:28Z,CONTRIBUTOR,"I'm on Ubuntu, unfortunately. :( Would it still be relevant?
I think I've narrowed things down a bit more.
Even `sqlite3_free(sqlite3_malloc(128))` segfaults -- this suggests to me that it's something about the sqlite3 library that was loaded, vs, say, getting the wrong db handle when I go spelunking in the Connection object.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1553615704,
https://github.com/simonw/datasette/issues/2001#issuecomment-1403053144,https://api.github.com/repos/simonw/datasette/issues/2001,1403053144,IC_kwDOBm6k_c5ToORY,193185,2023-01-25T03:34:53Z,2023-01-25T03:34:53Z,CONTRIBUTOR,"Your comment introduced me to this issue in sqlite and to the `ctypes` module - thanks!
> I also hope that the datasette developers will enable this mode in a test environment [...]
> perhaps we could figure out how to invoke it using `ctypes`
I'm not a Datasette developer, but I _am_ curious to learn more about getting unholy access to the sqlite C APIs inside of Datasette. (Such access could also help #1293, and if done without grovelling inside of pysqlite's Connection object for the db handle, could even be relatively safe.)
I experimented a bit. I came up with https://gist.github.com/cldellow/85bba507c314b127f85563869cd94820
If you run `python3 enable-strict-quoting-sqlite3.py`, it seems to set those flags correctly -- `SELECT ""foo""` fails where it would normally succeed.
But if you put it in a `plugins/` dir and run `datasette --plugins-dir plugins/`, it segfaults when it tries to call `sqlite3_db_config` on the connections created by Datasette.
I am... confused. I'm _pretty_ sure I'm using the same python and the same libsqlite3 in both scenarios, so I would expect it to work.
@gwk do you know anything that might help me debug the segfault? I gather that my approach of going grovelling inside of a `PyObject` is particularly dangerous, but I was thinking (a) it's necessary in order to test Datasette's use of the sqlite3 library and (b) even if it's not portable, it'd be good enough for running the tests on a single machine.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1553615704,
https://github.com/simonw/datasette/issues/1099#issuecomment-1402900354,https://api.github.com/repos/simonw/datasette/issues/1099,1402900354,IC_kwDOBm6k_c5Tno-C,536941,2023-01-25T00:58:26Z,2023-01-25T00:58:26Z,CONTRIBUTOR,"> My original idea for compound foreign keys was to turn both of those columns into links, but that doesn't fit here because `database_name` is already part of a different foreign key.
it's pretty hard to know what the right thing to do is if a field is part of multiple foreign keys.
but, if that's not the case, what about making each of the columns a link. seems like an improvement over the status quo.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743371103,
https://github.com/simonw/datasette/issues/1099#issuecomment-1402898291,https://api.github.com/repos/simonw/datasette/issues/1099,1402898291,IC_kwDOBm6k_c5Tnodz,536941,2023-01-25T00:55:06Z,2023-01-25T00:55:06Z,CONTRIBUTOR,"I went ahead and spiked something together, in #2003 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743371103,
https://github.com/simonw/datasette/pull/2003#issuecomment-1402898033,https://api.github.com/repos/simonw/datasette/issues/2003,1402898033,IC_kwDOBm6k_c5TnoZx,536941,2023-01-25T00:54:41Z,2023-01-25T00:54:41Z,CONTRIBUTOR,"@simonw, let me know what you think about this approach!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1555701851,
https://github.com/simonw/datasette/issues/1099#issuecomment-1402563930,https://api.github.com/repos/simonw/datasette/issues/1099,1402563930,IC_kwDOBm6k_c5TmW1a,536941,2023-01-24T20:11:11Z,2023-01-24T20:11:11Z,CONTRIBUTOR,"hi @simonw, this bug bit me today.
the UX for linking from a table to the foreign key seems tough!
the design in the other direction seems a lot easier, for a given primary key detail page, add links back to the tables that refer to the row.
would you be open to a PR that solved the second problem but not the first?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743371103,
https://github.com/simonw/datasette/issues/2000#issuecomment-1399847946,https://api.github.com/repos/simonw/datasette/issues/2000,1399847946,IC_kwDOBm6k_c5Tb_wK,193185,2023-01-23T06:08:00Z,2023-01-23T06:08:00Z,CONTRIBUTOR,"Actually, I discovered [your post](https://til.simonwillison.net/datasette/register-new-plugin-hooks) showing how a plugin can add a Datasette hook. That's wild! I've released `datasette-rewrite-sql` that adds this ability, albeit via monkey patching.
I had hoped to be able to expose `request` to the hook (or, even better `actor`) when the SQL was being run as a result of a user's HTTP request.
But some spelunking in the code makes me suspect that would actually require co-operation from Datasette itself. I'd be happy to be wrong and pointed in the right direction, though!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1552368054,
https://github.com/simonw/datasette/pull/1982#issuecomment-1376620851,https://api.github.com/repos/simonw/datasette/issues/1982,1376620851,IC_kwDOBm6k_c5SDZEz,49699333,2023-01-10T02:03:18Z,2023-01-10T02:03:18Z,CONTRIBUTOR,"Looks like sphinx is up-to-date now, so this is no longer needed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1525560504,
https://github.com/simonw/datasette/issues/1983#issuecomment-1375810027,https://api.github.com/repos/simonw/datasette/issues/1983,1375810027,IC_kwDOBm6k_c5SATHr,25778,2023-01-09T15:35:58Z,2023-01-09T15:35:58Z,CONTRIBUTOR,"Yes please, and thank you. I realized I was maybe getting myself in trouble using that, but I think it's a good way to standardize JSON handling.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1525815985,
https://github.com/simonw/datasette/issues/1978#issuecomment-1375708725,https://api.github.com/repos/simonw/datasette/issues/1978,1375708725,IC_kwDOBm6k_c5R_6Y1,25778,2023-01-09T14:30:00Z,2023-01-09T14:30:00Z,CONTRIBUTOR,Totally missed that issue. I can close this as a duplicate.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1522778923,
https://github.com/simonw/datasette/pull/1977#issuecomment-1375596856,https://api.github.com/repos/simonw/datasette/issues/1977,1375596856,IC_kwDOBm6k_c5R_fE4,49699333,2023-01-09T13:06:14Z,2023-01-09T13:06:14Z,CONTRIBUTOR,Superseded by #1982.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1522552817,
https://github.com/simonw/datasette/pull/1976#issuecomment-1373592231,https://api.github.com/repos/simonw/datasette/issues/1976,1373592231,IC_kwDOBm6k_c5R31qn,49699333,2023-01-06T13:02:15Z,2023-01-06T13:02:15Z,CONTRIBUTOR,Superseded by #1977.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1520712722,
https://github.com/simonw/datasette/pull/1974#issuecomment-1372188571,https://api.github.com/repos/simonw/datasette/issues/1974,1372188571,IC_kwDOBm6k_c5Rye-b,49699333,2023-01-05T13:02:40Z,2023-01-05T13:02:40Z,CONTRIBUTOR,Superseded by #1976.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1516376583,
https://github.com/simonw/datasette/issues/1973#issuecomment-1369044959,https://api.github.com/repos/simonw/datasette/issues/1973,1369044959,IC_kwDOBm6k_c5Rmfff,193185,2023-01-02T15:41:40Z,2023-01-02T15:41:40Z,CONTRIBUTOR,"Thanks for the response!
Yes, it does seem like a pretty nice developer experience--both the automagical labelling of fkeys, and the ability to index the row by column name in addition to column index.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1515815014,
https://github.com/simonw/datasette/issues/1614#issuecomment-1364345119,https://api.github.com/repos/simonw/datasette/issues/1614,1364345119,IC_kwDOBm6k_c5RUkEf,536941,2022-12-23T21:27:10Z,2022-12-23T21:27:10Z,CONTRIBUTOR,is this issue closed by #1893?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1115435536,
https://github.com/simonw/datasette/issues/1796#issuecomment-1364345071,https://api.github.com/repos/simonw/datasette/issues/1796,1364345071,IC_kwDOBm6k_c5RUkDv,536941,2022-12-23T21:27:02Z,2022-12-23T21:27:02Z,CONTRIBUTOR,@simonw is this issue closed by #1893?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1355148385,