{"id": 1200224939, "node_id": "I_kwDOBm6k_c5Hifqr", "number": 1707, "title": "[feature] expanded detail page", "user": {"value": 536941, "label": "fgregg"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-04-11T16:29:17Z", "updated_at": "2022-04-11T16:33:00Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "Right now, if click on the detail page for a row you get the info for the row and links to related tables:\r\n![Screenshot 2022-04-11 at 12-27-26 lm20 filing](https://user-images.githubusercontent.com/536941/162786802-90ac1a71-4624-47c4-ae55-b783f4f6c92d.png)\r\n\r\nIt would be very cool if there was an option to expand the rows of the related tables from within this detail view.\r\n\r\nIf you had that then datasette could fulfill a pretty common use case where you want to search for an entity and get a consolidate detail view about what you know about that entity.\r\n\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1707/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1193090967, "node_id": "I_kwDOBm6k_c5HHR-X", "number": 1699, "title": "Proposal: datasette query", "user": {"value": 25778, "label": "eyeseast"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2022-04-05T12:36:43Z", "updated_at": "2022-04-11T01:32:12Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "I started sketching out a plugin to add a `datasette query` subcommand to export data from the command line. This is based on discussions in #1356 and #1605. Before I get too far down this rabbit hole, I figure it's worth getting some feedback here (unless this should happen in `Discussions`). Here's what I'm thinking:\r\n\r\nAt its most basic, it will write the results of a query to STDOUT.\r\n\r\n```sh\r\ndatasette query -d data.db 'select * from data' > results.json\r\n```\r\n\r\nThis isn't much improvement over using [sqlite-utils](https://github.com/simonw/sqlite-utils). To make better use of datasette and its ecosystem, run `datasette query` using a canned query defined in a `metadata.yml` file.\r\n\r\nFor example, using the metadata file from [alltheplaces-datasette](https://github.com/eyeseast/alltheplaces-datasette/blob/main/metadata.yml):\r\n\r\n```sh\r\ncd alltheplaces-datasette\r\ndatasette query -d alltheplaces.db -m metadata.yml count_by_spider\r\n```\r\n\r\nThat query would be good to get as CSV, and we can auto-discover metadata and databases in the current directory:\r\n\r\n```sh\r\ncd alltheplaces-datasette\r\ndatasette query count_by_spider -f csv\r\n```\r\n\r\nIn this case, `count_by_spider` is a canned query defined on the `alltheplaces` database. If the same query is defined on multiple databases or its otherwise unclear which database `query` should use, pass the `-d` or `--database` option.\r\n\r\nIf a query takes parameters, I can pass them in at runtime, using the `--param` or `-p` option:\r\n\r\n```sh\r\ndatasette query -d data.db -p value something 'select * from neighborhoods where some_column = :value'\r\n```\r\n\r\nI'm very interested in feedback on this, including whether it should be a plugin or in Datasette core. (I don't have a strong opinion about this, but I'm prototyping it as a plugin to start.)", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1699/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1197925865, "node_id": "I_kwDOBm6k_c5HZuXp", "number": 1704, "title": "File PRs against incompatible plugins pinning to datasette<1.0", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 0, "created_at": "2022-04-08T23:15:30Z", "updated_at": "2022-04-08T23:15:30Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "As part of the preparation for the 1.0 release, test all existing known plugins against the alpha.\r\n\r\nFor any that break, submit a PR suggesting they pin to a version <1.0 - and include a link to the documentation on how to upgrade the plugin for 1.0.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1704/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1196327155, "node_id": "I_kwDOBm6k_c5HToDz", "number": 1702, "title": "Be more consistent with column quoting", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-04-07T16:59:20Z", "updated_at": "2022-04-07T16:59:20Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "This tutorial made me notice that Datasette is pretty inconsistent with how column quoting works: https://datasette.io/tutorials/learn-sql\r\n\r\nIt has examples of each of `\"table_name\"` and `[table_name]` and `table_name`, and it uses single quoted values too.\r\n\r\nDatasette should generate SQL as consistently as possible to support learners.\r\n\r\nThat tutorial should also provide a tiny bit of extra information about what's going on here.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1702/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1077620955, "node_id": "I_kwDOBm6k_c5AOzDb", "number": 1549, "title": "Redesign CSV export to improve usability", "user": {"value": 536941, "label": "fgregg"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 5, "created_at": "2021-12-11T19:02:12Z", "updated_at": "2022-04-04T11:17:13Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "*Original title: Set content type for CSV so that browsers will attempt to download instead opening in the browser*\r\n\r\nRight now, if the user clicks on the CSV related to a table or a query, the response header for the content type is \r\n\r\n\"content-type: text/plain; charset=utf-8\"\r\n\r\nMost browsers will try to open a file with this content-type in the browser. \r\n\r\nThis is not what most people want to do, and lots of folks don't know that if they want to download the CSV and open it in the a spreadsheet program they next need to save the page through their browser.\r\n\r\nIt would be great if the response header could be something like \r\n\r\n```\r\n'Content-type: text/csv');\r\n'Content-disposition: attachment;filename=MyVerySpecial.csv');\r\n```\r\n\r\nwhich would lead browsers to open a download dialog.\r\n\r\n\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1549/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1182227211, "node_id": "I_kwDOBm6k_c5Gd1sL", "number": 1692, "title": "[plugins][feature request]: Support additional script tag attributes when loading custom JS", "user": {"value": 9020979, "label": "hydrosquall"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2022-03-27T01:16:03Z", "updated_at": "2022-03-30T06:14:51Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "## Motivation\r\n\r\n- The build system for my new [plugin](https://github.com/hydrosquall/datasette-nteract-data-explorer) has two output JS files, one for browsers that support ES modules, one for browsers that don't. At present, I'm only passing one of them into Datasette.\r\n- I'd like to specify the non-es-module script as a fallback for older browsers. I don't want to load it by default, because browsers will only need one, and it's heavy, so for now I'm only supporting modern browsers. \r\n\r\nTo be able to support legacy browsers without slowing down users with modern browsers, I would like to be able to set additional HTML attributes on the tag fallback script, `nomodule` and `defer`. My injected scripts should look something like this:\r\n\r\n```html\r\n\r\n\r\n```\r\n\r\n## Proposal\r\n\r\nTo achieve this, I propose additional optional properties to the API accepted by the `extra_js_urls` hook and custom JS field the `metadata.json` [described here](https://docs.datasette.io/en/stable/custom_templates.html#custom-css-and-javascript). \r\n\r\nUnder this API, I'd write something like this to get the above HTML rendered in Datasette.\r\n\r\n```json\r\n{\r\n \"extra_js_urls\": [\r\n {\r\n \"url\": \"/index.my-es-module-bundle.js\",\r\n \"module\": true,\r\n },\r\n {\r\n \"url\": \"/index.my-legacy-fallback-bundle.js\",\r\n \"nomodule\": \"\",\r\n \"defer\": true\r\n }\r\n ]\r\n}\r\n```\r\n\r\n## Resources\r\n\r\n- [MDN on the script tag](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/script)\r\n - There may be other properties that could be added that are potentially valuable, like `async` or `referrerpolicy`, but I don't have an immediate need for those.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1692/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1185868354, "node_id": "I_kwDOBm6k_c5GrupC", "number": 1695, "title": "Option to un-filter facet not shown for `?col__exact=value`", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2022-03-30T04:44:02Z", "updated_at": "2022-03-30T04:46:18Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Spotted this on a page with `COUNTY__exact=Lee` in the URL:\r\n\r\n![CleanShot 2022-03-29 at 21 41 46@2x](https://user-images.githubusercontent.com/9599/160752849-a9039343-3770-4655-920b-f19e25687a57.png)\r\n\r\nWith `COUNTY=Lee` you get this instead:\r\n\r\n\"image\"\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1695/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1182141761, "node_id": "I_kwDOBm6k_c5Gdg1B", "number": 1690, "title": "Idea: `datasette.set_actor_cookie(response, actor)`", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2022-03-26T22:41:52Z", "updated_at": "2022-03-26T22:43:00Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "I just wrote this code in a plugin and it felt like it could benefit from an abstraction: https://github.com/simonw/datasette-auth0/blob/152e6eb21e96e9b73bd9c205f9749a1297d0ef0b/datasette_auth0/__init__.py#L79-L92\r\n\r\n```python\r\n redirect_response = Response.redirect(\"/\")\r\n expires_at = int(time.time()) + (24 * 60 * 60)\r\n redirect_response.set_cookie(\r\n \"ds_actor\",\r\n datasette.sign(\r\n {\r\n \"a\": profile_response.json(),\r\n \"e\": baseconv.base62.encode(expires_at),\r\n },\r\n \"actor\",\r\n ),\r\n )\r\n return redirect_response\r\n```\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1690/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1181364043, "node_id": "I_kwDOBm6k_c5Gai9L", "number": 1687, "title": "Make show_json.html or a similar mechanism stable for plugins", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-03-25T23:42:45Z", "updated_at": "2022-03-25T23:42:45Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "I used `show_json.html` in the new `datasette-packages` plugin, which means it will break if that template changes:\r\n- https://github.com/simonw/datasette-packages/issues/3\r\n\r\nIt would be useful if it (or something like it) was documented and stable for plugins to use.\r\n\r\nAlso relevant:\r\n- #878", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1687/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1181236173, "node_id": "I_kwDOCGYnMM5GaDvN", "number": 422, "title": "Reconsider not running convert functions against null values", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-03-25T20:22:40Z", "updated_at": "2022-03-25T20:23:21Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "I just got caught out by the fact that `None` values are not processed by the `.convert()` mechanism https://github.com/simonw/sqlite-utils/blob/0b7b80bd40fe86e4d66a04c9f607d94991c45c0b/sqlite_utils/db.py#L2504-L2510\r\n\r\nI had run this code while working on #420 and I wasn't sure why it didn't work:\r\n\r\n```\r\n$ sqlite-utils add-column content.db articles score float\r\n$ sqlite-utils convert content.db articles score '\r\nimport random\r\nrandom.seed(10)\r\n\r\ndef convert(value):\r\n global random\r\n return random.random()\r\n'\r\n```\r\nThe reason it didn't work is that the newly added `score` column was full of `null` values.\r\n\r\nI fixed it by doing this instead:\r\n\r\n $ sqlite-utils add-column content.db articles score float --not-null-default 1.0\r\n\r\nBut this indicates to me that the design of `convert()` here may be incorrect.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/422/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1181037277, "node_id": "I_kwDOBm6k_c5GZTLd", "number": 1686, "title": "heroku bails if app name specifed in datasette publish is the same as existing app", "user": {"value": 2115933, "label": "tlongers"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-03-25T17:10:34Z", "updated_at": "2022-03-25T17:10:34Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Seem that `heroku` does not accept an app overwrite triggered by specifying the app name using `datasette publish`, as below:\r\n\r\n```\r\ndatasette publish heroku some.db --name \"jazzy-name\" \r\n```\r\n\r\nThe resulting error has the below traceback:\r\n\r\n```\r\nCreating jazzy-name... !\r\n \u25b8 Name jazzy-name is already taken\r\nTraceback (most recent call last):\r\n File \"/opt/homebrew/bin/datasette\", line 33, in \r\n sys.exit(load_entry_point('datasette==0.60.1', 'console_scripts', 'datasette')())\r\n File \"/opt/homebrew/Cellar/datasette/0.60.1/libexec/lib/python3.10/site-packages/click/core.py\", line 1128, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/opt/homebrew/Cellar/datasette/0.60.1/libexec/lib/python3.10/site-packages/click/core.py\", line 1053, in main\r\n rv = self.invoke(ctx)\r\n File \"/opt/homebrew/Cellar/datasette/0.60.1/libexec/lib/python3.10/site-packages/click/core.py\", line 1659, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/opt/homebrew/Cellar/datasette/0.60.1/libexec/lib/python3.10/site-packages/click/core.py\", line 1659, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/opt/homebrew/Cellar/datasette/0.60.1/libexec/lib/python3.10/site-packages/click/core.py\", line 1395, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/opt/homebrew/Cellar/datasette/0.60.1/libexec/lib/python3.10/site-packages/click/core.py\", line 754, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/opt/homebrew/Cellar/datasette/0.60.1/libexec/lib/python3.10/site-packages/datasette/publish/heroku.py\", line 127, in heroku\r\n create_output = check_output(cmd).decode(\"utf8\")\r\n File \"/opt/homebrew/Cellar/python@3.10/3.10.2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py\", line 420, in check_output\r\n return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,\r\n File \"/opt/homebrew/Cellar/python@3.10/3.10.2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py\", line 524, in run\r\n raise CalledProcessError(retcode, process.args,\r\nsubprocess.CalledProcessError: Command '['heroku', 'apps:create', 'jazzy-name', '--json']' returned non-zero exit status 1.\r\n```\r\n\r\nIt's a solid failsafe, but does `datasette publish` have a way to force an overwrite?", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1686/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1179998071, "node_id": "I_kwDOBm6k_c5GVVd3", "number": 1684, "title": "Mechanism for disabling faceting on large tables only", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-03-24T20:06:11Z", "updated_at": "2022-03-24T20:13:19Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Forest turned off faceting on https://labordata.bunkum.us/ because it was causing performance problems on some of the huge tables - but it would be nice if it could still be an option on smaller tables such as https://labordata.bunkum.us/voluntary_recognitions-4421085/voluntary_recognitions\r\n\r\nOne option: a new setting that automatically disables faceting (and facet suggestion) for tables that have either more than X rows or that are so big that the count could not be completed within the time limit.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1684/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1177101697, "node_id": "I_kwDOBm6k_c5GKSWB", "number": 1681, "title": "Potential bug in numeric handling where_clause for filters", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2022-03-22T17:43:50Z", "updated_at": "2022-03-22T17:49:09Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "> Note that Datasette does already have special logic to convert parameters to integers for numeric comparisons like `>`:\r\n>\r\n> https://github.com/simonw/datasette/blob/c4c9dbd0386e46d2bf199f0ed34e4895c98cb78c/datasette/filters.py#L203-L212\r\n> \r\n> Though... it looks like there's a bug in that? It doesn't account for `float` values - `\"3.5\".isdigit()` return `False` - probably for the best, because `int(3.5)` would break that value anyway.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1671#issuecomment-1075432283_", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1681/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1174655187, "node_id": "I_kwDOBm6k_c5GA9DT", "number": 1671, "title": "Filters fail to work correctly against calculated numeric columns returned by SQL views because type affinity rules do not apply", "user": {"value": 9308268, "label": "rayvoelker"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 8, "created_at": "2022-03-20T19:17:24Z", "updated_at": "2022-03-22T17:43:12Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I found a strange behavior, and I'm not sure if it's related to views and boolean values perhaps, or if there's something else weird going on here, but I'll provide an example that may help show what I'm seeing happen.\r\n\r\n```bash\r\n#!/bin/bash\r\n\r\necho \"\\\"id\\\",\\\"expiration_date\\\"\r\n0,2018-01-04\r\n1,2019-01-05\r\n2,2020-01-06\r\n3,2021-01-07\r\n4,2022-01-08\r\n5,2023-01-09\r\n6,2024-01-10\r\n7,2025-01-11\r\n8,2026-01-12\r\n9,2027-01-13\r\n\" > test.csv\r\ncsvs-to-sqlite test.csv test.db\r\nsqlite-utils create-view --replace test.db test_view \"select id, expiration_date, case when julianday('NOW') >= julianday(expiration_date) then 1 else 0 end as has_expired FROM test\"\r\n```\r\n\r\n```bash\r\ndatasette test.db\r\n```\r\n![image](https://user-images.githubusercontent.com/9308268/159178745-9c6152f7-eac6-4bf9-bef5-a2d63d3ee13f.png)\r\n\r\n![image](https://user-images.githubusercontent.com/9308268/159178824-c8952137-270c-42a4-ad1c-f6ad2c51e499.png)\r\n\r\n![image](https://user-images.githubusercontent.com/9308268/159178877-23e00b36-443a-43ef-83e5-e0bdddd3fdcd.png)\r\n\r\n![image](https://user-images.githubusercontent.com/9308268/159178918-65922cc7-2514-4735-a72d-4904b99976d4.png)\r\n\r\nThanks again and let me know if you want me to provide anything else!", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1671/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 953218043, "node_id": "MDU6SXNzdWU5NTMyMTgwNDM=", "number": 1403, "title": "Labels explaining what hidden tables are for", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-07-26T19:29:22Z", "updated_at": "2022-03-21T22:20:37Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "A reasonable question: \"What are those hidden tables for?\"\r\n\r\nThis could be answered by adding a small piece of explanatory text to each table - based on if it's related to FTS or to SpatiaLite or configured to be hidden for some other reason.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1403/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1175894898, "node_id": "I_kwDOBm6k_c5GFrty", "number": 1680, "title": "Consider simplifying permissions for 1.0", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 0, "created_at": "2022-03-21T20:17:29Z", "updated_at": "2022-03-21T20:17:29Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Permission checks right now can express one of three opinions:\r\n\r\n- `False` means \"so not grant this permisson\"\r\n- `True` means \"grant this permission\"\r\n- `None` means \"I have no opinion\"\r\n\r\nBut... there's also a concept of a \"default\" for a given permission check, which might be `False` or `True`.\r\n\r\nI worry this is too complicated. Could this be simplified before 1.0? In particular the default concept.\r\n\r\nSee also:\r\n- #1676 ", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1680/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 780153562, "node_id": "MDU6SXNzdWU3ODAxNTM1NjI=", "number": 1177, "title": "Ability to stream all rows as newline-delimited JSON", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 1, "created_at": "2021-01-06T07:10:48Z", "updated_at": "2022-03-21T15:08:52Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "> Yet another use-case for this: I want to be able to stream newline-delimited JSON in order to better import into Pandas:\r\n> \r\n> pandas.read_json(\"https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_nl=on\", lines=True)\r\n_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1101#issuecomment-755128038_", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1177/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1174717287, "node_id": "I_kwDOBm6k_c5GBMNn", "number": 1674, "title": "Tweak design of /.json", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 1, "created_at": "2022-03-20T22:58:01Z", "updated_at": "2022-03-20T22:58:40Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "https://latest.datasette.io/.json\r\n\r\nCurrently:\r\n```json\r\n{\r\n \"_memory\": {\r\n \"name\": \"_memory\",\r\n \"hash\": null,\r\n \"color\": \"a6c7b9\",\r\n \"path\": \"/_memory\",\r\n \"tables_and_views_truncated\": [],\r\n \"tables_and_views_more\": false,\r\n \"tables_count\": 0,\r\n \"table_rows_sum\": 0,\r\n \"show_table_row_counts\": false,\r\n \"hidden_table_rows_sum\": 0,\r\n \"hidden_tables_count\": 0,\r\n \"views_count\": 0,\r\n \"private\": false\r\n },\r\n \"fixtures\": {\r\n \"name\": \"fixtures\",\r\n \"hash\": \"645005884646eb941c89997fbd1c0dd6be517cb1b493df9816ae497c0c5afbaa\",\r\n \"color\": \"645005\",\r\n \"path\": \"/fixtures\",\r\n \"tables_and_views_truncated\": [\r\n {\r\n \"name\": \"compound_three_primary_keys\",\r\n \"columns\": [\r\n \"pk1\",\r\n \"pk2\",\r\n \"pk3\",\r\n \"content\"\r\n ],\r\n \"primary_keys\": [\r\n \"pk1\",\r\n \"pk2\",\r\n \"pk3\"\r\n ],\r\n \"count\": 1001,\r\n \"hidden\": false,\r\n \"fts_table\": null,\r\n \"num_relationships_for_sorting\": 0,\r\n \"private\": false\r\n },\r\n```\r\nAs-of this issue the `\"path\"` key is confusing, it doesn't match what https://latest.datasette.io/-/databases returns:\r\n\r\n- #1668", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1674/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 910088936, "node_id": "MDU6SXNzdWU5MTAwODg5MzY=", "number": 1355, "title": "datasette --get should efficiently handle streaming CSV", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-06-03T04:40:40Z", "updated_at": "2022-03-20T22:38:53Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "It would be great if you could use `datasette --get` to run queries that return streaming CSV data without running out of RAM.\r\n\r\nCurrent implementation looks like it loads the entire result into memory first: https://github.com/simonw/datasette/blob/f78ebdc04537a6102316d6dbbf6c887565806078/datasette/cli.py#L546-L552", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1355/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1174708375, "node_id": "I_kwDOBm6k_c5GBKCX", "number": 1673, "title": "Streaming CSV spends a lot of time in `table_column_details`", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-03-20T22:25:28Z", "updated_at": "2022-03-20T22:34:06Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "At least I think it does. I tried running `py-spy top -p $PID` against a Datasette process that was trying to do:\r\n\r\n datasette covid.db --get '/covid/ny_times_us_counties.csv?_size=10&_stream=on'\r\n\r\nWhile investigating:\r\n- #1355\r\n\r\nAnd spotted this:\r\n```\r\ndatasette covid.db --get /covid/ny_times_us_counties.csv?_size=10&_stream=on' (python v3.10.2)\r\nTotal Samples 5800\r\nGIL: 71.00%, Active: 98.00%, Threads: 4\r\n\r\n %Own %Total OwnTime TotalTime Function (filename:line) \r\n 8.00% 8.00% 4.32s 4.38s sql_operation_in_thread (datasette/database.py:212)\r\n 5.00% 5.00% 3.77s 3.93s table_column_details (datasette/utils/__init__.py:614)\r\n 6.00% 6.00% 3.72s 3.72s _worker (concurrent/futures/thread.py:81)\r\n 7.00% 7.00% 2.98s 2.98s _read_from_self (asyncio/selector_events.py:120)\r\n 5.00% 6.00% 2.35s 2.49s detect_fts (datasette/utils/__init__.py:571)\r\n 4.00% 4.00% 1.34s 1.34s _write_to_self (asyncio/selector_events.py:140)\r\n```\r\nRelevant code: https://github.com/simonw/datasette/blob/798f075ef9b98819fdb564f9f79c78975a0f71e8/datasette/utils/__init__.py#L609-L625\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1673/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1174697144, "node_id": "I_kwDOBm6k_c5GBHS4", "number": 1672, "title": "Refactor CSV handling code out of DataView", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 1, "created_at": "2022-03-20T21:47:00Z", "updated_at": "2022-03-20T21:52:39Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "> I think the way to get rid of most of the remaining complexity in `DataView` is to refactor how CSV stuff works - pulling it in line with other export factors and extracting the streaming mechanism. Opening a fresh issue for that.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1660#issuecomment-1073355032_", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1672/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 688351054, "node_id": "MDU6SXNzdWU2ODgzNTEwNTQ=", "number": 140, "title": "Idea: insert-files mechanism for adding extra columns with fixed values", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-08-28T20:57:36Z", "updated_at": "2022-03-20T19:45:45Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Say for example you want to populate a `file_type` column with the value `gif`. That could work like this:\r\n\r\n```\r\nsqlite-utils insert-files gifs.db images *.gif \\\r\n -c path -c md5 -c last_modified:mtime \\\r\n -c file_type:text:gif --pk=path\r\n```\r\nSo a column defined as a `text` column with a value that follows a second colon.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/140/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1123393829, "node_id": "I_kwDODFE5qs5C9aEl", "number": 10, "title": "sqlite3.OperationalError: no such table: main.my_activity", "user": {"value": 69208826, "label": "glxblt14"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-02-03T17:59:29Z", "updated_at": "2022-03-20T02:38:07Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Hello,\r\nWhen i run the command `google-takeout-to-sqlite my-activity db.db takeout-20220203T174446Z-001.zip`, i get this error :\r\n```\r\nTraceback (most recent call last):\r\n File \"c:\\users\\julie\\appdata\\local\\programs\\python\\python39-32\\lib\\runpy.py\", line 197, in _run_module_as_main\r\n return _run_code(code, main_globals, None,\r\n File \"c:\\users\\julie\\appdata\\local\\programs\\python\\python39-32\\lib\\runpy.py\", line 87, in _run_code\r\n exec(code, run_globals)\r\n File \"C:\\Users\\julie\\AppData\\Local\\Programs\\Python\\Python39-32\\Scripts\\google-takeout-to-sqlite.exe\\__main__.py\", line 7, in \r\n File \"c:\\users\\julie\\appdata\\local\\programs\\python\\python39-32\\lib\\site-packages\\click\\core.py\", line 1128, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"c:\\users\\julie\\appdata\\local\\programs\\python\\python39-32\\lib\\site-packages\\click\\core.py\", line 1053, in main\r\n rv = self.invoke(ctx)\r\n File \"c:\\users\\julie\\appdata\\local\\programs\\python\\python39-32\\lib\\site-packages\\click\\core.py\", line 1659, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"c:\\users\\julie\\appdata\\local\\programs\\python\\python39-32\\lib\\site-packages\\click\\core.py\", line 1395, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"c:\\users\\julie\\appdata\\local\\programs\\python\\python39-32\\lib\\site-packages\\click\\core.py\", line 754, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"c:\\users\\julie\\appdata\\local\\programs\\python\\python39-32\\lib\\site-packages\\google_takeout_to_sqlite\\cli.py\", line 31, in my_activity\r\n utils.save_my_activity(db, zf)\r\n File \"c:\\users\\julie\\appdata\\local\\programs\\python\\python39-32\\lib\\site-packages\\google_takeout_to_sqlite\\utils.py\", line 19, in save_my_activity\r\n db[\"my_activity\"].create_index([\"time\"])\r\n File \"c:\\users\\julie\\appdata\\local\\programs\\python\\python39-32\\lib\\site-packages\\sqlite_utils\\db.py\", line 629, in create_index\r\n self.db.conn.execute(sql)\r\nsqlite3.OperationalError: no such table: main.my_activity\r\n```\r\nThank you for your help\r\nSorry for my bad English\r\nEDIT: i used the json format", "repo": {"value": 206649770, "label": "google-takeout-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/10/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 648435885, "node_id": "MDU6SXNzdWU2NDg0MzU4ODU=", "number": 878, "title": "New pattern for views that return either JSON or HTML, available for plugins", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 26, "created_at": "2020-06-30T19:26:13Z", "updated_at": "2022-03-19T16:19:30Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Can be part of #870 - refactoring existing views to use `register_routes()`.\r\n\r\n> I'm going to put the new `check_permissions()` method on `BaseView` as well. If I want that method to be available to plugins I can do so by turning that `BaseView` class into a documented API that plugins are encouraged to use themselves.\r\n_Originally posted by @simonw in https://github.com/simonw/datasette/issues/832#issuecomment-651995453_", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/878/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1065429936, "node_id": "I_kwDOBm6k_c4_gSuw", "number": 1532, "title": "Use datasette-table Web Component to guide the design of the JSON API for 1.0", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 4, "created_at": "2021-11-28T20:37:18Z", "updated_at": "2022-03-16T20:13:34Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "I realized that one of the reasons I'm having trouble committing to nailing down the JSON API for 1.0 is that I don't use it much myself - I use the `?_shape=array` one quite often, but I don't have any projects that are using the default, more fully-featured API.\r\n\r\nAs an experiment I built a Web Component for embedding Datasette tables on pages - https://github.com/simonw/datasette-table - and I think it's actually going to be a really useful tool for helping me dog food the v1.0 API design.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1532/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 531502365, "node_id": "MDU6SXNzdWU1MzE1MDIzNjU=", "number": 646, "title": "Make database level information from metadata.json available in the index.html template", "user": {"value": 18017473, "label": "lagolucas"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 3, "created_at": "2019-12-02T19:55:10Z", "updated_at": "2022-03-15T20:50:34Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Did a search on the issues here and didn't find anything related to what I want.\r\n\r\nI want to have information that is on the database level of the JSON like title, source and source_url, and use it on the index page.\r\n\r\nI tried some small tweaks on the python and html files, but failed to get that result.\r\n\r\nIs there a way? Thanks!", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/646/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 626593402, "node_id": "MDU6SXNzdWU2MjY1OTM0MDI=", "number": 780, "title": "Internals documentation for datasette.metadata() method", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 2, "created_at": "2020-05-28T15:14:22Z", "updated_at": "2022-03-15T20:50:34Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "https://github.com/simonw/datasette/blob/40885ef24e32d91502b6b8bbad1c7376f50f2830/datasette/app.py#L297-L328", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/780/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1054243511, "node_id": "I_kwDOBm6k_c4-1nq3", "number": 1509, "title": "Datasette 1.0 JSON API (and documentation)", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 3, "created_at": "2021-11-15T23:22:45Z", "updated_at": "2022-03-15T20:38:56Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "The new JSON API in a stable, documented form.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1509/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 646737558, "node_id": "MDU6SXNzdWU2NDY3Mzc1NTg=", "number": 870, "title": "Refactor default views to use register_routes", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 10, "created_at": "2020-06-27T18:53:12Z", "updated_at": "2022-03-15T20:07:18Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "It would be much cleaner if Datasette's default views were all registered using the new `register_routes()` plugin hook. Could dramatically reduce the code in `datasette/app.py`.\r\n\r\n> The ideal fix here would be to rework my `BaseView` subclass mechanism to work with `register_routes()` so that those views don't have any special privileges above plugin-provided views.\r\n_Originally posted by @simonw in https://github.com/simonw/datasette/issues/864#issuecomment-648580556_", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/870/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1058072543, "node_id": "I_kwDOBm6k_c4_EOff", "number": 1518, "title": "Complete refactor of TableView and table.html template", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 45, "created_at": "2021-11-19T02:55:16Z", "updated_at": "2022-03-15T18:35:49Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Split from #878. The current `TableView` class is by far the most complex part of Datasette, and the most difficult to work on: https://github.com/simonw/datasette/blob/0.59.2/datasette/views/table.py\r\n\r\nIn #878 I started exploring a new pattern for building views. In doing so it became clear that `TableView` is the first beast that I need to slay - if I can refactor that into something neat the pattern for building other views will emerge as a natural consequence.\r\n\r\nI've been trying to build this as a `register_routes()` plugin, as originally suggested in #870 - though unfortunately it looks like those plugins can't replace existing Datasette default views at the moment, see #1517. [UPDATE: I was wrong about this, plugins can over-ride default views just fine]\r\n\r\nI also know that I want to have a fully documented template context for `table.html` as a major step on the way to Datasette 1.0, see #1510.\r\n\r\nAll of this adds up to the `TableView` factor being a major project that will unblock a whole flurry of other things - so I'm going to work on that in this separate issue.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1518/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 675753042, "node_id": "MDU6SXNzdWU2NzU3NTMwNDI=", "number": 131, "title": "sqlite-utils insert: options for column types", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2020-08-09T18:59:11Z", "updated_at": "2022-03-15T13:21:42Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "The `insert` command currently results in string types for every column - at least when used against CSV or TSV inputs.\r\n\r\nIt would be useful if you could do the following:\r\n\r\n- automatically detects the column types based on eg the first 1000 records\r\n- explicitly state the rule for specific columns\r\n\r\n`--detect-types` could work for the former - or it could do that by default and allow opt-out using `--no-detect-types`\r\n\r\nFor specific columns maybe this:\r\n\r\n sqlite-utils insert db.db images images.tsv \\\r\n --tsv \\\r\n -c id int \\\r\n -c score float", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/131/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 930807135, "node_id": "MDU6SXNzdWU5MzA4MDcxMzU=", "number": 1384, "title": "Plugin hook for dynamic metadata", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 22, "created_at": "2021-06-26T22:36:03Z", "updated_at": "2022-03-14T00:36:42Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "@brandonrobertz contributed an implementation of this in PR #1368, which I just merged. Opening this ticket to track further work on this before it goes out in a Datasette release (likely preceded by an alpha).", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1384/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1160034488, "node_id": "I_kwDOCGYnMM5FJLi4", "number": 411, "title": "Support for generated columns", "user": {"value": 25778, "label": "eyeseast"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 8, "created_at": "2022-03-04T20:41:33Z", "updated_at": "2022-03-11T22:32:43Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "This is a fairly new feature -- SQLite version 3.31.0 (2020-01-22) -- that I, admittedly, haven't gotten to work yet. But it looks _incredibly_ useful: https://dgl.cx/2020/06/sqlite-json-support\r\n\r\nI'm not sure if this is an option on `add-column` or a separate command like `add-generated-column`. Either way, it needs an argument to populate it. It could be something like this:\r\n\r\n```sh\r\nsqlite-utils add-column data.db table-name generated --as 'json_extract(data, \"$.field\")' --virtual\r\n```\r\n\r\nMore here: https://www.sqlite.org/gencol.html", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/411/reactions\", \"total_count\": 2, \"+1\": 2, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1131295060, "node_id": "I_kwDOBm6k_c5DbjFU", "number": 1634, "title": "Update Dockerfile generated by `datasette publish`", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 4, "created_at": "2022-02-11T00:07:26Z", "updated_at": "2022-03-11T17:38:08Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "The generated `Dockerfile` currently looks something like this:\r\n```Dockerfile\r\nFROM python:3.8\r\nCOPY . /app\r\nWORKDIR /app\r\n\r\nENV DATASETTE_SECRET 'edab49cbc5d5f6f33238f54852037e3fee710821960b73edd2ce743454182ae2'\r\nRUN pip install -U datasette datasette-auth-passwords datasette-tiddlywiki datasette-graphql\r\nRUN datasette inspect fixtures.db other.db --inspect-file inspect-data.json\r\nENV PORT 8080\r\nEXPOSE 8080\r\nCMD datasette serve --host 0.0.0.0 -i fixtures.db -i other.db --cors --inspect-file inspect-data.json --metadata metadata.json --create --port $PORT /data/*.db\r\n```\r\nThis is still on Python 3.8, and it generates a pretty large image compared to the `Dockerfile` used for https://hub.docker.com/datasetteproject/datasette - https://github.com/simonw/datasette/blob/0.60.2/Dockerfile\r\n\r\nHere's the code that generates it: https://github.com/simonw/datasette/blob/7d24fd405f3c60e4c852c5d746c91aa2ba23cf5b/datasette/utils/__init__.py#L389-L400", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1634/reactions\", \"total_count\": 2, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 2, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 678760988, "node_id": "MDU6SXNzdWU2Nzg3NjA5ODg=", "number": 932, "title": "End-user documentation", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 6, "created_at": "2020-08-13T22:04:39Z", "updated_at": "2022-03-08T15:20:48Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Datasette's documentation is aimed at people who install and configure it.\r\n\r\nWhat about end users of preconfigured and deployed Datasette instances?\r\n\r\nSomething that can be linked to from the Datasette UI would be really useful.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/932/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1154399841, "node_id": "I_kwDOBm6k_c5Ezr5h", "number": 1645, "title": "Sensible `cache-control` headers for static assets, including those served by plugins", "user": {"value": 697092, "label": "curiousleo"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 4, "created_at": "2022-02-28T18:12:03Z", "updated_at": "2022-03-08T02:59:29Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "## What I'm seeing\r\n\r\nWith `default_cache_ttl = 86400`, I see the following:\r\n\r\nA table view returns `Cache-control: max-age=86400`:\r\n\r\n![Screenshot_20220228_190000](https://user-images.githubusercontent.com/697092/156034352-4d64683e-39c8-49af-81df-0217a5957bbd.png)\r\n\r\nA static asset returns no `Cache-control` header:\r\n\r\n![Screenshot_20220228_185933](https://user-images.githubusercontent.com/697092/156034363-d0b03cc2-5889-4ed2-b601-8c1846b8469a.png)\r\n\r\n## What I expected to see\r\n\r\nI expected the static asset to return a `Cache-control` header indicating that this response can be cached.\r\n\r\n## Why this matters\r\n\r\nI'm productionising a Datasette deployment right now and was looking into putting it behind a Varnish instance. I was surprised to see requests for static assets being served from Datasette rather than Varnish, this is what led me to look more closely at the response headers.\r\n\r\nWhile Datasette serves those static assets pretty quickly, I don't see why Datasette should serve them. By their nature, static assets like images and JS files are very cacheable, so it should be easy to serve them from a cache like Varnish.\r\n\r\n(Note that Varnish can easily be configured to override this header, enabling caching for static assets. But it would be better if this override was not necessary.)\r\n\r\n## Discussion\r\n\r\nIt seems clear to me that serving static assets without a `Cache-control` header is not ideal.\r\n\r\nI see two options here:\r\n\r\nA. Static assets use the same logic as table / SQL views to set the `Cache-control` header based on `default_cache_ttl`.\r\nB. An additional setting for static assets is introduced (`default_static_cache_ttl`, say).", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1645/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1161937073, "node_id": "I_kwDOBm6k_c5FQcCx", "number": 1653, "title": "Mechanism to default a table to sorting by multiple columns", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2022-03-07T21:20:11Z", "updated_at": "2022-03-07T21:23:39Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "### Discussed in https://github.com/simonw/datasette/discussions/1652\r\n\r\n
\r\n\r\nOriginally posted by **zaneselvans** March 7, 2022\r\nIt's easy to tell datasette to sort tables using a single column, as [described in the docs](https://docs.datasette.io/en/stable/metadata.html#setting-a-default-sort-order):\r\n\r\n```yaml\r\ndatabases:\r\n ferc1:\r\n tables:\r\n f1_edcfu_epda:\r\n sort: created_time\r\n```\r\n\r\nBut is there some way to tell it to sort using a composite key, like you would in an `ORDER BY` clause instead? For example, the way it's being done **[in this query](https://data.catalyst.coop/ferc1?sql=select%0D%0A++rowid%2C%0D%0A++respondent_id%2C%0D%0A++report_year%2C%0D%0A++spplmnt_num%2C%0D%0A++row_number%2C%0D%0A++row_seq%2C%0D%0A++row_prvlg%2C%0D%0A++acct_num%2C%0D%0A++depr_plnt_base%2C%0D%0A++est_avg_srvce_lf%2C%0D%0A++net_salvage%2C%0D%0A++apply_depr_rate%2C%0D%0A++mrtlty_crv_typ%2C%0D%0A++avg_remaining_lf%2C%0D%0A++report_prd%0D%0Afrom%0D%0A++f1_edcfu_epda%0D%0Awhere%0D%0A++respondent_id+%3D+210%0D%0A++AND+report_year+%3D+2020%0D%0Aorder+by%0D%0A++report_year%2C+report_prd%2C+respondent_id%2C+spplmnt_num%2C+row_number%0D%0Alimit%0D%0A++1000)** on our Datasette?\r\n\r\n```sql\r\nSELECT\r\n respondent_id,\r\n report_year,\r\n spplmnt_num,\r\n row_number,\r\n row_seq,\r\n row_prvlg,\r\n acct_num,\r\n depr_plnt_base,\r\n est_avg_srvce_lf,\r\n net_salvage,\r\n apply_depr_rate,\r\n mrtlty_crv_typ,\r\n avg_remaining_lf,\r\n report_prd\r\nFROM\r\n f1_edcfu_epda\r\nWHERE\r\n respondent_id = 210\r\n AND report_year = 2020\r\nORDER BY\r\n report_year, report_prd, respondent_id, spplmnt_num, row_number\r\nLIMIT\r\n 1000\r\n```\r\n\r\nThe problem here is that by default it's using `rowid` (the SQLite assigned autoincrementing integer key) to order the records, but the table **should** have a natural composite primary key, but the original database that this data is being migrated from doesn't enforce unique primary keys, so there are dupes, and we don't want to drop those rows, and the records are somehow getting jumbled in the database (the `rowid` ordering isn't lined up with the expected ordering based on the composite primary key, though it's close) and this jumbling is confusing to users that expect to see the data ordered based on the natural primary key.\r\n\r\nI've tried setting the `sort` metadata parameter to a list of column names, a tuple of column names, a quoted string of comma-separated column names, a quoted string of a tuple of column names...\r\n\r\n```yaml\r\ndatabases:\r\n ferc1:\r\n tables:\r\n f1_edcfu_epda:\r\n sort: \"(report_year, report_prd, respondent_id, spplmnt_num, row_number)\"\r\n```\r\n\r\nand they all give me server errors like:\r\n\r\n```\r\nCannot sort table by (report_year, report_prd, respondent_id, spplmnt_num, row_number)\r\n```
", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1653/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1160327106, "node_id": "PR_kwDODEm0Qs4z_V3w", "number": 65, "title": "Update Twitter dev link, clarify apps vs projects", "user": {"value": 2657547, "label": "rixx"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-03-05T11:56:08Z", "updated_at": "2022-03-05T11:56:08Z", "closed_at": null, "author_association": "FIRST_TIME_CONTRIBUTOR", "pull_request": "dogsheep/twitter-to-sqlite/pulls/65", "body": "Twitter pushes you heavily towards v2 projects instead of v1 apps \u2013 I know the README mentions v1 API compatibility at the top, but I still nearly got turned around here.", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/65/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1148725876, "node_id": "I_kwDOBm6k_c5EeCp0", "number": 1640, "title": "Support static assets where file length may change, e.g. logs", "user": {"value": 57859326, "label": "broccolihighkicks"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2022-02-24T00:34:42Z", "updated_at": "2022-03-05T01:19:25Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "This is a bit of an oxymoron.\r\n\r\nI am serving a log.txt file for a background process using the Datasette --static CLI. This is useful as I can observe a background process from the web UI to see any errors that occur (instead of spelunking the logs via docker exec/ssh etc).\r\n\r\nI get this error, which I think is because Datasette assumes that the size of the content does not change (but appending new log lines means the content length changes).\r\n\r\n\r\n```python\r\nTraceback (most recent call last):\r\n File \"/usr/local/lib/python3.9/site-packages/datasette/app.py\", line 1181, in route_path\r\n response = await view(request, send)\r\n File \"/usr/local/lib/python3.9/site-packages/datasette/utils/asgi.py\", line 305, in inner_static\r\n await asgi_send_file(send, full_path, chunk_size=chunk_size)\r\n File \"/usr/local/lib/python3.9/site-packages/datasette/utils/asgi.py\", line 280, in asgi_send_file\r\n await send(\r\n File \"/usr/local/lib/python3.9/site-packages/asgi_csrf.py\", line 104, in wrapped_send\r\n await send(event)\r\n File \"/usr/local/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py\", line 460, in send\r\n output = self.conn.send(event)\r\n File \"/usr/local/lib/python3.9/site-packages/h11/_connection.py\", line 468, in send\r\n data_list = self.send_with_data_passthrough(event)\r\n File \"/usr/local/lib/python3.9/site-packages/h11/_connection.py\", line 501, in send_with_data_passthrough\r\n writer(event, data_list.append)\r\n File \"/usr/local/lib/python3.9/site-packages/h11/_writers.py\", line 58, in __call__\r\n self.send_data(event.data, write)\r\n File \"/usr/local/lib/python3.9/site-packages/h11/_writers.py\", line 78, in send_data\r\n raise LocalProtocolError(\"Too much data for declared Content-Length\")\r\nh11._util.LocalProtocolError: Too much data for declared Content-Length\r\nERROR: Exception in ASGI application\r\nTraceback (most recent call last):\r\n File \"/usr/local/lib/python3.9/site-packages/datasette/app.py\", line 1181, in route_path\r\n response = await view(request, send)\r\n File \"/usr/local/lib/python3.9/site-packages/datasette/utils/asgi.py\", line 305, in inner_static\r\n await asgi_send_file(send, full_path, chunk_size=chunk_size)\r\n File \"/usr/local/lib/python3.9/site-packages/datasette/utils/asgi.py\", line 280, in asgi_send_file\r\n await send(\r\n File \"/usr/local/lib/python3.9/site-packages/asgi_csrf.py\", line 104, in wrapped_send\r\n await send(event)\r\n File \"/usr/local/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py\", line 460, in send\r\n output = self.conn.send(event)\r\n File \"/usr/local/lib/python3.9/site-packages/h11/_connection.py\", line 468, in send\r\n data_list = self.send_with_data_passthrough(event)\r\n File \"/usr/local/lib/python3.9/site-packages/h11/_connection.py\", line 501, in send_with_data_passthrough\r\n writer(event, data_list.append)\r\n File \"/usr/local/lib/python3.9/site-packages/h11/_writers.py\", line 58, in __call__\r\n self.send_data(event.data, write)\r\n File \"/usr/local/lib/python3.9/site-packages/h11/_writers.py\", line 78, in send_data\r\n raise LocalProtocolError(\"Too much data for declared Content-Length\")\r\nh11._util.LocalProtocolError: Too much data for declared Content-Length\r\n```\r\n\r\nThanks, I am finding Datasette very useful.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1640/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1149402080, "node_id": "PR_kwDODFdgUs4zaUta", "number": 70, "title": "scrape-dependents: enable paging through package menu option if present", "user": {"value": 36061055, "label": "stanbiryukov"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-02-24T15:07:25Z", "updated_at": "2022-02-24T15:07:25Z", "closed_at": null, "author_association": "FIRST_TIME_CONTRIBUTOR", "pull_request": "dogsheep/github-to-sqlite/pulls/70", "body": "Some repos organize network dependents by a Package toggle. This PR adds the ability to page through those options and scrape underlying dependents.", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/70/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1149310456, "node_id": "I_kwDOBm6k_c5EgRX4", "number": 1641, "title": "Tweak mobile keyboard settings", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-02-24T13:47:10Z", "updated_at": "2022-02-24T13:49:26Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "https://developer.apple.com/library/archive/documentation/StringsTextFonts/Conceptual/TextAndWebiPhoneOS/KeyboardManagement/KeyboardManagement.html#//apple_ref/doc/uid/TP40009542-CH5-SW12\r\n\r\n`autocorrect=\"off\"` is worth experimenting with.\r\n\r\nTwitter: https://twitter.com/forestgregg/status/1496842959563726852", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1641/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1148638868, "node_id": "I_kwDOBm6k_c5EdtaU", "number": 1639, "title": "Make datasette-redirect-forbidden unneccessary", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-02-23T22:18:46Z", "updated_at": "2022-02-23T22:18:46Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "I wrote `datasette-redirect-forbidden` today because I needed 403 errors to redirect to `/-/login` and it was the quickest way to solve that problem.\r\n\r\nThis should be a feature of Datasette core.\r\n\r\n- https://github.com/simonw/datasette-redirect-forbidden/issues/2", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1639/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1142107925, "node_id": "I_kwDOBm6k_c5EEy8V", "number": 1638, "title": "`filters_from_request` plugin hook docs should mention that returning an async function is allowed", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-02-18T00:08:26Z", "updated_at": "2022-02-18T00:08:26Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "https://docs.datasette.io/en/stable/plugin_hooks.html#filters-from-request-request-database-table-datasette doesn't mention that you can return an `async` function - but you can, and in fact Datasette itself uses that here: https://github.com/simonw/datasette/blob/aa7f0037a46eb76ae6fe9bf2a1f616c58738ecdf/datasette/filters.py#L43-L47", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1638/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 335200136, "node_id": "MDU6SXNzdWUzMzUyMDAxMzY=", "number": 327, "title": "Explore if SquashFS can be used to shrink size of packaged Docker containers", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2018-06-24T18:15:16Z", "updated_at": "2022-02-17T23:37:24Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Inspired by this article: https://cldellow.com/2018/06/22/sqlite-parquet-vtable.html#sqlite-database-indexed--squashed\r\n\r\nhttps://en.wikipedia.org/wiki/SquashFS is \"a compressed read-only file system for Linux\" - which means it could be a really nice fit for Datasette and its read-only SQLite databases.\r\n\r\nIt would be interesting to explore a Dockerfile recipe that used SquashFS to compress the SQLite database file that was bundled up by `datasette package` and friends.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/327/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1125297737, "node_id": "I_kwDOCGYnMM5DEq5J", "number": 402, "title": "Advanced class-based `conversions=` mechanism", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 14, "created_at": "2022-02-06T19:47:41Z", "updated_at": "2022-02-16T10:18:55Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "The `conversions=` parameter works like this at the moment: https://sqlite-utils.datasette.io/en/3.23/python-api.html#converting-column-values-using-sql-functions\r\n\r\n```python\r\ndb[\"places\"].insert(\r\n {\"name\": \"Wales\", \"geometry\": wkt},\r\n conversions={\"geometry\": \"GeomFromText(?, 4326)\"},\r\n)\r\n```\r\nThis proposal is to support values in that dictionary that are objects, not strings, which can represent more complex conversions - spun out from #399.\r\n\r\nNew proposed mechanism:\r\n```python\r\nfrom sqlite_utils.utils import LongitudeLatitude\r\n\r\ndb[\"places\"].insert(\r\n {\r\n \"name\": \"London\",\r\n \"point\": (-0.118092, 51.509865)\r\n },\r\n conversions={\"point\": LongitudeLatitude},\r\n)\r\n```\r\nHere `LongitudeLatitude` is a magical value which does TWO things: it sets up the `GeomFromText(?, 4326)` SQL function, and it handles converting the `(51.509865, -0.118092)` tuple into a `POINT({} {})` string.\r\n\r\nThis would involve a change to the `conversions=` contract - where it usually expects a SQL string fragment, but it can also take an object which combines that SQL string fragment with a Python conversion function.\r\n\r\nBest of all... this resolves the `lat, lon` v.s. `lon, lat` dilemma because you can use `from sqlite_utils.utils import LongitudeLatitude` OR `from sqlite_utils.utils import LatitudeLongitude` depending on which you prefer!\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030739566_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/402/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 764059235, "node_id": "MDU6SXNzdWU3NjQwNTkyMzU=", "number": 1143, "title": "More flexible CORS support in core, to encourage good security practices", "user": {"value": 114388, "label": "yurivish"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 6, "created_at": "2020-12-12T17:06:35Z", "updated_at": "2022-02-13T17:41:17Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "It would be nice if the `--cors` option accepted an origin regex to more securely allow secure local development.\r\n\r\nAs an example, Observable notebooks namespace every user's notebooks by their username and user content is served from username.observableusercontent.com, so you would set `--cors-origin username.observableusercontent.com` to restrict access to a local development Datasette instance to only your own notebooks, rather than exposing the data to any website that makes a request.\r\n\r\nThank you for all of your work on Datasette!", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1143/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1125576543, "node_id": "I_kwDOBm6k_c5DFu9f", "number": 1630, "title": "Review datasette.utils and decide which functions should be documented for 1.0", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 0, "created_at": "2022-02-07T06:39:52Z", "updated_at": "2022-02-07T06:39:52Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Follows:\r\n- #1176", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1630/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 734777631, "node_id": "MDU6SXNzdWU3MzQ3Nzc2MzE=", "number": 1080, "title": "\"View all\" option for facets, to provide a (paginated) list of ALL of the facet counts plus a link to view them", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 7, "created_at": "2020-11-02T19:55:06Z", "updated_at": "2022-02-04T06:25:18Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Can use `/database/-/...` namespace from #296", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1080/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1072792507, "node_id": "I_kwDOCGYnMM4_8YO7", "number": 352, "title": "`sqlite-utils insert --extract colname`", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-12-07T00:55:44Z", "updated_at": "2022-02-03T22:59:36Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Is there a reason I've not added `--extract` as an option for `sqlite-utils insert` next? There's a `extracts=` option for the various `table.insert()` etc methods - last line in this code block:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/213a0ff177f23a35f3b235386366ff132eb879f1/sqlite_utils/db.py#L2483-L2495", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/352/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1122446693, "node_id": "I_kwDOCGYnMM5C5y1l", "number": 394, "title": "Test against Python 3.11-dev", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-02-02T22:21:03Z", "updated_at": "2022-02-03T21:06:35Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Same as:\r\n- https://github.com/simonw/datasette/issues/1621", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/394/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1122557010, "node_id": "I_kwDOBm6k_c5C6NxS", "number": 1627, "title": "Get the tests passing against Windows", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-02-03T01:23:06Z", "updated_at": "2022-02-03T01:23:32Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "> OK, the tests do NOT pass against Windows! https://github.com/simonw/datasette/runs/5044105941\r\n>\r\n> \"image\"\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1626#issuecomment-1028515161_", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1627/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1122451096, "node_id": "PR_kwDOBm6k_c4x_mXy", "number": 1626, "title": "Try test suite against macOS and Windows", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2022-02-02T22:26:51Z", "updated_at": "2022-02-03T01:22:44Z", "closed_at": null, "author_association": "OWNER", "pull_request": "simonw/datasette/pulls/1626", "body": "Refs #1625", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1626/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1122450452, "node_id": "I_kwDOBm6k_c5C5zwU", "number": 1625, "title": "Try running tests against macOS and Windows in addition to Ubuntu", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-02-02T22:25:57Z", "updated_at": "2022-02-02T22:25:57Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "I already do this for `sqlite-utils`: https://github.com/simonw/sqlite-utils/blob/3.22.1/.github/workflows/test.yml\r\n\r\nRelated:\r\n- #1617\r\n- #1545", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1625/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1121121305, "node_id": "I_kwDOBm6k_c5C0vQZ", "number": 1618, "title": "Reconsider policy on blocking queries containing the string \"pragma\"", "user": {"value": 770231, "label": "strada"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2022-02-01T19:39:46Z", "updated_at": "2022-02-02T19:42:03Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "First of all, thanks for creating this cool project, and also supporting publishing to various hosting services out of the box.\r\n\r\nWhile testing out, I noticed legitimate queries such as \r\n```\r\nselect * from books where title like 'Pragmatic%'\r\n```\r\nor\r\n```\r\nselect * from books where title = 'The Pragmatic Programmer'\r\n```\r\nare blocked, due to the regular expression check here:\r\nhttps://github.com/simonw/datasette/blob/main/datasette/utils/__init__.py#L185\r\n\r\nExample as seen from a Datasette instance:\r\nhttps://fivethirtyeight.datasettes.com/polls?sql=select+*+from+books+where+title+like+%27Pragmatic%25%27%0D%0A\r\n\r\nI'd propose a regular expression like \r\n```\r\nre.compile(f\"pragma_(?!({'|'.join(allowed_pragmas)}))\"),\r\n```\r\ninstead of\r\n```\r\nre.compile(f\"pragma(?!_({'|'.join(allowed_pragmas)}))\"),\r\n```\r\n\r\nI can create a pull request with this change, unless the maintainers think it would allow unwanted queries to be executed.\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1618/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1096536240, "node_id": "I_kwDOBm6k_c5BW9Cw", "number": 1586, "title": "run analyze on all databases as part of start up or publishing", "user": {"value": 536941, "label": "fgregg"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-01-07T17:52:34Z", "updated_at": "2022-02-02T07:13:37Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "Running `analyze;` lets sqlite's query planner make *much* better use of any indices.\r\n\r\nIt might be nice if the analyze was run as part of the start up of \"serve\" or \"publish\".", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1586/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1113384383, "node_id": "I_kwDOBm6k_c5CXOW_", "number": 1611, "title": "Avoid ever running count(*) against SpatiaLite KNN table", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-01-25T03:32:54Z", "updated_at": "2022-02-02T06:45:47Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Got this in a trace:\r\n\r\n\"image\"\r\n\r\nLooks like running `count(*)` against KNN took 83s! It ignored the time limit. And still only returned a count of 0.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1611/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1114628238, "node_id": "I_kwDOBm6k_c5Cb-CO", "number": 1613, "title": "Improvements to help make Datasette a better tool for learning SQL", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2022-01-26T04:56:07Z", "updated_at": "2022-01-26T16:41:46Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Tracking issue for the general goal of making Datasette a better tool for learning SQL.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1613/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 328155946, "node_id": "MDU6SXNzdWUzMjgxNTU5NDY=", "number": 301, "title": "--spatialite option for \"datasette publish heroku\"", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2018-05-31T14:13:09Z", "updated_at": "2022-01-20T21:28:50Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Split off from #243. Need to figure out how to install and configure SpatiaLite on Heroku.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/301/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 838382890, "node_id": "MDU6SXNzdWU4MzgzODI4OTA=", "number": 1273, "title": "Refresh SpatiaLite documentation", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-03-23T06:05:55Z", "updated_at": "2022-01-20T21:28:50Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "https://docs.datasette.io/en/0.55/spatialite.html was written before I had tools like [geojson-to-sqlite](https://datasette.io/tools/geojson-to-sqlite) and [shapefile-to-sqlite](https://datasette.io/tools/shapefile-to-sqlite).", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1273/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 849975810, "node_id": "MDU6SXNzdWU4NDk5NzU4MTA=", "number": 1292, "title": "Research ctypes.util.find_library('spatialite')", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-04-04T22:36:59Z", "updated_at": "2022-01-20T21:28:50Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Spotted this in the Django SpatiaLite backend: https://github.com/django/django/blob/8f6a7a0e9e7c5404af6520ae606927e32415eb00/django/contrib/gis/db/backends/spatialite/base.py#L24-L36\r\n\r\n```python\r\nctypes.util.find_library('spatialite')\r\n```", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1292/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 910092577, "node_id": "MDU6SXNzdWU5MTAwOTI1Nzc=", "number": 1356, "title": "Research: syntactic sugar for using --get with SQL queries, maybe \"datasette query\"", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 10, "created_at": "2021-06-03T04:49:42Z", "updated_at": "2022-01-20T01:06:37Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Inspired by https://github.com/simonw/sqlite-utils/issues/264 - in particular this example:\r\n```\r\ndatasette covid.db --get='/covid.yaml?sql=select * from ny_times_us_counties limit 1' \r\n- date: '2020-01-21'\r\n county: Snohomish\r\n state: Washington\r\n fips: 53061\r\n cases: 1\r\n deaths: 0\r\n```\r\nHaving to construct that URL - including potentially URL escaping the SQL query - isn't a great developer experience.\r\n\r\nImagine if you could do this instead:\r\n\r\n datasette covid.db --query \"select * from ny_times_us_counties limit 1\" --format yaml\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1356/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1108300685, "node_id": "I_kwDOBm6k_c5CD1ON", "number": 1604, "title": "Option to assign a domain/subdomain using `datasette publish cloudrun`", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-01-19T16:21:17Z", "updated_at": "2022-01-19T16:23:54Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Looks like this API should be able to do that: https://twitter.com/steren/status/1483835859191304192 - https://cloud.google.com/run/docs/reference/rest/v1/namespaces.domainmappings/create", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1604/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1100015398, "node_id": "I_kwDOBm6k_c5BkOcm", "number": 1591, "title": "Maybe let plugins define custom serve options?", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2022-01-12T08:18:47Z", "updated_at": "2022-01-15T11:56:59Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "https://twitter.com/psychemedia/status/1481171650934714370\r\n\r\n> can extensions be passed their own cli args? eg `--ext-tiddlywiki-dbname tiddlywiki2.sqlite` ?\r\n\r\nI've thought something like this might be useful for other plugins in the past, too.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1591/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1102966378, "node_id": "I_kwDOBm6k_c5Bve5q", "number": 1599, "title": "Add architecture documentation", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-01-14T04:55:38Z", "updated_at": "2022-01-14T04:56:03Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Inspired by https://matklad.github.io/2021/02/06/ARCHITECTURE.md.html\r\n\r\nGood example: https://github.com/rust-analyzer/rust-analyzer/blob/d7c99931d05e3723d878bea5dc26766791fa4e69/docs/dev/architecture.md", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1599/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1102568047, "node_id": "I_kwDOBm6k_c5Bt9pv", "number": 1596, "title": "Documentation page warning of changes coming in 1.0", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-01-13T23:26:04Z", "updated_at": "2022-01-13T23:26:04Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "I should start this relatively soon.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1596/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 777140799, "node_id": "MDU6SXNzdWU3NzcxNDA3OTk=", "number": 1166, "title": "Adopt Prettier for JavaScript code formatting", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 10, "created_at": "2020-12-31T21:25:27Z", "updated_at": "2022-01-13T22:22:18Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "https://prettier.io/ - I'm going to go with 2 spaces.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1166/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 712202333, "node_id": "MDU6SXNzdWU3MTIyMDIzMzM=", "number": 982, "title": "SQL editor should allow execution of write queries, if you have permission", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-09-30T19:04:35Z", "updated_at": "2022-01-13T22:21:29Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "The `datasette-write` plugin provides this at the moment https://github.com/simonw/datasette-write - but it feels like it should be a built-in capability, protected by a default permission.\r\n\r\nUI concept: if you have write permission then the existing SQL editor gets an \"execute write\" checkbox underneath it.\r\n\r\nJavaScript can spot if you appear to be trying to execute an UPDATE or INSERT or DELETE query and check that checkbox for you.\r\n\r\nIf you link to a query page with a non-SELECT then that query will be displayed in the box ready for you to POST submit it. The page will also then get \"cannot be embedded\" headers to protect against clickjacking.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/982/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 733999615, "node_id": "MDU6SXNzdWU3MzM5OTk2MTU=", "number": 1079, "title": "Handle long breadcrumbs better with new menu", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-11-01T15:57:41Z", "updated_at": "2022-01-13T22:21:29Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "On this page when signed in as root: https://latest.datasette.io/fixtures/roadside_attraction_characteristics/1\r\n\r\n![EF921CB1-625F-4D04-A850-490B812A72B3](https://user-images.githubusercontent.com/9599/97807807-db0fbf80-1c17-11eb-9c77-ae5169b12c3d.jpeg)\r\n\r\n![A49D8B76-5ACF-4F71-A8B4-21A44F5C8D51](https://user-images.githubusercontent.com/9599/97807809-dea34680-1c17-11eb-9511-a49af56a4bd2.jpeg)\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1079/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 776634318, "node_id": "MDU6SXNzdWU3NzY2MzQzMTg=", "number": 1164, "title": "Mechanism for minifying JavaScript that ships with Datasette", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2020-12-30T20:59:06Z", "updated_at": "2022-01-13T22:21:29Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "> If I'm going to minify it I'll need to figure out a build step in Datasette itself so that I can easily work on that minified version.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/datasette/issues/983#issuecomment-752748496_", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1164/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 776635426, "node_id": "MDU6SXNzdWU3NzY2MzU0MjY=", "number": 1165, "title": "Mechanism for executing JavaScript unit tests", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2020-12-30T21:02:34Z", "updated_at": "2022-01-13T22:21:29Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "> I'm going to need to add JavaScript unit tests for this new plugin system.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/datasette/issues/983#issuecomment-752757289_", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1165/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 323718842, "node_id": "MDU6SXNzdWUzMjM3MTg4NDI=", "number": 268, "title": "Mechanism for ranking results from SQLite full-text search", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 12, "created_at": "2018-05-16T17:36:40Z", "updated_at": "2022-01-13T22:21:28Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "This isn't particularly straight-forward - all the more reason for Datasette to implement it for you. This article is helpful: http://charlesleifer.com/blog/using-sqlite-full-text-search-with-python/", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/268/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 707849175, "node_id": "MDU6SXNzdWU3MDc4NDkxNzU=", "number": 974, "title": "static assets and favicon aren't cached by the browser", "user": {"value": 45416, "label": "obra"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-09-24T04:44:55Z", "updated_at": "2022-01-13T22:21:28Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Using datasette to solve some frustrating problems with our fulfillment provider today, I was surprised to see repeated requests for assets under /-/static and the favicon. While it won't likely be a huge performance bottleneck, I bet datasette would feel a bit zippier if you had Uvicorn serving up some caching-related headers telling the browser it was safe to cache static assets.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/974/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 534629631, "node_id": "MDU6SXNzdWU1MzQ2Mjk2MzE=", "number": 650, "title": "Add a glossary to the documentation", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2019-12-09T00:23:45Z", "updated_at": "2022-01-13T22:04:56Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Call it `glossary.rst` - it can use a definition list something like this:\r\n```rst\r\n.. _glossary:\r\n\r\nGlossary\r\n========\r\n\r\nTerm\r\n A definition of the term.\r\n\r\nAnother term\r\n Another definition.\r\n```", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/650/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 895686039, "node_id": "MDU6SXNzdWU4OTU2ODYwMzk=", "number": 1336, "title": "Document turning on WAL for live served SQLite databases", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-05-19T17:08:58Z", "updated_at": "2022-01-13T21:55:59Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Datasette docs don't talk about WAL yet, which allows you to safely serve reads from a database file while it is accepting writes.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1336/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1100499619, "node_id": "I_kwDOBm6k_c5BmEqj", "number": 1592, "title": "Row pages should show links to foreign keys", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-01-12T15:50:20Z", "updated_at": "2022-01-12T15:52:17Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Refs #1518 refactor.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1592/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1097332098, "node_id": "I_kwDODEm0Qs5BZ_WC", "number": 64, "title": "Include all entities for tweets", "user": {"value": 111631, "label": "max"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-01-09T23:35:28Z", "updated_at": "2022-01-09T23:35:28Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Per our conversation [on Twitter](https://twitter.com/mschoening/status/1480312477246054401):\r\n\r\nIt would be neat if all entities (including URLs) were captured. This way you can ensure, that URLs are parsed out exactly the same way Twitter parses URLs \u2013 we all know parsing URLs with a regex ain't fun.\r\n\r\nRight now, I believe the tool filters out all entities that are not of type `media`.", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/64/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 988493790, "node_id": "MDExOlB1bGxSZXF1ZXN0NzI3MzkwODM1", "number": 36, "title": "Correct naming of tool in readme", "user": {"value": 2129, "label": "badboy"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-09-05T12:05:40Z", "updated_at": "2022-01-06T16:04:46Z", "closed_at": null, "author_association": "FIRST_TIME_CONTRIBUTOR", "pull_request": "dogsheep/dogsheep-photos/pulls/36", "body": null, "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/36/reactions\", \"total_count\": 2, \"+1\": 2, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1091850530, "node_id": "I_kwDODEm0Qs5BFFEi", "number": 63, "title": "Import archive error 'withheld_in_countries'", "user": {"value": 521097, "label": "pauloxnet"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-01-01T16:58:59Z", "updated_at": "2022-01-01T16:58:59Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Importing the twitter archive I received this error:\r\n```bash\r\n$ twitter-to-sqlite import archive.db twitter-2021-12-31-.zip \r\nbirdwatch-note-rating: not yet implemented\r\nbirdwatch-note: not yet implemented\r\nbranch-links: not yet implemented\r\ncommunity-tweet: not yet implemented\r\ncontact: not yet implemented\r\ndevice-token: not yet implemented\r\ndirect-message-mute: not yet implemented\r\nmute: not yet implemented\r\nperiscope-account-information: not yet implemented\r\nperiscope-ban-information: not yet implemented\r\nperiscope-broadcast-metadata: not yet implemented\r\nperiscope-comments-made-by-user: not yet implemented\r\nperiscope-expired-broadcasts: not yet implemented\r\nperiscope-followers: not yet implemented\r\nperiscope-profile-description: not yet implemented\r\nprofessional-data: not yet implemented\r\nprotected-history: not yet implemented\r\nreply-prompt: not yet implemented\r\nscreen-name-change: not yet implemented\r\nsmartblock: not yet implemented\r\nspaces-metadata: not yet implemented\r\nsso: not yet implemented\r\nTraceback (most recent call last):\r\n File \"/home/paulox/.virtualenvs/dogsheep/bin/twitter-to-sqlite\", line 8, in \r\n sys.exit(cli())\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py\", line 1128, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py\", line 1053, in main\r\n rv = self.invoke(ctx)\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py\", line 1659, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py\", line 1395, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py\", line 754, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/twitter_to_sqlite/cli.py\", line 759, in import_\r\n archive.import_from_file(db, filename, content)\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/twitter_to_sqlite/archive.py\", line 246, in import_from_file\r\n db[table_name].insert_all(rows, pk=pk, replace=True)\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/sqlite_utils/db.py\", line 2625, in insert_all\r\n self.insert_chunk(\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/sqlite_utils/db.py\", line 2406, in insert_chunk\r\n result = self.db.execute(query, params)\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/sqlite_utils/db.py\", line 422, in execute\r\n return self.conn.execute(sql, parameters)\r\nsqlite3.OperationalError: table archive_tweet has no column named withheld_in_countries\r\n```\r\n\r\nI found only a single tweet with the key `withheld_in_countries` in `tweet.js` that seems the problems:\r\n```JSON\r\n[\r\n{\r\n \"tweet\" : {\r\n \"retweeted\" : false,\r\n \"source\" : \"Twitter for Android\",\r\n \"entities\" : {\r\n \"hashtags\" : [\r\n {\r\n \"text\" : \"NowOnAndroid\",\r\n \"indices\" : [\r\n \"64\",\r\n \"77\"\r\n ]\r\n }\r\n ],\r\n \"symbols\" : [ ],\r\n \"user_mentions\" : [\r\n {\r\n \"name\" : \"Periscope\",\r\n \"screen_name\" : \"PeriscopeCo\",\r\n \"indices\" : [\r\n \"3\",\r\n \"15\"\r\n ],\r\n \"id_str\" : \"1111111111\",\r\n \"id\" : \"222222222\"\r\n }\r\n ],\r\n \"urls\" : [\r\n {\r\n \"url\" : \"https://t.co/xxxxxxxxx\",\r\n \"expanded_url\" : \"https://vine.co/v/xxxxxxxxx\",\r\n \"display_url\" : \"vine.co/v/xxxxxxxxxx\",\r\n \"indices\" : [\r\n \"78\",\r\n \"101\"\r\n ]\r\n }\r\n ]\r\n },\r\n \"display_text_range\" : [\r\n \"0\",\r\n \"101\"\r\n ],\r\n \"favorite_count\" : \"0\",\r\n \"id_str\" : \"1111111111111111111111\",\r\n \"truncated\" : false,\r\n \"retweet_count\" : \"0\",\r\n \"withheld_in_countries\" : [\r\n \"TR\"\r\n ],\r\n \"id\" : \"000000000000000000\",\r\n \"possibly_sensitive\" : false,\r\n \"created_at\" : \"Fri Aug 14 06:04:03 +0000 2015\",\r\n \"favorited\" : false,\r\n \"full_text\" : \"RT @periscopeco: Travel the world. LIVE. The Global Map is here #NowOnAndroid https://t.co/NZXdsPWROk\",\r\n \"lang\" : \"en\"\r\n }\r\n }\r\n ]\r\n```\r\n\r\nI solved the error removing the key from the `tweet.js` but I'm reporting this error to improve the project.", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/63/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1091838742, "node_id": "I_kwDOBm6k_c5BFCMW", "number": 1585, "title": "Fire base caching for `publish cloudrun`", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-01-01T15:38:15Z", "updated_at": "2022-01-01T15:40:38Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "https://gist.github.com/steren/03d3e58c58c9a53fd49bb78f58541872 has a recipe for this, via https://twitter.com/steren/status/1477038411114446848\r\n\r\nCould this enable easier vanity URLs of the format `https://$project_id.web.app/`? How about CDN caching?", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1585/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1091257796, "node_id": "I_kwDOBm6k_c5BC0XE", "number": 1584, "title": "give error with recursive sql", "user": {"value": 58088336, "label": "tunguyenatwork"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-12-30T18:53:16Z", "updated_at": "2021-12-30T18:53:16Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I got an error \"near \"WITH\": syntax error\" after I upgraded to version 0.59 from 0.52.4. This error is related to recursive sql. It works great on the previous version but it failed after upgraded. Below is an example of sql:\r\n\r\nWITH RECURSIVE manager_of(position, super_position) AS (SELECT position, case ifnull(INDIRECT_SUPER_POSITION,'') when '' then super_position else INDIRECT_SUPER_POSITION end as SUPER_POSITION FROM position where super_position<>'SGV000000001' and super_position!='' and position <> super_position),chain_manager_of_position(position, level) AS (SELECT super_position, 1 as level FROM manager_of WHERE super_position!='' and (position=:pos or position in (Select position from employee where employee=:ein)) UNION ALL SELECT super_position, level+1 as level FROM manager_of JOIN chain_manager_of_position USING(position)) SELECT * FROM chain_manager_of_position left join employee using(position) where employee is not NULL order by level limit 1", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1584/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1090810196, "node_id": "I_kwDOBm6k_c5BBHFU", "number": 1583, "title": "consider adding deletion step of cloudbuild artifacts to gcloud publish", "user": {"value": 536941, "label": "fgregg"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-12-30T00:33:23Z", "updated_at": "2021-12-30T00:34:16Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "right now, as part of the the publish process images and other artifacts are stored to gcloud's cloud storage before being deployed to cloudrun.\r\n\r\nafter successfully deploying, it would be nice if the the script deleted these artifacts. otherwise, if you have regularly scheduled build process, you can end up paying to store lots of out of date artifacts.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1583/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1090798237, "node_id": "I_kwDOCGYnMM5BBEKd", "number": 359, "title": "Use RETURNING if available to populate last_pk", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-12-29T23:43:23Z", "updated_at": "2021-12-29T23:43:23Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Inspired by this: https://news.ycombinator.com/item?id=29729283\r\n\r\n> Because SQLite is effectively serializing all the writes for us, we have zero locking in our code. We used to have to lock when inserting new items (to get the LastInsertRowId), but the newer version of SQLite supports the RETURNING keyword, so we don't even have to lock on inserts now.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/359/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 770598024, "node_id": "MDU6SXNzdWU3NzA1OTgwMjQ=", "number": 1152, "title": "Efficiently calculate list of databases/tables a user can view", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 12, "created_at": "2020-12-18T06:13:01Z", "updated_at": "2021-12-27T23:04:31Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "> The homepage currently performs a massive flurry of permission checks - one for each, database, table and view: https://github.com/simonw/datasette/blob/0.53/datasette/views/index.py#L21-L75\r\n> \r\n> A paginated version of this is a little daunting as the permission checks would have to be carried out in every single table just to calculate the count that will be paginated.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1150#issuecomment-747864831_", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1152/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1087919372, "node_id": "I_kwDOBm6k_c5A2FUM", "number": 1578, "title": "Confirm if documented nginx proxy config works for row pages with escaped characters in their primary key", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-12-23T18:27:59Z", "updated_at": "2021-12-24T21:33:19Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Found this while working on https://github.com/simonw/datasette-tiddlywiki\r\n\r\n\"image\"\r\n\r\nThen clicking on `/tiddlywiki/tiddlers/%24%3A%2FDefaultTiddlers` returns a 404.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1578/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 706001517, "node_id": "MDU6SXNzdWU3MDYwMDE1MTc=", "number": 163, "title": "Idea: conversions= could take Python functions", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-09-22T00:37:12Z", "updated_at": "2021-12-20T00:56:52Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Right now you use `conversions=` like this:\r\n\r\n```python\r\ndb[\"example\"].insert({\r\n \"name\": \"The Bigfoot Discovery Museum\"\r\n}, conversions={\"name\": \"upper(?)\"})\r\n```\r\nHow about if you could optionally provide a Python function (or a lambda) like this?\r\n```python\r\ndb[\"example\"].insert({\r\n \"name\": \"The Bigfoot Discovery Museum\"\r\n}, conversions={\"name\": lambda s: s.upper()})\r\n```\r\nThis would work by creating a random name for that function, registering it (similar to #162), executing the SQL and then un-registering the custom function at the end.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/163/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1083657868, "node_id": "I_kwDOBm6k_c5Al06M", "number": 1565, "title": "Documented JavaScript variables on different templates made available for plugins", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 8, "created_at": "2021-12-17T22:30:51Z", "updated_at": "2021-12-19T22:37:29Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "While working on https://github.com/simonw/datasette-leaflet-freedraw/issues/10 I found myself writing this atrocity to figure out the SQL query used for a specific table page:\r\n\r\n```javascript\r\nlet innerSql = Array.from(document.getElementsByTagName(\"span\")).filter(\r\n el => el.innerText == \"View and edit SQL\"\r\n)[0].parentElement.getAttribute(\"title\")\r\n```\r\nThis is obviously bad - it's very brittle, and will break if I ever change the text on that link (like localizing it for example).\r\n\r\nInstead, I think pages like that one should have a block of script at the bottom something like this:\r\n```javascript\r\nwindow.datasette = window.datasette || {};\r\ndatasette.view_name = 'table';\r\ndatasette.table_sql = 'select * from ...';\r\n```", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1565/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1084185188, "node_id": "I_kwDOBm6k_c5An1pk", "number": 1573, "title": "Make trace() a documented internal API", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-12-19T20:32:56Z", "updated_at": "2021-12-19T21:13:13Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "This should be documented so plugin authors can use it to add their own custom traces: https://github.com/simonw/datasette/blob/8f311d6c1d9f73f4ec643009767749c17b5ca5dd/datasette/tracer.py#L28-L52\r\n\r\nIncluding the new `kwargs` pattern I added in #1571: https://github.com/simonw/datasette/blob/f65817000fdf87ce8a0c23edc40784ebe33b5842/datasette/database.py#L128-L132", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1573/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 520681725, "node_id": "MDU6SXNzdWU1MjA2ODE3MjU=", "number": 621, "title": "Syntax for ?_through= that works as a form field", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2019-11-11T00:19:03Z", "updated_at": "2021-12-18T01:42:33Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "The current syntax for `?_through=` uses JSON to avoid any risk of confusion with table or column names that contain special characters.\r\n\r\nThis means you can't target a form field at it.\r\n\r\nWe should be able to support both - `?x.y.z=value` for tables and columns with \"regular\" names, falling back to the current JSON syntax for columns or tables that won't work with the key/value syntax.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/621/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 793002853, "node_id": "MDExOlB1bGxSZXF1ZXN0NTYwNzYwMTQ1", "number": 1204, "title": "WIP: Plugin includes", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-01-25T03:59:06Z", "updated_at": "2021-12-17T07:10:49Z", "closed_at": null, "author_association": "OWNER", "pull_request": "simonw/datasette/pulls/1204", "body": "Refs #1191\r\n\r\nNext steps:\r\n\r\n- [ ] Get comfortable that this pattern is the right way to go\r\n- [ ] Implement it for all of the other pages, not just the table page\r\n- [ ] Add a new set of plugin tests that exercise ALL of these new hook locations\r\n- [ ] Document, then ship", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1204/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 1, "state_reason": null} {"id": 636511683, "node_id": "MDU6SXNzdWU2MzY1MTE2ODM=", "number": 830, "title": "Redesign register_facet_classes plugin hook", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 3, "created_at": "2020-06-10T20:03:27Z", "updated_at": "2021-12-16T19:58:22Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Nothing uses this plugin hook yet, so the design is not yet proven.\r\n\r\nI'm going to build a real plugin against it and use that process to inform any design changes that may need to be made.\r\n\r\nI'll add a warning about this to the documentation.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/830/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1079111498, "node_id": "I_kwDOBm6k_c5AUe9K", "number": 1553, "title": "if csv export is truncated in non streaming mode set informative response header", "user": {"value": 536941, "label": "fgregg"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-12-13T22:50:44Z", "updated_at": "2021-12-16T19:17:28Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "streaming mode is currently not enabled for custom queries, so the queries will be truncated to max row limit.\r\n\r\nit would be great if a response is truncated that an header signalling that was set in the header.\r\n\r\ni need to write some pagination code for getting full results back for a custom query and it would make the code much better if i could reliably known when there is nothing more to limit/offset ", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1553/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 741231849, "node_id": "MDU6SXNzdWU3NDEyMzE4NDk=", "number": 1087, "title": "Idea: ?_extra=urls for getting back URLs to useful things", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-11-12T02:55:41Z", "updated_at": "2021-12-15T18:06:16Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Working on https://github.com/simonw/datasette-search-all/issues/10 made me realize that sometimes it can be difficult to calculate the URL for a database, table or row within Datasette.\r\n\r\nIt would be useful to have an optional extra JSON extension (using `?_extra=` from #262) that can help with this.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1087/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 705840673, "node_id": "MDU6SXNzdWU3MDU4NDA2NzM=", "number": 972, "title": "Support faceting against arbitrary SQL queries", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-09-21T19:00:43Z", "updated_at": "2021-12-15T18:02:20Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "> ... support for running facets against arbitrary custom SQL queries is half-done in that facets now execute against wrapped subqueries as-of ea66c45df96479ef66a89caa71fff1a97a862646\r\n> \r\n> https://github.com/simonw/datasette/blob/ea66c45df96479ef66a89caa71fff1a97a862646/datasette/facets.py#L192-L200\r\n_Originally posted by @simonw in https://github.com/simonw/datasette/issues/971#issuecomment-696307922_", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/972/reactions\", \"total_count\": 3, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 3, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1072106103, "node_id": "I_kwDOBm6k_c4_5wp3", "number": 1542, "title": "feature request: order and dependency of plugins (that use js)", "user": {"value": 33631, "label": "fs111"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-12-06T12:40:45Z", "updated_at": "2021-12-15T17:47:08Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I have been playing with datasette for the last couple of weeks and it is great! I am a big fan of `datasette-cluster-map` and wanted to enhance it a bit with a what I would call a sub-plugin. I basically want to add more controls to the map that cluster map provides. I have been looking into its code and how the plugin management works, but it seems what I am trying to do is not doable without hacks in js.\r\n\r\nBasically what would like to have is a way to say load my plugin after the plugins I depend on have been loaded and rendered. There seems to be no prior art where plugins have these dependencies on the js level so I was wondering if that could be added or if it exists how to do it.\r\n\r\nBasically what I want to do is:\r\n\r\nmy-awesome-plugin has a dependency on datastte-cluster-map. Whenever datasette cluster map has finished rendering on page load, call my plugin, but no earlier. To make that work datasette probably needs some total order in which way plugins are loaded intialized.\r\n\r\nSince I am new to datastte, I may be missing something obvious, so please let me know if the above makes no sense.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1542/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1077628073, "node_id": "I_kwDOBm6k_c5AO0yp", "number": 1550, "title": "Research option for returning all rows from arbitrary query", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-12-11T19:31:11Z", "updated_at": "2021-12-11T23:43:24Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Inspired by thinking about #1549 - returning ALL rows from an arbitrary query is a lot easier if you just run that query and keep iterating over the cursor.\r\n\r\nI've avoided doing that in the past because it could tie up a connection for a long time - but in private instances this wouldn't be such a problem.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1550/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1066603133, "node_id": "PR_kwDOCGYnMM4vKAzW", "number": 347, "title": "Test against pysqlite3 running SQLite 3.37", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2021-11-29T23:17:57Z", "updated_at": "2021-12-11T01:02:19Z", "closed_at": null, "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/347", "body": "Refs #346 and #344.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/347/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1068791148, "node_id": "I_kwDOBm6k_c4_tHVs", "number": 1540, "title": "Idea: hover to reveal details of linked row", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2021-12-01T19:28:07Z", "updated_at": "2021-12-09T23:38:39Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "\"fara__item_version__7_rows_where_where__item___5236\"\r\n\r\nHovering over that could work a little bit like GitHub issue links:\r\n\r\n![hover](https://user-images.githubusercontent.com/9599/144300537-9cd9e9af-ac16-42db-842f-37661bc94063.gif)\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1540/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 863884805, "node_id": "MDU6SXNzdWU4NjM4ODQ4MDU=", "number": 1304, "title": "Document how to send multiple values for \"Named parameters\" ", "user": {"value": 9308268, "label": "rayvoelker"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-04-21T13:19:06Z", "updated_at": "2021-12-08T03:23:14Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "https://docs.datasette.io/en/stable/sql_queries.html#named-parameters\r\n\r\nI thought that I had seen an example of how to do this example below, but I can't seem to find it\r\n\r\n```sql\r\nselect\r\n *\r\nfrom\r\n bib\r\nwhere\r\n bib.bib_record_num in (1008088,1008092)\r\n```\r\n\r\n```sql\r\nselect\r\n *\r\nfrom\r\n bib\r\nwhere\r\n bib.bib_record_num in (:bib_record_numbers)\r\n```\r\n![image](https://user-images.githubusercontent.com/9308268/115558839-2333a480-a281-11eb-85e6-ce3bada79140.png)\r\n\r\nhttps://ilsweb.cincinnatilibrary.org/collection-analysis/current_collection-204d100?sql=select%0D%0A++*%0D%0Afrom%0D%0A++bib%0D%0Awhere%0D%0A++bib.bib_record_num+in+%28%3Abib_record_numbers%29&bib_record_numbers=1008088%2C1008092\r\n\r\nOr, maybe this isn't a fully supported feature.\r\n\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1304/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1071531082, "node_id": "I_kwDOCGYnMM4_3kRK", "number": 349, "title": "A way of creating indexes on newly created tables", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-12-05T18:56:12Z", "updated_at": "2021-12-07T01:04:37Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "I'm writing code for https://github.com/simonw/git-history/issues/33 that creates a table inside a loop:\r\n\r\n```python\r\nitem_pk = db[item_table].lookup(\r\n {\"_item_id\": item_id},\r\n item_to_insert,\r\n column_order=(\"_id\", \"_item_id\"),\r\n pk=\"_id\",\r\n)\r\n```\r\nI need to look things up by `_item_id` on this table, which means I need an index on that column (the table can get very big).\r\n\r\nBut there's no mechanism in SQLite utils to detect if the table was created for the first time and add an index to it. And I don't want to run `CREATE INDEX IF NOT EXISTS` every time through the loop.\r\n\r\nThis should work like the `foreign_keys=` mechanism.\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/349/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1072435124, "node_id": "I_kwDOCGYnMM4_7A-0", "number": 350, "title": "Optional caching mechanism for table.lookup()", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-12-06T17:54:25Z", "updated_at": "2021-12-06T17:56:57Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Inspired by work on `git-history` where I used this pattern:\r\n```python\r\n column_name_to_id = {}\r\n\r\n def column_id(column):\r\n if column not in column_name_to_id:\r\n id = db[\"columns\"].lookup(\r\n {\"namespace\": namespace_id, \"name\": column},\r\n foreign_keys=((\"namespace\", \"namespaces\", \"id\"),),\r\n )\r\n column_name_to_id[column] = id\r\n return column_name_to_id[column]\r\n```\r\nIf you're going to be doing a large number of `table.lookup(...)` calls and you know that no other script will be modifying the database at the same time you can presumably get a big speedup using a Python in-memory cache - maybe even a LRU one to avoid memory bloat.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/350/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null}