{"id": 787098345, "node_id": "MDU6SXNzdWU3ODcwOTgzNDU=", "number": 1191, "title": "Ability for plugins to collaborate when adding extra HTML to blocks in default templates", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 12, "created_at": "2021-01-15T18:18:51Z", "updated_at": "2023-09-18T06:55:52Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Sometimes a plugin may want to add content to an existing default template - for example `datasette-search-all` adds a new search box at the top of `index.html`. I also want `datasette-upload-csvs` to add a CTA on the `database.html` page: https://github.com/simonw/datasette-upload-csvs/issues/18\r\n\r\nCurrently plugins can do this by providing a new version of the `index.html` template - but if multiple plugins try to do that only one of them will succeed.\r\n\r\nIt would be better if there were known areas of those templates which plugins could add additional content to, such that multiple plugins can use the same spot.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1191/reactions\", \"total_count\": 4, \"+1\": 4, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null}
{"id": 1656432059, "node_id": "PR_kwDOBm6k_c5NuBNG", "number": 2053, "title": "WIP new JSON for queries", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 12, "created_at": "2023-04-05T23:26:15Z", "updated_at": "2023-07-26T18:28:59Z", "closed_at": "2023-07-26T18:26:45Z", "author_association": "OWNER", "pull_request": "simonw/datasette/pulls/2053", "body": "Refs:\r\n- #2049\r\n\r\nTODO:\r\n\r\n- [x] Read queries JSON\r\n- Implement error display with `\"ok\": false` and an errors key\r\n- Read queries HTML\r\n- Read queries other formats (plugins)\r\n- Canned read queries (dispatched to from table)\r\n- Write queries (a canned query thing)\r\n- Implement different shapes, refactoring to share code with table\r\n- Implement a sensible subset of extras, also refactoring to share code with table\r\n- Get all tests passing\r\n\r\n<!-- readthedocs-preview datasette start -->\r\n----\r\n:books: Documentation preview :books:: https://datasette--2053.org.readthedocs.build/en/2053/\r\n\r\n<!-- readthedocs-preview datasette end -->", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2053/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 1, "state_reason": null}
{"id": 1219385669, "node_id": "I_kwDOBm6k_c5IrllF", "number": 1729, "title": "Implement ?_extra and new API design for TableView", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 8755003, "label": "Datasette 1.0a-next"}, "comments": 12, "created_at": "2022-04-28T22:28:14Z", "updated_at": "2022-12-13T05:29:07Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Part of:\r\n- #262\r\n- #1518", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1729/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null}
{"id": 1435294468, "node_id": "I_kwDOBm6k_c5VjNsE", "number": 1882, "title": "`/db/-/create` API for creating tables", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 8658075, "label": "Datasette 1.0a0"}, "comments": 12, "created_at": "2022-11-03T21:44:32Z", "updated_at": "2022-11-15T19:59:43Z", "closed_at": "2022-11-15T06:00:41Z", "author_association": "OWNER", "pull_request": null, "body": ">  It really feels like this should be accompanied by a `/db/-/create` API for creating tables. I had to add that to `sqlite-utils` eventually (initially it only supported creating by passing in an example document):\r\n>\r\n> https://sqlite-utils.datasette.io/en/stable/cli.html#cli-create-table\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1862#issuecomment-1299073433_\r\n      ", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1882/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1426001541, "node_id": "I_kwDOBm6k_c5U_w6F", "number": 1866, "title": "API for bulk inserting records into a table", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 8658075, "label": "Datasette 1.0a0"}, "comments": 12, "created_at": "2022-10-27T17:19:25Z", "updated_at": "2022-11-15T19:59:34Z", "closed_at": "2022-10-30T06:04:07Z", "author_association": "OWNER", "pull_request": null, "body": "Similar to https://github.com/simonw/datasette-insert/blob/0.8/README.md#inserting-data-and-creating-tables\r\n\r\nI expect this to become by far the most common way that data gets into a Datasette instance - more so than the individual row API in:\r\n- #1851 ", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1866/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1424378012, "node_id": "I_kwDOBm6k_c5U5kic", "number": 1860, "title": "SQL query field can't begin by a comment", "user": {"value": 562352, "label": "CharlesNepote"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 12, "created_at": "2022-10-26T16:55:31Z", "updated_at": "2022-10-27T18:57:37Z", "closed_at": "2022-10-27T04:21:40Z", "author_association": "NONE", "pull_request": null, "body": "![image](https://user-images.githubusercontent.com/562352/198085197-f26fcd61-4dac-4ca4-a346-e70f88a30ecc.png)\r\n\r\nSQL comments are **very** useful to explain the meaning of the query. It's currently impossible to put it at the beginning of the field as seen on the screen capture: it leads to an error: `Statement must be a SELECT`.\r\n\r\nIt would be great to make it possible because:\r\n* as the request is the title of the page:\r\n  * it eases the search with search engines\r\n  * it eases the search in the browsers' url field\r\n* it acts as a kind of title: the global meaning of the query is immediately understandable\r\n* some tools, such as Slack, are shortening long URLs and displaying the beginning of the URLs (eg. `https://example.org/products?sql=select+%28length%28data_quality_errors_ta[...]+%21%3D+%22%22+group+by+NB_of_issues+order+by+NB_of_issues+desc+limit+200`)\r\n\r\nBeginning a query with a comment is possible with SQLite.\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1860/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1352932716, "node_id": "I_kwDOCGYnMM5QpB1s", "number": 471, "title": "sqlite-utils query --functions mechanism for registering extra functions", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 8355157, "label": "3.29"}, "comments": 12, "created_at": "2022-08-27T03:57:53Z", "updated_at": "2022-09-07T03:46:26Z", "closed_at": "2022-08-27T05:10:57Z", "author_association": "OWNER", "pull_request": null, "body": "It would be really cool if you could register additional custom SQL functions for use with the `sqlite-utils query` command - something like this:\r\n\r\n```\r\nsqlite-utils data.db 'update images set domain = extract_domain(url)' --functions '\r\nfrom urllib.parse import urlparse\r\n\r\ndef extract_domain(url):\r\n    return urlparse(url).netloc\r\n'\r\n```\r\nEvery function defined in that code block would be registered with the connection, unless the name began with an underscore.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/471/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1178546862, "node_id": "I_kwDOCGYnMM5GPzKu", "number": 420, "title": "Document how to use a `--convert` function that runs initialization code first", "user": {"value": 770231, "label": "strada"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 12, "created_at": "2022-03-23T19:07:36Z", "updated_at": "2022-08-28T11:34:37Z", "closed_at": "2022-03-25T20:07:33Z", "author_association": "NONE", "pull_request": null, "body": "When I have an insert command with transform like this:\r\n\r\n```\r\ncat items.json | jq '.data' | sqlite-utils insert listings.db listings - --convert '\r\nd = enchant.Dict(\"en_US\")\r\nrow[\"is_dictionary_word\"] = d.check(row[\"name\"])\r\n'  --import=enchant --ignore\r\n```\r\n\r\nI noticed as the number of rows increases the operation becomes quite slow, likely due to the creation of the `d = enchant.Dict(\"en_US\")` object for each row. Is there a way to share that instance `d` between transform function calls, like a shared context?", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/420/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1250495688, "node_id": "I_kwDOCGYnMM5KiQzI", "number": 439, "title": "Misleading progress bar against utf-16-le CSV input", "user": {"value": 4068, "label": "frafra"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 12, "created_at": "2022-05-27T08:34:49Z", "updated_at": "2022-06-15T03:53:43Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "The program crashes without any error.\r\n```\r\nwget \"https://artsdatabanken.no/Fab2018/api/export/csv\"\r\nsqlite-utils create-database test.db\r\nsqlite-utils insert --csv --delimiter \";\" --encoding \"utf-16-le\" test test.db csv \r\n  [------------------------------------]    0%\r\n  [#################-------------------]   49%  00:00:01\r\n```\r\nI would like to highlight various issues:\r\n1. sqlite-utils catches exceptions without printing the stacktrace and/or reraising the exception, so there is no easy way to use `pdb` or similar to debug the program, solution: add a debug option\r\n2. Silent crash: this is related to (1.), and it happens when there is a catch-all mechanism; solution: let the program fail.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/439/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null}
{"id": 1213683988, "node_id": "I_kwDOBm6k_c5IV1kU", "number": 1718, "title": "Code examples in the documentation should be formatted with Black", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 12, "created_at": "2022-04-24T15:22:50Z", "updated_at": "2022-04-24T16:24:14Z", "closed_at": "2022-04-24T16:18:03Z", "author_association": "OWNER", "pull_request": null, "body": "For example on this page: https://docs.datasette.io/en/stable/writing_plugins.html#packaging-a-plugin\r\n\r\nI wonder if there's an easy way for me to enforce this for Sphinx documentation?", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1718/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1168995756, "node_id": "I_kwDOBm6k_c5FrXWs", "number": 1657, "title": "Tilde encoding: use ~ instead of - for dash-encoding", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 12, "created_at": "2022-03-14T22:55:17Z", "updated_at": "2022-03-15T18:25:11Z", "closed_at": "2022-03-15T18:01:58Z", "author_association": "OWNER", "pull_request": null, "body": "Refs #1439", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1657/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1087181951, "node_id": "I_kwDOBm6k_c5AzRR_", "number": 1576, "title": "Traces should include SQL executed by subtasks created with `asyncio.gather`", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 12, "created_at": "2021-12-22T20:52:02Z", "updated_at": "2022-02-05T05:21:35Z", "closed_at": "2022-02-05T05:19:53Z", "author_association": "OWNER", "pull_request": null, "body": "I tried running some parallel SQL queries using `asyncio.gather()` but the SQL that was executed didn't show up in the trace rendered by https://datasette.io/plugins/datasette-pretty-traces\r\n\r\nI realized that was because traces are keyed against the current task ID, which changes when a sub-task is run using `asyncio.gather` or similar.\r\n\r\nThe faceting and suggest faceting queries are missing from this trace:\r\n\r\n![image](https://user-images.githubusercontent.com/9599/147153855-2d611f07-922a-4d18-9e6e-4be89e010dc4.png)\r\n\r\n> The reason they aren't showing up in the traces is that traces are stored just for the currently executing `asyncio` task ID: https://github.com/simonw/datasette/blob/ace86566b28280091b3844cf5fbecd20158e9004/datasette/tracer.py#L13-L25\r\n>\r\n> This is so traces for other incoming requests don't end up mixed together. But there's no current mechanism to track async tasks that are effectively \"child tasks\" of the current request, and hence should be tracked the same.\r\n>\r\n> https://stackoverflow.com/a/69349501/6083 suggests that you pass the task ID as an argument to the child tasks that are executed using `asyncio.gather()` to work around this kind of problem.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1518#issuecomment-999870993_", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1576/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1109884720, "node_id": "I_kwDOBm6k_c5CJ38w", "number": 1609, "title": "Ensure \"pip install datasette\" still works with Python 3.6", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 12, "created_at": "2022-01-21T00:08:10Z", "updated_at": "2022-01-24T19:20:09Z", "closed_at": "2022-01-21T02:24:13Z", "author_association": "OWNER", "pull_request": null, "body": "## Original title: Can I keep \"pip install datasette\" working on Python 3.6?\r\n\r\nI dropped support for 3.6 in:\r\n- #1577\r\n\r\nI'm getting reports that `pip3 install datasette` throws an error on that Python, even though I haven't made that new release yet - presumably due to lack of pinning of Uvicorn: https://twitter.com/ldodds/status/1484289475195080706\r\n\r\nIs it possible to get `pip` on that version of Python to install the highest possible version of the packages that are still known to support Python 3.6?\r\n\r\nIf so, how?", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1609/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 323718842, "node_id": "MDU6SXNzdWUzMjM3MTg4NDI=", "number": 268, "title": "Mechanism for ranking results from SQLite full-text search", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 12, "created_at": "2018-05-16T17:36:40Z", "updated_at": "2022-01-13T22:21:28Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "This isn't particularly straight-forward - all the more reason for Datasette to implement it for you. This article is helpful: http://charlesleifer.com/blog/using-sqlite-full-text-search-with-python/", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/268/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null}
{"id": 770598024, "node_id": "MDU6SXNzdWU3NzA1OTgwMjQ=", "number": 1152, "title": "Efficiently calculate list of databases/tables a user can view", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 12, "created_at": "2020-12-18T06:13:01Z", "updated_at": "2021-12-27T23:04:31Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "> The homepage currently performs a massive flurry of permission checks - one for each, database, table and view: https://github.com/simonw/datasette/blob/0.53/datasette/views/index.py#L21-L75\r\n> \r\n> A paginated version of this is a little daunting as the permission checks would have to be carried out in every single table just to calculate the count that will be paginated.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1150#issuecomment-747864831_", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1152/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null}
{"id": 1055469073, "node_id": "I_kwDOBm6k_c4-6S4R", "number": 1513, "title": "Research: CTEs and union all to calculate facets AND query at the same time", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 12, "created_at": "2021-11-16T22:26:45Z", "updated_at": "2021-11-16T23:41:46Z", "closed_at": "2021-11-16T23:41:46Z", "author_association": "OWNER", "pull_request": null, "body": "Consider this page: https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_search=plant&_facet=owner&_facet=country_long&_facet=primary_fuel\r\n\r\nDatasette needs to run the main query for the rows on that page, a count query for the total query, then a separate query for each of those three specified facets.\r\n\r\nThis is a `_search=` query, so it needs to execute the FTS code once for the rows, again for the count, and then three more times for each of the facets.\r\n\r\nCould running that query as a CTE and doing the other queries as part of the same large query produce significant speed improvements?", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1513/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 440222719, "node_id": "MDU6SXNzdWU0NDAyMjI3MTk=", "number": 448, "title": "_facet_array should work against views", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 12, "created_at": "2019-05-03T21:08:04Z", "updated_at": "2021-11-16T01:32:05Z", "closed_at": "2021-11-16T01:19:40Z", "author_association": "OWNER", "pull_request": null, "body": "I created this view: https://json-view-facet-bug-demo-j7hipcg4aq-uc.a.run.app/russian-ads-8dbda00/ads_with_targets\r\n\r\n```\r\nCREATE VIEW ads_with_targets as select ads.*, json_group_array(targets.name) as target_names from ads\r\n  join ad_targets on ad_targets.ad_id = ads.id\r\n  join targets on ad_targets.target_id = targets.id\r\n  group by ad_targets.ad_id\r\n```\r\n\r\nWhen I try to apply faceting by array it appears to work at first: https://json-view-facet-bug-demo-j7hipcg4aq-uc.a.run.app/russian-ads/ads_with_targets?_facet_array=target_names\r\n\r\nBut actually it's doing the wrong thing - the SQL for the facets uses rowid, but rowid is not present on views at all! These results are incorrect, and clicking to select a facet will fail to produce any rows: https://json-view-facet-bug-demo-j7hipcg4aq-uc.a.run.app/russian-ads/ads_with_targets?_facet_array=target_names&target_names__arraycontains=people_who_match%3Ainterests%3AAfrican-American+Civil+Rights+Movement+%281954%E2%80%9468%29\r\n\r\nHere's the SQL it should be using when you select a facet (note that it does not use a rowid):\r\n\r\nhttps://json-view-facet-bug-demo-j7hipcg4aq-uc.a.run.app/russian-ads?sql=select+*+from+ads_with_targets+where+id+in+%28%0D%0A++++++++++++select+ads_with_targets.id+from+ads_with_targets%2C+json_each%28ads_with_targets.target_names%29+j%0D%0A++++++++++++where+j.value+%3D+%3Ap0%0D%0A++++++++%29+limit+101&p0=people_who_match%3Ainterests%3ABlack+%28Color%29\r\n\r\nSo we need to do something a lot smarter here. I'm not sure what the fix will look like, or even if it's feasible given that views don't have a rowid to hook into so the JSON faceting SQL may have to be completely rewritten.\r\n\r\n```\r\ndatasette publish cloudrun \\\r\n    russian-ads.db \\\r\n    --name json-view-facet-bug-demo \\\r\n    --branch master \\\r\n    --extra-options \"--config sql_time_limit_ms:5000 --config facet_time_limit_ms:5000\"\r\n```", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/448/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1005891028, "node_id": "I_kwDOCGYnMM479K3U", "number": 329, "title": "Rethink approach to [ and ] in column names (currently throws error)", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 12, "created_at": "2021-09-23T22:14:24Z", "updated_at": "2021-11-15T02:57:51Z", "closed_at": "2021-11-15T02:57:51Z", "author_association": "OWNER", "pull_request": null, "body": "> I think it's best to still keep `[` and `]` out of column names though. Transforming them into `(` and `)` seems reasonable - but should that happen here or in `sqlite-utils`? I think in `sqlite-utils`.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/datasette-app/issues/121#issuecomment-926200398_\r\n\r\nThis is a rethinking of the solution to:\r\n\r\n- https://github.com/simonw/sqlite-utils/issues/86", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/329/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 274615452, "node_id": "MDU6SXNzdWUyNzQ2MTU0NTI=", "number": 111, "title": "Add \u201cupdated\u201d to metadata", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 12, "created_at": "2017-11-16T18:22:20Z", "updated_at": "2021-09-21T22:48:27Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "To give an indication as to when the data was last updated.\r\n\r\nThis should be a field in the metadata that is then shown on the index page and in the footer, if it is set.\r\n\r\nAlso support setting it using an option to \u201cdatasette publish\u201d and \u201cdatasette package\u201d - which can either be a string or can be the magic string \u201ctoday\u201d to set it to today\u2019s date:\r\n\r\n    datasette publish file.db --updated=today", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/111/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null}
{"id": 906330187, "node_id": "MDU6SXNzdWU5MDYzMzAxODc=", "number": 260, "title": "Support creating descending order indexes", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 12, "created_at": "2021-05-29T03:42:59Z", "updated_at": "2021-05-29T05:01:39Z", "closed_at": "2021-05-29T05:01:39Z", "author_association": "OWNER", "pull_request": null, "body": "SQLite lets you create indexes in reverse order, which can have a surprisingly big impact on performance, see https://github.com/simonw/covid-19-datasette/issues/27\r\n\r\nI tried doing this using `sqlite-utils` like so, but it's didn't work:\r\n\r\n```python\r\ndb[\"ny_times_us_counties\"].create_index([\"date desc\"])\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/260/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 421546944, "node_id": "MDU6SXNzdWU0MjE1NDY5NDQ=", "number": 417, "title": "Datasette Library", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 12, "created_at": "2019-03-15T14:30:22Z", "updated_at": "2020-12-29T14:34:50Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "The ability to run Datasette in a mode where it automatically picks up new (or modified) files in a directory tree without needing to restart the server.\r\n\r\nSuggested command:\r\n\r\n    datasette library /path/to/mydbs/", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/417/reactions\", \"total_count\": 8, \"+1\": 8, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null}
{"id": 567902704, "node_id": "MDU6SXNzdWU1Njc5MDI3MDQ=", "number": 675, "title": "--cp option for datasette publish and datasette package for shipping additional files and directories", "user": {"value": 141844, "label": "aviflax"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 12, "created_at": "2020-02-19T22:55:56Z", "updated_at": "2020-12-28T18:49:21Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I\u2019m working on integrating Datasette into a documentation-oriented publishing workflow internally in my company, and in order to deploy the Docker image created by `datasette package` I need to add an additional file to the image \u2014 in my case, it\u2019s a sort of a deployment directive. I\u2019ve worked out a way to do this after the image has been created, but it\u2019s convoluted and brittle.\r\n\r\nSo it\u2019d be excellent if there was an additional option for this command, something like, like, `--copy`.\r\n\r\nI\u2019d envision it looking something like:\r\n\r\n```shell\r\n$ datasette package --copy /the/source/path:/the/target/path data.db\r\n```\r\n\r\nI\u2019d be happy to help design, specify, implement, and test this feature, if you\u2019d be interested.\r\n\r\nThanks for the fantastic tools!", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/675/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null}
{"id": 339505204, "node_id": "MDU6SXNzdWUzMzk1MDUyMDQ=", "number": 335, "title": "Package datasette for installation using homebrew", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 12, "created_at": "2018-07-09T15:45:03Z", "updated_at": "2020-08-11T16:54:06Z", "closed_at": "2020-08-11T16:54:06Z", "author_association": "OWNER", "pull_request": null, "body": "https://docs.brew.sh/Python-for-Formula-Authors describes how.\r\n\r\n> Applications should be installed into a Python virtualenv environment rooted in libexec. This prevents the app\u2019s Python modules from contaminating the system site-packages and vice versa.\r\n\r\nIt recommends using https://github.com/tdsmith/homebrew-pypi-poet", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/335/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 636722501, "node_id": "MDU6SXNzdWU2MzY3MjI1MDE=", "number": 832, "title": "Having view-table permission but NOT view-database should still grant access to /db/table", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5533512, "label": "Datasette 0.45"}, "comments": 12, "created_at": "2020-06-11T05:12:59Z", "updated_at": "2020-06-30T23:42:11Z", "closed_at": "2020-06-30T23:42:11Z", "author_association": "OWNER", "pull_request": null, "body": "Stumbled into this while working on `datasette-permissions-sql`. I had granted table permissions, but the permission check wasn't even executed because the user failed the previous `view-database` check.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/832/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 451513541, "node_id": "MDU6SXNzdWU0NTE1MTM1NDE=", "number": 498, "title": "Full text search of all tables at once?", "user": {"value": 7936571, "label": "chrismp"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 12, "created_at": "2019-06-03T14:24:43Z", "updated_at": "2020-05-30T17:26:02Z", "closed_at": "2020-05-30T17:26:02Z", "author_association": "NONE", "pull_request": null, "body": "Does datasette have a built-in way, in a browser, to do a full-text search of all columns, in all databases and tables, that have full-text search enabled? Is there a plugin that does this?", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/498/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 588108428, "node_id": "MDU6SXNzdWU1ODgxMDg0Mjg=", "number": 712, "title": "base_url doesn't entirely work for running Datasette inside Binder", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 12, "created_at": "2020-03-26T02:25:55Z", "updated_at": "2020-03-26T15:11:49Z", "closed_at": "2020-03-26T14:35:43Z", "author_association": "OWNER", "pull_request": null, "body": "> Thanks! I'm trying to launch Datasette from *within* a notebook using the jupyter-server-proxy and the new `base_url` parameter. While the assets load ok, and the breadcrumb navigation works, the facet links don't seem to use the `base_url`. Or have I missed something?\r\n\r\n_Originally posted by @wragge in https://github.com/simonw/datasette/issues/394#issuecomment-604166918_", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/712/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 463544206, "node_id": "MDU6SXNzdWU0NjM1NDQyMDY=", "number": 537, "title": "Populate \"endpoint\" key in ASGI scope", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 12, "created_at": "2019-07-03T04:54:47Z", "updated_at": "2019-07-22T06:03:18Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "This is a trick used by Starlette so that other layers of ASGI middleware can see which route was selected.\r\n\r\nThey added it here: https://github.com/encode/starlette/commit/34d0097feb6f057bd050d5057df5a2f96b97384e\r\n\r\nIf Datasette supports it as well we can benefit from it if we integrate this sentry_asgi middleware (probably as a `datasette-sentry` plugin): https://github.com/encode/sentry-asgi/blob/c6a42d44d31f85885b79e4ee898683ecf8104971/sentry_asgi/middleware.py#L34-L35", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/537/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null}
{"id": 459587155, "node_id": "MDExOlB1bGxSZXF1ZXN0MjkwODk3MTA0", "number": 518, "title": "Port Datasette from Sanic to ASGI + Uvicorn", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": {"value": 9599, "label": "simonw"}, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 12, "created_at": "2019-06-23T15:18:42Z", "updated_at": "2019-06-24T13:42:50Z", "closed_at": "2019-06-24T03:13:09Z", "author_association": "OWNER", "pull_request": "simonw/datasette/pulls/518", "body": "Most of the code here was fleshed out in comments on #272 (Port Datasette to ASGI) - this pull request will track the final pieces:\r\n\r\n- [x] Update test harness to more correctly simulate the `raw_path` issue\r\n- [x] Use `raw_path` so table names containing `/` can work correctly\r\n- [x] Bug: JSON not served with correct content-type\r\n- [x] Get ?_trace=1 working again\r\n- [x] Replacement for `@app.listener(\"before_server_start\")`\r\n- [x] Bug: `/fixtures/table%2Fwith%2Fslashes.csv?_format=json` downloads as CSV\r\n- [x] Replace Sanic request and response objects with my own classes, so I can remove Sanic dependency\r\n- [x] Final code tidy-up before merging to master", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/518/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null}
{"id": 299760684, "node_id": "MDU6SXNzdWUyOTk3NjA2ODQ=", "number": 185, "title": "Metadata should be a nested arbitrary KV store", "user": {"value": 222245, "label": "carlmjohnson"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 12, "created_at": "2018-02-23T16:02:07Z", "updated_at": "2019-05-13T18:33:33Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I started using the metadata feature and was surprised to find that values are not inherited from the root object down to specific databases and tables. This makes metadata much less useful and requires a lot of pointless duplication.\r\n\r\nIdeally, metadata should allow arbitrary key-value pairs, and there should be a way of accessing metadata either in an inherited or non-inherited manner. Something like `metadata.page.key` vs. `metadata.this.key` might work as an interface.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/185/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null}
{"id": 337141108, "node_id": "MDU6SXNzdWUzMzcxNDExMDg=", "number": 332, "title": "Sanely handle Infinity/-Infinity values in JSON using ?_json_infinity=1", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 12, "created_at": "2018-06-29T21:21:27Z", "updated_at": "2018-07-24T04:53:20Z", "closed_at": "2018-07-24T03:08:30Z", "author_association": "OWNER", "pull_request": null, "body": "It turns out if you load this CSV using `csvs-to-sqlite` you get an Infinity value in SQLite:\r\n\r\n```\r\nname,num\r\nsasha,10\r\nterry,Inf\r\ncathy,0.5\r\n```\r\n\r\n`csvs-to-sqlite infinity-bug.csv infinity-bug.db`\r\n\r\nI deployed this using:\r\n\r\n```\r\ndatasette publish now infinity-bug.db --name=datasette-infinity-bug --install=datasette-vega\r\n```\r\n\r\nDatasette outputs that as `Infinity` in the JSON format, which causes JavaScript errors.\r\n\r\nDemo\r\n* https://datasette-infinity-bug.now.sh/infinity-bug-0d0224e/infinity-bug - HTML view works\r\n* https://datasette-infinity-bug.now.sh/infinity-bug-0d0224e/infinity-bug.json?_shape=array - this outputs the following:\r\n\r\n```\r\n[\r\n    {\r\n        \"rowid\": 1,\r\n        \"name\": \"sasha\",\r\n        \"num\": 10.0\r\n    },\r\n    {\r\n        \"rowid\": 2,\r\n        \"name\": \"terry\",\r\n        \"num\": Infinity\r\n    },\r\n    {\r\n        \"rowid\": 3,\r\n        \"name\": \"cathy\",\r\n        \"num\": 0.5\r\n    }\r\n]\r\n```\r\n\r\nBut... in Firefox that gets rendered like this:\r\n\r\n![2018-06-29 at 4 20 pm](https://user-images.githubusercontent.com/9599/42115408-5d30f630-7bb8-11e8-8370-c8484801c49b.png)\r\n\r\nAnd if you click the \"Show charting options\" button you get this error in the console:\r\n\r\n```\r\nSyntaxError: JSON.parse: unexpected character at line 1 column 83 of the JSON data\r\n```\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/332/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}