{"id": 473083260, "node_id": "MDU6SXNzdWU0NzMwODMyNjA=", "number": 50, "title": "\"Too many SQL variables\" on large inserts", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2019-07-25T21:43:31Z", "updated_at": "2022-11-04T14:38:36Z", "closed_at": "2019-07-28T11:59:33Z", "author_association": "OWNER", "pull_request": null, "body": "Reported here: https://github.com/dogsheep/healthkit-to-sqlite/issues/9\r\n\r\nIt looks like there's a default limit of 999 variables - we need to be smart about that, maybe dynamically lower the batch size based on the number of columns.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/50/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 760312579, "node_id": "MDU6SXNzdWU3NjAzMTI1Nzk=", "number": 1134, "title": "\"_searchmode=raw\" throws an index out of range error when combined with \"_search_COLUMN\"", "user": {"value": 2181410, "label": "clausjuhl"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-12-09T13:05:37Z", "updated_at": "2020-12-10T05:57:17Z", "closed_at": "2020-12-09T19:56:55Z", "author_association": "NONE", "pull_request": null, "body": "Hi Simon!\r\nMaybe it's just me, but when [using _searchmode=raw (trying to enable wildcard-searching) in combination with the \"_search_COLUMN\"-table argument](https://byraadsarkivet.aarhus.dk/db/cases?_searchmode=raw&_search_title=sundhedsfrem*), I get a list index out of range error. [When combining with the simpler \"_search\"-argument everything works, including wildcard-seaches.](https://byraadsarkivet.aarhus.dk/db/cases?_search=sundhedsfrem*&_searchmode=raw). Here's the traceback:\r\n\r\n```\r\nTraceback (most recent call last):\r\n File \"/Users/cjk/.local/share/virtualenvs/minutes-jMDZ8Ssk/lib/python3.7/site-packages/datasette/utils/asgi.py\", line 122, in route_path\r\n return await view(new_scope, receive, send)\r\n File \"/Users/cjk/.local/share/virtualenvs/minutes-jMDZ8Ssk/lib/python3.7/site-packages/datasette/utils/asgi.py\", line 196, in view\r\n request, **scope[\"url_route\"][\"kwargs\"]\r\n File \"/Users/cjk/.local/share/virtualenvs/minutes-jMDZ8Ssk/lib/python3.7/site-packages/datasette/views/base.py\", line 204, in get\r\n request, database, hash, correct_hash_provided, **kwargs\r\n File \"/Users/cjk/.local/share/virtualenvs/minutes-jMDZ8Ssk/lib/python3.7/site-packages/datasette/views/base.py\", line 342, in view_get\r\n request, database, hash, **kwargs\r\n File \"/Users/cjk/.local/share/virtualenvs/minutes-jMDZ8Ssk/lib/python3.7/site-packages/datasette/views/table.py\", line 393, in data\r\n search_col = key.split(\"_search_\", 1)[1]\r\nIndexError: list index out of range\r\n\r\n```", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1134/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 683804172, "node_id": "MDU6SXNzdWU2ODM4MDQxNzI=", "number": 134, "title": "--load-extension option for sqlite-utils query", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-08-21T20:12:42Z", "updated_at": "2020-08-21T21:06:26Z", "closed_at": "2020-08-21T20:54:19Z", "author_association": "OWNER", "pull_request": null, "body": "I got this error:\r\n```\r\n% sqlite-utils calands.db 'create table superunits_with_maps_view_concrete as select * from superunits_with_maps_view'\r\nTraceback (most recent call last):\r\n...\r\n cursor = db.conn.execute(sql, dict(param))\r\nsqlite3.OperationalError: no such function: AsGeoJSON\r\n```\r\nA `--load-extension=/usr/local/lib/mod_spatialite.dylib` option (imitating the same option for Datasette) would help.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/134/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 752966476, "node_id": "MDU6SXNzdWU3NTI5NjY0NzY=", "number": 1114, "title": "--load-extension=spatialite not working with datasetteproject/datasette docker image", "user": {"value": 2182, "label": "danp"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-11-29T17:35:20Z", "updated_at": "2022-01-20T21:29:42Z", "closed_at": "2020-11-29T17:37:45Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "https://github.com/simonw/datasette/commit/6aa5886379dd9017215904fb28567b80018902f9 added the `--load-extension=spatialite` shortcut looking for the extension in these places:\r\n\r\nhttps://github.com/simonw/datasette/blob/12877d7a48e2aa28bb5e780f929a218f7265d849/datasette/utils/__init__.py#L56-L60\r\n\r\nHowever, in the datasetteproject/datasette docker image the file is at `/usr/local/lib/mod_spatialite.so`.\r\n\r\nThis results in the example command [here](https://docs.datasette.io/en/stable/installation.html#loading-spatialite) failing:\r\n\r\n```\r\n% docker run --rm -p 8001:8001 -v `pwd`:/mnt datasetteproject/datasette datasette -p 8001 -h 0.0.0.0 /mnt/data.db --load-extension=spatialite\r\nError: Could not find SpatiaLite extension\r\n```\r\n\r\nBut it does work when given an explicit path:\r\n\r\n```\r\n% docker run --rm -p 8001:8001 -v `pwd`:/mnt datasetteproject/datasette datasette -p 8001 -h 0.0.0.0 /mnt/data.db --load-extension=/usr/local/lib/mod_spatialite.so\r\nINFO: Started server process [1]\r\nINFO: Waiting for application startup.\r\nINFO: Application startup complete.\r\nINFO: Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit)\r\n...\r\n```\r\n\r\nPerhaps `SPATIALITE_PATHS` should include `/usr/local/lib/mod_spatialite.so`?", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1114/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 808843401, "node_id": "MDU6SXNzdWU4MDg4NDM0MDE=", "number": 1226, "title": "--port option should validate port is between 0 and 65535", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-02-15T22:01:33Z", "updated_at": "2021-02-18T18:41:27Z", "closed_at": "2021-02-18T18:41:27Z", "author_association": "OWNER", "pull_request": null, "body": "Currently throws an ugly error message:\r\n```\r\n(datasette-graphql) datasette-graphql % datasette fivethirtyeight.db -p 80094\r\nINFO: Started server process [45497]\r\nINFO: Waiting for application startup.\r\nINFO: Application startup complete.\r\nTraceback (most recent call last):\r\n File \"/Users/simon/.local/share/virtualenvs/datasette-graphql-n1OSJCS8/bin/datasette\", line 8, in \r\n sys.exit(cli())\r\n...\r\n server = await loop.create_server(\r\n File \"/Users/simon/.pyenv/versions/3.8.2/lib/python3.8/asyncio/base_events.py\", line 1461, in create_server\r\n sock.bind(sa)\r\nOverflowError: bind(): port must be 0-65535.\r\n```", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1226/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 490803176, "node_id": "MDU6SXNzdWU0OTA4MDMxNzY=", "number": 8, "title": "--sql and --attach options for feeding commands from SQL queries", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2019-09-08T20:35:49Z", "updated_at": "2020-03-20T23:13:01Z", "closed_at": "2020-03-20T23:13:01Z", "author_association": "MEMBER", "pull_request": null, "body": "Say you want to fetch Twitter profiles for a list of accounts that are stored in another database:\r\n\r\n $ twitter-to-sqlite users-lookup users.db --attach attending.db \\\r\n --sql \"select Twitter from attending.attendes where Twitter is not null\"\r\n\r\nThe SQL query you feed in is expected to return a list of screen names suitable for processing further by the command.\r\n\r\nShould be supported by all three of:\r\n\r\n- [x] `twitter-to-sqlite users-lookup`\r\n- [x] `twitter-to-sqlite user-timeline`\r\n- [x] `twitter-to-sqlite followers` and `friends`\r\n\r\nThe `--attach` option allows other SQLite databases to be attached to the connection. Without it the SQL query will have to read from the single attached database.", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/8/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 732634375, "node_id": "MDExOlB1bGxSZXF1ZXN0NTEyNTQ1MzY0", "number": 1061, "title": ".blob output renderer", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 6026070, "label": "0.51"}, "comments": 4, "created_at": "2020-10-29T20:25:08Z", "updated_at": "2020-10-29T22:01:40Z", "closed_at": "2020-10-29T22:01:39Z", "author_association": "OWNER", "pull_request": "simonw/datasette/pulls/1061", "body": "- [x] Remove the `/-/...blob/...` route I added in #1040 in place of the new `.blob` renderer URLs\r\n- [x] Link to new `.blob` download links on the arbitrary query page (using `_blob_hash=...`) - plus tests for this\r\n\r\nCloses #1050, Closes #1051", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1061/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 637253789, "node_id": "MDU6SXNzdWU2MzcyNTM3ODk=", "number": 833, "title": "/-/metadata and so on should respect view-instance permission", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5512395, "label": "Datasette 0.44"}, "comments": 4, "created_at": "2020-06-11T19:07:21Z", "updated_at": "2020-06-11T22:15:32Z", "closed_at": "2020-06-11T22:14:59Z", "author_association": "OWNER", "pull_request": null, "body": "The only URLs that should be available without authentication at all times are the `/-/static/` prefix, to allow for HTTP caching.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/833/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1450303205, "node_id": "I_kwDOBm6k_c5Wcd7l", "number": 1891, "title": "1.0a0 release notes", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 8658075, "label": "Datasette 1.0a0"}, "comments": 4, "created_at": "2022-11-15T19:58:20Z", "updated_at": "2022-11-29T19:23:41Z", "closed_at": "2022-11-29T19:23:41Z", "author_association": "OWNER", "pull_request": null, "body": "This release will mainly help preview the new Datasette write API:\r\n- #1850", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1891/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1318907685, "node_id": "I_kwDOBm6k_c5OnO8l", "number": 1773, "title": "500 error if sorted by a column not in the ?_col= list", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 8303187, "label": "Datasette 0.62"}, "comments": 4, "created_at": "2022-07-27T01:20:27Z", "updated_at": "2022-08-14T16:06:25Z", "closed_at": "2022-08-14T15:44:05Z", "author_association": "OWNER", "pull_request": null, "body": "For example: https://latest.datasette.io/fixtures/sortable?_sort_desc=sortable&_col=sortable_with_nulls\r\n\r\nThat's `?_sort_desc=sortable&_col=sortable_with_nulls`\r\n\r\n\"image\"\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1773/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 292011379, "node_id": "MDU6SXNzdWUyOTIwMTEzNzk=", "number": 184, "title": "500 from missing table name", "user": {"value": 222245, "label": "carlmjohnson"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2018-01-26T19:46:45Z", "updated_at": "2019-05-21T16:17:29Z", "closed_at": "2018-04-13T18:18:59Z", "author_association": "NONE", "pull_request": null, "body": "https://github.com/simonw/datasette/blob/56623e48da5412b25fb39cc26b9c743b684dd968/datasette/app.py#L517-L519 throws an error if it gets an empty list back. Simplest solution is to write a helper func that just says \r\n\r\n```python\r\nresult = list(await self.execute(name, sql, params)\r\nif result:\r\n return result[0][0]\r\n```\r\n\r\nand use it anywhere `[0][0]` is now.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/184/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 275089535, "node_id": "MDU6SXNzdWUyNzUwODk1MzU=", "number": 121, "title": "?_json=foo&_json=bar query string argument ", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2017-11-18T16:09:55Z", "updated_at": "2018-05-31T13:48:12Z", "closed_at": "2018-05-28T18:11:51Z", "author_association": "OWNER", "pull_request": null, "body": "Causes the specified columns in the output to be treated as JSON, and returned deserialized in the .json or .jsono response.\r\n\r\nThis will be particularly powerful when combined with https://sqlite.org/json1.html", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/121/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 995098231, "node_id": "MDU6SXNzdWU5OTUwOTgyMzE=", "number": 1470, "title": "?_sort=rowid with _next= returns error", "user": {"value": 19851673, "label": "eigenfoo"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-09-13T16:36:15Z", "updated_at": "2021-10-18T19:30:15Z", "closed_at": "2021-10-10T01:15:03Z", "author_association": "NONE", "pull_request": null, "body": "For example:\r\n\r\n- Go to https://cryptics.eigenfoo.xyz/clues/clues?_next=100 (this is the second page of results in a Datasette site)\r\n- Search anything using the FTS search bar. For example, searching for `hello` will take you to https://cryptics.eigenfoo.xyz/clues/clues?_search=hello&_sort=rowid&_next=100\r\n- A `500 Error: list index out of range` is raised.\r\n\r\nThis is because the search URL includes the `&_next=100` UTM parameter, carried over from where the FTS search was run. However, there isn't a second page in the search results, so a `list index out of range` error is raised. You can confirm that removing this UTM parameter from the URL returns the appropriate search results.\r\n\r\nThe FTS search request should strip any `_next` UTM parameter.\r\n\r\n---\r\n\r\n```bash\r\ndatasette, version 0.58.1\r\nsqlite-utils, version 3.17\r\n```", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1470/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1237586379, "node_id": "I_kwDOBm6k_c5JxBHL", "number": 1742, "title": "?_trace=1 fails with datasette-geojson for some reason", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-05-16T19:06:05Z", "updated_at": "2022-05-16T19:42:13Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "view-source:https://calands.datasettes.com/calands/CPAD_2020a_SuperUnits.geojson?_sort=id&id__exact=4&_labels=on&_trace=1 is showing me a blank page.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1742/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 277589569, "node_id": "MDU6SXNzdWUyNzc1ODk1Njk=", "number": 155, "title": "A primary key column that has foreign key restriction associated won't rendering label column", "user": {"value": 388154, "label": "wsxiaoys"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 2949431, "label": "Custom templates edition"}, "comments": 4, "created_at": "2017-11-29T00:40:02Z", "updated_at": "2017-12-07T05:39:53Z", "closed_at": "2017-12-07T05:39:53Z", "author_association": "NONE", "pull_request": null, "body": "", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/155/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1429030341, "node_id": "I_kwDOBm6k_c5VLUXF", "number": 1874, "title": "API to drop a table", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 8658075, "label": "Datasette 1.0a0"}, "comments": 4, "created_at": "2022-10-30T21:55:11Z", "updated_at": "2022-11-15T19:59:53Z", "closed_at": "2022-11-14T05:45:06Z", "author_association": "OWNER", "pull_request": null, "body": "`POST /db/table/-/drop`\r\n\r\nRequire `drop-table` permission.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1874/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 312620566, "node_id": "MDU6SXNzdWUzMTI2MjA1NjY=", "number": 199, "title": "Ability to apply sort on mobile in portrait mode", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2018-04-09T17:35:04Z", "updated_at": "2018-04-10T00:37:53Z", "closed_at": "2018-04-10T00:34:38Z", "author_association": "OWNER", "pull_request": null, "body": "Missed this in #189... on mobile in portrait mode we hide the column headers, which means you can't click them to sort! You can sort in landscape mode at least.\r\n\r\nNeed to come up with an alternative sort UI for portrait on mobile.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/199/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 309047460, "node_id": "MDU6SXNzdWUzMDkwNDc0NjA=", "number": 188, "title": "Ability to bundle metadata and templates inside the SQLite file", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2018-03-27T16:42:07Z", "updated_at": "2020-12-04T17:18:34Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "One of the nicest qualities of SQLite as a data format is that you get a single file which you can then backup or share with other people. \r\n\r\nDatasette breaks this a little once you start including custom metadata.json or template files and CSS.\r\n\r\nIt would be cool if there was an optional mechanism for baking that extra configuration into the SQLite file itself. That way entire datasette mini-applications (including canned queries and custom HTML and CSS) could be constructed as single .db files.\r\n\r\nSince datasette configuration is all file-based, one way to achieve that would be to support a \"datasette_files\" table which, if present is used to search for file contents by path.\r\n\r\nThis is inline with the philosophy described by https://www.sqlite.org/appfileformat.html\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/188/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 961367843, "node_id": "MDU6SXNzdWU5NjEzNjc4NDM=", "number": 1422, "title": "Ability to default to hiding the SQL for a canned query", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-08-05T02:51:39Z", "updated_at": "2021-08-07T05:32:29Z", "closed_at": "2021-08-07T05:32:29Z", "author_association": "OWNER", "pull_request": null, "body": "I'm working on a project with some HUGE (400+ lines of SQL) canned queries right now.\r\n\r\nAny time you land on the canned query page you have to scroll down a long distance to get to the results!\r\n\r\nWould be useful to be able to default to https://latest.datasette.io/fixtures/magic_parameters?_hide_sql=1 without needing the parameter.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1422/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 346028655, "node_id": "MDU6SXNzdWUzNDYwMjg2NTU=", "number": 356, "title": "Ability to display facet counts for many-to-many relationships", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2018-07-31T04:14:26Z", "updated_at": "2019-05-29T21:39:12Z", "closed_at": "2019-05-25T16:30:09Z", "author_association": "OWNER", "pull_request": null, "body": "Parent: #354", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/356/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1382457780, "node_id": "I_kwDOCGYnMM5SZqG0", "number": 490, "title": "Ability to insert multi-line files", "user": {"value": 6180701, "label": "jeqo"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-09-22T13:29:22Z", "updated_at": "2022-09-26T18:24:44Z", "closed_at": "2022-09-23T16:37:58Z", "author_association": "NONE", "pull_request": null, "body": "I was looking into how to parse application log files that contain multiline text (e.g. Java stack traces) into sqlite. \r\nI can see that at the moment `--lines` helps, but falls short when processing multi-line texts.\r\n\r\nI wonder if this functionality would be useful for sqlite-utils. A similar approach to Elastic logstash/filebeat can be adopted: https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html \r\n\r\nPotential changes:\r\n\r\n- add a `--multiline` option\r\n- additional properties for\r\n - multiline-pattern (regex expression)\r\n - multiline-negate: true/false\r\n - multiline-what: previous or next\r\n\r\nOr if this is achievable in a different way, please share. Thanks!", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/490/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 611540797, "node_id": "MDU6SXNzdWU2MTE1NDA3OTc=", "number": 751, "title": "Ability to set custom default _size on a per-table basis", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5471110, "label": "Datasette 0.43"}, "comments": 4, "created_at": "2020-05-04T00:13:03Z", "updated_at": "2020-05-28T05:00:22Z", "closed_at": "2020-05-28T05:00:20Z", "author_association": "OWNER", "pull_request": null, "body": "I have some tables where I'd like the default page size to be 10, without affecting the rest of my Datasette instance.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/751/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 273775212, "node_id": "MDU6SXNzdWUyNzM3NzUyMTI=", "number": 88, "title": "Add NHS England Hospitals example to wiki", "user": {"value": 15543, "label": "tomdyson"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2017-11-14T12:29:10Z", "updated_at": "2021-03-22T23:46:36Z", "closed_at": "2017-11-14T22:54:06Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "https://nhs-england-hospitals.now.sh\r\n\r\nand an associated map visualisation:\r\n\r\nhttp://run.plnkr.co/preview/cj9zlf1qc0003414y90ajkwpk/\r\n\r\nDatasette is wonderful!\r\n\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/88/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1065431383, "node_id": "I_kwDOBm6k_c4_gTFX", "number": 1533, "title": "Add `Link: rel=\"alternate\"` header pointing to JSON for a table/query", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 4, "created_at": "2021-11-28T20:43:25Z", "updated_at": "2022-02-02T07:56:51Z", "closed_at": "2022-02-02T07:49:33Z", "author_association": "OWNER", "pull_request": null, "body": "Originally explored in https://github.com/simonw/datasette-notebook/issues/2#issuecomment-980789406 - I wanted an efficient way to scan a list of URLs and figure out which if any of those corresponded to Datasette tables, canned queries or SQL output that could be represented as a table on a page.\r\n\r\nIt looks like a neat way to do that is with ` Link:` header like this:\r\n\r\n`Link: http://127.0.0.1:8058/fixtures/compound_three_primary_keys.json; rel=\"alternate\"; type=\"application/datasette+json\"`\r\n\r\nI can put a `\r\n----\r\n:books: Documentation preview :books:: https://sqlite-utils--604.org.readthedocs.build/en/604/\r\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/604/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1870672704, "node_id": "PR_kwDOBm6k_c5Y-7Em", "number": 2162, "title": "Add new `--internal internal.db` option, deprecate legacy `_internal` database", "user": {"value": 15178711, "label": "asg017"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2023-08-29T00:05:07Z", "updated_at": "2023-08-29T03:24:23Z", "closed_at": "2023-08-29T03:24:23Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/datasette/pulls/2162", "body": "refs #2157 \r\n\r\nThis PR adds a new `--internal` option to datasette serve. If provided, it is the path to a persistent internal database that Datasette core and Datasette plugins can use to store data, as discussed in the proposal issue. \r\n\r\nThis PR also removes and deprecates the previous in-memory `_internal` database. Those tables now appear in the `internal` database, with `core_` prefixes (ex `tables` in `_internal` is now `core_tables` in `internal`).\r\n\r\n\r\n## A note on the new `core_` tables\r\nHowever, one important notes about those new `core_` tables: If a `--internal` DB is passed in, that means those `core_` tables will persist across multiple Datasette instances. This wasn't the case before, since `_internal` was always an in-memory database created from scratch.\r\n\r\nI tried to put those `core_` tables as `TEMP` tables - after all, there's always one 1 `internal` DB connection at a time, so I figured it would work. But, since we use the `Database()` wrapper for the internal DB, it has two separate connections: a default read-only connection and a write connection that is created when a write operation occurs. Which meant the `TEMP` tables would be created by the write connection, but not available in the read-only connection. \r\n\r\nSo I had a brillant idea: Attach an in-memory named database with `cache=shared`, and create those tables there! \r\n\r\n```sql\r\nATTACH DATABASE 'file:datasette_internal_core?mode=memory&cache=shared' AS core;\r\n```\r\n\r\nWe'd run this on both the read-only connection and the write-only connection. That way, those tables would stay in memory, they'd communicate with the `cache=shared` feature, and we'd be good to go.\r\n\r\n\r\nHowever, I couldn't find an easy way to run a `ATTACH DATABASE` command on the read-only query. \r\n\r\nUsing `Database()` as a wrapper for the internal DB is pretty limiting - it's meant for Datasette \"data\" databases, where we want multiple readers and possibly 1 write connection at a time. But the internal database doesn't really require that kind of support - I think we could get away with a single read/write connection, but it seemed like too big of a rabbithole to go through now. \r\n\r\n\r\n\r\n----\r\n:books: Documentation preview :books:: https://datasette--2162.org.readthedocs.build/en/2162/\r\n\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2162/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1620164673, "node_id": "PR_kwDOCGYnMM5L08O8", "number": 531, "title": "Add paths for homebrew on Apple silicon", "user": {"value": 25778, "label": "eyeseast"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2023-03-11T22:27:52Z", "updated_at": "2023-04-09T01:49:44Z", "closed_at": "2023-04-09T01:49:43Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/531", "body": "This also passes in the extension path when specified in GIS methods. Wherever we know an extension path, we use `db.init_spatialite(find_spatialite() or load_extension)`.\r\n\r\n\r\n----\n:books: Documentation preview :books:: https://sqlite-utils--531.org.readthedocs.build/en/531/\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/531/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 965102534, "node_id": "MDU6SXNzdWU5NjUxMDI1MzQ=", "number": 311, "title": "Add reference documentation generated from docstrings", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-08-10T16:04:00Z", "updated_at": "2021-08-11T12:03:50Z", "closed_at": "2021-08-11T12:03:50Z", "author_association": "OWNER", "pull_request": null, "body": "Using https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html\r\n\r\nI'm not a big fan of this kind of documentation because it so often comes in place of narrative documentation - but the library has great narrative documentation now, so the reference documentation can link to it in places.\r\n\r\nThis will also encourage me to add good docstrings everywhere, useful for IDEs and suchlike.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/311/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 729017519, "node_id": "MDExOlB1bGxSZXF1ZXN0NTA5NTkwMjA1", "number": 1049, "title": "Add template block prior to extra URL loaders", "user": {"value": 82988, "label": "psychemedia"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-10-25T13:08:55Z", "updated_at": "2020-10-29T09:20:52Z", "closed_at": "2020-10-29T09:20:34Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/datasette/pulls/1049", "body": "To handle packages that require Javascript state setting prior to loading a package (eg [`thebelab`](https://thebelab.readthedocs.io/en/latest/examples/minimal_example.html), provide a template block before the URLs are loaded.", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1049/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 487987958, "node_id": "MDExOlB1bGxSZXF1ZXN0MzEzMTA1NjM0", "number": 57, "title": "Add triggers while enabling FTS", "user": {"value": 49260, "label": "amjith"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2019-09-02T04:23:40Z", "updated_at": "2019-09-03T01:03:59Z", "closed_at": "2019-09-02T23:42:29Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/57", "body": "This adds the option for a user to set up triggers in the database to keep their FTS table in sync with the parent table. \r\n\r\nRef: https://sqlite.org/fts5.html#external_content_and_contentless_tables\r\n\r\nI would prefer to make the creation of triggers the default behavior, but that will break existing usage where people have been calling `populate_fts` after inserting new rows.\r\n\r\nI am happy to make changes to the PR as you see fit. ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/57/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 325352370, "node_id": "MDExOlB1bGxSZXF1ZXN0MTg5NzA3Mzc0", "number": 279, "title": "Add version number support with Versioneer", "user": {"value": 198537, "label": "rgieseke"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2018-05-22T15:39:45Z", "updated_at": "2018-05-22T19:35:23Z", "closed_at": "2018-05-22T19:35:22Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/datasette/pulls/279", "body": "I think that's all for getting Versioneer support, I've been happily using it in a couple of projects ... \r\n\r\n```\r\nIn [2]: datasette.__version__\r\nOut[2]: '0.22+3.g6e12445'\r\n```\r\nRepo:\r\nhttps://github.com/warner/python-versioneer\r\n\r\nVersioneer Licence:\r\nPublic Domain (CC0-1.0)\r\n\r\nCloses #273\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/279/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 593751293, "node_id": "MDU6SXNzdWU1OTM3NTEyOTM=", "number": 97, "title": "Adding a \"recreate\" flag to the `Database` constructor", "user": {"value": 1448859, "label": "betatim"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-04-04T05:41:10Z", "updated_at": "2020-04-15T14:29:31Z", "closed_at": "2020-04-13T03:52:29Z", "author_association": "NONE", "pull_request": null, "body": "I have a [script](https://github.com/betatim/binder-datasette/blob/master/create-db.ipynb) that imports data into a sqlite DB. When I re-run that script I'd like to remove the existing sqlite DB, instead of adding to it. The pragmatic answer is to add the check and file deletion to my script.\r\n\r\nHowever I thought it would be easy and useful for others to add a `recreate=True` flag to `db = sqlite_utils.Database(\"binder-launches.db\")`. After taking a look at the code for it I am not so sure any more. This is because the connection string could be a URL (or \"connection string\") like `\"file:///tmp/foo.db\"`. I don't know what the equivalent of `os.path.exists()` is for a connection string or how to detect that something is a connection string and raise an error \"can't use recreate=True and conn_string at the same time\".\r\n\r\nDoes anyone have an idea/suggestion where to start investigating?", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/97/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1342430983, "node_id": "I_kwDOBm6k_c5QA98H", "number": 1786, "title": "Adjust height of textarea for no JS case", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-08-18T01:15:15Z", "updated_at": "2022-10-27T21:50:12Z", "closed_at": "2022-08-18T16:06:09Z", "author_association": "OWNER", "pull_request": null, "body": "Datasette Lite: https://lite.datasette.io/?sql=https://gist.githubusercontent.com/simonw/1f8a91123ccefd8844187225b1832d7a/raw/5069075b86aa79358fbab3d4482d1d269077d632/recipes.sql#/data?sql=select+id%2C+name%2C+ingredients%2C+%28%0A++select+json_group_array%28value%29+from+json_each%28ingredients%29%0A++where+value+in+%28select+value+from+json_each%28%3Ap0%29%29%0A%29+as+matching_ingredients%0Afrom+recipes%0Awhere+json_array_length%28matching_ingredients%29+%3E+0%0Aorder+by+json_array_length%28matching_ingredients%29+desc&p0=%5B%22sugar%22%2C+%22cheese%22%5D\r\n\r\n![46F8101E-8CE3-4F61-B200-F865E6B5DBCC](https://user-images.githubusercontent.com/9599/185270723-f55513b0-b561-434d-9d7c-4fe5be9756e0.jpeg)\r\n\r\n\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1786/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 449848803, "node_id": "MDU6SXNzdWU0NDk4NDg4MDM=", "number": 25, "title": "Allow .insert(..., foreign_keys=()) to auto-detect table and primary key", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2019-05-29T14:39:22Z", "updated_at": "2019-06-13T05:32:32Z", "closed_at": "2019-06-13T05:32:32Z", "author_association": "OWNER", "pull_request": null, "body": "The `foreign_keys=` argument currently takes a list of triples:\r\n```python\r\ndb[\"usages\"].insert_all(\r\n usages_to_insert,\r\n foreign_keys=(\r\n (\"line_id\", \"lines\", \"id\"),\r\n (\"definition_id\", \"definitions\", \"id\"),\r\n ),\r\n)\r\n```\r\nAs of #16 we have a mechanism for detecting the primary key column (the third item in this triple) - we should use that here too, so foreign keys can be optionally defined as a list of pairs.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/25/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 548591089, "node_id": "MDU6SXNzdWU1NDg1OTEwODk=", "number": 657, "title": "Allow creation of virtual tables at startup", "user": {"value": 1055831, "label": "dazzag24"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-01-12T16:10:55Z", "updated_at": "2021-01-15T20:24:35Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Hi,\r\n \r\nI've been experimenting with SQLite reading from huge datasets using this excellent Parquet extension from @cldellow.\r\nhttps://cldellow.com/2018/06/22/sqlite-parquet-vtable.html\r\nhttps://github.com/cldellow/sqlite-parquet-vtable\r\n\r\nThis works really well, but I was keen to see if I could combine datasette with this. Having previously experimented with the spatialite extension I knew that datasette supports loading extensions in the underlying sqlite instance. However I hit a blocker as the current design only allows SELECT statements to be executed and so I am unable to execute the crucial \r\n\r\nCREATE VIRTUAL TABLE .........\r\n\r\ncommand that is required to load the data from the parquet file into the table.\r\n\r\nIt seems like this would be a simple-ish change, but I don't know enough about the architecture of datasette to start implementing this myself? Could this be done as a datasette plugin? or would this require more fundamental changes at initialisation time?\r\n\r\nMy thoughts are that something at init time could detect that the user was loading a *.parquet file and then switch to a mode were it loads that via the \"CREATE VIRTUAL TABLE...\" rather than loading the *.db file in the default case??\r\n\r\nI'm happy to contribute code and testing, I just need some pointers on the best approach.\r\n\r\nThanks\r\nDarren", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/657/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1077322009, "node_id": "I_kwDOCGYnMM5ANqEZ", "number": 355, "title": "Allow users to pass a full convert() function definition", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-12-10T23:59:58Z", "updated_at": "2021-12-11T00:51:15Z", "closed_at": "2021-12-11T00:49:31Z", "author_association": "OWNER", "pull_request": null, "body": "> I think the fix for this is to change the rules about what code is accepted in both the `-` mode and the literal code string mode: you can pass in a Python expression, OR a fragment that gets turned into a function, OR code that implements its own `def convert(value)` function. So this would work too:\r\n> ```sh\r\n> sqlite-utils convert my.db mytable col1 '\r\n> def convert(value):\r\n> return value.upper()\r\n> '\r\n> ```\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991381679_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/355/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 544571092, "node_id": "MDU6SXNzdWU1NDQ1NzEwOTI=", "number": 15, "title": "Assets table with downloads", "user": {"value": 2029, "label": "garethr"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5225818, "label": "1.0"}, "comments": 4, "created_at": "2020-01-02T13:05:28Z", "updated_at": "2020-03-28T12:17:01Z", "closed_at": "2020-03-23T19:17:32Z", "author_association": "NONE", "pull_request": null, "body": "The `releases` command extracts the releases table, but data about the individual assets are locked up in the JSON document in the `assets` field. My main interest is in individual and aggregate download counts. I was wondering if creating a new table with a record per asset may be useful?\r\nIf so I'm happy to send a PR when I get a moment. Do you have opinions about that simply being part of the `releases` command or would you prefer a separate command as well?", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1223459734, "node_id": "I_kwDOBm6k_c5I7IOW", "number": 1737, "title": "Automated test for Pyodide compatibility", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-05-02T23:24:25Z", "updated_at": "2022-05-02T23:40:50Z", "closed_at": "2022-05-02T23:40:50Z", "author_association": "OWNER", "pull_request": null, "body": "Refs:\r\n- #1733\r\n\r\nNeed something in the test suite such that if Datasette breaks against Pyodide in the future we hear about it.\r\n\r\nI'm thinking this is an opportunity to use [shot-scraper javascript](https://github.com/simonw/shot-scraper#scraping-pages-using-javascript).", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1737/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 925305186, "node_id": "MDU6SXNzdWU5MjUzMDUxODY=", "number": 282, "title": "Automatic type detection for CSV data", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-06-19T03:33:21Z", "updated_at": "2021-06-19T04:42:03Z", "closed_at": "2021-06-19T04:38:00Z", "author_association": "OWNER", "pull_request": null, "body": "I've touched on this before in #179 - but now that I've added `sqlite-utils memory` this is much more important - because unlike with `sqlite-utils insert` the in-memory command doesn't give you the opportunity to fix any types you imported from CSV, so queries like `select * from stdin where age > 3` are never going to work correctly against these temporary in-memory tables.\r\n\r\nTeaching `sqlite-utils insert` to detect types for columns in a CSV file would be a backwards-compatibility breaking change. Teaching `sqlite-utils memory` that trick would not be, since it hasn't been included in a release yet.\r\n\r\nIt's a little inconsistent, but I'm going to have `sqlite-utils memory` default to detecting types while `sqlite-utils insert` does not. In each case this can be controlled by a new command-line option:\r\n\r\n cat file.csv | sqlite-utils memory - --no-detect-types\r\n\r\nTo opt-in for `sqlite-utils insert`:\r\n\r\n cat file.csv | sqlite-utils insert blah.db blah - --detect-types\r\n\r\nI'll have short options for these too: `-n` for `--no-detect-types` and `-d` for `--detect-types`.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/282/reactions\", \"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 1, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 864979486, "node_id": "MDExOlB1bGxSZXF1ZXN0NjIxMTE3OTc4", "number": 1306, "title": "Avoid error sorting by relationships if related tables are not allowed", "user": {"value": 416374, "label": "gfrmin"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-04-22T13:53:17Z", "updated_at": "2021-06-02T04:27:00Z", "closed_at": "2021-06-02T04:25:28Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/datasette/pulls/1306", "body": "Refs #1305", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1306/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1094981339, "node_id": "I_kwDOCGYnMM5BRBbb", "number": 363, "title": "Better error message if `--convert` code fails to return a dict", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-01-06T05:26:28Z", "updated_at": "2022-02-03T22:52:30Z", "closed_at": "2022-02-03T22:51:30Z", "author_association": "OWNER", "pull_request": null, "body": "Here's the traceback if your `--convert` function doesn't return a dict right now:\r\n```\r\n% sqlite-utils insert /tmp/all.db blah /tmp/log.log --convert 'all.upper()' --all \r\n\r\nTraceback (most recent call last):\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/bin/sqlite-utils\", line 33, in \r\n sys.exit(load_entry_point('sqlite-utils', 'console_scripts', 'sqlite-utils')())\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 1137, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 1062, in main\r\n rv = self.invoke(ctx)\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 1668, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 1404, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 763, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py\", line 949, in insert\r\n insert_upsert_implementation(\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py\", line 834, in insert_upsert_implementation\r\n db[table].insert_all(\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py\", line 2602, in insert_all\r\n first_record = next(records)\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py\", line 3044, in fix_square_braces\r\n for record in records:\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py\", line 831, in \r\n docs = (decode_base64_values(doc) for doc in docs)\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py\", line 86, in decode_base64_values\r\n to_fix = [\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py\", line 89, in \r\n if isinstance(doc[k], dict)\r\nTypeError: string indices must be integers\r\n```\r\nIt would be nicer if that returned a more useful error message.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/361#issuecomment-1006295276_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/363/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 944870799, "node_id": "MDU6SXNzdWU5NDQ4NzA3OTk=", "number": 1394, "title": "Big performance boost on faceting: skip the inner order by", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-07-14T23:32:29Z", "updated_at": "2021-07-16T02:23:32Z", "closed_at": "2021-07-15T00:05:50Z", "author_association": "OWNER", "pull_request": null, "body": "I just noticed something that could make for a huge performance improvement in faceting.\r\n\r\nThe default query used by Datasette when faceting looks like this:\r\n```sql\r\nselect\r\n country_long,\r\n count(*)\r\nfrom (\r\n select * from [global-power-plants] order by rowid\r\n)\r\nwhere\r\n country_long is not null\r\ngroup by\r\n country_long\r\norder by\r\n count(*) desc\r\n```\r\nHere it takes 53ms: https://global-power-plants.datasettes.com/global-power-plants?sql=select%0D%0A++country_long%2C%0D%0A++count%28*%29%0D%0Afrom+%28%0D%0A++select+*+from+%5Bglobal-power-plants%5D+order+by+rowid%0D%0A%29%0D%0Awhere%0D%0A++country_long+is+not+null%0D%0Agroup+by%0D%0A++country_long%0D%0Aorder+by%0D%0A++count%28*%29+desc\r\n\r\nNote that there's a `order by rowid` in there which isn't necessary - the order on that inner query doesn't matter since we're grouping and counting.\r\n\r\nI had assumed SQLite would optimize this away - but it turns out it doesn't! Consider this version of the query, with that pointless order by removed:\r\n```\r\nselect\r\n country_long,\r\n count(*)\r\nfrom (\r\n select * from [global-power-plants]\r\n)\r\nwhere\r\n country_long is not null\r\ngroup by\r\n country_long\r\norder by\r\n count(*) desc\r\n```\r\nhttps://global-power-plants.datasettes.com/global-power-plants?sql=select%0D%0A++country_long%2C%0D%0A++count%28*%29%0D%0Afrom+%28%0D%0A++select+*+from+%5Bglobal-power-plants%5D%0D%0A%29%0D%0Awhere%0D%0A++country_long+is+not+null%0D%0Agroup+by%0D%0A++country_long%0D%0Aorder+by%0D%0A++count%28*%29+desc runs in 7.2ms!\r\n\r\nI tried this optimization on a table with 2.5m rows in it - without the optimization it took 5 seconds, with the optimization it took 450ms. So this is a very significant improvement!", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1394/reactions\", \"total_count\": 2, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 1, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 884952179, "node_id": "MDU6SXNzdWU4ODQ5NTIxNzk=", "number": 1320, "title": "Can't use apt-get in Dockerfile when using datasetteproj/datasette as base", "user": {"value": 2670795, "label": "brandonrobertz"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-05-10T19:37:27Z", "updated_at": "2021-05-24T18:15:56Z", "closed_at": "2021-05-24T18:07:08Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "The datasette base Docker image is super convenient, but there's one problem: if any of the plugins you install require additional system dependencies (e.g., xz, git, curl) then any attempt to use apt in said Dockerfile results in an explosion:\r\n\r\n```\r\n$ docker-compose build\r\nBuilding server\r\n[+] Building 9.9s (7/9)\r\n => [internal] load build definition from Dockerfile 0.0s\r\n => => transferring dockerfile: 666B 0.0s\r\n => [internal] load .dockerignore 0.0s\r\n => => transferring context: 34B 0.0s\r\n => [internal] load metadata for docker.io/datasetteproject/datasette:latest 0.6s\r\n => [base 1/4] FROM docker.io/datasetteproject/datasette@sha256:2250d0fbe57b1d615a8d6df0c9d43deb9533532e00bac68854773d8ff8dcf00a 0.0s\r\n => [internal] load build context 1.8s\r\n => => transferring context: 2.44MB 1.8s\r\n => CACHED [base 2/4] WORKDIR /datasette 0.0s\r\n => ERROR [base 3/4] RUN apt-get update && apt-get install --no-install-recommends -y git ssh curl xz-utils 9.2s\r\n------\r\n > [base 3/4] RUN apt-get update && apt-get install --no-install-recommends -y git ssh curl xz-utils:\r\n#6 0.446 Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]\r\n#6 0.449 Get:2 http://deb.debian.org/debian buster InRelease [121 kB]\r\n#6 0.459 Get:3 http://httpredir.debian.org/debian sid InRelease [157 kB]\r\n#6 0.784 Get:4 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]\r\n#6 0.790 Get:5 http://httpredir.debian.org/debian sid/main amd64 Packages [8626 kB]\r\n#6 1.003 Get:6 http://deb.debian.org/debian buster/main amd64 Packages [7907 kB]\r\n#6 1.180 Get:7 http://security.debian.org/debian-security buster/updates/main amd64 Packages [286 kB]\r\n#6 7.095 Get:8 http://deb.debian.org/debian buster-updates/main amd64 Packages [10.9 kB]\r\n#6 8.058 Fetched 17.2 MB in 8s (2243 kB/s)\r\n#6 8.058 Reading package lists...\r\n#6 9.166 E: flAbsPath on /var/lib/dpkg/status failed - realpath (2: No such file or directory)\r\n#6 9.166 E: Could not open file - open (2: No such file or directory)\r\n#6 9.166 E: Problem opening\r\n#6 9.166 E: The package lists or status file could not be parsed or opened.\r\n```\r\n\r\nThe problem seems to be from completely wiping out `/var/lib/dpkg` in the upstream Dockerfile:\r\n\r\nhttps://github.com/simonw/datasette/blob/1b697539f5b53cec3fe13c0f4ada13ba655c88c7/Dockerfile#L18\r\n\r\nI've tested without removing the directory and apt works as expected.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1320/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 919314806, "node_id": "MDU6SXNzdWU5MTkzMTQ4MDY=", "number": 270, "title": "Cannot set type JSON", "user": {"value": 4068, "label": "frafra"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-06-11T23:53:22Z", "updated_at": "2021-06-16T17:34:49Z", "closed_at": "2021-06-16T15:47:06Z", "author_association": "NONE", "pull_request": null, "body": "It would be great if the column type could be set to JSON. That would not be different from handling a regular string. It would be something like `repr(value)` and it would work with both JSON and CSV inputs, no matter if `value` is a real list or just a string representing a list.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/270/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 569317377, "node_id": "MDU6SXNzdWU1NjkzMTczNzc=", "number": 681, "title": "Cashe-header missing in http-response", "user": {"value": 2181410, "label": "clausjuhl"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-02-22T10:50:45Z", "updated_at": "2020-02-24T20:53:57Z", "closed_at": "2020-02-24T20:53:56Z", "author_association": "NONE", "pull_request": null, "body": "Hi Simon. I need some help with both understanding and adding http-headers. If I call datasette on localhost with --config default_cache_ttl:120 and --cors, I only get the following response-headers:\r\n\r\naccess-control-allow-origin: *\r\ncontent-type: text/html; charset=utf-8\r\ndate: Sat, 22 Feb 2020 10:32:15 GMT\r\nreferrer-policy: no-referrer\r\nserver: uvicorn\r\ntransfer-encoding: chunked\r\n\r\nCors works, but no caching-header is set? Same thing happens if I use the command in a Dockerfile and run datasette with docker.\r\n\r\nSecond, how can one add headers to uvicorn? I've tried to add uvicorn commands to the Dockerfile, before the final datasette command, but it doesn't work. Is there any way to add headers to the uvicorn.run() command i datasette? I particular, I would like to add some of the missing security-headers:\r\n\r\n\"Screenshot\r\n\r\nThank you for a great product!", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/681/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 781262510, "node_id": "MDU6SXNzdWU3ODEyNjI1MTA=", "number": 1181, "title": "Certain database names results in 404: \"Database not found: None\"", "user": {"value": 1470389, "label": "jieter"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 6346396, "label": "Datasette 0.54"}, "comments": 4, "created_at": "2021-01-07T12:01:16Z", "updated_at": "2021-12-21T18:25:15Z", "closed_at": "2021-01-25T05:13:19Z", "author_association": "NONE", "pull_request": null, "body": "I have a file named `test-database (1).sqlite`. When requesting the home route `/`, I see datasette is able to read it correctly:\r\n\r\n\"Screenshot\r\n\r\nHowever, if I click any of the links, datasette replies with: `Error 404 Database not found: None`\r\n\r\nIt seems the hash is crucial, as renaming the file to `database (1).sqlite` makes the error go away.\r\n\r\nThis lines checks for a single dash:\r\nhttps://github.com/simonw/datasette/blob/97fb10c17dd007a275ab743742e93e932335ad67/datasette/views/base.py#L184\r\n\r\n```\r\n$ datasette test-database\\ \\(1\\).sqlite \r\nINFO: Started server process [68314]\r\nINFO: Waiting for application startup.\r\nINFO: Application startup complete.\r\nINFO: Uvicorn running on http://127.0.0.1:8001 (Press CTRL+C to quit)\r\nINFO: 127.0.0.1:54043 - \"GET /favicon.ico HTTP/1.1\" 200 OK\r\nINFO: 127.0.0.1:54043 - \"GET / HTTP/1.1\" 200 OK\r\n...\r\nINFO: 127.0.0.1:54044 - \"GET /favicon.ico HTTP/1.1\" 200 OK\r\nINFO: 127.0.0.1:54044 - \"GET /test-database (1) HTTP/1.1\" 404 Not Found\r\n\r\n```\r\nVersion:\r\n```\r\n$ datasette --version\r\ndatasette, version 0.53\r\n```\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1181/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1452495049, "node_id": "I_kwDOBm6k_c5Wk1DJ", "number": 1899, "title": "Clicking within the CodeMirror area below the SQL (i.e. when there's only a single line) doesn't cause the editor to get focused ", "user": {"value": 95570, "label": "bgrins"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-11-17T00:29:52Z", "updated_at": "2022-11-18T07:28:28Z", "closed_at": "2022-11-18T07:20:53Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "After the upgrade to 6 (#1893) I noticed this. I think it's because we're doing overflow:hidden to accomplish the CSS resizer.\r\n\r\nWhen there's a single line of SQL there's a gap below that line where clicking doesn't do anything. It should focus at the end of the line.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1899/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 610829227, "node_id": "MDU6SXNzdWU2MTA4MjkyMjc=", "number": 749, "title": "Cloud Run fails to serve database files larger than 32MB", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-05-01T16:06:46Z", "updated_at": "2020-12-03T00:31:15Z", "closed_at": "2020-12-03T00:31:14Z", "author_association": "OWNER", "pull_request": null, "body": "https://cloud.google.com/run/quotas lists the maximum response size as 32MB.\r\n\r\nI spotted a bug where attempting to download a database file larger than that from a Cloud Run deployment (in this case it was https://github-to-sqlite.dogsheep.net/github.db after I [accidentally increased the size of that database](https://github.com/dogsheep/github-to-sqlite/commit/630bdba68a23c0ac453e015518ef0bf41107a952)) returned a 500 error because of this.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/749/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1257724585, "node_id": "I_kwDOCGYnMM5K91qp", "number": 441, "title": "Combining `rows_where()` and `search()` to limit which rows are searched", "user": {"value": 1448859, "label": "betatim"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-06-02T06:01:55Z", "updated_at": "2022-06-14T21:57:57Z", "closed_at": "2022-06-14T21:54:38Z", "author_association": "NONE", "pull_request": null, "body": "What is the right way to limit a full text search query to some rows of a table?\r\n\r\nFor example, I have a table that contains the following columns: `title`, `content`, `owner` (each row represents a document). The `owner` column is a username. It feels right to store all documents in one table, instead of having one table per owner. In particular because I'd like to full text search all documents, only documents owned by one user and documents owned by a set of users.\r\n\r\nI tried to combine `.rows_where(\"owner = ?\", \"1234\")` and `.search()` from the `Table` class but I don't think that is meant to work. I discovered `.search_sql()` as a way to generate the FTS SQL statement. By hand I can edit it to add a `AND [original].[owner] = :owner` to the `where` clause. This seems to do what I want.\r\n\r\nMy two questions:\r\n1. is adding a `AND ...` to the `where` clause actually the right thing to do or should I be doing something else (my SQL skills are low)?\r\n2. is there a built-in to sqlite-utils way to achieve this?\r\n\r\nRight now I am thinking I will make my own version of `search_sql()` that generates a query that contains an additional `owner = :owner` for my particular use-case.\r\n\r\nBonus question: is this generally useful/something to add to sqlite-utils or too niche?", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/441/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 952179830, "node_id": "MDU6SXNzdWU5NTIxNzk4MzA=", "number": 2, "title": "Command for fetching Hacker News threads from the search API", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-07-25T02:00:45Z", "updated_at": "2021-07-25T03:12:57Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "I want to be able to fetch every item for a domain, e.g. https://news.ycombinator.com/from?site=simonwillison.net", "repo": {"value": 248903544, "label": "hacker-news-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/2/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 505928530, "node_id": "MDU6SXNzdWU1MDU5Mjg1MzA=", "number": 18, "title": "Command to import home-timeline", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2019-10-11T15:47:54Z", "updated_at": "2019-10-11T16:51:33Z", "closed_at": "2019-10-11T16:51:12Z", "author_association": "MEMBER", "pull_request": null, "body": "Feature request: https://twitter.com/johankj/status/1182563563136868352\r\n> Would it be possible to save all tweets in my timeline from the last X days? I would love to see how big a percentage some users are of my daily timeline as a metric on whether I should unfollow them/move them to a list.", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/18/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1087919372, "node_id": "I_kwDOBm6k_c5A2FUM", "number": 1578, "title": "Confirm if documented nginx proxy config works for row pages with escaped characters in their primary key", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-12-23T18:27:59Z", "updated_at": "2021-12-24T21:33:19Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Found this while working on https://github.com/simonw/datasette-tiddlywiki\r\n\r\n\"image\"\r\n\r\nThen clicking on `/tiddlywiki/tiddlers/%24%3A%2FDefaultTiddlers` returns a 404.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1578/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 274314940, "node_id": "MDU6SXNzdWUyNzQzMTQ5NDA=", "number": 105, "title": "Consider data-package as a format for metadata", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2017-11-15T21:43:34Z", "updated_at": "2017-11-20T19:50:53Z", "closed_at": "2017-11-20T19:50:53Z", "author_association": "OWNER", "pull_request": null, "body": "http://frictionlessdata.io/specs/data-package/", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/105/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1326087800, "node_id": "PR_kwDOCGYnMM48hI-_", "number": 460, "title": "Cross-link CLI to Python docs", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-08-02T16:18:28Z", "updated_at": "2022-08-18T21:58:10Z", "closed_at": "2022-08-18T21:58:07Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/460", "body": "Work in progress, partly to test the ReadTheDocs preview link action.\r\n\r\nRefs:\r\n- #426\r\n\r\n\r\n----\n:books: Documentation preview :books:: https://readthedocs-preview--460.org.readthedocs.build/en/460/\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/460/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1646734246, "node_id": "I_kwDOBm6k_c5iJyum", "number": 2049, "title": "Custom SQL queries should use new JSON ?_extra= format", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 8755003, "label": "Datasette 1.0a-next"}, "comments": 4, "created_at": "2023-03-30T00:42:53Z", "updated_at": "2023-04-05T23:29:27Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Related:\r\n- #262\r\n\r\nI've made the change to the table view, now I need the new format to work for arbitrary SQL queries too.\r\n\r\nNote that this incorporates both arbitrary SQL queries and canned queries.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2049/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 607067303, "node_id": "MDExOlB1bGxSZXF1ZXN0NDA5MTIzODk3", "number": 737, "title": "Custom pages mechanism, refs #648", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-04-26T17:31:41Z", "updated_at": "2020-04-26T18:46:43Z", "closed_at": "2020-04-26T18:46:43Z", "author_association": "OWNER", "pull_request": "simonw/datasette/pulls/737", "body": "Refs #648. TODO:\r\n- [x] Pass a `view_name` to `render_template()`\r\n- [x] Mechanism for custom status code / headers / redirect\r\n- [x] Documentation", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/737/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1686033652, "node_id": "I_kwDOBm6k_c5kftT0", "number": 2065, "title": "Datasette cannot be installed with Rye", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2023-04-27T03:35:42Z", "updated_at": "2023-04-27T05:09:36Z", "closed_at": "2023-04-27T05:09:36Z", "author_association": "OWNER", "pull_request": null, "body": "https://github.com/mitsuhiko/rye\r\n\r\nI tried this:\r\n\r\n rye install datasette\r\n\r\nBut now:\r\n\r\n```\r\n% ~/.rye/shims/datasette\r\nTraceback (most recent call last):\r\n File \"/Users/simon/.rye/shims/datasette\", line 5, in \r\n from datasette.cli import cli\r\n File \"/Users/simon/.rye/tools/datasette/lib/python3.11/site-packages/datasette/cli.py\", line 17, in \r\n from .app import (\r\n File \"/Users/simon/.rye/tools/datasette/lib/python3.11/site-packages/datasette/app.py\", line 14, in \r\n import pkg_resources\r\nModuleNotFoundError: No module named 'pkg_resources'\r\n```\r\nI think that's because `setuptools` is not included in Rye.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2065/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1553615704, "node_id": "I_kwDOBm6k_c5cmktY", "number": 2001, "title": "Datasette is not compatible with SQLite's strict quoting compilation option", "user": {"value": 406380, "label": "gwk"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2023-01-23T19:10:07Z", "updated_at": "2023-01-25T04:59:58Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I have linked Python3.11 on macOS against recent SQLite that was compiled using `-DSQLITE_DQS=0`. This option disables interpretation of double-quoted identifiers as string literals, described in the SQLite docs as a \"MySQL 3.x misfeature\". See https://www.sqlite.org/quirks.html#dblquote for background.\r\n\r\nDatasette uses the double-quote syntax in a number of key places, and is thus completely broken in this environment.\r\n\r\nMy experience was to `pip install datasette`, then run `datasette serve -I my-data.db`. When I visit `http://127.0.0.1:8001` I get a 500 response.\r\n\r\nThe error: `sqlite3.OperationalError: no such column: geometry_columns`\r\n\r\nThe responsible SQL: `'select 1 from sqlite_master where tbl_name = \"geometry_columns\"'`\r\n\r\nI then installed datasette from GitHub master in development mode and changed the offending SQL to use correct quotes: `\"select 1 from sqlite_master where tbl_name = 'geometry_columns'\"`.\r\n\r\nWith this change, I get a little further, but have the same problem with the first table name in my database (in my case, \"Meta\"):\r\n```\r\nOperationalError: no such column: Meta\r\nTraceback (most recent call last):\r\n File \"/Users/gwk/external/datasette/datasette/app.py\", line 1522, in route_path\r\n response = await view(request, send)\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/Users/gwk/external/datasette/datasette/views/base.py\", line 151, in view\r\n return await self.dispatch_request(request)\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/Users/gwk/external/datasette/datasette/views/base.py\", line 105, in dispatch_request\r\n response = await handler(request)\r\n ^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/Users/gwk/external/datasette/datasette/views/index.py\", line 70, in get\r\n \"fts_table\": await db.fts_table(table),\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/Users/gwk/external/datasette/datasette/database.py\", line 363, in fts_table\r\n return await self.execute_fn(lambda conn: detect_fts(conn, table))\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/Users/gwk/external/datasette/datasette/database.py\", line 213, in execute_fn\r\n return await asyncio.get_event_loop().run_in_executor(\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/usr/local/py/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/thread.py\", line 58, in run\r\n result = self.fn(*self.args, **self.kwargs)\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/Users/gwk/external/datasette/datasette/database.py\", line 211, in in_thread\r\n return fn(conn)\r\n ^^^^^^^^\r\n File \"/Users/gwk/external/datasette/datasette/database.py\", line 363, in \r\n return await self.execute_fn(lambda conn: detect_fts(conn, table))\r\n ^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/Users/gwk/external/datasette/datasette/utils/__init__.py\", line 588, in detect_fts\r\n rows = conn.execute(detect_fts_sql(table)).fetchall()\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\nsqlite3.OperationalError: no such column: Meta\r\nINFO: 127.0.0.1:50258 - \"GET / HTTP/1.1\" 500 Internal Server Error\r\n```\r\n\r\nI will try to continue playing with this, but I also hope that the datasette developers will enable this mode in a test environment as I am unlikely to be able to exercise all of the SQL in the codebase, or make a pull request very soon.\r\n\r\nNote that the DQS setting compile-time option can be overridden at runtime with calls to the C API:\r\n```\r\nsqlite3_db_config(db, SQLITE_DBCONFIG_DQS_DDL, 0, (void*)0);\r\nsqlite3_db_config(db, SQLITE_DBCONFIG_DQS_DML, 0, (void*)0);\r\n```\r\n\r\nAs far as I can tell, `sqlite3_db_config` is not exposed in Python, but perhaps we could figure out how to invoke it using `ctypes`.\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2001/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 338768551, "node_id": "MDU6SXNzdWUzMzg3Njg1NTE=", "number": 333, "title": "Datasette on Zeit Now returns http URLs for facet and next links", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2018-07-06T00:40:49Z", "updated_at": "2018-07-24T04:53:20Z", "closed_at": "2018-07-24T01:51:53Z", "author_association": "OWNER", "pull_request": null, "body": "e.g. on https://fivethirtyeight.datasettes.com/fivethirtyeight-ac35616/nba-elo%2Fnbaallelo.json?_facet=lg_id&_size=0\r\n```\r\n{\r\n \"facet_results\": {\r\n \"lg_id\": {\r\n \"name\": \"lg_id\",\r\n \"results\": [\r\n {\r\n \"value\": \"NBA\",\r\n \"label\": \"NBA\",\r\n \"count\": 118016,\r\n \"toggle_url\": \"http://fivethirtyeight.datasettes.com/fivethirtyeight-ac35616/nba-elo%2Fnbaallelo.json?_facet=lg_id&_size=1&lg_id=NBA\",\r\n \"selected\": false\r\n },\r\n {\r\n \"value\": \"ABA\",\r\n \"label\": \"ABA\",\r\n \"count\": 8298,\r\n \"toggle_url\": \"http://fivethirtyeight.datasettes.com/fivethirtyeight-ac35616/nba-elo%2Fnbaallelo.json?_facet=lg_id&_size=1&lg_id=ABA\",\r\n \"selected\": false\r\n }\r\n ],\r\n \"truncated\": false\r\n }\r\n },\r\n \"suggested_facets\": [\r\n {\r\n \"name\": \"_iscopy\",\r\n \"toggle_url\": \"/fivethirtyeight-ac35616/nba-elo%2Fnbaallelo.json?_facet=lg_id&_size=1&_facet=_iscopy\"\r\n }\r\n ],\r\n \"next_url\": \"http://fivethirtyeight.datasettes.com/fivethirtyeight-ac35616/nba-elo%2Fnbaallelo.json?_facet=lg_id&_size=1&_next=1\",\r\n}\r\n```\r\n`next_url` and `facet_results` both link to `http://` when they should link to `https://`.\r\n\r\nNote that suggested facets doesn't include the full URL at all, which is a consistency bug.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/333/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 520715188, "node_id": "MDU6SXNzdWU1MjA3MTUxODg=", "number": 622, "title": "Datasette should work with Python 3.8 (and drop compatibility with Python 3.5)", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2019-11-11T03:12:36Z", "updated_at": "2019-11-12T05:52:49Z", "closed_at": "2019-11-12T05:09:13Z", "author_association": "OWNER", "pull_request": null, "body": "See #595, #594, #404.\r\n\r\nThe big thing holding me back from ditching Python 3.5 was glitch.com - but they now offer Python 3.7: https://support.glitch.com/t/can-you-upgrade-python-to-latest-version/7980/25?u=simonw", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/622/reactions\", \"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 1, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1433576351, "node_id": "I_kwDOBm6k_c5VcqOf", "number": 1880, "title": "Datasette with many and large databases > Memory use", "user": {"value": 525934, "label": "amitkoth"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-11-02T18:10:27Z", "updated_at": "2022-11-16T17:50:29Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "> Datasette maintains an in-memory SQLite database with details of the the databases, tables and columns for all of the attached databases.\r\n\r\nThe above is from the docs ^. There's two problems here - the number of datasette \"instances\" in a single server/VM and the size of the database itself. We want the **opposite** of in-memory, including what happens on SQLlite - documented in https://www.sqlite.org/inmemorydb.html\r\n\r\nFrom the context in https://github.com/simonw/datasette/issues/1150 - does it mean datasette is memory-bound to the size of the dataset - which might be a deal-breaker for many large-scale use cases?\r\n\r\nIn an extreme case - let's say a single server had 100 SQLlite databases, which would enable 100 \"instances\" of datasette to run, one per client (e.g. in a SaaS multi-tenant environment). How could we achieve all these goals:\r\n\r\n1. Allow any _one_ of these 100 databases to grow to say 2Tb in size \r\n2. Have one datasette instance, which connects to 1 of the 100 instances, based on incoming credentials/tenant ID\r\n3. Minimize memory use entirely - both by datasette and SQLlite, such that almost all operations are executed in real-time on-disk with little to no memory consumption per-tenant, or per-database.\r\n\r\nAny ideas appreciated - we're looking to use this in a SaaS type of setting - many instances, single server.\r\n\r\n@simonw great work on datasette, in general! Possibly related to https://github.com/simonw/datasette/issues/1480 but we don't want use any kind of serverless infra - this is a long-running VM/server.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1880/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 459590021, "node_id": "MDU6SXNzdWU0NTk1OTAwMjE=", "number": 519, "title": "Decide what goes into Datasette 1.0", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 4, "created_at": "2019-06-23T15:47:41Z", "updated_at": "2021-11-15T23:26:11Z", "closed_at": "2021-11-15T23:26:11Z", "author_association": "OWNER", "pull_request": null, "body": "Datasette ASGI #272 is a big part of it... but 1.0 will generally be an indicator that Datasette is a stable platform for developers to write plugins and custom templates against. So lots to think about.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/519/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 443038584, "node_id": "MDU6SXNzdWU0NDMwMzg1ODQ=", "number": 465, "title": "Decide what to do about /-/inspect", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2019-05-11T21:39:46Z", "updated_at": "2019-06-28T16:34:33Z", "closed_at": "2019-06-28T16:34:33Z", "author_association": "OWNER", "pull_request": null, "body": "It's not clear to me what this endpoint should do now as a result of #419 - it's still useful to be able to introspect databases for tools like datasette-registry, but since we aren't pre-calculating introspection data any more I need to rethink the approach.\r\n\r\nFor one thing, this endpoint may need to be paginated. Or maybe it should be split up into separate endpoints for each connected database? Those should probably be paginated too seeing as fivethirtyeight has 400+ tables.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/465/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1425029275, "node_id": "I_kwDOBm6k_c5U8Dib", "number": 1864, "title": "Delete a single record from an existing table", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 8658075, "label": "Datasette 1.0a0"}, "comments": 4, "created_at": "2022-10-27T04:53:22Z", "updated_at": "2022-11-29T18:54:04Z", "closed_at": "2022-11-29T18:54:04Z", "author_association": "OWNER", "pull_request": null, "body": "API design:\r\n```\r\nPOST /db/table/row-pks/-/delete\r\nOr...\r\nDELETE /db/table/row-pks/-/delete\r\n```\r\nI'm just going to do `POST` for the moment, like I did here:\r\n- #1874\r\n\r\nPermission: `delete-row`\r\n\r\nStill needed:\r\n\r\n- [ ] Tests for rowid tables\r\n- [ ] Tests for compound primary keys", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1864/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 610192152, "node_id": "MDU6SXNzdWU2MTAxOTIxNTI=", "number": 747, "title": "Directory configuration mode should support metadata.yaml", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-04-30T16:05:30Z", "updated_at": "2020-04-30T19:04:19Z", "closed_at": "2020-04-30T19:04:19Z", "author_association": "OWNER", "pull_request": null, "body": "Refs #739 - `metadata.yml` or `metadata.yaml` should be detected in the same way as `metadata.json` is.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/747/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 855446829, "node_id": "MDExOlB1bGxSZXF1ZXN0NjEzMTc4OTY4", "number": 1296, "title": "Dockerfile: use Ubuntu 20.10 as base", "user": {"value": 82332573, "label": "tmcl-it"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-04-12T00:23:32Z", "updated_at": "2021-07-20T08:52:13Z", "closed_at": null, "author_association": "FIRST_TIME_CONTRIBUTOR", "pull_request": "simonw/datasette/pulls/1296", "body": "This PR changes the main Dockerfile to use ubuntu:20.10 as base image instead of python:3.9.2-slim-buster (itself based on debian:buster-slim).\r\n\r\nThe Dockerfile is essentially the one from https://github.com/simonw/datasette/issues/1249#issuecomment-803698983 with some additional cleanups to slim it down.\r\n\r\nThis fixes a couple of issues:\r\n1. The SQLite version in Debian Buster (2.6.0) doesn't support generated columns\r\n2. Installing SpatiaLite from the Debian sid repositories has the side effect of also installing updates to libc and libstdc++ from sid.\r\n\r\nAs a bonus, the Docker image becomes smaller:\r\n\r\n\r\n```\r\n$ docker image ls\r\nREPOSITORY TAG IMAGE ID CREATED SIZE\r\ndatasette 0.56-ubuntu f7aca255140a 5 hours ago 212MB\r\ndatasetteproject/datasette 0.56 efb3b282f390 13 days ago 258MB\r\n```\r\n\r\n### Reproduction of the first issue\r\n\r\n```\r\n$ curl -O https://latest.datasette.io/fixtures.db\r\n % Total % Received % Xferd Average Speed Time Time Time Current\r\n Dload Upload Total Spent Left Speed\r\n100 260k 0 260k 0 0 489k 0 --:--:-- --:--:-- --:--:-- 489k\r\n\r\n$ docker run -v `pwd`:/mnt datasetteproject/datasette:0.56 datasette /mnt/fixtures.db\r\nTraceback (most recent call last):\r\n File \"/usr/local/bin/datasette\", line 8, in \r\n sys.exit(cli())\r\n File \"/usr/local/lib/python3.9/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/usr/local/lib/python3.9/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/usr/local/lib/python3.9/site-packages/click/core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/usr/local/lib/python3.9/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/usr/local/lib/python3.9/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/usr/local/lib/python3.9/site-packages/datasette/cli.py\", line 544, in serve\r\n asyncio.get_event_loop().run_until_complete(check_databases(ds))\r\n File \"/usr/local/lib/python3.9/asyncio/base_events.py\", line 642, in run_until_complete\r\n return future.result()\r\n File \"/usr/local/lib/python3.9/site-packages/datasette/cli.py\", line 584, in check_databases\r\n await database.execute_fn(check_connection)\r\n File \"/usr/local/lib/python3.9/site-packages/datasette/database.py\", line 155, in execute_fn\r\n return await asyncio.get_event_loop().run_in_executor(\r\n File \"/usr/local/lib/python3.9/concurrent/futures/thread.py\", line 52, in run\r\n result = self.fn(*self.args, **self.kwargs)\r\n File \"/usr/local/lib/python3.9/site-packages/datasette/database.py\", line 153, in in_thread\r\n return fn(conn)\r\n File \"/usr/local/lib/python3.9/site-packages/datasette/utils/__init__.py\", line 892, in check_connection\r\n for r in conn.execute(\r\nsqlite3.DatabaseError: malformed database schema (generated_columns) - near \"AS\": syntax error\r\n```\r\n\r\nHere is the SQLite version:\r\n\r\n```\r\n$ docker run -v `pwd`:/mnt -it datasetteproject/datasette:0.56 /bin/bash\r\nroot@d9220d3b95dd:/# python3\r\nPython 3.9.2 (default, Mar 27 2021, 02:50:26) \r\n[GCC 8.3.0] on linux\r\nType \"help\", \"copyright\", \"credits\" or \"license\" for more information.\r\n>>> import sqlite3\r\n>>> sqlite3.version\r\n'2.6.0'\r\n```\r\n\r\n### Reproduction of the second issue\r\n\r\n```\r\n$ docker build . -t datasette --build-arg VERSION=0.55\r\n[...snip...]\r\nThe following packages will be upgraded:\r\n libc-bin libc6 libstdc++6\r\n[...snip...]\r\nUnpacking libc6:amd64 (2.31-11) over (2.28-10) ...\r\n[...snip...]\r\nUnpacking libstdc++6:amd64 (10.2.1-6) over (8.3.0-6) ...\r\n[...snip...]\r\n```\r\n\r\nBoth libc and libstdc++ are backwards compatible, so the image still works, but it will result in a combination of libraries and Python versions that exists only in the Datasette image, so it's likely untested. In addition, since Debian sid is an always-changing rolling-release, the versions of libc, libstdc++, Spatialite, and their dependencies change frequently, so the library versions in the Datasette image will depend on the day when it was built.\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1296/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1470509936, "node_id": "I_kwDOBm6k_c5XpjNw", "number": 1924, "title": "Docs for replace:true and ignore:true options for insert API", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7867486, "label": "Datasette 1.0a1"}, "comments": 4, "created_at": "2022-12-01T01:33:25Z", "updated_at": "2022-12-01T18:15:15Z", "closed_at": "2022-12-01T02:08:02Z", "author_association": "OWNER", "pull_request": null, "body": "Equivalent to https://sqlite-utils.datasette.io/en/stable/cli.html#insert-replacing-data", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1924/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 963527045, "node_id": "MDU6SXNzdWU5NjM1MjcwNDU=", "number": 1424, "title": "Document exceptions that can be raised by db.execute() and friends", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-08-08T22:23:25Z", "updated_at": "2021-08-08T22:27:31Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Not currently covered here: https://docs.datasette.io/en/stable/internals.html#await-db-execute-sql", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1424/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1126692066, "node_id": "I_kwDOCGYnMM5DJ_Ti", "number": 403, "title": "Document how to add a primary key to a rowid table using `sqlite-utils transform --pk`", "user": {"value": 536941, "label": "fgregg"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-02-08T01:39:40Z", "updated_at": "2022-02-09T04:22:43Z", "closed_at": "2022-02-08T19:33:59Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "*Original title: Add option for adding a new, serial, primary key*\r\n\r\nsometimes we have tables that don't have primary keys, but ought to have them. we *can* use rowid for that, but it would often be nicer to have an explicit primary key. using the current value of rowid would be fine.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/403/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 863884805, "node_id": "MDU6SXNzdWU4NjM4ODQ4MDU=", "number": 1304, "title": "Document how to send multiple values for \"Named parameters\" ", "user": {"value": 9308268, "label": "rayvoelker"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-04-21T13:19:06Z", "updated_at": "2021-12-08T03:23:14Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "https://docs.datasette.io/en/stable/sql_queries.html#named-parameters\r\n\r\nI thought that I had seen an example of how to do this example below, but I can't seem to find it\r\n\r\n```sql\r\nselect\r\n *\r\nfrom\r\n bib\r\nwhere\r\n bib.bib_record_num in (1008088,1008092)\r\n```\r\n\r\n```sql\r\nselect\r\n *\r\nfrom\r\n bib\r\nwhere\r\n bib.bib_record_num in (:bib_record_numbers)\r\n```\r\n![image](https://user-images.githubusercontent.com/9308268/115558839-2333a480-a281-11eb-85e6-ce3bada79140.png)\r\n\r\nhttps://ilsweb.cincinnatilibrary.org/collection-analysis/current_collection-204d100?sql=select%0D%0A++*%0D%0Afrom%0D%0A++bib%0D%0Awhere%0D%0A++bib.bib_record_num+in+%28%3Abib_record_numbers%29&bib_record_numbers=1008088%2C1008092\r\n\r\nOr, maybe this isn't a fully supported feature.\r\n\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1304/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 273248366, "node_id": "MDU6SXNzdWUyNzMyNDgzNjY=", "number": 69, "title": "Enforce pagination (or at least limits) for arbitrary custom SQL", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 2857392, "label": "Ship first public release"}, "comments": 4, "created_at": "2017-11-12T17:21:33Z", "updated_at": "2017-11-13T20:32:47Z", "closed_at": "2017-11-13T19:35:47Z", "author_association": "OWNER", "pull_request": null, "body": "It's way too easy to accidentally trigger a page that returns 100,000 rows at the moment. I need to use the LIMIT clause on views and custom SQL - I can support pagination \"next\" links using offset as well.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/69/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 626171242, "node_id": "MDU6SXNzdWU2MjYxNzEyNDI=", "number": 777, "title": "Error pages not correctly loading CSS", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5512395, "label": "Datasette 0.44"}, "comments": 4, "created_at": "2020-05-28T02:47:52Z", "updated_at": "2020-06-09T00:35:29Z", "closed_at": "2020-06-09T00:35:29Z", "author_association": "OWNER", "pull_request": null, "body": "e.g. https://latest.datasette.io/fixtures/compound_three_primary_keys.tsv?_size=max\r\n\r\n\"Error_404_and_All_plugin_hooks_should_have_unit_tests_\u00b7_Issue__773_\u00b7_simonw_datasette\"\r\n\r\nThe HTML starts like this:\r\n\r\n```html\r\n\r\n\r\n\r\n Error 404\r\n \r\n```", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/777/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 807174161, "node_id": "MDU6SXNzdWU4MDcxNzQxNjE=", "number": 227, "title": "Error reading csv files with large column data", "user": {"value": 295329, "label": "camallen"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-02-12T11:51:47Z", "updated_at": "2021-02-16T11:48:03Z", "closed_at": "2021-02-14T21:17:19Z", "author_association": "NONE", "pull_request": null, "body": "*Feel free to close this issue - I mostly added it for reference for future folks that run into this :)*\r\n\r\nI have a CSV file with one column that has very long strings. When i try to import this file via the `insert` command I get the following error: \r\n```\r\nsqlite-utils insert database.db table_name file_with_large_column.csv\r\n\r\nTraceback (most recent call last):\r\n File \"/usr/local/bin/sqlite-utils\", line 10, in \r\n sys.exit(cli())\r\n File \"/usr/local/lib/python3.7/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/usr/local/lib/python3.7/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/usr/local/lib/python3.7/site-packages/click/core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/usr/local/lib/python3.7/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/usr/local/lib/python3.7/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/usr/local/lib/python3.7/site-packages/sqlite_utils/cli.py\", line 774, in insert\r\n default=default,\r\n File \"/usr/local/lib/python3.7/site-packages/sqlite_utils/cli.py\", line 705, in insert_upsert_implementation\r\n docs, pk=pk, batch_size=batch_size, alter=alter, **extra_kwargs\r\n File \"/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py\", line 1852, in insert_all\r\n first_record = next(records)\r\n File \"/usr/local/lib/python3.7/site-packages/sqlite_utils/cli.py\", line 703, in \r\n docs = (decode_base64_values(doc) for doc in docs)\r\n File \"/usr/local/lib/python3.7/site-packages/sqlite_utils/cli.py\", line 681, in \r\n docs = (dict(zip(headers, row)) for row in reader)\r\n_csv.Error: field larger than field limit (131072)\r\n```\r\nBuilt with the docker image `datasetteproject/datasette:0.54` with the following versions:\r\n```\r\n# sqlite-utils --version\r\nsqlite-utils, version 3.4.1\r\n\r\n# datasette --version\r\ndatasette, version 0.54\r\n```\r\nIt appears this is a [known issue](https://stackoverflow.com/a/54517228/2761423) reading in csv files in python and [doesn't look to be modifiable](https://github.com/python/cpython/blob/ea46579067fd2d4e164d6605719ffec690c4d621/Modules/_csv.c#L1685) through system / env vars (i may be very wrong on this).\r\n\r\nNoting that using sqlite3 `import` command work without error (not using the python csv reader)\r\n```\r\nsqlite3 database.db\r\nsqlite> .mode csv\r\nsqlite> .import file_with_large_column.csv table_name\r\n```\r\nSadly I couldn't see an easy way around this while using the cli as it appears this value needs to be changed in python code. FWIW I've switched to using https://datasette.io/tools/csvs-to-sqlite for importing csv data and it's working well. \r\n\r\nFinally, I'm loving https://datasette.io/ thank you very much for an amazing tool and data ecosytem \ud83d\ude47\u200d\u2640\ufe0f ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/227/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 831751367, "node_id": "MDU6SXNzdWU4MzE3NTEzNjc=", "number": 246, "title": "Escaping FTS search strings", "user": {"value": 16001974, "label": "DeNeutoy"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-03-15T12:15:09Z", "updated_at": "2021-08-18T18:57:13Z", "closed_at": "2021-08-18T18:43:12Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "\r\nThanks for the excellent library, it's very nice to use!\r\n\r\nI've been building some in memory search functionality for a data annotation tool i'm making, and I got tripped up a little bit with escaping the full text search queries. First I tried using `db.quote(q)`, which doesn't work, because sqlite FTS has it's own (separate)[ query syntax](https://www2.sqlite.org/fts5.html#full_text_query_syntax). You can see this happening here also:\r\n\r\nhttp://search-24ways.herokuapp.com/24ways-f8f455f/articles?_search=acces%2A\r\n\r\nI got around this by aggressively escaping quotes inside the query string like this:\r\n\r\n```python\r\n quoted = q.replace('\"', '\"\"')\r\n quoted = f'\"{quoted}\"'\r\n print(quoted)\r\n results = db[\"data\"].search(quoted, columns=[\"id\"])\r\n return [x[\"id\"] for x in results]\r\n\r\n```\r\n\r\nThis works in the sense it doesn't crash, but it also removes access to the search query syntax. Given the well specified definition, it might be possible for sqlite-utils to provide a `db.quote_query(q)` which would intelligently escape a query whilst leaving the syntax intact. This would be very nice!\r\n\r\n\r\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/246/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 322283067, "node_id": "MDU6SXNzdWUzMjIyODMwNjc=", "number": 254, "title": "Escaping named parameters in canned queries", "user": {"value": 247131, "label": "philroche"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2018-05-11T12:43:30Z", "updated_at": "2020-05-10T14:54:14Z", "closed_at": "2020-05-10T14:54:13Z", "author_association": "NONE", "pull_request": null, "body": "Thank you very much for this project.\r\n\r\nI have created some canned queries but some of the filters include a colon eg. \"com.ubuntu.cloud:server:18.04:amd64\". When saved these colons are parsed as named parameters. \r\n\r\nIs there a way to escape colons in a canned query?", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/254/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 455852801, "node_id": "MDU6SXNzdWU0NTU4NTI4MDE=", "number": 507, "title": "Every datasette plugin on the ecosystem page should have a screenshot", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2019-06-13T17:02:51Z", "updated_at": "2020-09-17T02:47:35Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "https://github.com/simonw/datasette/blob/master/docs/ecosystem.rst", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/507/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 635108074, "node_id": "MDU6SXNzdWU2MzUxMDgwNzQ=", "number": 824, "title": "Example authentication plugin", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5512395, "label": "Datasette 0.44"}, "comments": 4, "created_at": "2020-06-09T04:49:53Z", "updated_at": "2020-06-12T00:11:51Z", "closed_at": "2020-06-12T00:11:50Z", "author_association": "OWNER", "pull_request": null, "body": "https://github.com/simonw/datasette-auth-github/issues/62 will work for this.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/824/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 546051181, "node_id": "MDU6SXNzdWU1NDYwNTExODE=", "number": 16, "title": "Exception running first command: IndexError: list index out of range", "user": {"value": 15092, "label": "jayvdb"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-01-07T03:01:58Z", "updated_at": "2020-04-14T18:37:21Z", "closed_at": "2020-04-14T18:37:21Z", "author_association": "NONE", "pull_request": null, "body": "Exception running first command without an existing db or auth.\r\n\r\n```py\r\n> mkdir ~/.github/coala\r\n> /usr/bin/github-to-sqlite repos ~/.github/coala coala\r\nTraceback (most recent call last):\r\n File \"/usr/bin/github-to-sqlite\", line 11, in \r\n load_entry_point('github-to-sqlite==0.6', 'console_scripts', 'github-to-sqlite')()\r\n File \"/usr/lib/python3.7/site-packages/click/core.py\", line 764, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/usr/lib/python3.7/site-packages/click/core.py\", line 717, in main\r\n rv = self.invoke(ctx)\r\n File \"/usr/lib/python3.7/site-packages/click/core.py\", line 1137, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/usr/lib/python3.7/site-packages/click/core.py\", line 956, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/usr/lib/python3.7/site-packages/click/core.py\", line 555, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/usr/lib/python3.7/site-packages/github_to_sqlite/cli.py\", line 163, in repos\r\n utils.save_repo(db, repo)\r\n File \"/usr/lib/python3.7/site-packages/github_to_sqlite/utils.py\", line 120, in save_repo\r\n to_save[\"owner\"] = save_user(db, to_save[\"owner\"])\r\n File \"/usr/lib/python3.7/site-packages/github_to_sqlite/utils.py\", line 61, in save_user\r\n return db[\"users\"].upsert(to_save, pk=\"id\", alter=True).last_pk\r\n File \"/usr/lib/python3.7/site-packages/sqlite_utils/db.py\", line 1135, in upsert\r\n extracts=extracts,\r\n File \"/usr/lib/python3.7/site-packages/sqlite_utils/db.py\", line 1162, in upsert_all\r\n upsert=True,\r\n File \"/usr/lib/python3.7/site-packages/sqlite_utils/db.py\", line 1105, in insert_all\r\n row = list(self.rows_where(\"rowid = ?\", [self.last_rowid]))[0]\r\nIndexError: list index out of range\r\n```", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/16/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 320592643, "node_id": "MDU6SXNzdWUzMjA1OTI2NDM=", "number": 251, "title": "Explore \"distinct values for column\" in inspect()", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2018-05-06T13:27:24Z", "updated_at": "2018-05-14T22:47:55Z", "closed_at": "2018-05-14T22:47:55Z", "author_association": "OWNER", "pull_request": null, "body": "A lot of datasets have columns which have a small number of possible values in them - this one for example: https://fivethirtyeight.datasettes.com/fivethirtyeight-2628db9?sql=select+distinct+category+from+%5Binconvenient-sequel%2Fratings%5D%3B\r\n\r\nDetecting these could be interesting as part of `.inspect()`, since it would allow for various UI enhancements like autocomplete / select box filters for those columns.\r\n\r\nThe problem is detecting them efficiently. `.inspect()` shouldn't spend 5 minutes churning through columns on giant tables trying to determine if they have a small collection of unique values.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/251/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 335200136, "node_id": "MDU6SXNzdWUzMzUyMDAxMzY=", "number": 327, "title": "Explore if SquashFS can be used to shrink size of packaged Docker containers", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2018-06-24T18:15:16Z", "updated_at": "2022-02-17T23:37:24Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Inspired by this article: https://cldellow.com/2018/06/22/sqlite-parquet-vtable.html#sqlite-database-indexed--squashed\r\n\r\nhttps://en.wikipedia.org/wiki/SquashFS is \"a compressed read-only file system for Linux\" - which means it could be a really nice fit for Datasette and its read-only SQLite databases.\r\n\r\nIt would be interesting to explore a Dockerfile recipe that used SquashFS to compress the SQLite database file that was bundled up by `datasette package` and friends.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/327/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 340396247, "node_id": "MDU6SXNzdWUzNDAzOTYyNDc=", "number": 339, "title": "Expose SANIC_RESPONSE_TIMEOUT config option in a sensible way", "user": {"value": 12617395, "label": "bsilverm"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2018-07-11T20:38:06Z", "updated_at": "2022-03-21T22:22:40Z", "closed_at": "2022-03-21T22:22:34Z", "author_association": "NONE", "pull_request": null, "body": "Is it possible to configure the sql_time_limit_ms beyond 60 seconds? It seems queries are still timing out at 60 seconds when sql_time_limit_ms is set to 180000. We have a very large data set and often encounter timeouts when testing new queries from the datasette UI. We are optimizing our database as much as we can, but still may require more than 60 seconds for complex queries.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/339/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1452364777, "node_id": "I_kwDOBm6k_c5WkVPp", "number": 1896, "title": "Extract logic for resolving a URL to a database / table / row", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 8658075, "label": "Datasette 1.0a0"}, "comments": 4, "created_at": "2022-11-16T22:25:20Z", "updated_at": "2022-11-18T22:57:47Z", "closed_at": "2022-11-18T22:56:55Z", "author_association": "OWNER", "pull_request": null, "body": "> In trying to write this I realize that there's a lot of duplicated code with delete row, specifically around resolving the incoming URL into a row (or a database or a table).\r\n>\r\n> Since this is so common, I think it's worth extracting the logic out first.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1863#issuecomment-1317755263_\r\n ", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1896/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 449854604, "node_id": "MDU6SXNzdWU0NDk4NTQ2MDQ=", "number": 492, "title": "Facets not correctly persisted in hidden form fields", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 4, "created_at": "2019-05-29T14:49:39Z", "updated_at": "2020-09-15T20:12:29Z", "closed_at": "2020-09-15T20:12:29Z", "author_association": "OWNER", "pull_request": null, "body": "Steps to reproduce: visit https://2a4b892.datasette.io/fixtures/roadside_attractions?_facet_m2m=attraction_characteristic and click \"Apply\"\r\n\r\nResult is a 500: `no such column: attraction_characteristic`\r\n\r\nThe error occurs because of this hidden HTML input:\r\n\r\n \r\n\r\nThis should be:\r\n\r\n ", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/492/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 519038979, "node_id": "MDU6SXNzdWU1MTkwMzg5Nzk=", "number": 10, "title": "Failed to import workout points", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2019-11-07T04:50:22Z", "updated_at": "2019-11-08T01:18:37Z", "closed_at": "2019-11-08T01:18:37Z", "author_association": "MEMBER", "pull_request": null, "body": "I just ran the script and it failed to import any `workout_points`, though it did import `workouts`.", "repo": {"value": 197882382, "label": "healthkit-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/10/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 609950090, "node_id": "MDU6SXNzdWU2MDk5NTAwOTA=", "number": 33, "title": "Fall back to authentication via ENV", "user": {"value": 2029, "label": "garethr"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-04-30T12:58:14Z", "updated_at": "2020-05-02T18:46:10Z", "closed_at": "2020-05-02T18:45:37Z", "author_association": "NONE", "pull_request": null, "body": "Would you accept a PR that falls back to looking for an environment variable for the GitHub token? Specifically a change here:\r\nhttps://github.com/dogsheep/github-to-sqlite/blob/c34d5a18bfc41fa08755ba3d5cf9fe09ff204238/github_to_sqlite/cli.py#L271\r\n\r\nI'd like to use `github-to-sqlite` in a GitHub Action workflow and this would be simpler than trying to fill out the prompt or generate a file with sensitive content.\r\n\r\nWanted to check first, I'm happy to submit a PR with tests and updates to the docs. ", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 845794436, "node_id": "MDU6SXNzdWU4NDU3OTQ0MzY=", "number": 1284, "title": "Feature or Documentation Request: Individual table as home page template", "user": {"value": 192568, "label": "mroswell"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-03-31T03:56:17Z", "updated_at": "2021-11-04T03:15:01Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "It would be great to have a sample showing how to move a single database that has a single table, to the index page. I'm trying it now, and find there is a real depth of Datasette and Python understanding that's required to be successful. \r\n\r\nI've got all the basic jinja concepts down... variables, template control structures, template inheritance, template overrides, css, html, the --template-dir and --static arguments, etc. \r\n\r\nBut copying the table.html file to index.html doesn't work. There are undocumented functions and filters... I can figure some of them out (yay, url_builder.py and utils/__init__.py!) but it's a slog better handled by a much stronger Python developer. \r\n\r\nOne sample would make a world of difference. The ideal form of this documentation would be a diff between the default table.html and how that would look if essentially moved to index.html. The use case is for everyone who wants to create a public-facing website to explore a single table at the root directory. (Maybe a second bit of documentation for people who have a single database with multiple tables.)\r\n\r\n(Hmm... might be cool to have a setting for that, where it happens automagically! If only one table, then home page is at the table level. if only one database, then home page is at the database level.... as an option.)\r\n\r\nI suppose I could ignore this, and somehow do this in the DNS settings once I hook up Vercel to a domain name, maybe.. and remove the breadcrumbs in table.html... but for now, a documentation request in the form of a diff... for viewing a single table (or a single database) at the root.\r\n\r\n(Actually, there's probably room for a whole expanded section on templates. Noticed some nice table metadata in one of the datasette examples, for instance... Hmm... maybe a whole library of solutions in one place... maybe a documentation hackathon! If that's of interest, of course it's a separate issue. )\r\n\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1284/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1479914599, "node_id": "I_kwDOCGYnMM5YNbRn", "number": 516, "title": "Feature request: output number of ignored/replaced rows for insert command", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-12-06T18:59:21Z", "updated_at": "2022-12-06T19:08:14Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "https://hachyderm.io/@briandorsey/109468185742876820\r\n\r\n> I'm fiddling with piping json to `insert -ignore` I'd love to see the count of records inserted & ignored, but didn't see a way to do that in the help/docs.\r\n>\r\n> Example: `xh \"https://hachyderm.io/api/v1/timelines/tag/rust?max_id=109443380308326328\" | sqlite-utils insert aoc.db aoc - --pk=id --ignore`", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/516/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1374626873, "node_id": "I_kwDOBm6k_c5R7yQ5", "number": 1810, "title": "Featured table(s) on the homepage", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-09-15T14:30:49Z", "updated_at": "2022-09-15T15:51:25Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Many Datasette instances mainly exist to serve a single table - for example:\r\n\r\n- https://global-power-plants.datasettes.com/global-power-plants/global-power-plants\r\n- https://laion-aesthetic.datasette.io/laion-aesthetic-6pls/images\r\n\r\nIt would be neat if the / homepage of those instances could be configured to highlight that specific table.\r\n\r\nOr maybe more than one?", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1810/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 903978133, "node_id": "MDU6SXNzdWU5MDM5NzgxMzM=", "number": 1343, "title": "Figure out how to publish alpha/beta releases to Docker Hub", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-05-27T16:42:17Z", "updated_at": "2021-05-27T16:46:37Z", "closed_at": "2021-05-27T16:45:41Z", "author_association": "OWNER", "pull_request": null, "body": "> It looks like all I need to do to ship an alpha version to Docker Hub is NOT point the `latest` tag at it after it goes live: https://github.com/simonw/datasette/blob/1a8972f9c012cd22b088c6b70661a9c3d3847853/.github/workflows/publish.yml#L75-L77\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1319#issuecomment-849780481_", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1343/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 423316403, "node_id": "MDU6SXNzdWU0MjMzMTY0MDM=", "number": 422, "title": "Figure out what to do about table counts in a mutable world", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2019-03-20T15:27:15Z", "updated_at": "2019-05-02T05:43:11Z", "closed_at": "2019-05-02T05:43:11Z", "author_association": "OWNER", "pull_request": null, "body": "In moving away from the existing static inspect method (see #420 and #419) the biggest thing lost is full table row counts. These can be expensive against large tables, but currently Datasette runs the `count (*) from x` query once at inspection time and then reuses it for every page.\r\n\r\nWe can run those counts with a timelimit, but this means that for larger tables we won't be able to show a count at all, which is disappointing.\r\n\r\nIs there a way we can find an approximate or lower bound count for a table?", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/422/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 756876238, "node_id": "MDExOlB1bGxSZXF1ZXN0NTMyMzQ4OTE5", "number": 1130, "title": "Fix footer not sticking to bottom in short pages", "user": {"value": 3243482, "label": "abdusco"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-12-04T07:29:01Z", "updated_at": "2021-06-15T13:27:48Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": "simonw/datasette/pulls/1130", "body": "Fixes https://github.com/simonw/datasette/issues/1129", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1130/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 465728430, "node_id": "MDExOlB1bGxSZXF1ZXN0Mjk1NzExNTA0", "number": 554, "title": "Fix static mounts using relative paths and prevent traversal exploits", "user": {"value": 3243482, "label": "abdusco"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2019-07-09T11:32:02Z", "updated_at": "2019-07-11T16:29:26Z", "closed_at": "2019-07-11T16:13:19Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/datasette/pulls/554", "body": "While debugging why my static mounts using a relative path (`--static mystatic:rel/path/to/dir`) not working, I noticed that the requests fail no matter what, returning 404 errors. \r\n\r\nThe reason is that datasette tries to prevent traversal exploits by checking if the path is relative to its registered directory. This check fails when the mount is a relative directory, because `/abs/dir/file` obviously not under `dir/file`. \r\n\r\nhttps://github.com/simonw/datasette/blob/81fa8b6cdc5457b42a224779e5291952314e8d20/datasette/utils/asgi.py#L303-L306\r\n\r\nThis also has the consequence of returning any requested file, because when `/abs/dir/../../evil.file` resolves `aiofiles` happily returns it to the client after it resolves the path itself. The solution is to make sure we're checking relativity of paths after they're fully resolved.\r\n\r\nI've implemented the mentioned changes and also updated the tests.", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/554/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 466996584, "node_id": "MDExOlB1bGxSZXF1ZXN0Mjk2NzM1MzIw", "number": 557, "title": "Get tests running on Windows using Travis CI", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2019-07-11T16:36:57Z", "updated_at": "2021-07-10T23:39:48Z", "closed_at": "2021-07-10T23:39:48Z", "author_association": "OWNER", "pull_request": "simonw/datasette/pulls/557", "body": "Refs #511", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/557/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 268591332, "node_id": "MDU6SXNzdWUyNjg1OTEzMzI=", "number": 42, "title": "Homepage UI for editing metadata file", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2017-10-26T00:22:03Z", "updated_at": "2017-12-10T03:02:14Z", "closed_at": "2017-12-10T03:02:14Z", "author_association": "OWNER", "pull_request": null, "body": "Since we are going to have a metadata file which sets the title/description/etc for each database, why not allow you to run the app in \u2014dev mode which makes the homepage into a WYSIWYG editor that can save to that file format.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/42/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1590183272, "node_id": "I_kwDOBm6k_c5eyEVo", "number": 2027, "title": "How to redirect from \"/\" to a specific db/table", "user": {"value": 1350673, "label": "dmick"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2023-02-18T03:14:01Z", "updated_at": "2023-03-08T04:42:22Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Using nginx to redirect public IP to the local uvicorn server as 'normal'. I can't figure out how to redirect such that '/' results in accessing the one db/table I want to serve; redirecting / to /db/table breaks some of the CSS; fooling with base_url doesn't seem to help. Can someone explain this, if it's possible?", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2027/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 706001517, "node_id": "MDU6SXNzdWU3MDYwMDE1MTc=", "number": 163, "title": "Idea: conversions= could take Python functions", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-09-22T00:37:12Z", "updated_at": "2021-12-20T00:56:52Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Right now you use `conversions=` like this:\r\n\r\n```python\r\ndb[\"example\"].insert({\r\n \"name\": \"The Bigfoot Discovery Museum\"\r\n}, conversions={\"name\": \"upper(?)\"})\r\n```\r\nHow about if you could optionally provide a Python function (or a lambda) like this?\r\n```python\r\ndb[\"example\"].insert({\r\n \"name\": \"The Bigfoot Discovery Museum\"\r\n}, conversions={\"name\": lambda s: s.upper()})\r\n```\r\nThis would work by creating a random name for that function, registering it (similar to #162), executing the SQL and then un-registering the custom function at the end.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/163/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1822937426, "node_id": "I_kwDOBm6k_c5sp9FS", "number": 2111, "title": "Implement new /content.json?sql=...", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 9700784, "label": "Datasette 1.0a3"}, "comments": 4, "created_at": "2023-07-26T18:22:39Z", "updated_at": "2023-08-08T02:00:37Z", "closed_at": "2023-08-08T02:00:22Z", "author_association": "OWNER", "pull_request": null, "body": "This will be the base that the remaining work builds on top of. Refs:\r\n- #2109 ", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2111/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1059219106, "node_id": "I_kwDOBm6k_c4_Imai", "number": 1524, "title": "Improve Apache proxy documentation, link to demo", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-11-20T20:03:14Z", "updated_at": "2021-11-20T23:34:03Z", "closed_at": "2021-11-20T23:34:03Z", "author_association": "OWNER", "pull_request": null, "body": "> The latest demo is now live at https://datasette-apache-proxy-demo.fly.dev/prefix/fixtures/sortable?_facet=pk2\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1519#issuecomment-974697824_\r\n\r\nI'm going to put out 0.59.3 bugfix release with this, but I'd like to first improve the documentation on https://docs.datasette.io/en/stable/deploying.html#apache-proxy-configuration to highlight the new demo.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1524/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}