{"id": 988493790, "node_id": "MDExOlB1bGxSZXF1ZXN0NzI3MzkwODM1", "number": 36, "title": "Correct naming of tool in readme", "user": {"value": 2129, "label": "badboy"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-09-05T12:05:40Z", "updated_at": "2022-01-06T16:04:46Z", "closed_at": null, "author_association": "FIRST_TIME_CONTRIBUTOR", "pull_request": "dogsheep/dogsheep-photos/pulls/36", "body": null, "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/36/reactions\", \"total_count\": 2, \"+1\": 2, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1091850530, "node_id": "I_kwDODEm0Qs5BFFEi", "number": 63, "title": "Import archive error 'withheld_in_countries'", "user": {"value": 521097, "label": "pauloxnet"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-01-01T16:58:59Z", "updated_at": "2022-01-01T16:58:59Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Importing the twitter archive I received this error:\r\n```bash\r\n$ twitter-to-sqlite import archive.db twitter-2021-12-31-.zip \r\nbirdwatch-note-rating: not yet implemented\r\nbirdwatch-note: not yet implemented\r\nbranch-links: not yet implemented\r\ncommunity-tweet: not yet implemented\r\ncontact: not yet implemented\r\ndevice-token: not yet implemented\r\ndirect-message-mute: not yet implemented\r\nmute: not yet implemented\r\nperiscope-account-information: not yet implemented\r\nperiscope-ban-information: not yet implemented\r\nperiscope-broadcast-metadata: not yet implemented\r\nperiscope-comments-made-by-user: not yet implemented\r\nperiscope-expired-broadcasts: not yet implemented\r\nperiscope-followers: not yet implemented\r\nperiscope-profile-description: not yet implemented\r\nprofessional-data: not yet implemented\r\nprotected-history: not yet implemented\r\nreply-prompt: not yet implemented\r\nscreen-name-change: not yet implemented\r\nsmartblock: not yet implemented\r\nspaces-metadata: not yet implemented\r\nsso: not yet implemented\r\nTraceback (most recent call last):\r\n File \"/home/paulox/.virtualenvs/dogsheep/bin/twitter-to-sqlite\", line 8, in \r\n sys.exit(cli())\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py\", line 1128, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py\", line 1053, in main\r\n rv = self.invoke(ctx)\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py\", line 1659, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py\", line 1395, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py\", line 754, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/twitter_to_sqlite/cli.py\", line 759, in import_\r\n archive.import_from_file(db, filename, content)\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/twitter_to_sqlite/archive.py\", line 246, in import_from_file\r\n db[table_name].insert_all(rows, pk=pk, replace=True)\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/sqlite_utils/db.py\", line 2625, in insert_all\r\n self.insert_chunk(\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/sqlite_utils/db.py\", line 2406, in insert_chunk\r\n result = self.db.execute(query, params)\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/sqlite_utils/db.py\", line 422, in execute\r\n return self.conn.execute(sql, parameters)\r\nsqlite3.OperationalError: table archive_tweet has no column named withheld_in_countries\r\n```\r\n\r\nI found only a single tweet with the key `withheld_in_countries` in `tweet.js` that seems the problems:\r\n```JSON\r\n[\r\n{\r\n \"tweet\" : {\r\n \"retweeted\" : false,\r\n \"source\" : \"Twitter for Android\",\r\n \"entities\" : {\r\n \"hashtags\" : [\r\n {\r\n \"text\" : \"NowOnAndroid\",\r\n \"indices\" : [\r\n \"64\",\r\n \"77\"\r\n ]\r\n }\r\n ],\r\n \"symbols\" : [ ],\r\n \"user_mentions\" : [\r\n {\r\n \"name\" : \"Periscope\",\r\n \"screen_name\" : \"PeriscopeCo\",\r\n \"indices\" : [\r\n \"3\",\r\n \"15\"\r\n ],\r\n \"id_str\" : \"1111111111\",\r\n \"id\" : \"222222222\"\r\n }\r\n ],\r\n \"urls\" : [\r\n {\r\n \"url\" : \"https://t.co/xxxxxxxxx\",\r\n \"expanded_url\" : \"https://vine.co/v/xxxxxxxxx\",\r\n \"display_url\" : \"vine.co/v/xxxxxxxxxx\",\r\n \"indices\" : [\r\n \"78\",\r\n \"101\"\r\n ]\r\n }\r\n ]\r\n },\r\n \"display_text_range\" : [\r\n \"0\",\r\n \"101\"\r\n ],\r\n \"favorite_count\" : \"0\",\r\n \"id_str\" : \"1111111111111111111111\",\r\n \"truncated\" : false,\r\n \"retweet_count\" : \"0\",\r\n \"withheld_in_countries\" : [\r\n \"TR\"\r\n ],\r\n \"id\" : \"000000000000000000\",\r\n \"possibly_sensitive\" : false,\r\n \"created_at\" : \"Fri Aug 14 06:04:03 +0000 2015\",\r\n \"favorited\" : false,\r\n \"full_text\" : \"RT @periscopeco: Travel the world. LIVE. The Global Map is here #NowOnAndroid https://t.co/NZXdsPWROk\",\r\n \"lang\" : \"en\"\r\n }\r\n }\r\n ]\r\n```\r\n\r\nI solved the error removing the key from the `tweet.js` but I'm reporting this error to improve the project.", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/63/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1091838742, "node_id": "I_kwDOBm6k_c5BFCMW", "number": 1585, "title": "Fire base caching for `publish cloudrun`", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-01-01T15:38:15Z", "updated_at": "2022-01-01T15:40:38Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "https://gist.github.com/steren/03d3e58c58c9a53fd49bb78f58541872 has a recipe for this, via https://twitter.com/steren/status/1477038411114446848\r\n\r\nCould this enable easier vanity URLs of the format `https://$project_id.web.app/`? How about CDN caching?", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1585/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1091257796, "node_id": "I_kwDOBm6k_c5BC0XE", "number": 1584, "title": "give error with recursive sql", "user": {"value": 58088336, "label": "tunguyenatwork"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-12-30T18:53:16Z", "updated_at": "2021-12-30T18:53:16Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I got an error \"near \"WITH\": syntax error\" after I upgraded to version 0.59 from 0.52.4. This error is related to recursive sql. It works great on the previous version but it failed after upgraded. Below is an example of sql:\r\n\r\nWITH RECURSIVE manager_of(position, super_position) AS (SELECT position, case ifnull(INDIRECT_SUPER_POSITION,'') when '' then super_position else INDIRECT_SUPER_POSITION end as SUPER_POSITION FROM position where super_position<>'SGV000000001' and super_position!='' and position <> super_position),chain_manager_of_position(position, level) AS (SELECT super_position, 1 as level FROM manager_of WHERE super_position!='' and (position=:pos or position in (Select position from employee where employee=:ein)) UNION ALL SELECT super_position, level+1 as level FROM manager_of JOIN chain_manager_of_position USING(position)) SELECT * FROM chain_manager_of_position left join employee using(position) where employee is not NULL order by level limit 1", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1584/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1090810196, "node_id": "I_kwDOBm6k_c5BBHFU", "number": 1583, "title": "consider adding deletion step of cloudbuild artifacts to gcloud publish", "user": {"value": 536941, "label": "fgregg"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-12-30T00:33:23Z", "updated_at": "2021-12-30T00:34:16Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "right now, as part of the the publish process images and other artifacts are stored to gcloud's cloud storage before being deployed to cloudrun.\r\n\r\nafter successfully deploying, it would be nice if the the script deleted these artifacts. otherwise, if you have regularly scheduled build process, you can end up paying to store lots of out of date artifacts.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1583/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1090798237, "node_id": "I_kwDOCGYnMM5BBEKd", "number": 359, "title": "Use RETURNING if available to populate last_pk", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-12-29T23:43:23Z", "updated_at": "2021-12-29T23:43:23Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Inspired by this: https://news.ycombinator.com/item?id=29729283\r\n\r\n> Because SQLite is effectively serializing all the writes for us, we have zero locking in our code. We used to have to lock when inserting new items (to get the LastInsertRowId), but the newer version of SQLite supports the RETURNING keyword, so we don't even have to lock on inserts now.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/359/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 770598024, "node_id": "MDU6SXNzdWU3NzA1OTgwMjQ=", "number": 1152, "title": "Efficiently calculate list of databases/tables a user can view", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 12, "created_at": "2020-12-18T06:13:01Z", "updated_at": "2021-12-27T23:04:31Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "> The homepage currently performs a massive flurry of permission checks - one for each, database, table and view: https://github.com/simonw/datasette/blob/0.53/datasette/views/index.py#L21-L75\r\n> \r\n> A paginated version of this is a little daunting as the permission checks would have to be carried out in every single table just to calculate the count that will be paginated.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1150#issuecomment-747864831_", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1152/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1087919372, "node_id": "I_kwDOBm6k_c5A2FUM", "number": 1578, "title": "Confirm if documented nginx proxy config works for row pages with escaped characters in their primary key", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-12-23T18:27:59Z", "updated_at": "2021-12-24T21:33:19Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Found this while working on https://github.com/simonw/datasette-tiddlywiki\r\n\r\n\"image\"\r\n\r\nThen clicking on `/tiddlywiki/tiddlers/%24%3A%2FDefaultTiddlers` returns a 404.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1578/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 706001517, "node_id": "MDU6SXNzdWU3MDYwMDE1MTc=", "number": 163, "title": "Idea: conversions= could take Python functions", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-09-22T00:37:12Z", "updated_at": "2021-12-20T00:56:52Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Right now you use `conversions=` like this:\r\n\r\n```python\r\ndb[\"example\"].insert({\r\n \"name\": \"The Bigfoot Discovery Museum\"\r\n}, conversions={\"name\": \"upper(?)\"})\r\n```\r\nHow about if you could optionally provide a Python function (or a lambda) like this?\r\n```python\r\ndb[\"example\"].insert({\r\n \"name\": \"The Bigfoot Discovery Museum\"\r\n}, conversions={\"name\": lambda s: s.upper()})\r\n```\r\nThis would work by creating a random name for that function, registering it (similar to #162), executing the SQL and then un-registering the custom function at the end.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/163/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1083657868, "node_id": "I_kwDOBm6k_c5Al06M", "number": 1565, "title": "Documented JavaScript variables on different templates made available for plugins", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 8, "created_at": "2021-12-17T22:30:51Z", "updated_at": "2021-12-19T22:37:29Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "While working on https://github.com/simonw/datasette-leaflet-freedraw/issues/10 I found myself writing this atrocity to figure out the SQL query used for a specific table page:\r\n\r\n```javascript\r\nlet innerSql = Array.from(document.getElementsByTagName(\"span\")).filter(\r\n el => el.innerText == \"View and edit SQL\"\r\n)[0].parentElement.getAttribute(\"title\")\r\n```\r\nThis is obviously bad - it's very brittle, and will break if I ever change the text on that link (like localizing it for example).\r\n\r\nInstead, I think pages like that one should have a block of script at the bottom something like this:\r\n```javascript\r\nwindow.datasette = window.datasette || {};\r\ndatasette.view_name = 'table';\r\ndatasette.table_sql = 'select * from ...';\r\n```", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1565/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1084185188, "node_id": "I_kwDOBm6k_c5An1pk", "number": 1573, "title": "Make trace() a documented internal API", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-12-19T20:32:56Z", "updated_at": "2021-12-19T21:13:13Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "This should be documented so plugin authors can use it to add their own custom traces: https://github.com/simonw/datasette/blob/8f311d6c1d9f73f4ec643009767749c17b5ca5dd/datasette/tracer.py#L28-L52\r\n\r\nIncluding the new `kwargs` pattern I added in #1571: https://github.com/simonw/datasette/blob/f65817000fdf87ce8a0c23edc40784ebe33b5842/datasette/database.py#L128-L132", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1573/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 520681725, "node_id": "MDU6SXNzdWU1MjA2ODE3MjU=", "number": 621, "title": "Syntax for ?_through= that works as a form field", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2019-11-11T00:19:03Z", "updated_at": "2021-12-18T01:42:33Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "The current syntax for `?_through=` uses JSON to avoid any risk of confusion with table or column names that contain special characters.\r\n\r\nThis means you can't target a form field at it.\r\n\r\nWe should be able to support both - `?x.y.z=value` for tables and columns with \"regular\" names, falling back to the current JSON syntax for columns or tables that won't work with the key/value syntax.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/621/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 793002853, "node_id": "MDExOlB1bGxSZXF1ZXN0NTYwNzYwMTQ1", "number": 1204, "title": "WIP: Plugin includes", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-01-25T03:59:06Z", "updated_at": "2021-12-17T07:10:49Z", "closed_at": null, "author_association": "OWNER", "pull_request": "simonw/datasette/pulls/1204", "body": "Refs #1191\r\n\r\nNext steps:\r\n\r\n- [ ] Get comfortable that this pattern is the right way to go\r\n- [ ] Implement it for all of the other pages, not just the table page\r\n- [ ] Add a new set of plugin tests that exercise ALL of these new hook locations\r\n- [ ] Document, then ship", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1204/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 1, "state_reason": null} {"id": 636511683, "node_id": "MDU6SXNzdWU2MzY1MTE2ODM=", "number": 830, "title": "Redesign register_facet_classes plugin hook", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 3, "created_at": "2020-06-10T20:03:27Z", "updated_at": "2021-12-16T19:58:22Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Nothing uses this plugin hook yet, so the design is not yet proven.\r\n\r\nI'm going to build a real plugin against it and use that process to inform any design changes that may need to be made.\r\n\r\nI'll add a warning about this to the documentation.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/830/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1079111498, "node_id": "I_kwDOBm6k_c5AUe9K", "number": 1553, "title": "if csv export is truncated in non streaming mode set informative response header", "user": {"value": 536941, "label": "fgregg"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-12-13T22:50:44Z", "updated_at": "2021-12-16T19:17:28Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "streaming mode is currently not enabled for custom queries, so the queries will be truncated to max row limit.\r\n\r\nit would be great if a response is truncated that an header signalling that was set in the header.\r\n\r\ni need to write some pagination code for getting full results back for a custom query and it would make the code much better if i could reliably known when there is nothing more to limit/offset ", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1553/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 741231849, "node_id": "MDU6SXNzdWU3NDEyMzE4NDk=", "number": 1087, "title": "Idea: ?_extra=urls for getting back URLs to useful things", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-11-12T02:55:41Z", "updated_at": "2021-12-15T18:06:16Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Working on https://github.com/simonw/datasette-search-all/issues/10 made me realize that sometimes it can be difficult to calculate the URL for a database, table or row within Datasette.\r\n\r\nIt would be useful to have an optional extra JSON extension (using `?_extra=` from #262) that can help with this.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1087/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 705840673, "node_id": "MDU6SXNzdWU3MDU4NDA2NzM=", "number": 972, "title": "Support faceting against arbitrary SQL queries", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-09-21T19:00:43Z", "updated_at": "2021-12-15T18:02:20Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "> ... support for running facets against arbitrary custom SQL queries is half-done in that facets now execute against wrapped subqueries as-of ea66c45df96479ef66a89caa71fff1a97a862646\r\n> \r\n> https://github.com/simonw/datasette/blob/ea66c45df96479ef66a89caa71fff1a97a862646/datasette/facets.py#L192-L200\r\n_Originally posted by @simonw in https://github.com/simonw/datasette/issues/971#issuecomment-696307922_", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/972/reactions\", \"total_count\": 3, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 3, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1072106103, "node_id": "I_kwDOBm6k_c4_5wp3", "number": 1542, "title": "feature request: order and dependency of plugins (that use js)", "user": {"value": 33631, "label": "fs111"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-12-06T12:40:45Z", "updated_at": "2021-12-15T17:47:08Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I have been playing with datasette for the last couple of weeks and it is great! I am a big fan of `datasette-cluster-map` and wanted to enhance it a bit with a what I would call a sub-plugin. I basically want to add more controls to the map that cluster map provides. I have been looking into its code and how the plugin management works, but it seems what I am trying to do is not doable without hacks in js.\r\n\r\nBasically what would like to have is a way to say load my plugin after the plugins I depend on have been loaded and rendered. There seems to be no prior art where plugins have these dependencies on the js level so I was wondering if that could be added or if it exists how to do it.\r\n\r\nBasically what I want to do is:\r\n\r\nmy-awesome-plugin has a dependency on datastte-cluster-map. Whenever datasette cluster map has finished rendering on page load, call my plugin, but no earlier. To make that work datasette probably needs some total order in which way plugins are loaded intialized.\r\n\r\nSince I am new to datastte, I may be missing something obvious, so please let me know if the above makes no sense.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1542/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1077628073, "node_id": "I_kwDOBm6k_c5AO0yp", "number": 1550, "title": "Research option for returning all rows from arbitrary query", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-12-11T19:31:11Z", "updated_at": "2021-12-11T23:43:24Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Inspired by thinking about #1549 - returning ALL rows from an arbitrary query is a lot easier if you just run that query and keep iterating over the cursor.\r\n\r\nI've avoided doing that in the past because it could tie up a connection for a long time - but in private instances this wouldn't be such a problem.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1550/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1066603133, "node_id": "PR_kwDOCGYnMM4vKAzW", "number": 347, "title": "Test against pysqlite3 running SQLite 3.37", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2021-11-29T23:17:57Z", "updated_at": "2021-12-11T01:02:19Z", "closed_at": null, "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/347", "body": "Refs #346 and #344.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/347/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1068791148, "node_id": "I_kwDOBm6k_c4_tHVs", "number": 1540, "title": "Idea: hover to reveal details of linked row", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2021-12-01T19:28:07Z", "updated_at": "2021-12-09T23:38:39Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "\"fara__item_version__7_rows_where_where__item___5236\"\r\n\r\nHovering over that could work a little bit like GitHub issue links:\r\n\r\n![hover](https://user-images.githubusercontent.com/9599/144300537-9cd9e9af-ac16-42db-842f-37661bc94063.gif)\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1540/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 863884805, "node_id": "MDU6SXNzdWU4NjM4ODQ4MDU=", "number": 1304, "title": "Document how to send multiple values for \"Named parameters\" ", "user": {"value": 9308268, "label": "rayvoelker"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-04-21T13:19:06Z", "updated_at": "2021-12-08T03:23:14Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "https://docs.datasette.io/en/stable/sql_queries.html#named-parameters\r\n\r\nI thought that I had seen an example of how to do this example below, but I can't seem to find it\r\n\r\n```sql\r\nselect\r\n *\r\nfrom\r\n bib\r\nwhere\r\n bib.bib_record_num in (1008088,1008092)\r\n```\r\n\r\n```sql\r\nselect\r\n *\r\nfrom\r\n bib\r\nwhere\r\n bib.bib_record_num in (:bib_record_numbers)\r\n```\r\n![image](https://user-images.githubusercontent.com/9308268/115558839-2333a480-a281-11eb-85e6-ce3bada79140.png)\r\n\r\nhttps://ilsweb.cincinnatilibrary.org/collection-analysis/current_collection-204d100?sql=select%0D%0A++*%0D%0Afrom%0D%0A++bib%0D%0Awhere%0D%0A++bib.bib_record_num+in+%28%3Abib_record_numbers%29&bib_record_numbers=1008088%2C1008092\r\n\r\nOr, maybe this isn't a fully supported feature.\r\n\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1304/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1071531082, "node_id": "I_kwDOCGYnMM4_3kRK", "number": 349, "title": "A way of creating indexes on newly created tables", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-12-05T18:56:12Z", "updated_at": "2021-12-07T01:04:37Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "I'm writing code for https://github.com/simonw/git-history/issues/33 that creates a table inside a loop:\r\n\r\n```python\r\nitem_pk = db[item_table].lookup(\r\n {\"_item_id\": item_id},\r\n item_to_insert,\r\n column_order=(\"_id\", \"_item_id\"),\r\n pk=\"_id\",\r\n)\r\n```\r\nI need to look things up by `_item_id` on this table, which means I need an index on that column (the table can get very big).\r\n\r\nBut there's no mechanism in SQLite utils to detect if the table was created for the first time and add an index to it. And I don't want to run `CREATE INDEX IF NOT EXISTS` every time through the loop.\r\n\r\nThis should work like the `foreign_keys=` mechanism.\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/349/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1072435124, "node_id": "I_kwDOCGYnMM4_7A-0", "number": 350, "title": "Optional caching mechanism for table.lookup()", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-12-06T17:54:25Z", "updated_at": "2021-12-06T17:56:57Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Inspired by work on `git-history` where I used this pattern:\r\n```python\r\n column_name_to_id = {}\r\n\r\n def column_id(column):\r\n if column not in column_name_to_id:\r\n id = db[\"columns\"].lookup(\r\n {\"namespace\": namespace_id, \"name\": column},\r\n foreign_keys=((\"namespace\", \"namespaces\", \"id\"),),\r\n )\r\n column_name_to_id[column] = id\r\n return column_name_to_id[column]\r\n```\r\nIf you're going to be doing a large number of `table.lookup(...)` calls and you know that no other script will be modifying the database at the same time you can presumably get a big speedup using a Python in-memory cache - maybe even a LRU one to avoid memory bloat.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/350/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 964322136, "node_id": "MDU6SXNzdWU5NjQzMjIxMzY=", "number": 1426, "title": "Manage /robots.txt in Datasette core, block robots by default", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2021-08-09T19:56:56Z", "updated_at": "2021-12-04T07:11:29Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "See accompanying Twitter thread: https://twitter.com/simonw/status/1424820203603431439\r\n\r\n> Datasette currently has a plugin for configuring robots.txt, but I'm beginning to think it should be part of core and crawlers should be blocked by default - having people explicitly opt-in to having their sites crawled and indexed feels a lot safer https://datasette.io/plugins/datasette-block-robots\r\n\r\nI have a lot of Datasettes deployed now, and tailing logs shows that they are being *hammered* by search engine crawlers even though many of them are not interesting enough to warrant indexing.\r\n\r\nI'm starting to think blocking crawlers would actually be a better default for most people, provided it was well documented and easy to understand how to allow them.\r\n\r\nDefault-deny is usually a better policy than default-allow!", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1426/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1071071397, "node_id": "I_kwDODFdgUs4_10Cl", "number": 69, "title": "View that combines issues and issue comments", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-12-04T00:34:33Z", "updated_at": "2021-12-04T00:34:52Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "I want to see a reverse chronologically ordered interface onto both issues and comments - essentially a unified log of comments and issues opened across one or multiple projects.", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/69/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1069881276, "node_id": "I_kwDOBm6k_c4_xRe8", "number": 1541, "title": "Different default layout for row page", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-12-02T18:56:36Z", "updated_at": "2021-12-02T18:56:54Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "The row page displays as a table even though it only has one table row.\r\n\r\nmaybe default to the same display as the narrow page version, even for wide pages?", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1541/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1067775061, "node_id": "I_kwDOBm6k_c4_pPRV", "number": 1539, "title": "Research PRAGMA query_only", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-11-30T23:30:24Z", "updated_at": "2021-11-30T23:30:24Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "https://www.sqlite.org/pragma.html#pragma_query_only\r\n\r\n> The query_only pragma prevents data changes on database files when enabled. When this pragma is enabled, any attempt to CREATE, DELETE, DROP, INSERT, or UPDATE will result in an [SQLITE_READONLY](https://www.sqlite.org/rescode.html#readonly) error. However, the database is not truly read-only. You can still run a [checkpoint](https://www.sqlite.org/wal.html#ckpt) or a [COMMIT](https://www.sqlite.org/lang_transaction.html) and the return value of the [sqlite3_db_readonly()](https://www.sqlite.org/c3ref/db_readonly.html) routine is not affected.\r\n\r\nWould it be worth adding this as an extra protection against accidental writes to a DB file over a read-only connection?", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1539/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1066563554, "node_id": "I_kwDOCGYnMM4_knfi", "number": 346, "title": "Way to test SQLite 3.37 (and potentially other versions) in CI", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2021-11-29T22:21:06Z", "updated_at": "2021-11-29T23:12:49Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "> Need to figure out a good pattern for testing this in CI too - it will currently skip the new tests if it doesn't have SQLite 3.37 or higher.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982076924_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/346/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1059209412, "node_id": "I_kwDOBm6k_c4_IkDE", "number": 1523, "title": "Come up with a more elegant solution for base_url than ds.urls.path()", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-11-20T19:05:22Z", "updated_at": "2021-11-20T19:05:22Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "While fixing #1519 I added a lot of ugly code that looks like this: https://github.com/simonw/datasette/blob/08947fa76433d18988aa1ee1d929bd8320c75fe2/datasette/facets.py#L228-L230\r\n\r\nSee these two commits in particular: fe687fd0207c4c56c4778d3e92e3505fc4b18172 and 08947fa76433d18988aa1ee1d929bd8320c75fe2\r\n\r\nIt would be great to come up with a less verbose and error-prone way of handling this problem.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1523/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1058803238, "node_id": "I_kwDOBm6k_c4_HA4m", "number": 1520, "title": "Pattern for avoiding accidental URL over-rides", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-11-19T18:28:05Z", "updated_at": "2021-11-19T18:29:26Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Following #1517 I'm experimenting with a plugin that does this:\r\n```python\r\n@hookimpl\r\ndef register_routes():\r\n return [\r\n (r\"/(?P[^/]+)/(?P[^/]+?)$\", Table().view),\r\n ]\r\n```\r\nThis is supposed to replace the default table page with new code... but there's a problem: `/-/versions` on that instance now returns 404 `Database '-' does not exist`!\r\n\r\nNeed to figure out a pattern to avoid that happening. Plugins get to add their routes before Datasette's default routes, which is why this is happening here.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1520/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1033678984, "node_id": "PR_kwDOBm6k_c4tjgJ8", "number": 1495, "title": "Allow routes to have extra options", "user": {"value": 536941, "label": "fgregg"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2021-10-22T15:00:45Z", "updated_at": "2021-11-19T15:36:27Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": "simonw/datasette/pulls/1495", "body": "Right now, datasette routes can only be a 2-tuple of `(regex, view_fn)`. \r\n\r\nIf it was possible for datasette to handle extra options, like [standard Django does](https://docs.djangoproject.com/en/3.2/topics/http/urls/#passing-extra-options-to-view-functions), it would add flexibility for plugin authors.\r\n\r\nFor example, if extra options were enabled, then it would be easy to make a single table the home page (#1284). This plugin would accomplish it.\r\n\r\n```python\r\nfrom datasette import hookimpl\r\nfrom datasette.views.table import TableView\r\n\r\n@hookimpl\r\ndef register_routes(datasette):\r\n return [\r\n (r\"^/$\", TableView.as_view(datasette), {'db_name': 'DB_NAME',\r\n 'table': 'TABLE_NAME'})\r\n ]\r\n```\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1495/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1056746091, "node_id": "I_kwDOBm6k_c4-_Kpr", "number": 1515, "title": "Handle foreign keys that point to a non-existent table", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-11-17T23:40:13Z", "updated_at": "2021-11-18T01:31:56Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Spotted in https://github.com/simonw/datasette-graphql/issues/79\r\n\r\nDemo: https://datasette-graphql-demo.datasette.io/fixtures/bad_foreign_key\r\n\r\nThe foreign key links to a 404 page.\r\n\r\n![B87009C7-CFCA-4DF9-8FBA-FA3E6CA28EC2](https://user-images.githubusercontent.com/9599/142334788-4d1a4acd-bc87-4426-b333-d46b221afcec.jpeg)\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1515/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 812704869, "node_id": "MDU6SXNzdWU4MTI3MDQ4Njk=", "number": 1237, "title": "?_pretty=1 option for pretty-printing JSON output", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 1, "created_at": "2021-02-20T20:54:40Z", "updated_at": "2021-11-16T18:28:33Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Suggested by @frankieroberto in https://github.com/simonw/datasette/issues/782#issuecomment-782746755", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1237/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 718540751, "node_id": "MDU6SXNzdWU3MTg1NDA3NTE=", "number": 1012, "title": "For 1.0 update trove classifier in setup.py", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 5, "created_at": "2020-10-10T05:52:08Z", "updated_at": "2021-11-16T13:18:36Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": " Development Status :: 5 - Production/Stable", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1012/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1052247023, "node_id": "I_kwDOBm6k_c4-uAPv", "number": 1505, "title": "Datasette should have an option to output CSV with semicolons", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-11-12T18:02:21Z", "updated_at": "2021-11-16T11:40:52Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": null, "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1505/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1054246919, "node_id": "I_kwDOBm6k_c4-1ogH", "number": 1511, "title": "Review plugin hooks for Datasette 1.0", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 1, "created_at": "2021-11-15T23:26:05Z", "updated_at": "2021-11-16T01:20:14Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "I need to perform a detailed review of the plugin interface - especially the plugin hooks like [register_facet_classes()](https://docs.datasette.io/en/stable/plugin_hooks.html#register-facet-classes) which I don't yet have complete confidence in.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1511/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 924748955, "node_id": "MDU6SXNzdWU5MjQ3NDg5NTU=", "number": 1380, "title": "Serve all db files in a folder", "user": {"value": 193463, "label": "stratosgear"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2021-06-18T10:03:32Z", "updated_at": "2021-11-13T08:09:11Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I tried to get the `serve` command to serve all the .db files in the `/mnt` folder but is seems that the server does not refresh the list of files.\r\n\r\nIn more detail:\r\n\r\n* Starting datasette as a docker container with: \r\n```\r\ndocker run -p 8001:8001 -v `pwd`:/mnt \\\r\n datasetteproject/datasette \\\r\n datasette -p 8001 -h 0.0.0.0 /mnt\r\n```\r\n\r\n* Datasette correctly serves all the *.db files found in the /mnt folder \r\n* When the server is running, if I copy a new file in the $PWD folder, Datasette does not seem to see the new files, forcing me to restart Docker.\r\n\r\nIs there an option/setting that I overlooked, or is this something missing?\r\n\r\nBTW, the `--reload` setting, although at first glance is what you think you need, does not seem to do anything in regards of seeing all *.db files.\r\n\r\nThanks!", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1380/reactions\", \"total_count\": 2, \"+1\": 2, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1051277222, "node_id": "I_kwDOBm6k_c4-qTem", "number": 1504, "title": "Link to ?_size=max at bottom of table page", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-11-11T19:06:33Z", "updated_at": "2021-11-11T19:06:33Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "This can have text such as \"Show 1,000 rows per page\", based on the max size limit setting. Would make it easier for people to see more data at once without having to know how to hack the URL, similar to the `...` for facet sizes I added in #1337.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1504/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1049946823, "node_id": "I_kwDOBm6k_c4-lOrH", "number": 1502, "title": "Full-text search: No support to unary \"-\" operator", "user": {"value": 516827, "label": "gustavorps"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-11-10T15:11:19Z", "updated_at": "2021-11-10T15:11:19Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Reference: https://www.sqlite.org/fts3.html#set_operations_using_the_standard_query_syntax\r\n\r\nTest: https://fara.datasettes.com/fara/FARA_All_ShortForms?_search=manafort+-freedman&_sort=rowid", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1502/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1046887492, "node_id": "PR_kwDODFE5qs4uMsMJ", "number": 9, "title": "Removed space from filename My Activity.json", "user": {"value": 91880982, "label": "widadmogral"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-11-08T00:04:31Z", "updated_at": "2021-11-08T00:04:31Z", "closed_at": null, "author_association": "FIRST_TIME_CONTRIBUTOR", "pull_request": "dogsheep/google-takeout-to-sqlite/pulls/9", "body": "File name from google takeout has no space. The code only runs without error if filename is \"MyActivity.json\" and not \"My Activity.json\". Is it a new change by Google?", "repo": {"value": 206649770, "label": "google-takeout-to-sqlite"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/9/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 845794436, "node_id": "MDU6SXNzdWU4NDU3OTQ0MzY=", "number": 1284, "title": "Feature or Documentation Request: Individual table as home page template", "user": {"value": 192568, "label": "mroswell"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-03-31T03:56:17Z", "updated_at": "2021-11-04T03:15:01Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "It would be great to have a sample showing how to move a single database that has a single table, to the index page. I'm trying it now, and find there is a real depth of Datasette and Python understanding that's required to be successful. \r\n\r\nI've got all the basic jinja concepts down... variables, template control structures, template inheritance, template overrides, css, html, the --template-dir and --static arguments, etc. \r\n\r\nBut copying the table.html file to index.html doesn't work. There are undocumented functions and filters... I can figure some of them out (yay, url_builder.py and utils/__init__.py!) but it's a slog better handled by a much stronger Python developer. \r\n\r\nOne sample would make a world of difference. The ideal form of this documentation would be a diff between the default table.html and how that would look if essentially moved to index.html. The use case is for everyone who wants to create a public-facing website to explore a single table at the root directory. (Maybe a second bit of documentation for people who have a single database with multiple tables.)\r\n\r\n(Hmm... might be cool to have a setting for that, where it happens automagically! If only one table, then home page is at the table level. if only one database, then home page is at the database level.... as an option.)\r\n\r\nI suppose I could ignore this, and somehow do this in the DNS settings once I hook up Vercel to a domain name, maybe.. and remove the breadcrumbs in table.html... but for now, a documentation request in the form of a diff... for viewing a single table (or a single database) at the root.\r\n\r\n(Actually, there's probably room for a whole expanded section on templates. Noticed some nice table metadata in one of the datasette examples, for instance... Hmm... maybe a whole library of solutions in one place... maybe a documentation hackathon! If that's of interest, of course it's a separate issue. )\r\n\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1284/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1042759769, "node_id": "PR_kwDOEhK-wc4uAJb9", "number": 15, "title": "include note tags in the export", "user": {"value": 436138, "label": "d-rep"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-11-02T20:04:31Z", "updated_at": "2021-11-02T20:04:31Z", "closed_at": null, "author_association": "FIRST_TIME_CONTRIBUTOR", "pull_request": "dogsheep/evernote-to-sqlite/pulls/15", "body": "When parsing the Evernote `` elements, the script will now also parse any nested `` elements, writing them out into a separate sqlite table.\r\n\r\nHere is an example of how to query the data after the script has run:\r\n```\r\nselect notes.*,\r\n\t(select group_concat(tag) from notes_tags where notes_tags.note_id=notes.id) as tags\r\nfrom notes;\r\n```\r\n\r\nMy .enex source file is 3+ years old so I am assuming the structure hasn't changed. Interestingly, my _notebook names_ show up in the _tags_ list where the tag name is prefixed with `notebook_`, so this could maybe help work around the first limitation mentioned in the [evernote-to-sqlite blog post](https://simonwillison.net/2020/Oct/16/building-evernote-sqlite-exporter/).\r\n", "repo": {"value": 303218369, "label": "evernote-to-sqlite"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/15/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 826064552, "node_id": "MDU6SXNzdWU4MjYwNjQ1NTI=", "number": 1253, "title": "Capture \"Ctrl + Enter\" or \"\u2318 + Enter\" to send SQL query?", "user": {"value": 9308268, "label": "rayvoelker"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-03-09T15:00:50Z", "updated_at": "2021-10-30T16:00:42Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "It appears as though \"Shift + Enter\" triggers the form submit action to submit SQL, but could that action be bound to the \"Ctrl + Enter\" or \"\u2318 + Enter\" action?\r\n\r\nI feel like that pattern already exists in a number of similar tools and could improve usability of the editor.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1253/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 836829560, "node_id": "MDU6SXNzdWU4MzY4Mjk1NjA=", "number": 248, "title": "support for Apache Arrow / parquet files I/O", "user": {"value": 649467, "label": "mhalle"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-03-20T14:59:30Z", "updated_at": "2021-10-28T23:46:48Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I just started looking at Apache Arrow using pyarrow for import and export of tabular datasets, and it looks quite compelling. It might be worth looking at for sqlite-utils and/or datasette.\r\n\r\nAs a test, I took a random jsonl data dump of a dataset I have with floats, strings, and ints and converted it to arrow's parquet format using the naive `pyarrow.parquet.write_file()` command, which has automatic type inferrence. It compressed down to 7% of the original size. Conversion of a 26MB JSON file and serializing it to parquet was eyeblink instantaneous. Parquet files are portable and can be directly imported into pandas and other analytics software. \r\n\r\nThe only hangup is the automatic type inference of the naive reader. It's great for general laziness and for parsing JSON columns (it correctly interpreted a table of mine with a JSON array). However, I did get an exception for a string column where most entries looked integer-like but had a couple values that weren't -- the reader tried to coerce all of them for some reason, even though the JSON type is string. Since the writer optionally takes a schema, it shouldn't be too hard to grab the sqlite header types. With some additional hinting, you might get datetime columns and JSON, which are native Arrow types. \r\n\r\nSomewhat tangentially, someone even wrote an sqlite vfs extension for Parquet: https://cldellow.com/2018/06/22/sqlite-parquet-vtable.html\r\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/248/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 817989436, "node_id": "MDU6SXNzdWU4MTc5ODk0MzY=", "number": 242, "title": "Async support", "user": {"value": 25778, "label": "eyeseast"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 13, "created_at": "2021-02-27T18:29:38Z", "updated_at": "2021-10-28T14:37:56Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "Following our conversation last week, want to note this here before I forget.\r\n\r\nI've had a couple situations where I'd like to do a bunch of updates in an async event loop, but I run into SQLite's issues with concurrent writes. This feels like something sqlite-utils could help with.\r\n\r\nPeeWee ORM has a [SQLite write queue](http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#sqliteq) that might be a good model. It's using threads or gevent, but I _think_ that approach would translate well enough to asyncio. \r\n\r\nHappy to help with this, too.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/242/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 950664971, "node_id": "MDU6SXNzdWU5NTA2NjQ5NzE=", "number": 1401, "title": "unordered list is not rendering bullet points in description_html on database page", "user": {"value": 536941, "label": "fgregg"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-07-22T13:24:18Z", "updated_at": "2021-10-23T13:09:10Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "Thanks for this tremendous package, @simonw!\r\n\r\nIn the `description_html` for a database, I [have an unordered list](https://github.com/labordata/warehouse/blob/fcea4502e5b615b0eb3e0bdcb45ec634abe20bb6/warehouse_metadata.yml#L19-L22).\r\n\r\nHowever, on the database page on the deployed site, it is not rendering this as a bulleted list.\r\n\r\n![Screenshot 2021-07-22 at 09-21-51 nlrb](https://user-images.githubusercontent.com/536941/126645923-2777b7f1-fd4c-4d2d-af70-a35e49a07675.png)\r\n\r\nPage here: https://labordata-warehouse.herokuapp.com/nlrb-9da4ae5\r\n\r\nThe documentation gives an [example of using an unordered list](https://docs.datasette.io/en/stable/metadata.html#using-yaml-for-metadata) in a `description_html`, so I expected this will work.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1401/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 969855774, "node_id": "MDU6SXNzdWU5Njk4NTU3NzQ=", "number": 1432, "title": "Rename Datasette.__init__(config=) parameter to settings=", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 8, "created_at": "2021-08-13T01:00:27Z", "updated_at": "2021-10-19T01:16:41Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "> While I'm doing this I should rename this internal variable to avoid confusion in the future:\r\n>\r\n> https://github.com/simonw/datasette/blob/e837095ef35ae155b4c78cc9a8b7133a48c94f03/datasette/app.py#L203\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1431#issuecomment-898072940_", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1432/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 951817328, "node_id": "MDU6SXNzdWU5NTE4MTczMjg=", "number": 12, "title": "403 when getting token", "user": {"value": 285352, "label": "treyhunner"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-07-23T18:43:26Z", "updated_at": "2021-10-12T18:31:57Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I tried to use https://your-foursquare-oauth-token.glitch.me/ to get my Swarm auth token and got a 403 after I clicked the Allow button:\r\n\r\n![image](https://user-images.githubusercontent.com/285352/126826478-60e53614-263d-40bb-9f1d-c1a676644eb0.png)\r\n\r\nI'm not sure if this is the right repo to report this in", "repo": {"value": 205429375, "label": "swarm-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/12/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 602533481, "node_id": "MDU6SXNzdWU2MDI1MzM0ODE=", "number": 3, "title": "Import EXIF data into SQLite - lens used, ISO, aperture etc", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 5324096, "label": "Apple Photos online and securely browsable"}, "comments": 2, "created_at": "2020-04-18T19:24:31Z", "updated_at": "2021-10-05T12:38:24Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/3/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1013506559, "node_id": "PR_kwDODFdgUs4skaNS", "number": 68, "title": "Add support for retrieving teams / members", "user": {"value": 68329, "label": "philwills"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-10-01T15:55:02Z", "updated_at": "2021-10-01T15:59:53Z", "closed_at": null, "author_association": "FIRST_TIME_CONTRIBUTOR", "pull_request": "dogsheep/github-to-sqlite/pulls/68", "body": "Adds a method for retrieving all the teams within an organisation and all the members in those teams. The latter is stored as a join table `team_members` beteween `teams` and `users`.", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/68/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 975161924, "node_id": "MDExOlB1bGxSZXF1ZXN0NzE2MzU3OTgy", "number": 66, "title": "Add --merged-by flag to pull-requests sub command", "user": {"value": 30531572, "label": "sarcasticadmin"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-08-20T00:57:55Z", "updated_at": "2021-09-28T21:50:31Z", "closed_at": null, "author_association": "FIRST_TIME_CONTRIBUTOR", "pull_request": "dogsheep/github-to-sqlite/pulls/66", "body": "## Description\r\n\r\nProposing a solution to the API limitation for `merged_by` in pull_requests. Specifically the following called out in the readme:\r\n\r\n```\r\nNote that the merged_by column on the pull_requests table will only be populated for pull requests that are loaded using the --pull-request option - the GitHub API does not return this field for pull requests that are loaded in bulk.\r\n```\r\n\r\nThis approach might cause larger repos to hit rate limits called out in https://github.com/dogsheep/github-to-sqlite/issues/51 but seems to work well in the repos I tested and included below.\r\n\r\n## Old Behavior\r\n- Had to list out the pull-requests individually via multiple `--pull-request` flags\r\n\r\n## New Behavior\r\n\r\n- `--merged-by` flag for getting 'merge_by' information out of pull-requests without having to specify individual PR numbers.\r\n\r\n# Testing\r\n\r\nPicking some repo that has more than one merger (datasette only has 1 \ud83d\ude09 )\r\n\r\n```\r\n$ github-to-sqlite pull-requests ./github.db opnsense/tools --merged-by\r\n$ echo \"select id, url, merged_by from pull_requests;\" | sqlite3 ./github.db \r\n83533612|https://github.com/opnsense/tools/pull/39|1915288\r\n102632885|https://github.com/opnsense/tools/pull/43|1915288\r\n149114810|https://github.com/opnsense/tools/pull/57|1915288\r\n160394495|https://github.com/opnsense/tools/pull/64|1915288\r\n163308408|https://github.com/opnsense/tools/pull/67|1915288\r\n169723264|https://github.com/opnsense/tools/pull/69|1915288\r\n171381422|https://github.com/opnsense/tools/pull/72|1915288\r\n179938195|https://github.com/opnsense/tools/pull/77|1915288\r\n196233824|https://github.com/opnsense/tools/pull/82|1915288\r\n215289964|https://github.com/opnsense/tools/pull/93|\r\n219696100|https://github.com/opnsense/tools/pull/97|1915288\r\n223664843|https://github.com/opnsense/tools/pull/99|\r\n228446172|https://github.com/opnsense/tools/pull/103|1915288\r\n238930434|https://github.com/opnsense/tools/pull/110|1915288\r\n255507110|https://github.com/opnsense/tools/pull/119|1915288\r\n255980675|https://github.com/opnsense/tools/pull/120|1915288\r\n261906770|https://github.com/opnsense/tools/pull/125|\r\n263800503|https://github.com/opnsense/tools/pull/127|1915288\r\n264038685|https://github.com/opnsense/tools/pull/128|1915288\r\n264696704|https://github.com/opnsense/tools/pull/129|1915288\r\n266660547|https://github.com/opnsense/tools/pull/130|1915288\r\n273120409|https://github.com/opnsense/tools/pull/133|1915288\r\n274370803|https://github.com/opnsense/tools/pull/135|\r\n276600629|https://github.com/opnsense/tools/pull/139|\r\n277303655|https://github.com/opnsense/tools/pull/141|1915288\r\n293033714|https://github.com/opnsense/tools/pull/145|\r\n294827649|https://github.com/opnsense/tools/pull/146|\r\n295140008|https://github.com/opnsense/tools/pull/147|1915288\r\n305690829|https://github.com/opnsense/tools/pull/150|9783985\r\n307077931|https://github.com/opnsense/tools/pull/152|1915288\r\n321782100|https://github.com/opnsense/tools/pull/155|\r\n337265672|https://github.com/opnsense/tools/pull/160|\r\n337267484|https://github.com/opnsense/tools/pull/161|1915288\r\n368251763|https://github.com/opnsense/tools/pull/169|\r\n428262505|https://github.com/opnsense/tools/pull/181|\r\n437557011|https://github.com/opnsense/tools/pull/182|1915288\r\n447079893|https://github.com/opnsense/tools/pull/185|\r\n461822092|https://github.com/opnsense/tools/pull/191|\r\n463290142|https://github.com/opnsense/tools/pull/193|1915288\r\n470112962|https://github.com/opnsense/tools/pull/194|1915288\r\n472644649|https://github.com/opnsense/tools/pull/195|1915288\r\n488696898|https://github.com/opnsense/tools/pull/198|\r\n513289902|https://github.com/opnsense/tools/pull/201|\r\n522530265|https://github.com/opnsense/tools/pull/203|\r\n564443347|https://github.com/opnsense/tools/pull/213|\r\n597579516|https://github.com/opnsense/tools/pull/220|1915288\r\n602860357|https://github.com/opnsense/tools/pull/221|1915288\r\n608744738|https://github.com/opnsense/tools/pull/222|1915288\r\n623279673|https://github.com/opnsense/tools/pull/228|1915288\r\n664656182|https://github.com/opnsense/tools/pull/233|\r\n664781786|https://github.com/opnsense/tools/pull/234|1915288\r\n670683636|https://github.com/opnsense/tools/pull/235|1915288\r\n683150764|https://github.com/opnsense/tools/pull/237|\r\n685016233|https://github.com/opnsense/tools/pull/238|\r\n687099825|https://github.com/opnsense/tools/pull/239|1915288\r\n715705652|https://github.com/opnsense/tools/pull/244|1915288\r\n715721248|https://github.com/opnsense/tools/pull/245|1915288\r\n```\r\n`userid` are now present for those PRs that were merged.\r\n\r\nWithout the flag the `merged_by` behavior remains missing as expected when get PRs bulk:\r\n\r\n```\r\n$ github-to-sqlite pull-requests ./github.db opnsense/tools\r\n$ echo \"select id, url, merged_by from pull_requests;\" | sqlite3 ./github.db \r\n83533612|https://github.com/opnsense/tools/pull/39|\r\n102632885|https://github.com/opnsense/tools/pull/43|\r\n149114810|https://github.com/opnsense/tools/pull/57|\r\n160394495|https://github.com/opnsense/tools/pull/64|\r\n163308408|https://github.com/opnsense/tools/pull/67|\r\n169723264|https://github.com/opnsense/tools/pull/69|\r\n171381422|https://github.com/opnsense/tools/pull/72|\r\n179938195|https://github.com/opnsense/tools/pull/77|\r\n196233824|https://github.com/opnsense/tools/pull/82|\r\n215289964|https://github.com/opnsense/tools/pull/93|\r\n219696100|https://github.com/opnsense/tools/pull/97|\r\n223664843|https://github.com/opnsense/tools/pull/99|\r\n228446172|https://github.com/opnsense/tools/pull/103|\r\n238930434|https://github.com/opnsense/tools/pull/110|\r\n255507110|https://github.com/opnsense/tools/pull/119|\r\n255980675|https://github.com/opnsense/tools/pull/120|\r\n261906770|https://github.com/opnsense/tools/pull/125|\r\n263800503|https://github.com/opnsense/tools/pull/127|\r\n264038685|https://github.com/opnsense/tools/pull/128|\r\n264696704|https://github.com/opnsense/tools/pull/129|\r\n266660547|https://github.com/opnsense/tools/pull/130|\r\n273120409|https://github.com/opnsense/tools/pull/133|\r\n274370803|https://github.com/opnsense/tools/pull/135|\r\n276600629|https://github.com/opnsense/tools/pull/139|\r\n277303655|https://github.com/opnsense/tools/pull/141|\r\n293033714|https://github.com/opnsense/tools/pull/145|\r\n294827649|https://github.com/opnsense/tools/pull/146|\r\n295140008|https://github.com/opnsense/tools/pull/147|\r\n305690829|https://github.com/opnsense/tools/pull/150|\r\n307077931|https://github.com/opnsense/tools/pull/152|\r\n321782100|https://github.com/opnsense/tools/pull/155|\r\n337265672|https://github.com/opnsense/tools/pull/160|\r\n337267484|https://github.com/opnsense/tools/pull/161|\r\n368251763|https://github.com/opnsense/tools/pull/169|\r\n428262505|https://github.com/opnsense/tools/pull/181|\r\n437557011|https://github.com/opnsense/tools/pull/182|\r\n447079893|https://github.com/opnsense/tools/pull/185|\r\n461822092|https://github.com/opnsense/tools/pull/191|\r\n463290142|https://github.com/opnsense/tools/pull/193|\r\n470112962|https://github.com/opnsense/tools/pull/194|\r\n472644649|https://github.com/opnsense/tools/pull/195|\r\n488696898|https://github.com/opnsense/tools/pull/198|\r\n513289902|https://github.com/opnsense/tools/pull/201|\r\n522530265|https://github.com/opnsense/tools/pull/203|\r\n564443347|https://github.com/opnsense/tools/pull/213|\r\n597579516|https://github.com/opnsense/tools/pull/220|\r\n602860357|https://github.com/opnsense/tools/pull/221|\r\n608744738|https://github.com/opnsense/tools/pull/222|\r\n623279673|https://github.com/opnsense/tools/pull/228|\r\n664656182|https://github.com/opnsense/tools/pull/233|\r\n664781786|https://github.com/opnsense/tools/pull/234|\r\n670683636|https://github.com/opnsense/tools/pull/235|\r\n683150764|https://github.com/opnsense/tools/pull/237|\r\n685016233|https://github.com/opnsense/tools/pull/238|\r\n687099825|https://github.com/opnsense/tools/pull/239|\r\n715705652|https://github.com/opnsense/tools/pull/244|\r\n715721248|https://github.com/opnsense/tools/pull/245|\r\n```\r\n\r\nIndividual PRs passed via `--pull-request` flag behaves as expected (unchanged):\r\n\r\n```\r\n$ github-to-sqlite pull-requests ./github.db opnsense/tools --pull-request 39 --pull-request 237\r\n$ echo \"select id, url, merged_by from pull_requests;\" | sqlite3 ./github.db\r\n83533612|https://github.com/opnsense/tools/pull/39|1915288\r\n683150764|https://github.com/opnsense/tools/pull/237|\r\n```\r\n> Picking 1 PR that has a merged_by (39) and one that does not (237)", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/66/reactions\", \"total_count\": 3, \"+1\": 2, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 1, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1006781949, "node_id": "I_kwDOBm6k_c48AkX9", "number": 1478, "title": "Documentation Request: Feature alternative ID instead of default ID", "user": {"value": 192568, "label": "mroswell"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-09-24T19:56:13Z", "updated_at": "2021-09-25T16:18:54Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "My data already has an ID that comes from a federal agency.\r\nWould love to have documentation on how to modify the template to:\r\n- Remove the generated ID from the table\r\n- Link the federal ID to the detail page\r\n- and to ensure that the JSON file uses that as the ID. I'd be happy to include the database ID in the export, but not as a key.\r\n\r\nI don't want to remove the ID from the database, though, because my experience with the federal agency is that data often has anomalies. I don't want all hell to break loose if they end up applying the same ID to multiple rows (which they haven't done yet). I just don't want it to display in the table or the data exports.\r\n\r\nPerhaps this isn't a template issue, maybe more of a db manipulation...\r\n\r\nMargie", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1478/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1006016302, "node_id": "I_kwDOBm6k_c479pcu", "number": 1477, "title": "Consider adding request to the documented default template context", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-09-24T02:34:09Z", "updated_at": "2021-09-24T02:34:09Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "I made a plugin for this today but I think perhaps it should be a default thing instead: https://datasette.io/plugins/datasette-template-request", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1477/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 274615452, "node_id": "MDU6SXNzdWUyNzQ2MTU0NTI=", "number": 111, "title": "Add \u201cupdated\u201d to metadata", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 12, "created_at": "2017-11-16T18:22:20Z", "updated_at": "2021-09-21T22:48:27Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "To give an indication as to when the data was last updated.\r\n\r\nThis should be a field in the metadata that is then shown on the index page and in the footer, if it is set.\r\n\r\nAlso support setting it using an option to \u201cdatasette publish\u201d and \u201cdatasette package\u201d - which can either be a string or can be the magic string \u201ctoday\u201d to set it to today\u2019s date:\r\n\r\n datasette publish file.db --updated=today", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/111/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1001104942, "node_id": "PR_kwDOBm6k_c4r-EVH", "number": 1475, "title": "feat: allow joins using _through in both directions", "user": {"value": 5268174, "label": "bram2000"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-09-20T15:28:20Z", "updated_at": "2021-09-20T15:28:20Z", "closed_at": null, "author_association": "FIRST_TIME_CONTRIBUTOR", "pull_request": "simonw/datasette/pulls/1475", "body": "Currently the `_through` clause can only work if the FK relationship is defined in a specific direction. I don't think there is any reason for this limitation, as an FK allows joining in both directions.\r\n\r\nThis is an admittedly hacky change to implement bidirectional joins using `_through`. It does work for our use-case, but I don't know if there are other implications that I haven't thought of. Also if this change is desirable we probably want to make the code a little nicer.", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1475/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 999902754, "node_id": "I_kwDOBm6k_c47mU4i", "number": 1473, "title": "base logo link visits `undefined` rather than href url", "user": {"value": 192568, "label": "mroswell"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-09-18T04:17:04Z", "updated_at": "2021-09-19T00:45:32Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "I have two connected sites:\r\nhttp://www.SaferOrToxic.org \r\n(a Hugo website)\r\nand:\r\nhttp://disinfectants.SaferOrToxic.org/disinfectants/listN\r\n(a datasette table page)\r\n\r\nThe latter is linked as \"The List\" in the former's menu.\r\n(I'd love a prettier URL, but that's what I've got.)\r\n\r\nOn:\r\nhttp://disinfectants.SaferOrToxic.org/disinfectants/listN\r\n... all the other menu links should point back to:\r\nhttps://www.SaferOrToxic.org \r\nAnd they do!\r\n\r\nBut the logo, for some reason--though it has an href pointing to:\r\nhttps://www.SaferOrToxic.org\r\nKeeps going to this instead:\r\nhttps://disinfectants.saferortoxic.org/disinfectants/undefined\r\n\r\nWhat is causing that? How can I fix it?\r\n\r\nIn #1284 back in March, I was doing battle with the index.html template, in a still unresolved issue. (I wanted only a single table page at the root.)\r\n\r\nBut I thought, well, if I can't resolve that, at least I could just point the main website to the datasette page (\"The List,\") and then have the List point back to the home website.\r\n\r\nThe menu hrefs to https://www.SaferOrToxic.org work just fine, exactly as they should, from the datasette page. Even the Home link works properly.\r\n\r\nBut the logo link keeps rewriting to: https://disinfectants.saferortoxic.org/disinfectants/undefined\r\n\r\nThis is the HTML:\r\n```\r\n\"Logo:\r\n```\r\n\r\n\r\nIs this somehow related to cloudflare?\r\nOr something in the datasette code?\r\n\r\nI'm starting to think it's a cloudflare issue.\r\n\"Screen\r\n\r\nCan I at least rule out it being a datasette issue?\r\n\r\nMy repository is here:\r\nhttps://github.com/mroswell/list-N\r\n\r\n(BTW, I couldn't figure out how to reference a local image, either, on the datasette side, which is why I'm using the image from the www home page.)\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1473/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 991191951, "node_id": "MDU6SXNzdWU5OTExOTE5NTE=", "number": 1464, "title": "clean checkout & clean environment has test failures", "user": {"value": 51016, "label": "ctb"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2021-09-08T14:16:23Z", "updated_at": "2021-09-13T22:17:17Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "I followed the instructions [here](https://docs.datasette.io/en/stable/contributing.html#setting-up-a-development-environment), and even after running `python update-docs-help.py` I get the following failed tests -- any thoughts?\r\n\r\n```\r\nFAILED tests/test_api.py::test_searchable[/fixtures/searchable.json?_search=te*+AND+do*&_searchmode=raw-expected_rows3]\r\nFAILED tests/test_api.py::test_searchmode[table_metadata1-_search=te*+AND+do*-expected_rows1]\r\nFAILED tests/test_api.py::test_searchmode[table_metadata2-_search=te*+AND+do*&_searchmode=raw-expected_rows2]\r\n```\r\n\r\nThis is with python 3.9.7 and lots of other packages, as in attached environment listing from `conda list`.\r\n[conda-installed.txt](https://github.com/simonw/datasette/files/7129487/conda-installed.txt)\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1464/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 991206402, "node_id": "MDExOlB1bGxSZXF1ZXN0NzI5NzA0NTM3", "number": 1465, "title": "add support for -o --get /path", "user": {"value": 51016, "label": "ctb"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-09-08T14:30:42Z", "updated_at": "2021-09-08T14:31:45Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": "simonw/datasette/pulls/1465", "body": "Fixes https://github.com/simonw/datasette/issues/1459\r\n\r\nAdds support for `--open --get /path` to be used in combination.\r\n\r\nIf `--open` is provided alone, datasette will open a web page to a default URL.\r\nIf `--get ` is provided alone, datasette will output the result of doing a GET to that URL and then exit.\r\nIf `--open --get ` are provided together, datasette will open a web page to that URL.\r\n\r\nTODO items:\r\n- [ ] update documentation\r\n- [ ] print out error message when `--root --open --get ` is used\r\n- [ ] adjust code to require that `` start with a `/` when `-o --get ` is used\r\n- [ ] add test(s)\r\n\r\nnote, '@CTB' is used in this PR to flag code that needs revisiting.", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1465/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 1, "state_reason": null} {"id": 990367646, "node_id": "MDU6SXNzdWU5OTAzNjc2NDY=", "number": 1462, "title": "Separate out \"debug\" options from \"root\" options", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-09-07T21:27:34Z", "updated_at": "2021-09-07T21:34:33Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "> I ditched \"root\" for \"admin\" because root by default gives you a whole bunch of stuff which I think could be confusing:\r\n> \r\n> \"Datasette___internal__temporary_and_datasette-app_\u2014_Electron_Helper__Renderer__\u25c2_npm_start_TMPDIR__var_folders_wr_hn3206rs1yzgq3r49bz8nvnh0000gn_T__XPC_FLAGS_0x0_\u2014_180\u00d758_and_datasette-app-support_\u2014_pipenv_shell_\u25b8_zsh_\u2014_147\u00d776\"\r\n>\r\n> Maybe the real problem here is that I'm conflating \"root\" permissions with \"debug\" options. Perhaps there should be an extra Datasette mode that unlocks debug tools for the root user?\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/datasette-app-support/issues/8#issuecomment-914638998_", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1462/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 989109888, "node_id": "MDU6SXNzdWU5ODkxMDk4ODg=", "number": 1460, "title": "Override column metadata with metadata from another column", "user": {"value": 72577720, "label": "MichaelTiemannOSC"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-09-06T12:13:33Z", "updated_at": "2021-09-06T12:13:33Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "I have a table from the PUDL project (https://github.com/catalyst-cooperative/pudl) that looks like this:\r\n\r\n```\r\nCREATE TABLE fuel_ferc1 (\r\n\tid INTEGER NOT NULL, \r\n\trecord_id TEXT, \r\n\tutility_id_ferc1 INTEGER, \r\n\treport_year INTEGER, \r\n\tplant_name_ferc1 TEXT, \r\n\tfuel_type_code_pudl VARCHAR(7), \r\n\tfuel_unit VARCHAR(7),\r\n\tfuel_qty_burned FLOAT,\r\n\tfuel_mmbtu_per_unit FLOAT, \r\n\tfuel_cost_per_unit_burned FLOAT, \r\n\tfuel_cost_per_unit_delivered FLOAT, \r\n\tfuel_cost_per_mmbtu FLOAT, \r\n\tPRIMARY KEY (id), \r\n\tFOREIGN KEY(plant_name_ferc1, utility_id_ferc1) REFERENCES plants_ferc1 (plant_name_ferc1, utility_id_ferc1), \r\n\tCONSTRAINT fuel_ferc1_fuel_type_code_pudl_enum CHECK (fuel_type_code_pudl IN ('coal', 'oil', 'gas', 'solar', 'wind', 'hydro', 'nuclear', 'waste', 'unknown')), \r\n\tCONSTRAINT fuel_ferc1_fuel_unit_enum CHECK (fuel_unit IN ('ton', 'mcf', 'bbl', 'gal', 'kgal', 'gramsU', 'kgU', 'klbs', 'btu', 'mmbtu', 'mwdth', 'mwhth', 'unknown'))\r\n);\r\n```\r\n\r\nNote that `fuel_unit` is a unit that **pint** can understand, and that `fuel_qty_burned` is a column of data that could be expressed in terms of actual units, not merely as a dimensionless number. Ditto the `fuel_cost_per_unit_...` columns. Is there a way to give a column a default metadata unit (such as *tons* or *USD/ton*) and then let that be overridden when the metadata in another column says *barrels* or *USD/gramsU*?\r\n\r\n@catalyst-cooperative", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1460/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 988556488, "node_id": "MDU6SXNzdWU5ODg1NTY0ODg=", "number": 1459, "title": "suggestion: allow `datasette --open` to take a relative URL", "user": {"value": 51016, "label": "ctb"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-09-05T17:17:07Z", "updated_at": "2021-09-05T19:59:15Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "(soft suggestion because I'm not sure I'm using datasette right yet)\r\n\r\nOver at https://github.com/ctb/2021-sourmash-datasette, I'm playing around with datasette, and I'm creating some static pages to send people to the right facets. There may well be better ways of achieving this end goal, and I will find out if so, I'm sure!\r\n\r\nBut regardless I think it might be neat to support an option to allow `-o/--open` to take a relative URL, that then gets appended to the hostname and port. This would let me improve my documentation. I don't see any downsides, either, but \ud83e\udd37 there may well be some :)\r\n\r\nHappy to dig in and provide a PR if it's of interest. I'm not sure off the top of my head how to support an optional value to a parameter in argparse - the current `-o` behavior is kinda nice so it'd be suboptimal to require a url for `-o`. Maybe `--open-url=` or something would work?\r\n\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1459/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 988552851, "node_id": "MDU6SXNzdWU5ODg1NTI4NTE=", "number": 1456, "title": "conda install results in non-functioning `datasette serve` due to out-of-date asgiref", "user": {"value": 51016, "label": "ctb"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-09-05T16:59:55Z", "updated_at": "2021-09-05T16:59:55Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "Over in https://github.com/ctb/2021-sourmash-datasette, I discovered that the following commands fail:\r\n```\r\nconda create -n datasette4 -y datasette=0.58.1\r\nconda activate datasette4\r\ndatasette gathertax.db\r\n```\r\nwith `ImportError: cannot import name 'WebSocketScope' from 'asgiref.typing'`.\r\n\r\nThis appears to be because asgiref 3.3.4 doesn't have WebSocketScope, but later versions do - a simple\r\n\r\n```\r\npip install asgiref==3.4.1\r\n```\r\n\r\nfixes the problem for me, at least to the point where I can run datasette and poke around as usual.\r\n\r\nI note that over in the conda-forge recipe, https://github.com/conda-forge/datasette-feedstock/blob/master/recipe/meta.yaml pins asgiref to < 3.4.0, but I'm not sure why - so I'm not sure how to best resolve this issue :).", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1456/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 987985935, "node_id": "MDExOlB1bGxSZXF1ZXN0NzI2OTkwNjgw", "number": 35, "title": "Support for Datasette's --base-url setting", "user": {"value": 2670795, "label": "brandonrobertz"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-09-03T17:47:45Z", "updated_at": "2021-09-03T17:47:45Z", "closed_at": null, "author_association": "FIRST_TIME_CONTRIBUTOR", "pull_request": "dogsheep/dogsheep-beta/pulls/35", "body": "This makes it so you can use Dogsheep if you're using Datasette with the `--base-url /some-path/` setting.", "repo": {"value": 197431109, "label": "dogsheep-beta"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-beta/issues/35/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 986829194, "node_id": "MDU6SXNzdWU5ODY4MjkxOTQ=", "number": 14, "title": "xml.etree.ElementTree.Parse Error - mismatched tag", "user": {"value": 46968, "label": "step21"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-09-02T14:46:36Z", "updated_at": "2021-09-02T14:53:11Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "This is an error message I get upon parsing the enex file of my Inbox. Please find the full error message below. Any hints welcome.\r\n\r\n```\r\nImporting from ENEX [##################------------------] 50% 00:00:50\r\nTraceback (most recent call last):\r\n File \"/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/bin/evernote-to-sqlite\", line 8, in \r\n sys.exit(cli())\r\n File \"/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/click/core.py\", line 1137, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/click/core.py\", line 1062, in main\r\n rv = self.invoke(ctx)\r\n File \"/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/click/core.py\", line 1668, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/click/core.py\", line 1404, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/click/core.py\", line 763, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/evernote_to_sqlite/cli.py\", line 30, in enex\r\n for tag, note in find_all_tags(fp, [\"note\"], progress_callback=bar.update):\r\n File \"/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/evernote_to_sqlite/utils.py\", line 17, in find_all_tags\r\n for event, el in parser.read_events():\r\n File \"/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/xml/etree/ElementTree.py\", line 1329, in read_events\r\n raise event\r\n File \"/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/xml/etree/ElementTree.py\", line 1301, in feed\r\n self._parser.feed(data)\r\nxml.etree.ElementTree.ParseError: mismatched tag: line 6837961, column 2\r\n```\r\n", "repo": {"value": 303218369, "label": "evernote-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/14/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 983221851, "node_id": "MDU6SXNzdWU5ODMyMjE4NTE=", "number": 34, "title": "Data folder as index command parameter", "user": {"value": 1223625, "label": "humrochagf"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-08-30T21:29:33Z", "updated_at": "2021-08-30T21:29:33Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Hi,\r\n\r\nFirst of all, thank you for this wonderful project :smile:\r\n\r\nI started to use dogsheep to make my personal data searchable, and by using the project I noticed an issue with the index command.\r\n\r\nIt always expects you are running it from the root folder from where the data is located, so I got some errors while trying to make it work on my setup.\r\n\r\nI separate all databases inside a `data` folder (I published my setup to be easier to follow: https://github.com/humrochagf/my-dogsheep)\r\n\r\nBefore, I configured `dogsheep.yml` to add the data folder to its path like this:\r\n\r\n```yml\r\ndata/twitter.db:\r\n tweets:\r\n sql: |-\r\n...\r\n```\r\n\r\nAnd running the index command like this:\r\n\r\n```\r\ndogsheep-beta index data/dogsheep.db dogsheep.yml\r\n```\r\n\r\nIt worked to the normal search feature with no problem this way, but when I started adding `display_sql` rules the app started to crash, because at datasette `get_database` it was looking for `data/twitter` and it only had a db called `twitter` there.\r\n\r\nSo my workaround to that was to cd into the data folder and run the indexer. You can check the way I'm doing it at this line of the makefile: https://github.com/humrochagf/my-dogsheep/blob/main/makefile#L3\r\n\r\nIt works but it would be nice to have an option to pass the path where the data is located to the index function.", "repo": {"value": 197431109, "label": "dogsheep-beta"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-beta/issues/34/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 982803408, "node_id": "MDU6SXNzdWU5ODI4MDM0MDg=", "number": 1454, "title": "Feature Request: Publish to IPFS", "user": {"value": 1560788, "label": "blitmap"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-08-30T13:36:18Z", "updated_at": "2021-08-30T13:36:18Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Hello,\r\n\r\nI am a huge fan of this being used for exploring data. I think it has a lot of flexibility not found in other tools.\r\n\r\nI'm not sure if what I'm asking for is possible: Can this be extended to publish to IPFS?\r\n\r\nIPFS is an attractive hosting option for decentralized journalism.\r\n\r\nFood for thought ~", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1454/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 981690086, "node_id": "MDExOlB1bGxSZXF1ZXN0NzIxNjg2NzIx", "number": 67, "title": "Replacing step ID key with step_id", "user": {"value": 16374374, "label": "jshcmpbll"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-08-28T01:26:41Z", "updated_at": "2021-08-28T01:27:00Z", "closed_at": null, "author_association": "FIRST_TIME_CONTRIBUTOR", "pull_request": "dogsheep/github-to-sqlite/pulls/67", "body": "Workflows that have an `id` in any step result in the following error when running `workflows`:\r\n\r\ne.g.`github-to-sqlite workflows github.db nixos/nixpkgs`\r\n\r\n```Traceback (most recent call last):\r\n File \"/usr/local/bin/github-to-sqlite\", line 8, in \r\n sys.exit(cli())\r\n File \"/usr/local/lib/python3.8/dist-packages/click/core.py\", line 1137, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/usr/local/lib/python3.8/dist-packages/click/core.py\", line 1062, in main\r\n rv = self.invoke(ctx)\r\n File \"/usr/local/lib/python3.8/dist-packages/click/core.py\", line 1668, in invoke```Traceback (most recent call last):\r\n File \"/usr/local/bin/github-to-sqlite\", line 8, in \r\n sys.exit(cli())\r\n File \"/usr/local/lib/python3.8/dist-packages/click/core.py\", line 1137, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/usr/local/lib/python3.8/dist-packages/click/core.py\", line 1062, in main\r\n rv = self.invoke(ctx)\r\n File \"/usr/local/lib/python3.8/dist-packages/click/core.py\", line 1668, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/usr/local/lib/python3.8/dist-packages/click/core.py\", line 1404, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/usr/local/lib/python3.8/dist-packages/click/core.py\", line 763, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/usr/local/lib/python3.8/dist-packages/github_to_sqlite/cli.py\", line 601, in workflows\r\n utils.save_workflow(db, repo_id, filename, content)\r\n File \"/usr/local/lib/python3.8/dist-packages/github_to_sqlite/utils.py\", line 865, in save_workflow\r\n db[\"steps\"].insert_all(\r\n File \"/usr/local/lib/python3.8/dist-packages/sqlite_utils/db.py\", line 2596, in insert_all\r\n self.insert_chunk(\r\n File \"/usr/local/lib/python3.8/dist-packages/sqlite_utils/db.py\", line 2378, in insert_chunk\r\n result = self.db.execute(query, params)\r\n File \"/usr/local/lib/python3.8/dist-packages/sqlite_utils/db.py\", line 419, in execute\r\n return self.conn.execute(sql, parameters)\r\nsqlite3.IntegrityError: datatype mismatch\r\n```\r\n\r\n - [Information about the ID key in a step for GHA](https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idstepsid)\r\n - [An example workflow from a public repo](https://github.com/NixOS/nixpkgs/blob/b4cc66827745e525ce7bb54659845ac89788a597/.github/workflows/direct-push.yml#L16)\r\n\r\n# Changes\r\nI'm proposing that the key for `id` in step is replaced with `step_id` so that it no longer interferes with the table `id` for tracking the record.\r\n\r\nSpecial thanks to @sarcasticadmin @egiffen and @ruebenramirez for helping a bit on this \ud83d\ude04 ", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/67/reactions\", \"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 1, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 724759588, "node_id": "MDU6SXNzdWU3MjQ3NTk1ODg=", "number": 29, "title": "Add search highlighting snippets", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2020-10-19T16:00:48Z", "updated_at": "2021-08-26T20:23:11Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "Like on https://til.simonwillison.net/til/search?q=Snippet", "repo": {"value": 197431109, "label": "dogsheep-beta"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-beta/issues/29/reactions\", \"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 1, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 602585497, "node_id": "MDU6SXNzdWU2MDI1ODU0OTc=", "number": 7, "title": "Integrate image content hashing", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-04-19T00:36:58Z", "updated_at": "2021-08-26T02:01:01Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "To spot duplicate images (where the file content differs such that the sha256 is no longer a match) it would be useful to calculate and store perceptual hashes of some sort.", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/7/reactions\", \"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 1, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 642572841, "node_id": "MDU6SXNzdWU2NDI1NzI4NDE=", "number": 859, "title": "Database page loads too slowly with many large tables (due to table counts)", "user": {"value": 3243482, "label": "abdusco"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 21, "created_at": "2020-06-21T14:23:17Z", "updated_at": "2021-08-25T21:59:55Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "Hey,\r\nI have a database that I save in HTML from couple of web scrapers. There are around 200k+, 50+ rows in a couple of tables, with sqlite file weighing around 600MB.\r\n\r\nThe app runs on a VPS with 2 core CPU, 4GB RAM and refreshing database page regularly takes more than 10 seconds. I was suspecting that counting tables was the culprit, but manually running `select count(*) from table_name` for the largest table finishes under a second.\r\n\r\nI've looked at the source code. There's a check for index page for mutable databases larger than 100MB\r\nhttps://github.com/simonw/datasette/blob/799c5d53570d773203527f19530cf772dc2eeb24/datasette/views/index.py#L15\r\n\r\nbut this check is not performed for database page. \r\nI've manually crippled `Database::table_counts` method\r\n```py\r\nasync def table_counts(self, limit=10):\r\n if not self.is_mutable and self.cached_table_counts is not None:\r\n return self.cached_table_counts\r\n # Try to get counts for each table, $limit timeout for each count\r\n counts = {}\r\n for table in await self.table_names():\r\n try:\r\n # table_count = (\r\n # await self.execute(\r\n # \"select count(*) from [{}]\".format(table),\r\n # custom_time_limit=limit,\r\n # )\r\n # ).rows[0][0]\r\n counts[table] = 10 # table_count\r\n # In some cases I saw \"SQL Logic Error\" here in addition to\r\n # QueryInterrupted - so we catch that too:\r\n except (QueryInterrupted, sqlite3.OperationalError, sqlite3.DatabaseError):\r\n counts[table] = None\r\n if not self.is_mutable:\r\n self.cached_table_counts = counts\r\n return counts\r\n```\r\n\r\nnow the page loads in <100ms.\r\n\r\nIs it possible to apply size check on database page too?\r\n\r\n
\r\n\r\n/-/versions output\r\n\r\n
\r\n{\r\n    \"python\": {\r\n        \"version\": \"3.8.0\",\r\n        \"full\": \"3.8.0 (default, Oct 28 2019, 16:14:01) \\n[GCC 8.3.0]\"\r\n    },\r\n    \"datasette\": {\r\n        \"version\": \"0.44\"\r\n    },\r\n    \"asgi\": \"3.0\",\r\n    \"uvicorn\": \"0.11.5\",\r\n    \"sqlite\": {\r\n        \"version\": \"3.22.0\",\r\n        \"fts_versions\": [\r\n            \"FTS5\",\r\n            \"FTS4\",\r\n            \"FTS3\"\r\n        ],\r\n        \"extensions\": {\r\n            \"json1\": null\r\n        },\r\n        \"compile_options\": [\r\n            \"COMPILER=gcc-7.4.0\",\r\n            \"ENABLE_COLUMN_METADATA\",\r\n            \"ENABLE_DBSTAT_VTAB\",\r\n            \"ENABLE_FTS3\",\r\n            \"ENABLE_FTS3_PARENTHESIS\",\r\n            \"ENABLE_FTS3_TOKENIZER\",\r\n            \"ENABLE_FTS4\",\r\n            \"ENABLE_FTS5\",\r\n            \"ENABLE_JSON1\",\r\n            \"ENABLE_LOAD_EXTENSION\",\r\n            \"ENABLE_PREUPDATE_HOOK\",\r\n            \"ENABLE_RTREE\",\r\n            \"ENABLE_SESSION\",\r\n            \"ENABLE_STMTVTAB\",\r\n            \"ENABLE_UNLOCK_NOTIFY\",\r\n            \"ENABLE_UPDATE_DELETE_LIMIT\",\r\n            \"HAVE_ISNAN\",\r\n            \"LIKE_DOESNT_MATCH_BLOBS\",\r\n            \"MAX_SCHEMA_RETRY=25\",\r\n            \"MAX_VARIABLE_NUMBER=250000\",\r\n            \"OMIT_LOOKASIDE\",\r\n            \"SECURE_DELETE\",\r\n            \"SOUNDEX\",\r\n            \"TEMP_STORE=1\",\r\n            \"THREADSAFE=1\"\r\n        ]\r\n    }\r\n}\r\n
\r\n
", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/859/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 743071410, "node_id": "MDExOlB1bGxSZXF1ZXN0NTIxMDU0NjEy", "number": 13, "title": "SQLite does not have case sensitive columns", "user": {"value": 1689944, "label": "tomaskrehlik"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-11-14T20:12:32Z", "updated_at": "2021-08-24T13:28:26Z", "closed_at": null, "author_association": "FIRST_TIME_CONTRIBUTOR", "pull_request": "dogsheep/healthkit-to-sqlite/pulls/13", "body": "This solves a weird issue when there is record with metadata key\r\nthat is only different in letter cases.\r\n\r\nSee the test for details.", "repo": {"value": 197882382, "label": "healthkit-to-sqlite"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/13/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 978086284, "node_id": "MDExOlB1bGxSZXF1ZXN0NzE4NzM0MTkx", "number": 22, "title": "Make sure that case-insensitive column names are unique", "user": {"value": 32016596, "label": "FabianHertwig"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-08-24T13:13:38Z", "updated_at": "2021-08-24T13:26:20Z", "closed_at": null, "author_association": "FIRST_TIME_CONTRIBUTOR", "pull_request": "dogsheep/healthkit-to-sqlite/pulls/22", "body": "This closes #21.\r\n\r\nWhen there are metadata entries with the same case insensitive string, then there is an error when trying to create a new column for that metadata entry in the database table, because a column with that case insensitive name already exists.\r\n\r\n```xml\r\n \r\n \r\n \r\n \r\n```\r\n\r\nThe code added in this PR checks if a key already exists in a record and if so adds a number at its end. The resulting column names look like the example below then. Interestingly, the column names viewed with Datasette are not case insensitive.\r\n\r\n```text\r\nstartDate, endDate, value, unit, sourceName, sourceVersion, creationDate, metadata_meal, metadata_Meal_2, metadata_Mahlzeit\r\n```\r\n", "repo": {"value": 197882382, "label": "healthkit-to-sqlite"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/22/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 977323133, "node_id": "MDU6SXNzdWU5NzczMjMxMzM=", "number": 1445, "title": "Ability to search for text across all columns in a table", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2021-08-23T18:50:48Z", "updated_at": "2021-08-23T19:10:17Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "When I'm working with new data I often find myself wanting to run a search for text embedded in ANY of the columns of a table, without having to even fully understand the schema first.\r\n\r\nI figured out a trick for doing that using a SQL-generated SQL query here: https://til.simonwillison.net/datasette/search-all-columns-trick\r\n\r\nBut maybe this should be a core Datasette feature? Or a plugin?", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1445/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 977128935, "node_id": "MDU6SXNzdWU5NzcxMjg5MzU=", "number": 21, "title": "Duplicate Column", "user": {"value": 32016596, "label": "FabianHertwig"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-08-23T15:00:44Z", "updated_at": "2021-08-23T17:00:59Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Hey, thank you for this repo!\r\n\r\nWhen I try to convert my export, I get a multiple column error. Here is the stack trace:\r\n\r\n```sh\r\n(.venv) (base) computer:bodyweight_app user$ healthkit-to-sqlite ./data/Health_export.zip ./data/healthkit.db\r\nImporting from HealthKit [###############################-----] 87% 00:00:22\r\nTraceback (most recent call last):\r\n File \"/MyProject/.venv/bin/healthkit-to-sqlite\", line 10, in \r\n sys.exit(cli())\r\n File \"/MyProject/.venv/lib/python3.7/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/MyProject/.venv/lib/python3.7/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/MyProject/.venv/lib/python3.7/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/MyProject/.venv/lib/python3.7/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/MyProject/.venv/lib/python3.7/site-packages/healthkit_to_sqlite/cli.py\", line 57, in cli\r\n convert_xml_to_sqlite(fp, db, progress_callback=bar.update, zipfile=zf)\r\n File \"/MyProject/.venv/lib/python3.7/site-packages/healthkit_to_sqlite/utils.py\", line 41, in convert_xml_to_sqlite\r\n write_records(records, db)\r\n File \"/MyProject/.venv/lib/python3.7/site-packages/healthkit_to_sqlite/utils.py\", line 146, in write_records\r\n batch_size=50,\r\n File \"/MyProject/.venv/lib/python3.7/site-packages/sqlite_utils/db.py\", line 2579, in insert_all\r\n extracts=extracts,\r\n File \"/MyProject/.venv/lib/python3.7/site-packages/sqlite_utils/db.py\", line 1246, in create\r\n extracts=extracts,\r\n File \"/MyProject/.venv/lib/python3.7/site-packages/sqlite_utils/db.py\", line 767, in create_table\r\n self.execute(sql)\r\n File \"/MyProject/.venv/lib/python3.7/site-packages/sqlite_utils/db.py\", line 421, in execute\r\n return self.conn.execute(sql)\r\nsqlite3.OperationalError: duplicate column name: metadata_Meal\r\n```", "repo": {"value": 197882382, "label": "healthkit-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/21/reactions\", \"total_count\": 5, \"+1\": 5, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 657572753, "node_id": "MDU6SXNzdWU2NTc1NzI3NTM=", "number": 894, "title": "?sort=colname~numeric to sort by by column cast to real", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 21, "created_at": "2020-07-15T18:47:48Z", "updated_at": "2021-08-20T02:07:53Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "If a text column actually contains numbers, being able to \"sort by column, treated as numeric\" would be really useful.\r\n\r\nProbably depends on column actions enabled by #690", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/894/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 975166271, "node_id": "MDU6SXNzdWU5NzUxNjYyNzE=", "number": 20, "title": "Add index on workout_points.date", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-08-20T01:08:04Z", "updated_at": "2021-08-20T01:12:48Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "Sorting that by date makes sense for seeing most recent points, and my DB has 2.5m points in so it's an expensive sort!", "repo": {"value": 197882382, "label": "healthkit-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/20/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 970463436, "node_id": "MDExOlB1bGxSZXF1ZXN0NzEyNDEyODgz", "number": 1434, "title": "Enrich arbitrary query results with foreign key links and column descriptions", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-08-13T14:43:01Z", "updated_at": "2021-08-19T21:18:58Z", "closed_at": null, "author_association": "OWNER", "pull_request": "simonw/datasette/pulls/1434", "body": "Refs #1293, follows #942.", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1434/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 974067156, "node_id": "MDU6SXNzdWU5NzQwNjcxNTY=", "number": 318, "title": "Research: handle gzipped CSV directly", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-08-18T21:23:04Z", "updated_at": "2021-08-18T21:25:30Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Would it be worthwhile for the `sqlite-utils` command-line tool to grow features to efficiently directly interact with gzipped CSV data?\r\n\r\nMaybe add `--gz` options to both `insert` and to the various commands that output query results.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/318/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 972918533, "node_id": "MDU6SXNzdWU5NzI5MTg1MzM=", "number": 1438, "title": "Query page .csv and .json links are not correctly URL-encoded on Vercel under unknown specific conditions", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2021-08-17T17:35:36Z", "updated_at": "2021-08-18T00:22:23Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "> Confirmed: https://thesession.vercel.app/thesession?sql=select+*+from+tunes+where+name+like+%22%25wise+maid%25%22%0D%0A is a page where the URL correctly encoded `%` as `%25` - but then in the HTML on that page that links to the CSV and JSON versions we get this:\r\n>\r\n> ```html\r\n>

This data as\r\n> json,\r\n> CSV\r\n>

\r\n> ```\r\nThose CSV and JSON links are incorrect.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/datasette-publish-vercel/issues/48#issuecomment-900497579_", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1438/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 970626625, "node_id": "MDU6SXNzdWU5NzA2MjY2MjU=", "number": 1435, "title": "Turn off suggest facets on tables with large numbers of columns", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-08-13T18:30:48Z", "updated_at": "2021-08-13T18:30:48Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "If a table has 200 columns it will take multiple seconds to try and suggest facets. I should either quit after the first 20 or not suggest facets at all.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1435/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 969548935, "node_id": "MDU6SXNzdWU5Njk1NDg5MzU=", "number": 1429, "title": "UI for setting `?_size=max` on table page", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-08-12T20:52:09Z", "updated_at": "2021-08-13T04:37:41Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "It defaults to 100 per page, but you can increase that to 1000 per page using `?_size=max` (or higher if `max_returned_rows` is set higher than that).\r\n\r\nBut... that's only available to people who know how to hack URLs.\r\n\r\nSolution: add a link that sets that option to the pagination block at the bottom of the table:\r\n\r\n\"data__states__28_999_rows\"\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1429/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 722816436, "node_id": "MDU6SXNzdWU3MjI4MTY0MzY=", "number": 186, "title": ".extract() shouldn't extract null values", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2020-10-16T02:41:08Z", "updated_at": "2021-08-12T12:32:14Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "This almost works, but it creates a rogue `type` record with a value of None.\r\n```\r\nIn [1]: import sqlite_utils\r\nIn [2]: db = sqlite_utils.Database(memory=True)\r\nIn [5]: db[\"creatures\"].insert_all([\r\n {\"id\": 1, \"name\": \"Simon\", \"type\": None},\r\n {\"id\": 2, \"name\": \"Natalie\", \"type\": None},\r\n {\"id\": 3, \"name\": \"Cleo\", \"type\": \"dog\"}], pk=\"id\")\r\nOut[5]: \r\nIn [7]: db[\"creatures\"].extract(\"type\")\r\nOut[7]:
\r\nIn [8]: list(db[\"creatures\"].rows)\r\nOut[8]: \r\n[{'id': 1, 'name': 'Simon', 'type_id': None},\r\n {'id': 2, 'name': 'Natalie', 'type_id': None},\r\n {'id': 3, 'name': 'Cleo', 'type_id': 2}]\r\nIn [9]: db[\"type\"]\r\nOut[9]:
\r\nIn [10]: list(db[\"type\"].rows)\r\nOut[10]: [{'id': 1, 'type': None}, {'id': 2, 'type': 'dog'}]\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/186/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 963527045, "node_id": "MDU6SXNzdWU5NjM1MjcwNDU=", "number": 1424, "title": "Document exceptions that can be raised by db.execute() and friends", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-08-08T22:23:25Z", "updated_at": "2021-08-08T22:27:31Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Not currently covered here: https://docs.datasette.io/en/stable/internals.html#await-db-execute-sql", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1424/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 959710008, "node_id": "MDU6SXNzdWU5NTk3MTAwMDg=", "number": 1419, "title": "`publish cloudrun` should deploy a more recent SQLite version", "user": {"value": 536941, "label": "fgregg"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-08-04T00:45:55Z", "updated_at": "2021-08-05T03:23:24Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "I recently changed from deploying a datasette using `datasette publish heroku` to `datasette publish cloudrun`. [A query that ran on the heroku site](https://odpr.bunkum.us/odpr-6c2f4fc?sql=with+pivot_members+as+%28%0D%0A++select%0D%0A++++f_num%2C%0D%0A++++max%28union_name%29+as+union_name%2C%0D%0A++++max%28aff_abbr%29+as+abbreviation%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2007%0D%0A++++%29+as+%222007%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2008%0D%0A++++%29+as+%222008%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2009%0D%0A++++%29+as+%222009%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2010%0D%0A++++%29+as+%222010%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2011%0D%0A++++%29+as+%222011%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2012%0D%0A++++%29+as+%222012%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2013%0D%0A++++%29+as+%222013%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2014%0D%0A++++%29+as+%222014%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2015%0D%0A++++%29+as+%222015%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2016%0D%0A++++%29+as+%222016%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2017%0D%0A++++%29+as+%222017%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2018%0D%0A++++%29+as+%222018%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2019%0D%0A++++%29+as+%222019%22%0D%0A++from%0D%0A++++lm_data%0D%0A++++left+join+ar_membership+using+%28rpt_id%29%0D%0A++where%0D%0A++++members+%21%3D+%27%27%0D%0A++++and+trim%28desig_name%29+%3D+%27NHQ%27%0D%0A++++and+union_name+not+in+%28%0D%0A++++++%27AFL-CIO%27%2C%0D%0A++++++%27CHANGE+TO+WIN%27%2C%0D%0A++++++%27FOOD+ALLIED+SVC+TRADES+DEPT+AFL-CIO%27%0D%0A++++%29%0D%0A++++and+f_num+not+in+%28387%2C+296%2C+123%2C+347%2C+531897%2C+30410%2C+49%29%0D%0A++++and+lower%28ar_membership.category%29+IN+%28%0D%0A++++++%27active+members%27%2C%0D%0A++++++%27active+professional%27%2C%0D%0A++++++%27regular+members%27%2C%0D%0A++++++%27full+time+member%27%2C%0D%0A++++++%27full+time+members%27%2C%0D%0A++++++%27active+education+support+professional%27%2C%0D%0A++++++%27active%27%2C%0D%0A++++++%27active+member%27%2C%0D%0A++++++%27members%27%2C%0D%0A++++++%27see+item+69%27%2C%0D%0A++++++%27regular%27%2C%0D%0A++++++%27dues+paying+members%27%2C%0D%0A++++++%27building+trades+journeyman%27%2C%0D%0A++++++%27full+per+capita+tax+payers%27%2C%0D%0A++++++%27active+postal%27%2C%0D%0A++++++%27active+membership%27%2C%0D%0A++++++%27full-time+member%27%2C%0D%0A++++++%27regular+active+member%27%2C%0D%0A++++++%27journeyman%27%2C%0D%0A++++++%27member%27%2C%0D%0A++++++%27journeymen%27%2C%0D%0A++++++%27one+half+per+capita+tax+payers%27%2C%0D%0A++++++%27schedule+1%27%2C%0D%0A++++++%27active+memebers%27%2C%0D%0A++++++%27active+members+-+us%27%2C%0D%0A++++++%27dues+paying+membership%27%0D%0A++++%29%0D%0A++GROUP+by%0D%0A++++f_num%0D%0A%29%0D%0Aselect%0D%0A++*%0D%0Afrom%0D%0A++pivot_members%0D%0Aorder+by%0D%0A++%222019%22+desc%3B), now throws a syntax error on the [cloudrun site](https://labordata.bunkum.us/odpr-6c2f4fc?sql=with+pivot_members+as+%28%0D%0A++select%0D%0A++++f_num%2C%0D%0A++++max%28union_name%29+as+union_name%2C%0D%0A++++max%28aff_abbr%29+as+abbreviation%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2007%0D%0A++++%29+as+%222007%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2008%0D%0A++++%29+as+%222008%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2009%0D%0A++++%29+as+%222009%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2010%0D%0A++++%29+as+%222010%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2011%0D%0A++++%29+as+%222011%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2012%0D%0A++++%29+as+%222012%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2013%0D%0A++++%29+as+%222013%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2014%0D%0A++++%29+as+%222014%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2015%0D%0A++++%29+as+%222015%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2016%0D%0A++++%29+as+%222016%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2017%0D%0A++++%29+as+%222017%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2018%0D%0A++++%29+as+%222018%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2019%0D%0A++++%29+as+%222019%22%0D%0A++from%0D%0A++++lm_data%0D%0A++++left+join+ar_membership+using+%28rpt_id%29%0D%0A++where%0D%0A++++members+%21%3D+%27%27%0D%0A++++and+trim%28desig_name%29+%3D+%27NHQ%27%0D%0A++++and+union_name+not+in+%28%0D%0A++++++%27AFL-CIO%27%2C%0D%0A++++++%27CHANGE+TO+WIN%27%2C%0D%0A++++++%27FOOD+ALLIED+SVC+TRADES+DEPT+AFL-CIO%27%0D%0A++++%29%0D%0A++++and+f_num+not+in+%28387%2C+296%2C+123%2C+347%2C+531897%2C+30410%2C+49%29%0D%0A++++and+lower%28ar_membership.category%29+IN+%28%0D%0A++++++%27active+members%27%2C%0D%0A++++++%27active+professional%27%2C%0D%0A++++++%27regular+members%27%2C%0D%0A++++++%27full+time+member%27%2C%0D%0A++++++%27full+time+members%27%2C%0D%0A++++++%27active+education+support+professional%27%2C%0D%0A++++++%27active%27%2C%0D%0A++++++%27active+member%27%2C%0D%0A++++++%27members%27%2C%0D%0A++++++%27see+item+69%27%2C%0D%0A++++++%27regular%27%2C%0D%0A++++++%27dues+paying+members%27%2C%0D%0A++++++%27building+trades+journeyman%27%2C%0D%0A++++++%27full+per+capita+tax+payers%27%2C%0D%0A++++++%27active+postal%27%2C%0D%0A++++++%27active+membership%27%2C%0D%0A++++++%27full-time+member%27%2C%0D%0A++++++%27regular+active+member%27%2C%0D%0A++++++%27journeyman%27%2C%0D%0A++++++%27member%27%2C%0D%0A++++++%27journeymen%27%2C%0D%0A++++++%27one+half+per+capita+tax+payers%27%2C%0D%0A++++++%27schedule+1%27%2C%0D%0A++++++%27active+memebers%27%2C%0D%0A++++++%27active+members+-+us%27%2C%0D%0A++++++%27dues+paying+membership%27%0D%0A++++%29%0D%0A++GROUP+by%0D%0A++++f_num%0D%0A%29%0D%0Aselect%0D%0A++*%0D%0Afrom%0D%0A++pivot_members%0D%0Aorder+by%0D%0A++%222019%22+desc%3B).\r\n\r\nI suspect this is because they are running different versions of sqlite3.\r\n\r\n- Heroku: sqlite3 3.31.1 ([-/versions](https://odpr.bunkum.us/-/versions))\r\n- Cloudrun: sqlite3 3.27.2 ([-/versions](https://labordata.bunkum.us/-/versions))\r\n\r\nIf so, it would be great to\r\n\r\n1. harmonize the sqlite3 versions across platforms\r\n2. update the docker files so as to update the sqlite3 version for cloudrun", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1419/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 961008507, "node_id": "MDU6SXNzdWU5NjEwMDg1MDc=", "number": 308, "title": "Add an interactive tutorial as a Jupyter notebook", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-08-04T20:34:22Z", "updated_at": "2021-08-04T21:30:59Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Can show people how to open this up in Binder.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/308/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 957347860, "node_id": "MDU6SXNzdWU5NTczNDc4NjA=", "number": 1412, "title": "Mention WAL mode in the documentation (plus backup tips)", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-08-01T00:27:11Z", "updated_at": "2021-08-01T00:27:11Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "This is useful for people who are deploying Datasette with any write functionality, especially if they might be using something like EFS.\r\n\r\nI can add a section about this to the bottom of https://docs.datasette.io/en/stable/deploying.html and then link to that section from both https://docs.datasette.io/en/stable/sql_queries.html#writable-canned-queries and https://docs.datasette.io/en/stable/internals.html#await-db-execute-write-sql-params-none-block-false\r\n\r\nAlso useful: mention that just copying a SQLite database file while it is being written to may not get a consistent file, so tell people to use one of the SQLite backup mechanisms instead.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1412/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 957315684, "node_id": "MDU6SXNzdWU5NTczMTU2ODQ=", "number": 1410, "title": "Rename settings to `default_allow_facet` and `default_allow_download` and `default_allow_csv_stream`", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 0, "created_at": "2021-07-31T20:27:12Z", "updated_at": "2021-07-31T20:27:49Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "> If I was prone to over-thinking (which I am) I'd note that `allow_facet` and `allow_download` and `allow_csv_stream` are all settings that do NOT have an equivalent in the newer permissions system, which is itself a little weird and inconsistent.\r\n>\r\n> So maybe there's a future task where I introduce those as both permissions and metadata `\"allow_x\"` blocks, then rename the settings themselves to be called `default_allow_facet` and `default_allow_download` and `default_allow_csv_stream`.\r\n>\r\n> If I was going to do that I should get it in before Datasette 1.0.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1409#issuecomment-890400425_", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1410/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 957302085, "node_id": "MDU6SXNzdWU5NTczMDIwODU=", "number": 1408, "title": "Review places in codebase that use os.chdir(), in particularly relating to tests", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-07-31T18:57:06Z", "updated_at": "2021-07-31T19:00:32Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "> To clarify: the core problem here is that an error is thrown any time you call `os.getcwd()` but the directory you are currently in has been deleted.\r\n>\r\n> `runner.isolated_filesystem()` assumes that the current directory in has not been deleted. But the various temporary directory utilities in `pytest` work by creating directories and then deleting them.\r\n>\r\n> Maybe there's a larger problem here that I play a bit fast and loose with `os.chdir()` in both the test suite and in various lines of code in Datasette itself (in particular in the publish commands)?\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1406#issuecomment-890390198_", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1408/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 947044667, "node_id": "MDU6SXNzdWU5NDcwNDQ2Njc=", "number": 1398, "title": "Documentation on using Datasette as a library", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-07-18T14:15:27Z", "updated_at": "2021-07-30T03:21:49Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Instantiating `Datasette()` directly is an increasingly interesting pattern. I do it in tests all the time, but thanks to `datasette.client` there are plenty of neat things you can do with it in a library context.\r\n\r\nMaybe support `from datasette import Datasette` for this.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1398/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 925384329, "node_id": "MDExOlB1bGxSZXF1ZXN0NjczODcyOTc0", "number": 7, "title": "Add instagram-to-sqlite", "user": {"value": 36654812, "label": "gavindsouza"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-06-19T12:26:16Z", "updated_at": "2021-07-28T07:58:59Z", "closed_at": null, "author_association": "FIRST_TIME_CONTRIBUTOR", "pull_request": "dogsheep/dogsheep.github.io/pulls/7", "body": "The tool covers only chat imports at the time of opening this PR but I'm planning to import everything else that I feel inquisitive about\r\n\r\nref: https://github.com/gavindsouza/instagram-to-sqlite", "repo": {"value": 214746582, "label": "dogsheep.github.io"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep.github.io/issues/7/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 813880401, "node_id": "MDExOlB1bGxSZXF1ZXN0NTc3OTUzNzI3", "number": 5, "title": "WIP: Add Gmail takeout mbox import", "user": {"value": 306240, "label": "UtahDave"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 25, "created_at": "2021-02-22T21:30:40Z", "updated_at": "2021-07-28T07:18:56Z", "closed_at": null, "author_association": "FIRST_TIME_CONTRIBUTOR", "pull_request": "dogsheep/google-takeout-to-sqlite/pulls/5", "body": "WIP\r\n\r\nThis PR adds the ability to import emails from a Gmail mbox export from Google Takeout.\r\n\r\nThis is my first PR to a datasette/dogsheep repo. I've tested this on my personal Google Takeout mbox with ~520,000 emails going back to 2004. This took around ~20 minutes to process.\r\n\r\nTo provide some feedback on the progress of the import I added the \"rich\" python module. I'm happy to remove that if adding a dependency is discouraged. However, I think it makes a nice addition to give feedback on the progress of a long import.\r\n\r\nDo we want to log emails that have errors when trying to import them?\r\n\r\nDealing with encodings with emails is a bit tricky. I'm very open to feedback on how to deal with those better. As well as any other feedback for improvements.", "repo": {"value": 206649770, "label": "google-takeout-to-sqlite"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 951185411, "node_id": "MDU6SXNzdWU5NTExODU0MTE=", "number": 1402, "title": "feature request: social meta tags", "user": {"value": 536941, "label": "fgregg"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-07-23T01:57:23Z", "updated_at": "2021-07-26T19:31:41Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "it would be very nice if the twitter, slack, and other social media could make rich cards when people post a link to a datasette instance\r\n\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1402/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 952189173, "node_id": "MDU6SXNzdWU5NTIxODkxNzM=", "number": 3, "title": "Use HN algolia endpoint to retrieve trees", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-07-25T03:35:27Z", "updated_at": "2021-07-25T18:41:17Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "The `trees` command currently has to make a request for every single comment. Algolia have an endpoint that bundles the entire thread together into a single request.\r\n\r\n`https://hn.algolia.com/api/v1/items/ID`\r\n\r\nHere's an example that loads quickly, with about 50 comments: https://hn.algolia.com/api/v1/items/27941108\r\n\r\nIt doesn't appear to use pagination at all - if a thread is big then the response is big.\r\n\r\nI ran this search to find some stories with more than 1000 comments: https://hn.algolia.com/api/v1/search?tags=story&numericFilters=num_comments%3E=1000\r\n\r\nHere's one: https://news.ycombinator.com/item?id=25015967 with 4759 comments. Hitting the API takes 41s and returns 3.7 MB of JSON!\r\n```\r\nwget 'https://hn.algolia.com/api/v1/items/25015967' 0.03s user 0.04s system 0% cpu 41.368 total\r\n/tmp % ls -lah 25015967 \r\n-rw-r--r-- 1 simon wheel 3.7M Jul 24 20:31 25015967\r\n```", "repo": {"value": 248903544, "label": "hacker-news-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/3/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 952179830, "node_id": "MDU6SXNzdWU5NTIxNzk4MzA=", "number": 2, "title": "Command for fetching Hacker News threads from the search API", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-07-25T02:00:45Z", "updated_at": "2021-07-25T03:12:57Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "I want to be able to fetch every item for a domain, e.g. https://news.ycombinator.com/from?site=simonwillison.net", "repo": {"value": 248903544, "label": "hacker-news-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/2/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 803333769, "node_id": "MDU6SXNzdWU4MDMzMzM3Njk=", "number": 32, "title": "KeyError: 'Contents' on running upload", "user": {"value": 11855322, "label": "robmarkcole"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-02-08T08:36:37Z", "updated_at": "2021-07-22T06:40:25Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Following the readme, on big sur, and having entered my auth creds via `dogsheep-photos s3-auth`:\r\n\r\n```\r\n(venv) (base) Robins-MacBook:datasette robin$ dogsheep-photos upload photos.db ~/Pictures/Photos\\ /Users/robin/Pictures/Library.photoslibrary --dry-run\r\n\r\nFetching existing keys from S3...\r\nTraceback (most recent call last):\r\n File \"/Users/robin/datasette/venv/bin/dogsheep-photos\", line 8, in \r\n sys.exit(cli())\r\n File \"/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/Users/robin/datasette/venv/lib/python3.8/site-packages/dogsheep_photos/cli.py\", line 96, in upload\r\n key.split(\".\")[0] for key in get_all_keys(client, creds[\"photos_s3_bucket\"])\r\n File \"/Users/robin/datasette/venv/lib/python3.8/site-packages/dogsheep_photos/utils.py\", line 46, in get_all_keys\r\n for row in page[\"Contents\"]:\r\nKeyError: 'Contents'\r\n```\r\n\r\nPossibly since the bucket is in `EU (London) eu-west-2` and this into is not requested?", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/32/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 855446829, "node_id": "MDExOlB1bGxSZXF1ZXN0NjEzMTc4OTY4", "number": 1296, "title": "Dockerfile: use Ubuntu 20.10 as base", "user": {"value": 82332573, "label": "tmcl-it"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-04-12T00:23:32Z", "updated_at": "2021-07-20T08:52:13Z", "closed_at": null, "author_association": "FIRST_TIME_CONTRIBUTOR", "pull_request": "simonw/datasette/pulls/1296", "body": "This PR changes the main Dockerfile to use ubuntu:20.10 as base image instead of python:3.9.2-slim-buster (itself based on debian:buster-slim).\r\n\r\nThe Dockerfile is essentially the one from https://github.com/simonw/datasette/issues/1249#issuecomment-803698983 with some additional cleanups to slim it down.\r\n\r\nThis fixes a couple of issues:\r\n1. The SQLite version in Debian Buster (2.6.0) doesn't support generated columns\r\n2. Installing SpatiaLite from the Debian sid repositories has the side effect of also installing updates to libc and libstdc++ from sid.\r\n\r\nAs a bonus, the Docker image becomes smaller:\r\n\r\n\r\n```\r\n$ docker image ls\r\nREPOSITORY TAG IMAGE ID CREATED SIZE\r\ndatasette 0.56-ubuntu f7aca255140a 5 hours ago 212MB\r\ndatasetteproject/datasette 0.56 efb3b282f390 13 days ago 258MB\r\n```\r\n\r\n### Reproduction of the first issue\r\n\r\n```\r\n$ curl -O https://latest.datasette.io/fixtures.db\r\n % Total % Received % Xferd Average Speed Time Time Time Current\r\n Dload Upload Total Spent Left Speed\r\n100 260k 0 260k 0 0 489k 0 --:--:-- --:--:-- --:--:-- 489k\r\n\r\n$ docker run -v `pwd`:/mnt datasetteproject/datasette:0.56 datasette /mnt/fixtures.db\r\nTraceback (most recent call last):\r\n File \"/usr/local/bin/datasette\", line 8, in \r\n sys.exit(cli())\r\n File \"/usr/local/lib/python3.9/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/usr/local/lib/python3.9/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/usr/local/lib/python3.9/site-packages/click/core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/usr/local/lib/python3.9/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/usr/local/lib/python3.9/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/usr/local/lib/python3.9/site-packages/datasette/cli.py\", line 544, in serve\r\n asyncio.get_event_loop().run_until_complete(check_databases(ds))\r\n File \"/usr/local/lib/python3.9/asyncio/base_events.py\", line 642, in run_until_complete\r\n return future.result()\r\n File \"/usr/local/lib/python3.9/site-packages/datasette/cli.py\", line 584, in check_databases\r\n await database.execute_fn(check_connection)\r\n File \"/usr/local/lib/python3.9/site-packages/datasette/database.py\", line 155, in execute_fn\r\n return await asyncio.get_event_loop().run_in_executor(\r\n File \"/usr/local/lib/python3.9/concurrent/futures/thread.py\", line 52, in run\r\n result = self.fn(*self.args, **self.kwargs)\r\n File \"/usr/local/lib/python3.9/site-packages/datasette/database.py\", line 153, in in_thread\r\n return fn(conn)\r\n File \"/usr/local/lib/python3.9/site-packages/datasette/utils/__init__.py\", line 892, in check_connection\r\n for r in conn.execute(\r\nsqlite3.DatabaseError: malformed database schema (generated_columns) - near \"AS\": syntax error\r\n```\r\n\r\nHere is the SQLite version:\r\n\r\n```\r\n$ docker run -v `pwd`:/mnt -it datasetteproject/datasette:0.56 /bin/bash\r\nroot@d9220d3b95dd:/# python3\r\nPython 3.9.2 (default, Mar 27 2021, 02:50:26) \r\n[GCC 8.3.0] on linux\r\nType \"help\", \"copyright\", \"credits\" or \"license\" for more information.\r\n>>> import sqlite3\r\n>>> sqlite3.version\r\n'2.6.0'\r\n```\r\n\r\n### Reproduction of the second issue\r\n\r\n```\r\n$ docker build . -t datasette --build-arg VERSION=0.55\r\n[...snip...]\r\nThe following packages will be upgraded:\r\n libc-bin libc6 libstdc++6\r\n[...snip...]\r\nUnpacking libc6:amd64 (2.31-11) over (2.28-10) ...\r\n[...snip...]\r\nUnpacking libstdc++6:amd64 (10.2.1-6) over (8.3.0-6) ...\r\n[...snip...]\r\n```\r\n\r\nBoth libc and libstdc++ are backwards compatible, so the image still works, but it will result in a combination of libraries and Python versions that exists only in the Datasette image, so it's likely untested. In addition, since Debian sid is an always-changing rolling-release, the versions of libc, libstdc++, Spatialite, and their dependencies change frequently, so the library versions in the Datasette image will depend on the day when it was built.\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1296/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 947596222, "node_id": "MDExOlB1bGxSZXF1ZXN0NjkyNTU3Mzgx", "number": 1399, "title": "Multiple sort", "user": {"value": 87192257, "label": "jgryko5"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-07-19T12:20:14Z", "updated_at": "2021-07-19T12:20:14Z", "closed_at": null, "author_association": "FIRST_TIME_CONTRIBUTOR", "pull_request": "simonw/datasette/pulls/1399", "body": "Closes #197.\r\nI have added support for sorting by multiple parameters as mentioned in the issue above, and together with that, a suggestion on how to implement such sorting in the user interface.", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1399/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 275125561, "node_id": "MDU6SXNzdWUyNzUxMjU1NjE=", "number": 123, "title": "Datasette serve should accept paths/URLs to CSVs and other file formats", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2017-11-19T02:05:48Z", "updated_at": "2021-07-19T00:04:32Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "This would remove the csvs-to-sqlite step which I end up using for almost everything.\r\n\r\nI'm hesitant to introduce pandas as a required dependency though since it require compiling numpy. Could build it so this option is only available if you have pandas installed.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/123/reactions\", \"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 1, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 792652391, "node_id": "MDU6SXNzdWU3OTI2NTIzOTE=", "number": 1199, "title": "Experiment with PRAGMA mmap_size=N", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-01-23T21:24:09Z", "updated_at": "2021-07-17T17:39:17Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "https://sqlite.org/mmap.html - SQLite supports memory-mapped I/O but it's disabled by default. The `PRAGMA mmap_size=N` option can be used to enable it.\r\n\r\nIt would be very interesting to understand the impact this could have on Datasette performance for various different shapes of data.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1199/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null}