{"id": 267515678, "node_id": "MDU6SXNzdWUyNjc1MTU2Nzg=", "number": 3, "title": "Make individual column valuables addressable, with smart content types", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2017-10-23T01:11:32Z", "updated_at": "2017-12-10T03:11:58Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Some SQLite databases embed images in columns. It would be cool if these had URLs.\r\n\r\n /database-name-7sha256/table-name/compound-pk/column\r\n /database-name-7sha256/table-name/compound-pk/column.json\r\n /database-name-7sha256/table-name/compound-pk/column.png\r\n /database-name-7sha256/table-name/compound-pk/column.gif\r\n /database-name-7sha256/table-name/compound-pk/column.txt\r\n\r\nThe one without an explicit file extension auto-detects the correct extension.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/3/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 268110769, "node_id": "MDU6SXNzdWUyNjgxMTA3Njk=", "number": 33, "title": "Use locust for benchmarking and load tests", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2017-10-24T17:00:09Z", "updated_at": "2017-12-10T03:12:16Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "https://github.com/locustio/locust\r\n\r\nNeeded for #32 ", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/33/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 274615452, "node_id": "MDU6SXNzdWUyNzQ2MTU0NTI=", "number": 111, "title": "Add \u201cupdated\u201d to metadata", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 12, "created_at": "2017-11-16T18:22:20Z", "updated_at": "2021-09-21T22:48:27Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "To give an indication as to when the data was last updated.\r\n\r\nThis should be a field in the metadata that is then shown on the index page and in the footer, if it is set.\r\n\r\nAlso support setting it using an option to \u201cdatasette publish\u201d and \u201cdatasette package\u201d - which can either be a string or can be the magic string \u201ctoday\u201d to set it to today\u2019s date:\r\n\r\n datasette publish file.db --updated=today", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/111/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 275125561, "node_id": "MDU6SXNzdWUyNzUxMjU1NjE=", "number": 123, "title": "Datasette serve should accept paths/URLs to CSVs and other file formats", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2017-11-19T02:05:48Z", "updated_at": "2021-07-19T00:04:32Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "This would remove the csvs-to-sqlite step which I end up using for almost everything.\r\n\r\nI'm hesitant to introduce pandas as a required dependency though since it require compiling numpy. Could build it so this option is only available if you have pandas installed.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/123/reactions\", \"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 1, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 275159710, "node_id": "MDU6SXNzdWUyNzUxNTk3MTA=", "number": 128, "title": "Every visualization should have an \"embed\" button", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2017-11-19T13:38:13Z", "updated_at": "2019-05-13T18:33:51Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "At least for the first round of visualizations, any time you construct one using the UI the result should include an \"embed this\" button that returns source code to copy and paste\r\n\r\nThese examples should use unpkg.com (or similarl) urls with SRI hashes, eg https://www.srihash.org - and should load data from the datasette JSON API.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/128/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 275415799, "node_id": "MDU6SXNzdWUyNzU0MTU3OTk=", "number": 137, "title": "Ability to combine multiple SQL queries on a single graph", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2017-11-20T16:26:57Z", "updated_at": "2019-05-13T18:33:51Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "This would make visualizations significantly more powerful. The interesting challenge will be around the URL design. It would be useful to be able to combine either multiple explicit SQL queries or multiple queries based on the filter string parameters passed to one or more table views.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/137/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 275755475, "node_id": "MDU6SXNzdWUyNzU3NTU0NzU=", "number": 140, "title": "Heatmap visualization plugin", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2017-11-21T15:34:23Z", "updated_at": "2019-05-13T18:33:51Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Could use https://github.com/scottbedard/svelte-heatmap", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/140/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 288438570, "node_id": "MDU6SXNzdWUyODg0Mzg1NzA=", "number": 179, "title": "More metadata options for template authors ", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2018-01-14T20:51:04Z", "updated_at": "2019-05-13T18:33:33Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "See this thread on Twitter: https://twitter.com/simonw/status/952637152797458432", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/179/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 309047460, "node_id": "MDU6SXNzdWUzMDkwNDc0NjA=", "number": 188, "title": "Ability to bundle metadata and templates inside the SQLite file", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2018-03-27T16:42:07Z", "updated_at": "2020-12-04T17:18:34Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "One of the nicest qualities of SQLite as a data format is that you get a single file which you can then backup or share with other people. \r\n\r\nDatasette breaks this a little once you start including custom metadata.json or template files and CSS.\r\n\r\nIt would be cool if there was an optional mechanism for baking that extra configuration into the SQLite file itself. That way entire datasette mini-applications (including canned queries and custom HTML and CSS) could be constructed as single .db files.\r\n\r\nSince datasette configuration is all file-based, one way to achieve that would be to support a \"datasette_files\" table which, if present is used to search for file contents by path.\r\n\r\nThis is inline with the philosophy described by https://www.sqlite.org/appfileformat.html\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/188/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 312395790, "node_id": "MDU6SXNzdWUzMTIzOTU3OTA=", "number": 197, "title": "Ability to sort by more than one column", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2018-04-09T05:13:30Z", "updated_at": "2018-07-10T17:45:37Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Split off from #189.\r\n\r\nI'd like to support \"sort by X descending, then by Y ascending if there are dupes for X\" as well. Suggested syntax for that:\r\n\r\n ?_sort_desc=X&_sort=Y\r\n\r\nwe currently only allow one argument to be sent. We should allow as many arguments as there are columns, for example:\r\n\r\n ?_sort=department&_sort_desc=precinct&_sort=age&_sort_desc=size", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/197/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 312396095, "node_id": "MDU6SXNzdWUzMTIzOTYwOTU=", "number": 198, "title": "Ability to sort with nulls last", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2018-04-09T05:15:40Z", "updated_at": "2018-07-10T17:45:37Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Split off from #189\r\n\r\nHere's how to do that in SQL: https://fivethirtyeight.datasettes.com/fivethirtyeight-2628db9?sql=select+rowid%2C+*+from+%5Bnfl-wide-receivers%2Fadvanced-historical%5D%0D%0Aorder+by+case+when+career_ranypa+is+null+then+1+else+0+end%2C+career_ranypa%2C+rowid\r\n\r\n order by case when career_ranypa is null then 1 else 0 end, career_ranypa", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/198/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 314771615, "node_id": "MDU6SXNzdWUzMTQ3NzE2MTU=", "number": 218, "title": "Support custom unit display in order to handle \"$10,000\"", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2018-04-16T18:39:31Z", "updated_at": "2018-07-10T17:45:38Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "I tried to get Datasette to display `$10,000` using the new units support but we currently only display units as a suffix:\r\n\r\nhttps://github.com/simonw/datasette/blob/10a34f995c70daa37a8a2aa02c3135a4b023a24c/datasette/app.py#L563-L572\r\n\r\nIt would be neat if there was a mechanism for specifying a custom unit display - maybe something like this:\r\n\r\n```\r\n{\r\n \"custom_units\": {\r\n \"us_dollar\": {\r\n \"unit\": \"us_dollar = [] = $\",\r\n \"format\": \"${:,}\"\r\n }\r\n }\r\n}\r\n```", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/218/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 316621102, "node_id": "MDU6SXNzdWUzMTY2MjExMDI=", "number": 235, "title": "Add limit on the size in KB of data returned from a single query", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2018-04-22T23:01:15Z", "updated_at": "2018-04-24T00:30:02Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Datasette limits the number of rows returned to 1,000 and limits the time spent executing a SQL query to 1000ms - and both of these limits can be customized.\r\n\r\nIt does not have a limit on the size of the response returned. It's possible to compose maliciously large SQL responses in a small number of rows using mechanisms like the `group_concat()` aggregate function. It would be good to avoid malicious SQL creating 100MB+ responses and potentially crashing the server.\r\n\r\nI think the easiest place to implement that is here:\r\n\r\nhttps://github.com/simonw/datasette/blob/f3f42957128c1e7ece584d45d9167f2ac003a3b8/datasette/app.py#L175-L190\r\n\r\nCurrently we use `cursor.fetchmany()` to fetch up to 1,001 rows at once. Instead, we could switch to iterating through `cursor.fetchone()` (or just using `for row in cursor`) and keeping a running tally of the size of the response as we go - maybe just using `rough_response_size += len(str(row))`. If that goes above a certain threshold we can terminate the response with an error, like we do with timelimits.\r\n\r\nThe bigger challenge here is understanding how well this approach works and what impact it will have on overall Datasette performance. I think I need #33 for this.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/235/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 317001500, "node_id": "MDU6SXNzdWUzMTcwMDE1MDA=", "number": 236, "title": "datasette publish lambda plugin", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 11, "created_at": "2018-04-23T22:10:30Z", "updated_at": "2023-03-12T14:04:15Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Refs #217 - create a publish plugin that can deploy to AWS Lambda.\r\n\r\nhttps://docs.aws.amazon.com/lambda/latest/dg/limits.html says lambda packages can be up to 50 MB, so this would only work with smaller databases (the command can check the filesize before attempting to package and deploy it).\r\n\r\nLambdas do get a 512 MB `/tmp` directory too, so for larger databases the function could start and then download up to 512MB from an S3 bucket - so the plugin could take an optional S3 bucket to write to and know how to upload the `.db` file there and then have the lambda download it on startup.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/236/reactions\", \"total_count\": 2, \"+1\": 2, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 318490133, "node_id": "MDU6SXNzdWUzMTg0OTAxMzM=", "number": 241, "title": "Default datasette logging format should be JSON", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2018-04-27T17:32:48Z", "updated_at": "2018-07-10T17:45:40Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Structured logs are better. Datasette should default to outputting it's HTTP access log lines as newline delimited JSON instead of the Sanic default format it uses at the moment.\r\n\r\nFor improved greppability these logs should have keys ordered in a consistent way. Python's JSON module can do this with ordered dictionaries.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/241/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 320132682, "node_id": "MDU6SXNzdWUzMjAxMzI2ODI=", "number": 250, "title": "Setup some issue templates", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2018-05-04T01:49:07Z", "updated_at": "2018-05-04T01:49:07Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "https://twitter.com/left_pad/status/99216385740464537\r\n\r\nI like the idea of using these to help people understand some of the ways I want to use issues.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/250/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 323223872, "node_id": "MDU6SXNzdWUzMjMyMjM4NzI=", "number": 260, "title": "Validate metadata.json on startup", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2018-05-15T13:42:56Z", "updated_at": "2023-06-21T12:51:22Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "It's easy to misspell the name of a database or table and then be puzzled when the metadata settings silently fail.\r\n\r\nTo avoid this, let's sanity check the provided metadata.json on startup and quit with a useful error message if we find any obvious mistakes.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/260/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 323658641, "node_id": "MDU6SXNzdWUzMjM2NTg2NDE=", "number": 262, "title": "Add ?_extra= mechanism for requesting extra properties in JSON", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 27, "created_at": "2018-05-16T14:55:42Z", "updated_at": "2023-03-29T06:22:22Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Datasette views currently work by creating a set of data that should be returned as JSON, then defining an additional, optional `template_data()` function which is called if the view is being rendered as HTML.\r\n\r\nThis `template_data()` function calculates extra template context variables which are necessary for the HTML view but should not be included in the JSON.\r\n\r\nExample of how that is used today: https://github.com/simonw/datasette/blob/2b79f2bdeb1efa86e0756e741292d625f91cb93d/datasette/views/table.py#L672-L704\r\n\r\nWith features like Facets in #255 I'm beginning to want to move more items into the `template_data()` - in the case of facets it's the `suggested_facets` array. This saves that feature from being calculated (involving several SQL queries) for the JSON case where it is unlikely to be used.\r\n\r\nBut... as an API user, I want to still optionally be able to access that information.\r\n\r\nSolution: Add a `?_extra=suggested_facets&_extra=table_metadata` argument which can be used to optionally request additional blocks to be added to the JSON API.\r\n\r\nThen redefine as many of the current `template_data()` features as extra arguments instead, and teach Datasette to return certain extras by default when rendering templates.\r\n\r\nThis could allow the JSON representation to be slimmed down further (removing e.g. the `table_definition` and `view_definition` keys) while still making that information available to API users who need it.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/262/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 323718842, "node_id": "MDU6SXNzdWUzMjM3MTg4NDI=", "number": 268, "title": "Mechanism for ranking results from SQLite full-text search", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 12, "created_at": "2018-05-16T17:36:40Z", "updated_at": "2022-01-13T22:21:28Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "This isn't particularly straight-forward - all the more reason for Datasette to implement it for you. This article is helpful: http://charlesleifer.com/blog/using-sqlite-full-text-search-with-python/", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/268/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 326599525, "node_id": "MDU6SXNzdWUzMjY1OTk1MjU=", "number": 286, "title": "Database hash should include current datasette version", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2018-05-25T17:03:42Z", "updated_at": "2018-05-25T17:07:36Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Right now deploying a new version of datasette doesn't invalidate existing URLs, so users may still see a cached copy of the old templates.\r\n\r\nWe can fix this by including the current datasette version in the input to the hash function (which currently just the database file contents).", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/286/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 326778161, "node_id": "MDU6SXNzdWUzMjY3NzgxNjE=", "number": 290, "title": "Consider increasing the default for num_sql_threads (currently 3)", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2018-05-27T00:52:41Z", "updated_at": "2018-05-27T00:52:41Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "I ran a very rough micro-benchmark on the new `num_sql_threads` config option (added in #285)\r\n\r\n datasette --config num_sql_threads:1 fivethirtyeight.db\r\n\r\nThen\r\n\r\n ab -n 100 -c 10 'http://127.0.0.1:8011/fivethirtyeight-2628db9/twitter-ratio%2Fsenators'\r\n\r\n| Number of threads | Requests/second |\r\n|---|---|\r\n| 1 | 4.57 |\r\n| 3 | 9.77 |\r\n| 10 | 13.53 |\r\n| 20 | 15.24 \r\n| 50 | 8.21 | \r\n\r\nThis was on my early 2018 OS X laptop. Need to benchmark in other common environments before making a decision on changing the default. That said, the default of 3 was a number I plucked out of thin air.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/290/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 327365110, "node_id": "MDU6SXNzdWUzMjczNjUxMTA=", "number": 294, "title": "inspect should record column types", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2018-05-29T15:10:41Z", "updated_at": "2019-06-28T16:45:28Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "For each table we want to know the columns, their order and what type they are.\r\n\r\nI'm going to break with SQLite defaults a little on this one and allow datasette to define additional types - to start with just a `geometry` type for columns that are detected as SpatiaLite geometries.\r\n\r\nPossible JSON design:\r\n\r\n \"columns\": [{\r\n \"name\": \"title\",\r\n \"type\": \"text\"\r\n }, ...]\r\n\r\nRefs #276", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/294/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 327395270, "node_id": "MDU6SXNzdWUzMjczOTUyNzA=", "number": 296, "title": "Per-database and per-table /-/ URL namespace", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2018-05-29T16:23:13Z", "updated_at": "2019-06-28T16:46:34Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Initially this will be for subsets of `/-/inspect` and `/-/metadata` but it will also give us a URL namespace for future features like `/-/facet` (expanded list of a specific facet, linked to from `...`) and `/-/graph`\r\n\r\nTo start:\r\n\r\n* `/dbname/-/inspect`\r\n* `/dbname/-/metadata`\r\n* `/dbname/tablename/-/inspect`\r\n* `/dbname/tablename/-/metadata`\r\n\r\nThis means we will no longer allow databases or tables to have the name `\"-\"` - I think that's OK\r\n\r\nWe will continue to support rows with a primary key of `\"-\"` at the following URL:\r\n\r\n* `/dbname/tablename/-`", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/296/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 328155946, "node_id": "MDU6SXNzdWUzMjgxNTU5NDY=", "number": 301, "title": "--spatialite option for \"datasette publish heroku\"", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2018-05-31T14:13:09Z", "updated_at": "2022-01-20T21:28:50Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Split off from #243. Need to figure out how to install and configure SpatiaLite on Heroku.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/301/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 335200136, "node_id": "MDU6SXNzdWUzMzUyMDAxMzY=", "number": 327, "title": "Explore if SquashFS can be used to shrink size of packaged Docker containers", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2018-06-24T18:15:16Z", "updated_at": "2022-02-17T23:37:24Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Inspired by this article: https://cldellow.com/2018/06/22/sqlite-parquet-vtable.html#sqlite-database-indexed--squashed\r\n\r\nhttps://en.wikipedia.org/wiki/SquashFS is \"a compressed read-only file system for Linux\" - which means it could be a really nice fit for Datasette and its read-only SQLite databases.\r\n\r\nIt would be interesting to explore a Dockerfile recipe that used SquashFS to compress the SQLite database file that was bundled up by `datasette package` and friends.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/327/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 344654623, "node_id": "MDU6SXNzdWUzNDQ2NTQ2MjM=", "number": 347, "title": "Rename \"datasette package\" to \"datasette publish docker\"", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2018-07-26T00:42:46Z", "updated_at": "2018-07-26T00:42:46Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/347/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 346026869, "node_id": "MDU6SXNzdWUzNDYwMjY4Njk=", "number": 354, "title": "Handle many-to-many relationships", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2018-07-31T04:03:13Z", "updated_at": "2020-11-24T19:51:18Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "This is a master tracking ticket for various many-2-many features.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/354/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 346027040, "node_id": "MDU6SXNzdWUzNDYwMjcwNDA=", "number": 355, "title": "Table view should support filtering via many-to-many relationships", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 10, "created_at": "2018-07-31T04:04:16Z", "updated_at": "2019-05-23T06:04:03Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Parent: #354 ", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/355/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 348043884, "node_id": "MDU6SXNzdWUzNDgwNDM4ODQ=", "number": 357, "title": "Plugin hook for loading metadata.json", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2018-08-06T19:00:01Z", "updated_at": "2020-06-21T22:19:58Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "For https://github.com/simonw/russian-ira-facebook-ads-datasette/tree/af6d956995e14afd585c35a6a06bb01da32043ba I wrote a script to convert YAML to JSON because YAML is a better format for embedding multi-line HTML descriptions and canned SQL statements.\r\n\r\nExample yaml metadata file: https://github.com/simonw/russian-ira-facebook-ads-datasette/blob/af6d956995e14afd585c35a6a06bb01da32043ba/russian-ads-metadata.yaml\r\n\r\nIt would be useful if Datasette could be fed a YAML file directly:\r\n\r\n datasette -m metadata.yaml\r\n\r\nQuestion is... should this be a native feature (hence adding a YAML dependency) or should it be handled by a `datasette-metadata-yaml` plugin, using a new plugin hook for loading metadata? If so, what would other use-cases for that plugin hook be?", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/357/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 400340905, "node_id": "MDU6SXNzdWU0MDAzNDA5MDU=", "number": 402, "title": "Use SQLITE_DBCONFIG_DEFENSIVE plus other recommendations from SQLite security docs", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2019-01-17T15:52:28Z", "updated_at": "2019-01-17T16:15:21Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "> Was just having a skim through the datasette source. Given that the vuln impacts shadow tables, wasn't sure whether these are also covered by the immutable flag. Latest release introduced a SQLITE_DBCONFIG_DEFENSIVE flag that they recommend setting: https://sqlite.org/security.html\r\n\r\nhttps://twitter.com/ignoredambience/status/1085926961413869568", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/402/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 421546944, "node_id": "MDU6SXNzdWU0MjE1NDY5NDQ=", "number": 417, "title": "Datasette Library", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 12, "created_at": "2019-03-15T14:30:22Z", "updated_at": "2020-12-29T14:34:50Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "The ability to run Datasette in a mode where it automatically picks up new (or modified) files in a directory tree without needing to restart the server.\r\n\r\nSuggested command:\r\n\r\n datasette library /path/to/mydbs/", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/417/reactions\", \"total_count\": 8, \"+1\": 8, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 426722204, "node_id": "MDU6SXNzdWU0MjY3MjIyMDQ=", "number": 423, "title": "?_search_col=X not reflected correctly in the UI", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-03-28T21:48:19Z", "updated_at": "2020-11-03T19:01:59Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "e.g. https://latest.datasette.io/fixtures/searchable?_search_text1=barry\r\n\r\n![2019-03-28 at 2 47 PM](https://user-images.githubusercontent.com/9599/55195035-84ebb800-5168-11e9-910b-fc9868bcd93e.png)\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/423/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 443021509, "node_id": "MDU6SXNzdWU0NDMwMjE1MDk=", "number": 461, "title": "Paginate + search for databases/tables on the homepage", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 4, "created_at": "2019-05-11T18:05:34Z", "updated_at": "2020-12-17T22:14:46Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Split out from #460 - in order to support large numbers of connected databases the homepage needs to be paginated.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/461/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 447408527, "node_id": "MDU6SXNzdWU0NDc0MDg1Mjc=", "number": 483, "title": "Option to facet by date using month or year", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2019-05-23T01:25:29Z", "updated_at": "2019-05-29T21:38:27Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Facet by date (from #481) can take datetimes and facet them by the day component.\r\n\r\nhttps://latest.datasette.io/fixtures/facetable?_facet_date=created\r\n\r\nI'd like to also be able to facet by month or year.\r\n\r\nI'm not sure what the best way to achieve this is. Could be two more Facet classes (YearFacet and MonthFacet) but I think it might be nicer if the existing DateFacet could take an optional argument that changed its behaviour. But... if I do that, do I expose it in the UI somewhere or is it only available to URL-hackers?", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/483/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 447451492, "node_id": "MDU6SXNzdWU0NDc0NTE0OTI=", "number": 484, "title": "Mechanism for displaying summary of m2m relationships in rows on table view", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2019-05-23T05:02:41Z", "updated_at": "2019-05-23T06:34:05Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Part of #354 (m2m support)\r\n\r\nIt would be fantastic if rows that are part of a m2m relationship could display it in an additional column in the table view.\r\n\r\nIt might look something like this: https://russian-ira-facebook-ads.datasettes.com/russian-ads-919cbfd/display_ads?_search=black+lives+matter\r\n\r\n\"russian-ads__display_ads__50_rows_where_where_search_matches__black_lives_matter_\"\r\n\r\nThat example [was achieved](https://github.com/simonw/russian-ira-facebook-ads-datasette/blob/daf51a8c50a78e8bc7971c211005fd85e66ccf64/russian-ads-metadata.yaml#L72-L77) using a custom SQL query and [datasette-json-html](https://github.com/simonw/datasette-json-html) - but I'd like this to be a built-in feature instead.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/484/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 447469253, "node_id": "MDU6SXNzdWU0NDc0NjkyNTM=", "number": 485, "title": "Improvements to table label detection ", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": {"value": 9599, "label": "simonw"}, "milestone": null, "comments": 10, "created_at": "2019-05-23T06:19:49Z", "updated_at": "2022-10-03T00:04:42Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Label detection doesn't work if the primary key is called pk rather than id, so this page doesn't work: https://latest.datasette.io/fixtures/roadside_attraction_characteristics\r\n\r\nCode is here: \r\n\r\nhttps://github.com/simonw/datasette/blob/cccea85be6aaaeadb31f3b588ec7f732628815f5/datasette/app.py#L644-L653", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/485/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 449445715, "node_id": "MDU6SXNzdWU0NDk0NDU3MTU=", "number": 491, "title": "Figure out how to use Firebase with cloudrun to enable vanity URLs and CDN caching", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-05-28T19:48:06Z", "updated_at": "2019-05-28T19:48:35Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "It looks like Firebase can solve a couple of problems with the existing `datasette publish cloudrun` hosting mechanism:\r\n\r\n* The URLs it produces aren't pretty enough. Firebase offers more control over vanity URLs.\r\n* CDN caching (as seen in `datasette publish now`) is great for improving performance and saving money on Cloud Run execution time.\r\n\r\nhttps://firebase.google.com/docs/hosting/cloud-run looks like it can help with both of these.\r\n\r\nLots of interesting questions:\r\n\r\n* Should this be a new `datasette publish firebase` command or should it instead be implemented as additional custom options to `datasette publish cloudrun`?\r\n* How much harder does it become to do account setup?\r\n* How much will this option cost users?", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/491/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 450032134, "node_id": "MDU6SXNzdWU0NTAwMzIxMzQ=", "number": 495, "title": "facet_m2m gets confused by multiple relationships", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-05-29T21:37:28Z", "updated_at": "2020-12-17T05:08:22Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "I got this for a database I was playing with:\r\n\r\n\"hackathon__hacks_project__24_rows\"\r\n\r\nI think this is because of these three tables:\r\n\r\n\"hackathon__select___from_sqlite_master_\"\r\n\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/495/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 455486286, "node_id": "MDU6SXNzdWU0NTU0ODYyODY=", "number": 26, "title": "Mechanism for turning nested JSON into foreign keys / many-to-many", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 14, "created_at": "2019-06-13T00:52:06Z", "updated_at": "2022-06-29T23:35:29Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "The GitHub JSON APIs have a really interesting convention with respect to related objects.\r\n\r\nConsider https://api.github.com/repos/simonw/sqlite-utils/issues - here's a truncated subset:\r\n```json\r\n {\r\n \"id\": 449818897,\r\n \"node_id\": \"MDU6SXNzdWU0NDk4MTg4OTc=\",\r\n \"number\": 24,\r\n \"title\": \"Additional Column Constraints?\",\r\n \"user\": {\r\n \"login\": \"IgnoredAmbience\",\r\n \"id\": 98555,\r\n \"node_id\": \"MDQ6VXNlcjk4NTU1\",\r\n \"avatar_url\": \"https://avatars0.githubusercontent.com/u/98555?v=4\",\r\n \"gravatar_id\": \"\"\r\n },\r\n \"labels\": [\r\n {\r\n \"id\": 993377884,\r\n \"node_id\": \"MDU6TGFiZWw5OTMzNzc4ODQ=\",\r\n \"url\": \"https://api.github.com/repos/simonw/sqlite-utils/labels/enhancement\",\r\n \"name\": \"enhancement\",\r\n \"color\": \"a2eeef\",\r\n \"default\": true\r\n }\r\n ],\r\n \"state\": \"open\"\r\n }\r\n```\r\nThe `user` column lists a complete user. The `labels` column has a list of labels.\r\n\r\nSince both user and label have populated `id` field this is actually enough information for us to create records for them AND set up the corresponding foreign key (for user) and m2m relationships (for labels).\r\n\r\nIt would be really neat if `sqlite-utils` had some kind of mechanism for correctly processing these kind of patterns.\r\n\r\nThanks to `jq` there's not much need for extra customization of the shape here - if we support a narrowly defined structure users can use `jq` to reshape arbitrary JSON to match.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/26/reactions\", \"total_count\": 4, \"+1\": 4, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 455852801, "node_id": "MDU6SXNzdWU0NTU4NTI4MDE=", "number": 507, "title": "Every datasette plugin on the ecosystem page should have a screenshot", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2019-06-13T17:02:51Z", "updated_at": "2020-09-17T02:47:35Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "https://github.com/simonw/datasette/blob/master/docs/ecosystem.rst", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/507/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 456569067, "node_id": "MDU6SXNzdWU0NTY1NjkwNjc=", "number": 510, "title": "Ability to facet by delimiter (e.g. comma separated fields)", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": {"value": 9599, "label": "simonw"}, "milestone": null, "comments": 1, "created_at": "2019-06-15T19:34:41Z", "updated_at": "2019-07-08T15:44:51Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "E.g. if a field contains \"Tags,With,Commas\" be able to facet them in the same way as `_facet_array=` lets you facet `[\"Tags\", \"With\", \"Commas\"]`", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/510/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 456578474, "node_id": "MDU6SXNzdWU0NTY1Nzg0NzQ=", "number": 511, "title": "Get Datasette tests passing on Windows in GitHub Actions", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 13, "created_at": "2019-06-15T21:41:58Z", "updated_at": "2021-07-11T17:23:05Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "This should almost happen as a side-effect or moving from Sanic to Uvicorn during the port to ASGI: #272 \r\n\r\nAdditional steps:\r\n\r\n- test it manually\r\n- update documentation\r\n- set up some form of Windows CI\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/511/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 459469278, "node_id": "MDU6SXNzdWU0NTk0NjkyNzg=", "number": 515, "title": "Try shrinking official image with docker-slim", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-06-22T12:25:37Z", "updated_at": "2019-06-22T12:25:37Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "This looks really promising: https://github.com/docker-slim/docker-slim\r\n\r\nIf it can shave substantial size from our official container reliably we could add it to the automated build process.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/515/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 459509126, "node_id": "MDU6SXNzdWU0NTk1MDkxMjY=", "number": 516, "title": "Enforce import sort order with isort", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 8, "created_at": "2019-06-22T20:35:50Z", "updated_at": "2023-08-23T02:15:36Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "I want to use isort to order imports. A few steps here:\r\n\r\n- [x] Add a .isort.cfg file (see below)\r\n- [x] Use `isort -rc` to reformat existing code\r\n- [ ] Commit this change\r\n- [x] Add a unit test that ensures future changes remain isort compatible", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/516/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 459622390, "node_id": "MDU6SXNzdWU0NTk2MjIzOTA=", "number": 522, "title": "Handle case-insensitive headers in a nicer way", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2019-06-23T21:56:34Z", "updated_at": "2019-06-26T18:48:53Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Spun out from https://github.com/simonw/datasette/pull/518#discussion_r296486289", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/522/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 460095928, "node_id": "MDU6SXNzdWU0NjAwOTU5Mjg=", "number": 528, "title": "Establish a pattern for Datasette plugins built on top of Pandas", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-06-24T21:05:52Z", "updated_at": "2019-06-24T21:05:52Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "The Pandas ecosystem is huge, varied and full of tools that are really good at doing interesting analysis on top of tabular data.\r\n\r\nPandas should not be a dependency of Datasette core, but I think there is a lot of potential in having plugins which use Pandas to apply interesting analysis to data sucked out of Datasette's SQLite tables.\r\n\r\nOne example ([thanks, Tony](https://twitter.com/psychemedia/status/1143259809715752962)): https://github.com/ResidentMario/missingno could form the basis of a fantastic plugin for getting a high-level overview of how complete each column in a table is.\r\n\r\nSome thought is needed here about what shape these kind of plugins might take, and what plugin hooks they would use.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/528/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 462117311, "node_id": "MDU6SXNzdWU0NjIxMTczMTE=", "number": 531, "title": "/database/-/inspect", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2019-06-28T16:33:41Z", "updated_at": "2019-07-08T15:43:57Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Build `/database/-/inspect` which shows tables, columns, column types and foreign keys\r\n\r\nIt won't show table counts. Or maybe it will include them optionally but only for `-i` databases, in a special area of the JSON reserved for immutable-only inspect details.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/datasette/issues/465#issuecomment-506797086_", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/531/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 463492815, "node_id": "MDU6SXNzdWU0NjM0OTI4MTU=", "number": 534, "title": "500 error on m2m facet detection", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2019-07-03T00:42:42Z", "updated_at": "2020-12-17T05:08:22Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "This may help debug:\r\n```\r\ndiff --git a/datasette/facets.py b/datasette/facets.py\r\nindex 76d73e5..07a4034 100644\r\n--- a/datasette/facets.py\r\n+++ b/datasette/facets.py\r\n@@ -499,11 +499,14 @@ class ManyToManyFacet(Facet):\r\n \"outgoing\"\r\n ]\r\n if len(other_table_outgoing_foreign_keys) == 2:\r\n- destination_table = [\r\n- t\r\n- for t in other_table_outgoing_foreign_keys\r\n- if t[\"other_table\"] != self.table\r\n- ][0][\"other_table\"]\r\n+ try:\r\n+ destination_table = [\r\n+ t\r\n+ for t in other_table_outgoing_foreign_keys\r\n+ if t[\"other_table\"] != self.table\r\n+ ][0][\"other_table\"]\r\n+ except IndexError:\r\n+ import pdb; pdb.pm()\r\n # Only suggest if it's not selected already\r\n if (\"_facet_m2m\", destination_table) in args:\r\n continue\r\n```", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/534/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 463544206, "node_id": "MDU6SXNzdWU0NjM1NDQyMDY=", "number": 537, "title": "Populate \"endpoint\" key in ASGI scope", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 12, "created_at": "2019-07-03T04:54:47Z", "updated_at": "2019-07-22T06:03:18Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "This is a trick used by Starlette so that other layers of ASGI middleware can see which route was selected.\r\n\r\nThey added it here: https://github.com/encode/starlette/commit/34d0097feb6f057bd050d5057df5a2f96b97384e\r\n\r\nIf Datasette supports it as well we can benefit from it if we integrate this sentry_asgi middleware (probably as a `datasette-sentry` plugin): https://github.com/encode/sentry-asgi/blob/c6a42d44d31f85885b79e4ee898683ecf8104971/sentry_asgi/middleware.py#L34-L35", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/537/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 464987783, "node_id": "MDExOlB1bGxSZXF1ZXN0Mjk1MTI3MjEz", "number": 546, "title": "Facet by delimiter", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-07-07T20:06:05Z", "updated_at": "2019-11-18T23:46:01Z", "closed_at": null, "author_association": "OWNER", "pull_request": "simonw/datasette/pulls/546", "body": "Refs #510", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/546/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 465003070, "node_id": "MDU6SXNzdWU0NjUwMDMwNzA=", "number": 551, "title": "Ship many-to-many faceting support (and facet-by-delimiter)", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-07-07T23:11:45Z", "updated_at": "2019-07-08T15:45:23Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/551/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 465019882, "node_id": "MDU6SXNzdWU0NjUwMTk4ODI=", "number": 552, "title": "Add --plugin-secret support to \"datasette package\"", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2019-07-08T01:46:47Z", "updated_at": "2019-07-08T01:47:30Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Split out from #544.\r\n\r\nI think I should combine this with #347 (renaming `datasette package` to `datasette publish docker`).", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/552/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 465327844, "node_id": "MDU6SXNzdWU0NjUzMjc4NDQ=", "number": 553, "title": "Potential improvements to facet-by-date", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2019-07-08T15:37:53Z", "updated_at": "2019-07-08T15:41:55Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "In addition to #483 Tobias had some useful suggestions on Twitter:\r\n\r\nhttps://twitter.com/rixxtr/status/1148253926476701696\r\n> I think for date facets, it might be more meaningful to order them by date, rather than by size? Or offer both? I'm *definitely* often interested in size-over-time, so https://data.rixx.de/django_tickets/tickets?_facet_date=created#facet-created \u2026 isn't all that helpful!\r\n\r\nScreenshot of that link:\r\n\r\n\"django_tickets__tickets__29_846_rows\"\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/553/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 472115381, "node_id": "MDU6SXNzdWU0NzIxMTUzODE=", "number": 49, "title": "extracts= should support multiple-column extracts", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 10, "created_at": "2019-07-24T07:06:41Z", "updated_at": "2020-10-16T19:18:19Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Lookup tables can be constructed on compound columns, but the `extracts=` option doesn't currently support that.\r\n\r\nRight now extracts can be defined in two ways:\r\n```python\r\n# Extract these columns into tables with the same name:\r\ndogs = db.table(\"dogs\", extracts=[\"breed\", \"most_recent_trophy\"])\r\n\r\n# Same as above but with custom table names:\r\ndogs = db.table(\"dogs\", extracts={\"breed\": \"Breeds\", \"most_recent_trophy\": \"Trophies\"})\r\n```\r\nNeed some kind of syntax for much more complicated extractions, like when two columns (say \"source\" and \"source_version\") are extracted into a single table.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/49/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 473288428, "node_id": "MDExOlB1bGxSZXF1ZXN0MzAxNDgzNjEz", "number": 564, "title": "First proof-of-concept of Datasette Library", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2019-07-26T10:22:26Z", "updated_at": "2023-02-07T15:14:11Z", "closed_at": null, "author_association": "OWNER", "pull_request": "simonw/datasette/pulls/564", "body": "Refs #417. Run it like this:\r\n\r\n datasette -d ~/Library\r\n\r\nUses a new plugin hook - available_databases()\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/564/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 1, "state_reason": null} {"id": 481885279, "node_id": "MDU6SXNzdWU0ODE4ODUyNzk=", "number": 569, "title": "More advanced connection pooling", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2019-08-17T13:20:41Z", "updated_at": "2019-10-02T22:44:37Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "We need a much smarter way of handling database connections.\r\n\r\nToday, connections are simple: Datasette runs a number of threads (defaults to 3) and each thread gets a threadlocal read-only (or immutable) connection to each attached database - opened on demand.\r\n\r\nFor Datasette Library (#417) I want to support potentially hundreds of attached databases. Datasette Edit (#567) is going to introduce a need for writable connections too.\r\n\r\nI'd also like to be able to run joins across multiple databases (#283) which further complicates things.\r\n\r\nSupporting thousands of open SQLite connections at once feels like it won't provide good enough performance (though I should benchmark that to be sure). Some kind of connection pooling is likely to be necessary.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/569/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 501773982, "node_id": "MDExOlB1bGxSZXF1ZXN0MzIzOTgzNzMy", "number": 579, "title": "New connection pooling", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2019-10-02T23:22:19Z", "updated_at": "2019-11-15T22:57:21Z", "closed_at": null, "author_association": "OWNER", "pull_request": "simonw/datasette/pulls/579", "body": "See #569", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/579/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 503053243, "node_id": "MDU6SXNzdWU1MDMwNTMyNDM=", "number": 582, "title": "Datasette should not completely crash if one SQLite database is malformed", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-10-06T05:11:43Z", "updated_at": "2019-10-06T05:11:43Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "If you run Datasette against a number of database files and one of them is malformed, you get this 500 error on the index page:\r\n\r\n\"Error_500\"\r\n\r\nIt would be better if Datasette still worked and listed the databases that were NOT malformed, then showed an inline error message just for the one that could not be accessed.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/582/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 507454958, "node_id": "MDU6SXNzdWU1MDc0NTQ5NTg=", "number": 596, "title": "Handle really wide tables better", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2019-10-15T20:05:46Z", "updated_at": "2022-09-07T00:58:41Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "If a table has hundreds of columns the Datasette UI starts getting unwieldy.\r\n\r\nAddressing this would be neat. One option would be to only select the first 30 columns by default and provide a UI for selecting more.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/596/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 516874735, "node_id": "MDU6SXNzdWU1MTY4NzQ3MzU=", "number": 613, "title": "Basic join support for table view", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2019-11-03T19:12:53Z", "updated_at": "2019-11-03T19:14:01Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "I think it would be possible to support basic foreign key joins on the table page.\r\n\r\nThe user could specify columns that should result in a join (from a set of suggestions similar to how facets work right now) and they could then be passed as `?_join=city_id` arguments.\r\n\r\nThis feature will make a lot of sense when combined with the ability to show / hide / customize columns, see #292", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/613/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 520667773, "node_id": "MDU6SXNzdWU1MjA2Njc3NzM=", "number": 620, "title": "Mechanism for indicating foreign key relationships in the table and query page URLs", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2019-11-10T22:26:27Z", "updated_at": "2021-04-05T03:57:22Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Datasette currently only inflates foreign keys (into names hyperlinks) if it detects them as foreign key constraints in the underlying database.\r\n\r\nIt would be useful if you could specify additional \"foreign keys\" using both `metadata.json` and the querystring - similar time how you can pass `?_fts_table=x` https://datasette.readthedocs.io/en/stable/full_text_search.html#configuring-full-text-search-for-a-table-or-view", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/620/reactions\", \"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 1}", "draft": null, "state_reason": null} {"id": 520681725, "node_id": "MDU6SXNzdWU1MjA2ODE3MjU=", "number": 621, "title": "Syntax for ?_through= that works as a form field", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2019-11-11T00:19:03Z", "updated_at": "2021-12-18T01:42:33Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "The current syntax for `?_through=` uses JSON to avoid any risk of confusion with table or column names that contain special characters.\r\n\r\nThis means you can't target a form field at it.\r\n\r\nWe should be able to support both - `?x.y.z=value` for tables and columns with \"regular\" names, falling back to the current JSON syntax for columns or tables that won't work with the key/value syntax.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/621/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 530468212, "node_id": "MDU6SXNzdWU1MzA0NjgyMTI=", "number": 643, "title": "Set up some basic benchmarks as part of the unit tests", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-11-29T19:24:19Z", "updated_at": "2019-11-29T19:24:19Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "https://pypi.org/project/pytest-benchmark/ looks great for this.\r\n\r\nHere's how to run it as a github action: https://github.com/rhysd/github-action-benchmark/blob/master/examples/pytest/README.md", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/643/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 534629631, "node_id": "MDU6SXNzdWU1MzQ2Mjk2MzE=", "number": 650, "title": "Add a glossary to the documentation", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2019-12-09T00:23:45Z", "updated_at": "2022-01-13T22:04:56Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Call it `glossary.rst` - it can use a definition list something like this:\r\n```rst\r\n.. _glossary:\r\n\r\nGlossary\r\n========\r\n\r\nTerm\r\n A definition of the term.\r\n\r\nAnother term\r\n Another definition.\r\n```", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/650/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 559964149, "node_id": "MDU6SXNzdWU1NTk5NjQxNDk=", "number": 665, "title": "Introduce a SQL statement parser in Python", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-02-04T20:36:05Z", "updated_at": "2020-02-04T20:36:48Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "#254 and #653 are both examples of problems that could be solved using a real SQL parser in Python.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/665/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 564833696, "node_id": "MDU6SXNzdWU1NjQ4MzM2OTY=", "number": 670, "title": "Prototoype for Datasette on PostgreSQL", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 15, "created_at": "2020-02-13T17:17:55Z", "updated_at": "2023-11-17T15:32:21Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "I thought this would never happen, but now that I'm deep in the weeds of running SQLite in production for Datasette Cloud I'm starting to reconsider my policy of only supporting SQLite.\r\n\r\nSome of the factors making me think PostgreSQL support could be worth the effort:\r\n- Serverless. I'm getting increasingly excited about writable-database use-cases for Datasette. If it could talk to PostgreSQL then users could easily deploy it on Heroku or other serverless providers that can talk to a managed RDS-style PostgreSQL.\r\n- Existing databases. Plenty of organizations have PostgreSQL databases. They can export to SQLite using [db-to-sqlite](https://github.com/simonw/db-to-sqlite) but that's a pretty big barrier to getting started - being able to run `datasette postgresql://connection-string` and start trying it out would be a massively better experience.\r\n- Data size. I keep running into use-cases where I want to run Datasette against many GBs of data. SQLite can do this but PostgreSQL is much more optimized for large data, especially given the existence of tools like Citus.\r\n- Marketing. Convincing people to trust their data to SQLite is potentially a big barrier to adoption. Even if I've convinced myself it's trustworthy I still have to convince everyone else.\r\n- It might not be that hard? If this required a ground-up rewrite it wouldn't be worth the effort, but I have a hunch that it may not be too hard - most of the SQL in Datasette should work on both databases since it's almost all portable SELECT statements. If Datasette did DML this would be a lot harder, but it doesn't.\r\n- Plugins! This feels like a natural surface for a plugin - at which point people could add MySQL support and suchlike in the future.\r\n\r\nThe above reasons feel strong enough to justify a prototype.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/670/reactions\", \"total_count\": 19, \"+1\": 14, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 5, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 565064079, "node_id": "MDExOlB1bGxSZXF1ZXN0Mzc1MTgwODMy", "number": 672, "title": "--dirs option for scanning directories for SQLite databases", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 15, "created_at": "2020-02-14T02:25:52Z", "updated_at": "2020-03-27T01:03:53Z", "closed_at": null, "author_association": "OWNER", "pull_request": "simonw/datasette/pulls/672", "body": "Refs #417.", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/672/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 573578548, "node_id": "MDU6SXNzdWU1NzM1Nzg1NDg=", "number": 89, "title": "Ability to customize columns used by extracts= feature", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-03-01T16:54:48Z", "updated_at": "2020-10-16T19:17:50Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "@simonw any thoughts on allow extracts to specify the lookup column name? If I'm understanding the documentation right, `.lookup()` allows you to define the \"value\" column (the documentation uses name), but when you use `extracts` keyword as part of `.insert()`, `.upsert()` etc. the lookup must be done against a column named \"value\". I have an existing lookup table that I've populated with columns \"id\" and \"name\" as opposed to \"id\" and \"value\", and seems I can't use `extracts=`, unless I'm missing something...\r\n\r\nInitial thought on how to do this would be to allow the dictionary value to be a tuple of table name column pair... so:\r\n```\r\ntable = db.table(\"trees\", extracts={\"species_id\": (\"Species\", \"name\"})\r\n```\r\n\r\nI haven't dug too much into the existing code yet, but does this make sense? Worth doing?\r\n\r\n_Originally posted by @chrishas35 in https://github.com/simonw/sqlite-utils/issues/46#issuecomment-592999503_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/89/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 574021194, "node_id": "MDU6SXNzdWU1NzQwMjExOTQ=", "number": 691, "title": "--reload sould reload server if code in --plugins-dir changes", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-03-02T14:42:21Z", "updated_at": "2020-06-14T02:35:17Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/691/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 574035432, "node_id": "MDU6SXNzdWU1NzQwMzU0MzI=", "number": 692, "title": "is_hidden_table context variable on table.html page", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-03-02T15:03:25Z", "updated_at": "2020-03-02T15:03:48Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "It's useful to know if a table is hidden when rendering that page. `datasette-configure-fts` for example may want to disallow enabling search on hidden tables.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/692/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 581795570, "node_id": "MDU6SXNzdWU1ODE3OTU1NzA=", "number": 93, "title": "Support more string values for types in .add_column()", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-03-15T19:32:49Z", "updated_at": "2020-09-24T20:36:46Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "https://sqlite-utils.readthedocs.io/en/2.4.2/python-api.html#adding-columns says:\r\n> SQLite types you can specify are \"TEXT\", \"INTEGER\", \"FLOAT\" or \"BLOB\".\r\n\r\nAs discovered in #92 this isn't the right list of values. I should expand this to match https://www.sqlite.org/datatype3.html", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/93/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 593006814, "node_id": "MDU6SXNzdWU1OTMwMDY4MTQ=", "number": 715, "title": "Refactor duplicate cell display logic", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-04-03T00:58:11Z", "updated_at": "2020-04-03T00:58:11Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "The logic for rendering cells in table view and in database (or canned query) view is currently very similar:\r\n\r\nhttps://github.com/simonw/datasette/blob/7656fd64d8b6a32ebc34d89c1b8711cc5ea240f7/datasette/views/base.py#L514-L539\r\n\r\nCompared with:\r\n\r\nhttps://github.com/simonw/datasette/blob/7656fd64d8b6a32ebc34d89c1b8711cc5ea240f7/datasette/views/table.py#L104-L195\r\n\r\nI'll be changing this a bit in #698 but I should still try to clean this up more further in the future.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/715/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 594237015, "node_id": "MDU6SXNzdWU1OTQyMzcwMTU=", "number": 718, "title": "Plugin idea: datasette-redirects", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-04-05T03:41:38Z", "updated_at": "2023-08-30T22:17:31Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "I just had to write a one-off custom plugin to redirect niche-musems.com to www.niche-museums.com (https://github.com/simonw/museums/issues/21) - it would be great if this kind of thing could be handled by a configurable plugin.\r\n\r\nhttps://github.com/simonw/museums/blob/6b1faf00c463b2228860d4d62d104b11935e01b1/plugins/redirect_www.py", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/718/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "reopened"} {"id": 607223136, "node_id": "MDU6SXNzdWU2MDcyMjMxMzY=", "number": 741, "title": "Replace \"datasette publish --extra-options\" with \"--setting\"", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 9, "created_at": "2020-04-27T04:29:04Z", "updated_at": "2022-05-12T19:21:16Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "See https://github.com/simonw/datasette-publish-now/issues/9#issuecomment-618155764 - the `--extra-options` mechanism is in practice just used to set `--config` options in data that you publish, but that means you end up with pretty messy looking commands:\r\n\r\n datasette publish my.db --extra-options=\"--config default_page_size:50 --config sql_time_limit_ms:3500\"\r\n\r\nA neater design would be to support `--config` as an option for `datasette publish` directly:\r\n\r\n datasette publish my.db --config default_page_size:50 --config sql_time_limit_ms:3500\r\n\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/741/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 613422636, "node_id": "MDU6SXNzdWU2MTM0MjI2MzY=", "number": 760, "title": "Way of seeing full schema for a database", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-05-06T15:46:08Z", "updated_at": "2020-05-06T23:49:06Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "I find myself wanting to quickly figure out all of the BLOB columns in a database.\r\n\r\nA `/-/schema` page showing the full schema (actually since it's per-database probably `/dbname/-/schema` or `/-/schema/dbname`) would be really handy.\r\n\r\nIt would need to be carefully constructed from various queries against `sqlite_master` - just doing `select * from sqlite_master where type='table'` isn't quite enough because I also want to show indexes, triggers etc.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/760/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 613491342, "node_id": "MDU6SXNzdWU2MTM0OTEzNDI=", "number": 762, "title": "Experiment with PRAGMA hard_heap_limit ", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-05-06T17:33:23Z", "updated_at": "2020-05-07T03:08:44Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "This was added in SQLite 2020-01-22 (3.31.0): https://www.sqlite.org/changes.html#version_3_31_0\r\n\r\n> Add the [sqlite3_hard_heap_limit64()](https://www.sqlite.org/c3ref/hard_heap_limit64.html) interface and the corresponding [PRAGMA hard_heap_limit](https://www.sqlite.org/pragma.html#pragma_hard_heap_limit) command. \r\n\r\nThis sounds like it could be a nice extra safety measure.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/762/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 616087149, "node_id": "MDU6SXNzdWU2MTYwODcxNDk=", "number": 765, "title": "publish heroku should default to currently tagged version", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-05-11T18:24:06Z", "updated_at": "2020-05-11T18:25:43Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Had a report that deploying to Heroku was using the previously installed version of Datasette, not the latest.\r\n\r\nCould be because of this:\r\n\r\nhttps://github.com/simonw/datasette/blob/af6c6c5d6f929f951c0e63bfd1c82e37a071b50f/datasette/publish/heroku.py#L172-L179\r\n\r\nHeroku documentation recommends pinning to specific versions https://devcenter.heroku.com/articles/python-pip\r\n\r\nSo... we could ensure we default to an install value of `[\"datasette>=current_tag\"]`.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/765/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 626211658, "node_id": "MDU6SXNzdWU2MjYyMTE2NTg=", "number": 778, "title": "Ability to configure keyset pagination for views and queries", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-05-28T04:48:56Z", "updated_at": "2020-10-02T02:26:25Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Currently views offer pagination, but it uses offset/limit - e.g. https://latest.datasette.io/fixtures/paginated_view?_next=100\r\n\r\nThis means pagination will perform poorly on deeper pages.\r\n\r\nIf a view is based on a table that has a primary key it should be possible to configure efficient keyset pagination that works the same way that table pagination works.\r\n\r\nThis may be as simple as configuring a column that can be treated as a \"primary key\" for the purpose of pagination using `metadata.json` - or with a `?_view_pk=colname` querystring argument.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/778/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 626582657, "node_id": "MDU6SXNzdWU2MjY1ODI2NTc=", "number": 779, "title": "Make human_description_en explicitly available to output renderers", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-05-28T14:59:54Z", "updated_at": "2020-05-28T14:59:54Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "`datasette-atom` uses this:\r\n\r\nhttps://github.com/simonw/datasette-atom/blob/df98a6c43a443224b6cd232f84703ec297ef046b/datasette_atom/__init__.py#L36-L37\r\n```python\r\n if data.get(\"human_description_en\"):\r\n title += \": \" + data[\"human_description_en\"]\r\n```\r\nIt's a nice way to generate a useful title for a filtered table.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/779/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 626593402, "node_id": "MDU6SXNzdWU2MjY1OTM0MDI=", "number": 780, "title": "Internals documentation for datasette.metadata() method", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 2, "created_at": "2020-05-28T15:14:22Z", "updated_at": "2022-03-15T20:50:34Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "https://github.com/simonw/datasette/blob/40885ef24e32d91502b6b8bbad1c7376f50f2830/datasette/app.py#L297-L328", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/780/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 628156527, "node_id": "MDU6SXNzdWU2MjgxNTY1Mjc=", "number": 789, "title": "Mechanism for enabling pluggy tracing", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-06-01T05:10:14Z", "updated_at": "2020-06-01T05:11:03Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Could be useful for debugging plugins: https://pluggy.readthedocs.io/en/latest/#call-tracing\r\n\r\nI tried this out by adding these two lines in `plugins.py`:\r\n```python\r\npm = pluggy.PluginManager(\"datasette\")\r\npm.add_hookspecs(hookspecs)\r\n# Added these:\r\npm.trace.root.setwriter(print)\r\npm.enable_tracing()\r\n```\r\nOutput looked something like this:\r\n```\r\nINFO: 127.0.0.1:52724 - \"GET /-/-/static/app.css HTTP/1.1\" 404 Not Found\r\n actor_from_request [hook]\r\n datasette: \r\n request: \r\n\r\n finish actor_from_request --> [] [hook]\r\n\r\n extra_body_script [hook]\r\n template: show_json.html\r\n database: None\r\n table: None\r\n view_name: json_data\r\n datasette: \r\n\r\n finish extra_body_script --> [] [hook]\r\n\r\n extra_template_vars [hook]\r\n template: show_json.html\r\n database: None\r\n table: None\r\n view_name: json_data\r\n request: \r\n datasette: \r\n\r\n finish extra_template_vars --> [] [hook]\r\n\r\n extra_css_urls [hook]\r\n template: show_json.html\r\n database: None\r\n table: None\r\n datasette: \r\n\r\n finish extra_css_urls --> [] [hook]\r\n\r\n extra_js_urls [hook]\r\n template: show_json.html\r\n database: None\r\n table: None\r\n datasette: \r\n\r\n finish extra_js_urls --> [] [hook]\r\n\r\nINFO: 127.0.0.1:52724 - \"GET /-/actor HTTP/1.1\" 200 OK\r\n actor_from_request [hook]\r\n datasette: \r\n request: \r\n\r\n finish actor_from_request --> [] [hook]\r\n```", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/789/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 628572716, "node_id": "MDU6SXNzdWU2Mjg1NzI3MTY=", "number": 791, "title": "Tutorial: building a something-interesting with writable canned queries", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-06-01T16:32:05Z", "updated_at": "2020-10-10T23:34:42Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Initial idea: TODO list, as a tutorial for #698 writable canned queries.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/791/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 632724154, "node_id": "MDU6SXNzdWU2MzI3MjQxNTQ=", "number": 805, "title": "Writable canned queries live demo on Glitch", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 11, "created_at": "2020-06-06T20:52:13Z", "updated_at": "2020-07-01T22:44:01Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Needs to run somewhere with a mutable disk drive, so not Cloud Run or Heroku or Vercel.\r\n\r\nI think I'll put it on Glitch.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/805/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 634663505, "node_id": "MDU6SXNzdWU2MzQ2NjM1MDU=", "number": 815, "title": "Group permission checks by request on /-/permissions debug page", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 8, "created_at": "2020-06-08T14:25:23Z", "updated_at": "2020-12-17T22:06:48Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Now that we're making a LOT more permission checks (on the DB index page we do a check for every listed table for example) the `/-/permissions` page gets filled up pretty quickly.\r\n\r\nCan make this more readable by grouping permission checks by request. Have most recent request at the top of the page but the permission requests within that page sorted chronologically by most recent last.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/815/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 636511683, "node_id": "MDU6SXNzdWU2MzY1MTE2ODM=", "number": 830, "title": "Redesign register_facet_classes plugin hook", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 3, "created_at": "2020-06-10T20:03:27Z", "updated_at": "2021-12-16T19:58:22Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Nothing uses this plugin hook yet, so the design is not yet proven.\r\n\r\nI'm going to build a real plugin against it and use that process to inform any design changes that may need to be made.\r\n\r\nI'll add a warning about this to the documentation.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/830/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 638238548, "node_id": "MDU6SXNzdWU2MzgyMzg1NDg=", "number": 845, "title": "Code coverage should ignore files in .coveragerc", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-06-13T21:45:42Z", "updated_at": "2020-06-13T21:46:03Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "I'm not sure why this is, but the code coverage I have running in a GitHub Action doesn't take my `.coveragerc` file into account. It should:\r\n\r\nhttps://github.com/simonw/datasette/blob/cf7a2bdb404734910ec07abc7571351a2d934828/.github/workflows/test-coverage.yml#L31-L35\r\n\r\nHere's the bit that's ignored:\r\n\r\nhttps://github.com/simonw/datasette/blob/cf7a2bdb404734910ec07abc7571351a2d934828/.coveragerc#L1-L2\r\n\r\nAs a result my coverage score is 84%, when it should be 92%:\r\n```\r\n2020-06-13T21:41:18.4404252Z ----------- coverage: platform linux, python 3.8.3-final-0 -----------\r\n2020-06-13T21:41:18.4404570Z Name Stmts Miss Cover\r\n2020-06-13T21:41:18.4404971Z --------------------------------------------------------\r\n2020-06-13T21:41:18.4405227Z datasette/__init__.py 3 0 100%\r\n2020-06-13T21:41:18.4405441Z datasette/__main__.py 3 3 0%\r\n2020-06-13T21:41:18.4405668Z datasette/_version.py 279 279 0%\r\n2020-06-13T21:41:18.4405921Z datasette/actor_auth_cookie.py 20 0 100%\r\n2020-06-13T21:41:18.4406135Z datasette/app.py 499 27 95%\r\n2020-06-13T21:41:18.4406343Z datasette/cli.py 162 45 72%\r\n2020-06-13T21:41:18.4406553Z datasette/database.py 236 17 93%\r\n2020-06-13T21:41:18.4406761Z datasette/default_permissions.py 40 0 100%\r\n2020-06-13T21:41:18.4406975Z datasette/facets.py 210 24 89%\r\n2020-06-13T21:41:18.4407186Z datasette/filters.py 122 7 94%\r\n2020-06-13T21:41:18.4407394Z datasette/hookspecs.py 34 0 100%\r\n2020-06-13T21:41:18.4407600Z datasette/inspect.py 36 23 36%\r\n2020-06-13T21:41:18.4407807Z datasette/plugins.py 34 6 82%\r\n2020-06-13T21:41:18.4408014Z datasette/publish/__init__.py 0 0 100%\r\n2020-06-13T21:41:18.4408240Z datasette/publish/cloudrun.py 57 2 96%\r\n2020-06-13T21:41:18.4408786Z datasette/publish/common.py 19 1 95%\r\n2020-06-13T21:41:18.4409029Z datasette/publish/heroku.py 97 13 87%\r\n2020-06-13T21:41:18.4409243Z datasette/renderer.py 63 4 94%\r\n2020-06-13T21:41:18.4409450Z datasette/sql_functions.py 5 0 100%\r\n2020-06-13T21:41:18.4410480Z datasette/tracer.py 87 16 82%\r\n2020-06-13T21:41:18.4410972Z datasette/utils/__init__.py 504 31 94%\r\n2020-06-13T21:41:18.4411755Z datasette/utils/asgi.py 264 24 91%\r\n2020-06-13T21:41:18.4412173Z datasette/utils/shutil_backport.py 44 44 0%\r\n2020-06-13T21:41:18.4412822Z datasette/version.py 4 0 100%\r\n2020-06-13T21:41:18.4413562Z datasette/views/__init__.py 0 0 100%\r\n2020-06-13T21:41:18.4414276Z datasette/views/base.py 288 19 93%\r\n2020-06-13T21:41:18.4414579Z datasette/views/database.py 120 2 98%\r\n2020-06-13T21:41:18.4414860Z datasette/views/index.py 57 2 96%\r\n2020-06-13T21:41:18.4415379Z datasette/views/special.py 72 16 78%\r\n2020-06-13T21:41:18.4418994Z datasette/views/table.py 418 18 96%\r\n2020-06-13T21:41:18.4428811Z --------------------------------------------------------\r\n2020-06-13T21:41:18.4430394Z TOTAL 3777 623 84%\r\n```", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/845/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 639993467, "node_id": "MDU6SXNzdWU2Mzk5OTM0Njc=", "number": 850, "title": "Proof of concept for Datasette on AWS Lambda with EFS", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 25, "created_at": "2020-06-16T21:48:31Z", "updated_at": "2020-06-16T23:52:16Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "https://aws.amazon.com/about-aws/whats-new/2020/06/aws-lambda-support-for-amazon-elastic-file-system-now-generally-/\r\n\r\nIf Datasette can run on Lambda with access to EFS it could both read AND write large databases there.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/850/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 642296989, "node_id": "MDU6SXNzdWU2NDIyOTY5ODk=", "number": 856, "title": "Consider pagination of canned queries", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-06-20T03:15:59Z", "updated_at": "2021-05-21T14:22:41Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "The new `canned_queries()` plugin hook from #852 combined with plugins like https://github.com/simonw/datasette-saved-queries could mean that some installations end up with hundreds or even thousands of canned queries. I should consider pagination or some other way of ensuring that this doesn't cause performance problems for Datasette.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/856/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 643510821, "node_id": "MDU6SXNzdWU2NDM1MTA4MjE=", "number": 862, "title": "Set an upper limit on total facet suggestion time for a page", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-06-23T03:57:55Z", "updated_at": "2020-06-23T03:58:48Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "If a table has 100 columns the facet suggestion code will currently run 100 times, taking a max of `facet_suggest_time_limit_ms` which defaults to 50ms per column:\r\n\r\nhttps://github.com/simonw/datasette/blob/000528192eaf891118932250141dabe7a1561ece/datasette/facets.py#L142-L162\r\n\r\nSo for 100 columns, that's 100 * 50ms = 5s total time that might be spent attempting to calculate facets on a large table!\r\n\r\nI should implement a hard upper limit on the total amount of time taken suggesting facets - probably of around 500ms. If it takes longer than that the remaining columns will not be considered.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/862/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 644161221, "node_id": "MDU6SXNzdWU2NDQxNjEyMjE=", "number": 117, "title": "Support for compound (composite) foreign keys", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-06-23T21:33:42Z", "updated_at": "2020-06-23T21:40:31Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "It turns out SQLite supports composite foreign keys: https://www.sqlite.org/foreignkeys.html#fk_composite\r\n\r\nTheir example looks like this:\r\n```sql\r\nCREATE TABLE album(\r\n albumartist TEXT,\r\n albumname TEXT,\r\n albumcover BINARY,\r\n PRIMARY KEY(albumartist, albumname)\r\n);\r\n\r\nCREATE TABLE song(\r\n songid INTEGER,\r\n songartist TEXT,\r\n songalbum TEXT,\r\n songname TEXT,\r\n FOREIGN KEY(songartist, songalbum) REFERENCES album(albumartist, albumname)\r\n);\r\n```\r\n\r\nHere's what that looks like in sqlite-utils:\r\n\r\n```\r\nIn [1]: import sqlite_utils \r\n\r\nIn [2]: import sqlite3 \r\n\r\nIn [3]: conn = sqlite3.connect(\":memory:\") \r\n\r\nIn [4]: conn \r\nOut[4]: \r\n\r\nIn [5]: conn.executescript(\"\"\" \r\n ...: CREATE TABLE album( \r\n ...: albumartist TEXT, \r\n ...: albumname TEXT, \r\n ...: albumcover BINARY, \r\n ...: PRIMARY KEY(albumartist, albumname) \r\n ...: ); \r\n ...: \r\n ...: CREATE TABLE song( \r\n ...: songid INTEGER, \r\n ...: songartist TEXT, \r\n ...: songalbum TEXT, \r\n ...: songname TEXT, \r\n ...: FOREIGN KEY(songartist, songalbum) REFERENCES album(albumartist, albumname) \r\n ...: ); \r\n ...: \"\"\") \r\nOut[5]: \r\n\r\nIn [6]: db = sqlite_utils.Database(conn) \r\n\r\nIn [7]: db.tables \r\nOut[7]: \r\n[,\r\n
]\r\n\r\nIn [8]: db.tables[0].foreign_keys \r\nOut[8]: []\r\n\r\nIn [9]: db.tables[1].foreign_keys \r\nOut[9]: \r\n[ForeignKey(table='song', column='songartist', other_table='album', other_column='albumartist'),\r\n ForeignKey(table='song', column='songalbum', other_table='album', other_column='albumname')]\r\n```\r\nThe table appears to have two separate foreign keys, when actually it has a single compound composite foreign key.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/117/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 646737558, "node_id": "MDU6SXNzdWU2NDY3Mzc1NTg=", "number": 870, "title": "Refactor default views to use register_routes", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 10, "created_at": "2020-06-27T18:53:12Z", "updated_at": "2022-03-15T20:07:18Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "It would be much cleaner if Datasette's default views were all registered using the new `register_routes()` plugin hook. Could dramatically reduce the code in `datasette/app.py`.\r\n\r\n> The ideal fix here would be to rework my `BaseView` subclass mechanism to work with `register_routes()` so that those views don't have any special privileges above plugin-provided views.\r\n_Originally posted by @simonw in https://github.com/simonw/datasette/issues/864#issuecomment-648580556_", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/870/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 647095487, "node_id": "MDU6SXNzdWU2NDcwOTU0ODc=", "number": 873, "title": "\"datasette -p 0 --root\" gives the wrong URL", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 14, "created_at": "2020-06-29T04:03:06Z", "updated_at": "2020-08-18T17:26:10Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "```\r\n$ datasette -p 0 --root\r\nhttp://127.0.0.1:0/-/auth-token?token=2d498c...\r\n```\r\nThe port is incorrect.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/873/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 648435885, "node_id": "MDU6SXNzdWU2NDg0MzU4ODU=", "number": 878, "title": "New pattern for views that return either JSON or HTML, available for plugins", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 26, "created_at": "2020-06-30T19:26:13Z", "updated_at": "2022-03-19T16:19:30Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Can be part of #870 - refactoring existing views to use `register_routes()`.\r\n\r\n> I'm going to put the new `check_permissions()` method on `BaseView` as well. If I want that method to be available to plugins I can do so by turning that `BaseView` class into a documented API that plugins are encouraged to use themselves.\r\n_Originally posted by @simonw in https://github.com/simonw/datasette/issues/832#issuecomment-651995453_", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/878/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 648659536, "node_id": "MDU6SXNzdWU2NDg2NTk1MzY=", "number": 881, "title": "Figure out why restore_working_directory is needed in some places", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-07-01T04:19:25Z", "updated_at": "2020-07-01T04:19:25Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "This is a frustrating workaround. I have a `restore_working_directory` fixture that I wrote to solve errors that look like this:\r\n```\r\n/Users/simon/Dropbox/Development/datasette/tests/test_publish_cloudrun.py:148: \r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _\r\n/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py:112: in __enter__\r\n return next(self.gen)\r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _\r\n\r\nself = \r\n\r\n @contextlib.contextmanager\r\n def isolated_filesystem(self):\r\n \"\"\"A context manager that creates a temporary folder and changes\r\n the current working directory to it for isolated filesystem tests.\r\n \"\"\"\r\n> cwd = os.getcwd()\r\nE FileNotFoundError: [Errno 2] No such file or directory\r\n```\r\nHere's an example of it in use: removing the `restore_working_directory` argument from this function causes the failure. https://github.com/simonw/datasette/blob/549b1c2063db48c4622ee5c7b478a1e3cbc1ac07/tests/test_plugins.py#L689-L690\r\n\r\nI'd like to not have to do this.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/881/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 649429772, "node_id": "MDU6SXNzdWU2NDk0Mjk3NzI=", "number": 886, "title": "Reconsider how _actor_X magic parameter deals with missing values", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-07-02T00:00:38Z", "updated_at": "2020-09-11T21:35:26Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "I had to build a custom `_actorornull` prefix for [datasette-saved-queries](https://github.com/simonw/datasette-saved-queries/blob/37c00e56ac398e1f9aa342d30357de013a9b37b4/datasette_saved_queries/__init__.py):\r\n```python\r\ndef actorornull(key, request):\r\n if request.actor is None:\r\n return None\r\n return request.actor.get(key)\r\n\r\n\r\n@hookimpl\r\ndef register_magic_parameters():\r\n return [\r\n (\"actorornull\", actorornull),\r\n ]\r\n```\r\nMaybe the `actor` magic in Datasette core should do that out of the box?\r\n\r\nhttps://github.com/simonw/datasette/blob/f1f581b7ffcd5d8f3ae6c1c654d813a6641410eb/datasette/default_magic_parameters.py#L14-L17\r\n\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/886/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 652961907, "node_id": "MDU6SXNzdWU2NTI5NjE5MDc=", "number": 121, "title": "Improved (and better documented) support for transactions", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-07-08T04:56:51Z", "updated_at": "2020-09-24T20:36:46Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655283393_\r\n\r\nWe should put some thought into how this library supports and encourages smart use of transactions.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/121/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 657572753, "node_id": "MDU6SXNzdWU2NTc1NzI3NTM=", "number": 894, "title": "?sort=colname~numeric to sort by by column cast to real", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 21, "created_at": "2020-07-15T18:47:48Z", "updated_at": "2021-08-20T02:07:53Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "If a text column actually contains numbers, being able to \"sort by column, treated as numeric\" would be really useful.\r\n\r\nProbably depends on column actions enabled by #690", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/894/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 659873662, "node_id": "MDU6SXNzdWU2NTk4NzM2NjI=", "number": 898, "title": "datasette.utils.testing module", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-07-18T03:53:24Z", "updated_at": "2020-07-18T03:57:46Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "The unit tests for plugins could benefit from reusing code from Datasette's own testing fixtures, e.g.:\r\n> I may need to borrow this function from Datasette for the tests:\r\n> https://github.com/simonw/datasette/blob/1f6a134369e6a7efaae9db469f15b1dd2b7f3709/tests/fixtures.py#L836-L851\r\n> \r\n> It's not importable (it lives in `fixtures.py` and not in the `datasette` package that gets packaged for PyPI) - maybe I should fix that in Datasette by adding a `from datasette.utils.testing` module.\r\n_Originally posted by @simonw in https://github.com/simonw/datasette-update-api/issues/4#issuecomment-660419182_", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/898/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 668064026, "node_id": "MDU6SXNzdWU2NjgwNjQwMjY=", "number": 911, "title": "Rethink the --name option to \"datasette publish\"", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 0, "created_at": "2020-07-29T18:49:49Z", "updated_at": "2020-07-29T18:49:49Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "`--name` works inconsistently across the different publish providers - on Cloud Run you should use `--service` instead for example. Need to review it across all of them and either remove it or clarify what it does.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/911/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 670209331, "node_id": "MDU6SXNzdWU2NzAyMDkzMzE=", "number": 913, "title": "Mechanism for passing additional options to `datasette my.db` that affect plugins", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2020-07-31T20:38:26Z", "updated_at": "2021-01-04T20:04:11Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "> It's a shame there's no obvious mechanism for passing additional options to `datasette my.db` that affect how plugins work.\r\n>\r\n>The only way I can think of at the moment is via environment variables:\r\n>\r\n> DATASETTE_INSERT_UNSAFE=1 datasette my.db\r\n>\r\n>This will have to do for the moment - it's ugly enough that people will at least know they are doing something unsafe, which is the goal here.\r\n_Originally posted by @simonw in https://github.com/simonw/datasette-insert/issues/15#issuecomment-667346438_", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/913/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null}