{"html_url": "https://github.com/simonw/datasette/issues/1670#issuecomment-1076652046", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1670", "id": 1076652046, "node_id": "IC_kwDOBm6k_c5ALGgO", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-23T18:02:30Z", "updated_at": "2022-03-23T18:02:30Z", "author_association": "OWNER", "body": "Two new things to add to the release notes from https://github.com/simonw/datasette/compare/0.61a0...main\r\n- https://github.com/simonw/datasette/issues/1678\r\n- https://github.com/simonw/datasette/issues/1675 (now also a documented API)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174423568, "label": "Ship Datasette 0.61"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1670#issuecomment-1076647495", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1670", "id": 1076647495, "node_id": "IC_kwDOBm6k_c5ALFZH", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-23T17:58:16Z", "updated_at": "2022-03-23T17:58:16Z", "author_association": "OWNER", "body": "I think the release notes are fine, but they need an opening paragraph highlighting the changes that are most likely to break backwards compatibility.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174423568, "label": "Ship Datasette 0.61"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1574#issuecomment-1076645636", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1574", "id": 1076645636, "node_id": "IC_kwDOBm6k_c5ALE8E", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-23T17:56:35Z", "updated_at": "2022-03-23T17:56:35Z", "author_association": "OWNER", "body": "I'd actually like to switch to slim as the default - I think Datasette should ship the smallest possible container that can still support extra packages being installed using `apt-get install`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1084193403, "label": "introduce new option for datasette package to use a slim base image"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1665#issuecomment-1076644362", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1665", "id": 1076644362, "node_id": "IC_kwDOBm6k_c5ALEoK", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-23T17:55:39Z", "updated_at": "2022-03-23T17:55:39Z", "author_association": "OWNER", "body": "Thanks for the PR - I spotted an error about this and went through and fixed this in all of my repos the other day: https://github.com/search?o=desc&q=user%3Asimonw+google-github-actions%2Fsetup-gcloud%40v0&s=committer-date&type=Commits", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1173828092, "label": "Pin setup-gcloud to v0 instead of master"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1670#issuecomment-1076638278", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1670", "id": 1076638278, "node_id": "IC_kwDOBm6k_c5ALDJG", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-23T17:50:55Z", "updated_at": "2022-03-23T17:50:55Z", "author_association": "OWNER", "body": "Release notes are mostly written for the alpha, just need to clean them up a bit https://github.com/simonw/datasette/blob/c4c9dbd0386e46d2bf199f0ed34e4895c98cb78c/docs/changelog.rst#061a0-2022-03-19", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174423568, "label": "Ship Datasette 0.61"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1681#issuecomment-1075438684", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1681", "id": 1075438684, "node_id": "IC_kwDOBm6k_c5AGeRc", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-22T17:45:50Z", "updated_at": "2022-03-22T17:49:09Z", "author_association": "OWNER", "body": "I would expect this to break against SQL views that include calculated columns though - something like this:\r\n\r\n```sql\r\ncreate view this_will_break as select pk + 1 as pk_plus_one, 0.5 as score from searchable;\r\n```\r\nConfirmed: the filter interface for that view plain doesn't work for any comparison against that table - except for `score > 0` since `0` is converted to an integer. `0.1` breaks though because it doesn't get converted as it doesn't match `.isdigit()`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1177101697, "label": "Potential bug in numeric handling where_clause for filters"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1681#issuecomment-1075437598", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1681", "id": 1075437598, "node_id": "IC_kwDOBm6k_c5AGeAe", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-22T17:44:42Z", "updated_at": "2022-03-22T17:45:04Z", "author_association": "OWNER", "body": "My hunch is that this mechanism doesn't actually do anything useful at all, because of the type conversion that automatically happens for data from tables based on the column type affinities, see:\r\n- #1671\r\n\r\nSo either remove the `self.numeric` type conversion bit entirely, or prove that it is necessary and upgrade it to be able to handle floating point values too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1177101697, "label": "Potential bug in numeric handling where_clause for filters"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1671#issuecomment-1075432283", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1671", "id": 1075432283, "node_id": "IC_kwDOBm6k_c5AGctb", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-22T17:39:04Z", "updated_at": "2022-03-22T17:43:12Z", "author_association": "OWNER", "body": "Note that Datasette does already have special logic to convert parameters to integers for numeric comparisons like `>`:\r\n\r\nhttps://github.com/simonw/datasette/blob/c4c9dbd0386e46d2bf199f0ed34e4895c98cb78c/datasette/filters.py#L203-L212\r\n\r\nThough... it looks like there's a bug in that? It doesn't account for `float` values - `\"3.5\".isdigit()` return `False` - probably for the best, because `int(3.5)` would break that value anyway.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174655187, "label": "Filters fail to work correctly against calculated numeric columns returned by SQL views because type affinity rules do not apply"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1671#issuecomment-1075435185", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1671", "id": 1075435185, "node_id": "IC_kwDOBm6k_c5AGdax", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-22T17:42:09Z", "updated_at": "2022-03-22T17:42:09Z", "author_association": "OWNER", "body": "Also made me realize that this query:\r\n```sql\r\nselect * from sortable where sortable > :p0\r\n```\r\nOnly works here thanks to the column affinity thing kicking in too: https://latest.datasette.io/fixtures?sql=select+*+from+sortable+where+sortable+%3E+%3Ap0&p0=70", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174655187, "label": "Filters fail to work correctly against calculated numeric columns returned by SQL views because type affinity rules do not apply"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1671#issuecomment-1075428030", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1671", "id": 1075428030, "node_id": "IC_kwDOBm6k_c5AGbq-", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-22T17:34:30Z", "updated_at": "2022-03-22T17:34:30Z", "author_association": "OWNER", "body": "No, I think I need to use `cast` - I can't think of any way to ask SQLite \"for this query, what types are the columns that will come back from it?\"\r\n\r\nEven the details from the `explain` trick explored in #1293 don't seem to come back with column type information: https://latest.datasette.io/fixtures?sql=explain+select+pk%2C+text1%2C+text2%2C+[name+with+.+and+spaces]+from+searchable_view+where+%22pk%22+%3D+%3Ap0&p0=1", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174655187, "label": "Filters fail to work correctly against calculated numeric columns returned by SQL views because type affinity rules do not apply"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1671#issuecomment-1075425513", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1671", "id": 1075425513, "node_id": "IC_kwDOBm6k_c5AGbDp", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-22T17:31:53Z", "updated_at": "2022-03-22T17:31:53Z", "author_association": "OWNER", "body": "The alternative to using `cast` here would be for Datasette to convert the `\"1\"` to a `1` in Python code before passing it as a param.\r\n\r\nThis feels a bit neater to me, but I still then need to solve the problem of how to identify the \"type\" of a column that I want to use in a query.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174655187, "label": "Filters fail to work correctly against calculated numeric columns returned by SQL views because type affinity rules do not apply"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/339#issuecomment-1074479932", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/339", "id": 1074479932, "node_id": "IC_kwDOBm6k_c5AC0M8", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T22:22:34Z", "updated_at": "2022-03-21T22:22:34Z", "author_association": "OWNER", "body": "Closing this as obsolete since Datasette no longer uses Sanic.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 340396247, "label": "Expose SANIC_RESPONSE_TIMEOUT config option in a sensible way"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/276#issuecomment-1074479768", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/276", "id": 1074479768, "node_id": "IC_kwDOBm6k_c5AC0KY", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T22:22:20Z", "updated_at": "2022-03-21T22:22:20Z", "author_association": "OWNER", "body": "I'm closing this issue because this is now solved by a number of neat plugins:\r\n\r\n- https://datasette.io/plugins/datasette-geojson-map shows the geometry from SpatiaLite columns on a map\r\n- https://datasette.io/plugins/datasette-leaflet-geojson can be used to display inline maps next to each column", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 324835838, "label": "Handle spatialite geometry columns better"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1671#issuecomment-1074478299", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1671", "id": 1074478299, "node_id": "IC_kwDOBm6k_c5ACzzb", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T22:20:26Z", "updated_at": "2022-03-21T22:20:26Z", "author_association": "OWNER", "body": "Thinking about options for fixing this...\r\n\r\nThe following query works fine:\r\n```sql\r\nselect * from test_view where cast(has_expired as text) = '1'\r\n```\r\nI don't want to start using this for every query, because one of the goals of Datasette is to help people who are learning SQL:\r\n- #1613\r\n\r\nIf someone clicks on \"View and edit SQL\" from a filtered table page I don't want them to have to wonder why that `cast` is there.\r\n\r\nBut... for querying views, the `cast` turns out to be necessary.\r\n\r\nSo one fix would be to get the SQL generating logic to use casts like this any time it is operating against a view.\r\n\r\nAn even better fix would be to detect which columns in a view come from a table and which ones might not, and only use casts for the columns that aren't definitely from a table.\r\n\r\nThe trick I was exploring here might be able to help with that:\r\n- #1293 ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174655187, "label": "Filters fail to work correctly against calculated numeric columns returned by SQL views because type affinity rules do not apply"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1671#issuecomment-1074470568", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1671", "id": 1074470568, "node_id": "IC_kwDOBm6k_c5ACx6o", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T22:11:14Z", "updated_at": "2022-03-21T22:12:49Z", "author_association": "OWNER", "body": "I wonder if this will be a problem with generated columns, or with SQLite strict tables?\r\n\r\nMy hunch is that strict tables will continue to work without any changes, because https://www.sqlite.org/stricttables.html says nothing about their impact on comparison operations. I should test this to make absolutely sure though.\r\n\r\nGenerated columns have a type, so my hunch is they will continue to work fine too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174655187, "label": "Filters fail to work correctly against calculated numeric columns returned by SQL views because type affinity rules do not apply"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1671#issuecomment-1074468450", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1671", "id": 1074468450, "node_id": "IC_kwDOBm6k_c5ACxZi", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T22:08:35Z", "updated_at": "2022-03-21T22:10:00Z", "author_association": "OWNER", "body": "Relevant section of the SQLite documentation: [3.2. Affinity Of Expressions](https://www.sqlite.org/datatype3.html#affinity_of_expressions):\r\n\r\n> When an expression is a simple reference to a column of a real table (not a [VIEW](https://www.sqlite.org/lang_createview.html) or subquery) then the expression has the same affinity as the table column.\r\n\r\nIn your example, `has_expired` is no longer a simple reference to a column of a real table, hence the bug.\r\n\r\nThen [4.2. Type Conversions Prior To Comparison](https://www.sqlite.org/datatype3.html#type_conversions_prior_to_comparison) fills in the rest:\r\n\r\n> SQLite may attempt to convert values between the storage classes INTEGER, REAL, and/or TEXT before performing a comparison. Whether or not any conversions are attempted before the comparison takes place depends on the type affinity of the operands. ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174655187, "label": "Filters fail to work correctly against calculated numeric columns returned by SQL views because type affinity rules do not apply"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1671#issuecomment-1074465536", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1671", "id": 1074465536, "node_id": "IC_kwDOBm6k_c5ACwsA", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T22:04:31Z", "updated_at": "2022-03-21T22:04:31Z", "author_association": "OWNER", "body": "Oh this is fascinating! I replicated the bug (thanks for the steps to reproduce) and it looks like this is down to the following:\r\n\r\n\"image\"\r\n\r\nAgainst views, `where has_expired = 1` returns different results from `where has_expired = '1'`\r\n\r\nThis doesn't happen against tables because of SQLite's [type affinity](https://www.sqlite.org/datatype3.html#type_affinity) mechanism, which handles the type conversion automatically.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174655187, "label": "Filters fail to work correctly against calculated numeric columns returned by SQL views because type affinity rules do not apply"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1679#issuecomment-1074459746", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1679", "id": 1074459746, "node_id": "IC_kwDOBm6k_c5ACvRi", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T21:55:45Z", "updated_at": "2022-03-21T21:55:45Z", "author_association": "OWNER", "body": "I'm going to change the original logic to set n=1 for times that are `<= 20ms` - and update the comments to make it more obvious what is happening.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1175854982, "label": "Research: how much overhead does the n=1 time limit have?"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1679#issuecomment-1074458506", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1679", "id": 1074458506, "node_id": "IC_kwDOBm6k_c5ACu-K", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T21:53:47Z", "updated_at": "2022-03-21T21:53:47Z", "author_association": "OWNER", "body": "Oh interesting, it turns out there is ONE place in the code that sets the `ms` to less than 20 - this test fixture: https://github.com/simonw/datasette/blob/4e47a2d894b96854348343374c8e97c9d7055cf6/tests/fixtures.py#L224-L226", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1175854982, "label": "Research: how much overhead does the n=1 time limit have?"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1679#issuecomment-1074454687", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1679", "id": 1074454687, "node_id": "IC_kwDOBm6k_c5ACuCf", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T21:48:02Z", "updated_at": "2022-03-21T21:48:02Z", "author_association": "OWNER", "body": "Here's another microbenchmark that measures how many nanoseconds it takes to run 1,000 vmops:\r\n\r\n```python\r\nimport sqlite3\r\nimport time\r\n\r\ndb = sqlite3.connect(\":memory:\")\r\n\r\ni = 0\r\nout = []\r\n\r\ndef count():\r\n global i\r\n i += 1000\r\n out.append(((i, time.perf_counter_ns())))\r\n\r\ndb.set_progress_handler(count, 1000)\r\n\r\nprint(\"Start:\", time.perf_counter_ns())\r\nall = db.execute(\"\"\"\r\nwith recursive counter(x) as (\r\n select 0\r\n union\r\n select x + 1 from counter\r\n)\r\nselect * from counter limit 10000;\r\n\"\"\").fetchall()\r\nprint(\"End:\", time.perf_counter_ns())\r\n\r\nprint()\r\nprint(\"So how long does it take to execute 1000 ops?\")\r\n\r\nprev_time_ns = None\r\nfor i, time_ns in out:\r\n if prev_time_ns is not None:\r\n print(time_ns - prev_time_ns, \"ns\")\r\n prev_time_ns = time_ns\r\n```\r\nRunning it:\r\n```\r\n% python nanobench.py\r\nStart: 330877620374821\r\nEnd: 330877632515822\r\n\r\nSo how long does it take to execute 1000 ops?\r\n47290 ns\r\n49573 ns\r\n48226 ns\r\n45674 ns\r\n53238 ns\r\n47313 ns\r\n52346 ns\r\n48689 ns\r\n47092 ns\r\n87596 ns\r\n69999 ns\r\n52522 ns\r\n52809 ns\r\n53259 ns\r\n52478 ns\r\n53478 ns\r\n65812 ns\r\n```\r\n87596ns is 0.087596ms - so even a measure rate of every 1000 ops is easily finely grained enough to capture differences of less than 0.1ms.\r\n\r\nIf anything I could bump that default 1000 up - and I can definitely eliminate the `if ms < 50` branch entirely.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1175854982, "label": "Research: how much overhead does the n=1 time limit have?"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1679#issuecomment-1074446576", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1679", "id": 1074446576, "node_id": "IC_kwDOBm6k_c5ACsDw", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T21:38:27Z", "updated_at": "2022-03-21T21:38:27Z", "author_association": "OWNER", "body": "OK here's a microbenchmark script:\r\n```python\r\nimport sqlite3\r\nimport timeit\r\n\r\ndb = sqlite3.connect(\":memory:\")\r\ndb_with_progress_handler_1 = sqlite3.connect(\":memory:\")\r\ndb_with_progress_handler_1000 = sqlite3.connect(\":memory:\")\r\n\r\ndb_with_progress_handler_1.set_progress_handler(lambda: None, 1)\r\ndb_with_progress_handler_1000.set_progress_handler(lambda: None, 1000)\r\n\r\ndef execute_query(db):\r\n cursor = db.execute(\"\"\"\r\n with recursive counter(x) as (\r\n select 0\r\n union\r\n select x + 1 from counter\r\n )\r\n select * from counter limit 10000;\r\n \"\"\")\r\n list(cursor.fetchall())\r\n\r\n\r\nprint(\"Without progress_handler\")\r\nprint(timeit.timeit(lambda: execute_query(db), number=100))\r\n\r\nprint(\"progress_handler every 1000 ops\")\r\nprint(timeit.timeit(lambda: execute_query(db_with_progress_handler_1000), number=100))\r\n\r\nprint(\"progress_handler every 1 op\")\r\nprint(timeit.timeit(lambda: execute_query(db_with_progress_handler_1), number=100))\r\n```\r\nResults:\r\n```\r\n% python3 bench.py\r\nWithout progress_handler\r\n0.8789225700311363\r\nprogress_handler every 1000 ops\r\n0.8829826560104266\r\nprogress_handler every 1 op\r\n2.8892734259716235\r\n```\r\n\r\nSo running every 1000 ops makes almost no difference at all, but running every single op is a 3.2x performance degradation.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1175854982, "label": "Research: how much overhead does the n=1 time limit have?"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1679#issuecomment-1074439309", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1679", "id": 1074439309, "node_id": "IC_kwDOBm6k_c5ACqSN", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T21:28:58Z", "updated_at": "2022-03-21T21:28:58Z", "author_association": "OWNER", "body": "David Raymond solved it there: https://sqlite.org/forum/forumpost/330c8532d8a88bcd\r\n\r\n> Don't forget to step through the results. All .execute() has done is prepared it.\r\n>\r\n> db.execute(query).fetchall()\r\n\r\nSure enough, adding that gets the VM steps number up to 190,007 which is close enough that I'm happy.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1175854982, "label": "Research: how much overhead does the n=1 time limit have?"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1676#issuecomment-1074378472", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1676", "id": 1074378472, "node_id": "IC_kwDOBm6k_c5ACbbo", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T20:18:10Z", "updated_at": "2022-03-21T20:18:10Z", "author_association": "OWNER", "body": "Maybe there is a better name for this method that helps emphasize its cascading nature.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1175690070, "label": "Reconsider ensure_permissions() logic, can it be less confusing?"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1679#issuecomment-1074347023", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1679", "id": 1074347023, "node_id": "IC_kwDOBm6k_c5ACTwP", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T19:48:59Z", "updated_at": "2022-03-21T19:48:59Z", "author_association": "OWNER", "body": "Posed a question about that here: https://sqlite.org/forum/forumpost/de9ff10fa7", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1175854982, "label": "Research: how much overhead does the n=1 time limit have?"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1679#issuecomment-1074341924", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1679", "id": 1074341924, "node_id": "IC_kwDOBm6k_c5ACSgk", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T19:42:08Z", "updated_at": "2022-03-21T19:42:08Z", "author_association": "OWNER", "body": "Here's the Python-C implementation of `set_progress_handler`: https://github.com/python/cpython/blob/4674fd4e938eb4a29ccd5b12c15455bd2a41c335/Modules/_sqlite/connection.c#L1177-L1201\r\n\r\nIt calls `sqlite3_progress_handler(self->db, n, progress_callback, ctx);`\r\n\r\nhttps://www.sqlite.org/c3ref/progress_handler.html says:\r\n\r\n> The parameter N is the approximate number of [virtual machine instructions](https://www.sqlite.org/opcode.html) that are evaluated between successive invocations of the callback X\r\n\r\nSo maybe VM-steps and virtual machine instructions are different things?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1175854982, "label": "Research: how much overhead does the n=1 time limit have?"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1679#issuecomment-1074337997", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1679", "id": 1074337997, "node_id": "IC_kwDOBm6k_c5ACRjN", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T19:37:08Z", "updated_at": "2022-03-21T19:37:08Z", "author_association": "OWNER", "body": "This is weird:\r\n```python\r\nimport sqlite3\r\n\r\ndb = sqlite3.connect(\":memory:\")\r\n\r\ni = 0\r\n\r\ndef count():\r\n global i\r\n i += 1\r\n\r\n\r\ndb.set_progress_handler(count, 1)\r\n\r\ndb.execute(\"\"\"\r\nwith recursive counter(x) as (\r\n select 0\r\n union\r\n select x + 1 from counter\r\n)\r\nselect * from counter limit 10000;\r\n\"\"\")\r\n\r\nprint(i)\r\n```\r\nOutputs `24`. But if you try the same thing in the SQLite console:\r\n```\r\nsqlite> .stats vmstep\r\nsqlite> with recursive counter(x) as (\r\n ...> select 0\r\n ...> union\r\n ...> select x + 1 from counter\r\n ...> )\r\n ...> select * from counter limit 10000;\r\n...\r\nVM-steps: 200007\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1175854982, "label": "Research: how much overhead does the n=1 time limit have?"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1679#issuecomment-1074332718", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1679", "id": 1074332718, "node_id": "IC_kwDOBm6k_c5ACQQu", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T19:31:10Z", "updated_at": "2022-03-21T19:31:10Z", "author_association": "OWNER", "body": "How long does it take for SQLite to execute 1000 opcodes anyway?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1175854982, "label": "Research: how much overhead does the n=1 time limit have?"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1679#issuecomment-1074332325", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1679", "id": 1074332325, "node_id": "IC_kwDOBm6k_c5ACQKl", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T19:30:44Z", "updated_at": "2022-03-21T19:30:44Z", "author_association": "OWNER", "body": "So it looks like even for facet suggestion `n=1000` always - it's never reduced to `n=1`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1175854982, "label": "Research: how much overhead does the n=1 time limit have?"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1679#issuecomment-1074331743", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1679", "id": 1074331743, "node_id": "IC_kwDOBm6k_c5ACQBf", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T19:30:05Z", "updated_at": "2022-03-21T19:30:05Z", "author_association": "OWNER", "body": "https://github.com/simonw/datasette/blob/1a7750eb29fd15dd2eea3b9f6e33028ce441b143/datasette/app.py#L118-L122 sets it to 50ms for facet suggestion but that's not going to pass `ms < 50`:\r\n\r\n```python\r\n Setting(\r\n \"facet_suggest_time_limit_ms\",\r\n 50,\r\n \"Time limit for calculating a suggested facet\",\r\n ),\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1175854982, "label": "Research: how much overhead does the n=1 time limit have?"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1660#issuecomment-1074321862", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1660", "id": 1074321862, "node_id": "IC_kwDOBm6k_c5ACNnG", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T19:19:01Z", "updated_at": "2022-03-21T19:19:01Z", "author_association": "OWNER", "body": "I've simplified this a ton now. I'm going to keep working on this in the long-term but I think this issue can be closed.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1170144879, "label": "Refactor and simplify Datasette routing and views"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1678#issuecomment-1074302559", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1678", "id": 1074302559, "node_id": "IC_kwDOBm6k_c5ACI5f", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T19:04:03Z", "updated_at": "2022-03-21T19:04:03Z", "author_association": "OWNER", "body": "Documentation: https://docs.datasette.io/en/latest/internals.html#await-check-visibility-actor-action-resource-none", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1175715988, "label": "Make `check_visibility()` a documented API"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1660#issuecomment-1074287177", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1660", "id": 1074287177, "node_id": "IC_kwDOBm6k_c5ACFJJ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T18:51:42Z", "updated_at": "2022-03-21T18:51:42Z", "author_association": "OWNER", "body": "`BaseView` is looking a LOT slimmer now that I've moved all of the permissions stuff out of it.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1170144879, "label": "Refactor and simplify Datasette routing and views"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/417#issuecomment-1074243540", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/417", "id": 1074243540, "node_id": "IC_kwDOCGYnMM5AB6fU", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T18:08:03Z", "updated_at": "2022-03-21T18:08:03Z", "author_association": "OWNER", "body": "I've not really thought about standards as much here as I should. It looks like there are two competing specs for newline-delimited JSON!\r\n\r\nhttp://ndjson.org/ is the one I've been using in `sqlite-utils` - and https://github.com/ndjson/ndjson-spec#31-serialization says:\r\n\r\n> The JSON texts MUST NOT contain newlines or carriage returns.\r\n\r\nhttps://jsonlines.org/ is the other one. It is slightly less clear, but it does say this:\r\n\r\n> 2. Each Line is a Valid JSON Value\r\n>\r\n> The most common values will be objects or arrays, but any JSON value is permitted.\r\n\r\nMy interpretation of both of these is that newlines in the middle of a JSON object shouldn't be allowed.\r\n\r\nSo what's `jq` doing here? It looks to me like that `jq` format is its own thing - it's not actually compatible with either of those two loose specs described above.\r\n\r\nThe `jq` docs seem to call this \"whitespace-separated JSON\": https://stedolan.github.io/jq/manual/v1.6/#Invokingjq\r\n\r\nThe thing I like about newline-delimited JSON is that it's really trivial to parse - loop through each line, run it through `json.loads()` and that's it. No need to try and unwrap JSON objects that might span multiple lines.\r\n\r\nUnless someone has written a robust Python implementation of a `jq`-compatible whitespace-separated JSON parser, I'm inclined to leave this as is. I'd be fine adding some documentation that helps point people towards `jq -c` though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1175744654, "label": "insert fails on JSONL with whitespace"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1677#issuecomment-1074184240", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1677", "id": 1074184240, "node_id": "IC_kwDOBm6k_c5ABsAw", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T17:20:17Z", "updated_at": "2022-03-21T17:20:17Z", "author_association": "OWNER", "body": "https://github.com/simonw/datasette/blob/e627510b760198ccedba9e5af47a771e847785c9/datasette/views/base.py#L69-L77\r\n\r\nThis is weirdly different from how `check_permissions()` used to work, in that it doesn't differentiate between `None` and `False`.\r\n\r\nhttps://github.com/simonw/datasette/blob/4a4164b81191dec35e423486a208b05a9edc65e4/datasette/views/base.py#L79-L103", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1175694248, "label": "Remove `check_permission()` from `BaseView`"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1676#issuecomment-1074180312", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1676", "id": 1074180312, "node_id": "IC_kwDOBm6k_c5ABrDY", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T17:16:45Z", "updated_at": "2022-03-21T17:16:45Z", "author_association": "OWNER", "body": "When looking at this code earlier I assumed that the following would check each permission in turn and fail if any of them failed:\r\n```python\r\nawait self.ds.ensure_permissions(\r\n request.actor,\r\n [\r\n (\"view-table\", (database, table)),\r\n (\"view-database\", database),\r\n \"view-instance\",\r\n ]\r\n)\r\n```\r\nBut it's not quite that simple: if any of them fail, it fails... but if an earlier one returns `True` the whole stack passes even if there would have been a failure later on!\r\n\r\nIf that is indeed the right abstraction, I need to work to make the documentation as clear as possible.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1175690070, "label": "Reconsider ensure_permissions() logic, can it be less confusing?"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1676#issuecomment-1074178865", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1676", "id": 1074178865, "node_id": "IC_kwDOBm6k_c5ABqsx", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T17:15:27Z", "updated_at": "2022-03-21T17:15:27Z", "author_association": "OWNER", "body": "This method here: https://github.com/simonw/datasette/blob/e627510b760198ccedba9e5af47a771e847785c9/datasette/app.py#L632-L664", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1175690070, "label": "Reconsider ensure_permissions() logic, can it be less confusing?"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1675#issuecomment-1074177827", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1675", "id": 1074177827, "node_id": "IC_kwDOBm6k_c5ABqcj", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T17:14:31Z", "updated_at": "2022-03-21T17:14:31Z", "author_association": "OWNER", "body": "Updated documentation: https://github.com/simonw/datasette/blob/e627510b760198ccedba9e5af47a771e847785c9/docs/internals.rst#await-ensure_permissionsactor-permissions\r\n\r\n> This method allows multiple permissions to be checked at onced. It raises a `datasette.Forbidden` exception if any of the checks are denied before one of them is explicitly granted.\r\n> \r\n> This is useful when you need to check multiple permissions at once. For example, an actor should be able to view a table if either one of the following checks returns `True` or not a single one of them returns `False`:\r\n\r\nThat's pretty hard to understand! I'm going to open a separate issue to reconsider if this is a useful enough abstraction given how confusing it is.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1175648453, "label": "Extract out `check_permissions()` from `BaseView"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1675#issuecomment-1074161523", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1675", "id": 1074161523, "node_id": "IC_kwDOBm6k_c5ABmdz", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T16:59:55Z", "updated_at": "2022-03-21T17:00:03Z", "author_association": "OWNER", "body": "Also calling that function `permissions_allowed()` is confusing because there is a plugin hook with a similar name already: https://docs.datasette.io/en/stable/plugin_hooks.html#permission-allowed-datasette-actor-action-resource", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1175648453, "label": "Extract out `check_permissions()` from `BaseView"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1675#issuecomment-1074158890", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1675", "id": 1074158890, "node_id": "IC_kwDOBm6k_c5ABl0q", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T16:57:15Z", "updated_at": "2022-03-21T16:57:15Z", "author_association": "OWNER", "body": "Idea: `ds.permission_allowed()` continues to just return `True` or `False`.\r\n\r\nA new `ds.ensure_permissions(...)` method is added which raises a `Forbidden` exception if a check fails (hence the different name)`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1175648453, "label": "Extract out `check_permissions()` from `BaseView"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1675#issuecomment-1074156779", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1675", "id": 1074156779, "node_id": "IC_kwDOBm6k_c5ABlTr", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T16:55:08Z", "updated_at": "2022-03-21T16:56:02Z", "author_association": "OWNER", "body": "One benefit of the current design of `check_permissions` that raises an exception is that the exception includes information on WHICH of the permission checks failed. Returning just `True` or `False` loses that information.\r\n\r\nI could return an object which evaluates to `False` but also carries extra information? Bit weird, I've never seen anything like that in other Python code.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1175648453, "label": "Extract out `check_permissions()` from `BaseView"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1675#issuecomment-1074143209", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1675", "id": 1074143209, "node_id": "IC_kwDOBm6k_c5ABh_p", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T16:46:05Z", "updated_at": "2022-03-21T16:46:05Z", "author_association": "OWNER", "body": "The other difference though is that `ds.permission_allowed(...)` works against an actor, while `check_permission()` works against a request (though just to access `request.actor`).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1175648453, "label": "Extract out `check_permissions()` from `BaseView"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1675#issuecomment-1074142617", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1675", "id": 1074142617, "node_id": "IC_kwDOBm6k_c5ABh2Z", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T16:45:27Z", "updated_at": "2022-03-21T16:45:27Z", "author_association": "OWNER", "body": "Though at that point `check_permission` is such a light wrapper around `self.ds.permission_allowed()` that there's little point in it existing at all.\r\n\r\nSo maybe `check_permisions()` becomes `ds.permissions_allowed()`.\r\n\r\n`permission_allowed()` v.s. `permissions_allowed()` is a bit of a subtle naming difference, but I think it works.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1175648453, "label": "Extract out `check_permissions()` from `BaseView"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1675#issuecomment-1074141457", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1675", "id": 1074141457, "node_id": "IC_kwDOBm6k_c5ABhkR", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T16:44:09Z", "updated_at": "2022-03-21T16:44:09Z", "author_association": "OWNER", "body": "A slightly odd thing about these methods is that they either fail silently or they raise a `Forbidden` exception.\r\n\r\nMaybe they should instead return `True` or `False` and the calling code could decide if it wants to raise the exception? That would make them more usable and a little less surprising.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1175648453, "label": "Extract out `check_permissions()` from `BaseView"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1660#issuecomment-1074136176", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1660", "id": 1074136176, "node_id": "IC_kwDOBm6k_c5ABgRw", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T16:38:46Z", "updated_at": "2022-03-21T16:38:46Z", "author_association": "OWNER", "body": "I'm going to refactor this stuff out and document it so it can be easily used by plugins:\r\n\r\nhttps://github.com/simonw/datasette/blob/4a4164b81191dec35e423486a208b05a9edc65e4/datasette/views/base.py#L69-L103", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1170144879, "label": "Refactor and simplify Datasette routing and views"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/526#issuecomment-1074019047", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/526", "id": 1074019047, "node_id": "IC_kwDOBm6k_c5ABDrn", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T15:09:56Z", "updated_at": "2022-03-21T15:09:56Z", "author_association": "OWNER", "body": "I should research how much overhead creating a new connection costs - it may be that an easy way to solve this is to create A dedicated connection for the query and then close that connection at the end.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 459882902, "label": "Stream all results for arbitrary SQL and canned queries"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1177#issuecomment-1074017633", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1177", "id": 1074017633, "node_id": "IC_kwDOBm6k_c5ABDVh", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T15:08:51Z", "updated_at": "2022-03-21T15:08:51Z", "author_association": "OWNER", "body": "Related:\r\n- #1062 ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 780153562, "label": "Ability to stream all rows as newline-delimited JSON"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/415#issuecomment-1073468996", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/415", "id": 1073468996, "node_id": "IC_kwDOCGYnMM4_-9ZE", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T04:14:42Z", "updated_at": "2022-03-21T04:14:42Z", "author_association": "OWNER", "body": "I can fix this like so:\r\n```\r\n% sqlite-utils convert demo.db demo foo '{\"foo\": \"bar\"}' --multi --dry-run\r\nabc\r\n --- becomes:\r\n{\"foo\": \"bar\"}\r\n\r\nWould affect 1 row\r\n```\r\nDiff is this:\r\n```diff\r\ndiff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py\r\nindex 0cf0468..b2a0440 100644\r\n--- a/sqlite_utils/cli.py\r\n+++ b/sqlite_utils/cli.py\r\n@@ -2676,7 +2676,10 @@ def convert(\r\n raise click.ClickException(str(e))\r\n if dry_run:\r\n # Pull first 20 values for first column and preview them\r\n- db.conn.create_function(\"preview_transform\", 1, lambda v: fn(v) if v else v)\r\n+ preview = lambda v: fn(v) if v else v\r\n+ if multi:\r\n+ preview = lambda v: json.dumps(fn(v), default=repr) if v else v\r\n+ db.conn.create_function(\"preview_transform\", 1, preview)\r\n sql = \"\"\"\r\n select\r\n [{column}] as value,\r\n```", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 1, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1171599874, "label": "Convert with `--multi` and `--dry-run` flag does not work"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/415#issuecomment-1073463375", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/415", "id": 1073463375, "node_id": "IC_kwDOCGYnMM4_-8BP", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T04:02:36Z", "updated_at": "2022-03-21T04:02:36Z", "author_association": "OWNER", "body": "Thanks for the really clear steps to reproduce!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1171599874, "label": "Convert with `--multi` and `--dry-run` flag does not work"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073456222", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/416", "id": 1073456222, "node_id": "IC_kwDOCGYnMM4_-6Re", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T03:45:52Z", "updated_at": "2022-03-21T03:45:52Z", "author_association": "OWNER", "body": "Needs tests and documentation.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1173023272, "label": "Options for how `r.parsedate()` should handle invalid dates"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073456155", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/416", "id": 1073456155, "node_id": "IC_kwDOCGYnMM4_-6Qb", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T03:45:37Z", "updated_at": "2022-03-21T03:45:37Z", "author_association": "OWNER", "body": "Prototype:\r\n```diff\r\ndiff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py\r\nindex 8255b56..0a3693e 100644\r\n--- a/sqlite_utils/cli.py\r\n+++ b/sqlite_utils/cli.py\r\n@@ -2583,7 +2583,11 @@ def _generate_convert_help():\r\n \"\"\"\r\n ).strip()\r\n recipe_names = [\r\n- n for n in dir(recipes) if not n.startswith(\"_\") and n not in (\"json\", \"parser\")\r\n+ n\r\n+ for n in dir(recipes)\r\n+ if not n.startswith(\"_\")\r\n+ and n not in (\"json\", \"parser\")\r\n+ and callable(getattr(recipes, n))\r\n ]\r\n for name in recipe_names:\r\n fn = getattr(recipes, name)\r\ndiff --git a/sqlite_utils/recipes.py b/sqlite_utils/recipes.py\r\nindex 6918661..569c30d 100644\r\n--- a/sqlite_utils/recipes.py\r\n+++ b/sqlite_utils/recipes.py\r\n@@ -1,17 +1,38 @@\r\n from dateutil import parser\r\n import json\r\n \r\n+IGNORE = object()\r\n+SET_NULL = object()\r\n \r\n-def parsedate(value, dayfirst=False, yearfirst=False):\r\n+\r\n+def parsedate(value, dayfirst=False, yearfirst=False, errors=None):\r\n \"Parse a date and convert it to ISO date format: yyyy-mm-dd\"\r\n- return (\r\n- parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).date().isoformat()\r\n- )\r\n+ try:\r\n+ return (\r\n+ parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst)\r\n+ .date()\r\n+ .isoformat()\r\n+ )\r\n+ except parser.ParserError:\r\n+ if errors is IGNORE:\r\n+ return value\r\n+ elif errors is SET_NULL:\r\n+ return None\r\n+ else:\r\n+ raise\r\n \r\n \r\n-def parsedatetime(value, dayfirst=False, yearfirst=False):\r\n+def parsedatetime(value, dayfirst=False, yearfirst=False, errors=None):\r\n \"Parse a datetime and convert it to ISO datetime format: yyyy-mm-ddTHH:MM:SS\"\r\n- return parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).isoformat()\r\n+ try:\r\n+ return parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).isoformat()\r\n+ except parser.ParserError:\r\n+ if errors is IGNORE:\r\n+ return value\r\n+ elif errors is SET_NULL:\r\n+ return None\r\n+ else:\r\n+ raise\r\n \r\n \r\n def jsonsplit(value, delimiter=\",\", type=str):\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1173023272, "label": "Options for how `r.parsedate()` should handle invalid dates"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073455905", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/416", "id": 1073455905, "node_id": "IC_kwDOCGYnMM4_-6Mh", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T03:44:47Z", "updated_at": "2022-03-21T03:45:00Z", "author_association": "OWNER", "body": "This is quite nice:\r\n```\r\n% sqlite-utils convert test-dates.db dates date \"r.parsedate(value, errors=r.IGNORE)\"\r\n [####################################] 100%\r\n% sqlite-utils rows test-dates.db dates \r\n[{\"id\": 1, \"date\": \"2016-03-15\"},\r\n {\"id\": 2, \"date\": \"2016-03-16\"},\r\n {\"id\": 3, \"date\": \"2016-03-17\"},\r\n {\"id\": 4, \"date\": \"2016-03-18\"},\r\n {\"id\": 5, \"date\": \"2016-03-19\"},\r\n {\"id\": 6, \"date\": \"2016-03-20\"},\r\n {\"id\": 7, \"date\": \"2016-03-21\"},\r\n {\"id\": 8, \"date\": \"2016-03-22\"},\r\n {\"id\": 9, \"date\": \"2016-03-23\"},\r\n {\"id\": 10, \"date\": \"//\"},\r\n {\"id\": 11, \"date\": \"2016-03-25\"},\r\n {\"id\": 12, \"date\": \"2016-03-26\"},\r\n {\"id\": 13, \"date\": \"2016-03-27\"},\r\n {\"id\": 14, \"date\": \"2016-03-28\"},\r\n {\"id\": 15, \"date\": \"2016-03-29\"},\r\n {\"id\": 16, \"date\": \"2016-03-30\"},\r\n {\"id\": 17, \"date\": \"2016-03-31\"},\r\n {\"id\": 18, \"date\": \"2016-04-01\"}]\r\n% sqlite-utils convert test-dates.db dates date \"r.parsedate(value, errors=r.SET_NULL)\"\r\n [####################################] 100%\r\n% sqlite-utils rows test-dates.db dates \r\n[{\"id\": 1, \"date\": \"2016-03-15\"},\r\n {\"id\": 2, \"date\": \"2016-03-16\"},\r\n {\"id\": 3, \"date\": \"2016-03-17\"},\r\n {\"id\": 4, \"date\": \"2016-03-18\"},\r\n {\"id\": 5, \"date\": \"2016-03-19\"},\r\n {\"id\": 6, \"date\": \"2016-03-20\"},\r\n {\"id\": 7, \"date\": \"2016-03-21\"},\r\n {\"id\": 8, \"date\": \"2016-03-22\"},\r\n {\"id\": 9, \"date\": \"2016-03-23\"},\r\n {\"id\": 10, \"date\": null},\r\n {\"id\": 11, \"date\": \"2016-03-25\"},\r\n {\"id\": 12, \"date\": \"2016-03-26\"},\r\n {\"id\": 13, \"date\": \"2016-03-27\"},\r\n {\"id\": 14, \"date\": \"2016-03-28\"},\r\n {\"id\": 15, \"date\": \"2016-03-29\"},\r\n {\"id\": 16, \"date\": \"2016-03-30\"},\r\n {\"id\": 17, \"date\": \"2016-03-31\"},\r\n {\"id\": 18, \"date\": \"2016-04-01\"}]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1173023272, "label": "Options for how `r.parsedate()` should handle invalid dates"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073453370", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/416", "id": 1073453370, "node_id": "IC_kwDOCGYnMM4_-5k6", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T03:41:06Z", "updated_at": "2022-03-21T03:41:06Z", "author_association": "OWNER", "body": "I'm going to try the `errors=r.IGNORE` option and see what that looks like once implemented.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1173023272, "label": "Options for how `r.parsedate()` should handle invalid dates"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073453230", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/416", "id": 1073453230, "node_id": "IC_kwDOCGYnMM4_-5iu", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T03:40:37Z", "updated_at": "2022-03-21T03:40:37Z", "author_association": "OWNER", "body": "I think the options here should be:\r\n\r\n- On error, raise an exception and revert the transaction (the current default)\r\n- On error, leave the value as-is\r\n- On error, set the value to `None`\r\n\r\nThese need to be indicated by parameters to the `r.parsedate()` function.\r\n\r\nSome design options:\r\n\r\n- `ignore=True` to ignore errors - but how does it know if it should leave the value or set it to `None`? This is similar to other `ignore=True` parameters elsewhere in the Python API.\r\n- `errors=\"ignore\"`, `errors=\"set-null\"` - I don't like magic string values very much, but this is similar to Python's `str.encode(errors=)` mechanism\r\n- `errors=r.IGNORE` - using constants, which at least avoids magic strings. The other one could be `errors=r.SET_NULL`\r\n- `error=lambda v: None` or `error=lambda v: v` - this is a bit confusing though, introducing another callback that gets to have a go at converting the error if the first callback failed? And what happens if that lambda itself raises an error?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1173023272, "label": "Options for how `r.parsedate()` should handle invalid dates"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073451659", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/416", "id": 1073451659, "node_id": "IC_kwDOCGYnMM4_-5KL", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T03:35:01Z", "updated_at": "2022-03-21T03:35:01Z", "author_association": "OWNER", "body": "I confirmed that if it fails for any value ALL values are left alone, since it runs in a transaction.\r\n\r\nHere's the code that does that:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/433813612ff9b4b501739fd7543bef0040dd51fe/sqlite_utils/db.py#L2523-L2526", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1173023272, "label": "Options for how `r.parsedate()` should handle invalid dates"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073450588", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/416", "id": 1073450588, "node_id": "IC_kwDOCGYnMM4_-45c", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T03:32:58Z", "updated_at": "2022-03-21T03:32:58Z", "author_association": "OWNER", "body": "Then I ran this to convert `2016-03-27` etc to `2016/03/27` so I could see which ones were later converted:\r\n\r\n sqlite-utils convert test-dates.db dates date 'value.replace(\"-\", \"/\")'\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1173023272, "label": "Options for how `r.parsedate()` should handle invalid dates"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073448904", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/416", "id": 1073448904, "node_id": "IC_kwDOCGYnMM4_-4fI", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-21T03:28:12Z", "updated_at": "2022-03-21T03:30:37Z", "author_association": "OWNER", "body": "Generating a test database using a pattern from https://www.geekytidbits.com/date-range-table-sqlite/\r\n```\r\nsqlite-utils create-database test-dates.db\r\nsqlite-utils create-table test-dates.db dates id integer date text --pk id\r\nsqlite-utils test-dates.db \"WITH RECURSIVE\r\n cnt(x) AS (\r\n SELECT 0\r\n UNION ALL\r\n SELECT x+1 FROM cnt\r\n LIMIT (SELECT ((julianday('2016-04-01') - julianday('2016-03-15'))) + 1)\r\n )\r\ninsert into dates (date) select date(julianday('2016-03-15'), '+' || x || ' days') as date FROM cnt;\"\r\n```\r\nAfter running that:\r\n```\r\n% sqlite-utils rows test-dates.db dates\r\n[{\"id\": 1, \"date\": \"2016-03-15\"},\r\n {\"id\": 2, \"date\": \"2016-03-16\"},\r\n {\"id\": 3, \"date\": \"2016-03-17\"},\r\n {\"id\": 4, \"date\": \"2016-03-18\"},\r\n {\"id\": 5, \"date\": \"2016-03-19\"},\r\n {\"id\": 6, \"date\": \"2016-03-20\"},\r\n {\"id\": 7, \"date\": \"2016-03-21\"},\r\n {\"id\": 8, \"date\": \"2016-03-22\"},\r\n {\"id\": 9, \"date\": \"2016-03-23\"},\r\n {\"id\": 10, \"date\": \"2016-03-24\"},\r\n {\"id\": 11, \"date\": \"2016-03-25\"},\r\n {\"id\": 12, \"date\": \"2016-03-26\"},\r\n {\"id\": 13, \"date\": \"2016-03-27\"},\r\n {\"id\": 14, \"date\": \"2016-03-28\"},\r\n {\"id\": 15, \"date\": \"2016-03-29\"},\r\n {\"id\": 16, \"date\": \"2016-03-30\"},\r\n {\"id\": 17, \"date\": \"2016-03-31\"},\r\n {\"id\": 18, \"date\": \"2016-04-01\"}]\r\n```\r\nThen to make one of them invalid:\r\n\r\n sqlite-utils test-dates.db \"update dates set date = '//' where id = 10\"", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1173023272, "label": "Options for how `r.parsedate()` should handle invalid dates"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1510#issuecomment-1073366630", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1510", "id": 1073366630, "node_id": "IC_kwDOBm6k_c4_-kZm", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-20T22:59:33Z", "updated_at": "2022-03-20T22:59:33Z", "author_association": "OWNER", "body": "I really like the idea of making this effectively the same thing as the fully documented, stable JSON API that comes as part of 1.0. If you want to know what will be available to your templates, consult the API documentation.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1054244712, "label": "Datasette 1.0 documented template context (maybe via API docs)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1674#issuecomment-1073366436", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1674", "id": 1073366436, "node_id": "IC_kwDOBm6k_c4_-kWk", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-20T22:58:40Z", "updated_at": "2022-03-20T22:58:40Z", "author_association": "OWNER", "body": "This will probably happen as part of turning this into an officially documented API that serves the template context for the homepage:\r\n- #1510", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174717287, "label": "Tweak design of /.json"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1355#issuecomment-1073362979", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1355", "id": 1073362979, "node_id": "IC_kwDOBm6k_c4_-jgj", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-20T22:38:53Z", "updated_at": "2022-03-20T22:38:53Z", "author_association": "OWNER", "body": "Built a research prototype:\r\n```diff\r\ndiff --git a/datasette/app.py b/datasette/app.py\r\nindex 5c8101a..5cd3e63 100644\r\n--- a/datasette/app.py\r\n+++ b/datasette/app.py\r\n@@ -1,6 +1,7 @@\r\n import asyncio\r\n import asgi_csrf\r\n import collections\r\n+import contextlib\r\n import datetime\r\n import functools\r\n import glob\r\n@@ -1490,3 +1491,11 @@ class DatasetteClient:\r\n return await client.request(\r\n method, self._fix(path, avoid_path_rewrites), **kwargs\r\n )\r\n+\r\n+ @contextlib.asynccontextmanager\r\n+ async def stream(self, method, path, **kwargs):\r\n+ async with httpx.AsyncClient(app=self.app) as client:\r\n+ print(\"async with as client\")\r\n+ async with client.stream(method, self._fix(path), **kwargs) as response:\r\n+ print(\"async with client.stream about to yield response\")\r\n+ yield response\r\ndiff --git a/datasette/cli.py b/datasette/cli.py\r\nindex 3c6e1b2..3025ead 100644\r\n--- a/datasette/cli.py\r\n+++ b/datasette/cli.py\r\n@@ -585,11 +585,19 @@ def serve(\r\n asyncio.get_event_loop().run_until_complete(check_databases(ds))\r\n \r\n if get:\r\n- client = TestClient(ds)\r\n- response = client.get(get)\r\n- click.echo(response.text)\r\n- exit_code = 0 if response.status == 200 else 1\r\n- sys.exit(exit_code)\r\n+\r\n+ async def _run_get():\r\n+ print(\"_run_get\")\r\n+ async with ds.client.stream(\"GET\", get) as response:\r\n+ print(\"Got response:\", response)\r\n+ async for chunk in response.aiter_bytes(chunk_size=1024):\r\n+ print(\" chunk\")\r\n+ sys.stdout.buffer.write(chunk)\r\n+ sys.stdout.buffer.flush()\r\n+ exit_code = 0 if response.status_code == 200 else 1\r\n+ sys.exit(exit_code)\r\n+\r\n+ asyncio.get_event_loop().run_until_complete(_run_get())\r\n return\r\n \r\n # Start the server\r\n```\r\nBut for some reason it didn't appear to stream out the response - it would print this out:\r\n```\r\n% datasette covid.db --get '/covid/ny_times_us_counties.csv?_size=10&_stream=on'\r\n_run_get\r\nasync with as client\r\n```\r\nAnd then hang. I would expect it to start printing out chunks of CSV data here, but instead it looks like it waited for everything to be generated before returning anything to the console.\r\n\r\nNo idea why. I dropped this for the moment.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 910088936, "label": "datasette --get should efficiently handle streaming CSV"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1673#issuecomment-1073361986", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1673", "id": 1073361986, "node_id": "IC_kwDOBm6k_c4_-jRC", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-20T22:31:41Z", "updated_at": "2022-03-20T22:34:06Z", "author_association": "OWNER", "body": "Maybe it's because `supports_table_xinfo()` creates a brand new in-memory SQLite connection every time you call it?\r\n\r\nhttps://github.com/simonw/datasette/blob/798f075ef9b98819fdb564f9f79c78975a0f71e8/datasette/utils/sqlite.py#L22-L35\r\n\r\nActually no, I'm caching that already:\r\n\r\nhttps://github.com/simonw/datasette/blob/798f075ef9b98819fdb564f9f79c78975a0f71e8/datasette/utils/sqlite.py#L12-L19", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174708375, "label": "Streaming CSV spends a lot of time in `table_column_details`"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1672#issuecomment-1073355818", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1672", "id": 1073355818, "node_id": "IC_kwDOBm6k_c4_-hwq", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-20T21:52:38Z", "updated_at": "2022-03-20T21:52:38Z", "author_association": "OWNER", "body": "That means taking on these issues:\r\n\r\n- https://github.com/simonw/datasette/issues/1101\r\n- https://github.com/simonw/datasette/issues/1096\r\n- https://github.com/simonw/datasette/issues/1062", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174697144, "label": "Refactor CSV handling code out of DataView"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1660#issuecomment-1073355032", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1660", "id": 1073355032, "node_id": "IC_kwDOBm6k_c4_-hkY", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-20T21:46:43Z", "updated_at": "2022-03-20T21:46:43Z", "author_association": "OWNER", "body": "I think the way to get rid of most of the remaining complexity in `DataView` is to refactor how CSV stuff works - pulling it in line with other export factors and extracting the streaming mechanism. Opening a fresh issue for that.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1170144879, "label": "Refactor and simplify Datasette routing and views"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/140#issuecomment-1073330388", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/140", "id": 1073330388, "node_id": "IC_kwDOCGYnMM4_-bjU", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-20T19:44:39Z", "updated_at": "2022-03-20T19:45:45Z", "author_association": "OWNER", "body": "Alternative idea for specifying types: accept a Python expression, then use Python type literal syntax. For example:\r\n\r\n```\r\nsqlite-utils insert-files gifs.db images *.gif \\\r\n -c path -c md5 -c last_modified:mtime \\\r\n -a file_type '\"gif\"'\r\n```\r\n\r\nWhere `-a` indicates an additional column.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 688351054, "label": "Idea: insert-files mechanism for adding extra columns with fixed values"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1669#issuecomment-1073143413", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1669", "id": 1073143413, "node_id": "IC_kwDOBm6k_c4_9t51", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-20T01:24:36Z", "updated_at": "2022-03-20T01:24:36Z", "author_association": "OWNER", "body": "https://github.com/simonw/datasette/releases/tag/0.61a0", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174404647, "label": "Release 0.61 alpha"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1669#issuecomment-1073137170", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1669", "id": 1073137170, "node_id": "IC_kwDOBm6k_c4_9sYS", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-20T00:35:52Z", "updated_at": "2022-03-20T00:35:52Z", "author_association": "OWNER", "body": "https://github.com/simonw/datasette/compare/0.60.2...main", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174404647, "label": "Release 0.61 alpha"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1668#issuecomment-1073136896", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1668", "id": 1073136896, "node_id": "IC_kwDOBm6k_c4_9sUA", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-20T00:33:23Z", "updated_at": "2022-03-20T00:33:23Z", "author_association": "OWNER", "body": "I'm going to release this as a 0.61 alpha so I can more easily depend on it from `datasette-hashed-urls`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174306154, "label": "Introduce concept of a database `route`, separate from its name"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1668#issuecomment-1073136686", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1668", "id": 1073136686, "node_id": "IC_kwDOBm6k_c4_9sQu", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-20T00:31:13Z", "updated_at": "2022-03-20T00:31:13Z", "author_association": "OWNER", "body": "That demo is now live:\r\n\r\n- https://latest.datasette.io/alternative-route\r\n- https://latest.datasette.io/alternative-route/attraction_characteristic", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174306154, "label": "Introduce concept of a database `route`, separate from its name"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1668#issuecomment-1073135433", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1668", "id": 1073135433, "node_id": "IC_kwDOBm6k_c4_9r9J", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-20T00:20:36Z", "updated_at": "2022-03-20T00:20:36Z", "author_association": "OWNER", "body": "Building this plugin instantly revealed that all of the links - on the homepage and the database page and so on - are incorrect:\r\n```python\r\nfrom datasette import hookimpl\r\n\r\n@hookimpl\r\ndef startup(datasette):\r\n db = datasette.get_database(\"fixtures2\")\r\n db.route = \"alternative-route\"\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174306154, "label": "Introduce concept of a database `route`, separate from its name"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1668#issuecomment-1073134816", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1668", "id": 1073134816, "node_id": "IC_kwDOBm6k_c4_9rzg", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-20T00:16:22Z", "updated_at": "2022-03-20T00:16:22Z", "author_association": "OWNER", "body": "I'm going to add a `fixtures2.db` database which has that as the name but `alternative-route` as the route. I'll set that up using a custom plugin in the `plugins/` folder that gets deployed by https://github.com/simonw/datasette/blob/main/.github/workflows/deploy-latest.yml", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174306154, "label": "Introduce concept of a database `route`, separate from its name"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1668#issuecomment-1073134206", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1668", "id": 1073134206, "node_id": "IC_kwDOBm6k_c4_9rp-", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-20T00:12:03Z", "updated_at": "2022-03-20T00:12:03Z", "author_association": "OWNER", "body": "I'd like to have a live demo of this up on `latest.datasette.io` too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174306154, "label": "Introduce concept of a database `route`, separate from its name"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1668#issuecomment-1073126264", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1668", "id": 1073126264, "node_id": "IC_kwDOBm6k_c4_9pt4", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-19T22:59:30Z", "updated_at": "2022-03-19T22:59:30Z", "author_association": "OWNER", "body": "Also need to update the `datasette.urls` methods that construct the URL to a database/table/row - they take the database name but they need to know to look for the route.\r\n\r\nNeed to add tests that check the links in the HTML and can confirm this is working correctly.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174306154, "label": "Introduce concept of a database `route`, separate from its name"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1668#issuecomment-1073125334", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1668", "id": 1073125334, "node_id": "IC_kwDOBm6k_c4_9pfW", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-19T22:53:55Z", "updated_at": "2022-03-19T22:53:55Z", "author_association": "OWNER", "body": "Need to update documentation in a few places - e.g. https://docs.datasette.io/en/stable/internals.html#remove-database-name\r\n\r\n> This removes a database that has been previously added. `name=` is the unique name of that database, used in its URL path.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174306154, "label": "Introduce concept of a database `route`, separate from its name"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1668#issuecomment-1073112104", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1668", "id": 1073112104, "node_id": "IC_kwDOBm6k_c4_9mQo", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-19T21:08:21Z", "updated_at": "2022-03-19T21:08:21Z", "author_association": "OWNER", "body": "I think I've got this working but I need to write a test for it that covers the rare case when the route is not the same thing as the database name.\r\n\r\nI'll do that with a new test.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174306154, "label": "Introduce concept of a database `route`, separate from its name"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1668#issuecomment-1073097394", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1668", "id": 1073097394, "node_id": "IC_kwDOBm6k_c4_9iqy", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-19T20:56:35Z", "updated_at": "2022-03-19T20:56:35Z", "author_association": "OWNER", "body": "I'm trying to think if there's any reason not to use `route` for this. Would I possibly want to use that noun for something else in the future? I like it more than `route_path` because it has no underscore.\r\n\r\nDecision made: I'm going with `route`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174306154, "label": "Introduce concept of a database `route`, separate from its name"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1667#issuecomment-1073076624", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1667", "id": 1073076624, "node_id": "IC_kwDOBm6k_c4_9dmQ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-19T20:31:44Z", "updated_at": "2022-03-19T20:31:44Z", "author_association": "OWNER", "body": "I can now read `format` from `request.url_vars` and delete this code entirely: https://github.com/simonw/datasette/blob/b9c2b1cfc8692b9700416db98721fa3ec982f6be/datasette/views/base.py#L375-L381", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174302994, "label": "Make route matched pattern groups more consistent"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1668#issuecomment-1073076187", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1668", "id": 1073076187, "node_id": "IC_kwDOBm6k_c4_9dfb", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-19T20:28:20Z", "updated_at": "2022-03-19T20:28:20Z", "author_association": "OWNER", "body": "I'm going to keep `path` as the path to the file on disk. I'll pick a new name for what is currently `path` in that undocumented JSON API.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174306154, "label": "Introduce concept of a database `route`, separate from its name"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1668#issuecomment-1073076136", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1668", "id": 1073076136, "node_id": "IC_kwDOBm6k_c4_9deo", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-19T20:27:44Z", "updated_at": "2022-03-19T20:27:44Z", "author_association": "OWNER", "body": "Pretty sure changing it will break some existing plugins though, including likely Datasette Desktop.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174306154, "label": "Introduce concept of a database `route`, separate from its name"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1668#issuecomment-1073076110", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1668", "id": 1073076110, "node_id": "IC_kwDOBm6k_c4_9deO", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-19T20:27:22Z", "updated_at": "2022-03-19T20:27:22Z", "author_association": "OWNER", "body": "The docs do currently describe `path` as the filesystem path here: https://docs.datasette.io/en/stable/internals.html#database-class\r\n\r\n\"image\"\r\n\r\nGood thing I'm not at 1.0 yet so I can change that!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174306154, "label": "Introduce concept of a database `route`, separate from its name"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1668#issuecomment-1073076015", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1668", "id": 1073076015, "node_id": "IC_kwDOBm6k_c4_9dcv", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-19T20:26:32Z", "updated_at": "2022-03-19T20:26:32Z", "author_association": "OWNER", "body": "I'm inclined to redefine `ds.path` to `ds.file_path` to fix this. Or `ds.filepath`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174306154, "label": "Introduce concept of a database `route`, separate from its name"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1668#issuecomment-1073075913", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1668", "id": 1073075913, "node_id": "IC_kwDOBm6k_c4_9dbJ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-19T20:25:46Z", "updated_at": "2022-03-19T20:26:08Z", "author_association": "OWNER", "body": "The output of `/.json` DOES use `path` to mean the URL path, not the path to the file on disk:\r\n\r\n```\r\n{\r\n \"fixtures.dot\": {\r\n \"name\": \"fixtures.dot\",\r\n \"hash\": null,\r\n \"color\": \"631f11\",\r\n \"path\": \"/fixtures~2Edot\",\r\n```\r\nSo that's a problem already: having `db.path` refer to something different from that JSON is inconsistent.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174306154, "label": "Introduce concept of a database `route`, separate from its name"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1668#issuecomment-1073075697", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1668", "id": 1073075697, "node_id": "IC_kwDOBm6k_c4_9dXx", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-19T20:24:06Z", "updated_at": "2022-03-19T20:24:06Z", "author_association": "OWNER", "body": "Right now if a database has a `.` in its name e.g. `fixtures.dot` the URL to that database is:\r\n\r\n /fixtures~2Edot\r\n\r\nBut the output on `/-/databases` doesn't reflect that, it still shows the name with the dot.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174306154, "label": "Introduce concept of a database `route`, separate from its name"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1660#issuecomment-1073073599", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1660", "id": 1073073599, "node_id": "IC_kwDOBm6k_c4_9c2_", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-19T20:06:40Z", "updated_at": "2022-03-19T20:06:40Z", "author_association": "OWNER", "body": "This blocks:\r\n- #1668", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1170144879, "label": "Refactor and simplify Datasette routing and views"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1668#issuecomment-1073073579", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1668", "id": 1073073579, "node_id": "IC_kwDOBm6k_c4_9c2r", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-19T20:06:27Z", "updated_at": "2022-03-19T20:06:27Z", "author_association": "OWNER", "body": "Marking this as blocked until #1660 is done.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174306154, "label": "Introduce concept of a database `route`, separate from its name"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1668#issuecomment-1073073547", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1668", "id": 1073073547, "node_id": "IC_kwDOBm6k_c4_9c2L", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-19T20:06:07Z", "updated_at": "2022-03-19T20:06:07Z", "author_association": "OWNER", "body": "Implementing this is a little tricky because there's a whole lot of code that expects the `database` captured by the URL routing to be the name used to look up the database in `datasette.databases` - or via `.get_database()`.\r\n\r\nThe `DataView.get()` method is a good example of the trickyness here. It even has code that dispatches out to plugin hooks that take `database` as a parameter.\r\n\r\nhttps://github.com/simonw/datasette/blob/61419388c134001118aaf7dfb913562d467d7913/datasette/views/base.py#L383-L555\r\n\r\nAll the more reason to get rid of that `BaseView -> DataView -> TableView` hierarchy entirely:\r\n- #1660", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174306154, "label": "Introduce concept of a database `route`, separate from its name"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1668#issuecomment-1073043433", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1668", "id": 1073043433, "node_id": "IC_kwDOBm6k_c4_9Vfp", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-19T16:54:55Z", "updated_at": "2022-03-19T20:01:19Z", "author_association": "OWNER", "body": "Options:\r\n- `route_path`\r\n- `url_path`\r\n- `route`\r\n\r\nI like `route_path`, or maybe `route`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174306154, "label": "Introduce concept of a database `route`, separate from its name"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1668#issuecomment-1073043713", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1668", "id": 1073043713, "node_id": "IC_kwDOBm6k_c4_9VkB", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-19T16:56:19Z", "updated_at": "2022-03-19T16:56:19Z", "author_association": "OWNER", "body": "Worth noting that the `name` right now is picked automatically to avoid conflicts:\r\n\r\nhttps://github.com/simonw/datasette/blob/61419388c134001118aaf7dfb913562d467d7913/datasette/app.py#L397-L413", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174306154, "label": "Introduce concept of a database `route`, separate from its name"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1668#issuecomment-1073043350", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1668", "id": 1073043350, "node_id": "IC_kwDOBm6k_c4_9VeW", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-19T16:54:26Z", "updated_at": "2022-03-19T16:54:26Z", "author_association": "OWNER", "body": "The `Database` class already has a `path` property but it means something else - it's the path to the `.db` file on disk:\r\n\r\nhttps://github.com/simonw/datasette/blob/61419388c134001118aaf7dfb913562d467d7913/datasette/database.py#L29-L50\r\n\r\nSo need a different name for the path-that-is-used-in-the-URL.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174306154, "label": "Introduce concept of a database `route`, separate from its name"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1667#issuecomment-1073042554", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1667", "id": 1073042554, "node_id": "IC_kwDOBm6k_c4_9VR6", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-19T16:50:01Z", "updated_at": "2022-03-19T16:52:35Z", "author_association": "OWNER", "body": "OK, I've made this more consistent - I still need to address the fact that `format` can be `.json` or `json` or not used at all before I close this issue.\r\n\r\nhttps://github.com/simonw/datasette/blob/61419388c134001118aaf7dfb913562d467d7913/tests/test_routes.py#L15-L35", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174302994, "label": "Make route matched pattern groups more consistent"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1667#issuecomment-1073040072", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1667", "id": 1073040072, "node_id": "IC_kwDOBm6k_c4_9UrI", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-19T16:34:02Z", "updated_at": "2022-03-19T16:34:02Z", "author_association": "OWNER", "body": "I called it `as_format` to avoid clashing with the Python built-in `format()` function when these things were turned into keyword arguments, but now that they're not I can use `format` instead.\r\n\r\nI think I'm going to go with `database`, `table`, `format` and `pks`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174302994, "label": "Make route matched pattern groups more consistent"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1666#issuecomment-1073039670", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1666", "id": 1073039670, "node_id": "IC_kwDOBm6k_c4_9Uk2", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-19T16:31:08Z", "updated_at": "2022-03-19T16:31:57Z", "author_association": "OWNER", "body": "This does make it more interesting - it also highlights how inconsistent the way the capturing works is. Especially `as_format` which can be `None` or `\"\"` or `.json` or `json` or not used at all in the case of `TableView`.\r\n\r\nhttps://github.com/simonw/datasette/blob/764738dfcb16cd98b0987d443f59d5baa9d3c332/tests/test_routes.py#L12-L36", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174162781, "label": "Refactor URL routing to enable testing"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1666#issuecomment-1073039241", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1666", "id": 1073039241, "node_id": "IC_kwDOBm6k_c4_9UeJ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-19T16:28:15Z", "updated_at": "2022-03-19T16:28:15Z", "author_association": "OWNER", "body": "This is more interesting if it also asserts against the captured matches from the pattern.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174162781, "label": "Refactor URL routing to enable testing"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/878#issuecomment-1073037939", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/878", "id": 1073037939, "node_id": "IC_kwDOBm6k_c4_9UJz", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-19T16:19:30Z", "updated_at": "2022-03-19T16:19:30Z", "author_association": "OWNER", "body": "On revisiting https://gist.github.com/simonw/281eac9c73b062c3469607ad86470eb2 a few months later I'm having second thoughts about using `@inject` on the `main()` method.\r\n\r\nBut I still like the pattern as a way to resolve more complex cases like \"to generate GeoJSON of the expanded view with labels, the label expansion code needs to run once at some before the GeoJSON formatting code does\".\r\n\r\nSo I'm going to stick with it a tiny bit longer, but maybe try to make it a lot more explicit when it's going to happen rather than having the main view methods themselves also use async DI.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 648435885, "label": "New pattern for views that return either JSON or HTML, available for plugins"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1561#issuecomment-1072939780", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1561", "id": 1072939780, "node_id": "IC_kwDOBm6k_c4_88ME", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-19T04:45:40Z", "updated_at": "2022-03-19T04:45:40Z", "author_association": "OWNER", "body": "I ended up moving hashed URL mode out to a plugin in:\r\n- #647\r\n\r\nIf you're still interested in using it with `_memory` please open an issue in that repo here: https://github.com/simonw/datasette-hashed-urls", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1082765654, "label": "add hash id to \"_memory\" url if hashed url mode is turned on and crossdb is also turned on"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1666#issuecomment-1072933875", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1666", "id": 1072933875, "node_id": "IC_kwDOBm6k_c4_86vz", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-19T04:03:42Z", "updated_at": "2022-03-19T04:03:42Z", "author_association": "OWNER", "body": "Tests so far: https://github.com/simonw/datasette/blob/711767bcd3c1e76a0861fe7f24069ff1c8efc97a/tests/test_routes.py#L12-L34", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1174162781, "label": "Refactor URL routing to enable testing"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1228#issuecomment-1072915936", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1228", "id": 1072915936, "node_id": "IC_kwDOBm6k_c4_82Xg", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-19T01:50:27Z", "updated_at": "2022-03-19T01:50:27Z", "author_association": "OWNER", "body": "Demo: https://latest.datasette.io/fixtures/facetable - which now has a column called `n`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 810397025, "label": "500 error caused by faceting if a column called `n` exists"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1228#issuecomment-1072908029", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1228", "id": 1072908029, "node_id": "IC_kwDOBm6k_c4_80b9", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-19T00:57:54Z", "updated_at": "2022-03-19T00:57:54Z", "author_association": "OWNER", "body": "Yes! That's the problem. I was able to replicate it like so:\r\n```\r\necho '[{ \r\n \"n\": \"one\",\r\n \"abc\": 1\r\n}, {\r\n \"n\": \"one\",\r\n \"abc\": 2\r\n}, {\r\n \"n\": \"two\",\r\n \"abc\": 3\r\n}]' | sqlite-utils insert column-called-n.db t -\r\n```\r\n\"image\"\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 810397025, "label": "500 error caused by faceting if a column called `n` exists"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1228#issuecomment-1072907680", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1228", "id": 1072907680, "node_id": "IC_kwDOBm6k_c4_80Wg", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-19T00:55:48Z", "updated_at": "2022-03-19T00:55:48Z", "author_association": "OWNER", "body": "... unless your data had a column called `n`?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 810397025, "label": "500 error caused by faceting if a column called `n` exists"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1228#issuecomment-1072907610", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1228", "id": 1072907610, "node_id": "IC_kwDOBm6k_c4_80Va", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-19T00:55:29Z", "updated_at": "2022-03-19T00:55:29Z", "author_association": "OWNER", "body": "It looks to me like something is causing the faceting query here to return a string when it was expected to return a number:\r\n\r\nhttps://github.com/simonw/datasette/blob/32963018e7edfab1233de7c7076c428d0e5c7813/datasette/facets.py#L153-L170\r\n\r\nI can't think of any way that a `count(*) as n` would turn into a string though!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 810397025, "label": "500 error caused by faceting if a column called `n` exists"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1605#issuecomment-1072907200", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1605", "id": 1072907200, "node_id": "IC_kwDOBm6k_c4_80PA", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-19T00:52:54Z", "updated_at": "2022-03-19T00:53:45Z", "author_association": "OWNER", "body": "Had a thought about the implementation of this: it could make a really neat plugin.\r\n\r\nSomething like `datasette-export` which adds a `export` command using https://docs.datasette.io/en/stable/plugin_hooks.html#register-commands-cli - then you could run:\r\n\r\n datasette export my-export-dir mydatabase.db -m metadata.json --template-dir templates/\r\n\r\nAnd the command would then:\r\n\r\n- Create a `Datasette()` instance with those databases/metadata/etc\r\n- Execute`await datasette.client.get(\"/\")` to get the homepage HTML\r\n- Parse the HTML using BeautifulSoup to find all `a[href]`, `link[href]`, `script[src]`, `img[src]` elements that reference a relative path as opposed to one that starts with `http://`\r\n- Write out the homepage to `my-export-dir/index.html`\r\n- Recursively fetch and dump all of the other pages and assets that it found too\r\n\r\nAll of that HTML parsing may be over-complicating things. It could alternatively accept options for which pages you want to export:\r\n\r\n```\r\ndatasette export my-export-dir \\\r\n mydatabase.db -m metadata.json --template-dir templates/ \\\r\n --path / \\\r\n --path /mydatabase ...\r\n```\r\n\r\nOr a really wild option: it could allow you to define the paths you want to export using a SQL query:\r\n\r\n```\r\ndatasette export my-export-dir \\\r\n mydatabase.db -m metadata.json --template-dir templates/ \\\r\n --sql \"\r\nselect '/' as path, 'index.html' as filename\r\n union all\r\nselect '/mydatabase/articles/' || id as path, 'article-' || id || '.html' as filename\r\nfrom articles\r\n union all\r\nselect '/mydatabase/tags/' || tag as path, 'tag-' || tag || '.html' as filename\r\nfrom tags\r\n\"\r\n```\r\nWhich would save these files:\r\n- `index.html` as the content of `/`\r\n- `article-1.html` (and more) as the content of `/mydatabase/articles/1`\r\n- `tag-python.html` (and more) as the content of `/mydatabase/tags/python`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1108671952, "label": "Scripted exports"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1662#issuecomment-1072905467", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1662", "id": 1072905467, "node_id": "IC_kwDOBm6k_c4_8zz7", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-19T00:42:23Z", "updated_at": "2022-03-19T00:42:23Z", "author_association": "OWNER", "body": "Those client-side SQLite tricks are _really_ neat.\r\n\r\n`datasette publish` defaults to configuring it so the raw SQLite database can be downloaded from `/fixtures.db` - and this issue updated it to be served with a CORS header that would allow client-side scripts to load the file:\r\n\r\n- #1057\r\n\r\nIf you're not going to run any server-side code at all you don't need Datasette for this - you can upload the SQLite database file to any static hosting with CORS headers and load it into the client that way.\r\n\r\nIn terms of static publishing, I do think there's something interesting about using Datasette to generate static sites. There's an issue discussing options for that over here:\r\n\r\n- #1605", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1170497629, "label": "[feature request] Publish to fully static website"}, "performed_via_github_app": null}