{"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/13#issuecomment-906646452", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13", "id": 906646452, "node_id": "IC_kwDOEhK-wc42ClO0", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-26T18:34:34Z", "updated_at": "2021-08-26T18:35:20Z", "author_association": "MEMBER", "body": "I tried this ampersand fix: https://regex101.com/r/ojU2H9/1\r\n```python\r\n\r\n# https://regex101.com/r/ojU2H9/1\r\n_invalid_ampersand_re = re.compile(r'&(?![a-z0-9]+;)')\r\n\r\ndef fix_bad_xml(xml):\r\n # More fixes for things like '&' not as part of an entity\r\n return _invalid_ampersand_re.sub('&', xml)\r\n```\r\n\r\nEven with that I'm still getting total garbage in the `` content - it's just HTML, not even trying to be XML.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 978743426, "label": "xml.etree.ElementTree.ParseError: not well-formed (invalid token)"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/13#issuecomment-906635938", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13", "id": 906635938, "node_id": "IC_kwDOEhK-wc42Ciqi", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-26T18:18:27Z", "updated_at": "2021-08-26T18:18:27Z", "author_association": "MEMBER", "body": "It looks like I was using the round-trip to dump the `` and ` Merging [#1448](https://codecov.io/gh/simonw/datasette/pull/1448?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (a211747) into [main](https://codecov.io/gh/simonw/datasette/commit/a1a33bb5822214be1cebd98cd858b2058d91a4aa?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (a1a33bb) will **not change** coverage.\n> The diff coverage is `n/a`.\n\n[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1448/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1448?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n\n```diff\n@@ Coverage Diff @@\n## main #1448 +/- ##\n=======================================\n Coverage 91.82% 91.82% \n=======================================\n Files 34 34 \n Lines 4418 4418 \n=======================================\n Hits 4057 4057 \n Misses 361 361 \n```\n\n\n\n------\n\n[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1448?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n> `\u0394 = absolute (impact)`, `\u00f8 = not affected`, `? = missing data`\n> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1448?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [a1a33bb...a211747](https://codecov.io/gh/simonw/datasette/pull/1448?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 980228553, "label": "Update pluggy requirement from ~=0.13.0 to >=0.13,<1.1"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/7#issuecomment-906015471", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/7", "id": 906015471, "node_id": "IC_kwDOD079W842ALLv", "user": {"value": 18232, "label": "dkam"}, "created_at": "2021-08-26T02:01:01Z", "updated_at": "2021-08-26T02:01:01Z", "author_association": "NONE", "body": "Perceptual hashes might be what you're after : http://phash.org", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602585497, "label": "Integrate image content hashing"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/859#issuecomment-905904540", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/859", "id": 905904540, "node_id": "IC_kwDOBm6k_c41_wGc", "user": {"value": 2670795, "label": "brandonrobertz"}, "created_at": "2021-08-25T21:59:14Z", "updated_at": "2021-08-25T21:59:55Z", "author_association": "CONTRIBUTOR", "body": "I did two tests: one with 1000 5-30mb DBs and a second with 20 multi gig DBs. For the second, I created them like so:\r\n`for i in {1..20}; do sqlite-generate db$i.db --tables ${i}00 --rows 100,2000 --columns 5,100 --pks 0 --fks 0; done`\r\n\r\nThis was for deciding whether to use lots of small DBs or to group things into a smaller number of bigger DBs. The second strategy wins.\r\n\r\nBy simply persisting the `_internal` DB to disk, I was able to avoid most of the performance issues I was experiencing previously. (To do this, I changed the `datasette/internal_db.py:init_internal_db` creates to if not exists, and changed the `_internal` DB instantiation in `datasette/app.py:Datasette.__init__` to a path with `is_mutable=True`.) Super rough, but the pages now load so I can continue testing ideas.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 642572841, "label": "Database page loads too slowly with many large tables (due to table counts)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/859#issuecomment-905900807", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/859", "id": 905900807, "node_id": "IC_kwDOBm6k_c41_vMH", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-25T21:51:10Z", "updated_at": "2021-08-25T21:51:10Z", "author_association": "OWNER", "body": "10-20 minutes to populate `_internal`! How many databases and tables is that for?\r\n\r\nI may have to rethink the `_internal` mechanism entirely. One possible alternative would be for the Datasette homepage to just show a list of available databases (maybe only if there are more than X connected) and then load in their metadata only the first time they are accessed.\r\n\r\nI need to get my own stress testing rig setup for this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 642572841, "label": "Database page loads too slowly with many large tables (due to table counts)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/859#issuecomment-905899177", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/859", "id": 905899177, "node_id": "IC_kwDOBm6k_c41_uyp", "user": {"value": 2670795, "label": "brandonrobertz"}, "created_at": "2021-08-25T21:48:00Z", "updated_at": "2021-08-25T21:48:00Z", "author_association": "CONTRIBUTOR", "body": "Upon first stab, there's two issues here:\r\n- DB/table/row counts (as discussed above). This isn't too bad if the DBs are actually above the MAX limit check.\r\n- Populating the internal DB. On first load of a giant set of DBs, it can take 10-20 mins to populate. By altering datasette and persisting the internal DB to disk, this problem is vastly improved, but I'm sure this will cause problems elsewhere.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 642572841, "label": "Database page loads too slowly with many large tables (due to table counts)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/323#issuecomment-905886797", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/323", "id": 905886797, "node_id": "IC_kwDOCGYnMM41_rxN", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-25T21:25:18Z", "updated_at": "2021-08-25T21:25:18Z", "author_association": "OWNER", "body": "As far as I can tell the Python `sqlite3` module doesn't actually have a mechanism for de-registering a custom SQL function.\r\n\r\nThis means that if I implement a mechanism whereby each call to `.convert()` registers a new SQL function with a random suffix (`convert_value_23424()` for example) those functions will stay registered - and if `.convert()` is called a large number of times the number of obsolete custom function registrations will grow without bounds.\r\n\r\nFor that reason, I'm going to `wontfix` this issue.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 979627285, "label": "`table.convert()` method should clean up after itself"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/13#issuecomment-905206234", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13", "id": 905206234, "node_id": "IC_kwDOEhK-wc419Fna", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-25T05:58:42Z", "updated_at": "2021-08-25T05:58:42Z", "author_association": "MEMBER", "body": "https://github.com/dogsheep/evernote-to-sqlite/blob/36a466f142e5bad52719851c2fbda0c05cd35b99/evernote_to_sqlite/utils.py#L34-L42\r\n\r\nNot sure why I was round-tripping the `content_xml` like that - I will try not doing that.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 978743426, "label": "xml.etree.ElementTree.ParseError: not well-formed (invalid token)"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/13#issuecomment-905203570", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13", "id": 905203570, "node_id": "IC_kwDOEhK-wc419E9y", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-25T05:51:22Z", "updated_at": "2021-08-25T05:53:27Z", "author_association": "MEMBER", "body": "The debugger showed me that it broke on a string that looked like this:\r\n```xml\r\n\r\n\r\n\r\n

\r\n \r\n Q3 2018 Reflection & Development\r\n \r\n

\r\n ...\r\n```\r\nYeah that is not valid XML!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 978743426, "label": "xml.etree.ElementTree.ParseError: not well-formed (invalid token)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1447#issuecomment-905097468", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1447", "id": 905097468, "node_id": "IC_kwDOBm6k_c418rD8", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-25T01:28:53Z", "updated_at": "2021-08-25T01:28:53Z", "author_association": "OWNER", "body": "Good catch, thanks!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 978614898, "label": "Remove underscore from search mode parameter name"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905043974", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/319", "id": 905043974, "node_id": "IC_kwDOCGYnMM418eAG", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-24T23:33:44Z", "updated_at": "2021-08-24T23:33:44Z", "author_association": "OWNER", "body": "Updated documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#inserting-data-from-files", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 976399638, "label": "[Enhancement] Please allow 'insert-files' to insert content as text."}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/321#issuecomment-905022931", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/321", "id": 905022931, "node_id": "IC_kwDOCGYnMM418Y3T", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2021-08-24T22:38:38Z", "updated_at": "2021-08-24T23:27:26Z", "author_association": "NONE", "body": "# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/321?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\n> Merging [#321](https://codecov.io/gh/simonw/sqlite-utils/pull/321?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (db2dd6d) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/9258f4bd8450c951900de998a7bf81ca9b45a014?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (9258f4b) will **increase** coverage by `0.16%`.\n> The diff coverage is `100.00%`.\n\n[![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/321/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/321?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n\n```diff\n@@ Coverage Diff @@\n## main #321 +/- ##\n==========================================\n+ Coverage 96.41% 96.58% +0.16% \n==========================================\n Files 5 5 \n Lines 2206 2223 +17 \n==========================================\n+ Hits 2127 2147 +20 \n+ Misses 79 76 -3 \n```\n\n\n| [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/321?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage \u0394 | |\n|---|---|---|\n| [sqlite\\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/321/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.69% <100.00%> (+0.42%)` | :arrow_up: |\n\n------\n\n[Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/321?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n> `\u0394 = absolute (impact)`, `\u00f8 = not affected`, `? = missing data`\n> Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/321?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [9258f4b...db2dd6d](https://codecov.io/gh/simonw/sqlite-utils/pull/321?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 978537855, "label": "Ability to insert file contents as text, in addition to blob"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/321#issuecomment-905040902", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/321", "id": 905040902, "node_id": "IC_kwDOCGYnMM418dQG", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-24T23:25:03Z", "updated_at": "2021-08-24T23:25:03Z", "author_association": "OWNER", "body": "I'm going to skip this test on windows.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 978537855, "label": "Ability to insert file contents as text, in addition to blob"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/321#issuecomment-905040307", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/321", "id": 905040307, "node_id": "IC_kwDOCGYnMM418dGz", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-24T23:23:36Z", "updated_at": "2021-08-24T23:23:36Z", "author_association": "OWNER", "body": "https://discuss.python.org/t/pep-597-use-utf-8-for-default-text-file-encoding/1819 says:\r\n\r\n> Currently, `TextIOWrapper` uses `locale.getpreferredencoding(False)` (hereinafter called \u201clocale encoding\u201d) when encoding is not specified.\r\n> ...\r\n> Package authors using macOS or Linux may forget that the default encoding\r\nis not always UTF-8.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 978537855, "label": "Ability to insert file contents as text, in addition to blob"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/321#issuecomment-905039576", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/321", "id": 905039576, "node_id": "IC_kwDOCGYnMM418c7Y", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-24T23:21:29Z", "updated_at": "2021-08-24T23:21:29Z", "author_association": "OWNER", "body": "Hah, the error here is actually:\r\n```\r\n> assert result.exit_code == 1, result.output\r\nE AssertionError: \r\nE \r\nE assert 0 == 1\r\nE + where 0 = .exit_code\r\n```\r\nSo I was expecting an error, but instead the command worked.\r\n\r\nI suspect this is because on Windows the default character set may not be UTF-8?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 978537855, "label": "Ability to insert file contents as text, in addition to blob"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/321#issuecomment-905037323", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/321", "id": 905037323, "node_id": "IC_kwDOCGYnMM418cYL", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-24T23:15:29Z", "updated_at": "2021-08-24T23:15:29Z", "author_association": "OWNER", "body": "Huh, tests are failing but only on Windows!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 978537855, "label": "Ability to insert file contents as text, in addition to blob"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905024066", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/319", "id": 905024066, "node_id": "IC_kwDOCGYnMM418ZJC", "user": {"value": 66709385, "label": "pjamargh"}, "created_at": "2021-08-24T22:41:39Z", "updated_at": "2021-08-24T22:41:39Z", "author_association": "NONE", "body": "I'm happy with this functionality left the way you describe. In my case the data is homogeneous but other cases would work just by being consistent on the encoding. Thanks a lot, Simon!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 976399638, "label": "[Enhancement] Please allow 'insert-files' to insert content as text."}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905021933", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/319", "id": 905021933, "node_id": "IC_kwDOCGYnMM418Ynt", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-24T22:36:04Z", "updated_at": "2021-08-24T22:36:04Z", "author_association": "OWNER", "body": "> Oh, I misread. Yes some files will not be valid UTF-8, I'd throw a warning and continue (not adding that file) but if you want to get more elaborate you could allow to define a policy on what to do. Not adding the file, index binary content or use a conversion policy like the ones available on Python's decode.\r\n\r\nI thought about supporting those different policies (with something like `--errors ignore`) but I feel like that's getting a little bit too deep into the weeds. Right now if you try to import an invalid file the behaviour is the same as for the `sqlite-utils insert` command (I added the same detailed error message):\r\n\r\n```\r\nError: Could not read file '/Users/simon/Dropbox/Development/sqlite-utils/data.txt' as text\r\n\r\n'utf-8' codec can't decode byte 0xe3 in position 83: invalid continuation byte\r\n\r\nThe input you provided uses a character encoding other than utf-8.\r\n\r\nYou can fix this by passing the --encoding= option with the encoding of the file.\r\n\r\nIf you do not know the encoding, running 'file filename.csv' may tell you.\r\n\r\nIt's often worth trying: --encoding=latin-1\r\n```\r\nIf someone has data that can't be translated to valid text using a known encoding, I'm happy leaving them to have to insert it into a `BLOB` column instead.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 976399638, "label": "[Enhancement] Please allow 'insert-files' to insert content as text."}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905021047", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/319", "id": 905021047, "node_id": "IC_kwDOCGYnMM418YZ3", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-24T22:33:48Z", "updated_at": "2021-08-24T22:33:48Z", "author_association": "OWNER", "body": "I had a few doubts about the design just now. Since `content_text` is supported as a special argument, an alternative way of handling the above would be:\r\n\r\n sqlite-utils insert-files /tmp/text.db files *.txt -c path -c content_text -c size\r\n\r\nThis does exactly the same thing as just using `--text` and not specifying any columns, because the actual implementation of `--text` is as follows:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/0c796cd945b146b7395ff5f553861400be503867/sqlite_utils/cli.py#L1851-L1855\r\n\r\nBut actually I think that's OK - ``--text`` is a useful shorthand that avoids you having to remember how to manually specify those columns with `-c`. So I'm going to leave the design as is.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 976399638, "label": "[Enhancement] Please allow 'insert-files' to insert content as text."}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905021010", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/319", "id": 905021010, "node_id": "IC_kwDOCGYnMM418YZS", "user": {"value": 66709385, "label": "pjamargh"}, "created_at": "2021-08-24T22:33:42Z", "updated_at": "2021-08-24T22:33:42Z", "author_association": "NONE", "body": "Oh, I misread. Yes some files will not be valid UTF-8, I'd throw a warning and continue (not adding that file) but if you want to get more elaborate you could allow to define a policy on what to do. Not adding the file, index binary content or use a conversion policy like the ones available on Python's decode.\r\n\r\nFrom https://stackoverflow.com/questions/24616678/unicodedecodeerror-in-python-when-reading-a-file-how-to-ignore-the-error-and-ju :\r\n - 'ignore' ignores errors. Note that ignoring encoding errors can lead to data loss.\r\n - 'replace' causes a replacement marker (such as '?') to be inserted where there is malformed data.\r\n - 'surrogateescape' will represent any incorrect bytes as code points in the Unicode Private Use Area ranging from U+DC80 to U+DCFF. These private code points will then be turned back into the same bytes when the surrogateescape error handler is used when writing data. This is useful for processing files in an unknown encoding.\r\n - 'xmlcharrefreplace' is only supported when writing to a file. Characters not supported by the encoding are replaced with the appropriate XML character reference &#nnn;.\r\n - 'backslashreplace' (also only supported when writing) replaces unsupported characters with Python\u2019s backslashed escape sequences.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 976399638, "label": "[Enhancement] Please allow 'insert-files' to insert content as text."}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905013183", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/319", "id": 905013183, "node_id": "IC_kwDOCGYnMM418We_", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-24T22:15:34Z", "updated_at": "2021-08-24T22:15:34Z", "author_association": "OWNER", "body": "Here's the error message I have working for invalid unicode:\r\n```\r\nsqlite-utils insert-files /tmp/text.db files *.txt --text\r\n [------------------------------------] 0%\r\nError: Could not read file '/Users/simon/Dropbox/Development/sqlite-utils/data.txt' as text\r\n\r\n'utf-8' codec can't decode byte 0xe3 in position 83: invalid continuation byte\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 976399638, "label": "[Enhancement] Please allow 'insert-files' to insert content as text."}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905013162", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/319", "id": 905013162, "node_id": "IC_kwDOCGYnMM418Weq", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-24T22:15:31Z", "updated_at": "2021-08-24T22:15:31Z", "author_association": "OWNER", "body": "I'm going to assume utf-8 but allow `--encoding` to be used to specify something different, since that option is already supported by other commands.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 976399638, "label": "[Enhancement] Please allow 'insert-files' to insert content as text."}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905003381", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/319", "id": 905003381, "node_id": "IC_kwDOCGYnMM418UF1", "user": {"value": 66709385, "label": "pjamargh"}, "created_at": "2021-08-24T21:56:49Z", "updated_at": "2021-08-24T21:56:49Z", "author_association": "NONE", "body": "I was thinking that an approach could be making FILE_COLUMNS a generator (_get_file_columns(mode)) or you can just have a different set of columns (is there something else that makes sense to be changed on the text scenario?).\r\n\r\nAbout UTF-8 I was referring to the encoding to use when reading files. This can be difficult to auto-detect but I believe that UTF-8 is pretty much the standard for text files.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 976399638, "label": "[Enhancement] Please allow 'insert-files' to insert content as text."}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905001586", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/319", "id": 905001586, "node_id": "IC_kwDOCGYnMM418Tpy", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-24T21:52:50Z", "updated_at": "2021-08-24T21:52:50Z", "author_association": "OWNER", "body": "Will need to re-title this section of the documentation: https://sqlite-utils.datasette.io/en/3.16/cli.html#inserting-binary-data-from-files - \"Inserting binary data from files\" will become \"Inserting data from files\"\r\n\r\nI'm OK with keeping the default as `BLOB` but I could add a `--text` option which stores the content as text instead.\r\n\r\nIf the text can't be stored as `utf-8` I'll probably raise an error.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 976399638, "label": "[Enhancement] Please allow 'insert-files' to insert content as text."}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/319#issuecomment-904999850", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/319", "id": 904999850, "node_id": "IC_kwDOCGYnMM418TOq", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-24T21:49:08Z", "updated_at": "2021-08-24T21:49:08Z", "author_association": "OWNER", "body": "This is a good idea.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 976399638, "label": "[Enhancement] Please allow 'insert-files' to insert content as text."}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/859#issuecomment-904982056", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/859", "id": 904982056, "node_id": "IC_kwDOBm6k_c418O4o", "user": {"value": 2670795, "label": "brandonrobertz"}, "created_at": "2021-08-24T21:15:04Z", "updated_at": "2021-08-24T21:15:30Z", "author_association": "CONTRIBUTOR", "body": "I'm running into issues with this as well. All other pages seem to work with lots of DBs except the home page, which absolutely tanks. Would be willing to put some work into this, if there's been any kind of progress on concepts on how this ought to work.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 642572841, "label": "Database page loads too slowly with many large tables (due to table counts)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1446#issuecomment-904954530", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1446", "id": 904954530, "node_id": "IC_kwDOBm6k_c418IKi", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-24T20:32:47Z", "updated_at": "2021-08-24T20:32:47Z", "author_association": "OWNER", "body": "Pasting that CSS into the styles editor in the developer tools on https://latest.datasette.io/ has the desired effect: footer at the bottom of the window unless the page is too long, in which case the footer is at the bottom of the scroll.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 978357984, "label": "Modify base.html template to support optional sticky footer"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1446#issuecomment-904866495", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1446", "id": 904866495, "node_id": "IC_kwDOBm6k_c417yq_", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-24T18:13:49Z", "updated_at": "2021-08-24T18:13:49Z", "author_association": "OWNER", "body": "OK, now the following optional CSS gives us a sticky footer:\r\n```css\r\nhtml, body {\r\n height: 100%;\r\n}\r\nbody {\r\n display: flex;\r\n flex-direction: column;\r\n}\r\n.not-footer {\r\n flex: 1 0 auto;\r\n}\r\nfooter {\r\n flex-shrink: 0;\r\n}\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 978357984, "label": "Modify base.html template to support optional sticky footer"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/healthkit-to-sqlite/pull/13#issuecomment-904642396", "issue_url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/13", "id": 904642396, "node_id": "IC_kwDOC8tyDs41679c", "user": {"value": 32016596, "label": "FabianHertwig"}, "created_at": "2021-08-24T13:27:40Z", "updated_at": "2021-08-24T13:28:26Z", "author_association": "NONE", "body": "This would fix #21 and make #22 obsolete.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 743071410, "label": "SQLite does not have case sensitive columns"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/healthkit-to-sqlite/pull/22#issuecomment-904641261", "issue_url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/22", "id": 904641261, "node_id": "IC_kwDOC8tyDs4167rt", "user": {"value": 32016596, "label": "FabianHertwig"}, "created_at": "2021-08-24T13:26:20Z", "updated_at": "2021-08-24T13:26:20Z", "author_association": "NONE", "body": "Did not see that #13 fixes the same issue in a similar way. You can decide which one to merge ;)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 978086284, "label": "Make sure that case-insensitive column names are unique"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1445#issuecomment-904037087", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1445", "id": 904037087, "node_id": "IC_kwDOBm6k_c414oLf", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-23T19:10:17Z", "updated_at": "2021-08-23T19:10:17Z", "author_association": "OWNER", "body": "Rather than trying to run that monstrosity in a single `union all` query, a better approach may be to use `fetch()` requests as seen in https://datasette.io/plugins/datasette-search-all", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 977323133, "label": "Ability to search for text across all columns in a table"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1445#issuecomment-904036200", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1445", "id": 904036200, "node_id": "IC_kwDOBm6k_c414n9o", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-23T19:08:54Z", "updated_at": "2021-08-23T19:08:54Z", "author_association": "OWNER", "body": "Figured out a query for searching across every column in every table! https://til.simonwillison.net/datasette/search-all-columns-trick#user-content-same-trick-for-the-entire-database\r\n\r\n```sql\r\nwith tables as (\r\n select\r\n name as table_name\r\n from\r\n sqlite_master\r\n where\r\n type = 'table'\r\n),\r\nqueries as (\r\n select\r\n 'select ''' || tables.table_name || ''' as _table, rowid from \"' || tables.table_name || '\" where ' || group_concat(\r\n '\"' || name || '\" like ''%'' || :search || ''%''',\r\n ' or '\r\n ) as query\r\n from\r\n pragma_table_info(tables.table_name),\r\n tables\r\n group by\r\n tables.table_name\r\n)\r\nselect\r\n group_concat(query, ' union all ')\r\nfrom\r\n queries\r\n```\r\nThe SQL query this generates for larger databases is _extremely_ long - but it does seem to work for smaller databases.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 977323133, "label": "Ability to search for text across all columns in a table"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1445#issuecomment-904027166", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1445", "id": 904027166, "node_id": "IC_kwDOBm6k_c414lwe", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-23T18:56:20Z", "updated_at": "2021-08-23T18:56:20Z", "author_association": "OWNER", "body": "A related but potentially even more useful ability would be running a search across every column of every table in a whole database. For anything less than a few 100MB this could be incredibly useful.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 977323133, "label": "Ability to search for text across all columns in a table"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1445#issuecomment-904026253", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1445", "id": 904026253, "node_id": "IC_kwDOBm6k_c414liN", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-23T18:54:49Z", "updated_at": "2021-08-23T18:54:49Z", "author_association": "OWNER", "body": "The bigger problem here is UI design. This feels like a pretty niche requirement to me, so adding a prominent search box to the table page (which already has the filters interface, plus the full-text search box for tables that have FTS configured) feels untidy.\r\n\r\nI could tuck it away in the table cog menu, but that's a weird place for something like this to live.\r\n\r\nMaybe add it as a new type of filter? Filters apply to specific columns though, so this would be the first filter that applied to _all_ columns - which doesn't really fit the existing filter interface very well.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 977323133, "label": "Ability to search for text across all columns in a table"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1445#issuecomment-904024939", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1445", "id": 904024939, "node_id": "IC_kwDOBm6k_c414lNr", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-23T18:52:35Z", "updated_at": "2021-08-23T18:52:35Z", "author_association": "OWNER", "body": "The downside of the current implementation of this trick is that it only works for exact LIKE partial matches in a specific table - if you search for `dog cat` and `dog` appears in `title` but `cat` appears in `description` you won't get back that result.\r\n\r\nI think that's fine though. If you want more advanced search there are other mechanisms you can use. This is meant to be a very quick and dirty starting point for exploring a brand new table.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 977323133, "label": "Ability to search for text across all columns in a table"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/healthkit-to-sqlite/issues/21#issuecomment-903950096", "issue_url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/21", "id": 903950096, "node_id": "IC_kwDOC8tyDs414S8Q", "user": {"value": 32016596, "label": "FabianHertwig"}, "created_at": "2021-08-23T17:00:59Z", "updated_at": "2021-08-23T17:00:59Z", "author_association": "NONE", "body": "I think the issue is that I have records like these:\r\n\r\n```xml\r\n \r\n \r\n \r\n \r\n```\r\n\r\nAnd if sqlite is case insensitive, then `metadata_meal` and `metadata_Meal` result in the same column.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 977128935, "label": "Duplicate Column"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/320#issuecomment-903288691", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/320", "id": 903288691, "node_id": "IC_kwDOCGYnMM411xdz", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-22T15:46:56Z", "updated_at": "2021-08-22T15:46:56Z", "author_association": "OWNER", "body": "Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#schema-analyze-dump-and-save", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 976405225, "label": "sqlite-utils memory --analyze option"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/320#issuecomment-903288430", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/320", "id": 903288430, "node_id": "IC_kwDOCGYnMM411xZu", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-22T15:44:55Z", "updated_at": "2021-08-22T15:45:52Z", "author_association": "OWNER", "body": "```\r\ncurl 'https://api.github.com/users/dogsheep/repos' | sqlite-utils memory - --analyze\r\n```\r\n```\r\nstdin.id: (1/73)\r\n\r\n Total rows: 13\r\n Null rows: 0\r\n Blank rows: 0\r\n\r\n Distinct values: 13\r\n\r\nstdin.node_id: (2/73)\r\n\r\n Total rows: 13\r\n Null rows: 0\r\n Blank rows: 0\r\n\r\n Distinct values: 13\r\n\r\nstdin.name: (3/73)\r\n\r\n Total rows: 13\r\n Null rows: 0\r\n Blank rows: 0\r\n\r\n Distinct values: 13\r\n\r\nstdin.full_name: (4/73)\r\n\r\n Total rows: 13\r\n Null rows: 0\r\n Blank rows: 0\r\n\r\n Distinct values: 13\r\n\r\nstdin.private: (5/73)\r\n\r\n Total rows: 13\r\n Null rows: 0\r\n Blank rows: 0\r\n\r\n Distinct values: 1\r\n\r\n Most common:\r\n 13: 0\r\n\r\nstdin.owner: (6/73)\r\n\r\n Total rows: 13\r\n Null rows: 0\r\n Blank rows: 0\r\n\r\n Distinct values: 1\r\n\r\n Most common:\r\n 13: {\"login\": \"dogsheep\", \"id\": 53015001, \"node_id\": \"MDEyOk9yZ2FuaXphdGlvbjUzMDE1MD...\r\n\r\nstdin.html_url: (7/73)\r\n\r\n Total rows: 13\r\n Null rows: 0\r\n Blank rows: 0\r\n\r\n Distinct values: 13\r\n\r\nstdin.description: (8/73)\r\n\r\n Total rows: 13\r\n Null rows: 0\r\n Blank rows: 0\r\n\r\n Distinct values: 13\r\n\r\nstdin.fork: (9/73)\r\n\r\n Total rows: 13\r\n Null rows: 0\r\n Blank rows: 0\r\n\r\n Distinct values: 1\r\n\r\n Most common:\r\n 13: 0\r\n\r\nstdin.url: (10/73)\r\n\r\n Total rows: 13\r\n Null rows: 0\r\n Blank rows: 0\r\n\r\n Distinct values: 13\r\n\r\nstdin.forks_url: (11/73)\r\n\r\n Total rows: 13\r\n Null rows: 0\r\n Blank rows: 0\r\n\r\n Distinct values: 13\r\n\r\nstdin.keys_url: (12/73)\r\n\r\n Total rows: 13\r\n...\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 976405225, "label": "sqlite-utils memory --analyze option"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/894#issuecomment-902375388", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/894", "id": 902375388, "node_id": "IC_kwDOBm6k_c41ySfc", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-20T02:07:53Z", "updated_at": "2021-08-20T02:07:53Z", "author_association": "OWNER", "body": "I could add these sorting links to the cog menu for any `TEXT` columns.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 657572753, "label": "?sort=colname~numeric to sort by by column cast to real"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/894#issuecomment-902375088", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/894", "id": 902375088, "node_id": "IC_kwDOBm6k_c41ySaw", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-20T02:07:13Z", "updated_at": "2021-08-20T02:07:26Z", "author_association": "OWNER", "body": "Maybe `?_sort_numeric=col` and `?_sort_numeric_desc=col` would be better here.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 657572753, "label": "?sort=colname~numeric to sort by by column cast to real"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/healthkit-to-sqlite/issues/20#issuecomment-902356871", "issue_url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/20", "id": 902356871, "node_id": "IC_kwDOC8tyDs41yN-H", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-20T01:12:48Z", "updated_at": "2021-08-20T01:12:48Z", "author_association": "MEMBER", "body": "Also on `workout_points.workout_id` to speed up queries to show all points in a specific workout.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 975166271, "label": "Add index on workout_points.date"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/healthkit-to-sqlite/issues/20#issuecomment-902355471", "issue_url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/20", "id": 902355471, "node_id": "IC_kwDOC8tyDs41yNoP", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-20T01:09:07Z", "updated_at": "2021-08-20T01:09:07Z", "author_association": "MEMBER", "body": "Workaround:\r\n\r\n sqlite-utils create-index healthkit.db workout_points -- -date\r\n\r\nSee https://sqlite-utils.datasette.io/en/stable/cli.html#creating-indexes", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 975166271, "label": "Add index on workout_points.date"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/pull/49#issuecomment-902330301", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/49", "id": 902330301, "node_id": "IC_kwDODEm0Qs41yHe9", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-20T00:01:56Z", "updated_at": "2021-08-20T00:01:56Z", "author_association": "MEMBER", "body": "Thanks!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 681575714, "label": "Document the use of --stop_after with favorites, refs #20"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902329884", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57", "id": 902329884, "node_id": "IC_kwDODEm0Qs41yHYc", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-20T00:01:05Z", "updated_at": "2021-08-20T00:01:05Z", "author_association": "MEMBER", "body": "Maybe Click changed something which meant that this broke things when it didn't used to?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 907645813, "label": "Error: Use either --since or --since_id, not both"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902329455", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57", "id": 902329455, "node_id": "IC_kwDODEm0Qs41yHRv", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-19T23:59:56Z", "updated_at": "2021-08-19T23:59:56Z", "author_association": "MEMBER", "body": "This looks like the bug to me:\r\n\r\nhttps://github.com/dogsheep/twitter-to-sqlite/blob/197e69cec40052c423a5ed071feb5f7cccea41b9/twitter_to_sqlite/cli.py#L239-L241\r\n\r\n`type=str, default=False`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 907645813, "label": "Error: Use either --since or --since_id, not both"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902328760", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57", "id": 902328760, "node_id": "IC_kwDODEm0Qs41yHG4", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-19T23:57:41Z", "updated_at": "2021-08-19T23:57:41Z", "author_association": "MEMBER", "body": "Weird, added debug code and got this: `{'screen_name': 'simonw', 'count': 200, 'since_id': 'False', 'tweet_mode': 'extended'}` - so maybe it's a `twitter-to-sqlite` bug where somehow the string `False` is being passed somewhere.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 907645813, "label": "Error: Use either --since or --since_id, not both"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902328369", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57", "id": 902328369, "node_id": "IC_kwDODEm0Qs41yHAx", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-19T23:56:26Z", "updated_at": "2021-08-19T23:56:26Z", "author_association": "MEMBER", "body": "https://developer.twitter.com/en/docs/twitter-api/v1/tweets/timelines/api-reference/get-statuses-user_timeline says the API has been replaced by the new v2 one, but it should still work - and the `since_id` parameter is still documented on that page.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 907645813, "label": "Error: Use either --since or --since_id, not both"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902327457", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57", "id": 902327457, "node_id": "IC_kwDODEm0Qs41yGyh", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-19T23:53:25Z", "updated_at": "2021-08-19T23:53:25Z", "author_association": "MEMBER", "body": "I'm getting this too. Looking into it now.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 907645813, "label": "Error: Use either --since or --since_id, not both"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1426#issuecomment-902263367", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1426", "id": 902263367, "node_id": "IC_kwDOBm6k_c41x3JH", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-19T21:33:51Z", "updated_at": "2021-08-19T21:36:28Z", "author_association": "OWNER", "body": "I was worried about if it's possible to allow access to `/fixtures` but deny access to `/fixtures?sql=...`\r\n\r\nFrom various answers on Stack Overflow it looks like this should handle that:\r\n\r\n```\r\nUser-agent: *\r\nDisallow: /fixtures?\r\n```\r\nI could use this for tables too - it may well be OK to access table index pages while still avoiding pagination, facets etc. I think this should block both query strings and row pages while allowing the table page itself:\r\n```\r\nUser-agent: *\r\nDisallow: /fixtures/searchable?\r\nDisallow: /fixtures/searchable/*\r\n```\r\nCould even accompany that with a `sitemap.xml` that explicitly lists all of the tables - which would mean adding sitemaps to Datasette core too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 964322136, "label": "Manage /robots.txt in Datasette core, block robots by default"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1426#issuecomment-902260338", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1426", "id": 902260338, "node_id": "IC_kwDOBm6k_c41x2Zy", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-19T21:28:25Z", "updated_at": "2021-08-19T21:29:40Z", "author_association": "OWNER", "body": "Actually it looks like you can send a `sitemap.xml` to Google using an unauthenticated GET request to:\r\n\r\n https://www.google.com/ping?sitemap=FULL_URL_OF_SITEMAP\r\n\r\nAccording to https://developers.google.com/search/docs/advanced/sitemaps/build-sitemap", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 964322136, "label": "Manage /robots.txt in Datasette core, block robots by default"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1426#issuecomment-902260799", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1426", "id": 902260799, "node_id": "IC_kwDOBm6k_c41x2g_", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-19T21:29:13Z", "updated_at": "2021-08-19T21:29:13Z", "author_association": "OWNER", "body": "Bing's equivalent is: https://www.bing.com/webmasters/help/Sitemaps-3b5cf6ed\r\n\r\n http://www.bing.com/ping?sitemap=FULL_URL_OF_SITEMAP", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 964322136, "label": "Manage /robots.txt in Datasette core, block robots by default"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1443#issuecomment-902258509", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1443", "id": 902258509, "node_id": "IC_kwDOBm6k_c41x19N", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-19T21:25:07Z", "updated_at": "2021-08-19T21:25:07Z", "author_association": "OWNER", "body": "https://docs.datasette.io/en/latest/internals.html#databases", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 974995592, "label": "datasette.databases should be a documented property"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1434#issuecomment-902254712", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1434", "id": 902254712, "node_id": "IC_kwDOBm6k_c41x1B4", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-19T21:18:31Z", "updated_at": "2021-08-19T21:18:57Z", "author_association": "OWNER", "body": "I deployed a demo to https://datasette-latest-query-info-j7hipcg4aq-uc.a.run.app using the mechanism from #1442.\r\n\r\ne.g. demo here: https://datasette-latest-query-info-j7hipcg4aq-uc.a.run.app/fixtures?sql=select+*+from+searchable", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 970463436, "label": "Enrich arbitrary query results with foreign key links and column descriptions"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1415#issuecomment-902251316", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1415", "id": 902251316, "node_id": "IC_kwDOBm6k_c41x0M0", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-19T21:14:15Z", "updated_at": "2021-08-19T21:14:15Z", "author_association": "OWNER", "body": "https://github.com/ahmetb/cloud-run-faq#how-do-i-continuously-deploy-to-cloud-run suggests the following:\r\n\r\n> - `roles/run.admin` to deploy applications\r\n> - `roles/iam.serviceAccountUser` on the service account that your app will use\r\n\r\nIt also links to https://cloud.google.com/run/docs/reference/iam/roles", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 959137143, "label": "feature request: document minimum permissions for service account for cloudrun"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1415#issuecomment-902250361", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1415", "id": 902250361, "node_id": "IC_kwDOBm6k_c41xz95", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-19T21:12:28Z", "updated_at": "2021-08-19T21:12:28Z", "author_association": "OWNER", "body": "I would love to know this too! I always find figuring out minimal permissions to be really difficult.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 959137143, "label": "feature request: document minimum permissions for service account for cloudrun"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1442#issuecomment-902243498", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1442", "id": 902243498, "node_id": "IC_kwDOBm6k_c41xySq", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-19T21:04:01Z", "updated_at": "2021-08-19T21:04:01Z", "author_association": "OWNER", "body": "That successfully deployed to https://datasette-latest-deploy-this-branch-j7hipcg4aq-uc.a.run.app/ even though the tests failed.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 974987856, "label": "Mechanism to cause specific branches to deploy their own demos"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1442#issuecomment-902239215", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1442", "id": 902239215, "node_id": "IC_kwDOBm6k_c41xxPv", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-19T20:56:46Z", "updated_at": "2021-08-19T20:56:46Z", "author_association": "OWNER", "body": "I'm going to only run the tests if it's a push to `main` - that way I can ship demo branches really quickly, even if they don't yet have passing tests.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 974987856, "label": "Mechanism to cause specific branches to deploy their own demos"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1442#issuecomment-902235714", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1442", "id": 902235714, "node_id": "IC_kwDOBm6k_c41xwZC", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-19T20:50:38Z", "updated_at": "2021-08-19T20:50:38Z", "author_association": "OWNER", "body": "Would this allow anyone to push a PR to this repo that would result in their code being deployed against my Cloud Run account? I'm reasonably confident that it would not, since the secrets would not be visible to their PR branch.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 974987856, "label": "Mechanism to cause specific branches to deploy their own demos"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1442#issuecomment-902231018", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1442", "id": 902231018, "node_id": "IC_kwDOBm6k_c41xvPq", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-19T20:42:08Z", "updated_at": "2021-08-19T20:42:08Z", "author_association": "OWNER", "body": "If I get this working I should document it on https://docs.datasette.io/en/stable/contributing.html", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 974987856, "label": "Mechanism to cause specific branches to deploy their own demos"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1442#issuecomment-902217726", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1442", "id": 902217726, "node_id": "IC_kwDOBm6k_c41xr_-", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-19T20:21:47Z", "updated_at": "2021-08-19T20:21:47Z", "author_association": "OWNER", "body": "I think the neatest way to implement this would be for the `on -> push -> branches` list to be the list of branches that should be deployed in this way. The rest of the code can react to that.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 974987856, "label": "Mechanism to cause specific branches to deploy their own demos"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1442#issuecomment-902191150", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1442", "id": 902191150, "node_id": "IC_kwDOBm6k_c41xlgu", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-19T19:43:05Z", "updated_at": "2021-08-19T19:43:59Z", "author_association": "OWNER", "body": "Maybe as simple as teaching https://github.com/simonw/datasette/blob/main/.github/workflows/deploy-latest.yml to run on pushes to ALL branches:\r\n\r\nhttps://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/.github/workflows/deploy-latest.yml#L3-L6\r\n\r\nAnd then quit early if the branch is not in some allow-list.\r\n\r\nIf it IS in the allow-list, use the name of the branch to dynamically construct the name of the Cloud Run service here: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/.github/workflows/deploy-latest.yml#L60\r\n\r\nNeed to skip the documentation build and deployment stuff for other branches though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 974987856, "label": "Mechanism to cause specific branches to deploy their own demos"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-901475812", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 901475812, "node_id": "IC_kwDOBm6k_c41u23k", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-18T22:41:19Z", "updated_at": "2021-08-18T22:41:19Z", "author_association": "OWNER", "body": "> Maybe I split this out into a separate Python library that gets tested against _every_ SQLite release I can possibly try it against, and then bakes out the supported release versions into the library code itself?\r\n\r\nI'm going to do this, and call the Python library `sqlite-explain`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/37#issuecomment-901452199", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/37", "id": 901452199, "node_id": "IC_kwDOCGYnMM41uxGn", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-18T21:48:57Z", "updated_at": "2021-08-18T21:48:57Z", "author_association": "OWNER", "body": "I did a bunch of work on this in #266. The library is now pretty thoroughly typed, and I even found a couple of bugs using `mypy` along the way: #313 and #315.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 465815372, "label": "Experiment with type hints"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/318#issuecomment-901440752", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/318", "id": 901440752, "node_id": "IC_kwDOCGYnMM41uuTw", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-18T21:25:30Z", "updated_at": "2021-08-18T21:25:30Z", "author_association": "OWNER", "body": "Some questions:\r\n\r\n- Should this support compression formats other than gzip?\r\n- Should `memory` learn to auto-detect gzipped data?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 974067156, "label": "Research: handle gzipped CSV directly"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/318#issuecomment-901440207", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/318", "id": 901440207, "node_id": "IC_kwDOCGYnMM41uuLP", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-18T21:24:28Z", "updated_at": "2021-08-18T21:24:49Z", "author_association": "OWNER", "body": "Something like this then:\r\n\r\n sqlite-utils file.db \"select * from t\" --csv --gz > t.csv.gz\r\n\r\nMaybe add a `-o t.csv.gz` option too so you don't have to use a `>`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 974067156, "label": "Research: handle gzipped CSV directly"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/295#issuecomment-901403298", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/295", "id": 901403298, "node_id": "IC_kwDOCGYnMM41ulKi", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-18T20:19:04Z", "updated_at": "2021-08-18T20:19:04Z", "author_association": "OWNER", "body": "Thanks, this was a bug.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 934123448, "label": "Insert with --tsv and --no-headers give error about --nl arguments"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/296#issuecomment-901399139", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/296", "id": 901399139, "node_id": "IC_kwDOCGYnMM41ukJj", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-18T20:12:34Z", "updated_at": "2021-08-18T20:13:12Z", "author_association": "OWNER", "body": "Documentation for `table.search(..., quote=True)`: https://sqlite-utils.datasette.io/en/latest/python-api.html#searching-with-table-search\r\n\r\nIn the API reference: https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Table.search\r\n\r\nAnd for the CLI `--quote` option: https://sqlite-utils.datasette.io/en/latest/cli.html#executing-searches", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 944326512, "label": "`table.search(..., quote=True)` parameter and `sqlite-utils search --quote` option"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/296#issuecomment-901398216", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/296", "id": 901398216, "node_id": "IC_kwDOCGYnMM41uj7I", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-18T20:11:01Z", "updated_at": "2021-08-18T20:11:01Z", "author_association": "OWNER", "body": "```\r\n% sqlite-utils search fixtures.db searchable 'dog\"'\r\nError: malformed MATCH expression: [dog\"]\r\n\r\nTry running this again with the --quote option\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 944326512, "label": "`table.search(..., quote=True)` parameter and `sqlite-utils search --quote` option"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/296#issuecomment-901390635", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/296", "id": 901390635, "node_id": "IC_kwDOCGYnMM41uiEr", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-18T19:58:53Z", "updated_at": "2021-08-18T19:58:53Z", "author_association": "OWNER", "body": "```\r\nsqlite-utils search fixtures.db searchable 'dog\"'\r\nError: malformed MATCH expression: [dog\"]\r\n```\r\nThis error message could suggest retrying with `--quote`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 944326512, "label": "`table.search(..., quote=True)` parameter and `sqlite-utils search --quote` option"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/296#issuecomment-901379930", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/296", "id": 901379930, "node_id": "IC_kwDOCGYnMM41ufda", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-18T19:40:38Z", "updated_at": "2021-08-18T19:40:38Z", "author_association": "OWNER", "body": "Also add `sqlite-utils search ... --quote` option.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 944326512, "label": "`table.search(..., quote=True)` parameter and `sqlite-utils search --quote` option"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/246#issuecomment-901353345", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/246", "id": 901353345, "node_id": "IC_kwDOCGYnMM41uY-B", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-18T18:57:13Z", "updated_at": "2021-08-18T18:57:13Z", "author_association": "OWNER", "body": "More documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#quoting-characters-for-use-in-search", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 831751367, "label": "Escaping FTS search strings"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/296#issuecomment-901338841", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/296", "id": 901338841, "node_id": "IC_kwDOCGYnMM41uVbZ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-18T18:33:26Z", "updated_at": "2021-08-18T18:45:12Z", "author_association": "OWNER", "body": "I think I'll do this as an optional `table.search(..., escape=True)` parameter.\r\n\r\nActually I'll do `quote=True` for consistency with the new `db.quote_fts()` method.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 944326512, "label": "`table.search(..., quote=True)` parameter and `sqlite-utils search --quote` option"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/246#issuecomment-901345800", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/246", "id": 901345800, "node_id": "IC_kwDOCGYnMM41uXII", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-18T18:44:48Z", "updated_at": "2021-08-18T18:44:48Z", "author_association": "OWNER", "body": "The `db.quote_fts(value)` method from #247 can now be used for this - documentation here: https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Database.quote_fts\r\n\r\nI'll be adding further improvements relating to this (a `table.search(q, quote=True)` parameter) in #296.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 831751367, "label": "Escaping FTS search strings"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/247#issuecomment-901344634", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/247", "id": 901344634, "node_id": "IC_kwDOCGYnMM41uW16", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2021-08-18T18:42:54Z", "updated_at": "2021-08-18T18:42:54Z", "author_association": "NONE", "body": "# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/247?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\n> Merging [#247](https://codecov.io/gh/simonw/sqlite-utils/pull/247?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (af989af) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/1fe73c898b44695052f1a9ca832818d50cecf662?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (1fe73c8) will **decrease** coverage by `0.03%`.\n> The diff coverage is `85.71%`.\n\n[![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/247/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/247?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n\n```diff\n@@ Coverage Diff @@\n## main #247 +/- ##\n==========================================\n- Coverage 96.28% 96.24% -0.04% \n==========================================\n Files 5 5 \n Lines 2179 2186 +7 \n==========================================\n+ Hits 2098 2104 +6 \n- Misses 81 82 +1 \n```\n\n\n| [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/247?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage \u0394 | |\n|---|---|---|\n| [sqlite\\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.84% <85.71%> (-0.08%)` | :arrow_down: |\n\n------\n\n[Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/247?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n> `\u0394 = absolute (impact)`, `\u00f8 = not affected`, `? = missing data`\n> Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/247?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [1fe73c8...af989af](https://codecov.io/gh/simonw/sqlite-utils/pull/247?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 832687563, "label": "FTS quote functionality from datasette"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/247#issuecomment-901338988", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/247", "id": 901338988, "node_id": "IC_kwDOCGYnMM41uVds", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-18T18:33:39Z", "updated_at": "2021-08-18T18:33:39Z", "author_association": "OWNER", "body": "This was also requested in #296.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 832687563, "label": "FTS quote functionality from datasette"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/296#issuecomment-901338356", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/296", "id": 901338356, "node_id": "IC_kwDOCGYnMM41uVT0", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-18T18:32:39Z", "updated_at": "2021-08-18T18:32:39Z", "author_association": "OWNER", "body": "This is a good call. I have a fix for this in Datasette but it's not in `sqlite-utils` yet: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/datasette/utils/__init__.py#L824-L835", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 944326512, "label": "`table.search(..., quote=True)` parameter and `sqlite-utils search --quote` option"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/317#issuecomment-901337305", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/317", "id": 901337305, "node_id": "IC_kwDOCGYnMM41uVDZ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-18T18:30:59Z", "updated_at": "2021-08-18T18:30:59Z", "author_association": "OWNER", "body": "I'm just going to remove this - I added it when the library was mostly undocumented, but it has comprehensive documentation now.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 972827346, "label": "Link to a better example on docs index"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1439#issuecomment-900715375", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1439", "id": 900715375, "node_id": "IC_kwDOBm6k_c41r9Nv", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-18T00:15:28Z", "updated_at": "2021-08-18T00:15:28Z", "author_association": "OWNER", "body": "Maybe I should use `-/` to encode forward slashes too, to defend against any ASGI servers that might not implement `raw_path` correctly.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 973139047, "label": "Rethink how .ext formats (v.s. ?_format=) works before 1.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1439#issuecomment-900714630", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1439", "id": 900714630, "node_id": "IC_kwDOBm6k_c41r9CG", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-18T00:13:33Z", "updated_at": "2021-08-18T00:13:33Z", "author_association": "OWNER", "body": "The documentation should definitely cover how table names become URLs, in case any third party code needs to be able to calculate this themselves.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 973139047, "label": "Rethink how .ext formats (v.s. ?_format=) works before 1.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1439#issuecomment-900712981", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1439", "id": 900712981, "node_id": "IC_kwDOBm6k_c41r8oV", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-18T00:09:59Z", "updated_at": "2021-08-18T00:12:32Z", "author_association": "OWNER", "body": "So given the original examples, a table called `table.csv` would have the following URLs:\r\n\r\n- `/db/table-.csv` - the HTML version\r\n- `/db/table-.csv.csv` - the CSV version\r\n- `/db/table-.csv.json` - the JSON version\r\n\r\nAnd if for some horific reason you had a table with the name `/db/table-.csv.csv` (so `/db/` was the first part of the actual table name in SQLite) the URLs would look like this:\r\n\r\n- `/db/%2Fdb%2Ftable---.csv-.csv` - the HTML version\r\n- `/db/%2Fdb%2Ftable---.csv-.csv.csv` - the CSV version\r\n- `/db/%2Fdb%2Ftable---.csv-.csv.json` - the JSON version", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 973139047, "label": "Rethink how .ext formats (v.s. ?_format=) works before 1.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1439#issuecomment-900711967", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1439", "id": 900711967, "node_id": "IC_kwDOBm6k_c41r8Yf", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-18T00:08:09Z", "updated_at": "2021-08-18T00:08:09Z", "author_association": "OWNER", "body": "Here's an alternative I just made up which I'm calling \"dot dash\" encoding:\r\n\r\n```python\r\ndef dot_dash_encode(s):\r\n return s.replace(\"-\", \"--\").replace(\".\", \"-.\")\r\n\r\ndef dot_dash_decode(s):\r\n return s.replace(\"-.\", \".\").replace(\"--\", \"-\")\r\n```\r\nAnd some examples:\r\n```python\r\nfor example in (\r\n \"hello\",\r\n \"hello.csv\",\r\n \"hello-and-so-on.csv\",\r\n \"hello-.csv\",\r\n \"hello--and--so--on-.csv\",\r\n \"hello.csv.\",\r\n \"hello.csv.-\",\r\n \"hello.csv.--\",\r\n):\r\n print(example)\r\n print(dot_dash_encode(example))\r\n print(example == dot_dash_decode(dot_dash_encode(example)))\r\n print()\r\n```\r\nOutputs:\r\n```\r\nhello\r\nhello\r\nTrue\r\n\r\nhello.csv\r\nhello-.csv\r\nTrue\r\n\r\nhello-and-so-on.csv\r\nhello--and--so--on-.csv\r\nTrue\r\n\r\nhello-.csv\r\nhello---.csv\r\nTrue\r\n\r\nhello--and--so--on-.csv\r\nhello----and----so----on---.csv\r\nTrue\r\n\r\nhello.csv.\r\nhello-.csv-.\r\nTrue\r\n\r\nhello.csv.-\r\nhello-.csv-.--\r\nTrue\r\n\r\nhello.csv.--\r\nhello-.csv-.----\r\nTrue\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 973139047, "label": "Rethink how .ext formats (v.s. ?_format=) works before 1.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1439#issuecomment-900709703", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1439", "id": 900709703, "node_id": "IC_kwDOBm6k_c41r71H", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-18T00:03:09Z", "updated_at": "2021-08-18T00:03:09Z", "author_association": "OWNER", "body": "But... what if I invent my own escaping scheme?\r\n\r\nI actually did this once before, in https://github.com/simonw/datasette/commit/9fdb47ca952b93b7b60adddb965ea6642b1ff523 - while I was working on porting Datasette to ASGI in https://github.com/simonw/datasette/issues/272#issuecomment-494192779 because ASGI didn't yet have the `raw_path` mechanism.\r\n\r\nI could bring that back - it looked like this:\r\n\r\n```\r\n \"table/and/slashes\" => \"tableU+002FandU+002Fslashes\"\r\n \"~table\" => \"U+007Etable\"\r\n \"+bobcats!\" => \"U+002Bbobcats!\"\r\n \"U+007Etable\" => \"UU+002B007Etable\"\r\n```\r\nBut I didn't particularly like it - it was quite verbose.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 973139047, "label": "Rethink how .ext formats (v.s. ?_format=) works before 1.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1439#issuecomment-900705226", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1439", "id": 900705226, "node_id": "IC_kwDOBm6k_c41r6vK", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-17T23:50:32Z", "updated_at": "2021-08-17T23:50:47Z", "author_association": "OWNER", "body": "An alternative solution would be to use some form of escaping for the characters that form the name of the table.\r\n\r\nThe obvious way to do this would be URL-encoding - but it doesn't hold for `.` characters. The hex for that is `%2E` but watch what happens with that in a URL:\r\n\r\n```\r\n# Against Cloud Run:\r\ncurl -s 'https://datasette.io/-/asgi-scope/foo/bar%2Fbaz%2E' | rg path\r\n 'path': '/-/asgi-scope/foo/bar/baz.',\r\n 'raw_path': b'/-/asgi-scope/foo/bar%2Fbaz.',\r\n 'root_path': '',\r\n# Against Vercel:\r\ncurl -s 'https://til.simonwillison.net/-/asgi-scope/foo/bar%2Fbaz%2E' | rg path\r\n 'path': '/-/asgi-scope/foo/bar%2Fbaz%2E',\r\n 'raw_path': b'/-/asgi-scope/foo/bar%2Fbaz%2E',\r\n 'root_path': '',\r\n```\r\nSurprisingly in this case Vercel DOES keep it intact, but Cloud Run does not.\r\n\r\nIt's still no good though: I need a solution that works on Vercel, Cloud Run and every other potential hosting provider too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 973139047, "label": "Rethink how .ext formats (v.s. ?_format=) works before 1.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1439#issuecomment-900699670", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1439", "id": 900699670, "node_id": "IC_kwDOBm6k_c41r5YW", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-17T23:34:23Z", "updated_at": "2021-08-17T23:34:23Z", "author_association": "OWNER", "body": "The challenge comes down to telling the difference between the following:\r\n\r\n- `/db/table` - an HTML table page\r\n- `/db/table.csv` - the CSV version of `/db/table`\r\n- `/db/table.csv` - no this one is actually a database table called `table.csv`\r\n- `/db/table.csv.csv` - the CSV version of `/db/table.csv`\r\n- `/db/table.csv.csv.csv` and so on...", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 973139047, "label": "Rethink how .ext formats (v.s. ?_format=) works before 1.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1438#issuecomment-900690998", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1438", "id": 900690998, "node_id": "IC_kwDOBm6k_c41r3Q2", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-17T23:11:16Z", "updated_at": "2021-08-17T23:12:25Z", "author_association": "OWNER", "body": "I have completely failed to replicate this initial bug - but it's still there on the `thesession.vercel.app` deployment (even though my own deployments to Vercel do not exhibit it). Here's a one-liner to replicate it against that deployment:\r\n\r\n`curl -s 'https://thesession.vercel.app/thesession?sql=select+*+from+tunes+where+name+like+%22%25wise+maid%25%22' | rg '.csv'`\r\n\r\nWhit outputs this:\r\n\r\n`

This data as json, CSV

`\r\n\r\nIt looks like, rather than being URL-encoded, the original query string is somehow making it through to Jinja and then being auto-escaped there.\r\n\r\nThe weird thing is that the equivalent query executed against my `til.simonwillison.net` Vercel instance does this:\r\n\r\n`curl -s 'https://til.simonwillison.net/fixtures?sql=select+*+from+searchable+where+text1+like+%22%25a%25%22' | rg '.csv'`\r\n\r\n`

This data as json, CSV

`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 972918533, "label": "Query page .csv and .json links are not correctly URL-encoded on Vercel under unknown specific conditions"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1438#issuecomment-900681413", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1438", "id": 900681413, "node_id": "IC_kwDOBm6k_c41r07F", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-17T22:47:44Z", "updated_at": "2021-08-17T22:47:44Z", "author_association": "OWNER", "body": "I deployed another copy of `fixtures.db` on Vercel at https://til.simonwillison.net/fixtures so I can compare it with `fixtures.db` on Cloud Run at https://latest.datasette.io/fixtures", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 972918533, "label": "Query page .csv and .json links are not correctly URL-encoded on Vercel under unknown specific conditions"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1438#issuecomment-900518343", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1438", "id": 900518343, "node_id": "IC_kwDOBm6k_c41rNHH", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-17T18:04:42Z", "updated_at": "2021-08-17T18:04:42Z", "author_association": "OWNER", "body": "Here's how `request.query_string` works: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/datasette/utils/asgi.py#L86-L88", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 972918533, "label": "Query page .csv and .json links are not correctly URL-encoded on Vercel under unknown specific conditions"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1438#issuecomment-900516826", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1438", "id": 900516826, "node_id": "IC_kwDOBm6k_c41rMva", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-17T18:02:27Z", "updated_at": "2021-08-17T18:02:27Z", "author_association": "OWNER", "body": "The key difference I can spot between Vercel and Cloud Run is that `+` in a query string gets converted to `%20` by Vercel before it gets to my app, but does not for Cloud Run:\r\n```\r\n# Vercel\r\n~ % curl -s 'https://til.simonwillison.net/-/asgi-scope?sql=select+*+from+tunes+where+name+like+%22%25wise+maid%25%22%0D%0A' | rg 'query_string' -C 2\r\n 'method': 'GET',\r\n 'path': '/-/asgi-scope',\r\n 'query_string': b'sql=select%20*%20from%20tunes%20where%20name%20like%20%22%25'\r\n b'wise%20maid%25%22%0D%0A',\r\n 'raw_path': b'/-/asgi-scope',\r\n\r\n# Cloud Run\r\n~ % curl -s 'https://latest-with-plugins.datasette.io/-/asgi-scope?sql=select+*+from+tunes+where+name+like+%22%25wise+maid%25%22%0D%0A' | rg 'query_string' -C 2\r\n 'method': 'GET',\r\n 'path': '/-/asgi-scope',\r\n 'query_string': b'sql=select+*+from+tunes+where+name+like+%22%25wise+maid%25%2'\r\n b'2%0D%0A',\r\n 'raw_path': b'/-/asgi-scope',\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 972918533, "label": "Query page .csv and .json links are not correctly URL-encoded on Vercel under unknown specific conditions"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1438#issuecomment-900513267", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1438", "id": 900513267, "node_id": "IC_kwDOBm6k_c41rL3z", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-17T17:57:05Z", "updated_at": "2021-08-17T17:57:05Z", "author_association": "OWNER", "body": "I'm having trouble replicating this bug outside of Vercel. Against Cloud Run: view-source:https://latest.datasette.io/fixtures?sql=select+*+from+searchable+where+text1+like+%22%25cat%25%22\r\n\r\nThe HTML here is:\r\n\r\n```html\r\n

This data as\r\n json, \r\n ...\r\n CSV\r\n

\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 972918533, "label": "Query page .csv and .json links are not correctly URL-encoded on Vercel under unknown specific conditions"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1438#issuecomment-900502364", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1438", "id": 900502364, "node_id": "IC_kwDOBm6k_c41rJNc", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-17T17:40:41Z", "updated_at": "2021-08-17T17:40:41Z", "author_association": "OWNER", "body": "Bug is likely in `path_with_format` itself: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/datasette/utils/__init__.py#L710-L729", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 972918533, "label": "Query page .csv and .json links are not correctly URL-encoded on Vercel under unknown specific conditions"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1438#issuecomment-900500824", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1438", "id": 900500824, "node_id": "IC_kwDOBm6k_c41rI1Y", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-17T17:38:16Z", "updated_at": "2021-08-17T17:38:16Z", "author_association": "OWNER", "body": "Relevant template code: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/datasette/templates/query.html#L71\r\n\r\n`renderers` comes from here: https://github.com/simonw/datasette/blob/2883098770fc66e50183b2b231edbde20848d4d6/datasette/views/base.py#L593-L608", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 972918533, "label": "Query page .csv and .json links are not correctly URL-encoded on Vercel under unknown specific conditions"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-899915829", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 899915829, "node_id": "IC_kwDOBm6k_c41o6A1", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-17T01:02:35Z", "updated_at": "2021-08-17T01:02:35Z", "author_association": "OWNER", "body": "New approach: this time I'm building a simplified executor for the bytecode operations themselves.\r\n```python\r\ndef execute_operations(operations, max_iterations = 100, trace=None):\r\n trace = trace or (lambda *args: None)\r\n registers: Dict[int, Any] = {}\r\n cursors: Dict[int, Tuple[str, Dict]] = {}\r\n instruction_pointer = 0\r\n iterations = 0\r\n result_row = None\r\n while True:\r\n iterations += 1\r\n if iterations > max_iterations:\r\n break\r\n operation = operations[instruction_pointer]\r\n trace(instruction_pointer, dict(operation))\r\n opcode = operation[\"opcode\"]\r\n if opcode == \"Init\":\r\n if operation[\"p2\"] != 0:\r\n instruction_pointer = operation[\"p2\"]\r\n continue\r\n else:\r\n instruction_pointer += 1\r\n continue\r\n elif opcode == \"Goto\":\r\n instruction_pointer = operation[\"p2\"]\r\n continue\r\n elif opcode == \"Halt\":\r\n break\r\n elif opcode == \"OpenRead\":\r\n cursors[operation[\"p1\"]] = (\"database_table\", {\r\n \"rootpage\": operation[\"p2\"],\r\n \"connection\": operation[\"p3\"],\r\n })\r\n elif opcode == \"OpenEphemeral\":\r\n cursors[operation[\"p1\"]] = (\"ephemeral\", {\r\n \"num_columns\": operation[\"p2\"],\r\n \"index_keys\": [],\r\n })\r\n elif opcode == \"MakeRecord\":\r\n registers[operation[\"p3\"]] = (\"MakeRecord\", {\r\n \"registers\": list(range(operation[\"p1\"] + operation[\"p2\"]))\r\n })\r\n elif opcode == \"IdxInsert\":\r\n record = registers[operation[\"p2\"]]\r\n cursors[operation[\"p1\"]][1][\"index_keys\"].append(record)\r\n elif opcode == \"Rowid\":\r\n registers[operation[\"p2\"]] = (\"rowid\", {\r\n \"table\": operation[\"p1\"]\r\n })\r\n elif opcode == \"Sequence\":\r\n registers[operation[\"p2\"]] = (\"sequence\", {\r\n \"next_from_cursor\": operation[\"p1\"]\r\n })\r\n elif opcode == \"Column\":\r\n registers[operation[\"p3\"]] = (\"column\", {\r\n \"cursor\": operation[\"p1\"],\r\n \"column_offset\": operation[\"p2\"]\r\n })\r\n elif opcode == \"ResultRow\":\r\n p1 = operation[\"p1\"]\r\n p2 = operation[\"p2\"]\r\n trace(\"ResultRow: \", list(range(p1, p1 + p2)), registers)\r\n result_row = [registers.get(i) for i in range(p1, p1 + p2)]\r\n elif opcode == \"Integer\":\r\n registers[operation[\"p2\"]] = (\"Integer\", operation[\"p1\"])\r\n elif opcode == \"String8\":\r\n registers[operation[\"p2\"]] = (\"String\", operation[\"p4\"])\r\n instruction_pointer += 1\r\n return {\"registers\": registers, \"cursors\": cursors, \"result_row\": result_row}\r\n```\r\nResults are promising!\r\n```\r\nexecute_operations(db.execute(\"explain select 'hello', 55, rowid, * from searchable\").fetchall())\r\n\r\n{'registers': {1: ('String', 'hello'),\r\n 2: ('Integer', 55),\r\n 3: ('rowid', {'table': 0}),\r\n 4: ('rowid', {'table': 0}),\r\n 5: ('column', {'cursor': 0, 'column_offset': 1}),\r\n 6: ('column', {'cursor': 0, 'column_offset': 2}),\r\n 7: ('column', {'cursor': 0, 'column_offset': 3})},\r\n 'cursors': {0: ('database_table', {'rootpage': 32, 'connection': 0})},\r\n 'result_row': [('String', 'hello'),\r\n ('Integer', 55),\r\n ('rowid', {'table': 0}),\r\n ('rowid', {'table': 0}),\r\n ('column', {'cursor': 0, 'column_offset': 1}),\r\n ('column', {'cursor': 0, 'column_offset': 2}),\r\n ('column', {'cursor': 0, 'column_offset': 3})]}\r\n```\r\nHere's what happens with a union across three tables:\r\n```\r\nexecute_operations(db.execute(f\"\"\"\r\nexplain select data as content from binary_data\r\nunion\r\nselect pk as content from complex_foreign_keys\r\nunion\r\nselect name as content from facet_cities\r\n\"\"\"}).fetchall())\r\n\r\n{'registers': {1: ('column', {'cursor': 4, 'column_offset': 0}),\r\n 2: ('MakeRecord', {'registers': [0, 1, 2, 3]}),\r\n 3: ('column', {'cursor': 0, 'column_offset': 1}),\r\n 4: ('column', {'cursor': 3, 'column_offset': 0})},\r\n 'cursors': {3: ('ephemeral',\r\n {'num_columns': 1,\r\n 'index_keys': [('MakeRecord', {'registers': [0, 1]}),\r\n ('MakeRecord', {'registers': [0, 1]}),\r\n ('MakeRecord', {'registers': [0, 1, 2, 3]})]}),\r\n 2: ('database_table', {'rootpage': 44, 'connection': 0}),\r\n 4: ('database_table', {'rootpage': 24, 'connection': 0}),\r\n 0: ('database_table', {'rootpage': 42, 'connection': 0})},\r\n 'result_row': [('column', {'cursor': 3, 'column_offset': 0})]}\r\n```\r\nNote how the result_row refers to cursor 3, which is an ephemeral table which had three different sets of `MakeRecord` index keys assigned to it - indicating that the output column is NOT from the same underlying table source.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1423#issuecomment-899749881", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1423", "id": 899749881, "node_id": "IC_kwDOBm6k_c41oRf5", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-16T19:07:02Z", "updated_at": "2021-08-16T19:07:02Z", "author_association": "OWNER", "body": "Demo: https://latest.datasette.io/fixtures/compound_three_primary_keys?_facet=content&_facet_size=max&_facet=pk1&_facet=pk2\r\n\r\n\"fixtures__compound_three_primary_keys__1_001_rows\"\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 962391325, "label": "Show count of facet values if ?_facet_size=max"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1423#issuecomment-899744109", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1423", "id": 899744109, "node_id": "IC_kwDOBm6k_c41oQFt", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-16T18:58:29Z", "updated_at": "2021-08-16T18:58:29Z", "author_association": "OWNER", "body": "I didn't bother with the tooltip, just the visible display if `?_facet_size=max`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 962391325, "label": "Show count of facet values if ?_facet_size=max"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-898961535", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 898961535, "node_id": "IC_kwDOBm6k_c41lRB_", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-14T21:37:24Z", "updated_at": "2021-08-14T21:37:24Z", "author_association": "OWNER", "body": "Did some more research into building SQLite custom versions via `pysqlite3` - here's what I figured out for macOS (which should hopefully work for Linux too): https://til.simonwillison.net/sqlite/build-specific-sqlite-pysqlite-macos", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-898936068", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 898936068, "node_id": "IC_kwDOBm6k_c41lK0E", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-14T17:44:54Z", "updated_at": "2021-08-14T17:44:54Z", "author_association": "OWNER", "body": "Another interesting query to consider: https://latest.datasette.io/fixtures?sql=explain+select+*+from++pragma_table_info%28+%27123_starts_with_digits%27%29\r\n\r\nThat one shows `VColumn` instead of `Column`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-898933865", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 898933865, "node_id": "IC_kwDOBm6k_c41lKRp", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-14T17:27:16Z", "updated_at": "2021-08-14T17:28:29Z", "author_association": "OWNER", "body": "Maybe I split this out into a separate Python library that gets tested against *every* SQLite release I can possibly try it against, and then bakes out the supported release versions into the library code itself?\r\n\r\nDatasette could depend on that library. The library could be released independently of Datasette any time a new SQLite version comes out.\r\n\r\nI could even run a separate git scraper repo that checks for new SQLite releases and submits PRs against the library when a new release comes out.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-898913629", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 898913629, "node_id": "IC_kwDOBm6k_c41lFVd", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-14T16:14:12Z", "updated_at": "2021-08-14T16:14:12Z", "author_association": "OWNER", "body": "I would feel a lot more comfortable about all of this if I had a robust mechanism for running the Datasette test suite against multiple versions of SQLite itself.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-898913554", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 898913554, "node_id": "IC_kwDOBm6k_c41lFUS", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-14T16:13:40Z", "updated_at": "2021-08-14T16:13:40Z", "author_association": "OWNER", "body": "I think I need to care about the following:\r\n\r\n- `ResultRow` and `Column` for the final result\r\n- `OpenRead` for opening tables\r\n- `OpenEphemeral` then `MakeRecord` and `IdxInsert` for writing records into ephemeral tables\r\n\r\n`Column` may reference either a table (from `OpenRead`) or an ephemeral table (from `OpenEphemeral`).\r\n\r\nThat *might* be enough.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null}