{"html_url": "https://github.com/dogsheep/pocket-to-sqlite/issues/10#issuecomment-1221623052", "issue_url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/10", "id": 1221623052, "node_id": "IC_kwDODLZ_YM5I0H0M", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-21T21:20:33Z", "updated_at": "2022-08-21T21:20:33Z", "author_association": "MEMBER", "body": "That was clearly the intention from the description of this issue:\r\n- #4", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1246826792, "label": "When running `auth` command, don't overwrite an existing auth.json file"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/pocket-to-sqlite/issues/10#issuecomment-1221622873", "issue_url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/10", "id": 1221622873, "node_id": "IC_kwDODLZ_YM5I0HxZ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-21T21:19:25Z", "updated_at": "2022-08-21T21:19:25Z", "author_association": "MEMBER", "body": "Agreed, that would be a much better implementation.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1246826792, "label": "When running `auth` command, don't overwrite an existing auth.json file"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/pocket-to-sqlite/issues/11#issuecomment-1221621529", "issue_url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/11", "id": 1221621529, "node_id": "IC_kwDODLZ_YM5I0HcZ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-21T21:10:15Z", "updated_at": "2022-08-21T21:11:26Z", "author_association": "MEMBER", "body": "Just saw that's what's implemented here already!\r\n- #7", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1345452427, "label": "-a option is used for \"--auth\" and for \"--all\""}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/pocket-to-sqlite/pull/7#issuecomment-1221621700", "issue_url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/7", "id": 1221621700, "node_id": "IC_kwDODLZ_YM5I0HfE", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-21T21:11:12Z", "updated_at": "2022-08-21T21:11:12Z", "author_association": "MEMBER", "body": "I thought this might need a documentation update but `--all` is already covered: https://github.com/dogsheep/pocket-to-sqlite/blob/0.2.1/README.md", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 750141615, "label": "Fixed conflicting CLI flags"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/pocket-to-sqlite/issues/11#issuecomment-1221621466", "issue_url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/11", "id": 1221621466, "node_id": "IC_kwDODLZ_YM5I0Hba", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-21T21:09:47Z", "updated_at": "2022-08-21T21:09:47Z", "author_association": "MEMBER", "body": "Great catch, thanks.\r\n\r\nI'm going to use it to mean `--auth` - since other tools in the Dogsheep family have the same convention.\r\n\r\n `--all` will be the only way to specify all.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1345452427, "label": "-a option is used for \"--auth\" and for \"--all\""}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/74#issuecomment-1188317682", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/74", "id": 1188317682, "node_id": "IC_kwDODFdgUs5G1Eny", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-07-18T21:14:22Z", "updated_at": "2022-07-18T21:14:22Z", "author_association": "MEMBER", "body": "That fixed it.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1308461063, "label": "500 error in github-to-sqlite demo"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/74#issuecomment-1188233729", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/74", "id": 1188233729, "node_id": "IC_kwDODFdgUs5G0wIB", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-07-18T19:51:02Z", "updated_at": "2022-07-18T19:51:02Z", "author_association": "MEMBER", "body": "Takes 30m to deploy the demo!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1308461063, "label": "500 error in github-to-sqlite demo"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/74#issuecomment-1188228964", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/74", "id": 1188228964, "node_id": "IC_kwDODFdgUs5G0u9k", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-07-18T19:45:30Z", "updated_at": "2022-07-18T19:47:35Z", "author_association": "MEMBER", "body": "`pycmarkgfm` doesn't implement the Markdown plugin extension I was using.\r\n\r\nI'm going to drop the GFM rendering from the demo, and just treat it as regular markdown.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1308461063, "label": "500 error in github-to-sqlite demo"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/74#issuecomment-1188223933", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/74", "id": 1188223933, "node_id": "IC_kwDODFdgUs5G0tu9", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-07-18T19:40:50Z", "updated_at": "2022-07-18T19:42:41Z", "author_association": "MEMBER", "body": "Here's how the demo is deployed: https://github.com/dogsheep/github-to-sqlite/blob/dbac2e5dd8a562b45d8255a265859cf8020ca22a/.github/workflows/deploy-demo.yml#L103-L119\r\n\r\nI'm suspicious of `py-gfm`, which is used like this:\r\n\r\nhttps://github.com/dogsheep/github-to-sqlite/blob/dbac2e5dd8a562b45d8255a265859cf8020ca22a/demo-metadata.json#L49-L51", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1308461063, "label": "500 error in github-to-sqlite demo"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/74#issuecomment-1188225625", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/74", "id": 1188225625, "node_id": "IC_kwDODFdgUs5G0uJZ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-07-18T19:41:52Z", "updated_at": "2022-07-18T19:41:52Z", "author_association": "MEMBER", "body": "https://github.com/Zopieux/py-gfm says that library is no longer maintained, and suggests https://github.com/Zopieux/pycmarkgfm as an alternative.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1308461063, "label": "500 error in github-to-sqlite demo"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/pull/73#issuecomment-1188223299", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/73", "id": 1188223299, "node_id": "IC_kwDODFdgUs5G0tlD", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-07-18T19:40:06Z", "updated_at": "2022-07-18T19:40:06Z", "author_association": "MEMBER", "body": "Thanks!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1261884917, "label": "Fixing 'NoneType' object has no attribute 'items'"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/72#issuecomment-1105474232", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/72", "id": 1105474232, "node_id": "IC_kwDODFdgUs5B5DK4", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-21T17:02:15Z", "updated_at": "2022-04-21T17:02:15Z", "author_association": "MEMBER", "body": "That's interesting - yeah it looks like the number of pages can be derived from the `Link` header, which is enough information to show a progress bar, probably using Click just to avoid adding another dependency.\r\n\r\nhttps://docs.github.com/en/rest/guides/traversing-with-pagination", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1211283427, "label": "feature: display progress bar when downloading multi-page responses"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/69#issuecomment-985928838", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/69", "id": 985928838, "node_id": "IC_kwDODFdgUs46xBSG", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-04T00:34:52Z", "updated_at": "2021-12-04T00:34:52Z", "author_association": "MEMBER", "body": "First attempt at this:\r\n```sql\r\nselect 'issues' as \"table\", id, node_id, title, user, created_at, body, repo\r\nfrom issues\r\nunion all\r\nselect 'issue_comments' as \"table\", issue_comments.id, issue_comments.node_id, '' as title, issue_comments.user, issue_comments.created_at, issue_comments.body, issues.repo\r\nfrom issue_comments join issues on issues.id = issue_comments.issue\r\norder by created_at desc\r\n```\r\nhttps://github-to-sqlite.dogsheep.net/github?sql=select+%27issues%27+as+%22table%22%2C+id%2C+node_id%2C+title%2C+user%2C+created_at%2C+body%2C+repo%0D%0Afrom+issues%0D%0Aunion+all%0D%0Aselect+%27issue_comments%27+as+%22table%22%2C+issue_comments.id%2C+issue_comments.node_id%2C+%27%27+as+title%2C+issue_comments.user%2C+issue_comments.created_at%2C+issue_comments.body%2C+issues.repo%0D%0Afrom+issue_comments+join+issues+on+issues.id+%3D+issue_comments.issue%0D%0Aorder+by+created_at+desc", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1071071397, "label": "View that combines issues and issue comments"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/pull/59#issuecomment-924209583", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/59", "id": 924209583, "node_id": "IC_kwDODEm0Qs43FlGv", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-09-21T17:37:34Z", "updated_at": "2021-09-21T17:37:34Z", "author_association": "MEMBER", "body": "Thanks for this!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 984942782, "label": "Fix for since_id bug, closes #58"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/13#issuecomment-906646452", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13", "id": 906646452, "node_id": "IC_kwDOEhK-wc42ClO0", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-26T18:34:34Z", "updated_at": "2021-08-26T18:35:20Z", "author_association": "MEMBER", "body": "I tried this ampersand fix: https://regex101.com/r/ojU2H9/1\r\n```python\r\n\r\n# https://regex101.com/r/ojU2H9/1\r\n_invalid_ampersand_re = re.compile(r'&(?![a-z0-9]+;)')\r\n\r\ndef fix_bad_xml(xml):\r\n # More fixes for things like '&' not as part of an entity\r\n return _invalid_ampersand_re.sub('&', xml)\r\n```\r\n\r\nEven with that I'm still getting total garbage in the `` content - it's just HTML, not even trying to be XML.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 978743426, "label": "xml.etree.ElementTree.ParseError: not well-formed (invalid token)"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/13#issuecomment-906635938", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13", "id": 906635938, "node_id": "IC_kwDOEhK-wc42Ciqi", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-26T18:18:27Z", "updated_at": "2021-08-26T18:18:27Z", "author_association": "MEMBER", "body": "It looks like I was using the round-trip to dump the `` and `\r\n\r\n\r\n

\r\n \r\n Q3 2018 Reflection & Development\r\n \r\n

\r\n ...\r\n```\r\nYeah that is not valid XML!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 978743426, "label": "xml.etree.ElementTree.ParseError: not well-formed (invalid token)"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/healthkit-to-sqlite/issues/20#issuecomment-902356871", "issue_url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/20", "id": 902356871, "node_id": "IC_kwDOC8tyDs41yN-H", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-20T01:12:48Z", "updated_at": "2021-08-20T01:12:48Z", "author_association": "MEMBER", "body": "Also on `workout_points.workout_id` to speed up queries to show all points in a specific workout.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 975166271, "label": "Add index on workout_points.date"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/healthkit-to-sqlite/issues/20#issuecomment-902355471", "issue_url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/20", "id": 902355471, "node_id": "IC_kwDOC8tyDs41yNoP", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-20T01:09:07Z", "updated_at": "2021-08-20T01:09:07Z", "author_association": "MEMBER", "body": "Workaround:\r\n\r\n sqlite-utils create-index healthkit.db workout_points -- -date\r\n\r\nSee https://sqlite-utils.datasette.io/en/stable/cli.html#creating-indexes", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 975166271, "label": "Add index on workout_points.date"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/pull/49#issuecomment-902330301", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/49", "id": 902330301, "node_id": "IC_kwDODEm0Qs41yHe9", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-20T00:01:56Z", "updated_at": "2021-08-20T00:01:56Z", "author_association": "MEMBER", "body": "Thanks!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 681575714, "label": "Document the use of --stop_after with favorites, refs #20"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902329884", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57", "id": 902329884, "node_id": "IC_kwDODEm0Qs41yHYc", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-20T00:01:05Z", "updated_at": "2021-08-20T00:01:05Z", "author_association": "MEMBER", "body": "Maybe Click changed something which meant that this broke things when it didn't used to?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 907645813, "label": "Error: Use either --since or --since_id, not both"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902329455", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57", "id": 902329455, "node_id": "IC_kwDODEm0Qs41yHRv", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-19T23:59:56Z", "updated_at": "2021-08-19T23:59:56Z", "author_association": "MEMBER", "body": "This looks like the bug to me:\r\n\r\nhttps://github.com/dogsheep/twitter-to-sqlite/blob/197e69cec40052c423a5ed071feb5f7cccea41b9/twitter_to_sqlite/cli.py#L239-L241\r\n\r\n`type=str, default=False`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 907645813, "label": "Error: Use either --since or --since_id, not both"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902328760", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57", "id": 902328760, "node_id": "IC_kwDODEm0Qs41yHG4", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-19T23:57:41Z", "updated_at": "2021-08-19T23:57:41Z", "author_association": "MEMBER", "body": "Weird, added debug code and got this: `{'screen_name': 'simonw', 'count': 200, 'since_id': 'False', 'tweet_mode': 'extended'}` - so maybe it's a `twitter-to-sqlite` bug where somehow the string `False` is being passed somewhere.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 907645813, "label": "Error: Use either --since or --since_id, not both"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902328369", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57", "id": 902328369, "node_id": "IC_kwDODEm0Qs41yHAx", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-19T23:56:26Z", "updated_at": "2021-08-19T23:56:26Z", "author_association": "MEMBER", "body": "https://developer.twitter.com/en/docs/twitter-api/v1/tweets/timelines/api-reference/get-statuses-user_timeline says the API has been replaced by the new v2 one, but it should still work - and the `since_id` parameter is still documented on that page.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 907645813, "label": "Error: Use either --since or --since_id, not both"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902327457", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57", "id": 902327457, "node_id": "IC_kwDODEm0Qs41yGyh", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-19T23:53:25Z", "updated_at": "2021-08-19T23:53:25Z", "author_association": "MEMBER", "body": "I'm getting this too. Looking into it now.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 907645813, "label": "Error: Use either --since or --since_id, not both"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/hacker-news-to-sqlite/issues/3#issuecomment-886241674", "issue_url": "https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/3", "id": 886241674, "node_id": "IC_kwDODtX3eM400vmK", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-25T18:41:17Z", "updated_at": "2021-07-25T18:41:17Z", "author_association": "MEMBER", "body": "Got a TIL out of this: https://til.simonwillison.net/jq/extracting-objects-recursively", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 952189173, "label": "Use HN algolia endpoint to retrieve trees"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/hacker-news-to-sqlite/issues/3#issuecomment-886237834", "issue_url": "https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/3", "id": 886237834, "node_id": "IC_kwDODtX3eM400uqK", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-25T18:05:32Z", "updated_at": "2021-07-25T18:05:32Z", "author_association": "MEMBER", "body": "If you hit the endpoint for a comment that's part of a thread you get that comment and its recursive children: https://hn.algolia.com/api/v1/items/27941552\r\n\r\nYou can tell that it's not the top-level because the `parent_id` isn't `null`. You can use `story_id` to figure out what the top-level item is.\r\n\r\n```json\r\n{\r\n \"id\": 27941552,\r\n \"created_at\": \"2021-07-24T15:08:39.000Z\",\r\n \"created_at_i\": 1627139319,\r\n \"type\": \"comment\",\r\n \"author\": \"nine_k\",\r\n \"title\": null,\r\n \"url\": null,\r\n \"text\": \"

I wish ...\",\r\n \"points\": null,\r\n \"parent_id\": 27941108,\r\n \"story_id\": 27941108\r\n}\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 952189173, "label": "Use HN algolia endpoint to retrieve trees"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/hacker-news-to-sqlite/issues/3#issuecomment-886142671", "issue_url": "https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/3", "id": 886142671, "node_id": "IC_kwDODtX3eM400XbP", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-25T03:51:05Z", "updated_at": "2021-07-25T03:51:05Z", "author_association": "MEMBER", "body": "Prototype:\r\n\r\n curl 'https://hn.algolia.com/api/v1/items/27941108' \\\r\n | jq '[recurse(.children[]) | del(.children)]' \\\r\n | sqlite-utils insert hn.db items - --pk id\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 952189173, "label": "Use HN algolia endpoint to retrieve trees"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/hacker-news-to-sqlite/issues/2#issuecomment-886140431", "issue_url": "https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/2", "id": 886140431, "node_id": "IC_kwDODtX3eM400W4P", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-25T03:12:57Z", "updated_at": "2021-07-25T03:12:57Z", "author_association": "MEMBER", "body": "I'm going to build a general-purpose `hacker-new-to-sqlite search ...` command, where one of the options is to search within the URL.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 952179830, "label": "Command for fetching Hacker News threads from the search API"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/hacker-news-to-sqlite/issues/2#issuecomment-886136224", "issue_url": "https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/2", "id": 886136224, "node_id": "IC_kwDODtX3eM400V2g", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-25T02:08:29Z", "updated_at": "2021-07-25T02:08:29Z", "author_association": "MEMBER", "body": "Prototype:\r\n\r\n curl \"https://hn.algolia.com/api/v1/search_by_date?query=simonwillison.net&restrictSearchableAttributes=url&hitsPerPage=1000\" | \\\r\n jq .hits | sqlite-utils insert hn.db items - --pk objectID --alter", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 952179830, "label": "Command for fetching Hacker News threads from the search API"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/hacker-news-to-sqlite/issues/2#issuecomment-886135922", "issue_url": "https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/2", "id": 886135922, "node_id": "IC_kwDODtX3eM400Vxy", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-25T02:06:20Z", "updated_at": "2021-07-25T02:06:20Z", "author_association": "MEMBER", "body": "https://hn.algolia.com/api/v1/search_by_date?query=simonwillison.net&restrictSearchableAttributes=url looks like it does what I want.\r\n\r\nhttps://hn.algolia.com/api/v1/search_by_date?query=simonwillison.net&restrictSearchableAttributes=url&hitsPerPage=1000 - returns 1000 at once.\r\n\r\nOtherwise you have to paginate using `&page=2` etc - up to `nbPages` pages.\r\n\r\nhttps://www.algolia.com/doc/api-reference/api-parameters/hitsPerPage/ says 1000 is the maximum.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 952179830, "label": "Command for fetching Hacker News threads from the search API"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/hacker-news-to-sqlite/issues/2#issuecomment-886135562", "issue_url": "https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/2", "id": 886135562, "node_id": "IC_kwDODtX3eM400VsK", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-25T02:01:11Z", "updated_at": "2021-07-25T02:01:11Z", "author_association": "MEMBER", "body": "That page doesn't have an API but does look easy to scrape.\r\n\r\nThe other option here is the HN Search API powered by Algolia, documented at https://hn.algolia.com/api", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 952179830, "label": "Command for fetching Hacker News threads from the search API"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/healthkit-to-sqlite/issues/12#issuecomment-879477586", "issue_url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/12", "id": 879477586, "node_id": "MDEyOklzc3VlQ29tbWVudDg3OTQ3NzU4Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-13T23:50:06Z", "updated_at": "2021-07-13T23:50:06Z", "author_association": "MEMBER", "body": "Unfortunately I don't think updating the database is practical, because the export doesn't include unique identifiers which can be used to update existing records and create new ones. Recreating from scratch works around that limitation.\r\n\r\nI've not explored workouts with SpatiaLite but that's a really good idea.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 727848625, "label": "Some workout columns should be float, not text"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/64#issuecomment-861042050", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64", "id": 861042050, "node_id": "MDEyOklzc3VlQ29tbWVudDg2MTA0MjA1MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-14T22:45:42Z", "updated_at": "2021-06-14T22:45:42Z", "author_association": "MEMBER", "body": "I'm definitely interested in supporting events in this tool - see #14.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 920636216, "label": "feature: support \"events\""}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/64#issuecomment-861041597", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64", "id": 861041597, "node_id": "MDEyOklzc3VlQ29tbWVudDg2MTA0MTU5Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-14T22:44:54Z", "updated_at": "2021-06-14T22:44:54Z", "author_association": "MEMBER", "body": "Have you found a way to access events in GraphQL? I can only see way to access a timeline of events for a single issue or a single pull request. See also https://github.community/t/get-event-equivalent-for-v4/13600/2", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 920636216, "label": "feature: support \"events\""}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/pull/59#issuecomment-844250232", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/59", "id": 844250232, "node_id": "MDEyOklzc3VlQ29tbWVudDg0NDI1MDIzMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-05-19T16:08:10Z", "updated_at": "2021-05-19T16:08:10Z", "author_association": "MEMBER", "body": "Thanks for catching this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771872303, "label": "Remove unneeded exists=True for -a/--auth flag."}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/pull/61#issuecomment-844249385", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/61", "id": 844249385, "node_id": "MDEyOklzc3VlQ29tbWVudDg0NDI0OTM4NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-05-19T16:07:06Z", "updated_at": "2021-05-19T16:07:06Z", "author_association": "MEMBER", "body": "Thanks!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 797108702, "label": "fixing typo in get cli help text"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790695126", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 790695126, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MDY5NTEyNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-04T15:20:42Z", "updated_at": "2021-03-04T15:20:42Z", "author_association": "MEMBER", "body": "I'm not sure why but my most recent import, when displayed in Datasette, looks like this:\r\n\r\n\"mbox__mbox_emails__753_446_rows\"\r\n\r\nSorting by `id` in the opposite order gives me the data I would expect - so it looks like a bunch of null/blank messages are being imported at some point and showing up first due to ID ordering.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790693674", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 790693674, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MDY5MzY3NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-04T15:18:36Z", "updated_at": "2021-03-04T15:18:36Z", "author_association": "MEMBER", "body": "I imported my 10GB mbox with 750,000 emails in it, ran this tool (with a hacked fix for the blob column problem) - and now a search that returns 92 results takes 25.37ms! This is fantastic.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790669767", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 790669767, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MDY2OTc2Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-04T14:46:06Z", "updated_at": "2021-03-04T14:46:06Z", "author_association": "MEMBER", "body": "Solution could be to pre-process that string by splitting on `(` and dropping everything afterwards, assuming that the `(...)` bit isn't necessary for correctly parsing the date.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790668263", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 790668263, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MDY2ODI2Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-04T14:43:58Z", "updated_at": "2021-03-04T14:43:58Z", "author_association": "MEMBER", "body": "I added this code to output a message ID on errors:\r\n```diff\r\n print(\"Errors: {}\".format(num_errors))\r\n print(traceback.format_exc())\r\n+ print(\"Message-Id: {}\".format(email.get(\"Message-Id\", \"None\")))\r\n continue\r\n```\r\nHaving found a message ID that had an error, I ran this command to see the context:\r\n\r\n rg --text --context 20 '44F289B0.000001.02100@SCHWARZE-DWFXMI' ~/gmail.mbox\r\n\r\nThis was for the following error:\r\n```\r\n File \"/Users/simon/Dropbox/Development/google-takeout-to-sqlite/google_takeout_to_sqlite/utils.py\", line 102, in get_mbox\r\n message[\"date\"] = get_message_date(email.get(\"Date\"), email.get_from())\r\n File \"/Users/simon/Dropbox/Development/google-takeout-to-sqlite/google_takeout_to_sqlite/utils.py\", line 178, in get_message_date\r\n datetime_tuple = email.utils.parsedate_tz(mail_date)\r\n File \"/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/email/_parseaddr.py\", line 50, in parsedate_tz\r\n res = _parsedate_tz(data)\r\n File \"/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/email/_parseaddr.py\", line 69, in _parsedate_tz\r\n data = data.split()\r\nAttributeError: 'Header' object has no attribute 'split'\r\n```\r\nHere's what I spotted in the `ripgrep` output:\r\n```\r\n177133570:Message-Id: <44F289B0.000001.02100@SCHWARZE-DWFXMI>\r\n177133571-Date: Mon, 28 Aug 2006 08:14:08 +0200 (Westeurop\ufffdische Sommerzeit)\r\n177133572-X-Mailer: IncrediMail (5002253)\r\n```\r\nSo it could it be that `_parsedate_tz` is having trouble with that `Mon, 28 Aug 2006 08:14:08 +0200 (Westeurop\ufffdische Sommerzeit)` string.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/issues/6#issuecomment-790384087", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/6", "id": 790384087, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MDM4NDA4Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-04T07:22:51Z", "updated_at": "2021-03-04T07:22:51Z", "author_association": "MEMBER", "body": "#3 also mentions the conflicting version with other tools.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 821841046, "label": "Upgrade to latest sqlite-utils"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790380839", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 790380839, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MDM4MDgzOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-04T07:17:05Z", "updated_at": "2021-03-04T07:17:05Z", "author_association": "MEMBER", "body": "Looks like you're doing this:\r\n```python\r\n elif message.get_content_type() == \"text/plain\":\r\n body = message.get_payload(decode=True)\r\n```\r\nSo presumably that decodes to a unicode string?\r\n\r\nI imagine the reason the column is a `BLOB` for me is that `sqlite-utils` determines the column type based on the first batch of items - https://github.com/simonw/sqlite-utils/blob/09c3386f55f766b135b6a1c00295646c4ae29bec/sqlite_utils/db.py#L1927-L1928 - and I got unlucky and had something in my first batch that wasn't a unicode string.\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790379629", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 790379629, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MDM3OTYyOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-04T07:14:41Z", "updated_at": "2021-03-04T07:14:41Z", "author_association": "MEMBER", "body": "Confirmed: removing the `len()` call does not speed things up, so it's reading through the entire file for some other purpose too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790378658", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 790378658, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MDM3ODY1OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-04T07:12:48Z", "updated_at": "2021-03-04T07:12:48Z", "author_association": "MEMBER", "body": "It looks like the `body` is being loaded into a BLOB column - so in Datasette default it looks like this:\r\n\r\n\"mbox__mbox_emails__753_446_rows\"\r\n\r\nIf I `datasette install datasette-render-binary` and then try again I get this:\r\n\r\n\"mbox__mbox_emails__753_446_rows\"\r\n\r\nIt would be great if we could store the `body` as unicode text instead. May have to do something clever to decode it based on some kind of charset header?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790373024", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 790373024, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MDM3MzAyNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-04T07:01:58Z", "updated_at": "2021-03-04T07:04:06Z", "author_association": "MEMBER", "body": "I got 9 warnings that look like this:\r\n```\r\nErrors: 1\r\nTraceback (most recent call last):\r\n File \"/Users/simon/Dropbox/Development/google-takeout-to-sqlite/google_takeout_to_sqlite/utils.py\", line 103, in get_mbox\r\n message[\"date\"] = get_message_date(email.get(\"Date\"), email.get_from())\r\n File \"/Users/simon/Dropbox/Development/google-takeout-to-sqlite/google_takeout_to_sqlite/utils.py\", line 167, in get_message_date\r\n datetime_tuple = email.utils.parsedate_tz(mail_date)\r\n File \"/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/email/_parseaddr.py\", line 50, in parsedate_tz\r\n res = _parsedate_tz(data)\r\n File \"/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/email/_parseaddr.py\", line 69, in _parsedate_tz\r\n data = data.split()\r\nAttributeError: 'Header' object has no attribute 'split'\r\n```\r\nIt would be useful if those warnings told me the message ID (or similar) of the affected message so I could grep for it in the `mbox` and see what was going on.\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790372621", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 790372621, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MDM3MjYyMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-04T07:01:18Z", "updated_at": "2021-03-04T07:01:18Z", "author_association": "MEMBER", "body": "I'm not sure if it would work, but there is an alternative pattern for showing a progress bar against a really large file that I've used in `healthkit-to-sqlite` - you set the progress bar size to the size of the file in bytes, then update a counter as you read the file.\r\n\r\nhttps://github.com/dogsheep/healthkit-to-sqlite/blob/3eb2b06bfe3b4faaf10e9cf9dfcb28e3d16c14ff/healthkit_to_sqlite/cli.py#L24-L57 and https://github.com/dogsheep/healthkit-to-sqlite/blob/3eb2b06bfe3b4faaf10e9cf9dfcb28e3d16c14ff/healthkit_to_sqlite/utils.py#L4-L19 (the `progress_callback()` bit) is where that happens.\r\n\r\nIt can be a bit of a convoluted pattern, and I'm not at all sure it would work for `mbox` files since it looks like that library has other reasons it needs to do a file scan rather than streaming it through one chunk of bytes at a time. So I imagine this would not work here.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790370485", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 790370485, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MDM3MDQ4NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-04T06:57:25Z", "updated_at": "2021-03-04T06:57:48Z", "author_association": "MEMBER", "body": "The command takes quite a while to start running, presumably because this line causes it to have to scan the WHOLE file in order to generate a count:\r\n\r\nhttps://github.com/dogsheep/google-takeout-to-sqlite/blob/a3de045eba0fae4b309da21aa3119102b0efc576/google_takeout_to_sqlite/utils.py#L66-L67\r\n\r\nI'm fine with waiting though. It's not like this is a command people run every day - and without that count we can't show a progress bar, which seems pretty important for a process that takes this long.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790369076", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 790369076, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MDM2OTA3Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-04T06:54:46Z", "updated_at": "2021-03-04T06:54:46Z", "author_association": "MEMBER", "body": "The Rich-powered progress bar is pretty:\r\n\r\n![rich](https://user-images.githubusercontent.com/9599/109923307-71f69200-7c73-11eb-9ee2-8f0a240f3994.gif)\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790312268", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 790312268, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MDMxMjI2OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-04T05:48:16Z", "updated_at": "2021-03-04T05:48:16Z", "author_association": "MEMBER", "body": "Wow, my mbox is a 10.35 GB download!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-786925280", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 786925280, "node_id": "MDEyOklzc3VlQ29tbWVudDc4NjkyNTI4MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-02-26T22:23:10Z", "updated_at": "2021-02-26T22:23:10Z", "author_association": "MEMBER", "body": "Thanks!\r\n\r\nI requested my Gmail export from takeout - once that arrives I'll test it against this and then merge the PR.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/evernote-to-sqlite/pull/10#issuecomment-777839351", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/10", "id": 777839351, "node_id": "MDEyOklzc3VlQ29tbWVudDc3NzgzOTM1MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-02-11T22:37:55Z", "updated_at": "2021-02-11T22:37:55Z", "author_association": "MEMBER", "body": "I've merged these changes by hand now, thanks!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770712149, "label": "BugFix for encoding and not update info."}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/7#issuecomment-777827396", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/7", "id": 777827396, "node_id": "MDEyOklzc3VlQ29tbWVudDc3NzgyNzM5Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-02-11T22:13:14Z", "updated_at": "2021-02-11T22:13:14Z", "author_association": "MEMBER", "body": "My best guess is that you have an older version of `sqlite-utils` installed here - the `replace=True` argument was added in version 2.0. I've bumped the dependency in `setup.py`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 743297582, "label": "evernote-to-sqlite on windows 10 give this error: TypeError: insert() got an unexpected keyword argument 'replace'"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/9#issuecomment-777821383", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/9", "id": 777821383, "node_id": "MDEyOklzc3VlQ29tbWVudDc3NzgyMTM4Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-02-11T22:01:28Z", "updated_at": "2021-02-11T22:01:28Z", "author_association": "MEMBER", "body": "Aha! I think I've figured out what's going on here.\r\n\r\nThe CData blocks containing the notes look like this:\r\n\r\n`

This note includes two images.

...`\r\n\r\nThe DTD at http://xml.evernote.com/pub/enml2.dtd includes some entities:\r\n\r\n```\r\n\r\n\r\n\r\n%HTMLlat1;\r\n\r\n\r\n%HTMLsymbol;\r\n\r\n\r\n%HTMLspecial;\r\n```\r\nSo I need to be able to handle all of those different entities. I think I can do that using `html.entities.entitydefs` from the Python standard library, which looks a bit like this:\r\n\r\n```python\r\n{'Aacute': '\u00c1',\r\n 'aacute': '\u00e1',\r\n 'Aacute;': '\u00c1',\r\n 'aacute;': '\u00e1',\r\n 'Abreve;': '\u0102',\r\n 'abreve;': '\u0103',\r\n 'ac;': '\u223e',\r\n 'acd;': '\u223f',\r\n# ...\r\n}\r\n```\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 748372469, "label": "ParseError: undefined entity š"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/11#issuecomment-777798330", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/11", "id": 777798330, "node_id": "MDEyOklzc3VlQ29tbWVudDc3Nzc5ODMzMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-02-11T21:18:58Z", "updated_at": "2021-02-11T21:18:58Z", "author_association": "MEMBER", "body": "Thanks for the fix!", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 792851444, "label": "XML parse error"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/60#issuecomment-770071568", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/60", "id": 770071568, "node_id": "MDEyOklzc3VlQ29tbWVudDc3MDA3MTU2OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-01-29T21:56:15Z", "updated_at": "2021-01-29T21:56:15Z", "author_association": "MEMBER", "body": "I really like the way you're using pipes here - really smart. It's similar to how I build the demo database in this GitHub Actions workflow:\r\n\r\nhttps://github.com/dogsheep/github-to-sqlite/blob/62dfd3bc4014b108200001ef4bc746feb6f33b45/.github/workflows/deploy-demo.yml#L52-L82\r\n\r\n`twitter-to-sqlite` actually has a mechanism for doing this kind of thing, documented at https://github.com/dogsheep/twitter-to-sqlite#providing-input-from-a-sql-query-with---sql-and---attach\r\n\r\nIt lets you do things like:\r\n\r\n```\r\n$ twitter-to-sqlite users-lookup my.db --sql=\"select follower_id from following\" --ids\r\n```\r\nMaybe I should add something similar to `github-to-sqlite`? Feels like it could be really useful.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 797097140, "label": "Use Data from SQLite in other commands"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/56#issuecomment-769957751", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/56", "id": 769957751, "node_id": "MDEyOklzc3VlQ29tbWVudDc2OTk1Nzc1MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-01-29T17:59:40Z", "updated_at": "2021-01-29T17:59:40Z", "author_association": "MEMBER", "body": "This is interesting - how did you create that initial table? Was this using the `twitter-to-sqlite import archive.db ~/Downloads/twitter-2019-06-25-b31f2.zip` command, or something else?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 796736607, "label": "Not all quoted statuses get fetched?"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/swarm-to-sqlite/issues/11#issuecomment-761967094", "issue_url": "https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/11", "id": 761967094, "node_id": "MDEyOklzc3VlQ29tbWVudDc2MTk2NzA5NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-01-18T04:11:13Z", "updated_at": "2021-01-18T04:11:13Z", "author_association": "MEMBER", "body": "I just got a similar error:\r\n\r\n```\r\n File \"/home/dogsheep/datasette-venv/lib/python3.8/site-packages/swarm_to_sqlite/utils.py\", line 79, in save_checkin\r\n checkins_table.m2m(\"users\", user, m2m_table=\"with\", pk=\"id\")\r\n File \"/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py\", line 2048, in m2m\r\n id = other_table.insert(record, pk=pk, replace=True).last_pk\r\n File \"/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py\", line 1781, in insert\r\n return self.insert_all(\r\n File \"/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py\", line 1899, in insert_all\r\n self.insert_chunk(\r\n File \"/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py\", line 1709, in insert_chunk\r\n result = self.db.execute(query, params)\r\n File \"/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py\", line 226, in execute\r\n return self.conn.execute(sql, parameters)\r\npysqlite3.dbapi2.OperationalError: table users has no column named countryCode\r\n\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 743400216, "label": "Error thrown: sqlite3.OperationalError: table users has no column named lastName"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/31#issuecomment-748426877", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/31", "id": 748426877, "node_id": "MDEyOklzc3VlQ29tbWVudDc0ODQyNjg3Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-19T06:16:11Z", "updated_at": "2020-12-19T06:16:11Z", "author_association": "MEMBER", "body": "Here's why:\r\n\r\n if \"fts5\" in str(e):\r\n\r\nBut the error being raised here is:\r\n\r\n sqlite3.OperationalError: no such column: to\r\n\r\nI'm going to attempt the escaped on on every error.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771316301, "label": "Searching for \"github-to-sqlite\" throws an error"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/31#issuecomment-748426663", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/31", "id": 748426663, "node_id": "MDEyOklzc3VlQ29tbWVudDc0ODQyNjY2Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-19T06:14:06Z", "updated_at": "2020-12-19T06:14:06Z", "author_association": "MEMBER", "body": "Looks like I already do that here: https://github.com/dogsheep/dogsheep-beta/blob/9ba4401017ac24ffa3bc1db38e0910ea49de7616/dogsheep_beta/__init__.py#L141-L146", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771316301, "label": "Searching for \"github-to-sqlite\" throws an error"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/31#issuecomment-748426581", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/31", "id": 748426581, "node_id": "MDEyOklzc3VlQ29tbWVudDc0ODQyNjU4MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-19T06:13:17Z", "updated_at": "2020-12-19T06:13:17Z", "author_association": "MEMBER", "body": "One fix for this could be to try running the raw query, but if it throws an error run it again with the query escaped.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771316301, "label": "Searching for \"github-to-sqlite\" throws an error"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/31#issuecomment-748426501", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/31", "id": 748426501, "node_id": "MDEyOklzc3VlQ29tbWVudDc0ODQyNjUwMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-19T06:12:22Z", "updated_at": "2020-12-19T06:12:22Z", "author_association": "MEMBER", "body": "I deliberately added support for advanced FTS in https://github.com/dogsheep/dogsheep-beta/commit/cbb2491b85d7ff416d6d429b60109e6c2d6d50b9 for #13 but that's the cause of this bug.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771316301, "label": "Searching for \"github-to-sqlite\" throws an error"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/issues/2#issuecomment-747126777", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/2", "id": 747126777, "node_id": "MDEyOklzc3VlQ29tbWVudDc0NzEyNjc3Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-17T00:36:52Z", "updated_at": "2020-12-17T00:36:52Z", "author_association": "MEMBER", "body": "The memory profiler tricks I used in https://github.com/dogsheep/healthkit-to-sqlite/issues/7 could help figure out what's going on here.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 769376447, "label": "killed by oomkiller on large location-history"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/29#issuecomment-747034481", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/29", "id": 747034481, "node_id": "MDEyOklzc3VlQ29tbWVudDc0NzAzNDQ4MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-16T21:17:05Z", "updated_at": "2020-12-16T21:17:05Z", "author_association": "MEMBER", "body": "I'm just going to add `q` for the moment.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 724759588, "label": "Add search highlighting snippets"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/29#issuecomment-747031608", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/29", "id": 747031608, "node_id": "MDEyOklzc3VlQ29tbWVudDc0NzAzMTYwOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-16T21:15:18Z", "updated_at": "2020-12-16T21:15:18Z", "author_association": "MEMBER", "body": "Should I pass any other details to the `display_sql` here as well?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 724759588, "label": "Add search highlighting snippets"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/29#issuecomment-747030964", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/29", "id": 747030964, "node_id": "MDEyOklzc3VlQ29tbWVudDc0NzAzMDk2NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-16T21:14:54Z", "updated_at": "2020-12-16T21:14:54Z", "author_association": "MEMBER", "body": "To do this I'll need the search term to be passed to the `display_sql` SQL query:\r\nhttps://github.com/dogsheep/dogsheep-beta/blob/4890ec87b5e2ec48940f32c9ad1f5aae25c75a4d/dogsheep_beta/__init__.py#L164-L171", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 724759588, "label": "Add search highlighting snippets"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/29#issuecomment-747029636", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/29", "id": 747029636, "node_id": "MDEyOklzc3VlQ29tbWVudDc0NzAyOTYzNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-16T21:14:03Z", "updated_at": "2020-12-16T21:14:03Z", "author_association": "MEMBER", "body": "I think I can do this as a cunning trick in `display_sql`. Consider this example query: https://til.simonwillison.net/tils?sql=select%0D%0A++path%2C%0D%0A++snippet%28til_fts%2C+-1%2C+%27b4de2a49c8%27%2C+%278c94a2ed4b%27%2C+%27...%27%2C+60%29+as+snippet%0D%0Afrom%0D%0A++til%0D%0A++join+til_fts+on+til.rowid+%3D+til_fts.rowid%0D%0Awhere%0D%0A++til_fts+match+escape_fts%28%3Aq%29%0D%0A++and+path+%3D+%27asgi_lifespan-test-httpx.md%27%0D%0A&q=pytest\r\n\r\n```sql\r\nselect\r\n path,\r\n snippet(til_fts, -1, 'b4de2a49c8', '8c94a2ed4b', '...', 60) as snippet\r\nfrom\r\n til\r\n join til_fts on til.rowid = til_fts.rowid\r\nwhere\r\n til_fts match escape_fts(:q)\r\n and path = 'asgi_lifespan-test-httpx.md'\r\n```\r\nThe `and path = 'asgi_lifespan-test-httpx.md'` bit means we only get back a specific document - but the snippet highlighting is applied to it.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 724759588, "label": "Add search highlighting snippets"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/58#issuecomment-746735889", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/58", "id": 746735889, "node_id": "MDEyOklzc3VlQ29tbWVudDc0NjczNTg4OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-16T17:59:50Z", "updated_at": "2020-12-16T17:59:50Z", "author_association": "MEMBER", "body": "I don't want to add a full HTML parser (like BeautifulSoup) as a dependency for this feature. Since the HTML comes from a single, trusted source (GitHub) I could probably handle this using [regular expressions](https://stackoverflow.com/a/1732454).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 769150394, "label": "Readme HTML has broken internal links"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/58#issuecomment-746734412", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/58", "id": 746734412, "node_id": "MDEyOklzc3VlQ29tbWVudDc0NjczNDQxMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-16T17:58:56Z", "updated_at": "2020-12-16T17:58:56Z", "author_association": "MEMBER", "body": "I'm going to rewrite those `` links to `` - but only if a corresponding `id=\"user-content-filtering-tables\"` element exists.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 769150394, "label": "Readme HTML has broken internal links"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/pull/29#issuecomment-739058820", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/29", "id": 739058820, "node_id": "MDEyOklzc3VlQ29tbWVudDczOTA1ODgyMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-04T22:32:35Z", "updated_at": "2020-12-04T22:32:35Z", "author_association": "MEMBER", "body": "Thanks for this!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 638375985, "label": "Fixed bug in SQL query for photo scores"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/53#issuecomment-735485677", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/53", "id": 735485677, "node_id": "MDEyOklzc3VlQ29tbWVudDczNTQ4NTY3Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-11-30T00:36:09Z", "updated_at": "2020-11-30T00:36:09Z", "author_association": "MEMBER", "body": "Given rate limits (see #51) this command might be better implemented by running a `git clone` into a temporary directory - doing so would retrieve all of the files in one go.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 753000405, "label": "Command for fetching file contents"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/51#issuecomment-735484186", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/51", "id": 735484186, "node_id": "MDEyOklzc3VlQ29tbWVudDczNTQ4NDE4Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-11-30T00:29:31Z", "updated_at": "2020-11-30T00:29:31Z", "author_association": "MEMBER", "body": "This just caused a failure in deploying the demo: https://github.com/dogsheep/github-to-sqlite/runs/1471304407?check_suite_focus=true\r\n\r\n```\r\n File \"/opt/hostedtoolcache/Python/3.8.6/x64/bin/github-to-sqlite\", line 33, in \r\n sys.exit(load_entry_point('github-to-sqlite', 'console_scripts', 'github-to-sqlite')())\r\n File \"/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/click/core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py\", line 142, in issue_comments\r\n for comment in utils.fetch_issue_comments(repo, token, issue):\r\n File \"/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py\", line 380, in fetch_issue_comments\r\n for comments in paginate(url, headers):\r\n File \"/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py\", line 472, in paginate\r\n raise GitHubError.from_response(response)\r\ngithub_to_sqlite.utils.GitHubError: ('API rate limit exceeded for user ID 9599.', 403)\r\nError: Process completed with exit code 1.\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 703246031, "label": "github-to-sqlite should handle rate limits better"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/46#issuecomment-735483820", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46", "id": 735483820, "node_id": "MDEyOklzc3VlQ29tbWVudDczNTQ4MzgyMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-11-30T00:27:47Z", "updated_at": "2020-11-30T00:27:47Z", "author_association": "MEMBER", "body": "So it looks like anything that pulls reviews needs to pull each review, then for each one pull the comments.\r\n\r\nI'm going to consider this blocked on smarter rate limit handling in #51.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 664485022, "label": "Feature: pull request reviews and comments"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/46#issuecomment-735483604", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46", "id": 735483604, "node_id": "MDEyOklzc3VlQ29tbWVudDczNTQ4MzYwNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-11-30T00:26:50Z", "updated_at": "2020-11-30T00:26:50Z", "author_association": "MEMBER", "body": "It seems like there's a lot missing from that - those aren't particularly interesting given the data that is returned.\r\n\r\nFrom the docs at https://docs.github.com/en/free-pro-team@latest/rest/reference/pulls#reviews it looks like each review consists of multiple comments, and the comments are where the useful material is - https://docs.github.com/en/free-pro-team@latest/rest/reference/pulls#list-comments-for-a-pull-request-review\r\n\r\n`github-to-sqlite get https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48/reviews/503368921/comments --accept 'application/vnd.github.v3+json'`\r\n\r\n```json\r\n[\r\n {\r\n \"id\": 500603838,\r\n \"node_id\": \"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDUwMDYwMzgzOA==\",\r\n \"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/comments/500603838\",\r\n \"pull_request_review_id\": 503368921,\r\n \"diff_hunk\": \"@@ -0,0 +1,370 @@\\n+[\\n+ {\\n+ \\\"url\\\": \\\"https://api.github.com/repos/simonw/datasette/pulls/571\\\",\\n+ \\\"id\\\": 313384926,\\n+ \\\"node_id\\\": \\\"MDExOlB1bGxSZXF1ZXN0MzEzMzg0OTI2\\\",\\n+ \\\"html_url\\\": \\\"https://github.com/simonw/datasette/pull/571\\\",\\n+ \\\"diff_url\\\": \\\"https://github.com/simonw/datasette/pull/571.diff\\\",\\n+ \\\"patch_url\\\": \\\"https://github.com/simonw/datasette/pull/571.patch\\\",\\n+ \\\"issue_url\\\": \\\"https://api.github.com/repos/simonw/datasette/issues/571\\\",\\n+ \\\"number\\\": 571,\\n+ \\\"state\\\": \\\"closed\\\",\\n+ \\\"locked\\\": false,\\n+ \\\"title\\\": \\\"detect_fts now works with alternative table escaping\\\",\\n+ \\\"user\\\": {\\n+ \\\"login\\\": \\\"simonw\\\",\\n+ \\\"id\\\": 9599,\\n+ \\\"node_id\\\": \\\"MDQ6VXNlcjk1OTk=\\\",\\n+ \\\"avatar_url\\\": \\\"https://avatars0.githubusercontent.com/u/9599?v=4\\\",\\n+ \\\"gravatar_id\\\": \\\"\\\",\\n+ \\\"url\\\": \\\"https://api.github.com/users/simonw\\\",\\n+ \\\"html_url\\\": \\\"https://github.com/simonw\\\",\\n+ \\\"followers_url\\\": \\\"https://api.github.com/users/simonw/followers\\\",\\n+ \\\"following_url\\\": \\\"https://api.github.com/users/simonw/following{/other_user}\\\",\\n+ \\\"gists_url\\\": \\\"https://api.github.com/users/simonw/gists{/gist_id}\\\",\\n+ \\\"starred_url\\\": \\\"https://api.github.com/users/simonw/starred{/owner}{/repo}\\\",\\n+ \\\"subscriptions_url\\\": \\\"https://api.github.com/users/simonw/subscriptions\\\",\\n+ \\\"organizations_url\\\": \\\"https://api.github.com/users/simonw/orgs\\\",\\n+ \\\"repos_url\\\": \\\"https://api.github.com/users/simonw/repos\\\",\\n+ \\\"events_url\\\": \\\"https://api.github.com/users/simonw/events{/privacy}\\\",\\n+ \\\"received_events_url\\\": \\\"https://api.github.com/users/simonw/received_events\\\",\\n+ \\\"type\\\": \\\"User\\\",\\n+ \\\"site_admin\\\": false\\n+ },\\n+ \\\"body\\\": \\\"Fixes #570\\\",\\n+ \\\"created_at\\\": \\\"2019-09-03T00:23:39Z\\\",\\n+ \\\"updated_at\\\": \\\"2019-09-03T00:32:28Z\\\",\\n+ \\\"closed_at\\\": \\\"2019-09-03T00:32:28Z\\\",\\n+ \\\"merged_at\\\": \\\"2019-09-03T00:32:28Z\\\",\\n+ \\\"merge_commit_sha\\\": \\\"2dc5c8dc259a0606162673d394ba8cc1c6f54428\\\",\\n+ \\\"assignee\\\": null,\\n+ \\\"assignees\\\": [\\n+\\n+ ],\\n+ \\\"requested_reviewers\\\": [\\n+\\n+ ],\\n+ \\\"requested_teams\\\": [\\n+\\n+ ],\\n+ \\\"labels\\\": [\\n+\\n+ ],\\n+ \\\"milestone\\\": null,\\n+ \\\"draft\\\": false,\\n+ \\\"commits_url\\\": \\\"https://api.github.com/repos/simonw/datasette/pulls/571/commits\\\",\\n+ \\\"review_comments_url\\\": \\\"https://api.github.com/repos/simonw/datasette/pulls/571/comments\\\",\\n+ \\\"review_comment_url\\\": \\\"https://api.github.com/repos/simonw/datasette/pulls/comments{/number}\\\",\\n+ \\\"comments_url\\\": \\\"https://api.github.com/repos/simonw/datasette/issues/571/comments\\\",\\n+ \\\"statuses_url\\\": \\\"https://api.github.com/repos/simonw/datasette/statuses/a85239f69261c10f1a9f90514c8b5d113cb94585\\\",\\n+ \\\"head\\\": {\\n+ \\\"label\\\": \\\"simonw:detect-fts\\\",\\n+ \\\"ref\\\": \\\"detect-fts\\\",\\n+ \\\"sha\\\": \\\"a85239f69261c10f1a9f90514c8b5d113cb94585\\\",\\n+ \\\"user\\\": {\\n+ \\\"login\\\": \\\"simonw\\\",\\n+ \\\"id\\\": 9599,\\n+ \\\"node_id\\\": \\\"MDQ6VXNlcjk1OTk=\\\",\\n+ \\\"avatar_url\\\": \\\"https://avatars0.githubusercontent.com/u/9599?v=4\\\",\\n+ \\\"gravatar_id\\\": \\\"\\\",\\n+ \\\"url\\\": \\\"https://api.github.com/users/simonw\\\",\\n+ \\\"html_url\\\": \\\"https://github.com/simonw\\\",\\n+ \\\"followers_url\\\": \\\"https://api.github.com/users/simonw/followers\\\",\\n+ \\\"following_url\\\": \\\"https://api.github.com/users/simonw/following{/other_user}\\\",\\n+ \\\"gists_url\\\": \\\"https://api.github.com/users/simonw/gists{/gist_id}\\\",\\n+ \\\"starred_url\\\": \\\"https://api.github.com/users/simonw/starred{/owner}{/repo}\\\",\\n+ \\\"subscriptions_url\\\": \\\"https://api.github.com/users/simonw/subscriptions\\\",\\n+ \\\"organizations_url\\\": \\\"https://api.github.com/users/simonw/orgs\\\",\\n+ \\\"repos_url\\\": \\\"https://api.github.com/users/simonw/repos\\\",\\n+ \\\"events_url\\\": \\\"https://api.github.com/users/simonw/events{/privacy}\\\",\\n+ \\\"received_events_url\\\": \\\"https://api.github.com/users/simonw/received_events\\\",\\n+ \\\"type\\\": \\\"User\\\",\\n+ \\\"site_admin\\\": false\\n+ },\\n+ \\\"repo\\\": {\\n+ \\\"id\\\": 107914493,\\n+ \\\"node_id\\\": \\\"MDEwOlJlcG9zaXRvcnkxMDc5MTQ0OTM=\\\",\\n+ \\\"name\\\": \\\"datasette\\\",\\n+ \\\"full_name\\\": \\\"simonw/datasette\\\",\\n+ \\\"private\\\": false,\\n+ \\\"owner\\\": {\\n+ \\\"login\\\": \\\"simonw\\\",\\n+ \\\"id\\\": 9599,\\n+ \\\"node_id\\\": \\\"MDQ6VXNlcjk1OTk=\\\",\\n+ \\\"avatar_url\\\": \\\"https://avatars0.githubusercontent.com/u/9599?v=4\\\",\\n+ \\\"gravatar_id\\\": \\\"\\\",\\n+ \\\"url\\\": \\\"https://api.github.com/users/simonw\\\",\\n+ \\\"html_url\\\": \\\"https://github.com/simonw\\\",\\n+ \\\"followers_url\\\": \\\"https://api.github.com/users/simonw/followers\\\",\\n+ \\\"following_url\\\": \\\"https://api.github.com/users/simonw/following{/other_user}\\\",\\n+ \\\"gists_url\\\": \\\"https://api.github.com/users/simonw/gists{/gist_id}\\\",\\n+ \\\"starred_url\\\": \\\"https://api.github.com/users/simonw/starred{/owner}{/repo}\\\",\\n+ \\\"subscriptions_url\\\": \\\"https://api.github.com/users/simonw/subscriptions\\\",\\n+ \\\"organizations_url\\\": \\\"https://api.github.com/users/simonw/orgs\\\",\\n+ \\\"repos_url\\\": \\\"https://api.github.com/users/simonw/repos\\\",\\n+ \\\"events_url\\\": \\\"https://api.github.com/users/simonw/events{/privacy}\\\",\\n+ \\\"received_events_url\\\": \\\"https://api.github.com/users/simonw/received_events\\\",\\n+ \\\"type\\\": \\\"User\\\",\\n+ \\\"site_admin\\\": false\\n+ },\\n+ \\\"html_url\\\": \\\"https://github.com/simonw/datasette\\\",\\n+ \\\"description\\\": \\\"An open source multi-tool for exploring and publishing data\\\",\\n+ \\\"fork\\\": false,\\n+ \\\"url\\\": \\\"https://api.github.com/repos/simonw/datasette\\\",\\n+ \\\"forks_url\\\": \\\"https://api.github.com/repos/simonw/datasette/forks\\\",\\n+ \\\"keys_url\\\": \\\"https://api.github.com/repos/simonw/datasette/keys{/key_id}\\\",\\n+ \\\"collaborators_url\\\": \\\"https://api.github.com/repos/simonw/datasette/collaborators{/collaborator}\\\",\\n+ \\\"teams_url\\\": \\\"https://api.github.com/repos/simonw/datasette/teams\\\",\\n+ \\\"hooks_url\\\": \\\"https://api.github.com/repos/simonw/datasette/hooks\\\",\\n+ \\\"issue_events_url\\\": \\\"https://api.github.com/repos/simonw/datasette/issues/events{/number}\\\",\\n+ \\\"events_url\\\": \\\"https://api.github.com/repos/simonw/datasette/events\\\",\\n+ \\\"assignees_url\\\": \\\"https://api.github.com/repos/simonw/datasette/assignees{/user}\\\",\\n+ \\\"branches_url\\\": \\\"https://api.github.com/repos/simonw/datasette/branches{/branch}\\\",\\n+ \\\"tags_url\\\": \\\"https://api.github.com/repos/simonw/datasette/tags\\\",\\n+ \\\"blobs_url\\\": \\\"https://api.github.com/repos/simonw/datasette/git/blobs{/sha}\\\",\\n+ \\\"git_tags_url\\\": \\\"https://api.github.com/repos/simonw/datasette/git/tags{/sha}\\\",\\n+ \\\"git_refs_url\\\": \\\"https://api.github.com/repos/simonw/datasette/git/refs{/sha}\\\",\\n+ \\\"trees_url\\\": \\\"https://api.github.com/repos/simonw/datasette/git/trees{/sha}\\\",\\n+ \\\"statuses_url\\\": \\\"https://api.github.com/repos/simonw/datasette/statuses/{sha}\\\",\\n+ \\\"languages_url\\\": \\\"https://api.github.com/repos/simonw/datasette/languages\\\",\\n+ \\\"stargazers_url\\\": \\\"https://api.github.com/repos/simonw/datasette/stargazers\\\",\\n+ \\\"contributors_url\\\": \\\"https://api.github.com/repos/simonw/datasette/contributors\\\",\\n+ \\\"subscribers_url\\\": \\\"https://api.github.com/repos/simonw/datasette/subscribers\\\",\\n+ \\\"subscription_url\\\": \\\"https://api.github.com/repos/simonw/datasette/subscription\\\",\\n+ \\\"commits_url\\\": \\\"https://api.github.com/repos/simonw/datasette/commits{/sha}\\\",\\n+ \\\"git_commits_url\\\": \\\"https://api.github.com/repos/simonw/datasette/git/commits{/sha}\\\",\\n+ \\\"comments_url\\\": \\\"https://api.github.com/repos/simonw/datasette/comments{/number}\\\",\\n+ \\\"issue_comment_url\\\": \\\"https://api.github.com/repos/simonw/datasette/issues/comments{/number}\\\",\\n+ \\\"contents_url\\\": \\\"https://api.github.com/repos/simonw/datasette/contents/{+path}\\\",\\n+ \\\"compare_url\\\": \\\"https://api.github.com/repos/simonw/datasette/compare/{base}...{head}\\\",\\n+ \\\"merges_url\\\": \\\"https://api.github.com/repos/simonw/datasette/merges\\\",\\n+ \\\"archive_url\\\": \\\"https://api.github.com/repos/simonw/datasette/{archive_format}{/ref}\\\",\\n+ \\\"downloads_url\\\": \\\"https://api.github.com/repos/simonw/datasette/downloads\\\",\\n+ \\\"issues_url\\\": \\\"https://api.github.com/repos/simonw/datasette/issues{/number}\\\",\\n+ \\\"pulls_url\\\": \\\"https://api.github.com/repos/simonw/datasette/pulls{/number}\\\",\\n+ \\\"milestones_url\\\": \\\"https://api.github.com/repos/simonw/datasette/milestones{/number}\\\",\\n+ \\\"notifications_url\\\": \\\"https://api.github.com/repos/simonw/datasette/notifications{?since,all,participating}\\\",\\n+ \\\"labels_url\\\": \\\"https://api.github.com/repos/simonw/datasette/labels{/name}\\\",\\n+ \\\"releases_url\\\": \\\"https://api.github.com/repos/simonw/datasette/releases{/id}\\\",\\n+ \\\"deployments_url\\\": \\\"https://api.github.com/repos/simonw/datasette/deployments\\\",\\n+ \\\"created_at\\\": \\\"2017-10-23T00:39:03Z\\\",\\n+ \\\"updated_at\\\": \\\"2020-07-27T20:42:15Z\\\",\\n+ \\\"pushed_at\\\": \\\"2020-07-26T01:21:05Z\\\",\\n+ \\\"git_url\\\": \\\"git://github.com/simonw/datasette.git\\\",\\n+ \\\"ssh_url\\\": \\\"git@github.com:simonw/datasette.git\\\",\\n+ \\\"clone_url\\\": \\\"https://github.com/simonw/datasette.git\\\",\\n+ \\\"svn_url\\\": \\\"https://github.com/simonw/datasette\\\",\\n+ \\\"homepage\\\": \\\"http://datasette.readthedocs.io/\\\",\\n+ \\\"size\\\": 3487,\\n+ \\\"stargazers_count\\\": 3642,\\n+ \\\"watchers_count\\\": 3642,\\n+ \\\"language\\\": \\\"Python\\\",\\n+ \\\"has_issues\\\": true,\\n+ \\\"has_projects\\\": false,\\n+ \\\"has_downloads\\\": true,\\n+ \\\"has_wiki\\\": true,\\n+ \\\"has_pages\\\": false,\\n+ \\\"forks_count\\\": 206,\\n+ \\\"mirror_url\\\": null,\\n+ \\\"archived\\\": false,\\n+ \\\"disabled\\\": false,\\n+ \\\"open_issues_count\\\": 190,\\n+ \\\"license\\\": {\\n+ \\\"key\\\": \\\"apache-2.0\\\",\\n+ \\\"name\\\": \\\"Apache License 2.0\\\",\\n+ \\\"spdx_id\\\": \\\"Apache-2.0\\\",\\n+ \\\"url\\\": \\\"https://api.github.com/licenses/apache-2.0\\\",\\n+ \\\"node_id\\\": \\\"MDc6TGljZW5zZTI=\\\"\\n+ },\\n+ \\\"forks\\\": 206,\\n+ \\\"open_issues\\\": 190,\\n+ \\\"watchers\\\": 3642,\\n+ \\\"default_branch\\\": \\\"master\\\"\\n+ }\\n+ },\\n+ \\\"base\\\": {\\n+ \\\"label\\\": \\\"simonw:master\\\",\\n+ \\\"ref\\\": \\\"master\\\",\\n+ \\\"sha\\\": \\\"f04deebec4f3842f7bd610cd5859de529f77d50e\\\",\\n+ \\\"user\\\": {\\n+ \\\"login\\\": \\\"simonw\\\",\\n+ \\\"id\\\": 9599,\\n+ \\\"node_id\\\": \\\"MDQ6VXNlcjk1OTk=\\\",\\n+ \\\"avatar_url\\\": \\\"https://avatars0.githubusercontent.com/u/9599?v=4\\\",\\n+ \\\"gravatar_id\\\": \\\"\\\",\\n+ \\\"url\\\": \\\"https://api.github.com/users/simonw\\\",\\n+ \\\"html_url\\\": \\\"https://github.com/simonw\\\",\\n+ \\\"followers_url\\\": \\\"https://api.github.com/users/simonw/followers\\\",\\n+ \\\"following_url\\\": \\\"https://api.github.com/users/simonw/following{/other_user}\\\",\\n+ \\\"gists_url\\\": \\\"https://api.github.com/users/simonw/gists{/gist_id}\\\",\\n+ \\\"starred_url\\\": \\\"https://api.github.com/users/simonw/starred{/owner}{/repo}\\\",\\n+ \\\"subscriptions_url\\\": \\\"https://api.github.com/users/simonw/subscriptions\\\",\\n+ \\\"organizations_url\\\": \\\"https://api.github.com/users/simonw/orgs\\\",\\n+ \\\"repos_url\\\": \\\"https://api.github.com/users/simonw/repos\\\",\\n+ \\\"events_url\\\": \\\"https://api.github.com/users/simonw/events{/privacy}\\\",\\n+ \\\"received_events_url\\\": \\\"https://api.github.com/users/simonw/received_events\\\",\\n+ \\\"type\\\": \\\"User\\\",\\n+ \\\"site_admin\\\": false\\n+ },\\n+ \\\"repo\\\": {\\n+ \\\"id\\\": 107914493,\\n+ \\\"node_id\\\": \\\"MDEwOlJlcG9zaXRvcnkxMDc5MTQ0OTM=\\\",\\n+ \\\"name\\\": \\\"datasette\\\",\\n+ \\\"full_name\\\": \\\"simonw/datasette\\\",\\n+ \\\"private\\\": false,\\n+ \\\"owner\\\": {\\n+ \\\"login\\\": \\\"simonw\\\",\\n+ \\\"id\\\": 9599,\\n+ \\\"node_id\\\": \\\"MDQ6VXNlcjk1OTk=\\\",\\n+ \\\"avatar_url\\\": \\\"https://avatars0.githubusercontent.com/u/9599?v=4\\\",\\n+ \\\"gravatar_id\\\": \\\"\\\",\\n+ \\\"url\\\": \\\"https://api.github.com/users/simonw\\\",\\n+ \\\"html_url\\\": \\\"https://github.com/simonw\\\",\\n+ \\\"followers_url\\\": \\\"https://api.github.com/users/simonw/followers\\\",\\n+ \\\"following_url\\\": \\\"https://api.github.com/users/simonw/following{/other_user}\\\",\\n+ \\\"gists_url\\\": \\\"https://api.github.com/users/simonw/gists{/gist_id}\\\",\\n+ \\\"starred_url\\\": \\\"https://api.github.com/users/simonw/starred{/owner}{/repo}\\\",\\n+ \\\"subscriptions_url\\\": \\\"https://api.github.com/users/simonw/subscriptions\\\",\\n+ \\\"organizations_url\\\": \\\"https://api.github.com/users/simonw/orgs\\\",\\n+ \\\"repos_url\\\": \\\"https://api.github.com/users/simonw/repos\\\",\\n+ \\\"events_url\\\": \\\"https://api.github.com/users/simonw/events{/privacy}\\\",\\n+ \\\"received_events_url\\\": \\\"https://api.github.com/users/simonw/received_events\\\",\\n+ \\\"type\\\": \\\"User\\\",\\n+ \\\"site_admin\\\": false\\n+ },\\n+ \\\"html_url\\\": \\\"https://github.com/simonw/datasette\\\",\\n+ \\\"description\\\": \\\"An open source multi-tool for exploring and publishing data\\\",\\n+ \\\"fork\\\": false,\\n+ \\\"url\\\": \\\"https://api.github.com/repos/simonw/datasette\\\",\\n+ \\\"forks_url\\\": \\\"https://api.github.com/repos/simonw/datasette/forks\\\",\\n+ \\\"keys_url\\\": \\\"https://api.github.com/repos/simonw/datasette/keys{/key_id}\\\",\\n+ \\\"collaborators_url\\\": \\\"https://api.github.com/repos/simonw/datasette/collaborators{/collaborator}\\\",\\n+ \\\"teams_url\\\": \\\"https://api.github.com/repos/simonw/datasette/teams\\\",\\n+ \\\"hooks_url\\\": \\\"https://api.github.com/repos/simonw/datasette/hooks\\\",\\n+ \\\"issue_events_url\\\": \\\"https://api.github.com/repos/simonw/datasette/issues/events{/number}\\\",\\n+ \\\"events_url\\\": \\\"https://api.github.com/repos/simonw/datasette/events\\\",\\n+ \\\"assignees_url\\\": \\\"https://api.github.com/repos/simonw/datasette/assignees{/user}\\\",\\n+ \\\"branches_url\\\": \\\"https://api.github.com/repos/simonw/datasette/branches{/branch}\\\",\\n+ \\\"tags_url\\\": \\\"https://api.github.com/repos/simonw/datasette/tags\\\",\\n+ \\\"blobs_url\\\": \\\"https://api.github.com/repos/simonw/datasette/git/blobs{/sha}\\\",\\n+ \\\"git_tags_url\\\": \\\"https://api.github.com/repos/simonw/datasette/git/tags{/sha}\\\",\\n+ \\\"git_refs_url\\\": \\\"https://api.github.com/repos/simonw/datasette/git/refs{/sha}\\\",\\n+ \\\"trees_url\\\": \\\"https://api.github.com/repos/simonw/datasette/git/trees{/sha}\\\",\\n+ \\\"statuses_url\\\": \\\"https://api.github.com/repos/simonw/datasette/statuses/{sha}\\\",\\n+ \\\"languages_url\\\": \\\"https://api.github.com/repos/simonw/datasette/languages\\\",\\n+ \\\"stargazers_url\\\": \\\"https://api.github.com/repos/simonw/datasette/stargazers\\\",\\n+ \\\"contributors_url\\\": \\\"https://api.github.com/repos/simonw/datasette/contributors\\\",\\n+ \\\"subscribers_url\\\": \\\"https://api.github.com/repos/simonw/datasette/subscribers\\\",\\n+ \\\"subscription_url\\\": \\\"https://api.github.com/repos/simonw/datasette/subscription\\\",\\n+ \\\"commits_url\\\": \\\"https://api.github.com/repos/simonw/datasette/commits{/sha}\\\",\\n+ \\\"git_commits_url\\\": \\\"https://api.github.com/repos/simonw/datasette/git/commits{/sha}\\\",\\n+ \\\"comments_url\\\": \\\"https://api.github.com/repos/simonw/datasette/comments{/number}\\\",\\n+ \\\"issue_comment_url\\\": \\\"https://api.github.com/repos/simonw/datasette/issues/comments{/number}\\\",\\n+ \\\"contents_url\\\": \\\"https://api.github.com/repos/simonw/datasette/contents/{+path}\\\",\\n+ \\\"compare_url\\\": \\\"https://api.github.com/repos/simonw/datasette/compare/{base}...{head}\\\",\\n+ \\\"merges_url\\\": \\\"https://api.github.com/repos/simonw/datasette/merges\\\",\\n+ \\\"archive_url\\\": \\\"https://api.github.com/repos/simonw/datasette/{archive_format}{/ref}\\\",\\n+ \\\"downloads_url\\\": \\\"https://api.github.com/repos/simonw/datasette/downloads\\\",\\n+ \\\"issues_url\\\": \\\"https://api.github.com/repos/simonw/datasette/issues{/number}\\\",\\n+ \\\"pulls_url\\\": \\\"https://api.github.com/repos/simonw/datasette/pulls{/number}\\\",\\n+ \\\"milestones_url\\\": \\\"https://api.github.com/repos/simonw/datasette/milestones{/number}\\\",\\n+ \\\"notifications_url\\\": \\\"https://api.github.com/repos/simonw/datasette/notifications{?since,all,participating}\\\",\\n+ \\\"labels_url\\\": \\\"https://api.github.com/repos/simonw/datasette/labels{/name}\\\",\\n+ \\\"releases_url\\\": \\\"https://api.github.com/repos/simonw/datasette/releases{/id}\\\",\\n+ \\\"deployments_url\\\": \\\"https://api.github.com/repos/simonw/datasette/deployments\\\",\\n+ \\\"created_at\\\": \\\"2017-10-23T00:39:03Z\\\",\\n+ \\\"updated_at\\\": \\\"2020-07-27T20:42:15Z\\\",\\n+ \\\"pushed_at\\\": \\\"2020-07-26T01:21:05Z\\\",\\n+ \\\"git_url\\\": \\\"git://github.com/simonw/datasette.git\\\",\\n+ \\\"ssh_url\\\": \\\"git@github.com:simonw/datasette.git\\\",\\n+ \\\"clone_url\\\": \\\"https://github.com/simonw/datasette.git\\\",\\n+ \\\"svn_url\\\": \\\"https://github.com/simonw/datasette\\\",\\n+ \\\"homepage\\\": \\\"http://datasette.readthedocs.io/\\\",\\n+ \\\"size\\\": 3487,\\n+ \\\"stargazers_count\\\": 3642,\\n+ \\\"watchers_count\\\": 3642,\\n+ \\\"language\\\": \\\"Python\\\",\\n+ \\\"has_issues\\\": true,\\n+ \\\"has_projects\\\": false,\\n+ \\\"has_downloads\\\": true,\\n+ \\\"has_wiki\\\": true,\\n+ \\\"has_pages\\\": false,\\n+ \\\"forks_count\\\": 206,\\n+ \\\"mirror_url\\\": null,\\n+ \\\"archived\\\": false,\\n+ \\\"disabled\\\": false,\\n+ \\\"open_issues_count\\\": 190,\\n+ \\\"license\\\": {\\n+ \\\"key\\\": \\\"apache-2.0\\\",\\n+ \\\"name\\\": \\\"Apache License 2.0\\\",\\n+ \\\"spdx_id\\\": \\\"Apache-2.0\\\",\\n+ \\\"url\\\": \\\"https://api.github.com/licenses/apache-2.0\\\",\\n+ \\\"node_id\\\": \\\"MDc6TGljZW5zZTI=\\\"\\n+ },\\n+ \\\"forks\\\": 206,\\n+ \\\"open_issues\\\": 190,\\n+ \\\"watchers\\\": 3642,\\n+ \\\"default_branch\\\": \\\"master\\\"\\n+ }\\n+ },\\n+ \\\"_links\\\": {\\n+ \\\"self\\\": {\\n+ \\\"href\\\": \\\"https://api.github.com/repos/simonw/datasette/pulls/571\\\"\\n+ },\\n+ \\\"html\\\": {\\n+ \\\"href\\\": \\\"https://github.com/simonw/datasette/pull/571\\\"\\n+ },\\n+ \\\"issue\\\": {\\n+ \\\"href\\\": \\\"https://api.github.com/repos/simonw/datasette/issues/571\\\"\\n+ },\\n+ \\\"comments\\\": {\\n+ \\\"href\\\": \\\"https://api.github.com/repos/simonw/datasette/issues/571/comments\\\"\\n+ },\\n+ \\\"review_comments\\\": {\\n+ \\\"href\\\": \\\"https://api.github.com/repos/simonw/datasette/pulls/571/comments\\\"\\n+ },\\n+ \\\"review_comment\\\": {\\n+ \\\"href\\\": \\\"https://api.github.com/repos/simonw/datasette/pulls/comments{/number}\\\"\\n+ },\\n+ \\\"commits\\\": {\\n+ \\\"href\\\": \\\"https://api.github.com/repos/simonw/datasette/pulls/571/commits\\\"\\n+ },\\n+ \\\"statuses\\\": {\\n+ \\\"href\\\": \\\"https://api.github.com/repos/simonw/datasette/statuses/a85239f69261c10f1a9f90514c8b5d113cb94585\\\"\\n+ }\\n+ },\\n+ \\\"author_association\\\": \\\"OWNER\\\",\\n+ \\\"active_lock_reason\\\": null,\\n+ \\\"merged\\\": true,\\n+ \\\"mergeable\\\": null,\\n+ \\\"rebaseable\\\": null,\\n+ \\\"mergeable_state\\\": \\\"unknown\\\",\\n+ \\\"merged_by\\\": {\",\r\n \"path\": \"tests/pull_requests.json\",\r\n \"position\": 342,\r\n \"original_position\": 342,\r\n \"commit_id\": \"3a0d5c498f9faae4e40aab204cd01b965a4f61f3\",\r\n \"user\": {\r\n \"login\": \"simonw\",\r\n \"id\": 9599,\r\n \"node_id\": \"MDQ6VXNlcjk1OTk=\",\r\n \"avatar_url\": \"https://avatars0.githubusercontent.com/u/9599?u=5968723deb1a55b82620e106f5ca58e9b11a0942&v=4\",\r\n \"gravatar_id\": \"\",\r\n \"url\": \"https://api.github.com/users/simonw\",\r\n \"html_url\": \"https://github.com/simonw\",\r\n \"followers_url\": \"https://api.github.com/users/simonw/followers\",\r\n \"following_url\": \"https://api.github.com/users/simonw/following{/other_user}\",\r\n \"gists_url\": \"https://api.github.com/users/simonw/gists{/gist_id}\",\r\n \"starred_url\": \"https://api.github.com/users/simonw/starred{/owner}{/repo}\",\r\n \"subscriptions_url\": \"https://api.github.com/users/simonw/subscriptions\",\r\n \"organizations_url\": \"https://api.github.com/users/simonw/orgs\",\r\n \"repos_url\": \"https://api.github.com/users/simonw/repos\",\r\n \"events_url\": \"https://api.github.com/users/simonw/events{/privacy}\",\r\n \"received_events_url\": \"https://api.github.com/users/simonw/received_events\",\r\n \"type\": \"User\",\r\n \"site_admin\": false\r\n },\r\n \"body\": \"Running this should create a `merged_by` column on the `pull_requests` table which is a foreign key to the `users` table.\",\r\n \"created_at\": \"2020-10-06T21:22:47Z\",\r\n \"updated_at\": \"2020-10-20T20:56:33Z\",\r\n \"html_url\": \"https://github.com/dogsheep/github-to-sqlite/pull/48#discussion_r500603838\",\r\n \"pull_request_url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48\",\r\n \"author_association\": \"MEMBER\",\r\n \"_links\": {\r\n \"self\": {\r\n \"href\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/comments/500603838\"\r\n },\r\n \"html\": {\r\n \"href\": \"https://github.com/dogsheep/github-to-sqlite/pull/48#discussion_r500603838\"\r\n },\r\n \"pull_request\": {\r\n \"href\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48\"\r\n }\r\n },\r\n \"original_commit_id\": \"4f33b850bd37829262dd29e1c520afffebedc19c\"\r\n },\r\n {\r\n \"id\": 500606198,\r\n \"node_id\": \"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDUwMDYwNjE5OA==\",\r\n \"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/comments/500606198\",\r\n \"pull_request_review_id\": 503368921,\r\n \"diff_hunk\": \"@@ -0,0 +1,124 @@\\n+from github_to_sqlite import utils\\n+import pytest\\n+import pathlib\\n+import sqlite_utils\\n+from sqlite_utils.db import ForeignKey\\n+import json\\n+\\n+\\n+@pytest.fixture\\n+def pull_requests():\\n+ return json.load(open(pathlib.Path(__file__).parent / \\\"pull_requests.json\\\"))\\n+\\n+\\n+@pytest.fixture\\n+def db(pull_requests):\\n+ db = sqlite_utils.Database(memory=True)\\n+ db[\\\"repos\\\"].insert(\\n+ {\\\"id\\\": 1},\\n+ pk=\\\"id\\\",\\n+ columns={\\\"organization\\\": int, \\\"topics\\\": str, \\\"name\\\": str, \\\"description\\\": str},\\n+ )\\n+ utils.save_pull_requests(db, pull_requests, {\\\"id\\\": 1})\\n+ return db\\n+\\n+\\n+def test_tables(db):\\n+ assert {\\\"pull_requests\\\", \\\"users\\\", \\\"repos\\\", \\\"milestones\\\"} == set(\\n+ db.table_names()\\n+ )\\n+ assert {\\n+ ForeignKey(\\n+ table=\\\"pull_requests\\\", column=\\\"repo\\\", other_table=\\\"repos\\\", other_column=\\\"id\\\"\\n+ ),\\n+ ForeignKey(\\n+ table=\\\"pull_requests\\\",\\n+ column=\\\"milestone\\\",\\n+ other_table=\\\"milestones\\\",\\n+ other_column=\\\"id\\\",\\n+ ),\\n+ ForeignKey(\\n+ table=\\\"pull_requests\\\", column=\\\"assignee\\\", other_table=\\\"users\\\", other_column=\\\"id\\\"\\n+ ),\\n+ ForeignKey(\\n+ table=\\\"pull_requests\\\", column=\\\"user\\\", other_table=\\\"users\\\", other_column=\\\"id\\\"\\n+ ),\\n+ } == set(db[\\\"pull_requests\\\"].foreign_keys)\\n+\\n+\\n+def test_pull_requests(db):\\n+ pull_request_rows = list(db[\\\"pull_requests\\\"].rows)\\n+ assert [\\n+ {\\n+ 'id': 313384926,\\n+ 'node_id': 'MDExOlB1bGxSZXF1ZXN0MzEzMzg0OTI2',\\n+ 'number': 571,\\n+ 'state': 'closed',\\n+ 'locked': 0,\\n+ 'title': 'detect_fts now works with alternative table escaping',\\n+ 'user': 9599,\\n+ 'body': 'Fixes #570',\\n+ 'created_at': '2019-09-03T00:23:39Z',\\n+ 'updated_at': '2019-09-03T00:32:28Z',\\n+ 'closed_at': '2019-09-03T00:32:28Z',\\n+ 'merged_at': '2019-09-03T00:32:28Z',\\n+ 'merge_commit_sha': '2dc5c8dc259a0606162673d394ba8cc1c6f54428',\\n+ 'assignee': None,\\n+ 'milestone': None,\\n+ 'draft': 0,\\n+ 'head': 'a85239f69261c10f1a9f90514c8b5d113cb94585',\\n+ 'base': 'f04deebec4f3842f7bd610cd5859de529f77d50e',\\n+ 'author_association': 'OWNER',\\n+ 'merged': 1,\\n+ 'mergeable': None,\\n+ 'rebaseable': None,\\n+ 'mergeable_state': 'unknown',\\n+ 'merged_by': '{\\\"login\\\": \\\"simonw\\\", \\\"id\\\": 9599, \\\"node_id\\\": \\\"MDQ6VXNlcjk1OTk=\\\", \\\"avatar_url\\\": \\\"https://avatars0.githubusercontent.com/u/9599?v=4\\\", \\\"gravatar_id\\\": \\\"\\\", \\\"url\\\": \\\"https://api.github.com/users/simonw\\\", \\\"html_url\\\": \\\"https://github.com/simonw\\\", \\\"followers_url\\\": \\\"https://api.github.com/users/simonw/followers\\\", \\\"following_url\\\": \\\"https://api.github.com/users/simonw/following{/other_user}\\\", \\\"gists_url\\\": \\\"https://api.github.com/users/simonw/gists{/gist_id}\\\", \\\"starred_url\\\": \\\"https://api.github.com/users/simonw/starred{/owner}{/repo}\\\", \\\"subscriptions_url\\\": \\\"https://api.github.com/users/simonw/subscriptions\\\", \\\"organizations_url\\\": \\\"https://api.github.com/users/simonw/orgs\\\", \\\"repos_url\\\": \\\"https://api.github.com/users/simonw/repos\\\", \\\"events_url\\\": \\\"https://api.github.com/users/simonw/events{/privacy}\\\", \\\"received_events_url\\\": \\\"https://api.github.com/users/simonw/received_events\\\", \\\"type\\\": \\\"User\\\", \\\"site_admin\\\": false}',\",\r\n \"path\": \"tests/test_pull_requests.py\",\r\n \"position\": null,\r\n \"original_position\": 76,\r\n \"commit_id\": \"3a0d5c498f9faae4e40aab204cd01b965a4f61f3\",\r\n \"user\": {\r\n \"login\": \"simonw\",\r\n \"id\": 9599,\r\n \"node_id\": \"MDQ6VXNlcjk1OTk=\",\r\n \"avatar_url\": \"https://avatars0.githubusercontent.com/u/9599?u=5968723deb1a55b82620e106f5ca58e9b11a0942&v=4\",\r\n \"gravatar_id\": \"\",\r\n \"url\": \"https://api.github.com/users/simonw\",\r\n \"html_url\": \"https://github.com/simonw\",\r\n \"followers_url\": \"https://api.github.com/users/simonw/followers\",\r\n \"following_url\": \"https://api.github.com/users/simonw/following{/other_user}\",\r\n \"gists_url\": \"https://api.github.com/users/simonw/gists{/gist_id}\",\r\n \"starred_url\": \"https://api.github.com/users/simonw/starred{/owner}{/repo}\",\r\n \"subscriptions_url\": \"https://api.github.com/users/simonw/subscriptions\",\r\n \"organizations_url\": \"https://api.github.com/users/simonw/orgs\",\r\n \"repos_url\": \"https://api.github.com/users/simonw/repos\",\r\n \"events_url\": \"https://api.github.com/users/simonw/events{/privacy}\",\r\n \"received_events_url\": \"https://api.github.com/users/simonw/received_events\",\r\n \"type\": \"User\",\r\n \"site_admin\": false\r\n },\r\n \"body\": \"See above - this should be 9599, an integer reference to the row in the users table.\",\r\n \"created_at\": \"2020-10-06T21:27:43Z\",\r\n \"updated_at\": \"2020-10-20T20:56:33Z\",\r\n \"html_url\": \"https://github.com/dogsheep/github-to-sqlite/pull/48#discussion_r500606198\",\r\n \"pull_request_url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48\",\r\n \"author_association\": \"MEMBER\",\r\n \"_links\": {\r\n \"self\": {\r\n \"href\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/comments/500606198\"\r\n },\r\n \"html\": {\r\n \"href\": \"https://github.com/dogsheep/github-to-sqlite/pull/48#discussion_r500606198\"\r\n },\r\n \"pull_request\": {\r\n \"href\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48\"\r\n }\r\n },\r\n \"original_commit_id\": \"4f33b850bd37829262dd29e1c520afffebedc19c\"\r\n },\r\n {\r\n \"id\": 500606665,\r\n \"node_id\": \"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDUwMDYwNjY2NQ==\",\r\n \"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/comments/500606665\",\r\n \"pull_request_review_id\": 503368921,\r\n \"diff_hunk\": \"@@ -0,0 +1,124 @@\\n+from github_to_sqlite import utils\\n+import pytest\\n+import pathlib\\n+import sqlite_utils\\n+from sqlite_utils.db import ForeignKey\\n+import json\\n+\\n+\\n+@pytest.fixture\\n+def pull_requests():\\n+ return json.load(open(pathlib.Path(__file__).parent / \\\"pull_requests.json\\\"))\\n+\\n+\\n+@pytest.fixture\\n+def db(pull_requests):\\n+ db = sqlite_utils.Database(memory=True)\\n+ db[\\\"repos\\\"].insert(\\n+ {\\\"id\\\": 1},\\n+ pk=\\\"id\\\",\\n+ columns={\\\"organization\\\": int, \\\"topics\\\": str, \\\"name\\\": str, \\\"description\\\": str},\\n+ )\\n+ utils.save_pull_requests(db, pull_requests, {\\\"id\\\": 1})\\n+ return db\\n+\\n+\\n+def test_tables(db):\\n+ assert {\\\"pull_requests\\\", \\\"users\\\", \\\"repos\\\", \\\"milestones\\\"} == set(\\n+ db.table_names()\\n+ )\\n+ assert {\\n+ ForeignKey(\\n+ table=\\\"pull_requests\\\", column=\\\"repo\\\", other_table=\\\"repos\\\", other_column=\\\"id\\\"\\n+ ),\\n+ ForeignKey(\\n+ table=\\\"pull_requests\\\",\\n+ column=\\\"milestone\\\",\\n+ other_table=\\\"milestones\\\",\\n+ other_column=\\\"id\\\",\\n+ ),\\n+ ForeignKey(\\n+ table=\\\"pull_requests\\\", column=\\\"assignee\\\", other_table=\\\"users\\\", other_column=\\\"id\\\"\\n+ ),\\n+ ForeignKey(\\n+ table=\\\"pull_requests\\\", column=\\\"user\\\", other_table=\\\"users\\\", other_column=\\\"id\\\"\\n+ ),\\n+ } == set(db[\\\"pull_requests\\\"].foreign_keys)\\n+\\n+\\n+def test_pull_requests(db):\\n+ pull_request_rows = list(db[\\\"pull_requests\\\"].rows)\\n+ assert [\\n+ {\\n+ 'id': 313384926,\",\r\n \"path\": \"tests/test_pull_requests.py\",\r\n \"position\": null,\r\n \"original_position\": 53,\r\n \"commit_id\": \"3a0d5c498f9faae4e40aab204cd01b965a4f61f3\",\r\n \"user\": {\r\n \"login\": \"simonw\",\r\n \"id\": 9599,\r\n \"node_id\": \"MDQ6VXNlcjk1OTk=\",\r\n \"avatar_url\": \"https://avatars0.githubusercontent.com/u/9599?u=5968723deb1a55b82620e106f5ca58e9b11a0942&v=4\",\r\n \"gravatar_id\": \"\",\r\n \"url\": \"https://api.github.com/users/simonw\",\r\n \"html_url\": \"https://github.com/simonw\",\r\n \"followers_url\": \"https://api.github.com/users/simonw/followers\",\r\n \"following_url\": \"https://api.github.com/users/simonw/following{/other_user}\",\r\n \"gists_url\": \"https://api.github.com/users/simonw/gists{/gist_id}\",\r\n \"starred_url\": \"https://api.github.com/users/simonw/starred{/owner}{/repo}\",\r\n \"subscriptions_url\": \"https://api.github.com/users/simonw/subscriptions\",\r\n \"organizations_url\": \"https://api.github.com/users/simonw/orgs\",\r\n \"repos_url\": \"https://api.github.com/users/simonw/repos\",\r\n \"events_url\": \"https://api.github.com/users/simonw/events{/privacy}\",\r\n \"received_events_url\": \"https://api.github.com/users/simonw/received_events\",\r\n \"type\": \"User\",\r\n \"site_admin\": false\r\n },\r\n \"body\": \"Minor detail: I use Black for this repo, which requires double quotes - running \\\"black .\\\" in the root directory (with the latest version of Black) should handle this for you.\",\r\n \"created_at\": \"2020-10-06T21:28:31Z\",\r\n \"updated_at\": \"2020-10-20T20:56:33Z\",\r\n \"html_url\": \"https://github.com/dogsheep/github-to-sqlite/pull/48#discussion_r500606665\",\r\n \"pull_request_url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48\",\r\n \"author_association\": \"MEMBER\",\r\n \"_links\": {\r\n \"self\": {\r\n \"href\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/comments/500606665\"\r\n },\r\n \"html\": {\r\n \"href\": \"https://github.com/dogsheep/github-to-sqlite/pull/48#discussion_r500606665\"\r\n },\r\n \"pull_request\": {\r\n \"href\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48\"\r\n }\r\n },\r\n \"original_commit_id\": \"4f33b850bd37829262dd29e1c520afffebedc19c\"\r\n }\r\n]\r\n```\r\nThat's a lot more interesting.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 664485022, "label": "Feature: pull request reviews and comments"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/46#issuecomment-735482546", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46", "id": 735482546, "node_id": "MDEyOklzc3VlQ29tbWVudDczNTQ4MjU0Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-11-30T00:22:02Z", "updated_at": "2020-11-30T00:22:02Z", "author_association": "MEMBER", "body": "As for reviews... here's the output of `github-to-sqlite get https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48/reviews --accept 'application/vnd.github.v3+json'`\r\n\r\n```json\r\n[\r\n {\r\n \"id\": 503368921,\r\n \"node_id\": \"MDE3OlB1bGxSZXF1ZXN0UmV2aWV3NTAzMzY4OTIx\",\r\n \"user\": {\r\n \"login\": \"simonw\",\r\n \"id\": 9599,\r\n \"node_id\": \"MDQ6VXNlcjk1OTk=\",\r\n \"avatar_url\": \"https://avatars0.githubusercontent.com/u/9599?u=5968723deb1a55b82620e106f5ca58e9b11a0942&v=4\",\r\n \"gravatar_id\": \"\",\r\n \"url\": \"https://api.github.com/users/simonw\",\r\n \"html_url\": \"https://github.com/simonw\",\r\n \"followers_url\": \"https://api.github.com/users/simonw/followers\",\r\n \"following_url\": \"https://api.github.com/users/simonw/following{/other_user}\",\r\n \"gists_url\": \"https://api.github.com/users/simonw/gists{/gist_id}\",\r\n \"starred_url\": \"https://api.github.com/users/simonw/starred{/owner}{/repo}\",\r\n \"subscriptions_url\": \"https://api.github.com/users/simonw/subscriptions\",\r\n \"organizations_url\": \"https://api.github.com/users/simonw/orgs\",\r\n \"repos_url\": \"https://api.github.com/users/simonw/repos\",\r\n \"events_url\": \"https://api.github.com/users/simonw/events{/privacy}\",\r\n \"received_events_url\": \"https://api.github.com/users/simonw/received_events\",\r\n \"type\": \"User\",\r\n \"site_admin\": false\r\n },\r\n \"body\": \"\",\r\n \"state\": \"CHANGES_REQUESTED\",\r\n \"html_url\": \"https://github.com/dogsheep/github-to-sqlite/pull/48#pullrequestreview-503368921\",\r\n \"pull_request_url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48\",\r\n \"author_association\": \"MEMBER\",\r\n \"_links\": {\r\n \"html\": {\r\n \"href\": \"https://github.com/dogsheep/github-to-sqlite/pull/48#pullrequestreview-503368921\"\r\n },\r\n \"pull_request\": {\r\n \"href\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48\"\r\n }\r\n },\r\n \"submitted_at\": \"2020-10-06T21:28:40Z\",\r\n \"commit_id\": \"4f33b850bd37829262dd29e1c520afffebedc19c\"\r\n },\r\n {\r\n \"id\": 513118561,\r\n \"node_id\": \"MDE3OlB1bGxSZXF1ZXN0UmV2aWV3NTEzMTE4NTYx\",\r\n \"user\": {\r\n \"login\": \"adamjonas\",\r\n \"id\": 755825,\r\n \"node_id\": \"MDQ6VXNlcjc1NTgyNQ==\",\r\n \"avatar_url\": \"https://avatars1.githubusercontent.com/u/755825?v=4\",\r\n \"gravatar_id\": \"\",\r\n \"url\": \"https://api.github.com/users/adamjonas\",\r\n \"html_url\": \"https://github.com/adamjonas\",\r\n \"followers_url\": \"https://api.github.com/users/adamjonas/followers\",\r\n \"following_url\": \"https://api.github.com/users/adamjonas/following{/other_user}\",\r\n \"gists_url\": \"https://api.github.com/users/adamjonas/gists{/gist_id}\",\r\n \"starred_url\": \"https://api.github.com/users/adamjonas/starred{/owner}{/repo}\",\r\n \"subscriptions_url\": \"https://api.github.com/users/adamjonas/subscriptions\",\r\n \"organizations_url\": \"https://api.github.com/users/adamjonas/orgs\",\r\n \"repos_url\": \"https://api.github.com/users/adamjonas/repos\",\r\n \"events_url\": \"https://api.github.com/users/adamjonas/events{/privacy}\",\r\n \"received_events_url\": \"https://api.github.com/users/adamjonas/received_events\",\r\n \"type\": \"User\",\r\n \"site_admin\": false\r\n },\r\n \"body\": \"\",\r\n \"state\": \"COMMENTED\",\r\n \"html_url\": \"https://github.com/dogsheep/github-to-sqlite/pull/48#pullrequestreview-513118561\",\r\n \"pull_request_url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48\",\r\n \"author_association\": \"CONTRIBUTOR\",\r\n \"_links\": {\r\n \"html\": {\r\n \"href\": \"https://github.com/dogsheep/github-to-sqlite/pull/48#pullrequestreview-513118561\"\r\n },\r\n \"pull_request\": {\r\n \"href\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48\"\r\n }\r\n },\r\n \"submitted_at\": \"2020-10-20T20:45:05Z\",\r\n \"commit_id\": \"4f33b850bd37829262dd29e1c520afffebedc19c\"\r\n },\r\n {\r\n \"id\": 513127529,\r\n \"node_id\": \"MDE3OlB1bGxSZXF1ZXN0UmV2aWV3NTEzMTI3NTI5\",\r\n \"user\": {\r\n \"login\": \"adamjonas\",\r\n \"id\": 755825,\r\n \"node_id\": \"MDQ6VXNlcjc1NTgyNQ==\",\r\n \"avatar_url\": \"https://avatars1.githubusercontent.com/u/755825?v=4\",\r\n \"gravatar_id\": \"\",\r\n \"url\": \"https://api.github.com/users/adamjonas\",\r\n \"html_url\": \"https://github.com/adamjonas\",\r\n \"followers_url\": \"https://api.github.com/users/adamjonas/followers\",\r\n \"following_url\": \"https://api.github.com/users/adamjonas/following{/other_user}\",\r\n \"gists_url\": \"https://api.github.com/users/adamjonas/gists{/gist_id}\",\r\n \"starred_url\": \"https://api.github.com/users/adamjonas/starred{/owner}{/repo}\",\r\n \"subscriptions_url\": \"https://api.github.com/users/adamjonas/subscriptions\",\r\n \"organizations_url\": \"https://api.github.com/users/adamjonas/orgs\",\r\n \"repos_url\": \"https://api.github.com/users/adamjonas/repos\",\r\n \"events_url\": \"https://api.github.com/users/adamjonas/events{/privacy}\",\r\n \"received_events_url\": \"https://api.github.com/users/adamjonas/received_events\",\r\n \"type\": \"User\",\r\n \"site_admin\": false\r\n },\r\n \"body\": \"\",\r\n \"state\": \"COMMENTED\",\r\n \"html_url\": \"https://github.com/dogsheep/github-to-sqlite/pull/48#pullrequestreview-513127529\",\r\n \"pull_request_url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48\",\r\n \"author_association\": \"CONTRIBUTOR\",\r\n \"_links\": {\r\n \"html\": {\r\n \"href\": \"https://github.com/dogsheep/github-to-sqlite/pull/48#pullrequestreview-513127529\"\r\n },\r\n \"pull_request\": {\r\n \"href\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48\"\r\n }\r\n },\r\n \"submitted_at\": \"2020-10-20T20:57:33Z\",\r\n \"commit_id\": \"3a0d5c498f9faae4e40aab204cd01b965a4f61f3\"\r\n }\r\n]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 664485022, "label": "Feature: pull request reviews and comments"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/46#issuecomment-735482187", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46", "id": 735482187, "node_id": "MDEyOklzc3VlQ29tbWVudDczNTQ4MjE4Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-11-30T00:20:11Z", "updated_at": "2020-11-30T00:20:11Z", "author_association": "MEMBER", "body": "Pull request are now added, thanks to @adamjonas.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 664485022, "label": "Feature: pull request reviews and comments"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/54#issuecomment-735465708", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/54", "id": 735465708, "node_id": "MDEyOklzc3VlQ29tbWVudDczNTQ2NTcwOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-11-29T22:08:46Z", "updated_at": "2020-11-29T22:08:46Z", "author_association": "MEMBER", "body": "Demo:\r\n- https://github-to-sqlite.dogsheep.net/github/steps?_facet=repo\r\n- https://github-to-sqlite.dogsheep.net/github/workflows\r\n- https://github-to-sqlite.dogsheep.net/github/jobs", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 753026003, "label": "github-to-sqlite workflows command"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/54#issuecomment-735464493", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/54", "id": 735464493, "node_id": "MDEyOklzc3VlQ29tbWVudDczNTQ2NDQ5Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-11-29T21:57:32Z", "updated_at": "2020-11-29T21:57:32Z", "author_association": "MEMBER", "body": "`$ github-to-sqlite workflows github.db simonw/datasette dogsheep/github-to-sqlite`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 753026003, "label": "github-to-sqlite workflows command"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/54#issuecomment-735464438", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/54", "id": 735464438, "node_id": "MDEyOklzc3VlQ29tbWVudDczNTQ2NDQzOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-11-29T21:57:08Z", "updated_at": "2020-11-29T21:57:08Z", "author_association": "MEMBER", "body": "Inspired by this tweet from Michael Heap https://twitter.com/mheap/status/1333108608817631238", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 753026003, "label": "github-to-sqlite workflows command"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/swarm-to-sqlite/issues/11#issuecomment-727692413", "issue_url": "https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/11", "id": 727692413, "node_id": "MDEyOklzc3VlQ29tbWVudDcyNzY5MjQxMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-11-16T02:15:22Z", "updated_at": "2020-11-16T02:15:22Z", "author_association": "MEMBER", "body": "Thanks, I'll look into this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 743400216, "label": "Error thrown: sqlite3.OperationalError: table users has no column named lastName"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/29#issuecomment-712266834", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/29", "id": 712266834, "node_id": "MDEyOklzc3VlQ29tbWVudDcxMjI2NjgzNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-19T16:01:23Z", "updated_at": "2020-10-19T16:01:23Z", "author_association": "MEMBER", "body": "Might just be a documented pattern for how to configure this in YAML templates.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 724759588, "label": "Add search highlighting snippets"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-711569063", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50", "id": 711569063, "node_id": "MDEyOklzc3VlQ29tbWVudDcxMTU2OTA2Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-19T05:01:29Z", "updated_at": "2020-10-19T05:01:29Z", "author_association": "MEMBER", "body": "Demo of `--accept`:\r\n\r\n github-to-sqlite get /repos/simonw/datasette/readme --accept 'application/vnd.github.VERSION.html'\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 703218756, "label": "Commands for making authenticated API calls"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/28#issuecomment-711089647", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/28", "id": 711089647, "node_id": "MDEyOklzc3VlQ29tbWVudDcxMTA4OTY0Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-17T22:43:13Z", "updated_at": "2020-10-17T22:43:13Z", "author_association": "MEMBER", "body": "Since my personal Dogsheep uses Datasette authentication, I'm going to need to pass through cookies. https://github.com/simonw/datasette/issues/1020 will solve that in the future but for now I need to solve it explicitly.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 723861683, "label": "Switch to using datasette.client"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/healthkit-to-sqlite/issues/11#issuecomment-711081703", "issue_url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11", "id": 711081703, "node_id": "MDEyOklzc3VlQ29tbWVudDcxMTA4MTcwMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-17T21:18:35Z", "updated_at": "2020-10-17T21:18:35Z", "author_association": "MEMBER", "body": "OK, if you upgrade to the just-released 1.0 this should work (it worked against my Spanish export).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 723838331, "label": "export.xml file name varies with different language settings"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/healthkit-to-sqlite/issues/11#issuecomment-711079760", "issue_url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11", "id": 711079760, "node_id": "MDEyOklzc3VlQ29tbWVudDcxMTA3OTc2MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-17T21:00:05Z", "updated_at": "2020-10-17T21:00:05Z", "author_association": "MEMBER", "body": "Checking for either `\r\n\"\r\nTEMPLATE = \"\"\"\r\n
{}
\r\n\"\"\".strip()\r\n\r\nEN_MEDIA_SCRIPT = \"\"\"\r\nArray.from(document.querySelectorAll('en-media')).forEach(el => {\r\n let hash = el.getAttribute('hash');\r\n let type = el.getAttribute('type');\r\n let path = `/evernote/resources_data/${hash}.json?_shape=array`;\r\n fetch(path).then(r => r.json()).then(rows => {\r\n let b64 = rows[0].data.encoded;\r\n let data = `data:${type};base64,${b64}`;\r\n el.innerHTML = ``;\r\n });\r\n});\r\n\"\"\"\r\n\r\n\r\n@hookimpl\r\ndef render_cell(value, table):\r\n if not table:\r\n # Don't render content from arbitrary SQL queries, could be XSS hole\r\n return\r\n if not value or not isinstance(value, str):\r\n return\r\n value = value.strip()\r\n if value.startswith(START) and value.endswith(END):\r\n trimmed = value[len(START) : -len(END)]\r\n trimmed = trimmed.split(\">\", 1)[1]\r\n # Replace those horrible double newlines\r\n trimmed = trimmed.replace(\"

\", \"
\")\r\n return jinja2.Markup(TEMPLATE.format(trimmed))\r\n\r\n\r\n@hookimpl\r\ndef extra_body_script():\r\n return EN_MEDIA_SCRIPT\r\n```\r\nIt works!\r\n\r\nIt does however demonstrate that Evernote's \"clip this webpage\" feature means there is a LOT of weird HTML that can get into a note. It looks like they've filtered out the scripts but I wouldn't bet on it - they certainly don't filter out many of the inline styles. So running Bleach is almost certainly a good idea.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 718938889, "label": "Figure out how to display images from tags inline in Datasette"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/swarm-to-sqlite/issues/8#issuecomment-707332912", "issue_url": "https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/8", "id": 707332912, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNzMzMjkxMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-12T20:35:06Z", "updated_at": "2020-10-12T20:35:06Z", "author_association": "MEMBER", "body": "Shipped a fix for this in [swarm-to-sqlite 0.3.2](https://github.com/dogsheep/swarm-to-sqlite/releases/tag/0.3.2).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 648245071, "label": "Error thrown: table photos has no column named hasSticker"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/4#issuecomment-706786548", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/4", "id": 706786548, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNjc4NjU0OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-11T23:39:46Z", "updated_at": "2020-10-11T23:39:46Z", "author_association": "MEMBER", "body": "Should have used porter stemming for this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 718938508, "label": "Configure FTS + add an index on the date columns"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/6#issuecomment-706785201", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/6", "id": 706785201, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNjc4NTIwMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-11T23:29:39Z", "updated_at": "2020-10-11T23:29:39Z", "author_association": "MEMBER", "body": "It looks to me like each of those `` blocks has a number of guesses in order of confidence:\r\n```xml\r\n \r\n wonders,\r\n wanders,\r\n wonders ?\r\n wonders\r\n wonders.\r\n \r\n```\r\nSo maybe the best approach here is to just take the first `t` element within each `item`.\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 718949182, "label": "Better handling of OCR data"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/6#issuecomment-706785086", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/6", "id": 706785086, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNjc4NTA4Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-11T23:28:50Z", "updated_at": "2020-10-11T23:28:50Z", "author_association": "MEMBER", "body": "The XML for the OCR stuff is a bit weird. Currently I'm doing this to it:\r\n\r\nhttps://github.com/dogsheep/evernote-to-sqlite/blob/c33d7b043a45eb3e88676e5fa3ce31755199d9f8/evernote_to_sqlite/utils.py#L70-L78\r\n\r\nThis can produce some odd results, for example:\r\n\r\n> Sure 'Sure, 'Sure. Sure, Sure. sure sure. sure ? If you If Yau [you live jive In m 1n an area devoid of natural wonders, wanders, wonders ? wonders wonders. your mind will be blown, blown' blown. blown ? -e i ? ,1 IL it ? at ? KY ? fl ft bat at\r\n\r\nWhich came from this image:\r\n\r\n![image](https://user-images.githubusercontent.com/9599/95692952-5dd7c880-0bde-11eb-939a-d10b800a4105.png)\r\n\r\nThe XML for that is:\r\n\r\n```xml\r\n\r\n \r\n Sure\r\n 'Sure,\r\n 'Sure.\r\n Sure,\r\n Sure.\r\n \r\n \r\n sure\r\n sure.\r\n sure ?\r\n \r\n \r\n If you\r\n If Yau\r\n [you\r\n \r\n \r\n live\r\n jive\r\n \r\n \r\n In\r\n m\r\n 1n\r\n \r\n \r\n an\r\n \r\n \r\n area\r\n \r\n \r\n devoid\r\n \r\n \r\n of\r\n \r\n \r\n natural\r\n \r\n \r\n wonders,\r\n wanders,\r\n wonders ?\r\n wonders\r\n wonders.\r\n \r\n \r\n your\r\n \r\n \r\n mind\r\n \r\n \r\n will\r\n \r\n \r\n be\r\n \r\n \r\n blown,\r\n blown'\r\n blown.\r\n blown ?\r\n \r\n \r\n -e\r\n \r\n \r\n i ?\r\n \r\n \r\n ,1\r\n \r\n \r\n IL\r\n \r\n \r\n it ?\r\n at ?\r\n KY ?\r\n \r\n \r\n fl\r\n ft\r\n bat\r\n at\r\n \r\n\r\n```\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 718949182, "label": "Better handling of OCR data"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/4#issuecomment-706784028", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/4", "id": 706784028, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNjc4NDAyOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-11T23:20:32Z", "updated_at": "2020-10-11T23:20:32Z", "author_association": "MEMBER", "body": "I haven't done the FTS on OCR yet. I'm going to move that to another ticket because it requires more thought.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 718938508, "label": "Configure FTS + add an index on the date columns"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706776808", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5", "id": 706776808, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNjc3NjgwOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-11T22:23:14Z", "updated_at": "2020-10-11T22:23:14Z", "author_association": "MEMBER", "body": "... but it's still important to be able to get to the rendered note directly from the browse notes `/evernote/notes` page. Maybe use a simple `render_cell()` hook that just knows how to generate the link to the rendered note page?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 718938889, "label": "Figure out how to display images from tags inline in Datasette"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706776680", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5", "id": 706776680, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNjc3NjY4MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-11T22:22:16Z", "updated_at": "2020-10-11T22:22:16Z", "author_association": "MEMBER", "body": "Maybe the best way do this is with a custom route, `/-/evernote/note-id` - that way I can clean the HTML and resolve the other things in the `` structure without using `render_cell()` and the like. My concern about using `render_cell()` is that it could lead to weird security problems when combined with `?sql=` queries.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 718938889, "label": "Figure out how to display images from tags inline in Datasette"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706776447", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5", "id": 706776447, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNjc3NjQ0Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-11T22:20:32Z", "updated_at": "2020-10-11T22:20:32Z", "author_association": "MEMBER", "body": "Or... I could do this client-side. JavaScript that looks for `` tags and fetches the data using `fetch()` wouldn't be too hard to write.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 718938889, "label": "Figure out how to display images from tags inline in Datasette"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706776242", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5", "id": 706776242, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNjc3NjI0Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-11T22:18:30Z", "updated_at": "2020-10-11T22:19:48Z", "author_association": "MEMBER", "body": "Alternatively, rather than relying on `datasette-media` this could base64-embed the images. `evernote-to-sqlite` could register itself as a Datasette plugin that knows how to do this.\r\n\r\nMaybe rename the column to `evernote_content` and register a render cell hook that knows how to rewrite those note bodies so that they are visible?\r\n\r\nMight need to feed them through Bleach too, just in case any nasty code can get into them.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 718938889, "label": "Figure out how to display images from tags inline in Datasette"}, "performed_via_github_app": null}