issue_comments
525 rows where author_association = "MEMBER" sorted by updated_at descending
This data as json, CSV (advanced)
user 1
- simonw 497
id | html_url | issue_url | node_id | user | created_at | updated_at ▲ | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
1221623052 | https://github.com/dogsheep/pocket-to-sqlite/issues/10#issuecomment-1221623052 | https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/10 | IC_kwDODLZ_YM5I0H0M | simonw 9599 | 2022-08-21T21:20:33Z | 2022-08-21T21:20:33Z | MEMBER | That was clearly the intention from the description of this issue: - #4 |
{ "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
When running `auth` command, don't overwrite an existing auth.json file 1246826792 | |
1221622873 | https://github.com/dogsheep/pocket-to-sqlite/issues/10#issuecomment-1221622873 | https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/10 | IC_kwDODLZ_YM5I0HxZ | simonw 9599 | 2022-08-21T21:19:25Z | 2022-08-21T21:19:25Z | MEMBER | Agreed, that would be a much better implementation. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
When running `auth` command, don't overwrite an existing auth.json file 1246826792 | |
1221621529 | https://github.com/dogsheep/pocket-to-sqlite/issues/11#issuecomment-1221621529 | https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/11 | IC_kwDODLZ_YM5I0HcZ | simonw 9599 | 2022-08-21T21:10:15Z | 2022-08-21T21:11:26Z | MEMBER | Just saw that's what's implemented here already! - #7 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
-a option is used for "--auth" and for "--all" 1345452427 | |
1221621700 | https://github.com/dogsheep/pocket-to-sqlite/pull/7#issuecomment-1221621700 | https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/7 | IC_kwDODLZ_YM5I0HfE | simonw 9599 | 2022-08-21T21:11:12Z | 2022-08-21T21:11:12Z | MEMBER | I thought this might need a documentation update but |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Fixed conflicting CLI flags 750141615 | |
1221621466 | https://github.com/dogsheep/pocket-to-sqlite/issues/11#issuecomment-1221621466 | https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/11 | IC_kwDODLZ_YM5I0Hba | simonw 9599 | 2022-08-21T21:09:47Z | 2022-08-21T21:09:47Z | MEMBER | Great catch, thanks. I'm going to use it to mean
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
-a option is used for "--auth" and for "--all" 1345452427 | |
1188317682 | https://github.com/dogsheep/github-to-sqlite/issues/74#issuecomment-1188317682 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/74 | IC_kwDODFdgUs5G1Eny | simonw 9599 | 2022-07-18T21:14:22Z | 2022-07-18T21:14:22Z | MEMBER | That fixed it. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
500 error in github-to-sqlite demo 1308461063 | |
1188233729 | https://github.com/dogsheep/github-to-sqlite/issues/74#issuecomment-1188233729 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/74 | IC_kwDODFdgUs5G0wIB | simonw 9599 | 2022-07-18T19:51:02Z | 2022-07-18T19:51:02Z | MEMBER | Takes 30m to deploy the demo! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
500 error in github-to-sqlite demo 1308461063 | |
1188228964 | https://github.com/dogsheep/github-to-sqlite/issues/74#issuecomment-1188228964 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/74 | IC_kwDODFdgUs5G0u9k | simonw 9599 | 2022-07-18T19:45:30Z | 2022-07-18T19:47:35Z | MEMBER |
I'm going to drop the GFM rendering from the demo, and just treat it as regular markdown. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
500 error in github-to-sqlite demo 1308461063 | |
1188223933 | https://github.com/dogsheep/github-to-sqlite/issues/74#issuecomment-1188223933 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/74 | IC_kwDODFdgUs5G0tu9 | simonw 9599 | 2022-07-18T19:40:50Z | 2022-07-18T19:42:41Z | MEMBER | Here's how the demo is deployed: https://github.com/dogsheep/github-to-sqlite/blob/dbac2e5dd8a562b45d8255a265859cf8020ca22a/.github/workflows/deploy-demo.yml#L103-L119 I'm suspicious of |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
500 error in github-to-sqlite demo 1308461063 | |
1188225625 | https://github.com/dogsheep/github-to-sqlite/issues/74#issuecomment-1188225625 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/74 | IC_kwDODFdgUs5G0uJZ | simonw 9599 | 2022-07-18T19:41:52Z | 2022-07-18T19:41:52Z | MEMBER | https://github.com/Zopieux/py-gfm says that library is no longer maintained, and suggests https://github.com/Zopieux/pycmarkgfm as an alternative. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
500 error in github-to-sqlite demo 1308461063 | |
1188223299 | https://github.com/dogsheep/github-to-sqlite/pull/73#issuecomment-1188223299 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/73 | IC_kwDODFdgUs5G0tlD | simonw 9599 | 2022-07-18T19:40:06Z | 2022-07-18T19:40:06Z | MEMBER | Thanks! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Fixing 'NoneType' object has no attribute 'items' 1261884917 | |
1105474232 | https://github.com/dogsheep/github-to-sqlite/issues/72#issuecomment-1105474232 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/72 | IC_kwDODFdgUs5B5DK4 | simonw 9599 | 2022-04-21T17:02:15Z | 2022-04-21T17:02:15Z | MEMBER | That's interesting - yeah it looks like the number of pages can be derived from the https://docs.github.com/en/rest/guides/traversing-with-pagination |
{ "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
feature: display progress bar when downloading multi-page responses 1211283427 | |
985928838 | https://github.com/dogsheep/github-to-sqlite/issues/69#issuecomment-985928838 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/69 | IC_kwDODFdgUs46xBSG | simonw 9599 | 2021-12-04T00:34:52Z | 2021-12-04T00:34:52Z | MEMBER | First attempt at this:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
View that combines issues and issue comments 1071071397 | |
924209583 | https://github.com/dogsheep/twitter-to-sqlite/pull/59#issuecomment-924209583 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/59 | IC_kwDODEm0Qs43FlGv | simonw 9599 | 2021-09-21T17:37:34Z | 2021-09-21T17:37:34Z | MEMBER | Thanks for this! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Fix for since_id bug, closes #58 984942782 | |
906646452 | https://github.com/dogsheep/evernote-to-sqlite/issues/13#issuecomment-906646452 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13 | IC_kwDOEhK-wc42ClO0 | simonw 9599 | 2021-08-26T18:34:34Z | 2021-08-26T18:35:20Z | MEMBER | I tried this ampersand fix: https://regex101.com/r/ojU2H9/1 ```python https://regex101.com/r/ojU2H9/1_invalid_ampersand_re = re.compile(r'&(?![a-z0-9]+;)') def fix_bad_xml(xml): # More fixes for things like '&' not as part of an entity return _invalid_ampersand_re.sub('&', xml) ``` Even with that I'm still getting total garbage in the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
xml.etree.ElementTree.ParseError: not well-formed (invalid token) 978743426 | |
906635938 | https://github.com/dogsheep/evernote-to-sqlite/issues/13#issuecomment-906635938 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13 | IC_kwDOEhK-wc42Ciqi | simonw 9599 | 2021-08-26T18:18:27Z | 2021-08-26T18:18:27Z | MEMBER | It looks like I was using the round-trip to dump the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
xml.etree.ElementTree.ParseError: not well-formed (invalid token) 978743426 | |
905206234 | https://github.com/dogsheep/evernote-to-sqlite/issues/13#issuecomment-905206234 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13 | IC_kwDOEhK-wc419Fna | simonw 9599 | 2021-08-25T05:58:42Z | 2021-08-25T05:58:42Z | MEMBER | Not sure why I was round-tripping the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
xml.etree.ElementTree.ParseError: not well-formed (invalid token) 978743426 | |
905203570 | https://github.com/dogsheep/evernote-to-sqlite/issues/13#issuecomment-905203570 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13 | IC_kwDOEhK-wc419E9y | simonw 9599 | 2021-08-25T05:51:22Z | 2021-08-25T05:53:27Z | MEMBER | The debugger showed me that it broke on a string that looked like this: ```xml <en-note>
Q3 2018 Reflection & Development... ``` Yeah that is not valid XML! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
xml.etree.ElementTree.ParseError: not well-formed (invalid token) 978743426 | |
902356871 | https://github.com/dogsheep/healthkit-to-sqlite/issues/20#issuecomment-902356871 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/20 | IC_kwDOC8tyDs41yN-H | simonw 9599 | 2021-08-20T01:12:48Z | 2021-08-20T01:12:48Z | MEMBER | Also on |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add index on workout_points.date 975166271 | |
902355471 | https://github.com/dogsheep/healthkit-to-sqlite/issues/20#issuecomment-902355471 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/20 | IC_kwDOC8tyDs41yNoP | simonw 9599 | 2021-08-20T01:09:07Z | 2021-08-20T01:09:07Z | MEMBER | Workaround:
See https://sqlite-utils.datasette.io/en/stable/cli.html#creating-indexes |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add index on workout_points.date 975166271 | |
902330301 | https://github.com/dogsheep/twitter-to-sqlite/pull/49#issuecomment-902330301 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/49 | IC_kwDODEm0Qs41yHe9 | simonw 9599 | 2021-08-20T00:01:56Z | 2021-08-20T00:01:56Z | MEMBER | Thanks! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Document the use of --stop_after with favorites, refs #20 681575714 | |
902329884 | https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902329884 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57 | IC_kwDODEm0Qs41yHYc | simonw 9599 | 2021-08-20T00:01:05Z | 2021-08-20T00:01:05Z | MEMBER | Maybe Click changed something which meant that this broke things when it didn't used to? |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Error: Use either --since or --since_id, not both 907645813 | |
902329455 | https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902329455 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57 | IC_kwDODEm0Qs41yHRv | simonw 9599 | 2021-08-19T23:59:56Z | 2021-08-19T23:59:56Z | MEMBER | This looks like the bug to me:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Error: Use either --since or --since_id, not both 907645813 | |
902328760 | https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902328760 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57 | IC_kwDODEm0Qs41yHG4 | simonw 9599 | 2021-08-19T23:57:41Z | 2021-08-19T23:57:41Z | MEMBER | Weird, added debug code and got this: |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Error: Use either --since or --since_id, not both 907645813 | |
902328369 | https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902328369 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57 | IC_kwDODEm0Qs41yHAx | simonw 9599 | 2021-08-19T23:56:26Z | 2021-08-19T23:56:26Z | MEMBER | https://developer.twitter.com/en/docs/twitter-api/v1/tweets/timelines/api-reference/get-statuses-user_timeline says the API has been replaced by the new v2 one, but it should still work - and the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Error: Use either --since or --since_id, not both 907645813 | |
902327457 | https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902327457 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57 | IC_kwDODEm0Qs41yGyh | simonw 9599 | 2021-08-19T23:53:25Z | 2021-08-19T23:53:25Z | MEMBER | I'm getting this too. Looking into it now. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Error: Use either --since or --since_id, not both 907645813 | |
886241674 | https://github.com/dogsheep/hacker-news-to-sqlite/issues/3#issuecomment-886241674 | https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/3 | IC_kwDODtX3eM400vmK | simonw 9599 | 2021-07-25T18:41:17Z | 2021-07-25T18:41:17Z | MEMBER | Got a TIL out of this: https://til.simonwillison.net/jq/extracting-objects-recursively |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Use HN algolia endpoint to retrieve trees 952189173 | |
886237834 | https://github.com/dogsheep/hacker-news-to-sqlite/issues/3#issuecomment-886237834 | https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/3 | IC_kwDODtX3eM400uqK | simonw 9599 | 2021-07-25T18:05:32Z | 2021-07-25T18:05:32Z | MEMBER | If you hit the endpoint for a comment that's part of a thread you get that comment and its recursive children: https://hn.algolia.com/api/v1/items/27941552 You can tell that it's not the top-level because the
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Use HN algolia endpoint to retrieve trees 952189173 | |
886142671 | https://github.com/dogsheep/hacker-news-to-sqlite/issues/3#issuecomment-886142671 | https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/3 | IC_kwDODtX3eM400XbP | simonw 9599 | 2021-07-25T03:51:05Z | 2021-07-25T03:51:05Z | MEMBER | Prototype:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Use HN algolia endpoint to retrieve trees 952189173 | |
886140431 | https://github.com/dogsheep/hacker-news-to-sqlite/issues/2#issuecomment-886140431 | https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/2 | IC_kwDODtX3eM400W4P | simonw 9599 | 2021-07-25T03:12:57Z | 2021-07-25T03:12:57Z | MEMBER | I'm going to build a general-purpose |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Command for fetching Hacker News threads from the search API 952179830 | |
886136224 | https://github.com/dogsheep/hacker-news-to-sqlite/issues/2#issuecomment-886136224 | https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/2 | IC_kwDODtX3eM400V2g | simonw 9599 | 2021-07-25T02:08:29Z | 2021-07-25T02:08:29Z | MEMBER | Prototype:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Command for fetching Hacker News threads from the search API 952179830 | |
886135922 | https://github.com/dogsheep/hacker-news-to-sqlite/issues/2#issuecomment-886135922 | https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/2 | IC_kwDODtX3eM400Vxy | simonw 9599 | 2021-07-25T02:06:20Z | 2021-07-25T02:06:20Z | MEMBER | https://hn.algolia.com/api/v1/search_by_date?query=simonwillison.net&restrictSearchableAttributes=url looks like it does what I want. https://hn.algolia.com/api/v1/search_by_date?query=simonwillison.net&restrictSearchableAttributes=url&hitsPerPage=1000 - returns 1000 at once. Otherwise you have to paginate using https://www.algolia.com/doc/api-reference/api-parameters/hitsPerPage/ says 1000 is the maximum. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Command for fetching Hacker News threads from the search API 952179830 | |
886135562 | https://github.com/dogsheep/hacker-news-to-sqlite/issues/2#issuecomment-886135562 | https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/2 | IC_kwDODtX3eM400VsK | simonw 9599 | 2021-07-25T02:01:11Z | 2021-07-25T02:01:11Z | MEMBER | That page doesn't have an API but does look easy to scrape. The other option here is the HN Search API powered by Algolia, documented at https://hn.algolia.com/api |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Command for fetching Hacker News threads from the search API 952179830 | |
879477586 | https://github.com/dogsheep/healthkit-to-sqlite/issues/12#issuecomment-879477586 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/12 | MDEyOklzc3VlQ29tbWVudDg3OTQ3NzU4Ng== | simonw 9599 | 2021-07-13T23:50:06Z | 2021-07-13T23:50:06Z | MEMBER | Unfortunately I don't think updating the database is practical, because the export doesn't include unique identifiers which can be used to update existing records and create new ones. Recreating from scratch works around that limitation. I've not explored workouts with SpatiaLite but that's a really good idea. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Some workout columns should be float, not text 727848625 | |
861042050 | https://github.com/dogsheep/github-to-sqlite/issues/64#issuecomment-861042050 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64 | MDEyOklzc3VlQ29tbWVudDg2MTA0MjA1MA== | simonw 9599 | 2021-06-14T22:45:42Z | 2021-06-14T22:45:42Z | MEMBER | I'm definitely interested in supporting events in this tool - see #14. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
feature: support "events" 920636216 | |
861041597 | https://github.com/dogsheep/github-to-sqlite/issues/64#issuecomment-861041597 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64 | MDEyOklzc3VlQ29tbWVudDg2MTA0MTU5Nw== | simonw 9599 | 2021-06-14T22:44:54Z | 2021-06-14T22:44:54Z | MEMBER | Have you found a way to access events in GraphQL? I can only see way to access a timeline of events for a single issue or a single pull request. See also https://github.community/t/get-event-equivalent-for-v4/13600/2 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
feature: support "events" 920636216 | |
844250232 | https://github.com/dogsheep/github-to-sqlite/pull/59#issuecomment-844250232 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/59 | MDEyOklzc3VlQ29tbWVudDg0NDI1MDIzMg== | simonw 9599 | 2021-05-19T16:08:10Z | 2021-05-19T16:08:10Z | MEMBER | Thanks for catching this. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Remove unneeded exists=True for -a/--auth flag. 771872303 | |
844249385 | https://github.com/dogsheep/github-to-sqlite/pull/61#issuecomment-844249385 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/61 | MDEyOklzc3VlQ29tbWVudDg0NDI0OTM4NQ== | simonw 9599 | 2021-05-19T16:07:06Z | 2021-05-19T16:07:06Z | MEMBER | Thanks! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
fixing typo in get cli help text 797108702 | |
790695126 | https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790695126 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDc5MDY5NTEyNg== | simonw 9599 | 2021-03-04T15:20:42Z | 2021-03-04T15:20:42Z | MEMBER | I'm not sure why but my most recent import, when displayed in Datasette, looks like this: Sorting by |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
WIP: Add Gmail takeout mbox import 813880401 | |
790693674 | https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790693674 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDc5MDY5MzY3NA== | simonw 9599 | 2021-03-04T15:18:36Z | 2021-03-04T15:18:36Z | MEMBER | I imported my 10GB mbox with 750,000 emails in it, ran this tool (with a hacked fix for the blob column problem) - and now a search that returns 92 results takes 25.37ms! This is fantastic. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
WIP: Add Gmail takeout mbox import 813880401 | |
790669767 | https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790669767 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDc5MDY2OTc2Nw== | simonw 9599 | 2021-03-04T14:46:06Z | 2021-03-04T14:46:06Z | MEMBER | Solution could be to pre-process that string by splitting on |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
WIP: Add Gmail takeout mbox import 813880401 | |
790668263 | https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790668263 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDc5MDY2ODI2Mw== | simonw 9599 | 2021-03-04T14:43:58Z | 2021-03-04T14:43:58Z | MEMBER | I added this code to output a message ID on errors:
This was for the following error:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
WIP: Add Gmail takeout mbox import 813880401 | |
790384087 | https://github.com/dogsheep/google-takeout-to-sqlite/issues/6#issuecomment-790384087 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/6 | MDEyOklzc3VlQ29tbWVudDc5MDM4NDA4Nw== | simonw 9599 | 2021-03-04T07:22:51Z | 2021-03-04T07:22:51Z | MEMBER | 3 also mentions the conflicting version with other tools. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Upgrade to latest sqlite-utils 821841046 | |
790380839 | https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790380839 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDc5MDM4MDgzOQ== | simonw 9599 | 2021-03-04T07:17:05Z | 2021-03-04T07:17:05Z | MEMBER | Looks like you're doing this:
I imagine the reason the column is a |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
WIP: Add Gmail takeout mbox import 813880401 | |
790379629 | https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790379629 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDc5MDM3OTYyOQ== | simonw 9599 | 2021-03-04T07:14:41Z | 2021-03-04T07:14:41Z | MEMBER | Confirmed: removing the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
WIP: Add Gmail takeout mbox import 813880401 | |
790378658 | https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790378658 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDc5MDM3ODY1OA== | simonw 9599 | 2021-03-04T07:12:48Z | 2021-03-04T07:12:48Z | MEMBER | It looks like the If I It would be great if we could store the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
WIP: Add Gmail takeout mbox import 813880401 | |
790373024 | https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790373024 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDc5MDM3MzAyNA== | simonw 9599 | 2021-03-04T07:01:58Z | 2021-03-04T07:04:06Z | MEMBER | I got 9 warnings that look like this:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
WIP: Add Gmail takeout mbox import 813880401 | |
790372621 | https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790372621 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDc5MDM3MjYyMQ== | simonw 9599 | 2021-03-04T07:01:18Z | 2021-03-04T07:01:18Z | MEMBER | I'm not sure if it would work, but there is an alternative pattern for showing a progress bar against a really large file that I've used in https://github.com/dogsheep/healthkit-to-sqlite/blob/3eb2b06bfe3b4faaf10e9cf9dfcb28e3d16c14ff/healthkit_to_sqlite/cli.py#L24-L57 and https://github.com/dogsheep/healthkit-to-sqlite/blob/3eb2b06bfe3b4faaf10e9cf9dfcb28e3d16c14ff/healthkit_to_sqlite/utils.py#L4-L19 (the It can be a bit of a convoluted pattern, and I'm not at all sure it would work for |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
WIP: Add Gmail takeout mbox import 813880401 | |
790370485 | https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790370485 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDc5MDM3MDQ4NQ== | simonw 9599 | 2021-03-04T06:57:25Z | 2021-03-04T06:57:48Z | MEMBER | The command takes quite a while to start running, presumably because this line causes it to have to scan the WHOLE file in order to generate a count: I'm fine with waiting though. It's not like this is a command people run every day - and without that count we can't show a progress bar, which seems pretty important for a process that takes this long. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
WIP: Add Gmail takeout mbox import 813880401 | |
790369076 | https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790369076 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDc5MDM2OTA3Ng== | simonw 9599 | 2021-03-04T06:54:46Z | 2021-03-04T06:54:46Z | MEMBER | The Rich-powered progress bar is pretty: |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
WIP: Add Gmail takeout mbox import 813880401 | |
790312268 | https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790312268 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDc5MDMxMjI2OA== | simonw 9599 | 2021-03-04T05:48:16Z | 2021-03-04T05:48:16Z | MEMBER | Wow, my mbox is a 10.35 GB download! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
WIP: Add Gmail takeout mbox import 813880401 | |
786925280 | https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-786925280 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDc4NjkyNTI4MA== | simonw 9599 | 2021-02-26T22:23:10Z | 2021-02-26T22:23:10Z | MEMBER | Thanks! I requested my Gmail export from takeout - once that arrives I'll test it against this and then merge the PR. |
{ "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
WIP: Add Gmail takeout mbox import 813880401 | |
777839351 | https://github.com/dogsheep/evernote-to-sqlite/pull/10#issuecomment-777839351 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/10 | MDEyOklzc3VlQ29tbWVudDc3NzgzOTM1MQ== | simonw 9599 | 2021-02-11T22:37:55Z | 2021-02-11T22:37:55Z | MEMBER | I've merged these changes by hand now, thanks! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
BugFix for encoding and not update info. 770712149 | |
777827396 | https://github.com/dogsheep/evernote-to-sqlite/issues/7#issuecomment-777827396 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/7 | MDEyOklzc3VlQ29tbWVudDc3NzgyNzM5Ng== | simonw 9599 | 2021-02-11T22:13:14Z | 2021-02-11T22:13:14Z | MEMBER | My best guess is that you have an older version of |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
evernote-to-sqlite on windows 10 give this error: TypeError: insert() got an unexpected keyword argument 'replace' 743297582 | |
777821383 | https://github.com/dogsheep/evernote-to-sqlite/issues/9#issuecomment-777821383 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/9 | MDEyOklzc3VlQ29tbWVudDc3NzgyMTM4Mw== | simonw 9599 | 2021-02-11T22:01:28Z | 2021-02-11T22:01:28Z | MEMBER | Aha! I think I've figured out what's going on here. The CData blocks containing the notes look like this:
The DTD at http://xml.evernote.com/pub/enml2.dtd includes some entities: ``` %HTMLlat1; %HTMLsymbol; %HTMLspecial;
```python {'Aacute': 'Á', 'aacute': 'á', 'Aacute;': 'Á', 'aacute;': 'á', 'Abreve;': 'Ă', 'abreve;': 'ă', 'ac;': '∾', 'acd;': '∿', ...} ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
ParseError: undefined entity š 748372469 | |
777798330 | https://github.com/dogsheep/evernote-to-sqlite/issues/11#issuecomment-777798330 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/11 | MDEyOklzc3VlQ29tbWVudDc3Nzc5ODMzMA== | simonw 9599 | 2021-02-11T21:18:58Z | 2021-02-11T21:18:58Z | MEMBER | Thanks for the fix! |
{ "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
XML parse error 792851444 | |
770071568 | https://github.com/dogsheep/github-to-sqlite/issues/60#issuecomment-770071568 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/60 | MDEyOklzc3VlQ29tbWVudDc3MDA3MTU2OA== | simonw 9599 | 2021-01-29T21:56:15Z | 2021-01-29T21:56:15Z | MEMBER | I really like the way you're using pipes here - really smart. It's similar to how I build the demo database in this GitHub Actions workflow:
It lets you do things like:
|
{ "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Use Data from SQLite in other commands 797097140 | |
769957751 | https://github.com/dogsheep/twitter-to-sqlite/issues/56#issuecomment-769957751 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/56 | MDEyOklzc3VlQ29tbWVudDc2OTk1Nzc1MQ== | simonw 9599 | 2021-01-29T17:59:40Z | 2021-01-29T17:59:40Z | MEMBER | This is interesting - how did you create that initial table? Was this using the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Not all quoted statuses get fetched? 796736607 | |
761967094 | https://github.com/dogsheep/swarm-to-sqlite/issues/11#issuecomment-761967094 | https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/11 | MDEyOklzc3VlQ29tbWVudDc2MTk2NzA5NA== | simonw 9599 | 2021-01-18T04:11:13Z | 2021-01-18T04:11:13Z | MEMBER | I just got a similar error: ``` File "/home/dogsheep/datasette-venv/lib/python3.8/site-packages/swarm_to_sqlite/utils.py", line 79, in save_checkin checkins_table.m2m("users", user, m2m_table="with", pk="id") File "/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py", line 2048, in m2m id = other_table.insert(record, pk=pk, replace=True).last_pk File "/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py", line 1781, in insert return self.insert_all( File "/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py", line 1899, in insert_all self.insert_chunk( File "/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py", line 1709, in insert_chunk result = self.db.execute(query, params) File "/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py", line 226, in execute return self.conn.execute(sql, parameters) pysqlite3.dbapi2.OperationalError: table users has no column named countryCode ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Error thrown: sqlite3.OperationalError: table users has no column named lastName 743400216 | |
748426877 | https://github.com/dogsheep/dogsheep-beta/issues/31#issuecomment-748426877 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/31 | MDEyOklzc3VlQ29tbWVudDc0ODQyNjg3Nw== | simonw 9599 | 2020-12-19T06:16:11Z | 2020-12-19T06:16:11Z | MEMBER | Here's why:
But the error being raised here is:
I'm going to attempt the escaped on on every error. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Searching for "github-to-sqlite" throws an error 771316301 | |
748426663 | https://github.com/dogsheep/dogsheep-beta/issues/31#issuecomment-748426663 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/31 | MDEyOklzc3VlQ29tbWVudDc0ODQyNjY2Mw== | simonw 9599 | 2020-12-19T06:14:06Z | 2020-12-19T06:14:06Z | MEMBER | Looks like I already do that here: https://github.com/dogsheep/dogsheep-beta/blob/9ba4401017ac24ffa3bc1db38e0910ea49de7616/dogsheep_beta/init.py#L141-L146 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Searching for "github-to-sqlite" throws an error 771316301 | |
748426581 | https://github.com/dogsheep/dogsheep-beta/issues/31#issuecomment-748426581 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/31 | MDEyOklzc3VlQ29tbWVudDc0ODQyNjU4MQ== | simonw 9599 | 2020-12-19T06:13:17Z | 2020-12-19T06:13:17Z | MEMBER | One fix for this could be to try running the raw query, but if it throws an error run it again with the query escaped. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Searching for "github-to-sqlite" throws an error 771316301 | |
748426501 | https://github.com/dogsheep/dogsheep-beta/issues/31#issuecomment-748426501 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/31 | MDEyOklzc3VlQ29tbWVudDc0ODQyNjUwMQ== | simonw 9599 | 2020-12-19T06:12:22Z | 2020-12-19T06:12:22Z | MEMBER | I deliberately added support for advanced FTS in https://github.com/dogsheep/dogsheep-beta/commit/cbb2491b85d7ff416d6d429b60109e6c2d6d50b9 for #13 but that's the cause of this bug. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Searching for "github-to-sqlite" throws an error 771316301 | |
747126777 | https://github.com/dogsheep/google-takeout-to-sqlite/issues/2#issuecomment-747126777 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/2 | MDEyOklzc3VlQ29tbWVudDc0NzEyNjc3Nw== | simonw 9599 | 2020-12-17T00:36:52Z | 2020-12-17T00:36:52Z | MEMBER | The memory profiler tricks I used in https://github.com/dogsheep/healthkit-to-sqlite/issues/7 could help figure out what's going on here. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
killed by oomkiller on large location-history 769376447 | |
747034481 | https://github.com/dogsheep/dogsheep-beta/issues/29#issuecomment-747034481 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/29 | MDEyOklzc3VlQ29tbWVudDc0NzAzNDQ4MQ== | simonw 9599 | 2020-12-16T21:17:05Z | 2020-12-16T21:17:05Z | MEMBER | I'm just going to add |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add search highlighting snippets 724759588 | |
747031608 | https://github.com/dogsheep/dogsheep-beta/issues/29#issuecomment-747031608 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/29 | MDEyOklzc3VlQ29tbWVudDc0NzAzMTYwOA== | simonw 9599 | 2020-12-16T21:15:18Z | 2020-12-16T21:15:18Z | MEMBER | Should I pass any other details to the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add search highlighting snippets 724759588 | |
747030964 | https://github.com/dogsheep/dogsheep-beta/issues/29#issuecomment-747030964 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/29 | MDEyOklzc3VlQ29tbWVudDc0NzAzMDk2NA== | simonw 9599 | 2020-12-16T21:14:54Z | 2020-12-16T21:14:54Z | MEMBER | To do this I'll need the search term to be passed to the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add search highlighting snippets 724759588 | |
747029636 | https://github.com/dogsheep/dogsheep-beta/issues/29#issuecomment-747029636 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/29 | MDEyOklzc3VlQ29tbWVudDc0NzAyOTYzNg== | simonw 9599 | 2020-12-16T21:14:03Z | 2020-12-16T21:14:03Z | MEMBER | I think I can do this as a cunning trick in
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add search highlighting snippets 724759588 | |
746735889 | https://github.com/dogsheep/github-to-sqlite/issues/58#issuecomment-746735889 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/58 | MDEyOklzc3VlQ29tbWVudDc0NjczNTg4OQ== | simonw 9599 | 2020-12-16T17:59:50Z | 2020-12-16T17:59:50Z | MEMBER | I don't want to add a full HTML parser (like BeautifulSoup) as a dependency for this feature. Since the HTML comes from a single, trusted source (GitHub) I could probably handle this using regular expressions. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Readme HTML has broken internal links 769150394 | |
746734412 | https://github.com/dogsheep/github-to-sqlite/issues/58#issuecomment-746734412 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/58 | MDEyOklzc3VlQ29tbWVudDc0NjczNDQxMg== | simonw 9599 | 2020-12-16T17:58:56Z | 2020-12-16T17:58:56Z | MEMBER | I'm going to rewrite those |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Readme HTML has broken internal links 769150394 | |
739058820 | https://github.com/dogsheep/dogsheep-photos/pull/29#issuecomment-739058820 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/29 | MDEyOklzc3VlQ29tbWVudDczOTA1ODgyMA== | simonw 9599 | 2020-12-04T22:32:35Z | 2020-12-04T22:32:35Z | MEMBER | Thanks for this! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Fixed bug in SQL query for photo scores 638375985 | |
735485677 | https://github.com/dogsheep/github-to-sqlite/issues/53#issuecomment-735485677 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/53 | MDEyOklzc3VlQ29tbWVudDczNTQ4NTY3Nw== | simonw 9599 | 2020-11-30T00:36:09Z | 2020-11-30T00:36:09Z | MEMBER | Given rate limits (see #51) this command might be better implemented by running a |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Command for fetching file contents 753000405 | |
735484186 | https://github.com/dogsheep/github-to-sqlite/issues/51#issuecomment-735484186 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/51 | MDEyOklzc3VlQ29tbWVudDczNTQ4NDE4Ng== | simonw 9599 | 2020-11-30T00:29:31Z | 2020-11-30T00:29:31Z | MEMBER | This just caused a failure in deploying the demo: https://github.com/dogsheep/github-to-sqlite/runs/1471304407?check_suite_focus=true
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
github-to-sqlite should handle rate limits better 703246031 | |
735483820 | https://github.com/dogsheep/github-to-sqlite/issues/46#issuecomment-735483820 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46 | MDEyOklzc3VlQ29tbWVudDczNTQ4MzgyMA== | simonw 9599 | 2020-11-30T00:27:47Z | 2020-11-30T00:27:47Z | MEMBER | So it looks like anything that pulls reviews needs to pull each review, then for each one pull the comments. I'm going to consider this blocked on smarter rate limit handling in #51. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Feature: pull request reviews and comments 664485022 | |
735483604 | https://github.com/dogsheep/github-to-sqlite/issues/46#issuecomment-735483604 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46 | MDEyOklzc3VlQ29tbWVudDczNTQ4MzYwNA== | simonw 9599 | 2020-11-30T00:26:50Z | 2020-11-30T00:26:50Z | MEMBER | It seems like there's a lot missing from that - those aren't particularly interesting given the data that is returned. From the docs at https://docs.github.com/en/free-pro-team@latest/rest/reference/pulls#reviews it looks like each review consists of multiple comments, and the comments are where the useful material is - https://docs.github.com/en/free-pro-team@latest/rest/reference/pulls#list-comments-for-a-pull-request-review
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Feature: pull request reviews and comments 664485022 | |
735482546 | https://github.com/dogsheep/github-to-sqlite/issues/46#issuecomment-735482546 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46 | MDEyOklzc3VlQ29tbWVudDczNTQ4MjU0Ng== | simonw 9599 | 2020-11-30T00:22:02Z | 2020-11-30T00:22:02Z | MEMBER | As for reviews... here's the output of
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Feature: pull request reviews and comments 664485022 | |
735482187 | https://github.com/dogsheep/github-to-sqlite/issues/46#issuecomment-735482187 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46 | MDEyOklzc3VlQ29tbWVudDczNTQ4MjE4Nw== | simonw 9599 | 2020-11-30T00:20:11Z | 2020-11-30T00:20:11Z | MEMBER | Pull request are now added, thanks to @adamjonas. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Feature: pull request reviews and comments 664485022 | |
735465708 | https://github.com/dogsheep/github-to-sqlite/issues/54#issuecomment-735465708 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/54 | MDEyOklzc3VlQ29tbWVudDczNTQ2NTcwOA== | simonw 9599 | 2020-11-29T22:08:46Z | 2020-11-29T22:08:46Z | MEMBER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
github-to-sqlite workflows command 753026003 | ||
735464493 | https://github.com/dogsheep/github-to-sqlite/issues/54#issuecomment-735464493 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/54 | MDEyOklzc3VlQ29tbWVudDczNTQ2NDQ5Mw== | simonw 9599 | 2020-11-29T21:57:32Z | 2020-11-29T21:57:32Z | MEMBER |
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
github-to-sqlite workflows command 753026003 | |
735464438 | https://github.com/dogsheep/github-to-sqlite/issues/54#issuecomment-735464438 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/54 | MDEyOklzc3VlQ29tbWVudDczNTQ2NDQzOA== | simonw 9599 | 2020-11-29T21:57:08Z | 2020-11-29T21:57:08Z | MEMBER | Inspired by this tweet from Michael Heap https://twitter.com/mheap/status/1333108608817631238 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
github-to-sqlite workflows command 753026003 | |
727692413 | https://github.com/dogsheep/swarm-to-sqlite/issues/11#issuecomment-727692413 | https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/11 | MDEyOklzc3VlQ29tbWVudDcyNzY5MjQxMw== | simonw 9599 | 2020-11-16T02:15:22Z | 2020-11-16T02:15:22Z | MEMBER | Thanks, I'll look into this. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Error thrown: sqlite3.OperationalError: table users has no column named lastName 743400216 | |
712266834 | https://github.com/dogsheep/dogsheep-beta/issues/29#issuecomment-712266834 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/29 | MDEyOklzc3VlQ29tbWVudDcxMjI2NjgzNA== | simonw 9599 | 2020-10-19T16:01:23Z | 2020-10-19T16:01:23Z | MEMBER | Might just be a documented pattern for how to configure this in YAML templates. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add search highlighting snippets 724759588 | |
711569063 | https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-711569063 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50 | MDEyOklzc3VlQ29tbWVudDcxMTU2OTA2Mw== | simonw 9599 | 2020-10-19T05:01:29Z | 2020-10-19T05:01:29Z | MEMBER | Demo of
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Commands for making authenticated API calls 703218756 | |
711089647 | https://github.com/dogsheep/dogsheep-beta/issues/28#issuecomment-711089647 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/28 | MDEyOklzc3VlQ29tbWVudDcxMTA4OTY0Nw== | simonw 9599 | 2020-10-17T22:43:13Z | 2020-10-17T22:43:13Z | MEMBER | Since my personal Dogsheep uses Datasette authentication, I'm going to need to pass through cookies. https://github.com/simonw/datasette/issues/1020 will solve that in the future but for now I need to solve it explicitly. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Switch to using datasette.client 723861683 | |
711081703 | https://github.com/dogsheep/healthkit-to-sqlite/issues/11#issuecomment-711081703 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11 | MDEyOklzc3VlQ29tbWVudDcxMTA4MTcwMw== | simonw 9599 | 2020-10-17T21:18:35Z | 2020-10-17T21:18:35Z | MEMBER | OK, if you upgrade to the just-released 1.0 this should work (it worked against my Spanish export). |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
export.xml file name varies with different language settings 723838331 | |
711079760 | https://github.com/dogsheep/healthkit-to-sqlite/issues/11#issuecomment-711079760 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11 | MDEyOklzc3VlQ29tbWVudDcxMTA3OTc2MA== | simonw 9599 | 2020-10-17T21:00:05Z | 2020-10-17T21:00:05Z | MEMBER | Checking for either |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
export.xml file name varies with different language settings 723838331 | |
711079056 | https://github.com/dogsheep/healthkit-to-sqlite/issues/11#issuecomment-711079056 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11 | MDEyOklzc3VlQ29tbWVudDcxMTA3OTA1Ng== | simonw 9599 | 2020-10-17T20:53:00Z | 2020-10-17T20:53:00Z | MEMBER | I think the safest thing is to sniff the first few lines of the file. Those should be the same no matter the language that was used: ```xml <!DOCTYPE HealthData [ ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
export.xml file name varies with different language settings 723838331 | |
711078917 | https://github.com/dogsheep/healthkit-to-sqlite/issues/11#issuecomment-711078917 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11 | MDEyOklzc3VlQ29tbWVudDcxMTA3ODkxNw== | simonw 9599 | 2020-10-17T20:51:55Z | 2020-10-17T20:52:03Z | MEMBER | I switched my phone to Spanish and ran an export - I got a file called |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
export.xml file name varies with different language settings 723838331 | |
711074306 | https://github.com/dogsheep/healthkit-to-sqlite/issues/11#issuecomment-711074306 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11 | MDEyOklzc3VlQ29tbWVudDcxMTA3NDMwNg== | simonw 9599 | 2020-10-17T20:16:22Z | 2020-10-17T20:16:22Z | MEMBER | The "first XML file in the root" solution is probably easier though! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
export.xml file name varies with different language settings 723838331 | |
711074031 | https://github.com/dogsheep/healthkit-to-sqlite/issues/11#issuecomment-711074031 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11 | MDEyOklzc3VlQ29tbWVudDcxMTA3NDAzMQ== | simonw 9599 | 2020-10-17T20:14:01Z | 2020-10-17T20:14:01Z | MEMBER | I'd be happy to teach the tool to look for |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
export.xml file name varies with different language settings 723838331 | |
706834800 | https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706834800 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDcwNjgzNDgwMA== | simonw 9599 | 2020-10-12T03:24:57Z | 2020-10-16T20:16:28Z | MEMBER | Here's my first attempt at a plugin for this: ```python from datasette import hookimpl import jinja2 START = "<en-note" END = "</en-note>" TEMPLATE = """ {}
""".strip()
EN_MEDIA_SCRIPT = """
Array.from(document.querySelectorAll('en-media')).forEach(el => {
let hash = el.getAttribute('hash');
let type = el.getAttribute('type');
let path = @hookimpl def render_cell(value, table): if not table: # Don't render content from arbitrary SQL queries, could be XSS hole return if not value or not isinstance(value, str): return value = value.strip() if value.startswith(START) and value.endswith(END): trimmed = value[len(START) : -len(END)] trimmed = trimmed.split(">", 1)[1] # Replace those horrible double newlines trimmed = trimmed.replace(" ") return jinja2.Markup(TEMPLATE.format(trimmed)) @hookimpl def extra_body_script(): return EN_MEDIA_SCRIPT ``` It works! It does however demonstrate that Evernote's "clip this webpage" feature means there is a LOT of weird HTML that can get into a note. It looks like they've filtered out the scripts but I wouldn't bet on it - they certainly don't filter out many of the inline styles. So running Bleach is almost certainly a good idea. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Figure out how to display images from <en-media> tags inline in Datasette 718938889 | |
707332912 | https://github.com/dogsheep/swarm-to-sqlite/issues/8#issuecomment-707332912 | https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/8 | MDEyOklzc3VlQ29tbWVudDcwNzMzMjkxMg== | simonw 9599 | 2020-10-12T20:35:06Z | 2020-10-12T20:35:06Z | MEMBER | Shipped a fix for this in swarm-to-sqlite 0.3.2. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Error thrown: table photos has no column named hasSticker 648245071 | |
706786548 | https://github.com/dogsheep/evernote-to-sqlite/issues/4#issuecomment-706786548 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/4 | MDEyOklzc3VlQ29tbWVudDcwNjc4NjU0OA== | simonw 9599 | 2020-10-11T23:39:46Z | 2020-10-11T23:39:46Z | MEMBER | Should have used porter stemming for this. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Configure FTS + add an index on the date columns 718938508 | |
706785201 | https://github.com/dogsheep/evernote-to-sqlite/issues/6#issuecomment-706785201 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/6 | MDEyOklzc3VlQ29tbWVudDcwNjc4NTIwMQ== | simonw 9599 | 2020-10-11T23:29:39Z | 2020-10-11T23:29:39Z | MEMBER | It looks to me like each of those |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Better handling of OCR data 718949182 | |
706785086 | https://github.com/dogsheep/evernote-to-sqlite/issues/6#issuecomment-706785086 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/6 | MDEyOklzc3VlQ29tbWVudDcwNjc4NTA4Ng== | simonw 9599 | 2020-10-11T23:28:50Z | 2020-10-11T23:28:50Z | MEMBER | The XML for the OCR stuff is a bit weird. Currently I'm doing this to it: This can produce some odd results, for example:
Which came from this image: The XML for that is:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Better handling of OCR data 718949182 | |
706784028 | https://github.com/dogsheep/evernote-to-sqlite/issues/4#issuecomment-706784028 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/4 | MDEyOklzc3VlQ29tbWVudDcwNjc4NDAyOA== | simonw 9599 | 2020-10-11T23:20:32Z | 2020-10-11T23:20:32Z | MEMBER | I haven't done the FTS on OCR yet. I'm going to move that to another ticket because it requires more thought. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Configure FTS + add an index on the date columns 718938508 | |
706776808 | https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706776808 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDcwNjc3NjgwOA== | simonw 9599 | 2020-10-11T22:23:14Z | 2020-10-11T22:23:14Z | MEMBER | ... but it's still important to be able to get to the rendered note directly from the browse notes |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Figure out how to display images from <en-media> tags inline in Datasette 718938889 | |
706776680 | https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706776680 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDcwNjc3NjY4MA== | simonw 9599 | 2020-10-11T22:22:16Z | 2020-10-11T22:22:16Z | MEMBER | Maybe the best way do this is with a custom route, |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Figure out how to display images from <en-media> tags inline in Datasette 718938889 | |
706776447 | https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706776447 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDcwNjc3NjQ0Nw== | simonw 9599 | 2020-10-11T22:20:32Z | 2020-10-11T22:20:32Z | MEMBER | Or... I could do this client-side. JavaScript that looks for |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Figure out how to display images from <en-media> tags inline in Datasette 718938889 | |
706776242 | https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706776242 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDcwNjc3NjI0Mg== | simonw 9599 | 2020-10-11T22:18:30Z | 2020-10-11T22:19:48Z | MEMBER | Alternatively, rather than relying on Maybe rename the column to Might need to feed them through Bleach too, just in case any nasty code can get into them. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Figure out how to display images from <en-media> tags inline in Datasette 718938889 |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issue_comments] ( [html_url] TEXT, [issue_url] TEXT, [id] INTEGER PRIMARY KEY, [node_id] TEXT, [user] INTEGER REFERENCES [users]([id]), [created_at] TEXT, [updated_at] TEXT, [author_association] TEXT, [body] TEXT, [reactions] TEXT, [issue] INTEGER REFERENCES [issues]([id]) , [performed_via_github_app] TEXT); CREATE INDEX [idx_issue_comments_issue] ON [issue_comments] ([issue]); CREATE INDEX [idx_issue_comments_user] ON [issue_comments] ([user]);
issue >30