html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app https://github.com/dogsheep/dogsheep-photos/issues/13#issuecomment-620273692,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/13,620273692,MDEyOklzc3VlQ29tbWVudDYyMDI3MzY5Mg==,9599,simonw,2020-04-27T22:42:50Z,2020-04-27T22:42:50Z,MEMBER,"``` >>> def ext_counts(directory): ... counts = {} ... for path in pathlib.Path(directory).glob(""**/*""): ... ext = path.suffix ... counts[ext] = counts.get(ext, 0) + 1 ... return counts ... >>> >>> ext_counts(""/Users/simon/Pictures/Photos Library.photoslibrary/originals"") {'': 16, '.heic': 15478, '.jpeg': 21691, '.mov': 946, '.png': 2262, '.gif': 38, '.mp4': 116, '.aae': 2} ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607888367,Also upload movie files, https://github.com/dogsheep/dogsheep-photos/issues/12#issuecomment-618796564,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/12,618796564,MDEyOklzc3VlQ29tbWVudDYxODc5NjU2NA==,9599,simonw,2020-04-24T04:35:25Z,2020-04-24T04:35:25Z,MEMBER,Code: https://github.com/dogsheep/photos-to-sqlite/blob/a388cf1f1b6b67752d669466cda8b171b6582171/photos_to_sqlite/cli.py#L109-L114,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",606033104,"If less than 500MB, show size in MB not GB", https://github.com/dogsheep/dogsheep-photos/issues/9#issuecomment-618725155,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/9,618725155,MDEyOklzc3VlQ29tbWVudDYxODcyNTE1NQ==,9599,simonw,2020-04-23T23:39:14Z,2020-04-23T23:39:14Z,MEMBER,"A few minutes later... ``` Fetching existing keys from S3... Got 22,446 existing keys Calculating hashes [####################################] 100% 22,441 hashed files, 610 are not yet in S3 Uploading 0.99 GB Uploading 610 photos [------------------------------------] 1/610 03:10:35 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",605938063,"upload command should be resumable, should only upload photos not already uploaded", https://github.com/dogsheep/dogsheep-photos/issues/9#issuecomment-618724149,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/9,618724149,MDEyOklzc3VlQ29tbWVudDYxODcyNDE0OQ==,9599,simonw,2020-04-23T23:35:29Z,2020-04-23T23:35:29Z,MEMBER,"``` % photos-to-sqlite upload photos.db ~/Pictures/Photos\ Library.photoslibrary/originals Fetching existing keys from S3... Got 22,446 existing keys Calculating hashes [####--------------------------------] 13% 00:04:14 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",605938063,"upload command should be resumable, should only upload photos not already uploaded", https://github.com/dogsheep/dogsheep-photos/issues/8#issuecomment-618100658,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/8,618100658,MDEyOklzc3VlQ29tbWVudDYxODEwMDY1OA==,9599,simonw,2020-04-23T00:03:35Z,2020-04-23T00:03:35Z,MEMBER,"Also MD5 isn't guaranteed for the ETag: > If an object is created by either the Multipart Upload or Part Copy operation, the ETag is not an MD5 digest, regardless of the method of encryption. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",605147638,Should I have used MD5 instead of SHA256?, https://github.com/dogsheep/dogsheep-photos/issues/8#issuecomment-618100434,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/8,618100434,MDEyOklzc3VlQ29tbWVudDYxODEwMDQzNA==,9599,simonw,2020-04-23T00:02:53Z,2020-04-23T00:02:53Z,MEMBER,"I don't think it matters one way or the other - I'm storing the sha256 in the filename, so the fact that I could read the MD5 back from the list bucket operation doesn't give me any benefits.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",605147638,Should I have used MD5 instead of SHA256?, https://github.com/dogsheep/github-to-sqlite/issues/31#issuecomment-617491607,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/31,617491607,MDEyOklzc3VlQ29tbWVudDYxNzQ5MTYwNw==,9599,simonw,2020-04-22T01:20:19Z,2020-04-22T01:20:19Z,MEMBER,"https://github-to-sqlite.dogsheep.net/github/milestones now link to repo: And so do issues: https://github-to-sqlite.dogsheep.net/github/issues ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603624862,Issue and milestone should have foreign key to repo, https://github.com/dogsheep/github-to-sqlite/issues/32#issuecomment-617490914,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/32,617490914,MDEyOklzc3VlQ29tbWVudDYxNzQ5MDkxNA==,9599,simonw,2020-04-22T01:17:44Z,2020-04-22T01:17:44Z,MEMBER,"https://github-to-sqlite.dogsheep.net/github?sql=select+html_url%2C+id%2C+issue+from+issue_comments+order+by+updated_at+desc+limit+101 now shows issues. And https://github-to-sqlite.dogsheep.net/github/issue_comments links to them: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",604222295,Issue comments don't appear to populate issues foreign key, https://github.com/dogsheep/github-to-sqlite/issues/32#issuecomment-617369247,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/32,617369247,MDEyOklzc3VlQ29tbWVudDYxNzM2OTI0Nw==,9599,simonw,2020-04-21T19:33:03Z,2020-04-21T19:33:03Z,MEMBER,Caused by #31.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",604222295,Issue comments don't appear to populate issues foreign key, https://github.com/dogsheep/github-to-sqlite/issues/32#issuecomment-617364956,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/32,617364956,MDEyOklzc3VlQ29tbWVudDYxNzM2NDk1Ng==,9599,simonw,2020-04-21T19:24:45Z,2020-04-21T19:24:45Z,MEMBER,"That's because I just broke this code: https://github.com/dogsheep/github-to-sqlite/blob/2cf75a0a036719eb7e57fdc7c5c2ea0f4c26978a/github_to_sqlite/utils.py#L131-L139 It expects the `repo` column to be `simonw/datasette` but it's now an ID instead. I should add a test for this as part of the fix.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",604222295,Issue comments don't appear to populate issues foreign key, https://github.com/dogsheep/github-to-sqlite/issues/31#issuecomment-617348174,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/31,617348174,MDEyOklzc3VlQ29tbWVudDYxNzM0ODE3NA==,9599,simonw,2020-04-21T18:50:29Z,2020-04-21T18:50:29Z,MEMBER,"Since this represents a breaking schema change for anyone running SQL queries against these tables, I'm going to do a major version bump to 2.0 when I release this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603624862,Issue and milestone should have foreign key to repo, https://github.com/dogsheep/github-to-sqlite/issues/31#issuecomment-616884647,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/31,616884647,MDEyOklzc3VlQ29tbWVudDYxNjg4NDY0Nw==,9599,simonw,2020-04-21T00:49:16Z,2020-04-21T00:50:20Z,MEMBER,"The API just gives us the `repository_url`: https://api.github.com/repos/simonw/datasette/issues ![Mozilla_Firefox_and_Topic__Week_2__Discussion__Submit_your_six_story_points_here](https://user-images.githubusercontent.com/9599/79812950-283cdb80-832f-11ea-8759-9633087d1e7e.png) We currently turn that into a `simonw/datasette` string here: https://github.com/dogsheep/github-to-sqlite/blob/e0e8d8caa9657b04bfb8a2cf16c9b580f38b1805/github_to_sqlite/utils.py#L43-L46","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603624862,Issue and milestone should have foreign key to repo, https://github.com/dogsheep/github-to-sqlite/issues/30#issuecomment-616883726,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/30,616883726,MDEyOklzc3VlQ29tbWVudDYxNjg4MzcyNg==,9599,simonw,2020-04-21T00:45:23Z,2020-04-21T00:45:23Z,MEMBER,"Demo of fix: https://github-to-sqlite.dogsheep.net/github/issues?assignee__notblank=1&milestone__notblank=1 ![github__issues__4_rows_where_where_assignee_is_not_blank_and_milestone_is_not_blank_sorted_by_updated_at_descending](https://user-images.githubusercontent.com/9599/79812758-b49ace80-832e-11ea-81db-bdf993b872cc.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603618244,Issues milestone column is the wrong type, https://github.com/dogsheep/github-to-sqlite/issues/29#issuecomment-616883275,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/29,616883275,MDEyOklzc3VlQ29tbWVudDYxNjg4MzI3NQ==,9599,simonw,2020-04-21T00:43:28Z,2020-04-21T00:43:28Z,MEMBER,"I'm copying repo from issue, which surprisingly is a string, not an integer ID.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603617013,Milestones should have foreign key to creator and repo, https://github.com/dogsheep/github-to-sqlite/issues/30#issuecomment-616879753,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/30,616879753,MDEyOklzc3VlQ29tbWVudDYxNjg3OTc1Mw==,9599,simonw,2020-04-21T00:29:29Z,2020-04-21T00:29:29Z,MEMBER,`assignee` looks like it's the incorrect type too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603618244,Issues milestone column is the wrong type, https://github.com/dogsheep/twitter-to-sqlite/issues/45#issuecomment-616029262,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/45,616029262,MDEyOklzc3VlQ29tbWVudDYxNjAyOTI2Mg==,9599,simonw,2020-04-19T04:39:21Z,2020-04-19T04:39:21Z,MEMBER,"![44714E00-8CC5-46CD-9E48-1F4DD148FCC8](https://user-images.githubusercontent.com/9599/79679696-09b6d300-81bd-11ea-80e4-0653d92e4f58.jpeg) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602619330,Use raise_for_status() everywhere, https://github.com/dogsheep/dogsheep-photos/issues/7#issuecomment-615993178,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/7,615993178,MDEyOklzc3VlQ29tbWVudDYxNTk5MzE3OA==,9599,simonw,2020-04-19T00:37:08Z,2020-04-19T00:37:08Z,MEMBER,https://pypi.org/project/ImageHash/ Is one option.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602585497,Integrate image content hashing, https://github.com/dogsheep/dogsheep-photos/issues/6#issuecomment-615983393,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/6,615983393,MDEyOklzc3VlQ29tbWVudDYxNTk4MzM5Mw==,9599,simonw,2020-04-18T23:53:10Z,2020-04-18T23:53:10Z,MEMBER,"``` $ photos-to-sqlite upload photos3.db ~/Pictures/Photos\ Library.photoslibrary/Masters/2020 Uploading 2.09 GB [##----------------------------------] 6% 00:36:37 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602575575,Add progress bar to upload command, https://github.com/dogsheep/dogsheep-photos/issues/6#issuecomment-615979923,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/6,615979923,MDEyOklzc3VlQ29tbWVudDYxNTk3OTkyMw==,9599,simonw,2020-04-18T23:36:02Z,2020-04-18T23:36:02Z,MEMBER,"I'll use a Click progress bar. To do this I need to first calculate the sum number of bytes in the photos that are going to be uploaded, then run the upload.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602575575,Add progress bar to upload command, https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615957385,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4,615957385,MDEyOklzc3VlQ29tbWVudDYxNTk1NzM4NQ==,9599,simonw,2020-04-18T21:56:16Z,2020-04-18T21:58:11Z,MEMBER,Got this working! I'll do EXIF in a separate ticket #3.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533539,Upload all my photos to a secure S3 bucket, https://github.com/dogsheep/dogsheep-photos/issues/5#issuecomment-615949574,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/5,615949574,MDEyOklzc3VlQ29tbWVudDYxNTk0OTU3NA==,9599,simonw,2020-04-18T21:06:07Z,2020-04-18T21:06:07Z,MEMBER,"``` $ photos-to-sqlite s3-auth Create S3 credentials and paste them here: Access key ID: xxx Secret access key: yyy $ cat auth.json { ""access_key_id"": ""xxx"", ""secret_access_key"": ""yyy"" } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602551638,photos-to-sqlite s3-auth command, https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615948102,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4,615948102,MDEyOklzc3VlQ29tbWVudDYxNTk0ODEwMg==,9599,simonw,2020-04-18T20:56:59Z,2020-04-18T20:56:59Z,MEMBER,"I'm going to start with this: `photos-to-sqlite upload photos.db ~/path/to/directory` This will scan the provided directory (and all sub-directories) for image files. It will then: * Calculate a sha256 of the contents of that file * Upload the file to a key that's `sha256.jpg` or `.heic` * Upload a `sha256.json` file with the original path to the image * Add that image to a `uploads` table in `photos.db` Stretch goal: grab the EXIF data and include that in the `.json` upload AND the `uploads` database table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533539,Upload all my photos to a secure S3 bucket, https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615947370,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4,615947370,MDEyOklzc3VlQ29tbWVudDYxNTk0NzM3MA==,9599,simonw,2020-04-18T20:52:13Z,2020-04-18T20:52:13Z,MEMBER,"This is great! I now have a key that can upload photos, and a separate key that can download photos OR generate signed URLs to access those photos. Next step: a script that starts uploading my photos.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533539,Upload all my photos to a secure S3 bucket, https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615947229,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4,615947229,MDEyOklzc3VlQ29tbWVudDYxNTk0NzIyOQ==,9599,simonw,2020-04-18T20:51:26Z,2020-04-18T20:51:26Z,MEMBER,"Running the upload again like this resulted in the correct content-type: ```python client.upload_file( ""/Users/simonw/Desktop/this_is_fine.jpg"", ""dogsheep-photos-simon"", ""this_is_fine.jpg"", ExtraArgs={ ""ContentType"": ""image/jpeg"" } ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533539,Upload all my photos to a secure S3 bucket, https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615946537,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4,615946537,MDEyOklzc3VlQ29tbWVudDYxNTk0NjUzNw==,9599,simonw,2020-04-18T20:48:13Z,2020-04-18T20:48:13Z,MEMBER,"How about generating a signed URL? ```python read_client.generate_presigned_url( ""get_object"", Params={ ""Bucket"": ""dogsheep-photos-simon"", ""Key"": ""this_is_fine.jpg"", }, ExpiresIn=600 ) ``` Gave me https://dogsheep-photos-simon.s3.amazonaws.com/this_is_fine.jpg?AWSAccessKeyId=AKIAWXFXAIOZNZ3JFO7I&Signature=x1zrS4w4OTGAACd7yHp9mYqXvN8%3D&Expires=1587243398 Which does this: ``` ~ $ curl -i 'https://dogsheep-photos-simon.s3.amazonaws.com/this_is_fine.jpg?AWSAccessKeyId=AKIAWXFXAIOZNZ3JFO7I&Signature=x1zrS4w4OTGAACd7yHp9mYqXvN8%3D&Expires=1587243398' HTTP/1.1 307 Temporary Redirect x-amz-bucket-region: us-west-1 x-amz-request-id: E78CD859AEE21D33 x-amz-id-2: 648mx+1+YSGga7NDOU7Q6isfsKnEPWOLC+DI4+x2o9FCc6pSCdIaoHJUbFMI8Vsuh1ADtx46ymU= Location: https://dogsheep-photos-simon.s3-us-west-1.amazonaws.com/this_is_fine.jpg?AWSAccessKeyId=AKIAWXFXAIOZNZ3JFO7I&Signature=x1zrS4w4OTGAACd7yHp9mYqXvN8%3D&Expires=1587243398 Content-Type: application/xml Transfer-Encoding: chunked Date: Sat, 18 Apr 2020 20:47:21 GMT Server: AmazonS3 TemporaryRedirectPlease re-send this request to the specified temporary endpoint. Continue to use the original request endpoint for future requests.dogsheep-photos-simon.s3-us-west-1.amazonaws.comdogsheep-photos-simonE78CD859AEE21D33648mx+1+YSGga7NDOU7Q6isfsKnEPWOLC+DI4+x2o9FCc6pSCdIaoHJUbFMI8Vsuh1ADtx46ymU=~ $ ``` So it redirects to another URL... which returns this: ``` ~ $ curl -i 'https://dogsheep-photos-simon.s3-us-west-1.amazonaws.com/this_is_fine.jpg?AWSAccessKeyId=AKIAWXFXAIOZNZ3JFO7I&Signature=x1zrS4w4OTGAACd7yHp9mYqXvN8%3D&Expires=1587243398' HTTP/1.1 200 OK x-amz-id-2: XafOl6mswj3yz0GJC9+Ptot1ll5sROVwqsMc10CUUfgpaUANTdIx2GhnONb5d1GVFJ6wlS2j3UY= x-amz-request-id: 258387C180411AFE Date: Sat, 18 Apr 2020 20:47:52 GMT Last-Modified: Sat, 18 Apr 2020 20:37:35 GMT ETag: ""ee04081c3182a44a1c6944e94012e977"" Accept-Ranges: bytes Content-Type: binary/octet-stream Content-Length: 53072 Server: AmazonS3 ????JFIF??C ``` So that worked! It did come back with `Content-Type: binary/octet-stream` though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533539,Upload all my photos to a secure S3 bucket, https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615945056,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4,615945056,MDEyOklzc3VlQ29tbWVudDYxNTk0NTA1Ng==,9599,simonw,2020-04-18T20:42:41Z,2020-04-18T20:42:41Z,MEMBER,"But... `list_objects` failed for both of my keys (read and write): ![Dogsheep_Photos_S3_access](https://user-images.githubusercontent.com/9599/79670798-75c41780-817a-11ea-9907-2cbc4a2e497c.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533539,Upload all my photos to a secure S3 bucket, https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615944806,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4,615944806,MDEyOklzc3VlQ29tbWVudDYxNTk0NDgwNg==,9599,simonw,2020-04-18T20:41:39Z,2020-04-18T20:41:39Z,MEMBER,"This worked! ![Dogsheep_Photos_S3_access](https://user-images.githubusercontent.com/9599/79670712-d868e380-8179-11ea-82a5-5dfd17356113.png) And this worked: ![Dogsheep_Photos_S3_access](https://user-images.githubusercontent.com/9599/79670777-50370e00-817a-11ea-83cd-18ebf5702878.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533539,Upload all my photos to a secure S3 bucket, https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615942116,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4,615942116,MDEyOklzc3VlQ29tbWVudDYxNTk0MjExNg==,9599,simonw,2020-04-18T20:30:56Z,2020-04-18T20:30:56Z,MEMBER,"Next step: attempt a programmatic upload using the `dogsheep-photos-simon-read-write` credentials from a Jupyter notebook. Also attempt a programmatic bucket listing and read using `dogsheep-photos-simon-read` credentials.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533539,Upload all my photos to a secure S3 bucket, https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615941746,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4,615941746,MDEyOklzc3VlQ29tbWVudDYxNTk0MTc0Ng==,9599,simonw,2020-04-18T20:29:36Z,2020-04-18T20:29:36Z,MEMBER,"I'm going to create another user just for Transmit, with full S3 access. name: `dogsheep-photos-simon-s3-all-access` Rather than creating a group for that user, I'm trying the ""Attach existing policies directly"" option: ![IAM_Management_Console](https://user-images.githubusercontent.com/9599/79670182-03513880-8176-11ea-811a-c80aefb4538a.png) That user DID work with Transmit. I uploaded a test HEIC image. I used Transmit to copy a signed URL for it. ``` ~ $ curl -i 'https://dogsheep-photos-simon.s3.us-west-1.amazonaws.com/IMG_7195.HEIC?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAWXFXAI...' | head -n 100 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0HTTP/1.1 200 OK x-amz-id-2: gBOCYqZfbNAnv0R/uJ++qm2NbW5SgD4TapgF9RQjzzeDIThcCz/BkKU+YoxlG4NJHlcmMgAHyh4= x-amz-request-id: C2FE7FCC3BD53A84 Date: Sat, 18 Apr 2020 20:28:54 GMT Last-Modified: Sat, 18 Apr 2020 20:13:49 GMT ETag: ""fe3e081239a123ef745517878c53b854"" Accept-Ranges: bytes Content-Type: image/heic Content-Length: 1913097 Server: AmazonS3 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533539,Upload all my photos to a secure S3 bucket, https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615936880,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4,615936880,MDEyOklzc3VlQ29tbWVudDYxNTkzNjg4MA==,9599,simonw,2020-04-18T20:04:31Z,2020-04-18T20:04:31Z,MEMBER,"Next step: create two IAM users, one for each of those groups. https://console.aws.amazon.com/iam/home#/users$new?step=details ![IAM_Management_Console](https://user-images.githubusercontent.com/9599/79669931-1bc05380-8174-11ea-9657-0e0c6a692d42.png) ![IAM_Management_Console](https://user-images.githubusercontent.com/9599/79669941-27137f00-8174-11ea-8ce7-249f0d4f96f6.png) I copied the keys into a secure note in 1password. Couldn't get into Transmit with them though! https://library.panic.com/transmit/transmit5/iam-roles/ may help.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533539,Upload all my photos to a secure S3 bucket, https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615935577,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4,615935577,MDEyOklzc3VlQ29tbWVudDYxNTkzNTU3Nw==,9599,simonw,2020-04-18T19:54:59Z,2020-04-18T19:55:30Z,MEMBER,"Creating IAM groups called `dogsheep-photos-simon-read-write` and `dogsheep-photos-simon-read`: https://console.aws.amazon.com/iam/home#/groups - I created them with no attached policies. Now I can attach an ""inline policy"" to each one. For the read-write group I go here: https://console.aws.amazon.com/iam/home#/groups/dogsheep-photos-simon-read-write ![IAM_Management_Console](https://user-images.githubusercontent.com/9599/79669703-2d086080-8172-11ea-9597-83e0b155193e.png) Example policies are here: https://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html For the read-write one I went with: ```json { ""Version"": ""2012-10-17"", ""Statement"": [ { ""Effect"": ""Allow"", ""Action"": ""s3:*"", ""Resource"": [ ""arn:aws:s3:::dogsheep-photos-simon/*"" ] } ] } ``` For the read-only policy I'm going to guess that this is appropriate: ```json { ""Version"": ""2012-10-17"", ""Statement"": [ { ""Effect"": ""Allow"", ""Action"": [ ""s3:GetObject*"", ""s3:ListBucket"" ], ""Resource"": [ ""arn:aws:s3:::dogsheep-photos-simon/*"" ] } ] } ``` I tried the policy simulator to test this out: https://policysim.aws.amazon.com/home/index.jsp?#groups/dogsheep-photos-simon-read - this worked: ![IAM_Policy_Simulator](https://user-images.githubusercontent.com/9599/79669893-cd12b980-8173-11ea-8dfb-5660ce3652da.png)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533539,Upload all my photos to a secure S3 bucket, https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615933273,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4,615933273,MDEyOklzc3VlQ29tbWVudDYxNTkzMzI3Mw==,9599,simonw,2020-04-18T19:37:33Z,2020-04-18T19:37:33Z,MEMBER,"https://console.aws.amazon.com/s3/bucket/create?region=us-west-1 ![S3_Management_Console](https://user-images.githubusercontent.com/9599/79669552-33e2a380-8171-11ea-9ab5-5785d34f652a.png) I created it with no public read-write access. I plan to use signed URLs via a transforming proxy to access images for display on the web.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533539,Upload all my photos to a secure S3 bucket, https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615932204,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4,615932204,MDEyOklzc3VlQ29tbWVudDYxNTkzMjIwNA==,9599,simonw,2020-04-18T19:29:22Z,2020-04-18T19:34:44Z,MEMBER,I'm going to call my bucket `dogsheep-photos-simon`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533539,Upload all my photos to a secure S3 bucket, https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615932007,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4,615932007,MDEyOklzc3VlQ29tbWVudDYxNTkzMjAwNw==,9599,simonw,2020-04-18T19:27:55Z,2020-04-18T19:27:55Z,MEMBER,"Research thread: https://twitter.com/simonw/status/1249049694984011776 > I want to build some software that lets people store their own data in their own S3 bucket, but if possible I'd like not to have to teach people the incantations needed to get their bucket setup and minimum-permission credentials figures out https://testdriven.io/blog/storing-django-static-and-media-files-on-amazon-s3/ looks useful","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533539,Upload all my photos to a secure S3 bucket, https://github.com/dogsheep/dogsheep-photos/issues/2#issuecomment-615931488,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/2,615931488,MDEyOklzc3VlQ29tbWVudDYxNTkzMTQ4OA==,9599,simonw,2020-04-18T19:24:02Z,2020-04-18T19:24:02Z,MEMBER,I made a start on this last week with a https://github.com/simonw/heic-to-jpeg proxy.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533352,Ability to convert HEIC images to JPEG, https://github.com/dogsheep/github-to-sqlite/issues/28#issuecomment-615886206,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/28,615886206,MDEyOklzc3VlQ29tbWVudDYxNTg4NjIwNg==,9599,simonw,2020-04-18T15:04:59Z,2020-04-18T15:04:59Z,MEMBER,"Demo: https://github-to-sqlite.dogsheep.net/github/contributors Documentation: https://github.com/dogsheep/github-to-sqlite/blob/13f8868fb5efa01c263b24f6dd91c617e6e938e1/README.md#fetching-contributors-to-a-repository ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",601333634,Pull repository contributors, https://github.com/dogsheep/github-to-sqlite/issues/28#issuecomment-615883687,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/28,615883687,MDEyOklzc3VlQ29tbWVudDYxNTg4MzY4Nw==,9599,simonw,2020-04-18T14:49:58Z,2020-04-18T14:49:58Z,MEMBER,"That happened trying to pull contributors for `dogsheep/beta` - an empty repository. Turns out it was returning a `204 no content`: ``` ~ $ curl -i 'https://api.github.com/repos/dogsheep/beta/contributors' HTTP/1.1 204 No Content ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",601333634,Pull repository contributors, https://github.com/dogsheep/github-to-sqlite/issues/28#issuecomment-615883040,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/28,615883040,MDEyOklzc3VlQ29tbWVudDYxNTg4MzA0MA==,9599,simonw,2020-04-18T14:45:38Z,2020-04-18T14:45:38Z,MEMBER,"``` File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py"", line 829, in __call__ return self.main(*args, **kwargs) File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py"", line 782, in main rv = self.invoke(ctx) File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py"", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py"", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py"", line 610, in invoke return callback(*args, **kwargs) File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py"", line 219, in contributors utils.save_contributors(db, contributors, repo_full[""id""]) File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py"", line 354, in save_contributors for contributor in contributors: File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py"", line 228, in fetch_contributors for contributors in paginate(url, headers): File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py"", line 286, in paginate data = response.json() File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/requests/models.py"", line 898, in json return complexjson.loads(self.text, **kwargs) File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/json/__init__.py"", line 357, in loads return _default_decoder.decode(s) File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/json/decoder.py"", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/json/decoder.py"", line 355, in raw_decode raise JSONDecodeError(""Expecting value"", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",601333634,Pull repository contributors, https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-615519409,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27,615519409,MDEyOklzc3VlQ29tbWVudDYxNTUxOTQwOQ==,9599,simonw,2020-04-18T00:19:16Z,2020-04-18T00:19:16Z,MEMBER,"``` $ github-to-sqlite repos b.db dogsheep $ sqlite3 b.db '.schema repos' CREATE TABLE [repos] ( [id] INTEGER PRIMARY KEY, ... [permissions] TEXT, [organization] INTEGER REFERENCES [users]([id]), FOREIGN KEY(license) REFERENCES licenses(key) ); ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",601330277,Repos have a big blob of JSON in the organization column, https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-615518606,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27,615518606,MDEyOklzc3VlQ29tbWVudDYxNTUxODYwNg==,9599,simonw,2020-04-18T00:14:32Z,2020-04-18T00:14:32Z,MEMBER,https://github.com/simonw/sqlite-utils/issues/100 is done and released in sqlite-utils 2.7.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",601330277,Repos have a big blob of JSON in the organization column, https://github.com/dogsheep/twitter-to-sqlite/issues/43#issuecomment-615513491,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/43,615513491,MDEyOklzc3VlQ29tbWVudDYxNTUxMzQ5MQ==,9599,simonw,2020-04-17T23:48:28Z,2020-04-17T23:48:28Z,MEMBER,Released in 0.21.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602176870,"""twitter-to-sqlite lists"" command for retrieving a user's owned lists", https://github.com/dogsheep/twitter-to-sqlite/issues/37#issuecomment-615510361,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/37,615510361,MDEyOklzc3VlQ29tbWVudDYxNTUxMDM2MQ==,9599,simonw,2020-04-17T23:38:27Z,2020-04-17T23:38:27Z,MEMBER,That's a bit tricky since I'd have to rewrite the internals of a bunch of other commands. For the moment I'll exit the script with an error but at least it will be a decent error!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585353598,"Handle ""User not found"" error", https://github.com/dogsheep/twitter-to-sqlite/issues/37#issuecomment-615509803,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/37,615509803,MDEyOklzc3VlQ29tbWVudDYxNTUwOTgwMw==,9599,simonw,2020-04-17T23:36:40Z,2020-04-17T23:36:40Z,MEMBER,"I'm going to print a warning to stderr, skip and continue - because if you have 100 screen names and only one of them is invalid you should still execute for the other 99.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585353598,"Handle ""User not found"" error", https://github.com/dogsheep/twitter-to-sqlite/issues/37#issuecomment-615509578,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/37,615509578,MDEyOklzc3VlQ29tbWVudDYxNTUwOTU3OA==,9599,simonw,2020-04-17T23:36:00Z,2020-04-17T23:36:00Z,MEMBER,"``` $ twitter-to-sqlite user-timeline doggo.db doggoenthuonetuh Traceback (most recent call last): ... File ""/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/utils.py"", line 272, in transform_user user[""created_at""] = parser.parse(user[""created_at""]) KeyError: 'created_at' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585353598,"Handle ""User not found"" error", https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-614843406,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27,614843406,MDEyOklzc3VlQ29tbWVudDYxNDg0MzQwNg==,9599,simonw,2020-04-16T19:11:53Z,2020-04-16T19:20:23Z,MEMBER,"This didn't quite work: the column type is incorrect, so the foreign key relationship isn't sticking: https://github-to-sqlite.dogsheep.net/github/repos?organization=53015001 `[organization] TEXT REFERENCES [users]([id])` - should be `INTEGER`. The problem is that if the first repo inserted has no organization it's set to `null`, which `sqlite-utils` derives as a `TEXT` column. One solution would be to create the column explicitly with a type, but this could get messy. I think I want a new sqlite-utils feature for this instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",601330277,Repos have a big blob of JSON in the organization column, https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-614831842,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27,614831842,MDEyOklzc3VlQ29tbWVudDYxNDgzMTg0Mg==,9599,simonw,2020-04-16T18:48:18Z,2020-04-16T18:48:18Z,MEMBER,I'm going to make `organization` another foreign key to the `users` table just in case it IS possible (maybe with GitHub Enterprise or similar?),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",601330277,Repos have a big blob of JSON in the organization column, https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-614831451,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27,614831451,MDEyOklzc3VlQ29tbWVudDYxNDgzMTQ1MQ==,9599,simonw,2020-04-16T18:47:25Z,2020-04-16T18:47:25Z,MEMBER,Is it possible for a repo to have an `owner` that differs from its `organization`?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",601330277,Repos have a big blob of JSON in the organization column, https://github.com/dogsheep/github-to-sqlite/issues/25#issuecomment-614810417,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/25,614810417,MDEyOklzc3VlQ29tbWVudDYxNDgxMDQxNw==,9599,simonw,2020-04-16T18:07:11Z,2020-04-16T18:07:11Z,MEMBER,Turns out the main problem was #26 - now fixed.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",601265023,Improvements to demo instance, https://github.com/dogsheep/github-to-sqlite/issues/26#issuecomment-614795712,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/26,614795712,MDEyOklzc3VlQ29tbWVudDYxNDc5NTcxMg==,9599,simonw,2020-04-16T17:40:27Z,2020-04-16T17:40:27Z,MEMBER,Aha! it was missing from the `fetch_repo()` function.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",601271612,Topics are missing from repositories, https://github.com/dogsheep/github-to-sqlite/issues/26#issuecomment-614794739,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/26,614794739,MDEyOklzc3VlQ29tbWVudDYxNDc5NDczOQ==,9599,simonw,2020-04-16T17:38:28Z,2020-04-16T17:38:28Z,MEMBER,I'm already doing this here: https://github.com/dogsheep/github-to-sqlite/blob/c4aaa50e167cfa9021c7c94260bc3e89e10947bf/github_to_sqlite/utils.py#L246-L250,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",601271612,Topics are missing from repositories, https://github.com/dogsheep/github-to-sqlite/issues/14#issuecomment-613641947,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14,613641947,MDEyOklzc3VlQ29tbWVudDYxMzY0MTk0Nw==,9599,simonw,2020-04-14T19:38:24Z,2020-04-14T19:38:34Z,MEMBER,"Since events include payloads with full object representations in them (for issues, repos and more) running this command every few minutes may be all it takes to keep a constant copy of everything updated in a very rate-limit friendly manner (thanks to the ETags).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",530491074,Command for importing events, https://github.com/dogsheep/github-to-sqlite/issues/16#issuecomment-613611455,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/16,613611455,MDEyOklzc3VlQ29tbWVudDYxMzYxMTQ1NQ==,9599,simonw,2020-04-14T18:37:21Z,2020-04-14T18:37:21Z,MEMBER,"This should have been fixed by #20 and #23 @jayvdb I'm definitely interested in this tool working as a library - it's purely designed as a CLI tool at the moment, but cleaning it up to work better as a dependency is totally in-scope for the project. https://sqlite-utils.readthedocs.io/ is an example of a tool I've built that works for both. Feel free to open a new issue here with some notes on what you would need for this to work as a library for your project!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",546051181,Exception running first command: IndexError: list index out of range, https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-607019151,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40,607019151,MDEyOklzc3VlQ29tbWVudDYwNzAxOTE1MQ==,9599,simonw,2020-04-01T04:11:10Z,2020-04-01T04:11:10Z,MEMBER,"In testing this collects a LOT of data. I'm going to skip tracking favourites_count and statuses_count and just track followers, friends and listed instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",590669793,Feature: record history of follower counts, https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-607011421,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40,607011421,MDEyOklzc3VlQ29tbWVudDYwNzAxMTQyMQ==,9599,simonw,2020-04-01T03:47:37Z,2020-04-01T03:55:08Z,MEMBER,"Actually a single table with a `type` integer ID referencing a `count_history_types` table would better match the way I implemented the `since_ids` table: https://github.com/dogsheep/twitter-to-sqlite/blob/4b6c8d8c1cc6fefdb566ec8506157133f47c569a/twitter_to_sqlite/utils.py#L331-L341 In which case the compound primary key would be `type`, `user`, `datetime`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",590669793,Feature: record history of follower counts, https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-607011972,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40,607011972,MDEyOklzc3VlQ29tbWVudDYwNzAxMTk3Mg==,9599,simonw,2020-04-01T03:49:02Z,2020-04-01T03:50:01Z,MEMBER,"I want the datetime value to look like `2020-04-01T03:34:58+00:00` (the format returned by the Twitter API which I am storing in other tables at the moment). ``` >>> datetime.utcnow().isoformat().split('.')[0] + '+00:00' '2020-04-01T03:49:52+00:00' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",590669793,Feature: record history of follower counts, https://github.com/dogsheep/twitter-to-sqlite/issues/10#issuecomment-607010791,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10,607010791,MDEyOklzc3VlQ29tbWVudDYwNzAxMDc5MQ==,9599,simonw,2020-04-01T03:45:48Z,2020-04-01T03:45:48Z,MEMBER,I'm happy with the recent work I did on this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",492297930,Rethink progress bars for various commands, https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-607010634,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39,607010634,MDEyOklzc3VlQ29tbWVudDYwNzAxMDYzNA==,9599,simonw,2020-04-01T03:45:16Z,2020-04-01T03:45:16Z,MEMBER,"OK, fix is applied to everything now.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",590666760,--since feature can be confused by retweets, https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-607003655,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39,607003655,MDEyOklzc3VlQ29tbWVudDYwNzAwMzY1NQ==,9599,simonw,2020-04-01T03:18:00Z,2020-04-01T03:18:00Z,MEMBER,I've got this working for the `user-timeline` command.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",590666760,--since feature can be confused by retweets, https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606850453,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39,606850453,MDEyOklzc3VlQ29tbWVudDYwNjg1MDQ1Mw==,9599,simonw,2020-03-31T20:14:58Z,2020-04-01T03:03:50Z,MEMBER,Actually I'll hard-code the population of `since_id_types` to get known ID constants.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",590666760,--since feature can be confused by retweets, https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606998669,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39,606998669,MDEyOklzc3VlQ29tbWVudDYwNjk5ODY2OQ==,9599,simonw,2020-04-01T02:57:36Z,2020-04-01T02:57:36Z,MEMBER,"The tricky thing here is thinking about the interaction between the recorded since_id and a desire to run the initial import. The first time you run `twitter-to-sqlite user-timeline db.db username` we want to fetch as many tweets from that user as possible - probably around 3,200 before the API limitations cut us off. We need to record the maximum ID from those as the `since_id` - which we will see on the very first page we paginate through. That way next time we run the command with `--since` we will only fetch new tweets. But what happens if our initial import is cancelled after only a few tweets? We risk never pulling in the rest of the tweets. Not sure if I need to solve this at all or if I should instead trust users to run the command a second time without `--since` if they think they didn't retrieve anything the first time. I had considered letting `--stop_after=` over-ride `--since` but that doesn't actually make sense - if you send a since_id to the Twitter API you'll never get back more tweets than exist after that ID, so the `--stop_after` would not make a meaningful difference.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",590666760,--since feature can be confused by retweets, https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606850008,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39,606850008,MDEyOklzc3VlQ29tbWVudDYwNjg1MDAwOA==,9599,simonw,2020-03-31T20:13:59Z,2020-04-01T00:23:00Z,MEMBER,"Table design for `since_ids` table: type | key | since_id --- | --- | --- 1 | 124324 | 2347239847293 2 | 99ff9cefff5cbfd804f7cd43e2b27ced8addbe8d | 2125947927344 Primary compound key on `(category, key)` `type` is also a foreign key to a `since_id_types` table with `id` and `name` columns (probably created using https://sqlite-utils.readthedocs.io/en/stable/python-api.html#working-with-lookup-tables )","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",590666760,--since feature can be confused by retweets, https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606843224,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39,606843224,MDEyOklzc3VlQ29tbWVudDYwNjg0MzIyNA==,9599,simonw,2020-03-31T19:59:11Z,2020-03-31T20:06:32Z,MEMBER,"Or... have a single `since_ids` table to track since values, and have its primary key be a string that looks something like this: `user:123145` `home:23441` `mentions:23425` `search:99ff9cefff5cbfd804f7cd43e2b27ced8addbe8d` That last example would use the hash generated here: https://github.com/dogsheep/twitter-to-sqlite/blob/810cb2af5a175837204389fd7f4b5721f8b325ab/twitter_to_sqlite/cli.py#L792-L808","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",590666760,--since feature can be confused by retweets, https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606844521,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39,606844521,MDEyOklzc3VlQ29tbWVudDYwNjg0NDUyMQ==,9599,simonw,2020-03-31T20:01:39Z,2020-03-31T20:01:39Z,MEMBER,"I think `utils.fetch_timeline()` grows a new argument, `since_key`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",590666760,--since feature can be confused by retweets, https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606824992,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39,606824992,MDEyOklzc3VlQ29tbWVudDYwNjgyNDk5Mg==,9599,simonw,2020-03-31T19:24:23Z,2020-03-31T19:24:23Z,MEMBER,"The `--since` option is actually used by four commands: * `user-timeline` * `home-timeline` * `mentions-timeline` * `search` All of them use the same `fetch_timeline()` utility function under the hood. I should move the logic that looks up the last `since_id` into that shared function. Question: should I have a table for each of those four methods or a single table that is used by them all? I'm leaning towards four separate tables.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",590666760,--since feature can be confused by retweets, https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606309165,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39,606309165,MDEyOklzc3VlQ29tbWVudDYwNjMwOTE2NQ==,9599,simonw,2020-03-30T23:41:31Z,2020-03-30T23:41:31Z,MEMBER,I like the separate `user_timeline_since` table solution.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",590666760,--since feature can be confused by retweets, https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-606307376,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40,606307376,MDEyOklzc3VlQ29tbWVudDYwNjMwNzM3Ng==,9599,simonw,2020-03-30T23:35:40Z,2020-03-30T23:39:15Z,MEMBER,"I think five separate tables: * followers_count_history * friends_count_history * listed_count_history * favourites_count_history * statuses_count_history Each with the following structure: * datetime (ISO UTC) * user (ID, foreign key to users) * count (integer) I'm tempted to have a compound primary key here - user, datetime ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",590669793,Feature: record history of follower counts, https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-606307019,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40,606307019,MDEyOklzc3VlQ29tbWVudDYwNjMwNzAxOQ==,9599,simonw,2020-03-30T23:34:27Z,2020-03-30T23:34:27Z,MEMBER,"The count properties available for a user are: * followers_count * friends_count * listed_count * favourites_count * statuses_count May as well track history for all of them? Should be pretty cheap to store.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",590669793,Feature: record history of follower counts, https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606305701,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39,606305701,MDEyOklzc3VlQ29tbWVudDYwNjMwNTcwMQ==,9599,simonw,2020-03-30T23:30:27Z,2020-03-30T23:30:27Z,MEMBER,A better alternative would be to maintain a separate table with the last seen since value for when we ran `user-timeline` for any specific user.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",590666760,--since feature can be confused by retweets, https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606304837,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39,606304837,MDEyOklzc3VlQ29tbWVudDYwNjMwNDgzNw==,9599,simonw,2020-03-30T23:27:50Z,2020-03-30T23:29:31Z,MEMBER,"One option would be something like this: ```sql select max(id) from tweets where user = ? and not exists (select id from tweets where retweeted_status = id) and not exists (select id from tweets where quoted_status = id) and not exists (select id from tweets where in_reply_to_status_id = id) ``` Might be a good idea to index those columns (after confirming that doing so would indeed speed up the query).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",590666760,--since feature can be confused by retweets, https://github.com/dogsheep/swarm-to-sqlite/pull/6#issuecomment-605382373,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/6,605382373,MDEyOklzc3VlQ29tbWVudDYwNTM4MjM3Mw==,9599,simonw,2020-03-28T02:27:32Z,2020-03-28T02:27:32Z,MEMBER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",543355051,don't break if source is missing, https://github.com/dogsheep/pocket-to-sqlite/issues/2#issuecomment-605338322,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/2,605338322,MDEyOklzc3VlQ29tbWVudDYwNTMzODMyMg==,9599,simonw,2020-03-27T22:18:02Z,2020-03-27T22:18:02Z,MEMBER,Just needs documentation now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503234169,Track and use the 'since' value, https://github.com/dogsheep/pocket-to-sqlite/issues/2#issuecomment-605337941,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/2,605337941,MDEyOklzc3VlQ29tbWVudDYwNTMzNzk0MQ==,9599,simonw,2020-03-27T22:16:32Z,2020-03-27T22:16:32Z,MEMBER,"Need to test this. I have 7,394 items in my database right now. I'm going to save a new thing. Then I ran this: ``` pocket-to-sqlite fetch pocket-simon.db ``` And it worked!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503234169,Track and use the 'since' value, https://github.com/dogsheep/pocket-to-sqlite/issues/1#issuecomment-605327655,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/1,605327655,MDEyOklzc3VlQ29tbWVudDYwNTMyNzY1NQ==,9599,simonw,2020-03-27T21:42:49Z,2020-03-27T21:42:49Z,MEMBER,Or maybe it was because of the current Google Cloud outage? https://news.ycombinator.com/item?id=22706677,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503233021,Use better pagination (and implement progress bar), https://github.com/dogsheep/pocket-to-sqlite/issues/1#issuecomment-605325897,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/1,605325897,MDEyOklzc3VlQ29tbWVudDYwNTMyNTg5Nw==,9599,simonw,2020-03-27T21:37:26Z,2020-03-27T21:38:37Z,MEMBER,"I keep getting 503 errors even though I appear to be staying within the rate limit: ``` {'Date': 'Fri, 27 Mar 2020 21:35:57 GMT', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Server': 'Apache/2.4.25 (Debian)', 'Content-Location': 'get.php', 'Vary': 'negotiate', 'TCN': 'choice', 'Set-Cookie': '...; httponly', 'X-Frame-Options': 'SAMEORIGIN', 'Status': '200 OK', 'X-Limit-Key-Limit': '10000', 'X-Limit-Key-Remaining': '9960', 'X-Limit-Key-Reset': '282', 'X-Source': 'Pocket', 'P3P': 'policyref=""/w3c/p3p.xml"", CP=""ALL CURa ADMa DEVa OUR IND UNI COM NAV INT STA PRE""'} [##----------------------------------] 6% 06:49:27 {'Date': 'Fri, 27 Mar 2020 21:36:06 GMT', 'Content-Type': 'text/html; charset=UTF-8', 'Content-Length': '23', 'Connection': 'keep-alive', 'Server': 'Apache/2.4.25 (Debian)', 'Content-Location': 'get.php', 'Vary': 'negotiate', 'TCN': 'choice', 'Set-Cookie': '...', 'X-Frame-Options': 'SAMEORIGIN', 'X-Error': 'Pocket is currently under heavy load. Please wait a moment and try again.', 'X-Error-Code': '199', 'Status': '503 Service Unavailable', 'X-Source': 'Pocket', 'P3P': 'policyref=""/w3c/p3p.xml"", CP=""ALL CURa ADMa DEVa OUR IND UNI COM NAV INT STA PRE""'} ``` I'm going to try doing a few automatic retries any time I see a 503 error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503233021,Use better pagination (and implement progress bar), https://github.com/dogsheep/pocket-to-sqlite/issues/1#issuecomment-605316146,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/1,605316146,MDEyOklzc3VlQ29tbWVudDYwNTMxNjE0Ng==,9599,simonw,2020-03-27T21:09:15Z,2020-03-27T21:09:22Z,MEMBER,"For a progress bar I need to know how many total items there are. I found an undocumented API for this! `/v3/stats` which returns: ```json { ""count_list"": 7394, ""count_read"": 1016, ""count_unread"": 6378, ""status"": 1 } ``` I guessed this based on the documented v2 API: https://getpocket.com/api/v2_docs/#stats","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503233021,Use better pagination (and implement progress bar), https://github.com/dogsheep/github-to-sqlite/issues/23#issuecomment-602928533,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/23,602928533,MDEyOklzc3VlQ29tbWVudDYwMjkyODUzMw==,9599,simonw,2020-03-24T00:15:49Z,2020-03-24T00:15:49Z,MEMBER,https://github.com/dogsheep/github-to-sqlite/releases/tag/1.0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",586595839,Release 1.0, https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602924714,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13,602924714,MDEyOklzc3VlQ29tbWVudDYwMjkyNDcxNA==,9599,simonw,2020-03-24T00:03:25Z,2020-03-24T00:03:25Z,MEMBER,This is good enough for the 1.0 release.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521275281,Set up a live demo Datasette instance, https://github.com/dogsheep/github-to-sqlite/issues/21#issuecomment-602920163,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/21,602920163,MDEyOklzc3VlQ29tbWVudDYwMjkyMDE2Mw==,9599,simonw,2020-03-23T23:48:22Z,2020-03-23T23:48:22Z,MEMBER,"I'm happy with this pattern: https://github.com/dogsheep/github-to-sqlite/blob/f78c4e9baaf0970ffab266ba780df7240aae9f32/github_to_sqlite/utils.py#L4-L18","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",586561727,Turn GitHub API errors into exceptions, https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602918689,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13,602918689,MDEyOklzc3VlQ29tbWVudDYwMjkxODY4OQ==,9599,simonw,2020-03-23T23:43:39Z,2020-03-23T23:47:50Z,MEMBER,I pointed https://github-to-sqlite.dogsheep.net/ at it. May take a few minutes for the new certificate to provision though.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521275281,Set up a live demo Datasette instance, https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602919058,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13,602919058,MDEyOklzc3VlQ29tbWVudDYwMjkxOTA1OA==,9599,simonw,2020-03-23T23:44:48Z,2020-03-23T23:44:48Z,MEMBER,Next step: use a `metadata.json` file to add some extras. And add the `datasette-render-markdown` plugin as soon as I ship https://github.com/simonw/datasette-render-markdown/issues/2 (GFM support).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521275281,Set up a live demo Datasette instance, https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602917713,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13,602917713,MDEyOklzc3VlQ29tbWVudDYwMjkxNzcxMw==,9599,simonw,2020-03-23T23:40:29Z,2020-03-23T23:40:29Z,MEMBER,"Most recently updated issues across all Dogsheep repos, with faceting: https://github-to-sqlite-j7hipcg4aq-uc.a.run.app/github/issues?_facet=repo&_facet=user&_facet=state&_facet=author_association&_facet=type&_sort_desc=updated_at","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521275281,Set up a live demo Datasette instance, https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602916947,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13,602916947,MDEyOklzc3VlQ29tbWVudDYwMjkxNjk0Nw==,9599,simonw,2020-03-23T23:38:06Z,2020-03-23T23:38:06Z,MEMBER,Woohoo! https://github-to-sqlite-j7hipcg4aq-uc.a.run.app/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521275281,Set up a live demo Datasette instance, https://github.com/dogsheep/github-to-sqlite/issues/21#issuecomment-602896434,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/21,602896434,MDEyOklzc3VlQ29tbWVudDYwMjg5NjQzNA==,9599,simonw,2020-03-23T22:43:37Z,2020-03-23T22:43:37Z,MEMBER,"I'm going to do this now to help figure out the latest error in #13: ``` Traceback (most recent call last): File ""/opt/hostedtoolcache/Python/3.8.2/x64/bin/github-to-sqlite"", line 11, in load_entry_point('github-to-sqlite', 'console_scripts', 'github-to-sqlite')() File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py"", line 829, in __call__ return self.main(*args, **kwargs) File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py"", line 782, in main rv = self.invoke(ctx) File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py"", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py"", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py"", line 610, in invoke return callback(*args, **kwargs) File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py"", line 237, in commits utils.save_commits(db, commits, repo_full[""id""]) File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py"", line 345, in save_commits for commit in commits: File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py"", line 207, in fetch_commits if stop_when(commit): File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py"", line 224, in stop_when db[""commits""].get(commit[""sha""]) TypeError: string indices must be integers ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",586561727,Turn GitHub API errors into exceptions, https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602895896,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13,602895896,MDEyOklzc3VlQ29tbWVudDYwMjg5NTg5Ng==,9599,simonw,2020-03-23T22:42:25Z,2020-03-23T22:42:25Z,MEMBER,"Urgh this is such a mess! I should have done this on a branch / pull request to avoid polluting my main master history, but never mind.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521275281,Set up a live demo Datasette instance, https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602862967,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13,602862967,MDEyOklzc3VlQ29tbWVudDYwMjg2Mjk2Nw==,9599,simonw,2020-03-23T21:22:04Z,2020-03-23T21:22:04Z,MEMBER,Following these instructions: https://simonwillison.net/2020/Jan/21/github-actions-cloud-run/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521275281,Set up a live demo Datasette instance, https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602862236,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13,602862236,MDEyOklzc3VlQ29tbWVudDYwMjg2MjIzNg==,9599,simonw,2020-03-23T21:20:26Z,2020-03-23T21:20:26Z,MEMBER,I'll run the `commits` and `issues` and `issue-comments` commands in addition to the `releases` command.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521275281,Set up a live demo Datasette instance, https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602861730,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13,602861730,MDEyOklzc3VlQ29tbWVudDYwMjg2MTczMA==,9599,simonw,2020-03-23T21:19:16Z,2020-03-23T21:19:42Z,MEMBER,I'm going to do this with a scheduled GitHub Action in this repo.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521275281,Set up a live demo Datasette instance, https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602813731,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18,602813731,MDEyOklzc3VlQ29tbWVudDYwMjgxMzczMQ==,9599,simonw,2020-03-23T19:38:03Z,2020-03-23T20:44:49Z,MEMBER,"So it turns out `""author""` and `""committer""` on the commit are `null` if the email address in the nested `""commit""` doesn't match an existing GitHub user. Maybe I should be storing the nested data somewhere as well?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585411547,Commits in GitHub API can have null author, https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602846293,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18,602846293,MDEyOklzc3VlQ29tbWVudDYwMjg0NjI5Mw==,9599,simonw,2020-03-23T20:44:40Z,2020-03-23T20:44:40Z,MEMBER,I implemented the `raw_authors` idea.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585411547,Commits in GitHub API can have null author, https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602815120,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18,602815120,MDEyOklzc3VlQ29tbWVudDYwMjgxNTEyMA==,9599,simonw,2020-03-23T19:40:55Z,2020-03-23T19:43:19Z,MEMBER,I could pull a pk-hashed version of the name/email into separate `raw_author` and `raw_committer` columns perhaps - against a `commit_authors` table. Could be interesting.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585411547,Commits in GitHub API can have null author, https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602811785,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18,602811785,MDEyOklzc3VlQ29tbWVudDYwMjgxMTc4NQ==,9599,simonw,2020-03-23T19:33:56Z,2020-03-23T19:34:06Z,MEMBER,"```json [ { ""sha"": ""a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0"", ""node_id"": ""MDY6Q29tbWl0OTMyMDk6YThkYzkxNDA4OWQzOTlkOWI1MjJlYmI1MWI2N2Y5YWMyZThhYTZiMA=="", ""commit"": { ""author"": { ""name"": ""Simon Willison"", ""email"": ""simon@..."", ""date"": ""2008-12-18T23:17:12Z"" }, ""committer"": { ""name"": ""Simon Willison"", ""email"": ""simon@..."", ""date"": ""2008-12-18T23:17:12Z"" }, ""message"": ""First commit"", ""tree"": { ""sha"": ""ac2dfb75e2592c59165c2880f3f7a16dafd452a1"", ""url"": ""https://api.github.com/repos/simonw/simonw.github.com/git/trees/ac2dfb75e2592c59165c2880f3f7a16dafd452a1"" }, ""url"": ""https://api.github.com/repos/simonw/simonw.github.com/git/commits/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0"", ""comment_count"": 0, ""verification"": { ""verified"": false, ""reason"": ""unsigned"", ""signature"": null, ""payload"": null } }, ""url"": ""https://api.github.com/repos/simonw/simonw.github.com/commits/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0"", ""html_url"": ""https://github.com/simonw/simonw.github.com/commit/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0"", ""comments_url"": ""https://api.github.com/repos/simonw/simonw.github.com/commits/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0/comments"", ""author"": null, ""committer"": null, ""parents"": [] } ] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585411547,Commits in GitHub API can have null author, https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602811468,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18,602811468,MDEyOklzc3VlQ29tbWVudDYwMjgxMTQ2OA==,9599,simonw,2020-03-23T19:33:14Z,2020-03-23T19:33:14Z,MEMBER,Found one: https://api.github.com/repos/simonw/simonw.github.com/commits,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585411547,Commits in GitHub API can have null author, https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602807178,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18,602807178,MDEyOklzc3VlQ29tbWVudDYwMjgwNzE3OA==,9599,simonw,2020-03-23T19:24:43Z,2020-03-23T19:24:43Z,MEMBER,I need to find an example before I work on this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585411547,Commits in GitHub API can have null author, https://github.com/dogsheep/github-to-sqlite/issues/5#issuecomment-602806530,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/5,602806530,MDEyOklzc3VlQ29tbWVudDYwMjgwNjUzMA==,9599,simonw,2020-03-23T19:23:30Z,2020-03-23T19:23:30Z,MEMBER,Yeah `alter=True` means that this isn't necessary.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",493671014,"Add ""incomplete"" boolean to users table for incomplete profiles", https://github.com/dogsheep/github-to-sqlite/issues/19#issuecomment-602782437,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/19,602782437,MDEyOklzc3VlQ29tbWVudDYwMjc4MjQzNw==,9599,simonw,2020-03-23T18:36:12Z,2020-03-23T18:51:23Z,MEMBER,"* `commits`: `message` * `issue_comments`: `body` * `issues`: `title`, `body` * `labels`: `name`, `description` * `licenses`: `name` * `milestones`: `title`, `description` * `releases`: `name`, `body` * `repos`: `name`, `description` * `users`: `login`, `name` (maybe `company`, `bio`)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585850715,"Enable full-text search for more stuff (like commits, issues and issue_comments)", https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-602312054,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15,602312054,MDEyOklzc3VlQ29tbWVudDYwMjMxMjA1NA==,9599,simonw,2020-03-23T01:07:10Z,2020-03-23T01:07:10Z,MEMBER,"Each asset looks like this: ```json { ""url"": ""https://api.github.com/repos/instrumenta/conftest/releases/assets/11811946"", ""id"": 11811946, ""node_id"": ""MDEyOlJlbGVhc2VBc3NldDExODExOTQ2"", ""name"": ""checksums.txt"", ""label"": """", ""uploader"": { ""login"": ""garethr"", ""id"": 2029, ""node_id"": ""MDQ6VXNlcjIwMjk="", ""avatar_url"": ""https://avatars2.githubusercontent.com/u/2029?v=4"", ""gravatar_id"": """", ""url"": ""https://api.github.com/users/garethr"", ""html_url"": ""https://github.com/garethr"", ""followers_url"": ""https://api.github.com/users/garethr/followers"", ""following_url"": ""https://api.github.com/users/garethr/following{/other_user}"", ""gists_url"": ""https://api.github.com/users/garethr/gists{/gist_id}"", ""starred_url"": ""https://api.github.com/users/garethr/starred{/owner}{/repo}"", ""subscriptions_url"": ""https://api.github.com/users/garethr/subscriptions"", ""organizations_url"": ""https://api.github.com/users/garethr/orgs"", ""repos_url"": ""https://api.github.com/users/garethr/repos"", ""events_url"": ""https://api.github.com/users/garethr/events{/privacy}"", ""received_events_url"": ""https://api.github.com/users/garethr/received_events"", ""type"": ""User"", ""site_admin"": false }, ""content_type"": ""text/plain; charset=utf-8"", ""state"": ""uploaded"", ""size"": 600, ""download_count"": 2, ""created_at"": ""2019-03-30T16:56:44Z"", ""updated_at"": ""2019-03-30T16:56:44Z"", ""browser_download_url"": ""https://github.com/instrumenta/conftest/releases/download/v0.1.0/checksums.txt"" } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",544571092,Assets table with downloads, https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-602311701,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15,602311701,MDEyOklzc3VlQ29tbWVudDYwMjMxMTcwMQ==,9599,simonw,2020-03-23T01:05:15Z,2020-03-23T01:06:14Z,MEMBER,"None of my own releases use assets (they are all pushed to PyPI instead) but I spotted that your project here uses assets, so I'll test against that: https://github.com/instrumenta/conftest/releases/tag/v0.18.0 ``` github-to-sqlite releases releases.db instrumenta/conftest ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",544571092,Assets table with downloads, https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-602309825,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15,602309825,MDEyOklzc3VlQ29tbWVudDYwMjMwOTgyNQ==,9599,simonw,2020-03-23T00:52:38Z,2020-03-23T00:52:38Z,MEMBER,Splitting assets out into a separate table totally makes sense to me. They can still be fetched as part of the `releases` command.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",544571092,Assets table with downloads, https://github.com/dogsheep/github-to-sqlite/issues/19#issuecomment-602304531,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/19,602304531,MDEyOklzc3VlQ29tbWVudDYwMjMwNDUzMQ==,9599,simonw,2020-03-23T00:20:28Z,2020-03-23T00:20:28Z,MEMBER,"Current FTS code: https://github.com/dogsheep/github-to-sqlite/blob/e35eec4343aa560c58c1634cc228d0d46c442304/github_to_sqlite/utils.py#L136-L143","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585850715,"Enable full-text search for more stuff (like commits, issues and issue_comments)", https://github.com/dogsheep/hacker-news-to-sqlite/issues/1#issuecomment-602094386,https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/1,602094386,MDEyOklzc3VlQ29tbWVudDYwMjA5NDM4Ng==,9599,simonw,2020-03-21T19:47:46Z,2020-03-21T19:47:46Z,MEMBER,Released as 0.3.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585526292,Set up full text search, https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-601980957,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18,601980957,MDEyOklzc3VlQ29tbWVudDYwMTk4MDk1Nw==,9599,simonw,2020-03-21T02:30:03Z,2020-03-21T02:30:03Z,MEMBER,"https://github.community/t5/GitHub-API-Development-and/Request-for-commits-quot-author-quot-null-and-quot-committer/m-p/35842/highlight/true#M3372 > Commits aren't always associated with a GitHub user. For example, perhaps a friend of mine and I were working on a project together. I have a GitHub account and my friend doesn't. If we both add commits to the repository using our own email addresses and names and then I push the repository to GitHub, my commits will be associated with my GitHub user account but my friends' commits will show up with `author` and `committer` as `null`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585411547,Commits in GitHub API can have null author, https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-601980724,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18,601980724,MDEyOklzc3VlQ29tbWVudDYwMTk4MDcyNA==,9599,simonw,2020-03-21T02:28:05Z,2020-03-21T02:28:05Z,MEMBER,"This suggests that `commit[""author""]` can be `None` in some cases?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585411547,Commits in GitHub API can have null author, https://github.com/dogsheep/github-to-sqlite/issues/16#issuecomment-601979709,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/16,601979709,MDEyOklzc3VlQ29tbWVudDYwMTk3OTcwOQ==,9599,simonw,2020-03-21T02:20:08Z,2020-03-21T02:20:08Z,MEMBER,I'm pretty sure this means I need to upgrade the underlying `sqlite-utils` dependency. I saw the same bug in `twitter-to-sqlite`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",546051181,Exception running first command: IndexError: list index out of range, https://github.com/dogsheep/twitter-to-sqlite/issues/38#issuecomment-601939656,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/38,601939656,MDEyOklzc3VlQ29tbWVudDYwMTkzOTY1Ng==,9599,simonw,2020-03-20T22:37:04Z,2020-03-20T22:37:04Z,MEMBER,"New formatting: ``` @Alaska_DHSS [####################################] 3205 @BWhippleKS [####################################] 1 @CAPublicHealth [####################################] 2 @CDPHE [####################################] 67 @CHFSKy [####################################] 3216 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585359363,Screen name display for user-timeline is uneven, https://github.com/dogsheep/twitter-to-sqlite/issues/35#issuecomment-601892338,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35,601892338,MDEyOklzc3VlQ29tbWVudDYwMTg5MjMzOA==,9599,simonw,2020-03-20T20:17:00Z,2020-03-20T20:17:00Z,MEMBER,Released as 0.18.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585282212,twitter-to-sqlite user-timeline [screen_names] --sql / --attach, https://github.com/dogsheep/twitter-to-sqlite/issues/35#issuecomment-601890795,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35,601890795,MDEyOklzc3VlQ29tbWVudDYwMTg5MDc5NQ==,9599,simonw,2020-03-20T20:13:08Z,2020-03-20T20:13:08Z,MEMBER,Needs documentation.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585282212,twitter-to-sqlite user-timeline [screen_names] --sql / --attach, https://github.com/dogsheep/twitter-to-sqlite/issues/35#issuecomment-601875405,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35,601875405,MDEyOklzc3VlQ29tbWVudDYwMTg3NTQwNQ==,9599,simonw,2020-03-20T19:30:09Z,2020-03-20T19:32:33Z,MEMBER,"I'm going to redesign the command a little. Right now it defaults to fetching the authenticated user's timeline and allows `--screen_name` to be used to fetch for one other user instead. I think the design of `users-lookup` is much better. I'm going to copy that: ``` twitter-to-sqlite user-timeline users.db simonw cleopaws twitter-to-sqlite user-timeline users.db 12497 3166449535 --ids ``` It will also accept `--since` and `--sql` and `--alter` and `--since_id`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585282212,twitter-to-sqlite user-timeline [screen_names] --sql / --attach, https://github.com/dogsheep/twitter-to-sqlite/issues/35#issuecomment-601876051,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35,601876051,MDEyOklzc3VlQ29tbWVudDYwMTg3NjA1MQ==,9599,simonw,2020-03-20T19:31:43Z,2020-03-20T19:31:43Z,MEMBER,I'm going to keep `--screen_name` working for backwards compatibility. Using `hidden=True` on it should hide it from the help output.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585282212,twitter-to-sqlite user-timeline [screen_names] --sql / --attach, https://github.com/dogsheep/twitter-to-sqlite/issues/35#issuecomment-601875524,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35,601875524,MDEyOklzc3VlQ29tbWVudDYwMTg3NTUyNA==,9599,simonw,2020-03-20T19:30:27Z,2020-03-20T19:30:27Z,MEMBER,"I can give it a snazzier progress bar to, as requested by #10.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585282212,twitter-to-sqlite user-timeline [screen_names] --sql / --attach, https://github.com/dogsheep/twitter-to-sqlite/pull/33#issuecomment-601871930,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/33,601871930,MDEyOklzc3VlQ29tbWVudDYwMTg3MTkzMA==,9599,simonw,2020-03-20T19:21:41Z,2020-03-20T19:21:41Z,MEMBER,Made obsolete by https://github.com/dogsheep/twitter-to-sqlite/commit/d3887d1535c3edd59d3ebf6aea4495bacd82e59b,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",561469252,Upgrade to sqlite-utils 2.2.1, https://github.com/dogsheep/twitter-to-sqlite/issues/34#issuecomment-601871579,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/34,601871579,MDEyOklzc3VlQ29tbWVudDYwMTg3MTU3OQ==,9599,simonw,2020-03-20T19:20:52Z,2020-03-20T19:20:52Z,MEMBER,Cause was the redefining of how `upsert()` works in sqlite-utils 2.x.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585266763,IndexError running user-timeline command, https://github.com/dogsheep/twitter-to-sqlite/issues/34#issuecomment-601861908,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/34,601861908,MDEyOklzc3VlQ29tbWVudDYwMTg2MTkwOA==,9599,simonw,2020-03-20T18:56:44Z,2020-03-20T18:56:44Z,MEMBER,"Could this be a bug in `sqlite-utils`? This table has a primary key, so why is it running a query on `rowid = ?`?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585266763,IndexError running user-timeline command, https://github.com/dogsheep/github-to-sqlite/issues/17#issuecomment-597358364,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/17,597358364,MDEyOklzc3VlQ29tbWVudDU5NzM1ODM2NA==,9599,simonw,2020-03-10T22:50:20Z,2020-03-11T01:18:36Z,MEMBER,By default it will stop when it sees a commit that has already been stored. You will be able to over-ride that behaviour using `--all`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",578883725,Command for importing commits, https://github.com/dogsheep/github-to-sqlite/issues/17#issuecomment-597354514,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/17,597354514,MDEyOklzc3VlQ29tbWVudDU5NzM1NDUxNA==,9599,simonw,2020-03-10T22:37:45Z,2020-03-10T22:37:45Z,MEMBER,I should add an option to stop the moment you see a commit you have fetched before.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",578883725,Command for importing commits, https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-594155249,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12,594155249,MDEyOklzc3VlQ29tbWVudDU5NDE1NTI0OQ==,9599,simonw,2020-03-03T20:35:17Z,2020-03-03T20:35:17Z,MEMBER,"`swarm-to-sqlite` has an example of adding views here: https://github.com/dogsheep/swarm-to-sqlite/blob/f2c89dd613fb8a7f14e5267ccc2145463b996190/swarm_to_sqlite/utils.py#L141 I think that approach can be approved by first checking if the view exists, then dropping it, then recreating it. Could even try to see if the view exists and matches what we were going to set it to and do nothing if that is the case.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520756546,Add this view for seeing new releases, https://github.com/dogsheep/github-to-sqlite/pull/8#issuecomment-594154644,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/8,594154644,MDEyOklzc3VlQ29tbWVudDU5NDE1NDY0NA==,9599,simonw,2020-03-03T20:33:57Z,2020-03-03T20:33:57Z,MEMBER,`sqlite-utils` supports proper upserts now so this problem should be easy to fix.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",516763727,"stargazers command, refs #4", https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-594151327,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12,594151327,MDEyOklzc3VlQ29tbWVudDU5NDE1MTMyNw==,9599,simonw,2020-03-03T20:26:15Z,2020-03-03T20:32:23Z,MEMBER,"Better version (since this also includes JSON array of repository topics): ```sql CREATE VIEW recent_releases AS select repos.rowid as rowid, json_object(""label"", repos.full_name, ""href"", repos.html_url) as repo, json_object( ""href"", releases.html_url, ""label"", releases.name ) as release, substr(releases.published_at, 0, 11) as date, releases.body as body_markdown, releases.published_at, coalesce(repos.topics, '[]') as topics from releases join repos on repos.id = releases.repo order by releases.published_at desc ``` That `repos.rowid as rowid` bit is necessary because otherwise clicking on a link in facet-by-topic doesn't return any results.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520756546,Add this view for seeing new releases, https://github.com/dogsheep/github-to-sqlite/issues/14#issuecomment-559902818,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14,559902818,MDEyOklzc3VlQ29tbWVudDU1OTkwMjgxOA==,9599,simonw,2019-11-30T01:32:38Z,2019-11-30T01:32:38Z,MEMBER,"Prototype: ``` pip install sqlite-utils paginate-json paginate-json ""https://api.github.com/users/simonw/events"" | sqlite-utils insert /tmp/events.db events - --pk=id ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",530491074,Command for importing events, https://github.com/dogsheep/github-to-sqlite/issues/14#issuecomment-559883311,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14,559883311,MDEyOklzc3VlQ29tbWVudDU1OTg4MzMxMQ==,9599,simonw,2019-11-29T21:30:37Z,2019-11-29T21:30:37Z,MEMBER,"I should build the command to persist ETags and obey their polling guidelines: > Events are optimized for polling with the ""ETag"" header. If no new events have been triggered, you will see a ""304 Not Modified"" response, and your current rate limit will be untouched. There is also an ""X-Poll-Interval"" header that specifies how often (in seconds) you are allowed to poll. In times of high server load, the time may increase. Please obey the header.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",530491074,Command for importing events, https://github.com/dogsheep/twitter-to-sqlite/issues/31#issuecomment-552135263,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/31,552135263,MDEyOklzc3VlQ29tbWVudDU1MjEzNTI2Mw==,9599,simonw,2019-11-09T20:38:35Z,2019-11-09T20:38:35Z,MEMBER,Command still needs documentation and a bit more testing.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520508502,"""friends"" command (similar to ""followers"")", https://github.com/dogsheep/twitter-to-sqlite/issues/28#issuecomment-552133488,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/28,552133488,MDEyOklzc3VlQ29tbWVudDU1MjEzMzQ4OA==,9599,simonw,2019-11-09T20:15:42Z,2019-11-09T20:15:42Z,MEMBER,Released: https://github.com/dogsheep/twitter-to-sqlite/releases/tag/0.15,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",515658861,Add indexes to followers table, https://github.com/dogsheep/twitter-to-sqlite/issues/30#issuecomment-552133468,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/30,552133468,MDEyOklzc3VlQ29tbWVudDU1MjEzMzQ2OA==,9599,simonw,2019-11-09T20:15:27Z,2019-11-09T20:15:27Z,MEMBER,Released: https://github.com/dogsheep/twitter-to-sqlite/releases/tag/0.15,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",518739697,`followers` fails because `transform_user` is called twice, https://github.com/dogsheep/twitter-to-sqlite/issues/29#issuecomment-552133449,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/29,552133449,MDEyOklzc3VlQ29tbWVudDU1MjEzMzQ0OQ==,9599,simonw,2019-11-09T20:15:15Z,2019-11-09T20:15:15Z,MEMBER,Released: https://github.com/dogsheep/twitter-to-sqlite/releases/tag/0.15,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",518725064,`import` command fails on empty files, https://github.com/dogsheep/twitter-to-sqlite/issues/30#issuecomment-552131798,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/30,552131798,MDEyOklzc3VlQ29tbWVudDU1MjEzMTc5OA==,9599,simonw,2019-11-09T19:54:45Z,2019-11-09T19:54:45Z,MEMBER,Good catch - not sure how that bug crept in. Removing line 116 looks like the right fix to me.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",518739697,`followers` fails because `transform_user` is called twice, https://github.com/dogsheep/twitter-to-sqlite/issues/29#issuecomment-552129921,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/29,552129921,MDEyOklzc3VlQ29tbWVudDU1MjEyOTkyMQ==,9599,simonw,2019-11-09T19:30:42Z,2019-11-09T19:30:42Z,MEMBER,"Confirmed, that seems to fix it: ``` (twitter-to-sqlite) ~/Dropbox/Development/twitter-to-sqlite $ twitter-to-sqlite import blah.db ~/Dropbox/dogsheep/twitter-2019-06-25-b31f246100821b551f2f9a23f21ac6fb565dab49dd23a35630cabbf2b94a1f03/account-suspension.js Traceback (most recent call last): File ""/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/bin/twitter-to-sqlite"", line 11, in load_entry_point('twitter-to-sqlite', 'console_scripts', 'twitter-to-sqlite')() File ""/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py"", line 764, in __call__ return self.main(*args, **kwargs) File ""/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py"", line 717, in main rv = self.invoke(ctx) File ""/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py"", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File ""/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py"", line 956, in invoke return ctx.invoke(self.callback, **ctx.params) File ""/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py"", line 555, in invoke return callback(*args, **kwargs) File ""/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/cli.py"", line 633, in import_ archive.import_from_file(db, path.name, open(path, ""rb"").read()) File ""/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/archive.py"", line 224, in import_from_file db[table_name].upsert_all(rows, hash_id=""pk"") File ""/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/sqlite_utils/db.py"", line 1094, in upsert_all extracts=extracts, File ""/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/sqlite_utils/db.py"", line 961, in insert_all first_record = next(records) StopIteration (twitter-to-sqlite) ~/Dropbox/Development/twitter-to-sqlite $ pip install -U sqlite-utils Collecting sqlite-utils Using cached https://files.pythonhosted.org/packages/ee/a2/1b135010c7ac8e2d7545f659e9e6c6ede0f406f20b52e08d5817e1e31a9a/sqlite_utils-1.12.1-py3-none-any.whl Requirement already satisfied, skipping upgrade: click in /Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages (from sqlite-utils) (7.0) Requirement already satisfied, skipping upgrade: tabulate in /Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages (from sqlite-utils) (0.8.5) Requirement already satisfied, skipping upgrade: click-default-group in /Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages (from sqlite-utils) (1.2.2) Installing collected packages: sqlite-utils Found existing installation: sqlite-utils 1.11 Uninstalling sqlite-utils-1.11: Successfully uninstalled sqlite-utils-1.11 Successfully installed sqlite-utils-1.12.1 (twitter-to-sqlite) ~/Dropbox/Development/twitter-to-sqlite $ twitter-to-sqlite import blah.db ~/Dropbox/dogsheep/twitter-2019-06-25-b31f246100821b551f2f9a23f21ac6fb565dab49dd23a35630cabbf2b94a1f03/account-suspension.js (twitter-to-sqlite) ~/Dropbox/Development/twitter-to-sqlite $ ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",518725064,`import` command fails on empty files, https://github.com/dogsheep/twitter-to-sqlite/issues/29#issuecomment-552129686,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/29,552129686,MDEyOklzc3VlQ29tbWVudDU1MjEyOTY4Ng==,9599,simonw,2019-11-09T19:27:39Z,2019-11-09T19:27:39Z,MEMBER,I think this is fixed by the latest version of `sqlite-utils` - https://sqlite-utils.readthedocs.io/en/latest/changelog.html#v1-12-1 - I'll bump the dependency.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",518725064,`import` command fails on empty files, https://github.com/dogsheep/healthkit-to-sqlite/issues/10#issuecomment-550828084,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/10,550828084,MDEyOklzc3VlQ29tbWVudDU1MDgyODA4NA==,9599,simonw,2019-11-07T05:49:24Z,2019-11-07T05:49:24Z,MEMBER,"So the fix there is going to be to detect the new `FileReference` element and load the corresponding points data from it. This will be a little tricky because that function will need access to the zip file. It probably won't work at all for the mode where the `export.xml` file is passed directly using the `--xml` option.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",519038979,Failed to import workout points, https://github.com/dogsheep/healthkit-to-sqlite/issues/10#issuecomment-550824838,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/10,550824838,MDEyOklzc3VlQ29tbWVudDU1MDgyNDgzOA==,9599,simonw,2019-11-07T05:47:07Z,2019-11-07T05:47:07Z,MEMBER,"Relevant code: https://github.com/dogsheep/healthkit-to-sqlite/blob/d16f45f06fbae6ec8a78cc9ca7b5b7db0413f139/healthkit_to_sqlite/utils.py#L58-L64","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",519038979,Failed to import workout points, https://github.com/dogsheep/healthkit-to-sqlite/issues/10#issuecomment-550783316,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/10,550783316,MDEyOklzc3VlQ29tbWVudDU1MDc4MzMxNg==,9599,simonw,2019-11-07T05:16:56Z,2019-11-07T05:34:29Z,MEMBER,"It looks like Apple changed the location of these in iOS 13 - they are now in separate `.gpx` files: ![2FF70E95-CDEE-4241-A5C5-EE95A862E519](https://user-images.githubusercontent.com/9599/68362042-be12e000-00da-11ea-8925-7397410332d8.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",519038979,Failed to import workout points, https://github.com/dogsheep/healthkit-to-sqlite/issues/10#issuecomment-550806302,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/10,550806302,MDEyOklzc3VlQ29tbWVudDU1MDgwNjMwMg==,9599,simonw,2019-11-07T05:33:31Z,2019-11-07T05:33:31Z,MEMBER,"The XML now includes references to these new files: ![CBBA54FC-51FB-4BB3-927C-C2CA99237B04](https://user-images.githubusercontent.com/9599/68362716-121ec400-00dd-11ea-9846-387c7cd64c8b.jpeg) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",519038979,Failed to import workout points, https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-550388354,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4,550388354,MDEyOklzc3VlQ29tbWVudDU1MDM4ODM1NA==,9599,simonw,2019-11-06T16:26:55Z,2019-11-06T16:26:55Z,MEMBER,"Here's a query I figured out using a window function that shows cumulative stargazers over time: ```sql select yyyymmdd, sum(n) over ( order by yyyymmdd rows unbounded preceding ) as cumulative_count from ( select substr(starred_at, 0, 11) as yyyymmdd, count(*) as n from stars group by yyyymmdd ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",493670730,Command to fetch stargazers for one or more repos, https://github.com/dogsheep/github-to-sqlite/pull/8#issuecomment-549233778,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/8,549233778,MDEyOklzc3VlQ29tbWVudDU0OTIzMzc3OA==,9599,simonw,2019-11-04T06:14:40Z,2019-11-04T06:14:40Z,MEMBER,"Spotted a tricky problem: running `github-to-sqlite starred stargazers.db` results in an incomplete `simonw` record. It creates a proper record for me thanks to this bit: https://github.com/dogsheep/github-to-sqlite/blob/ea07274667a08c67907e8bfbbccb6f0fb95ce817/github_to_sqlite/cli.py#L120-L126 But then... when it gets to the `datasette` repository which I have starred it over-writes my full user record with one that's missing most of the details, thanks to this bit: https://github.com/dogsheep/github-to-sqlite/blob/ea07274667a08c67907e8bfbbccb6f0fb95ce817/github_to_sqlite/utils.py#L117-L124 I need to find a way of NOT over-writing a good record with a thinner one. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",516763727,"stargazers command, refs #4", https://github.com/dogsheep/github-to-sqlite/pull/8#issuecomment-549230583,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/8,549230583,MDEyOklzc3VlQ29tbWVudDU0OTIzMDU4Mw==,9599,simonw,2019-11-04T05:49:26Z,2019-11-04T05:49:26Z,MEMBER,Adding the view from #10 would be useful here too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",516763727,"stargazers command, refs #4", https://github.com/dogsheep/github-to-sqlite/issues/10#issuecomment-549230337,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/10,549230337,MDEyOklzc3VlQ29tbWVudDU0OTIzMDMzNw==,9599,simonw,2019-11-04T05:47:18Z,2019-11-04T05:47:18Z,MEMBER,This definition isn't quite right - it's not pulling the identity of the user who starred the repo (`users.login` ends up being the owner login instead).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",516967682,Add this repos_starred view, https://github.com/dogsheep/twitter-to-sqlite/issues/3#issuecomment-549228535,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3,549228535,MDEyOklzc3VlQ29tbWVudDU0OTIyODUzNQ==,9599,simonw,2019-11-04T05:31:55Z,2019-11-04T05:31:55Z,MEMBER,Documented here: https://github.com/dogsheep/twitter-to-sqlite/blob/801c0c2daf17d8abce9dcb5d8d610410e7e25dbe/README.md#running-searches,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",488833975,Command for running a search and saving tweets for that search, https://github.com/dogsheep/twitter-to-sqlite/issues/3#issuecomment-549226399,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3,549226399,MDEyOklzc3VlQ29tbWVudDU0OTIyNjM5OQ==,9599,simonw,2019-11-04T05:11:57Z,2019-11-04T05:11:57Z,MEMBER,I'm going to add a `hash` column to `search_runs` to support that. It's going to be the sha1 hash of the key-ordered JSON of the search arguments used by that run. Then `--since` can look for an identical hash and use it to identify the highest last fetched tweet to use in `since_id`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",488833975,Command for running a search and saving tweets for that search, https://github.com/dogsheep/twitter-to-sqlite/issues/3#issuecomment-549096321,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3,549096321,MDEyOklzc3VlQ29tbWVudDU0OTA5NjMyMQ==,9599,simonw,2019-11-03T01:27:55Z,2019-11-03T01:28:17Z,MEMBER,"It would be neat if this could support `--since`, with that argument automatically finding the maximum tweet ID from a previous search that used the same exact arguments (using the `search_runs` table).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",488833975,Command for running a search and saving tweets for that search, https://github.com/dogsheep/twitter-to-sqlite/issues/27#issuecomment-549095641,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/27,549095641,MDEyOklzc3VlQ29tbWVudDU0OTA5NTY0MQ==,9599,simonw,2019-11-03T01:12:58Z,2019-11-03T01:12:58Z,MEMBER,"It looks like Twitter really want you to subscribe to a premium API for this kind of thing and consume retweets via webhooks: https://developer.twitter.com/en/docs/accounts-and-users/subscribe-account-activity/api-reference I'm going to give up on this for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",514459062,retweets-of-me command, https://github.com/dogsheep/twitter-to-sqlite/issues/27#issuecomment-549095463,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/27,549095463,MDEyOklzc3VlQ29tbWVudDU0OTA5NTQ2Mw==,9599,simonw,2019-11-03T01:10:52Z,2019-11-03T01:10:52Z,MEMBER,"I imagine it won't, since the data I would be recording and then passing to `since_id` would be the highest ID of my own tweets that have been retweeted at least once. So it won't be able to spot if I should check for fresh retweets of a given tweet.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",514459062,retweets-of-me command, https://github.com/dogsheep/twitter-to-sqlite/issues/27#issuecomment-549095317,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/27,549095317,MDEyOklzc3VlQ29tbWVudDU0OTA5NTMxNw==,9599,simonw,2019-11-03T01:08:10Z,2019-11-03T01:08:10Z,MEMBER,"Hmm... one thing that could be useful is that `retweets_of_me` can support a `--since` parameter - so if run frequently it should hopefully let us know which tweets we would need to run `statuses/retweets/:id.json` against. I'm not sure if the `--since` parameter would show me a tweet that was previously retweeted but has now been retweeted again. I'll have a bit of a test and see.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",514459062,retweets-of-me command, https://github.com/dogsheep/twitter-to-sqlite/issues/27#issuecomment-549095217,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/27,549095217,MDEyOklzc3VlQ29tbWVudDU0OTA5NTIxNw==,9599,simonw,2019-11-03T01:06:25Z,2019-11-03T01:06:25Z,MEMBER,"Wow, that `retweets_of_me` endpoint is almost completely useless: ``` $ twitter-to-sqlite fetch https://api.twitter.com/1.1/statuses/retweets_of_me.json ``` It returns my own tweets that have been retweeted, but with no indication at all of who retweeted them. It looks like this needs to be combined with this API - https://developer.twitter.com/en/docs/tweets/post-and-engage/api-reference/get-statuses-retweets-id - to fetch the details of up to 100 recent users who actually DID retweet an individual status. But that has a one-every-12-seconds rate limit on it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",514459062,retweets-of-me command, https://github.com/dogsheep/github-to-sqlite/issues/5#issuecomment-549094229,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/5,549094229,MDEyOklzc3VlQ29tbWVudDU0OTA5NDIyOQ==,9599,simonw,2019-11-03T00:44:03Z,2019-11-03T00:44:03Z,MEMBER,Might not need an incomplete boolean - may be possible to handle this with `alter=True` and then by filtering for users with null values in certain columns.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",493671014,"Add ""incomplete"" boolean to users table for incomplete profiles", https://github.com/dogsheep/github-to-sqlite/pull/8#issuecomment-549094195,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/8,549094195,MDEyOklzc3VlQ29tbWVudDU0OTA5NDE5NQ==,9599,simonw,2019-11-03T00:43:16Z,2019-11-03T00:43:28Z,MEMBER,"Also need to take #5 into account - if this command creates incomplete user records, how do we repair them? And make sure that if we run this command first any future commands that populate users don't break (probably just a case of using `alter=True` in a few places).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",516763727,"stargazers command, refs #4", https://github.com/dogsheep/twitter-to-sqlite/issues/26#issuecomment-547713287,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/26,547713287,MDEyOklzc3VlQ29tbWVudDU0NzcxMzI4Nw==,9599,simonw,2019-10-30T02:36:13Z,2019-10-30T02:36:13Z,MEMBER,"Shipped this in 0.13: https://github.com/dogsheep/twitter-to-sqlite/releases/tag/0.13 See also this Twitter thread: https://twitter.com/simonw/status/1189369677509623809","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",513074501,Command for importing mentions timeline, https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-544648863,https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1,544648863,MDEyOklzc3VlQ29tbWVudDU0NDY0ODg2Mw==,9599,simonw,2019-10-21T18:36:03Z,2019-10-21T18:36:03Z,MEMBER," ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",496415321,Figure out some interesting example SQL queries, https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-544646516,https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1,544646516,MDEyOklzc3VlQ29tbWVudDU0NDY0NjUxNg==,9599,simonw,2019-10-21T18:30:14Z,2019-10-21T18:30:14Z,MEMBER,"Thanks to help from Dr. Laura Cantino at Science Hack Day San Francisco I've been able to pull together this query: ```sql select rsid, genotype, case genotype when 'AA' then 'brown eye color, 80% of the time' when 'AG' then 'brown eye color' when 'GG' then 'blue eye color, 99% of the time' end as interpretation from genome where rsid = 'rs12913832' ``` See also https://www.snpedia.com/index.php/Rs12913832 - in particular this table: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",496415321,Figure out some interesting example SQL queries, https://github.com/dogsheep/twitter-to-sqlite/issues/20#issuecomment-544335363,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/20,544335363,MDEyOklzc3VlQ29tbWVudDU0NDMzNTM2Mw==,9599,simonw,2019-10-21T03:32:04Z,2019-10-21T03:32:04Z,MEMBER,"In case anyone is interested, here's an extract from the crontab I'm running these under at the moment: ``` 1,11,21,31,41,51 * * * * /home/ubuntu/datasette-venv/bin/twitter-to-sqlite user-timeline /home/ubuntu/twitter.db -a /home/ubuntu/auth.json --since 2,7,12,17,22,27,32,37,42,47,52,57 * * * * /home/ubuntu/datasette-venv/bin/twitter-to-sqlite home-timeline /home/ubuntu/timeline.db -a /home/ubuntu/auth.json --since 6,16,26,36,46,56 * * * * /home/ubuntu/datasette-venv/bin/twitter-to-sqlite favorites /home/ubuntu/twitter.db -a /home/ubuntu/auth.json --stop_after=50 ```","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",506268945,--since support for various commands for refresh-by-cron, https://github.com/dogsheep/twitter-to-sqlite/issues/3#issuecomment-543290744,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3,543290744,MDEyOklzc3VlQ29tbWVudDU0MzI5MDc0NA==,9599,simonw,2019-10-17T17:57:14Z,2019-10-17T17:57:14Z,MEMBER,I have a working command now. I'm going to ship it early because it could do with some other people trying it out.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",488833975,Command for running a search and saving tweets for that search, https://github.com/dogsheep/twitter-to-sqlite/issues/3#issuecomment-543273540,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3,543273540,MDEyOklzc3VlQ29tbWVudDU0MzI3MzU0MA==,9599,simonw,2019-10-17T17:12:51Z,2019-10-17T17:12:51Z,MEMBER,"Just importing tweets here isn't enough - how are we supposed to know which tweets were imported by which search? So I think the right thing to do here is to also create a `search_runs` table, which records each individual run of this tool (with a timestamp and the search terms used). Then have a `search_runs_tweets` m2m table which shows which Tweets were found by that search.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",488833975,Command for running a search and saving tweets for that search, https://github.com/dogsheep/twitter-to-sqlite/issues/10#issuecomment-543271000,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10,543271000,MDEyOklzc3VlQ29tbWVudDU0MzI3MTAwMA==,9599,simonw,2019-10-17T17:05:59Z,2019-10-17T17:05:59Z,MEMBER,Looks like tqdm already does a TTY check here: https://github.com/tqdm/tqdm/blob/89b73bdc30c099c5b53725806e7edf3a121c9b3a/tqdm/std.py#L889-L890,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",492297930,Rethink progress bars for various commands, https://github.com/dogsheep/twitter-to-sqlite/issues/10#issuecomment-543270714,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10,543270714,MDEyOklzc3VlQ29tbWVudDU0MzI3MDcxNA==,9599,simonw,2019-10-17T17:05:16Z,2019-10-17T17:05:16Z,MEMBER,"https://github.com/pallets/click/blob/716a5be90f56ce6cd506bb53d5739d09374b1636/click/_termui_impl.py#L93 is how Click does this: ``` self.is_hidden = not isatty(self.file) ``` Where `isatty` is a Click utility function: `from ._compat import isatty`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",492297930,Rethink progress bars for various commands, https://github.com/dogsheep/twitter-to-sqlite/issues/10#issuecomment-543269396,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10,543269396,MDEyOklzc3VlQ29tbWVudDU0MzI2OTM5Ng==,9599,simonw,2019-10-17T17:02:07Z,2019-10-17T17:02:07Z,MEMBER,A neat trick that Click does is detecting if an interactive terminal is attached and NOT showing a progress bar if there isn't one. Need to figure out how to do that with tqdm.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",492297930,Rethink progress bars for various commands, https://github.com/dogsheep/twitter-to-sqlite/issues/25#issuecomment-543266947,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/25,543266947,MDEyOklzc3VlQ29tbWVudDU0MzI2Njk0Nw==,9599,simonw,2019-10-17T16:56:06Z,2019-10-17T16:56:06Z,MEMBER,I wrote a test that proves that this is a problem. Should be an easy fix though.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",508578780,Ensure migrations don't accidentally create foreign key twice, https://github.com/dogsheep/twitter-to-sqlite/issues/25#issuecomment-543265058,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/25,543265058,MDEyOklzc3VlQ29tbWVudDU0MzI2NTA1OA==,9599,simonw,2019-10-17T16:51:12Z,2019-10-17T16:51:12Z,MEMBER,"This migration function only runs if there is a table called `tweets` and the migration has not run before. I think this can happen if the database has just been freshly created (by a command that fetches the user's user timeline for example) and is then run a SECOND time.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",508578780,Ensure migrations don't accidentally create foreign key twice, https://github.com/dogsheep/twitter-to-sqlite/issues/23#issuecomment-543222239,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/23,543222239,MDEyOklzc3VlQ29tbWVudDU0MzIyMjIzOQ==,9599,simonw,2019-10-17T15:12:33Z,2019-10-17T15:12:33Z,MEMBER,"Migrations will run only if you open a database that previously existed (as opposed to opening a brand new empty database). This means that the first time you run a command against a fresh database, migrations will not run and the `migrations` table will not be created. The _second_ time you run any command against that database the migrations will execute and populate the `migrations` table. This also means that each migration needs to be able to sanity check the database to see if it should run or not. If it should NOT run, it will do nothing but still be marked as having executed by adding to the `migrations` table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",508190730,Extremely simple migration system, https://github.com/dogsheep/twitter-to-sqlite/issues/23#issuecomment-543217890,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/23,543217890,MDEyOklzc3VlQ29tbWVudDU0MzIxNzg5MA==,9599,simonw,2019-10-17T15:03:10Z,2019-10-17T15:03:10Z,MEMBER,"Thinking about this further: the concept of migrations may end up being in direct conflict with the `sqlite-utils` concept of creating tables on demand the first time they are used - and of creating table schemas automatically to fit the shape of the JSON that is being inserted into them. I'm going to forge ahead anyway and build this because I think it will be an interesting exploration, but it's very likely this turns out to be a bad idea in the long run!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",508190730,Extremely simple migration system, https://github.com/dogsheep/swarm-to-sqlite/issues/3#issuecomment-542882604,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/3,542882604,MDEyOklzc3VlQ29tbWVudDU0Mjg4MjYwNA==,9599,simonw,2019-10-16T20:41:23Z,2019-10-16T20:41:23Z,MEMBER,Documented here: https://github.com/dogsheep/swarm-to-sqlite/blob/0.2/README.md#usage,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",487600595,Option to fetch only checkins more recent than the current max checkin, https://github.com/dogsheep/swarm-to-sqlite/issues/3#issuecomment-542876047,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/3,542876047,MDEyOklzc3VlQ29tbWVudDU0Mjg3NjA0Nw==,9599,simonw,2019-10-16T20:23:36Z,2019-10-16T20:23:36Z,MEMBER,I'm going to go with `--since=1d/2w/3h` for this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",487600595,Option to fetch only checkins more recent than the current max checkin, https://github.com/dogsheep/swarm-to-sqlite/issues/3#issuecomment-542875885,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/3,542875885,MDEyOklzc3VlQ29tbWVudDU0Mjg3NTg4NQ==,9599,simonw,2019-10-16T20:23:08Z,2019-10-16T20:23:08Z,MEMBER,"https://developer.foursquare.com/docs/api/users/checkins documents `afterTimestamp`: > Retrieve the first results to follow these seconds since epoch. This should be useful for paging forward in time, or when polling for changes. To avoid missing results when polling, we recommend subtracting several seconds from the last poll time and then de-duplicating.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",487600595,Option to fetch only checkins more recent than the current max checkin, https://github.com/dogsheep/twitter-to-sqlite/issues/12#issuecomment-542858025,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/12,542858025,MDEyOklzc3VlQ29tbWVudDU0Mjg1ODAyNQ==,9599,simonw,2019-10-16T19:35:31Z,2019-10-16T19:36:09Z,MEMBER,Maybe this means I need an `upgrade` command to apply these kinds of migrations? Total feature creep!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503053800,"Extract ""source"" into a separate lookup table", https://github.com/dogsheep/twitter-to-sqlite/issues/12#issuecomment-542855427,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/12,542855427,MDEyOklzc3VlQ29tbWVudDU0Mjg1NTQyNw==,9599,simonw,2019-10-16T19:27:55Z,2019-10-16T19:27:55Z,MEMBER,I can do that by keeping `source` as a `TEXT` column but turning it into a non-enforced foreign key against a new `sources` table. Then I can run code that scans that column for any values beginning with a `<` and converts them.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503053800,"Extract ""source"" into a separate lookup table", https://github.com/dogsheep/twitter-to-sqlite/issues/12#issuecomment-542855081,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/12,542855081,MDEyOklzc3VlQ29tbWVudDU0Mjg1NTA4MQ==,9599,simonw,2019-10-16T19:26:56Z,2019-10-16T19:26:56Z,MEMBER,This may be the first case where I want to be able to repair existing databases rather than discarding their contents.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503053800,"Extract ""source"" into a separate lookup table", https://github.com/dogsheep/twitter-to-sqlite/issues/20#issuecomment-542854749,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/20,542854749,MDEyOklzc3VlQ29tbWVudDU0Mjg1NDc0OQ==,9599,simonw,2019-10-16T19:26:01Z,2019-10-16T19:26:01Z,MEMBER,"I'm not going to do this for ""accounts that have followed me"" and ""new accounts that I have followed"" - instead I will recommend running the `friend_ids` and `followers_ids` commands on a daily basis since that data doesn't really change much by the hour. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",506268945,--since support for various commands for refresh-by-cron, https://github.com/dogsheep/twitter-to-sqlite/issues/19#issuecomment-542849963,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/19,542849963,MDEyOklzc3VlQ29tbWVudDU0Mjg0OTk2Mw==,9599,simonw,2019-10-16T19:13:06Z,2019-10-16T19:13:06Z,MEMBER,Updated documentation: https://github.com/dogsheep/twitter-to-sqlite/blob/fced2a9b67d2cbdf9817f1eb75f7c28e413c963b/README.md#retrieving-tweets-from-your-home-timeline,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",506087267,since_id support for home-timeline, https://github.com/dogsheep/twitter-to-sqlite/issues/19#issuecomment-542832952,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/19,542832952,MDEyOklzc3VlQ29tbWVudDU0MjgzMjk1Mg==,9599,simonw,2019-10-16T18:30:11Z,2019-10-16T18:30:11Z,MEMBER,"The `--since` option will derive the `since_id` from the max ID in the `timeline_tweets` table: $ twitter-to-sqlite home-timeline --since The `--since_id=xxx` option lets you specify that ID directly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",506087267,since_id support for home-timeline, https://github.com/dogsheep/twitter-to-sqlite/issues/21#issuecomment-542333836,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/21,542333836,MDEyOklzc3VlQ29tbWVudDU0MjMzMzgzNg==,9599,simonw,2019-10-15T18:00:48Z,2019-10-15T18:00:48Z,MEMBER,I'll use `html.unescape()` for this: https://docs.python.org/3/library/html.html#html.unescape,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",506432572,Fix & escapes in tweet text, https://github.com/dogsheep/twitter-to-sqlite/issues/10#issuecomment-541748580,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10,541748580,MDEyOklzc3VlQ29tbWVudDU0MTc0ODU4MA==,9599,simonw,2019-10-14T15:30:44Z,2019-10-14T15:30:44Z,MEMBER,Had several recommendations for https://github.com/tqdm/tqdm which is what goodreads-to-sqlite uses.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",492297930,Rethink progress bars for various commands, https://github.com/dogsheep/github-to-sqlite/issues/7#issuecomment-541721437,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/7,541721437,MDEyOklzc3VlQ29tbWVudDU0MTcyMTQzNw==,9599,simonw,2019-10-14T14:44:12Z,2019-10-14T14:44:12Z,MEMBER,Docs: https://github.com/dogsheep/github-to-sqlite/blob/0.5/README.md#retrieving-issue-comments-for-a-repository,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",506276893,issue-comments command for importing issue comments, https://github.com/dogsheep/twitter-to-sqlite/issues/3#issuecomment-541493242,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3,541493242,MDEyOklzc3VlQ29tbWVudDU0MTQ5MzI0Mg==,9599,simonw,2019-10-14T03:35:36Z,2019-10-14T03:35:36Z,MEMBER,"https://developer.twitter.com/en/docs/tweets/search/api-reference/get-search-tweets ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",488833975,Command for running a search and saving tweets for that search, https://github.com/dogsheep/twitter-to-sqlite/issues/20#issuecomment-541388038,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/20,541388038,MDEyOklzc3VlQ29tbWVudDU0MTM4ODAzOA==,9599,simonw,2019-10-13T05:31:58Z,2019-10-13T05:31:58Z,MEMBER,For favourites a `--stop_after=200` option is probably good enough.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",506268945,--since support for various commands for refresh-by-cron, https://github.com/dogsheep/github-to-sqlite/issues/6#issuecomment-541387941,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/6,541387941,MDEyOklzc3VlQ29tbWVudDU0MTM4Nzk0MQ==,9599,simonw,2019-10-13T05:30:19Z,2019-10-13T05:30:19Z,MEMBER,Fix released in 0.5: https://github.com/dogsheep/github-to-sqlite/releases/tag/0.5,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",504238461,sqlite3.OperationalError: table users has no column named bio, https://github.com/dogsheep/github-to-sqlite/issues/6#issuecomment-541387822,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/6,541387822,MDEyOklzc3VlQ29tbWVudDU0MTM4NzgyMg==,9599,simonw,2019-10-13T05:27:39Z,2019-10-13T05:27:39Z,MEMBER,This should be fixed by https://github.com/dogsheep/github-to-sqlite/commit/552543a74970f8a3a3f87f887be23a0c6eb1cb5b,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",504238461,sqlite3.OperationalError: table users has no column named bio, https://github.com/dogsheep/twitter-to-sqlite/issues/19#issuecomment-541248629,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/19,541248629,MDEyOklzc3VlQ29tbWVudDU0MTI0ODYyOQ==,9599,simonw,2019-10-11T22:48:56Z,2019-10-11T22:48:56Z,MEMBER,"`since_id` documented here: https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-home_timeline > Returns results with an ID greater than (that is, more recent than) the specified ID. There are limits to the number of Tweets which can be accessed through the API. If the limit of Tweets has occured since the since_id, the since_id will be forced to the oldest ID available.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",506087267,since_id support for home-timeline, https://github.com/dogsheep/twitter-to-sqlite/issues/18#issuecomment-541119834,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/18,541119834,MDEyOklzc3VlQ29tbWVudDU0MTExOTgzNA==,9599,simonw,2019-10-11T15:51:22Z,2019-10-11T16:51:33Z,MEMBER,"In order to support multiple user timelines being saved in the same database, I'm going to import the tweets into the `tweets` table AND add a new `timeline_tweets` table recording that a specific tweet showed up in a specific user's timeline.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",505928530,Command to import home-timeline, https://github.com/dogsheep/twitter-to-sqlite/issues/18#issuecomment-541141169,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/18,541141169,MDEyOklzc3VlQ29tbWVudDU0MTE0MTE2OQ==,9599,simonw,2019-10-11T16:51:29Z,2019-10-11T16:51:29Z,MEMBER,Documented here: https://github.com/dogsheep/twitter-to-sqlite/blob/master/README.md#retrieving-tweets-from-your-home-timeline,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",505928530,Command to import home-timeline, https://github.com/dogsheep/twitter-to-sqlite/issues/18#issuecomment-541118934,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/18,541118934,MDEyOklzc3VlQ29tbWVudDU0MTExODkzNA==,9599,simonw,2019-10-11T15:48:54Z,2019-10-11T15:48:54Z,MEMBER,Rate limit is tight: 15 requests every 15 mins!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",505928530,Command to import home-timeline, https://github.com/dogsheep/twitter-to-sqlite/issues/18#issuecomment-541118773,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/18,541118773,MDEyOklzc3VlQ29tbWVudDU0MTExODc3Mw==,9599,simonw,2019-10-11T15:48:31Z,2019-10-11T15:48:31Z,MEMBER,https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-home_timeline,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",505928530,Command to import home-timeline, https://github.com/dogsheep/twitter-to-sqlite/issues/17#issuecomment-541112588,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/17,541112588,MDEyOklzc3VlQ29tbWVudDU0MTExMjU4OA==,9599,simonw,2019-10-11T15:31:30Z,2019-10-11T15:31:30Z,MEMBER,"No need for an option: > This command will delete and recreate all of your `archive-*` tables every time you run it. If this is not what you want, run the command against a fresh SQLite database rather than running it again one that already exists.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",505674949,import command should empty all archive-* tables first, https://github.com/dogsheep/twitter-to-sqlite/issues/17#issuecomment-541112108,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/17,541112108,MDEyOklzc3VlQ29tbWVudDU0MTExMjEwOA==,9599,simonw,2019-10-11T15:30:15Z,2019-10-11T15:30:15Z,MEMBER,It should delete the tables entirely. That way it will work even if the table schema has changed.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",505674949,import command should empty all archive-* tables first, https://github.com/dogsheep/twitter-to-sqlite/issues/4#issuecomment-540879620,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/4,540879620,MDEyOklzc3VlQ29tbWVudDU0MDg3OTYyMA==,9599,simonw,2019-10-11T02:59:16Z,2019-10-11T02:59:16Z,MEMBER,Also import ad preferences and all that other junk.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",488835586,Command for importing data from a Twitter Export file, https://github.com/dogsheep/pocket-to-sqlite/issues/2#issuecomment-538847796,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/2,538847796,MDEyOklzc3VlQ29tbWVudDUzODg0Nzc5Ng==,9599,simonw,2019-10-07T05:43:30Z,2019-10-07T05:43:30Z,MEMBER,We can persist the `since` value in its own single-row table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503234169,Track and use the 'since' value, https://github.com/dogsheep/pocket-to-sqlite/issues/1#issuecomment-538847446,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/1,538847446,MDEyOklzc3VlQ29tbWVudDUzODg0NzQ0Ng==,9599,simonw,2019-10-07T05:41:17Z,2019-10-07T05:41:17Z,MEMBER,"Prototype code: ```python offset = 0 fetched = [] size = 500 while True: page = requests.get(""https://getpocket.com/v3/get"", { ""consumer_key"": consumer_key, ""access_token"": access_token, ""sort"": ""oldest"", ""detailType"": ""complete"", ""count"": size, ""offset"": offset, }).json() print(offset) fetched.append(page) offset += size if not len(page[""list""]): break ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503233021,Use better pagination (and implement progress bar), https://github.com/dogsheep/twitter-to-sqlite/issues/13#issuecomment-538804815,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/13,538804815,MDEyOklzc3VlQ29tbWVudDUzODgwNDgxNQ==,9599,simonw,2019-10-07T00:33:49Z,2019-10-07T00:33:49Z,MEMBER,Documentation: https://github.com/dogsheep/twitter-to-sqlite#retrieve-tweets-in-bulk,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503085013,statuses-lookup command, https://github.com/dogsheep/twitter-to-sqlite/issues/11#issuecomment-538711918,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/11,538711918,MDEyOklzc3VlQ29tbWVudDUzODcxMTkxOA==,9599,simonw,2019-10-06T04:54:17Z,2019-10-06T04:54:17Z,MEMBER,Shipped in 0.6. Here's the documentation: https://github.com/dogsheep/twitter-to-sqlite#capturing-tweets-in-real-time-with-track-and-follow,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503045221,Commands for recording real-time tweets from the streaming API, https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-531517138,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4,531517138,MDEyOklzc3VlQ29tbWVudDUzMTUxNzEzOA==,9599,simonw,2019-09-14T21:59:59Z,2019-09-14T21:59:59Z,MEMBER,"Paginate through https://api.github.com/repos/simonw/datasette/stargazers Send `Accept: application/vnd.github.v3.star+json` to get the `starred_at` dates.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",493670730,Command to fetch stargazers for one or more repos, https://github.com/dogsheep/github-to-sqlite/issues/3#issuecomment-531517083,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/3,531517083,MDEyOklzc3VlQ29tbWVudDUzMTUxNzA4Mw==,9599,simonw,2019-09-14T21:58:42Z,2019-09-14T21:58:42Z,MEMBER,Split stargazers into #4,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",493670426,Command to fetch all repos belonging to a user or organization, https://github.com/dogsheep/github-to-sqlite/issues/3#issuecomment-531516956,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/3,531516956,MDEyOklzc3VlQ29tbWVudDUzMTUxNjk1Ng==,9599,simonw,2019-09-14T21:56:31Z,2019-09-14T21:56:31Z,MEMBER,"https://api.github.com/users/simonw/repos It would be useful to be able to fetch stargazers, forks etc as well. Not sure if that should be a separate command or a `--stargazers` option to this command. Probably a separate command since `issues` is a separate command already.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",493670426,Command to fetch all repos belonging to a user or organization, https://github.com/dogsheep/twitter-to-sqlite/issues/8#issuecomment-530417631,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/8,530417631,MDEyOklzc3VlQ29tbWVudDUzMDQxNzYzMQ==,9599,simonw,2019-09-11T14:52:44Z,2019-09-14T19:09:22Z,MEMBER,- [x] This needs documentation.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",490803176,--sql and --attach options for feeding commands from SQL queries, https://github.com/dogsheep/twitter-to-sqlite/issues/8#issuecomment-531404891,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/8,531404891,MDEyOklzc3VlQ29tbWVudDUzMTQwNDg5MQ==,9599,simonw,2019-09-13T22:01:57Z,2019-09-13T22:01:57Z,MEMBER,I also wrote about this in https://simonwillison.net/2019/Sep/13/weeknotestwitter-sqlite-datasette-rure/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",490803176,--sql and --attach options for feeding commands from SQL queries, https://github.com/dogsheep/twitter-to-sqlite/issues/9#issuecomment-530028567,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/9,530028567,MDEyOklzc3VlQ29tbWVudDUzMDAyODU2Nw==,9599,simonw,2019-09-10T16:59:25Z,2019-09-10T16:59:25Z,MEMBER,"By default in SQLite foreign key constraints are not enforced (you need to run `PRAGMA foreign_keys = ON;` to enforce them). We will take advantage of this - even though the `following` table has foreign keys against user we will allow IDs to populate that table without a corresponding user record. In the future we may add a command that can backfill missing user records.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",491791152,followers-ids and friends-ids subcommands, https://github.com/dogsheep/twitter-to-sqlite/issues/8#issuecomment-529240286,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/8,529240286,MDEyOklzc3VlQ29tbWVudDUyOTI0MDI4Ng==,9599,simonw,2019-09-08T20:48:33Z,2019-09-08T20:48:33Z,MEMBER,"```ATTACH DATABASE ""file:blah.db?mode=ro"" AS foo```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",490803176,--sql and --attach options for feeding commands from SQL queries, https://github.com/dogsheep/twitter-to-sqlite/issues/8#issuecomment-529239307,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/8,529239307,MDEyOklzc3VlQ29tbWVudDUyOTIzOTMwNw==,9599,simonw,2019-09-08T20:36:49Z,2019-09-08T20:36:49Z,MEMBER,"`--attach` can optionally take a name for the database connection alias like this: $ twitter-to-sqlite users-lookup users.db --attach foo:attending.db ... If you omit the `alias:` bit the stem of the database (without the file extension) will be used.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",490803176,--sql and --attach options for feeding commands from SQL queries, https://github.com/dogsheep/twitter-to-sqlite/issues/2#issuecomment-527990908,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/2,527990908,MDEyOklzc3VlQ29tbWVudDUyNzk5MDkwOA==,9599,simonw,2019-09-04T16:57:24Z,2019-09-04T16:57:24Z,MEMBER,"I just tried this using `max_id=` pagination as described in [Working with timelines](https://developer.twitter.com/en/docs/tweets/timelines/guides/working-with-timelines) and I got back all 17,759 of my tweets.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",488833698,"""twitter-to-sqlite user-timeline"" command for pulling tweets by a specific user", https://github.com/dogsheep/twitter-to-sqlite/issues/2#issuecomment-527955302,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/2,527955302,MDEyOklzc3VlQ29tbWVudDUyNzk1NTMwMg==,9599,simonw,2019-09-04T15:32:39Z,2019-09-04T15:32:39Z,MEMBER,Rate limit is 900 / 15 minutes which is 1 call per second.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",488833698,"""twitter-to-sqlite user-timeline"" command for pulling tweets by a specific user", https://github.com/dogsheep/twitter-to-sqlite/issues/2#issuecomment-527954898,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/2,527954898,MDEyOklzc3VlQ29tbWVudDUyNzk1NDg5OA==,9599,simonw,2019-09-04T15:31:46Z,2019-09-04T15:31:46Z,MEMBER,I'm going to call this `twitter-to-sqlite user-timeline` to reflect the language used to describe the API endpoint: https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-user_timeline.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",488833698,"""twitter-to-sqlite user-timeline"" command for pulling tweets by a specific user", https://github.com/dogsheep/twitter-to-sqlite/issues/5#issuecomment-527684202,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/5,527684202,MDEyOklzc3VlQ29tbWVudDUyNzY4NDIwMg==,9599,simonw,2019-09-03T23:56:28Z,2019-09-03T23:56:28Z,MEMBER,I previously used betamax here: https://github.com/simonw/github-contents/blob/master/test_github_contents.py,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",488874815,Write tests that simulate the Twitter API, https://github.com/dogsheep/twitter-to-sqlite/issues/4#issuecomment-527682713,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/4,527682713,MDEyOklzc3VlQ29tbWVudDUyNzY4MjcxMw==,9599,simonw,2019-09-03T23:48:57Z,2019-09-03T23:48:57Z,MEMBER,"One interesting challenge here is that the JSON format for tweets in the archive is subtly different from the JSON format currently returned by the API. If we want to keep the tweets in the same database table (which feels like the right thing to me) we'll need to handle this. One thing we can do is have a column for `from_archive` which is set to 1 for tweets that were recovered from the archive. We can also ensure that tweets from the API always over-write the version that came from the archive (using `.upsert()`) while tweets from the archive use `.insert(..., ignore=True)` to avoid over-writing a better version that came from the API.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",488835586,Command for importing data from a Twitter Export file, https://github.com/dogsheep/swarm-to-sqlite/issues/3#issuecomment-527200332,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/3,527200332,MDEyOklzc3VlQ29tbWVudDUyNzIwMDMzMg==,9599,simonw,2019-09-02T16:32:20Z,2019-09-02T16:32:39Z,MEMBER,"Also needed: an option for ""fetch all checkins created within the last X days"". This should help provide support for that Swarm feature where you can retroactively checkin to places in the past.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",487600595,Option to fetch only checkins more recent than the current max checkin, https://github.com/dogsheep/swarm-to-sqlite/issues/4#issuecomment-526853542,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/4,526853542,MDEyOklzc3VlQ29tbWVudDUyNjg1MzU0Mg==,9599,simonw,2019-08-31T18:06:32Z,2019-08-31T18:06:32Z,MEMBER,"https://your-foursquare-oauth-token.glitch.me/ Source code: https://glitch.com/~your-foursquare-oauth-token","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",487601121,Online tool for getting a Foursquare OAuth token, https://github.com/dogsheep/swarm-to-sqlite/issues/2#issuecomment-526701674,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/2,526701674,MDEyOklzc3VlQ29tbWVudDUyNjcwMTY3NA==,9599,simonw,2019-08-30T18:24:26Z,2019-08-30T18:24:26Z,MEMBER,I renamed `--file` to `--load` in 0e5b6025c6f9823ff81aa8aae1cbff5c45e57baf,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",487598468,--save option to dump checkins to a JSON file on disk, https://github.com/dogsheep/healthkit-to-sqlite/issues/9#issuecomment-515322294,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/9,515322294,MDEyOklzc3VlQ29tbWVudDUxNTMyMjI5NA==,9599,simonw,2019-07-26T06:07:12Z,2019-07-26T06:07:12Z,MEMBER,@tholo this should be fixed in just-released version 0.3.2 - could you run a `pip install -U healthkit-to-sqlite` and let me know if it works for you now?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472429048,Too many SQL variables, https://github.com/dogsheep/healthkit-to-sqlite/issues/9#issuecomment-515226724,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/9,515226724,MDEyOklzc3VlQ29tbWVudDUxNTIyNjcyNA==,9599,simonw,2019-07-25T21:46:01Z,2019-07-25T21:46:01Z,MEMBER,I can work around this here (prior to the fix in sqlite-utils) by setting the batch size to something a bit lower here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472429048,Too many SQL variables, https://github.com/dogsheep/healthkit-to-sqlite/issues/7#issuecomment-514500253,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/7,514500253,MDEyOklzc3VlQ29tbWVudDUxNDUwMDI1Mw==,9599,simonw,2019-07-24T06:34:28Z,2019-07-24T06:34:28Z,MEMBER,"Clearing the root element each time saved even more: ","{""total_count"": 2, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 2, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472097220,Script uses a lot of RAM, https://github.com/dogsheep/healthkit-to-sqlite/issues/7#issuecomment-514498221,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/7,514498221,MDEyOklzc3VlQ29tbWVudDUxNDQ5ODIyMQ==,9599,simonw,2019-07-24T06:26:49Z,2019-07-24T06:26:49Z,MEMBER,"Adding `el.clear()` got me a huge improvement: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472097220,Script uses a lot of RAM, https://github.com/dogsheep/healthkit-to-sqlite/issues/7#issuecomment-514496725,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/7,514496725,MDEyOklzc3VlQ29tbWVudDUxNDQ5NjcyNQ==,9599,simonw,2019-07-24T06:20:59Z,2019-07-24T06:20:59Z,MEMBER,"I'm using https://pypi.org/project/memory-profiler/ to explore this in more detail: ``` $ pip install memory-profiler matplotlib ``` Then: ``` $ mprof run healthkit-to-sqlite ~/Downloads/healthkit-export.zip healthkit.db $ mprof plot ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472097220,Script uses a lot of RAM, https://github.com/dogsheep/healthkit-to-sqlite/issues/6#issuecomment-513626742,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/6,513626742,MDEyOklzc3VlQ29tbWVudDUxMzYyNjc0Mg==,9599,simonw,2019-07-22T03:28:55Z,2019-07-22T03:28:55Z,MEMBER,"Here's what it looks like now as separate tables: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470856782,Break up records into different tables for each type, https://github.com/dogsheep/healthkit-to-sqlite/issues/5#issuecomment-513625406,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/5,513625406,MDEyOklzc3VlQ29tbWVudDUxMzYyNTQwNg==,9599,simonw,2019-07-22T03:20:16Z,2019-07-22T03:20:16Z,MEMBER,"It now renders like this: ``` Importing from HealthKit [#-----------------------------------] 5% 00:01:33 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470691622,Add progress bar, https://github.com/dogsheep/healthkit-to-sqlite/issues/5#issuecomment-513514978,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/5,513514978,MDEyOklzc3VlQ29tbWVudDUxMzUxNDk3OA==,9599,simonw,2019-07-21T02:55:12Z,2019-07-21T02:55:12Z,MEMBER,I'm going to show this by default. Users can pass `-s` or `--silent` to disable the progress bar.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470691622,Add progress bar, https://github.com/dogsheep/healthkit-to-sqlite/issues/4#issuecomment-513440090,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/4,513440090,MDEyOklzc3VlQ29tbWVudDUxMzQ0MDA5MA==,9599,simonw,2019-07-20T06:11:50Z,2019-07-20T06:11:50Z,MEMBER,"Some examples: https://github.com/dogsheep/healthkit-to-sqlite/blob/d016e70c31cf84ba0f5ec3102546db54a51aaffb/tests/export.xml#L4-L13","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470640505,Import Records, https://github.com/dogsheep/healthkit-to-sqlite/issues/2#issuecomment-513439411,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/2,513439411,MDEyOklzc3VlQ29tbWVudDUxMzQzOTQxMQ==,9599,simonw,2019-07-20T05:58:57Z,2019-07-20T05:58:57Z,MEMBER,"```python 'Workout': {'attr_counts': {'creationDate': 663, 'device': 230, 'duration': 663, 'durationUnit': 663, 'endDate': 663, 'sourceName': 663, 'sourceVersion': 663, 'startDate': 663, 'totalDistance': 663, 'totalDistanceUnit': 663, 'totalEnergyBurned': 663, 'totalEnergyBurnedUnit': 663, 'workoutActivityType': 663}, 'child_counts': {'MetadataEntry': 1928, 'WorkoutEvent': 2094, 'WorkoutRoute': 340}, 'count': 663, 'parent_counts': {'HealthData': 663}}, 'WorkoutEvent': {'attr_counts': {'date': 2094, 'duration': 837, 'durationUnit': 837, 'type': 2094}, 'child_counts': {}, 'count': 2094, 'parent_counts': {'Workout': 2094}}, 'WorkoutRoute': {'attr_counts': {'creationDate': 340, 'endDate': 340, 'sourceName': 340, 'sourceVersion': 340, 'startDate': 340}, 'child_counts': {'Location': 398683, 'MetadataEntry': 546}, 'count': 340, 'parent_counts': {'Workout': 340}}} ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470637152,Import workouts, https://github.com/dogsheep/healthkit-to-sqlite/issues/1#issuecomment-513437463,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/1,513437463,MDEyOklzc3VlQ29tbWVudDUxMzQzNzQ2Mw==,9599,simonw,2019-07-20T05:19:59Z,2019-07-20T05:19:59Z,MEMBER,"I ran xml_analyser against the XML HealthKit `export.xml` file and got the following results: ```python { 'ActivitySummary': {'attr_counts': {'activeEnergyBurned': 980, 'activeEnergyBurnedGoal': 980, 'activeEnergyBurnedUnit': 980, 'appleExerciseTime': 980, 'appleExerciseTimeGoal': 980, 'appleStandHours': 980, 'appleStandHoursGoal': 980, 'dateComponents': 980}, 'child_counts': {}, 'count': 980, 'parent_counts': {'HealthData': 980}}, 'Correlation': {'attr_counts': {'creationDate': 1, 'endDate': 1, 'sourceName': 1, 'sourceVersion': 1, 'startDate': 1, 'type': 1}, 'child_counts': {'MetadataEntry': 1, 'Record': 2}, 'count': 1, 'parent_counts': {'HealthData': 1}}, 'ExportDate': {'attr_counts': {'value': 1}, 'child_counts': {}, 'count': 1, 'parent_counts': {'HealthData': 1}}, 'HealthData': {'attr_counts': {'locale': 1}, 'child_counts': {'ActivitySummary': 980, 'Correlation': 1, 'ExportDate': 1, 'Me': 1, 'Record': 2672231, 'Workout': 663}, 'count': 1, 'parent_counts': {}}, 'HeartRateVariabilityMetadataList': {'attr_counts': {}, 'child_counts': {'InstantaneousBeatsPerMinute': 93653}, 'count': 2318, 'parent_counts': {'Record': 2318}}, 'InstantaneousBeatsPerMinute': {'attr_counts': {'bpm': 93653, 'time': 93653}, 'child_counts': {}, 'count': 93653, 'parent_counts': {'HeartRateVariabilityMetadataList': 93653}}, 'Location': {'attr_counts': {'altitude': 398683, 'course': 398683, 'date': 398683, 'horizontalAccuracy': 398683, 'latitude': 398683, 'longitude': 398683, 'speed': 398683, 'verticalAccuracy': 398683}, 'child_counts': {}, 'count': 398683, 'parent_counts': {'WorkoutRoute': 398683}}, 'Me': {'attr_counts': {'HKCharacteristicTypeIdentifierBiologicalSex': 1, 'HKCharacteristicTypeIdentifierBloodType': 1, 'HKCharacteristicTypeIdentifierDateOfBirth': 1, 'HKCharacteristicTypeIdentifierFitzpatrickSkinType': 1}, 'child_counts': {}, 'count': 1, 'parent_counts': {'HealthData': 1}}, 'MetadataEntry': {'attr_counts': {'key': 290449, 'value': 290449}, 'child_counts': {}, 'count': 290449, 'parent_counts': {'Correlation': 1, 'Record': 287974, 'Workout': 1928, 'WorkoutRoute': 546}}, 'Record': {'attr_counts': {'creationDate': 2672233, 'device': 2665111, 'endDate': 2672233, 'sourceName': 2672233, 'sourceVersion': 2671779, 'startDate': 2672233, 'type': 2672233, 'unit': 2650012, 'value': 2672232}, 'child_counts': {'HeartRateVariabilityMetadataList': 2318, 'MetadataEntry': 287974}, 'count': 2672233, 'parent_counts': {'Correlation': 2, 'HealthData': 2672231}}, 'Workout': {'attr_counts': {'creationDate': 663, 'device': 230, 'duration': 663, 'durationUnit': 663, 'endDate': 663, 'sourceName': 663, 'sourceVersion': 663, 'startDate': 663, 'totalDistance': 663, 'totalDistanceUnit': 663, 'totalEnergyBurned': 663, 'totalEnergyBurnedUnit': 663, 'workoutActivityType': 663}, 'child_counts': {'MetadataEntry': 1928, 'WorkoutEvent': 2094, 'WorkoutRoute': 340}, 'count': 663, 'parent_counts': {'HealthData': 663}}, 'WorkoutEvent': {'attr_counts': {'date': 2094, 'duration': 837, 'durationUnit': 837, 'type': 2094}, 'child_counts': {}, 'count': 2094, 'parent_counts': {'Workout': 2094}}, 'WorkoutRoute': {'attr_counts': {'creationDate': 340, 'endDate': 340, 'sourceName': 340, 'sourceVersion': 340, 'startDate': 340}, 'child_counts': {'Location': 398683, 'MetadataEntry': 546}, 'count': 340, 'parent_counts': {'Workout': 340}}} ``` The most interesting bit is this: ```python 'HealthData': {'attr_counts': {'locale': 1}, 'child_counts': {'ActivitySummary': 980, 'Correlation': 1, 'ExportDate': 1, 'Me': 1, 'Record': 2672231, 'Workout': 663}, ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470637068,Use XML Analyser to figure out the structure of the export XML,