{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/13#issuecomment-620273692", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/13", "id": 620273692, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMDI3MzY5Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-27T22:42:50Z", "updated_at": "2020-04-27T22:42:50Z", "author_association": "MEMBER", "body": "```\r\n>>> def ext_counts(directory):\r\n... counts = {}\r\n... for path in pathlib.Path(directory).glob(\"**/*\"):\r\n... ext = path.suffix\r\n... counts[ext] = counts.get(ext, 0) + 1\r\n... return counts\r\n... \r\n>>> \r\n>>> ext_counts(\"/Users/simon/Pictures/Photos Library.photoslibrary/originals\")\r\n{'': 16, '.heic': 15478, '.jpeg': 21691, '.mov': 946, '.png': 2262, '.gif': 38, '.mp4': 116, '.aae': 2}\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 607888367, "label": "Also upload movie files"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/12#issuecomment-618796564", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/12", "id": 618796564, "node_id": "MDEyOklzc3VlQ29tbWVudDYxODc5NjU2NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-24T04:35:25Z", "updated_at": "2020-04-24T04:35:25Z", "author_association": "MEMBER", "body": "Code: https://github.com/dogsheep/photos-to-sqlite/blob/a388cf1f1b6b67752d669466cda8b171b6582171/photos_to_sqlite/cli.py#L109-L114", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 606033104, "label": "If less than 500MB, show size in MB not GB"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/9#issuecomment-618725155", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/9", "id": 618725155, "node_id": "MDEyOklzc3VlQ29tbWVudDYxODcyNTE1NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-23T23:39:14Z", "updated_at": "2020-04-23T23:39:14Z", "author_association": "MEMBER", "body": "A few minutes later...\r\n```\r\nFetching existing keys from S3...\r\nGot 22,446 existing keys\r\nCalculating hashes [####################################] 100% \r\n22,441 hashed files, 610 are not yet in S3\r\nUploading 0.99 GB\r\nUploading 610 photos [------------------------------------] 1/610 03:10:35\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 605938063, "label": "upload command should be resumable, should only upload photos not already uploaded"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/9#issuecomment-618724149", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/9", "id": 618724149, "node_id": "MDEyOklzc3VlQ29tbWVudDYxODcyNDE0OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-23T23:35:29Z", "updated_at": "2020-04-23T23:35:29Z", "author_association": "MEMBER", "body": "```\r\n% photos-to-sqlite upload photos.db ~/Pictures/Photos\\ Library.photoslibrary/originals\r\nFetching existing keys from S3...\r\nGot 22,446 existing keys\r\nCalculating hashes [####--------------------------------] 13% 00:04:14\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 605938063, "label": "upload command should be resumable, should only upload photos not already uploaded"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/8#issuecomment-618100658", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/8", "id": 618100658, "node_id": "MDEyOklzc3VlQ29tbWVudDYxODEwMDY1OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-23T00:03:35Z", "updated_at": "2020-04-23T00:03:35Z", "author_association": "MEMBER", "body": "Also MD5 isn't guaranteed for the ETag:\r\n> If an object is created by either the Multipart Upload or Part Copy operation, the ETag is not an MD5 digest, regardless of the method of encryption.\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 605147638, "label": "Should I have used MD5 instead of SHA256?"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/8#issuecomment-618100434", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/8", "id": 618100434, "node_id": "MDEyOklzc3VlQ29tbWVudDYxODEwMDQzNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-23T00:02:53Z", "updated_at": "2020-04-23T00:02:53Z", "author_association": "MEMBER", "body": "I don't think it matters one way or the other - I'm storing the sha256 in the filename, so the fact that I could read the MD5 back from the list bucket operation doesn't give me any benefits.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 605147638, "label": "Should I have used MD5 instead of SHA256?"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/31#issuecomment-617491607", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/31", "id": 617491607, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNzQ5MTYwNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-22T01:20:19Z", "updated_at": "2020-04-22T01:20:19Z", "author_association": "MEMBER", "body": "https://github-to-sqlite.dogsheep.net/github/milestones now link to repo:\r\n\r\n![\"Screen](\"https://user-images.githubusercontent.com/9599/79930145-956a7280-83fc-11ea-9b47-b0c5674a0d4f.png\")
\r\n\r\nAnd so do issues: https://github-to-sqlite.dogsheep.net/github/issues\r\n\r\n![\"Screen](\"https://user-images.githubusercontent.com/9599/79930200-bd59d600-83fc-11ea-8fb8-b40772140409.png\")
\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 603624862, "label": "Issue and milestone should have foreign key to repo"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/32#issuecomment-617490914", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/32", "id": 617490914, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNzQ5MDkxNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-22T01:17:44Z", "updated_at": "2020-04-22T01:17:44Z", "author_association": "MEMBER", "body": "https://github-to-sqlite.dogsheep.net/github?sql=select+html_url%2C+id%2C+issue+from+issue_comments+order+by+updated_at+desc+limit+101 now shows issues.\r\n\r\nAnd https://github-to-sqlite.dogsheep.net/github/issue_comments links to them:\r\n\r\n![\"Screen](\"https://user-images.githubusercontent.com/9599/79930066-6227e380-83fc-11ea-8416-8b9d55660cb5.png\")
\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 604222295, "label": "Issue comments don't appear to populate issues foreign key"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/32#issuecomment-617369247", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/32", "id": 617369247, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNzM2OTI0Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-21T19:33:03Z", "updated_at": "2020-04-21T19:33:03Z", "author_association": "MEMBER", "body": "Caused by #31.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 604222295, "label": "Issue comments don't appear to populate issues foreign key"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/32#issuecomment-617364956", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/32", "id": 617364956, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNzM2NDk1Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-21T19:24:45Z", "updated_at": "2020-04-21T19:24:45Z", "author_association": "MEMBER", "body": "That's because I just broke this code:\r\n\r\nhttps://github.com/dogsheep/github-to-sqlite/blob/2cf75a0a036719eb7e57fdc7c5c2ea0f4c26978a/github_to_sqlite/utils.py#L131-L139\r\n\r\nIt expects the `repo` column to be `simonw/datasette` but it's now an ID instead.\r\n\r\nI should add a test for this as part of the fix.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 604222295, "label": "Issue comments don't appear to populate issues foreign key"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/31#issuecomment-617348174", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/31", "id": 617348174, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNzM0ODE3NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-21T18:50:29Z", "updated_at": "2020-04-21T18:50:29Z", "author_association": "MEMBER", "body": "Since this represents a breaking schema change for anyone running SQL queries against these tables, I'm going to do a major version bump to 2.0 when I release this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 603624862, "label": "Issue and milestone should have foreign key to repo"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/31#issuecomment-616884647", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/31", "id": 616884647, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNjg4NDY0Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-21T00:49:16Z", "updated_at": "2020-04-21T00:50:20Z", "author_association": "MEMBER", "body": "The API just gives us the `repository_url`: https://api.github.com/repos/simonw/datasette/issues\r\n\r\n\r\n\r\nWe currently turn that into a `simonw/datasette` string here:\r\nhttps://github.com/dogsheep/github-to-sqlite/blob/e0e8d8caa9657b04bfb8a2cf16c9b580f38b1805/github_to_sqlite/utils.py#L43-L46", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 603624862, "label": "Issue and milestone should have foreign key to repo"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/30#issuecomment-616883726", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/30", "id": 616883726, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNjg4MzcyNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-21T00:45:23Z", "updated_at": "2020-04-21T00:45:23Z", "author_association": "MEMBER", "body": "Demo of fix: https://github-to-sqlite.dogsheep.net/github/issues?assignee__notblank=1&milestone__notblank=1\r\n\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 603618244, "label": "Issues milestone column is the wrong type"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/29#issuecomment-616883275", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/29", "id": 616883275, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNjg4MzI3NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-21T00:43:28Z", "updated_at": "2020-04-21T00:43:28Z", "author_association": "MEMBER", "body": "I'm copying repo from issue, which surprisingly is a string, not an integer ID.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 603617013, "label": "Milestones should have foreign key to creator and repo"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/30#issuecomment-616879753", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/30", "id": 616879753, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNjg3OTc1Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-21T00:29:29Z", "updated_at": "2020-04-21T00:29:29Z", "author_association": "MEMBER", "body": "`assignee` looks like it's the incorrect type too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 603618244, "label": "Issues milestone column is the wrong type"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/45#issuecomment-616029262", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/45", "id": 616029262, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNjAyOTI2Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-19T04:39:21Z", "updated_at": "2020-04-19T04:39:21Z", "author_association": "MEMBER", "body": "\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602619330, "label": "Use raise_for_status() everywhere"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/7#issuecomment-615993178", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/7", "id": 615993178, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk5MzE3OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-19T00:37:08Z", "updated_at": "2020-04-19T00:37:08Z", "author_association": "MEMBER", "body": "https://pypi.org/project/ImageHash/ Is one option.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602585497, "label": "Integrate image content hashing"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/6#issuecomment-615983393", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/6", "id": 615983393, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk4MzM5Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T23:53:10Z", "updated_at": "2020-04-18T23:53:10Z", "author_association": "MEMBER", "body": "```\r\n$ photos-to-sqlite upload photos3.db ~/Pictures/Photos\\ Library.photoslibrary/Masters/2020\r\nUploading 2.09 GB [##----------------------------------] 6% 00:36:37\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602575575, "label": "Add progress bar to upload command"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/6#issuecomment-615979923", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/6", "id": 615979923, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk3OTkyMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T23:36:02Z", "updated_at": "2020-04-18T23:36:02Z", "author_association": "MEMBER", "body": "I'll use a Click progress bar. To do this I need to first calculate the sum number of bytes in the photos that are going to be uploaded, then run the upload.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602575575, "label": "Add progress bar to upload command"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615957385", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615957385, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk1NzM4NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T21:56:16Z", "updated_at": "2020-04-18T21:58:11Z", "author_association": "MEMBER", "body": "Got this working! I'll do EXIF in a separate ticket #3.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/5#issuecomment-615949574", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/5", "id": 615949574, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk0OTU3NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T21:06:07Z", "updated_at": "2020-04-18T21:06:07Z", "author_association": "MEMBER", "body": "```\r\n$ photos-to-sqlite s3-auth\r\nCreate S3 credentials and paste them here:\r\n\r\nAccess key ID: xxx\r\nSecret access key: yyy\r\n$ cat auth.json \r\n{\r\n \"access_key_id\": \"xxx\",\r\n \"secret_access_key\": \"yyy\"\r\n}\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602551638, "label": "photos-to-sqlite s3-auth command"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615948102", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615948102, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk0ODEwMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T20:56:59Z", "updated_at": "2020-04-18T20:56:59Z", "author_association": "MEMBER", "body": "I'm going to start with this:\r\n\r\n`photos-to-sqlite upload photos.db ~/path/to/directory`\r\n\r\nThis will scan the provided directory (and all sub-directories) for image files. It will then:\r\n\r\n* Calculate a sha256 of the contents of that file\r\n* Upload the file to a key that's `sha256.jpg` or `.heic`\r\n* Upload a `sha256.json` file with the original path to the image\r\n* Add that image to a `uploads` table in `photos.db`\r\n\r\nStretch goal: grab the EXIF data and include that in the `.json` upload AND the `uploads` database table.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615947370", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615947370, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk0NzM3MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T20:52:13Z", "updated_at": "2020-04-18T20:52:13Z", "author_association": "MEMBER", "body": "This is great! I now have a key that can upload photos, and a separate key that can download photos OR generate signed URLs to access those photos.\r\n\r\nNext step: a script that starts uploading my photos.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615947229", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615947229, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk0NzIyOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T20:51:26Z", "updated_at": "2020-04-18T20:51:26Z", "author_association": "MEMBER", "body": "Running the upload again like this resulted in the correct content-type:\r\n```python\r\nclient.upload_file(\r\n \"/Users/simonw/Desktop/this_is_fine.jpg\",\r\n \"dogsheep-photos-simon\",\r\n \"this_is_fine.jpg\",\r\n ExtraArgs={\r\n \"ContentType\": \"image/jpeg\"\r\n }\r\n)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615946537", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615946537, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk0NjUzNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T20:48:13Z", "updated_at": "2020-04-18T20:48:13Z", "author_association": "MEMBER", "body": "How about generating a signed URL?\r\n```python\r\nread_client.generate_presigned_url(\r\n \"get_object\",\r\n Params={\r\n \"Bucket\": \"dogsheep-photos-simon\",\r\n \"Key\": \"this_is_fine.jpg\",\r\n },\r\n ExpiresIn=600\r\n)\r\n```\r\nGave me https://dogsheep-photos-simon.s3.amazonaws.com/this_is_fine.jpg?AWSAccessKeyId=AKIAWXFXAIOZNZ3JFO7I&Signature=x1zrS4w4OTGAACd7yHp9mYqXvN8%3D&Expires=1587243398\r\n\r\nWhich does this:\r\n\r\n```\r\n~ $ curl -i 'https://dogsheep-photos-simon.s3.amazonaws.com/this_is_fine.jpg?AWSAccessKeyId=AKIAWXFXAIOZNZ3JFO7I&Signature=x1zrS4w4OTGAACd7yHp9mYqXvN8%3D&Expires=1587243398'\r\nHTTP/1.1 307 Temporary Redirect\r\nx-amz-bucket-region: us-west-1\r\nx-amz-request-id: E78CD859AEE21D33\r\nx-amz-id-2: 648mx+1+YSGga7NDOU7Q6isfsKnEPWOLC+DI4+x2o9FCc6pSCdIaoHJUbFMI8Vsuh1ADtx46ymU=\r\nLocation: https://dogsheep-photos-simon.s3-us-west-1.amazonaws.com/this_is_fine.jpg?AWSAccessKeyId=AKIAWXFXAIOZNZ3JFO7I&Signature=x1zrS4w4OTGAACd7yHp9mYqXvN8%3D&Expires=1587243398\r\nContent-Type: application/xml\r\nTransfer-Encoding: chunked\r\nDate: Sat, 18 Apr 2020 20:47:21 GMT\r\nServer: AmazonS3\r\n\r\n\r\nTemporaryRedirectPlease re-send this request to the specified temporary endpoint. Continue to use the original request endpoint for future requests.dogsheep-photos-simon.s3-us-west-1.amazonaws.comdogsheep-photos-simonE78CD859AEE21D33648mx+1+YSGga7NDOU7Q6isfsKnEPWOLC+DI4+x2o9FCc6pSCdIaoHJUbFMI8Vsuh1ADtx46ymU=~ $ \r\n```\r\nSo it redirects to another URL... which returns this:\r\n```\r\n~ $ curl -i 'https://dogsheep-photos-simon.s3-us-west-1.amazonaws.com/this_is_fine.jpg?AWSAccessKeyId=AKIAWXFXAIOZNZ3JFO7I&Signature=x1zrS4w4OTGAACd7yHp9mYqXvN8%3D&Expires=1587243398'\r\nHTTP/1.1 200 OK\r\nx-amz-id-2: XafOl6mswj3yz0GJC9+Ptot1ll5sROVwqsMc10CUUfgpaUANTdIx2GhnONb5d1GVFJ6wlS2j3UY=\r\nx-amz-request-id: 258387C180411AFE\r\nDate: Sat, 18 Apr 2020 20:47:52 GMT\r\nLast-Modified: Sat, 18 Apr 2020 20:37:35 GMT\r\nETag: \"ee04081c3182a44a1c6944e94012e977\"\r\nAccept-Ranges: bytes\r\nContent-Type: binary/octet-stream\r\nContent-Length: 53072\r\nServer: AmazonS3\r\n\r\n????JFIF??C\r\n```\r\nSo that worked! It did come back with `Content-Type: binary/octet-stream` though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615945056", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615945056, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk0NTA1Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T20:42:41Z", "updated_at": "2020-04-18T20:42:41Z", "author_association": "MEMBER", "body": "But... `list_objects` failed for both of my keys (read and write):\r\n\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615944806", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615944806, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk0NDgwNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T20:41:39Z", "updated_at": "2020-04-18T20:41:39Z", "author_association": "MEMBER", "body": "This worked!\r\n\r\n\r\n\r\nAnd this worked:\r\n\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615942116", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615942116, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk0MjExNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T20:30:56Z", "updated_at": "2020-04-18T20:30:56Z", "author_association": "MEMBER", "body": "Next step: attempt a programmatic upload using the `dogsheep-photos-simon-read-write` credentials from a Jupyter notebook.\r\n\r\nAlso attempt a programmatic bucket listing and read using `dogsheep-photos-simon-read` credentials.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615941746", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615941746, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk0MTc0Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T20:29:36Z", "updated_at": "2020-04-18T20:29:36Z", "author_association": "MEMBER", "body": "I'm going to create another user just for Transmit, with full S3 access.\r\n\r\nname: `dogsheep-photos-simon-s3-all-access`\r\n\r\nRather than creating a group for that user, I'm trying the \"Attach existing policies directly\" option:\r\n\r\n\r\n\r\nThat user DID work with Transmit. I uploaded a test HEIC image. I used Transmit to copy a signed URL for it.\r\n\r\n```\r\n~ $ curl -i 'https://dogsheep-photos-simon.s3.us-west-1.amazonaws.com/IMG_7195.HEIC?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAWXFXAI...' | head -n 100\r\n % Total % Received % Xferd Average Speed Time Time Time Current\r\n Dload Upload Total Spent Left Speed\r\n 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0HTTP/1.1 200 OK\r\nx-amz-id-2: gBOCYqZfbNAnv0R/uJ++qm2NbW5SgD4TapgF9RQjzzeDIThcCz/BkKU+YoxlG4NJHlcmMgAHyh4=\r\nx-amz-request-id: C2FE7FCC3BD53A84\r\nDate: Sat, 18 Apr 2020 20:28:54 GMT\r\nLast-Modified: Sat, 18 Apr 2020 20:13:49 GMT\r\nETag: \"fe3e081239a123ef745517878c53b854\"\r\nAccept-Ranges: bytes\r\nContent-Type: image/heic\r\nContent-Length: 1913097\r\nServer: AmazonS3\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615936880", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615936880, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTkzNjg4MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T20:04:31Z", "updated_at": "2020-04-18T20:04:31Z", "author_association": "MEMBER", "body": "Next step: create two IAM users, one for each of those groups.\r\n\r\nhttps://console.aws.amazon.com/iam/home#/users$new?step=details\r\n\r\n\r\n\r\n\r\n\r\nI copied the keys into a secure note in 1password.\r\n\r\nCouldn't get into Transmit with them though! https://library.panic.com/transmit/transmit5/iam-roles/ may help.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615935577", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615935577, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTkzNTU3Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T19:54:59Z", "updated_at": "2020-04-18T19:55:30Z", "author_association": "MEMBER", "body": "Creating IAM groups called `dogsheep-photos-simon-read-write` and `dogsheep-photos-simon-read`: https://console.aws.amazon.com/iam/home#/groups - I created them with no attached policies.\r\n\r\nNow I can attach an \"inline policy\" to each one. For the read-write group I go here:\r\n\r\nhttps://console.aws.amazon.com/iam/home#/groups/dogsheep-photos-simon-read-write\r\n\r\n\r\n\r\nExample policies are here: https://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html\r\n\r\nFor the read-write one I went with:\r\n```json\r\n{\r\n \"Version\": \"2012-10-17\",\r\n \"Statement\": [\r\n {\r\n \"Effect\": \"Allow\",\r\n \"Action\": \"s3:*\",\r\n \"Resource\": [\r\n \"arn:aws:s3:::dogsheep-photos-simon/*\"\r\n ]\r\n }\r\n ]\r\n}\r\n```\r\nFor the read-only policy I'm going to guess that this is appropriate:\r\n\r\n```json\r\n{\r\n \"Version\": \"2012-10-17\",\r\n \"Statement\": [\r\n {\r\n \"Effect\": \"Allow\",\r\n \"Action\": [\r\n \"s3:GetObject*\",\r\n \"s3:ListBucket\"\r\n ],\r\n \"Resource\": [\r\n \"arn:aws:s3:::dogsheep-photos-simon/*\"\r\n ]\r\n }\r\n ]\r\n}\r\n```\r\nI tried the policy simulator to test this out: https://policysim.aws.amazon.com/home/index.jsp?#groups/dogsheep-photos-simon-read - this worked:\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615933273", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615933273, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTkzMzI3Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T19:37:33Z", "updated_at": "2020-04-18T19:37:33Z", "author_association": "MEMBER", "body": "https://console.aws.amazon.com/s3/bucket/create?region=us-west-1\r\n\r\n\r\n\r\nI created it with no public read-write access. I plan to use signed URLs via a transforming proxy to access images for display on the web.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615932204", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615932204, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTkzMjIwNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T19:29:22Z", "updated_at": "2020-04-18T19:34:44Z", "author_association": "MEMBER", "body": "I'm going to call my bucket `dogsheep-photos-simon`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615932007", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615932007, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTkzMjAwNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T19:27:55Z", "updated_at": "2020-04-18T19:27:55Z", "author_association": "MEMBER", "body": "Research thread: https://twitter.com/simonw/status/1249049694984011776\r\n> I want to build some software that lets people store their own data in their own S3 bucket, but if possible I'd like not to have to teach people the incantations needed to get their bucket setup and minimum-permission credentials figures out\r\n\r\nhttps://testdriven.io/blog/storing-django-static-and-media-files-on-amazon-s3/ looks useful", "reactions": "{\"total_count\": 2, \"+1\": 2, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/2#issuecomment-615931488", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/2", "id": 615931488, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTkzMTQ4OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T19:24:02Z", "updated_at": "2020-04-18T19:24:02Z", "author_association": "MEMBER", "body": "I made a start on this last week with a https://github.com/simonw/heic-to-jpeg proxy.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533352, "label": "Ability to convert HEIC images to JPEG"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/28#issuecomment-615886206", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/28", "id": 615886206, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTg4NjIwNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T15:04:59Z", "updated_at": "2020-04-18T15:04:59Z", "author_association": "MEMBER", "body": "Demo: https://github-to-sqlite.dogsheep.net/github/contributors\r\n\r\nDocumentation: https://github.com/dogsheep/github-to-sqlite/blob/13f8868fb5efa01c263b24f6dd91c617e6e938e1/README.md#fetching-contributors-to-a-repository ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601333634, "label": "Pull repository contributors"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/28#issuecomment-615883687", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/28", "id": 615883687, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTg4MzY4Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T14:49:58Z", "updated_at": "2020-04-18T14:49:58Z", "author_association": "MEMBER", "body": "That happened trying to pull contributors for `dogsheep/beta` - an empty repository. Turns out it was returning a `204 no content`:\r\n```\r\n~ $ curl -i 'https://api.github.com/repos/dogsheep/beta/contributors'\r\nHTTP/1.1 204 No Content\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601333634, "label": "Pull repository contributors"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/28#issuecomment-615883040", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/28", "id": 615883040, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTg4MzA0MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T14:45:38Z", "updated_at": "2020-04-18T14:45:38Z", "author_association": "MEMBER", "body": "```\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py\", line 219, in contributors\r\n utils.save_contributors(db, contributors, repo_full[\"id\"])\r\n File \"/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py\", line 354, in save_contributors\r\n for contributor in contributors:\r\n File \"/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py\", line 228, in fetch_contributors\r\n for contributors in paginate(url, headers):\r\n File \"/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py\", line 286, in paginate\r\n data = response.json()\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/requests/models.py\", line 898, in json\r\n return complexjson.loads(self.text, **kwargs)\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/json/__init__.py\", line 357, in loads\r\n return _default_decoder.decode(s)\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/json/decoder.py\", line 337, in decode\r\n obj, end = self.raw_decode(s, idx=_w(s, 0).end())\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/json/decoder.py\", line 355, in raw_decode\r\n raise JSONDecodeError(\"Expecting value\", s, err.value) from None\r\njson.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601333634, "label": "Pull repository contributors"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-615519409", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27", "id": 615519409, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTUxOTQwOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T00:19:16Z", "updated_at": "2020-04-18T00:19:16Z", "author_association": "MEMBER", "body": "```\r\n$ github-to-sqlite repos b.db dogsheep\r\n$ sqlite3 b.db '.schema repos'\r\nCREATE TABLE [repos] (\r\n [id] INTEGER PRIMARY KEY,\r\n...\r\n [permissions] TEXT,\r\n [organization] INTEGER REFERENCES [users]([id]),\r\n FOREIGN KEY(license) REFERENCES licenses(key)\r\n);\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601330277, "label": "Repos have a big blob of JSON in the organization column"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-615518606", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27", "id": 615518606, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTUxODYwNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T00:14:32Z", "updated_at": "2020-04-18T00:14:32Z", "author_association": "MEMBER", "body": "https://github.com/simonw/sqlite-utils/issues/100 is done and released in sqlite-utils 2.7.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601330277, "label": "Repos have a big blob of JSON in the organization column"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/43#issuecomment-615513491", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/43", "id": 615513491, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTUxMzQ5MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-17T23:48:28Z", "updated_at": "2020-04-17T23:48:28Z", "author_association": "MEMBER", "body": "Released in 0.21.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602176870, "label": "\"twitter-to-sqlite lists\" command for retrieving a user's owned lists"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/37#issuecomment-615510361", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/37", "id": 615510361, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTUxMDM2MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-17T23:38:27Z", "updated_at": "2020-04-17T23:38:27Z", "author_association": "MEMBER", "body": "That's a bit tricky since I'd have to rewrite the internals of a bunch of other commands. For the moment I'll exit the script with an error but at least it will be a decent error!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585353598, "label": "Handle \"User not found\" error"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/37#issuecomment-615509803", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/37", "id": 615509803, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTUwOTgwMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-17T23:36:40Z", "updated_at": "2020-04-17T23:36:40Z", "author_association": "MEMBER", "body": "I'm going to print a warning to stderr, skip and continue - because if you have 100 screen names and only one of them is invalid you should still execute for the other 99.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585353598, "label": "Handle \"User not found\" error"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/37#issuecomment-615509578", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/37", "id": 615509578, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTUwOTU3OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-17T23:36:00Z", "updated_at": "2020-04-17T23:36:00Z", "author_association": "MEMBER", "body": "```\r\n$ twitter-to-sqlite user-timeline doggo.db doggoenthuonetuh\r\nTraceback (most recent call last):\r\n...\r\n File \"/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/utils.py\", line 272, in transform_user\r\n user[\"created_at\"] = parser.parse(user[\"created_at\"])\r\nKeyError: 'created_at'\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585353598, "label": "Handle \"User not found\" error"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-614843406", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27", "id": 614843406, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNDg0MzQwNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-16T19:11:53Z", "updated_at": "2020-04-16T19:20:23Z", "author_association": "MEMBER", "body": "This didn't quite work: the column type is incorrect, so the foreign key relationship isn't sticking:\r\n\r\nhttps://github-to-sqlite.dogsheep.net/github/repos?organization=53015001\r\n\r\n`[organization] TEXT REFERENCES [users]([id])` - should be `INTEGER`.\r\n\r\nThe problem is that if the first repo inserted has no organization it's set to `null`, which `sqlite-utils` derives as a `TEXT` column.\r\n\r\nOne solution would be to create the column explicitly with a type, but this could get messy.\r\n\r\nI think I want a new sqlite-utils feature for this instead.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601330277, "label": "Repos have a big blob of JSON in the organization column"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-614831842", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27", "id": 614831842, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNDgzMTg0Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-16T18:48:18Z", "updated_at": "2020-04-16T18:48:18Z", "author_association": "MEMBER", "body": "I'm going to make `organization` another foreign key to the `users` table just in case it IS possible (maybe with GitHub Enterprise or similar?)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601330277, "label": "Repos have a big blob of JSON in the organization column"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-614831451", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27", "id": 614831451, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNDgzMTQ1MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-16T18:47:25Z", "updated_at": "2020-04-16T18:47:25Z", "author_association": "MEMBER", "body": "Is it possible for a repo to have an `owner` that differs from its `organization`?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601330277, "label": "Repos have a big blob of JSON in the organization column"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/25#issuecomment-614810417", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/25", "id": 614810417, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNDgxMDQxNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-16T18:07:11Z", "updated_at": "2020-04-16T18:07:11Z", "author_association": "MEMBER", "body": "Turns out the main problem was #26 - now fixed.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601265023, "label": "Improvements to demo instance"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/26#issuecomment-614795712", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/26", "id": 614795712, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNDc5NTcxMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-16T17:40:27Z", "updated_at": "2020-04-16T17:40:27Z", "author_association": "MEMBER", "body": "Aha! it was missing from the `fetch_repo()` function.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601271612, "label": "Topics are missing from repositories"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/26#issuecomment-614794739", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/26", "id": 614794739, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNDc5NDczOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-16T17:38:28Z", "updated_at": "2020-04-16T17:38:28Z", "author_association": "MEMBER", "body": "I'm already doing this here: https://github.com/dogsheep/github-to-sqlite/blob/c4aaa50e167cfa9021c7c94260bc3e89e10947bf/github_to_sqlite/utils.py#L246-L250", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601271612, "label": "Topics are missing from repositories"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/14#issuecomment-613641947", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14", "id": 613641947, "node_id": "MDEyOklzc3VlQ29tbWVudDYxMzY0MTk0Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-14T19:38:24Z", "updated_at": "2020-04-14T19:38:34Z", "author_association": "MEMBER", "body": "Since events include payloads with full object representations in them (for issues, repos and more) running this command every few minutes may be all it takes to keep a constant copy of everything updated in a very rate-limit friendly manner (thanks to the ETags).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 530491074, "label": "Command for importing events"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/16#issuecomment-613611455", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/16", "id": 613611455, "node_id": "MDEyOklzc3VlQ29tbWVudDYxMzYxMTQ1NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-14T18:37:21Z", "updated_at": "2020-04-14T18:37:21Z", "author_association": "MEMBER", "body": "This should have been fixed by #20 and #23 \r\n\r\n@jayvdb I'm definitely interested in this tool working as a library - it's purely designed as a CLI tool at the moment, but cleaning it up to work better as a dependency is totally in-scope for the project. https://sqlite-utils.readthedocs.io/ is an example of a tool I've built that works for both.\r\n\r\nFeel free to open a new issue here with some notes on what you would need for this to work as a library for your project!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 546051181, "label": "Exception running first command: IndexError: list index out of range"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-607019151", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40", "id": 607019151, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNzAxOTE1MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-01T04:11:10Z", "updated_at": "2020-04-01T04:11:10Z", "author_association": "MEMBER", "body": "In testing this collects a LOT of data. I'm going to skip tracking favourites_count and statuses_count and just track followers, friends and listed instead.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590669793, "label": "Feature: record history of follower counts"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-607011421", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40", "id": 607011421, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNzAxMTQyMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-01T03:47:37Z", "updated_at": "2020-04-01T03:55:08Z", "author_association": "MEMBER", "body": "Actually a single table with a `type` integer ID referencing a `count_history_types` table would better match the way I implemented the `since_ids` table:\r\n\r\nhttps://github.com/dogsheep/twitter-to-sqlite/blob/4b6c8d8c1cc6fefdb566ec8506157133f47c569a/twitter_to_sqlite/utils.py#L331-L341\r\n\r\nIn which case the compound primary key would be `type`, `user`, `datetime`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590669793, "label": "Feature: record history of follower counts"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-607011972", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40", "id": 607011972, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNzAxMTk3Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-01T03:49:02Z", "updated_at": "2020-04-01T03:50:01Z", "author_association": "MEMBER", "body": "I want the datetime value to look like `2020-04-01T03:34:58+00:00` (the format returned by the Twitter API which I am storing in other tables at the moment).\r\n```\r\n>>> datetime.utcnow().isoformat().split('.')[0] + '+00:00'\r\n'2020-04-01T03:49:52+00:00'\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590669793, "label": "Feature: record history of follower counts"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/10#issuecomment-607010791", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10", "id": 607010791, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNzAxMDc5MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-01T03:45:48Z", "updated_at": "2020-04-01T03:45:48Z", "author_association": "MEMBER", "body": "I'm happy with the recent work I did on this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 492297930, "label": "Rethink progress bars for various commands"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-607010634", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 607010634, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNzAxMDYzNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-01T03:45:16Z", "updated_at": "2020-04-01T03:45:16Z", "author_association": "MEMBER", "body": "OK, fix is applied to everything now.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-607003655", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 607003655, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNzAwMzY1NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-01T03:18:00Z", "updated_at": "2020-04-01T03:18:00Z", "author_association": "MEMBER", "body": "I've got this working for the `user-timeline` command.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606850453", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606850453, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjg1MDQ1Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-31T20:14:58Z", "updated_at": "2020-04-01T03:03:50Z", "author_association": "MEMBER", "body": "Actually I'll hard-code the population of `since_id_types` to get known ID constants.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606998669", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606998669, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjk5ODY2OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-01T02:57:36Z", "updated_at": "2020-04-01T02:57:36Z", "author_association": "MEMBER", "body": "The tricky thing here is thinking about the interaction between the recorded since_id and a desire to run the initial import.\r\n\r\nThe first time you run `twitter-to-sqlite user-timeline db.db username` we want to fetch as many tweets from that user as possible - probably around 3,200 before the API limitations cut us off.\r\n\r\nWe need to record the maximum ID from those as the `since_id` - which we will see on the very first page we paginate through. That way next time we run the command with `--since` we will only fetch new tweets.\r\n\r\nBut what happens if our initial import is cancelled after only a few tweets? We risk never pulling in the rest of the tweets.\r\n\r\nNot sure if I need to solve this at all or if I should instead trust users to run the command a second time without `--since` if they think they didn't retrieve anything the first time.\r\n\r\nI had considered letting `--stop_after=` over-ride `--since` but that doesn't actually make sense - if you send a since_id to the Twitter API you'll never get back more tweets than exist after that ID, so the `--stop_after` would not make a meaningful difference.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606850008", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606850008, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjg1MDAwOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-31T20:13:59Z", "updated_at": "2020-04-01T00:23:00Z", "author_association": "MEMBER", "body": "Table design for `since_ids` table:\r\n\r\ntype | key | since_id\r\n--- | --- | ---\r\n1 | 124324 | 2347239847293\r\n2 | 99ff9cefff5cbfd804f7cd43e2b27ced8addbe8d | 2125947927344\r\n\r\nPrimary compound key on `(category, key)`\r\n\r\n`type` is also a foreign key to a `since_id_types` table with `id` and `name` columns (probably created using https://sqlite-utils.readthedocs.io/en/stable/python-api.html#working-with-lookup-tables )", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606843224", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606843224, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjg0MzIyNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-31T19:59:11Z", "updated_at": "2020-03-31T20:06:32Z", "author_association": "MEMBER", "body": "Or... have a single `since_ids` table to track since values, and have its primary key be a string that looks something like this:\r\n\r\n`user:123145`\r\n`home:23441`\r\n`mentions:23425`\r\n`search:99ff9cefff5cbfd804f7cd43e2b27ced8addbe8d`\r\n\r\nThat last example would use the hash generated here:\r\n\r\nhttps://github.com/dogsheep/twitter-to-sqlite/blob/810cb2af5a175837204389fd7f4b5721f8b325ab/twitter_to_sqlite/cli.py#L792-L808", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606844521", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606844521, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjg0NDUyMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-31T20:01:39Z", "updated_at": "2020-03-31T20:01:39Z", "author_association": "MEMBER", "body": "I think `utils.fetch_timeline()` grows a new argument, `since_key`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606824992", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606824992, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjgyNDk5Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-31T19:24:23Z", "updated_at": "2020-03-31T19:24:23Z", "author_association": "MEMBER", "body": "The `--since` option is actually used by four commands:\r\n\r\n* `user-timeline`\r\n* `home-timeline`\r\n* `mentions-timeline`\r\n* `search`\r\n\r\nAll of them use the same `fetch_timeline()` utility function under the hood. I should move the logic that looks up the last `since_id` into that shared function.\r\n\r\nQuestion: should I have a table for each of those four methods or a single table that is used by them all? I'm leaning towards four separate tables.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606309165", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606309165, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjMwOTE2NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-30T23:41:31Z", "updated_at": "2020-03-30T23:41:31Z", "author_association": "MEMBER", "body": "I like the separate `user_timeline_since` table solution.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-606307376", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40", "id": 606307376, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjMwNzM3Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-30T23:35:40Z", "updated_at": "2020-03-30T23:39:15Z", "author_association": "MEMBER", "body": "I think five separate tables:\r\n\r\n* followers_count_history\r\n* friends_count_history\r\n* listed_count_history\r\n* favourites_count_history\r\n* statuses_count_history\r\n\r\nEach with the following structure:\r\n\r\n* datetime (ISO UTC)\r\n* user (ID, foreign key to users)\r\n* count (integer)\r\n\r\nI'm tempted to have a compound primary key here - user, datetime\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590669793, "label": "Feature: record history of follower counts"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-606307019", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40", "id": 606307019, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjMwNzAxOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-30T23:34:27Z", "updated_at": "2020-03-30T23:34:27Z", "author_association": "MEMBER", "body": "The count properties available for a user are:\r\n\r\n* followers_count\r\n* friends_count\r\n* listed_count\r\n* favourites_count\r\n* statuses_count\r\n\r\nMay as well track history for all of them? Should be pretty cheap to store.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590669793, "label": "Feature: record history of follower counts"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606305701", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606305701, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjMwNTcwMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-30T23:30:27Z", "updated_at": "2020-03-30T23:30:27Z", "author_association": "MEMBER", "body": "A better alternative would be to maintain a separate table with the last seen since value for when we ran `user-timeline` for any specific user.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606304837", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606304837, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjMwNDgzNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-30T23:27:50Z", "updated_at": "2020-03-30T23:29:31Z", "author_association": "MEMBER", "body": "One option would be something like this:\r\n```sql\r\nselect max(id) from tweets\r\nwhere user = ?\r\nand not exists (select id from tweets where retweeted_status = id)\r\nand not exists (select id from tweets where quoted_status = id)\r\nand not exists (select id from tweets where in_reply_to_status_id = id)\r\n```\r\nMight be a good idea to index those columns (after confirming that doing so would indeed speed up the query).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/swarm-to-sqlite/pull/6#issuecomment-605382373", "issue_url": "https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/6", "id": 605382373, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNTM4MjM3Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-28T02:27:32Z", "updated_at": "2020-03-28T02:27:32Z", "author_association": "MEMBER", "body": "Thanks!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 543355051, "label": "don't break if source is missing"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/pocket-to-sqlite/issues/2#issuecomment-605338322", "issue_url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/2", "id": 605338322, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNTMzODMyMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-27T22:18:02Z", "updated_at": "2020-03-27T22:18:02Z", "author_association": "MEMBER", "body": "Just needs documentation now.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 503234169, "label": "Track and use the 'since' value"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/pocket-to-sqlite/issues/2#issuecomment-605337941", "issue_url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/2", "id": 605337941, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNTMzNzk0MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-27T22:16:32Z", "updated_at": "2020-03-27T22:16:32Z", "author_association": "MEMBER", "body": "Need to test this. I have 7,394 items in my database right now. I'm going to save a new thing.\r\n\r\nThen I ran this:\r\n```\r\npocket-to-sqlite fetch pocket-simon.db\r\n```\r\nAnd it worked!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 503234169, "label": "Track and use the 'since' value"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/pocket-to-sqlite/issues/1#issuecomment-605327655", "issue_url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/1", "id": 605327655, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNTMyNzY1NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-27T21:42:49Z", "updated_at": "2020-03-27T21:42:49Z", "author_association": "MEMBER", "body": "Or maybe it was because of the current Google Cloud outage? https://news.ycombinator.com/item?id=22706677", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 503233021, "label": "Use better pagination (and implement progress bar)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/pocket-to-sqlite/issues/1#issuecomment-605325897", "issue_url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/1", "id": 605325897, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNTMyNTg5Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-27T21:37:26Z", "updated_at": "2020-03-27T21:38:37Z", "author_association": "MEMBER", "body": "I keep getting 503 errors even though I appear to be staying within the rate limit:\r\n```\r\n{'Date': 'Fri, 27 Mar 2020 21:35:57 GMT', 'Content-Type': 'application/json', 'Transfer-Encoding':\r\n'chunked', 'Connection': 'keep-alive', 'Server': 'Apache/2.4.25 (Debian)', 'Content-Location':\r\n'get.php', 'Vary': 'negotiate', 'TCN': 'choice', 'Set-Cookie': '...; httponly',\r\n'X-Frame-Options': 'SAMEORIGIN', 'Status': '200 OK', 'X-Limit-Key-Limit': '10000',\r\n'X-Limit-Key-Remaining': '9960', 'X-Limit-Key-Reset': '282', 'X-Source': 'Pocket',\r\n'P3P': 'policyref=\"/w3c/p3p.xml\", CP=\"ALL CURa ADMa DEVa OUR IND UNI COM NAV INT STA PRE\"'}\r\n\r\n [##----------------------------------] 6% 06:49:27\r\n\r\n{'Date': 'Fri, 27 Mar 2020 21:36:06 GMT', 'Content-Type': 'text/html; charset=UTF-8', 'Content-Length':\r\n'23', 'Connection': 'keep-alive', 'Server': 'Apache/2.4.25 (Debian)', 'Content-Location': 'get.php',\r\n'Vary': 'negotiate', 'TCN': 'choice', 'Set-Cookie': '...', 'X-Frame-Options': 'SAMEORIGIN',\r\n'X-Error': 'Pocket is currently under heavy load. Please wait a moment and try again.',\r\n'X-Error-Code': '199', 'Status': '503 Service Unavailable', 'X-Source': 'Pocket',\r\n'P3P': 'policyref=\"/w3c/p3p.xml\", CP=\"ALL CURa ADMa DEVa OUR IND UNI COM NAV INT STA PRE\"'}\r\n```\r\nI'm going to try doing a few automatic retries any time I see a 503 error.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 503233021, "label": "Use better pagination (and implement progress bar)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/pocket-to-sqlite/issues/1#issuecomment-605316146", "issue_url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/1", "id": 605316146, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNTMxNjE0Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-27T21:09:15Z", "updated_at": "2020-03-27T21:09:22Z", "author_association": "MEMBER", "body": "For a progress bar I need to know how many total items there are.\r\n\r\nI found an undocumented API for this! `/v3/stats` which returns:\r\n```json\r\n{\r\n \"count_list\": 7394,\r\n \"count_read\": 1016,\r\n \"count_unread\": 6378,\r\n \"status\": 1\r\n}\r\n```\r\nI guessed this based on the documented v2 API: https://getpocket.com/api/v2_docs/#stats", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 503233021, "label": "Use better pagination (and implement progress bar)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/23#issuecomment-602928533", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/23", "id": 602928533, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjkyODUzMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-24T00:15:49Z", "updated_at": "2020-03-24T00:15:49Z", "author_association": "MEMBER", "body": "https://github.com/dogsheep/github-to-sqlite/releases/tag/1.0", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 586595839, "label": "Release 1.0"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602924714", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602924714, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjkyNDcxNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-24T00:03:25Z", "updated_at": "2020-03-24T00:03:25Z", "author_association": "MEMBER", "body": "This is good enough for the 1.0 release.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/21#issuecomment-602920163", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/21", "id": 602920163, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjkyMDE2Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T23:48:22Z", "updated_at": "2020-03-23T23:48:22Z", "author_association": "MEMBER", "body": "I'm happy with this pattern:\r\nhttps://github.com/dogsheep/github-to-sqlite/blob/f78c4e9baaf0970ffab266ba780df7240aae9f32/github_to_sqlite/utils.py#L4-L18", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 586561727, "label": "Turn GitHub API errors into exceptions"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602918689", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602918689, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjkxODY4OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T23:43:39Z", "updated_at": "2020-03-23T23:47:50Z", "author_association": "MEMBER", "body": "I pointed https://github-to-sqlite.dogsheep.net/ at it. May take a few minutes for the new certificate to provision though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602919058", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602919058, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjkxOTA1OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T23:44:48Z", "updated_at": "2020-03-23T23:44:48Z", "author_association": "MEMBER", "body": "Next step: use a `metadata.json` file to add some extras. And add the `datasette-render-markdown` plugin as soon as I ship https://github.com/simonw/datasette-render-markdown/issues/2 (GFM support).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602917713", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602917713, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjkxNzcxMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T23:40:29Z", "updated_at": "2020-03-23T23:40:29Z", "author_association": "MEMBER", "body": "Most recently updated issues across all Dogsheep repos, with faceting: https://github-to-sqlite-j7hipcg4aq-uc.a.run.app/github/issues?_facet=repo&_facet=user&_facet=state&_facet=author_association&_facet=type&_sort_desc=updated_at", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602916947", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602916947, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjkxNjk0Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T23:38:06Z", "updated_at": "2020-03-23T23:38:06Z", "author_association": "MEMBER", "body": "Woohoo! https://github-to-sqlite-j7hipcg4aq-uc.a.run.app/", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/21#issuecomment-602896434", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/21", "id": 602896434, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjg5NjQzNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T22:43:37Z", "updated_at": "2020-03-23T22:43:37Z", "author_association": "MEMBER", "body": "I'm going to do this now to help figure out the latest error in #13:\r\n```\r\nTraceback (most recent call last):\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/bin/github-to-sqlite\", line 11, in \r\n load_entry_point('github-to-sqlite', 'console_scripts', 'github-to-sqlite')()\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py\", line 237, in commits\r\n utils.save_commits(db, commits, repo_full[\"id\"])\r\n File \"/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py\", line 345, in save_commits\r\n for commit in commits:\r\n File \"/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py\", line 207, in fetch_commits\r\n if stop_when(commit):\r\n File \"/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py\", line 224, in stop_when\r\n db[\"commits\"].get(commit[\"sha\"])\r\nTypeError: string indices must be integers\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 586561727, "label": "Turn GitHub API errors into exceptions"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602895896", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602895896, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjg5NTg5Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T22:42:25Z", "updated_at": "2020-03-23T22:42:25Z", "author_association": "MEMBER", "body": "Urgh this is such a mess! I should have done this on a branch / pull request to avoid polluting my main master history, but never mind.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602862967", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602862967, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjg2Mjk2Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T21:22:04Z", "updated_at": "2020-03-23T21:22:04Z", "author_association": "MEMBER", "body": "Following these instructions: https://simonwillison.net/2020/Jan/21/github-actions-cloud-run/", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602862236", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602862236, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjg2MjIzNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T21:20:26Z", "updated_at": "2020-03-23T21:20:26Z", "author_association": "MEMBER", "body": "I'll run the `commits` and `issues` and `issue-comments` commands in addition to the `releases` command.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602861730", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602861730, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjg2MTczMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T21:19:16Z", "updated_at": "2020-03-23T21:19:42Z", "author_association": "MEMBER", "body": "I'm going to do this with a scheduled GitHub Action in this repo.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602813731", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18", "id": 602813731, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjgxMzczMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T19:38:03Z", "updated_at": "2020-03-23T20:44:49Z", "author_association": "MEMBER", "body": "So it turns out `\"author\"` and `\"committer\"` on the commit are `null` if the email address in the nested `\"commit\"` doesn't match an existing GitHub user.\r\n\r\nMaybe I should be storing the nested data somewhere as well?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585411547, "label": "Commits in GitHub API can have null author"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602846293", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18", "id": 602846293, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjg0NjI5Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T20:44:40Z", "updated_at": "2020-03-23T20:44:40Z", "author_association": "MEMBER", "body": "I implemented the `raw_authors` idea.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585411547, "label": "Commits in GitHub API can have null author"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602815120", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18", "id": 602815120, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjgxNTEyMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T19:40:55Z", "updated_at": "2020-03-23T19:43:19Z", "author_association": "MEMBER", "body": "I could pull a pk-hashed version of the name/email into separate `raw_author` and `raw_committer` columns perhaps - against a `commit_authors` table. Could be interesting.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585411547, "label": "Commits in GitHub API can have null author"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602811785", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18", "id": 602811785, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjgxMTc4NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T19:33:56Z", "updated_at": "2020-03-23T19:34:06Z", "author_association": "MEMBER", "body": "```json\r\n[\r\n {\r\n \"sha\": \"a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0\",\r\n \"node_id\": \"MDY6Q29tbWl0OTMyMDk6YThkYzkxNDA4OWQzOTlkOWI1MjJlYmI1MWI2N2Y5YWMyZThhYTZiMA==\",\r\n \"commit\": {\r\n \"author\": {\r\n \"name\": \"Simon Willison\",\r\n \"email\": \"simon@...\",\r\n \"date\": \"2008-12-18T23:17:12Z\"\r\n },\r\n \"committer\": {\r\n \"name\": \"Simon Willison\",\r\n \"email\": \"simon@...\",\r\n \"date\": \"2008-12-18T23:17:12Z\"\r\n },\r\n \"message\": \"First commit\",\r\n \"tree\": {\r\n \"sha\": \"ac2dfb75e2592c59165c2880f3f7a16dafd452a1\",\r\n \"url\": \"https://api.github.com/repos/simonw/simonw.github.com/git/trees/ac2dfb75e2592c59165c2880f3f7a16dafd452a1\"\r\n },\r\n \"url\": \"https://api.github.com/repos/simonw/simonw.github.com/git/commits/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0\",\r\n \"comment_count\": 0,\r\n \"verification\": {\r\n \"verified\": false,\r\n \"reason\": \"unsigned\",\r\n \"signature\": null,\r\n \"payload\": null\r\n }\r\n },\r\n \"url\": \"https://api.github.com/repos/simonw/simonw.github.com/commits/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0\",\r\n \"html_url\": \"https://github.com/simonw/simonw.github.com/commit/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0\",\r\n \"comments_url\": \"https://api.github.com/repos/simonw/simonw.github.com/commits/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0/comments\",\r\n \"author\": null,\r\n \"committer\": null,\r\n \"parents\": []\r\n }\r\n]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585411547, "label": "Commits in GitHub API can have null author"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602811468", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18", "id": 602811468, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjgxMTQ2OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T19:33:14Z", "updated_at": "2020-03-23T19:33:14Z", "author_association": "MEMBER", "body": "Found one: https://api.github.com/repos/simonw/simonw.github.com/commits", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585411547, "label": "Commits in GitHub API can have null author"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602807178", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18", "id": 602807178, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjgwNzE3OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T19:24:43Z", "updated_at": "2020-03-23T19:24:43Z", "author_association": "MEMBER", "body": "I need to find an example before I work on this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585411547, "label": "Commits in GitHub API can have null author"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/5#issuecomment-602806530", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/5", "id": 602806530, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjgwNjUzMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T19:23:30Z", "updated_at": "2020-03-23T19:23:30Z", "author_association": "MEMBER", "body": "Yeah `alter=True` means that this isn't necessary.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 493671014, "label": "Add \"incomplete\" boolean to users table for incomplete profiles"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/19#issuecomment-602782437", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/19", "id": 602782437, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjc4MjQzNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T18:36:12Z", "updated_at": "2020-03-23T18:51:23Z", "author_association": "MEMBER", "body": "* `commits`: `message`\r\n* `issue_comments`: `body`\r\n* `issues`: `title`, `body`\r\n* `labels`: `name`, `description`\r\n* `licenses`: `name`\r\n* `milestones`: `title`, `description`\r\n* `releases`: `name`, `body`\r\n* `repos`: `name`, `description`\r\n* `users`: `login`, `name` (maybe `company`, `bio`)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585850715, "label": "Enable full-text search for more stuff (like commits, issues and issue_comments)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-602312054", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15", "id": 602312054, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjMxMjA1NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T01:07:10Z", "updated_at": "2020-03-23T01:07:10Z", "author_association": "MEMBER", "body": "Each asset looks like this:\r\n```json\r\n {\r\n \"url\": \"https://api.github.com/repos/instrumenta/conftest/releases/assets/11811946\",\r\n \"id\": 11811946,\r\n \"node_id\": \"MDEyOlJlbGVhc2VBc3NldDExODExOTQ2\",\r\n \"name\": \"checksums.txt\",\r\n \"label\": \"\",\r\n \"uploader\": {\r\n \"login\": \"garethr\",\r\n \"id\": 2029,\r\n \"node_id\": \"MDQ6VXNlcjIwMjk=\",\r\n \"avatar_url\": \"https://avatars2.githubusercontent.com/u/2029?v=4\",\r\n \"gravatar_id\": \"\",\r\n \"url\": \"https://api.github.com/users/garethr\",\r\n \"html_url\": \"https://github.com/garethr\",\r\n \"followers_url\": \"https://api.github.com/users/garethr/followers\",\r\n \"following_url\": \"https://api.github.com/users/garethr/following{/other_user}\",\r\n \"gists_url\": \"https://api.github.com/users/garethr/gists{/gist_id}\",\r\n \"starred_url\": \"https://api.github.com/users/garethr/starred{/owner}{/repo}\",\r\n \"subscriptions_url\": \"https://api.github.com/users/garethr/subscriptions\",\r\n \"organizations_url\": \"https://api.github.com/users/garethr/orgs\",\r\n \"repos_url\": \"https://api.github.com/users/garethr/repos\",\r\n \"events_url\": \"https://api.github.com/users/garethr/events{/privacy}\",\r\n \"received_events_url\": \"https://api.github.com/users/garethr/received_events\",\r\n \"type\": \"User\",\r\n \"site_admin\": false\r\n },\r\n \"content_type\": \"text/plain; charset=utf-8\",\r\n \"state\": \"uploaded\",\r\n \"size\": 600,\r\n \"download_count\": 2,\r\n \"created_at\": \"2019-03-30T16:56:44Z\",\r\n \"updated_at\": \"2019-03-30T16:56:44Z\",\r\n \"browser_download_url\": \"https://github.com/instrumenta/conftest/releases/download/v0.1.0/checksums.txt\"\r\n }\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 544571092, "label": "Assets table with downloads"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-602311701", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15", "id": 602311701, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjMxMTcwMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T01:05:15Z", "updated_at": "2020-03-23T01:06:14Z", "author_association": "MEMBER", "body": "None of my own releases use assets (they are all pushed to PyPI instead) but I spotted that your project here uses assets, so I'll test against that: https://github.com/instrumenta/conftest/releases/tag/v0.18.0\r\n\r\n```\r\ngithub-to-sqlite releases releases.db instrumenta/conftest\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 544571092, "label": "Assets table with downloads"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-602309825", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15", "id": 602309825, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjMwOTgyNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T00:52:38Z", "updated_at": "2020-03-23T00:52:38Z", "author_association": "MEMBER", "body": "Splitting assets out into a separate table totally makes sense to me. They can still be fetched as part of the `releases` command.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 544571092, "label": "Assets table with downloads"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/19#issuecomment-602304531", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/19", "id": 602304531, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjMwNDUzMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T00:20:28Z", "updated_at": "2020-03-23T00:20:28Z", "author_association": "MEMBER", "body": "Current FTS code:\r\nhttps://github.com/dogsheep/github-to-sqlite/blob/e35eec4343aa560c58c1634cc228d0d46c442304/github_to_sqlite/utils.py#L136-L143", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585850715, "label": "Enable full-text search for more stuff (like commits, issues and issue_comments)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/hacker-news-to-sqlite/issues/1#issuecomment-602094386", "issue_url": "https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/1", "id": 602094386, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjA5NDM4Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-21T19:47:46Z", "updated_at": "2020-03-21T19:47:46Z", "author_association": "MEMBER", "body": "Released as 0.3.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585526292, "label": "Set up full text search"}, "performed_via_github_app": null}