{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/34#issuecomment-601871579", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/34", "id": 601871579, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMTg3MTU3OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-20T19:20:52Z", "updated_at": "2020-03-20T19:20:52Z", "author_association": "MEMBER", "body": "Cause was the redefining of how `upsert()` works in sqlite-utils 2.x.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585266763, "label": "IndexError running user-timeline command"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/pull/33#issuecomment-601871930", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/33", "id": 601871930, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMTg3MTkzMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-20T19:21:41Z", "updated_at": "2020-03-20T19:21:41Z", "author_association": "MEMBER", "body": "Made obsolete by https://github.com/dogsheep/twitter-to-sqlite/commit/d3887d1535c3edd59d3ebf6aea4495bacd82e59b", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 561469252, "label": "Upgrade to sqlite-utils 2.2.1"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/35#issuecomment-601875405", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35", "id": 601875405, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMTg3NTQwNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-20T19:30:09Z", "updated_at": "2020-03-20T19:32:33Z", "author_association": "MEMBER", "body": "I'm going to redesign the command a little. Right now it defaults to fetching the authenticated user's timeline and allows `--screen_name` to be used to fetch for one other user instead.\r\n\r\nI think the design of `users-lookup` is much better. I'm going to copy that:\r\n\r\n```\r\ntwitter-to-sqlite user-timeline users.db simonw cleopaws\r\ntwitter-to-sqlite user-timeline users.db 12497 3166449535 --ids\r\n```\r\nIt will also accept `--since` and `--sql` and `--alter` and `--since_id`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585282212, "label": "twitter-to-sqlite user-timeline [screen_names] --sql / --attach"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/35#issuecomment-601875524", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35", "id": 601875524, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMTg3NTUyNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-20T19:30:27Z", "updated_at": "2020-03-20T19:30:27Z", "author_association": "MEMBER", "body": "I can give it a snazzier progress bar to, as requested by #10.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585282212, "label": "twitter-to-sqlite user-timeline [screen_names] --sql / --attach"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/35#issuecomment-601876051", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35", "id": 601876051, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMTg3NjA1MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-20T19:31:43Z", "updated_at": "2020-03-20T19:31:43Z", "author_association": "MEMBER", "body": "I'm going to keep `--screen_name` working for backwards compatibility. Using `hidden=True` on it should hide it from the help output.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585282212, "label": "twitter-to-sqlite user-timeline [screen_names] --sql / --attach"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/35#issuecomment-601890795", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35", "id": 601890795, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMTg5MDc5NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-20T20:13:08Z", "updated_at": "2020-03-20T20:13:08Z", "author_association": "MEMBER", "body": "Needs documentation.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585282212, "label": "twitter-to-sqlite user-timeline [screen_names] --sql / --attach"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/35#issuecomment-601892338", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35", "id": 601892338, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMTg5MjMzOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-20T20:17:00Z", "updated_at": "2020-03-20T20:17:00Z", "author_association": "MEMBER", "body": "Released as 0.18.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585282212, "label": "twitter-to-sqlite user-timeline [screen_names] --sql / --attach"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/38#issuecomment-601939656", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/38", "id": 601939656, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMTkzOTY1Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-20T22:37:04Z", "updated_at": "2020-03-20T22:37:04Z", "author_association": "MEMBER", "body": "New formatting:\r\n```\r\n@Alaska_DHSS [####################################] 3205\r\n@BWhippleKS [####################################] 1\r\n@CAPublicHealth [####################################] 2\r\n@CDPHE [####################################] 67\r\n@CHFSKy [####################################] 3216\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585359363, "label": "Screen name display for user-timeline is uneven"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/16#issuecomment-601979709", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/16", "id": 601979709, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMTk3OTcwOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-21T02:20:08Z", "updated_at": "2020-03-21T02:20:08Z", "author_association": "MEMBER", "body": "I'm pretty sure this means I need to upgrade the underlying `sqlite-utils` dependency. I saw the same bug in `twitter-to-sqlite`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 546051181, "label": "Exception running first command: IndexError: list index out of range"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-601980724", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18", "id": 601980724, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMTk4MDcyNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-21T02:28:05Z", "updated_at": "2020-03-21T02:28:05Z", "author_association": "MEMBER", "body": "This suggests that `commit[\"author\"]` can be `None` in some cases?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585411547, "label": "Commits in GitHub API can have null author"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-601980957", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18", "id": 601980957, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMTk4MDk1Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-21T02:30:03Z", "updated_at": "2020-03-21T02:30:03Z", "author_association": "MEMBER", "body": "https://github.community/t5/GitHub-API-Development-and/Request-for-commits-quot-author-quot-null-and-quot-committer/m-p/35842/highlight/true#M3372\r\n> Commits aren't always associated with a GitHub user. For example, perhaps a friend of mine and I were working on a project together. I have a GitHub account and my friend doesn't. If we both add commits to the repository using our own email addresses and names and then I push the repository to GitHub, my commits will be associated with my GitHub user account but my friends' commits will show up with `author` and `committer` as `null`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585411547, "label": "Commits in GitHub API can have null author"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/hacker-news-to-sqlite/issues/1#issuecomment-602094386", "issue_url": "https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/1", "id": 602094386, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjA5NDM4Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-21T19:47:46Z", "updated_at": "2020-03-21T19:47:46Z", "author_association": "MEMBER", "body": "Released as 0.3.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585526292, "label": "Set up full text search"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/19#issuecomment-602304531", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/19", "id": 602304531, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjMwNDUzMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T00:20:28Z", "updated_at": "2020-03-23T00:20:28Z", "author_association": "MEMBER", "body": "Current FTS code:\r\nhttps://github.com/dogsheep/github-to-sqlite/blob/e35eec4343aa560c58c1634cc228d0d46c442304/github_to_sqlite/utils.py#L136-L143", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585850715, "label": "Enable full-text search for more stuff (like commits, issues and issue_comments)"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-602309825", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15", "id": 602309825, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjMwOTgyNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T00:52:38Z", "updated_at": "2020-03-23T00:52:38Z", "author_association": "MEMBER", "body": "Splitting assets out into a separate table totally makes sense to me. They can still be fetched as part of the `releases` command.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 544571092, "label": "Assets table with downloads"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-602311701", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15", "id": 602311701, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjMxMTcwMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T01:05:15Z", "updated_at": "2020-03-23T01:06:14Z", "author_association": "MEMBER", "body": "None of my own releases use assets (they are all pushed to PyPI instead) but I spotted that your project here uses assets, so I'll test against that: https://github.com/instrumenta/conftest/releases/tag/v0.18.0\r\n\r\n```\r\ngithub-to-sqlite releases releases.db instrumenta/conftest\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 544571092, "label": "Assets table with downloads"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-602312054", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15", "id": 602312054, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjMxMjA1NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T01:07:10Z", "updated_at": "2020-03-23T01:07:10Z", "author_association": "MEMBER", "body": "Each asset looks like this:\r\n```json\r\n {\r\n \"url\": \"https://api.github.com/repos/instrumenta/conftest/releases/assets/11811946\",\r\n \"id\": 11811946,\r\n \"node_id\": \"MDEyOlJlbGVhc2VBc3NldDExODExOTQ2\",\r\n \"name\": \"checksums.txt\",\r\n \"label\": \"\",\r\n \"uploader\": {\r\n \"login\": \"garethr\",\r\n \"id\": 2029,\r\n \"node_id\": \"MDQ6VXNlcjIwMjk=\",\r\n \"avatar_url\": \"https://avatars2.githubusercontent.com/u/2029?v=4\",\r\n \"gravatar_id\": \"\",\r\n \"url\": \"https://api.github.com/users/garethr\",\r\n \"html_url\": \"https://github.com/garethr\",\r\n \"followers_url\": \"https://api.github.com/users/garethr/followers\",\r\n \"following_url\": \"https://api.github.com/users/garethr/following{/other_user}\",\r\n \"gists_url\": \"https://api.github.com/users/garethr/gists{/gist_id}\",\r\n \"starred_url\": \"https://api.github.com/users/garethr/starred{/owner}{/repo}\",\r\n \"subscriptions_url\": \"https://api.github.com/users/garethr/subscriptions\",\r\n \"organizations_url\": \"https://api.github.com/users/garethr/orgs\",\r\n \"repos_url\": \"https://api.github.com/users/garethr/repos\",\r\n \"events_url\": \"https://api.github.com/users/garethr/events{/privacy}\",\r\n \"received_events_url\": \"https://api.github.com/users/garethr/received_events\",\r\n \"type\": \"User\",\r\n \"site_admin\": false\r\n },\r\n \"content_type\": \"text/plain; charset=utf-8\",\r\n \"state\": \"uploaded\",\r\n \"size\": 600,\r\n \"download_count\": 2,\r\n \"created_at\": \"2019-03-30T16:56:44Z\",\r\n \"updated_at\": \"2019-03-30T16:56:44Z\",\r\n \"browser_download_url\": \"https://github.com/instrumenta/conftest/releases/download/v0.1.0/checksums.txt\"\r\n }\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 544571092, "label": "Assets table with downloads"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/19#issuecomment-602782437", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/19", "id": 602782437, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjc4MjQzNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T18:36:12Z", "updated_at": "2020-03-23T18:51:23Z", "author_association": "MEMBER", "body": "* `commits`: `message`\r\n* `issue_comments`: `body`\r\n* `issues`: `title`, `body`\r\n* `labels`: `name`, `description`\r\n* `licenses`: `name`\r\n* `milestones`: `title`, `description`\r\n* `releases`: `name`, `body`\r\n* `repos`: `name`, `description`\r\n* `users`: `login`, `name` (maybe `company`, `bio`)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585850715, "label": "Enable full-text search for more stuff (like commits, issues and issue_comments)"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/5#issuecomment-602806530", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/5", "id": 602806530, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjgwNjUzMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T19:23:30Z", "updated_at": "2020-03-23T19:23:30Z", "author_association": "MEMBER", "body": "Yeah `alter=True` means that this isn't necessary.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 493671014, "label": "Add \"incomplete\" boolean to users table for incomplete profiles"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602807178", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18", "id": 602807178, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjgwNzE3OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T19:24:43Z", "updated_at": "2020-03-23T19:24:43Z", "author_association": "MEMBER", "body": "I need to find an example before I work on this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585411547, "label": "Commits in GitHub API can have null author"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602811468", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18", "id": 602811468, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjgxMTQ2OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T19:33:14Z", "updated_at": "2020-03-23T19:33:14Z", "author_association": "MEMBER", "body": "Found one: https://api.github.com/repos/simonw/simonw.github.com/commits", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585411547, "label": "Commits in GitHub API can have null author"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602811785", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18", "id": 602811785, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjgxMTc4NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T19:33:56Z", "updated_at": "2020-03-23T19:34:06Z", "author_association": "MEMBER", "body": "```json\r\n[\r\n {\r\n \"sha\": \"a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0\",\r\n \"node_id\": \"MDY6Q29tbWl0OTMyMDk6YThkYzkxNDA4OWQzOTlkOWI1MjJlYmI1MWI2N2Y5YWMyZThhYTZiMA==\",\r\n \"commit\": {\r\n \"author\": {\r\n \"name\": \"Simon Willison\",\r\n \"email\": \"simon@...\",\r\n \"date\": \"2008-12-18T23:17:12Z\"\r\n },\r\n \"committer\": {\r\n \"name\": \"Simon Willison\",\r\n \"email\": \"simon@...\",\r\n \"date\": \"2008-12-18T23:17:12Z\"\r\n },\r\n \"message\": \"First commit\",\r\n \"tree\": {\r\n \"sha\": \"ac2dfb75e2592c59165c2880f3f7a16dafd452a1\",\r\n \"url\": \"https://api.github.com/repos/simonw/simonw.github.com/git/trees/ac2dfb75e2592c59165c2880f3f7a16dafd452a1\"\r\n },\r\n \"url\": \"https://api.github.com/repos/simonw/simonw.github.com/git/commits/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0\",\r\n \"comment_count\": 0,\r\n \"verification\": {\r\n \"verified\": false,\r\n \"reason\": \"unsigned\",\r\n \"signature\": null,\r\n \"payload\": null\r\n }\r\n },\r\n \"url\": \"https://api.github.com/repos/simonw/simonw.github.com/commits/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0\",\r\n \"html_url\": \"https://github.com/simonw/simonw.github.com/commit/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0\",\r\n \"comments_url\": \"https://api.github.com/repos/simonw/simonw.github.com/commits/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0/comments\",\r\n \"author\": null,\r\n \"committer\": null,\r\n \"parents\": []\r\n }\r\n]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585411547, "label": "Commits in GitHub API can have null author"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602813731", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18", "id": 602813731, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjgxMzczMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T19:38:03Z", "updated_at": "2020-03-23T20:44:49Z", "author_association": "MEMBER", "body": "So it turns out `\"author\"` and `\"committer\"` on the commit are `null` if the email address in the nested `\"commit\"` doesn't match an existing GitHub user.\r\n\r\nMaybe I should be storing the nested data somewhere as well?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585411547, "label": "Commits in GitHub API can have null author"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602815120", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18", "id": 602815120, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjgxNTEyMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T19:40:55Z", "updated_at": "2020-03-23T19:43:19Z", "author_association": "MEMBER", "body": "I could pull a pk-hashed version of the name/email into separate `raw_author` and `raw_committer` columns perhaps - against a `commit_authors` table. Could be interesting.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585411547, "label": "Commits in GitHub API can have null author"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602846293", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18", "id": 602846293, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjg0NjI5Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T20:44:40Z", "updated_at": "2020-03-23T20:44:40Z", "author_association": "MEMBER", "body": "I implemented the `raw_authors` idea.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585411547, "label": "Commits in GitHub API can have null author"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602861730", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602861730, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjg2MTczMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T21:19:16Z", "updated_at": "2020-03-23T21:19:42Z", "author_association": "MEMBER", "body": "I'm going to do this with a scheduled GitHub Action in this repo.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602862236", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602862236, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjg2MjIzNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T21:20:26Z", "updated_at": "2020-03-23T21:20:26Z", "author_association": "MEMBER", "body": "I'll run the `commits` and `issues` and `issue-comments` commands in addition to the `releases` command.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602862967", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602862967, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjg2Mjk2Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T21:22:04Z", "updated_at": "2020-03-23T21:22:04Z", "author_association": "MEMBER", "body": "Following these instructions: https://simonwillison.net/2020/Jan/21/github-actions-cloud-run/", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602895896", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602895896, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjg5NTg5Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T22:42:25Z", "updated_at": "2020-03-23T22:42:25Z", "author_association": "MEMBER", "body": "Urgh this is such a mess! I should have done this on a branch / pull request to avoid polluting my main master history, but never mind.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/21#issuecomment-602896434", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/21", "id": 602896434, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjg5NjQzNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T22:43:37Z", "updated_at": "2020-03-23T22:43:37Z", "author_association": "MEMBER", "body": "I'm going to do this now to help figure out the latest error in #13:\r\n```\r\nTraceback (most recent call last):\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/bin/github-to-sqlite\", line 11, in \r\n load_entry_point('github-to-sqlite', 'console_scripts', 'github-to-sqlite')()\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py\", line 237, in commits\r\n utils.save_commits(db, commits, repo_full[\"id\"])\r\n File \"/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py\", line 345, in save_commits\r\n for commit in commits:\r\n File \"/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py\", line 207, in fetch_commits\r\n if stop_when(commit):\r\n File \"/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py\", line 224, in stop_when\r\n db[\"commits\"].get(commit[\"sha\"])\r\nTypeError: string indices must be integers\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 586561727, "label": "Turn GitHub API errors into exceptions"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602916947", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602916947, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjkxNjk0Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T23:38:06Z", "updated_at": "2020-03-23T23:38:06Z", "author_association": "MEMBER", "body": "Woohoo! https://github-to-sqlite-j7hipcg4aq-uc.a.run.app/", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602917713", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602917713, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjkxNzcxMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T23:40:29Z", "updated_at": "2020-03-23T23:40:29Z", "author_association": "MEMBER", "body": "Most recently updated issues across all Dogsheep repos, with faceting: https://github-to-sqlite-j7hipcg4aq-uc.a.run.app/github/issues?_facet=repo&_facet=user&_facet=state&_facet=author_association&_facet=type&_sort_desc=updated_at", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602918689", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602918689, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjkxODY4OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T23:43:39Z", "updated_at": "2020-03-23T23:47:50Z", "author_association": "MEMBER", "body": "I pointed https://github-to-sqlite.dogsheep.net/ at it. May take a few minutes for the new certificate to provision though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602919058", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602919058, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjkxOTA1OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T23:44:48Z", "updated_at": "2020-03-23T23:44:48Z", "author_association": "MEMBER", "body": "Next step: use a `metadata.json` file to add some extras. And add the `datasette-render-markdown` plugin as soon as I ship https://github.com/simonw/datasette-render-markdown/issues/2 (GFM support).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/21#issuecomment-602920163", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/21", "id": 602920163, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjkyMDE2Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T23:48:22Z", "updated_at": "2020-03-23T23:48:22Z", "author_association": "MEMBER", "body": "I'm happy with this pattern:\r\nhttps://github.com/dogsheep/github-to-sqlite/blob/f78c4e9baaf0970ffab266ba780df7240aae9f32/github_to_sqlite/utils.py#L4-L18", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 586561727, "label": "Turn GitHub API errors into exceptions"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602924714", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602924714, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjkyNDcxNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-24T00:03:25Z", "updated_at": "2020-03-24T00:03:25Z", "author_association": "MEMBER", "body": "This is good enough for the 1.0 release.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/23#issuecomment-602928533", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/23", "id": 602928533, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjkyODUzMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-24T00:15:49Z", "updated_at": "2020-03-24T00:15:49Z", "author_association": "MEMBER", "body": "https://github.com/dogsheep/github-to-sqlite/releases/tag/1.0", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 586595839, "label": "Release 1.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/pocket-to-sqlite/issues/1#issuecomment-605316146", "issue_url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/1", "id": 605316146, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNTMxNjE0Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-27T21:09:15Z", "updated_at": "2020-03-27T21:09:22Z", "author_association": "MEMBER", "body": "For a progress bar I need to know how many total items there are.\r\n\r\nI found an undocumented API for this! `/v3/stats` which returns:\r\n```json\r\n{\r\n \"count_list\": 7394,\r\n \"count_read\": 1016,\r\n \"count_unread\": 6378,\r\n \"status\": 1\r\n}\r\n```\r\nI guessed this based on the documented v2 API: https://getpocket.com/api/v2_docs/#stats", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 503233021, "label": "Use better pagination (and implement progress bar)"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/pocket-to-sqlite/issues/1#issuecomment-605325897", "issue_url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/1", "id": 605325897, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNTMyNTg5Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-27T21:37:26Z", "updated_at": "2020-03-27T21:38:37Z", "author_association": "MEMBER", "body": "I keep getting 503 errors even though I appear to be staying within the rate limit:\r\n```\r\n{'Date': 'Fri, 27 Mar 2020 21:35:57 GMT', 'Content-Type': 'application/json', 'Transfer-Encoding':\r\n'chunked', 'Connection': 'keep-alive', 'Server': 'Apache/2.4.25 (Debian)', 'Content-Location':\r\n'get.php', 'Vary': 'negotiate', 'TCN': 'choice', 'Set-Cookie': '...; httponly',\r\n'X-Frame-Options': 'SAMEORIGIN', 'Status': '200 OK', 'X-Limit-Key-Limit': '10000',\r\n'X-Limit-Key-Remaining': '9960', 'X-Limit-Key-Reset': '282', 'X-Source': 'Pocket',\r\n'P3P': 'policyref=\"/w3c/p3p.xml\", CP=\"ALL CURa ADMa DEVa OUR IND UNI COM NAV INT STA PRE\"'}\r\n\r\n [##----------------------------------] 6% 06:49:27\r\n\r\n{'Date': 'Fri, 27 Mar 2020 21:36:06 GMT', 'Content-Type': 'text/html; charset=UTF-8', 'Content-Length':\r\n'23', 'Connection': 'keep-alive', 'Server': 'Apache/2.4.25 (Debian)', 'Content-Location': 'get.php',\r\n'Vary': 'negotiate', 'TCN': 'choice', 'Set-Cookie': '...', 'X-Frame-Options': 'SAMEORIGIN',\r\n'X-Error': 'Pocket is currently under heavy load. Please wait a moment and try again.',\r\n'X-Error-Code': '199', 'Status': '503 Service Unavailable', 'X-Source': 'Pocket',\r\n'P3P': 'policyref=\"/w3c/p3p.xml\", CP=\"ALL CURa ADMa DEVa OUR IND UNI COM NAV INT STA PRE\"'}\r\n```\r\nI'm going to try doing a few automatic retries any time I see a 503 error.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 503233021, "label": "Use better pagination (and implement progress bar)"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/pocket-to-sqlite/issues/1#issuecomment-605327655", "issue_url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/1", "id": 605327655, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNTMyNzY1NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-27T21:42:49Z", "updated_at": "2020-03-27T21:42:49Z", "author_association": "MEMBER", "body": "Or maybe it was because of the current Google Cloud outage? https://news.ycombinator.com/item?id=22706677", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 503233021, "label": "Use better pagination (and implement progress bar)"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/pocket-to-sqlite/issues/2#issuecomment-605337941", "issue_url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/2", "id": 605337941, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNTMzNzk0MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-27T22:16:32Z", "updated_at": "2020-03-27T22:16:32Z", "author_association": "MEMBER", "body": "Need to test this. I have 7,394 items in my database right now. I'm going to save a new thing.\r\n\r\nThen I ran this:\r\n```\r\npocket-to-sqlite fetch pocket-simon.db\r\n```\r\nAnd it worked!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 503234169, "label": "Track and use the 'since' value"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/pocket-to-sqlite/issues/2#issuecomment-605338322", "issue_url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/2", "id": 605338322, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNTMzODMyMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-27T22:18:02Z", "updated_at": "2020-03-27T22:18:02Z", "author_association": "MEMBER", "body": "Just needs documentation now.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 503234169, "label": "Track and use the 'since' value"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/swarm-to-sqlite/pull/6#issuecomment-605382373", "issue_url": "https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/6", "id": 605382373, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNTM4MjM3Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-28T02:27:32Z", "updated_at": "2020-03-28T02:27:32Z", "author_association": "MEMBER", "body": "Thanks!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 543355051, "label": "don't break if source is missing"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606304837", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606304837, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjMwNDgzNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-30T23:27:50Z", "updated_at": "2020-03-30T23:29:31Z", "author_association": "MEMBER", "body": "One option would be something like this:\r\n```sql\r\nselect max(id) from tweets\r\nwhere user = ?\r\nand not exists (select id from tweets where retweeted_status = id)\r\nand not exists (select id from tweets where quoted_status = id)\r\nand not exists (select id from tweets where in_reply_to_status_id = id)\r\n```\r\nMight be a good idea to index those columns (after confirming that doing so would indeed speed up the query).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606305701", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606305701, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjMwNTcwMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-30T23:30:27Z", "updated_at": "2020-03-30T23:30:27Z", "author_association": "MEMBER", "body": "A better alternative would be to maintain a separate table with the last seen since value for when we ran `user-timeline` for any specific user.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-606307019", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40", "id": 606307019, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjMwNzAxOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-30T23:34:27Z", "updated_at": "2020-03-30T23:34:27Z", "author_association": "MEMBER", "body": "The count properties available for a user are:\r\n\r\n* followers_count\r\n* friends_count\r\n* listed_count\r\n* favourites_count\r\n* statuses_count\r\n\r\nMay as well track history for all of them? Should be pretty cheap to store.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590669793, "label": "Feature: record history of follower counts"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-606307376", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40", "id": 606307376, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjMwNzM3Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-30T23:35:40Z", "updated_at": "2020-03-30T23:39:15Z", "author_association": "MEMBER", "body": "I think five separate tables:\r\n\r\n* followers_count_history\r\n* friends_count_history\r\n* listed_count_history\r\n* favourites_count_history\r\n* statuses_count_history\r\n\r\nEach with the following structure:\r\n\r\n* datetime (ISO UTC)\r\n* user (ID, foreign key to users)\r\n* count (integer)\r\n\r\nI'm tempted to have a compound primary key here - user, datetime\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590669793, "label": "Feature: record history of follower counts"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606309165", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606309165, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjMwOTE2NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-30T23:41:31Z", "updated_at": "2020-03-30T23:41:31Z", "author_association": "MEMBER", "body": "I like the separate `user_timeline_since` table solution.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606824992", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606824992, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjgyNDk5Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-31T19:24:23Z", "updated_at": "2020-03-31T19:24:23Z", "author_association": "MEMBER", "body": "The `--since` option is actually used by four commands:\r\n\r\n* `user-timeline`\r\n* `home-timeline`\r\n* `mentions-timeline`\r\n* `search`\r\n\r\nAll of them use the same `fetch_timeline()` utility function under the hood. I should move the logic that looks up the last `since_id` into that shared function.\r\n\r\nQuestion: should I have a table for each of those four methods or a single table that is used by them all? I'm leaning towards four separate tables.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606843224", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606843224, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjg0MzIyNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-31T19:59:11Z", "updated_at": "2020-03-31T20:06:32Z", "author_association": "MEMBER", "body": "Or... have a single `since_ids` table to track since values, and have its primary key be a string that looks something like this:\r\n\r\n`user:123145`\r\n`home:23441`\r\n`mentions:23425`\r\n`search:99ff9cefff5cbfd804f7cd43e2b27ced8addbe8d`\r\n\r\nThat last example would use the hash generated here:\r\n\r\nhttps://github.com/dogsheep/twitter-to-sqlite/blob/810cb2af5a175837204389fd7f4b5721f8b325ab/twitter_to_sqlite/cli.py#L792-L808", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606844521", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606844521, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjg0NDUyMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-31T20:01:39Z", "updated_at": "2020-03-31T20:01:39Z", "author_association": "MEMBER", "body": "I think `utils.fetch_timeline()` grows a new argument, `since_key`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606850008", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606850008, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjg1MDAwOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-31T20:13:59Z", "updated_at": "2020-04-01T00:23:00Z", "author_association": "MEMBER", "body": "Table design for `since_ids` table:\r\n\r\ntype | key | since_id\r\n--- | --- | ---\r\n1 | 124324 | 2347239847293\r\n2 | 99ff9cefff5cbfd804f7cd43e2b27ced8addbe8d | 2125947927344\r\n\r\nPrimary compound key on `(category, key)`\r\n\r\n`type` is also a foreign key to a `since_id_types` table with `id` and `name` columns (probably created using https://sqlite-utils.readthedocs.io/en/stable/python-api.html#working-with-lookup-tables )", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606850453", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606850453, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjg1MDQ1Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-31T20:14:58Z", "updated_at": "2020-04-01T03:03:50Z", "author_association": "MEMBER", "body": "Actually I'll hard-code the population of `since_id_types` to get known ID constants.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606998669", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606998669, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjk5ODY2OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-01T02:57:36Z", "updated_at": "2020-04-01T02:57:36Z", "author_association": "MEMBER", "body": "The tricky thing here is thinking about the interaction between the recorded since_id and a desire to run the initial import.\r\n\r\nThe first time you run `twitter-to-sqlite user-timeline db.db username` we want to fetch as many tweets from that user as possible - probably around 3,200 before the API limitations cut us off.\r\n\r\nWe need to record the maximum ID from those as the `since_id` - which we will see on the very first page we paginate through. That way next time we run the command with `--since` we will only fetch new tweets.\r\n\r\nBut what happens if our initial import is cancelled after only a few tweets? We risk never pulling in the rest of the tweets.\r\n\r\nNot sure if I need to solve this at all or if I should instead trust users to run the command a second time without `--since` if they think they didn't retrieve anything the first time.\r\n\r\nI had considered letting `--stop_after=` over-ride `--since` but that doesn't actually make sense - if you send a since_id to the Twitter API you'll never get back more tweets than exist after that ID, so the `--stop_after` would not make a meaningful difference.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-607003655", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 607003655, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNzAwMzY1NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-01T03:18:00Z", "updated_at": "2020-04-01T03:18:00Z", "author_association": "MEMBER", "body": "I've got this working for the `user-timeline` command.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-607010634", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 607010634, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNzAxMDYzNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-01T03:45:16Z", "updated_at": "2020-04-01T03:45:16Z", "author_association": "MEMBER", "body": "OK, fix is applied to everything now.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/10#issuecomment-607010791", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10", "id": 607010791, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNzAxMDc5MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-01T03:45:48Z", "updated_at": "2020-04-01T03:45:48Z", "author_association": "MEMBER", "body": "I'm happy with the recent work I did on this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 492297930, "label": "Rethink progress bars for various commands"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-607011421", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40", "id": 607011421, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNzAxMTQyMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-01T03:47:37Z", "updated_at": "2020-04-01T03:55:08Z", "author_association": "MEMBER", "body": "Actually a single table with a `type` integer ID referencing a `count_history_types` table would better match the way I implemented the `since_ids` table:\r\n\r\nhttps://github.com/dogsheep/twitter-to-sqlite/blob/4b6c8d8c1cc6fefdb566ec8506157133f47c569a/twitter_to_sqlite/utils.py#L331-L341\r\n\r\nIn which case the compound primary key would be `type`, `user`, `datetime`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590669793, "label": "Feature: record history of follower counts"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-607011972", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40", "id": 607011972, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNzAxMTk3Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-01T03:49:02Z", "updated_at": "2020-04-01T03:50:01Z", "author_association": "MEMBER", "body": "I want the datetime value to look like `2020-04-01T03:34:58+00:00` (the format returned by the Twitter API which I am storing in other tables at the moment).\r\n```\r\n>>> datetime.utcnow().isoformat().split('.')[0] + '+00:00'\r\n'2020-04-01T03:49:52+00:00'\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590669793, "label": "Feature: record history of follower counts"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-607019151", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40", "id": 607019151, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNzAxOTE1MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-01T04:11:10Z", "updated_at": "2020-04-01T04:11:10Z", "author_association": "MEMBER", "body": "In testing this collects a LOT of data. I'm going to skip tracking favourites_count and statuses_count and just track followers, friends and listed instead.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590669793, "label": "Feature: record history of follower counts"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/16#issuecomment-613611455", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/16", "id": 613611455, "node_id": "MDEyOklzc3VlQ29tbWVudDYxMzYxMTQ1NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-14T18:37:21Z", "updated_at": "2020-04-14T18:37:21Z", "author_association": "MEMBER", "body": "This should have been fixed by #20 and #23 \r\n\r\n@jayvdb I'm definitely interested in this tool working as a library - it's purely designed as a CLI tool at the moment, but cleaning it up to work better as a dependency is totally in-scope for the project. https://sqlite-utils.readthedocs.io/ is an example of a tool I've built that works for both.\r\n\r\nFeel free to open a new issue here with some notes on what you would need for this to work as a library for your project!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 546051181, "label": "Exception running first command: IndexError: list index out of range"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/14#issuecomment-613641947", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14", "id": 613641947, "node_id": "MDEyOklzc3VlQ29tbWVudDYxMzY0MTk0Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-14T19:38:24Z", "updated_at": "2020-04-14T19:38:34Z", "author_association": "MEMBER", "body": "Since events include payloads with full object representations in them (for issues, repos and more) running this command every few minutes may be all it takes to keep a constant copy of everything updated in a very rate-limit friendly manner (thanks to the ETags).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 530491074, "label": "Command for importing events"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/26#issuecomment-614794739", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/26", "id": 614794739, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNDc5NDczOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-16T17:38:28Z", "updated_at": "2020-04-16T17:38:28Z", "author_association": "MEMBER", "body": "I'm already doing this here: https://github.com/dogsheep/github-to-sqlite/blob/c4aaa50e167cfa9021c7c94260bc3e89e10947bf/github_to_sqlite/utils.py#L246-L250", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601271612, "label": "Topics are missing from repositories"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/26#issuecomment-614795712", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/26", "id": 614795712, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNDc5NTcxMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-16T17:40:27Z", "updated_at": "2020-04-16T17:40:27Z", "author_association": "MEMBER", "body": "Aha! it was missing from the `fetch_repo()` function.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601271612, "label": "Topics are missing from repositories"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/25#issuecomment-614810417", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/25", "id": 614810417, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNDgxMDQxNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-16T18:07:11Z", "updated_at": "2020-04-16T18:07:11Z", "author_association": "MEMBER", "body": "Turns out the main problem was #26 - now fixed.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601265023, "label": "Improvements to demo instance"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-614831451", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27", "id": 614831451, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNDgzMTQ1MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-16T18:47:25Z", "updated_at": "2020-04-16T18:47:25Z", "author_association": "MEMBER", "body": "Is it possible for a repo to have an `owner` that differs from its `organization`?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601330277, "label": "Repos have a big blob of JSON in the organization column"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-614831842", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27", "id": 614831842, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNDgzMTg0Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-16T18:48:18Z", "updated_at": "2020-04-16T18:48:18Z", "author_association": "MEMBER", "body": "I'm going to make `organization` another foreign key to the `users` table just in case it IS possible (maybe with GitHub Enterprise or similar?)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601330277, "label": "Repos have a big blob of JSON in the organization column"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-614843406", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27", "id": 614843406, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNDg0MzQwNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-16T19:11:53Z", "updated_at": "2020-04-16T19:20:23Z", "author_association": "MEMBER", "body": "This didn't quite work: the column type is incorrect, so the foreign key relationship isn't sticking:\r\n\r\nhttps://github-to-sqlite.dogsheep.net/github/repos?organization=53015001\r\n\r\n`[organization] TEXT REFERENCES [users]([id])` - should be `INTEGER`.\r\n\r\nThe problem is that if the first repo inserted has no organization it's set to `null`, which `sqlite-utils` derives as a `TEXT` column.\r\n\r\nOne solution would be to create the column explicitly with a type, but this could get messy.\r\n\r\nI think I want a new sqlite-utils feature for this instead.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601330277, "label": "Repos have a big blob of JSON in the organization column"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/37#issuecomment-615509578", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/37", "id": 615509578, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTUwOTU3OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-17T23:36:00Z", "updated_at": "2020-04-17T23:36:00Z", "author_association": "MEMBER", "body": "```\r\n$ twitter-to-sqlite user-timeline doggo.db doggoenthuonetuh\r\nTraceback (most recent call last):\r\n...\r\n File \"/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/utils.py\", line 272, in transform_user\r\n user[\"created_at\"] = parser.parse(user[\"created_at\"])\r\nKeyError: 'created_at'\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585353598, "label": "Handle \"User not found\" error"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/37#issuecomment-615509803", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/37", "id": 615509803, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTUwOTgwMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-17T23:36:40Z", "updated_at": "2020-04-17T23:36:40Z", "author_association": "MEMBER", "body": "I'm going to print a warning to stderr, skip and continue - because if you have 100 screen names and only one of them is invalid you should still execute for the other 99.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585353598, "label": "Handle \"User not found\" error"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/37#issuecomment-615510361", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/37", "id": 615510361, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTUxMDM2MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-17T23:38:27Z", "updated_at": "2020-04-17T23:38:27Z", "author_association": "MEMBER", "body": "That's a bit tricky since I'd have to rewrite the internals of a bunch of other commands. For the moment I'll exit the script with an error but at least it will be a decent error!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585353598, "label": "Handle \"User not found\" error"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/43#issuecomment-615513491", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/43", "id": 615513491, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTUxMzQ5MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-17T23:48:28Z", "updated_at": "2020-04-17T23:48:28Z", "author_association": "MEMBER", "body": "Released in 0.21.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602176870, "label": "\"twitter-to-sqlite lists\" command for retrieving a user's owned lists"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-615518606", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27", "id": 615518606, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTUxODYwNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T00:14:32Z", "updated_at": "2020-04-18T00:14:32Z", "author_association": "MEMBER", "body": "https://github.com/simonw/sqlite-utils/issues/100 is done and released in sqlite-utils 2.7.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601330277, "label": "Repos have a big blob of JSON in the organization column"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-615519409", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27", "id": 615519409, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTUxOTQwOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T00:19:16Z", "updated_at": "2020-04-18T00:19:16Z", "author_association": "MEMBER", "body": "```\r\n$ github-to-sqlite repos b.db dogsheep\r\n$ sqlite3 b.db '.schema repos'\r\nCREATE TABLE [repos] (\r\n [id] INTEGER PRIMARY KEY,\r\n...\r\n [permissions] TEXT,\r\n [organization] INTEGER REFERENCES [users]([id]),\r\n FOREIGN KEY(license) REFERENCES licenses(key)\r\n);\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601330277, "label": "Repos have a big blob of JSON in the organization column"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/28#issuecomment-615883040", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/28", "id": 615883040, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTg4MzA0MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T14:45:38Z", "updated_at": "2020-04-18T14:45:38Z", "author_association": "MEMBER", "body": "```\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py\", line 219, in contributors\r\n utils.save_contributors(db, contributors, repo_full[\"id\"])\r\n File \"/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py\", line 354, in save_contributors\r\n for contributor in contributors:\r\n File \"/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py\", line 228, in fetch_contributors\r\n for contributors in paginate(url, headers):\r\n File \"/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py\", line 286, in paginate\r\n data = response.json()\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/requests/models.py\", line 898, in json\r\n return complexjson.loads(self.text, **kwargs)\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/json/__init__.py\", line 357, in loads\r\n return _default_decoder.decode(s)\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/json/decoder.py\", line 337, in decode\r\n obj, end = self.raw_decode(s, idx=_w(s, 0).end())\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/json/decoder.py\", line 355, in raw_decode\r\n raise JSONDecodeError(\"Expecting value\", s, err.value) from None\r\njson.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601333634, "label": "Pull repository contributors"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/28#issuecomment-615883687", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/28", "id": 615883687, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTg4MzY4Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T14:49:58Z", "updated_at": "2020-04-18T14:49:58Z", "author_association": "MEMBER", "body": "That happened trying to pull contributors for `dogsheep/beta` - an empty repository. Turns out it was returning a `204 no content`:\r\n```\r\n~ $ curl -i 'https://api.github.com/repos/dogsheep/beta/contributors'\r\nHTTP/1.1 204 No Content\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601333634, "label": "Pull repository contributors"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/28#issuecomment-615886206", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/28", "id": 615886206, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTg4NjIwNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T15:04:59Z", "updated_at": "2020-04-18T15:04:59Z", "author_association": "MEMBER", "body": "Demo: https://github-to-sqlite.dogsheep.net/github/contributors\r\n\r\nDocumentation: https://github.com/dogsheep/github-to-sqlite/blob/13f8868fb5efa01c263b24f6dd91c617e6e938e1/README.md#fetching-contributors-to-a-repository ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601333634, "label": "Pull repository contributors"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/2#issuecomment-615931488", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/2", "id": 615931488, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTkzMTQ4OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T19:24:02Z", "updated_at": "2020-04-18T19:24:02Z", "author_association": "MEMBER", "body": "I made a start on this last week with a https://github.com/simonw/heic-to-jpeg proxy.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533352, "label": "Ability to convert HEIC images to JPEG"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615932007", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615932007, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTkzMjAwNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T19:27:55Z", "updated_at": "2020-04-18T19:27:55Z", "author_association": "MEMBER", "body": "Research thread: https://twitter.com/simonw/status/1249049694984011776\r\n> I want to build some software that lets people store their own data in their own S3 bucket, but if possible I'd like not to have to teach people the incantations needed to get their bucket setup and minimum-permission credentials figures out\r\n\r\nhttps://testdriven.io/blog/storing-django-static-and-media-files-on-amazon-s3/ looks useful", "reactions": "{\"total_count\": 2, \"+1\": 2, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615932204", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615932204, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTkzMjIwNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T19:29:22Z", "updated_at": "2020-04-18T19:34:44Z", "author_association": "MEMBER", "body": "I'm going to call my bucket `dogsheep-photos-simon`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615933273", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615933273, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTkzMzI3Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T19:37:33Z", "updated_at": "2020-04-18T19:37:33Z", "author_association": "MEMBER", "body": "https://console.aws.amazon.com/s3/bucket/create?region=us-west-1\r\n\r\n![S3_Management_Console](https://user-images.githubusercontent.com/9599/79669552-33e2a380-8171-11ea-9ab5-5785d34f652a.png)\r\n\r\nI created it with no public read-write access. I plan to use signed URLs via a transforming proxy to access images for display on the web.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615935577", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615935577, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTkzNTU3Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T19:54:59Z", "updated_at": "2020-04-18T19:55:30Z", "author_association": "MEMBER", "body": "Creating IAM groups called `dogsheep-photos-simon-read-write` and `dogsheep-photos-simon-read`: https://console.aws.amazon.com/iam/home#/groups - I created them with no attached policies.\r\n\r\nNow I can attach an \"inline policy\" to each one. For the read-write group I go here:\r\n\r\nhttps://console.aws.amazon.com/iam/home#/groups/dogsheep-photos-simon-read-write\r\n\r\n![IAM_Management_Console](https://user-images.githubusercontent.com/9599/79669703-2d086080-8172-11ea-9597-83e0b155193e.png)\r\n\r\nExample policies are here: https://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html\r\n\r\nFor the read-write one I went with:\r\n```json\r\n{\r\n \"Version\": \"2012-10-17\",\r\n \"Statement\": [\r\n {\r\n \"Effect\": \"Allow\",\r\n \"Action\": \"s3:*\",\r\n \"Resource\": [\r\n \"arn:aws:s3:::dogsheep-photos-simon/*\"\r\n ]\r\n }\r\n ]\r\n}\r\n```\r\nFor the read-only policy I'm going to guess that this is appropriate:\r\n\r\n```json\r\n{\r\n \"Version\": \"2012-10-17\",\r\n \"Statement\": [\r\n {\r\n \"Effect\": \"Allow\",\r\n \"Action\": [\r\n \"s3:GetObject*\",\r\n \"s3:ListBucket\"\r\n ],\r\n \"Resource\": [\r\n \"arn:aws:s3:::dogsheep-photos-simon/*\"\r\n ]\r\n }\r\n ]\r\n}\r\n```\r\nI tried the policy simulator to test this out: https://policysim.aws.amazon.com/home/index.jsp?#groups/dogsheep-photos-simon-read - this worked:\r\n\r\n![IAM_Policy_Simulator](https://user-images.githubusercontent.com/9599/79669893-cd12b980-8173-11ea-8dfb-5660ce3652da.png)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615936880", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615936880, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTkzNjg4MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T20:04:31Z", "updated_at": "2020-04-18T20:04:31Z", "author_association": "MEMBER", "body": "Next step: create two IAM users, one for each of those groups.\r\n\r\nhttps://console.aws.amazon.com/iam/home#/users$new?step=details\r\n\r\n![IAM_Management_Console](https://user-images.githubusercontent.com/9599/79669931-1bc05380-8174-11ea-9657-0e0c6a692d42.png)\r\n\r\n![IAM_Management_Console](https://user-images.githubusercontent.com/9599/79669941-27137f00-8174-11ea-8ce7-249f0d4f96f6.png)\r\n\r\nI copied the keys into a secure note in 1password.\r\n\r\nCouldn't get into Transmit with them though! https://library.panic.com/transmit/transmit5/iam-roles/ may help.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615941746", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615941746, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk0MTc0Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T20:29:36Z", "updated_at": "2020-04-18T20:29:36Z", "author_association": "MEMBER", "body": "I'm going to create another user just for Transmit, with full S3 access.\r\n\r\nname: `dogsheep-photos-simon-s3-all-access`\r\n\r\nRather than creating a group for that user, I'm trying the \"Attach existing policies directly\" option:\r\n\r\n![IAM_Management_Console](https://user-images.githubusercontent.com/9599/79670182-03513880-8176-11ea-811a-c80aefb4538a.png)\r\n\r\nThat user DID work with Transmit. I uploaded a test HEIC image. I used Transmit to copy a signed URL for it.\r\n\r\n```\r\n~ $ curl -i 'https://dogsheep-photos-simon.s3.us-west-1.amazonaws.com/IMG_7195.HEIC?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAWXFXAI...' | head -n 100\r\n % Total % Received % Xferd Average Speed Time Time Time Current\r\n Dload Upload Total Spent Left Speed\r\n 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0HTTP/1.1 200 OK\r\nx-amz-id-2: gBOCYqZfbNAnv0R/uJ++qm2NbW5SgD4TapgF9RQjzzeDIThcCz/BkKU+YoxlG4NJHlcmMgAHyh4=\r\nx-amz-request-id: C2FE7FCC3BD53A84\r\nDate: Sat, 18 Apr 2020 20:28:54 GMT\r\nLast-Modified: Sat, 18 Apr 2020 20:13:49 GMT\r\nETag: \"fe3e081239a123ef745517878c53b854\"\r\nAccept-Ranges: bytes\r\nContent-Type: image/heic\r\nContent-Length: 1913097\r\nServer: AmazonS3\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615942116", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615942116, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk0MjExNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T20:30:56Z", "updated_at": "2020-04-18T20:30:56Z", "author_association": "MEMBER", "body": "Next step: attempt a programmatic upload using the `dogsheep-photos-simon-read-write` credentials from a Jupyter notebook.\r\n\r\nAlso attempt a programmatic bucket listing and read using `dogsheep-photos-simon-read` credentials.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615944806", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615944806, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk0NDgwNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T20:41:39Z", "updated_at": "2020-04-18T20:41:39Z", "author_association": "MEMBER", "body": "This worked!\r\n\r\n![Dogsheep_Photos_S3_access](https://user-images.githubusercontent.com/9599/79670712-d868e380-8179-11ea-82a5-5dfd17356113.png)\r\n\r\nAnd this worked:\r\n\r\n![Dogsheep_Photos_S3_access](https://user-images.githubusercontent.com/9599/79670777-50370e00-817a-11ea-83cd-18ebf5702878.png)\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615945056", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615945056, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk0NTA1Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T20:42:41Z", "updated_at": "2020-04-18T20:42:41Z", "author_association": "MEMBER", "body": "But... `list_objects` failed for both of my keys (read and write):\r\n\r\n![Dogsheep_Photos_S3_access](https://user-images.githubusercontent.com/9599/79670798-75c41780-817a-11ea-9907-2cbc4a2e497c.png)\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615946537", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615946537, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk0NjUzNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T20:48:13Z", "updated_at": "2020-04-18T20:48:13Z", "author_association": "MEMBER", "body": "How about generating a signed URL?\r\n```python\r\nread_client.generate_presigned_url(\r\n \"get_object\",\r\n Params={\r\n \"Bucket\": \"dogsheep-photos-simon\",\r\n \"Key\": \"this_is_fine.jpg\",\r\n },\r\n ExpiresIn=600\r\n)\r\n```\r\nGave me https://dogsheep-photos-simon.s3.amazonaws.com/this_is_fine.jpg?AWSAccessKeyId=AKIAWXFXAIOZNZ3JFO7I&Signature=x1zrS4w4OTGAACd7yHp9mYqXvN8%3D&Expires=1587243398\r\n\r\nWhich does this:\r\n\r\n```\r\n~ $ curl -i 'https://dogsheep-photos-simon.s3.amazonaws.com/this_is_fine.jpg?AWSAccessKeyId=AKIAWXFXAIOZNZ3JFO7I&Signature=x1zrS4w4OTGAACd7yHp9mYqXvN8%3D&Expires=1587243398'\r\nHTTP/1.1 307 Temporary Redirect\r\nx-amz-bucket-region: us-west-1\r\nx-amz-request-id: E78CD859AEE21D33\r\nx-amz-id-2: 648mx+1+YSGga7NDOU7Q6isfsKnEPWOLC+DI4+x2o9FCc6pSCdIaoHJUbFMI8Vsuh1ADtx46ymU=\r\nLocation: https://dogsheep-photos-simon.s3-us-west-1.amazonaws.com/this_is_fine.jpg?AWSAccessKeyId=AKIAWXFXAIOZNZ3JFO7I&Signature=x1zrS4w4OTGAACd7yHp9mYqXvN8%3D&Expires=1587243398\r\nContent-Type: application/xml\r\nTransfer-Encoding: chunked\r\nDate: Sat, 18 Apr 2020 20:47:21 GMT\r\nServer: AmazonS3\r\n\r\n\r\nTemporaryRedirectPlease re-send this request to the specified temporary endpoint. Continue to use the original request endpoint for future requests.dogsheep-photos-simon.s3-us-west-1.amazonaws.comdogsheep-photos-simonE78CD859AEE21D33648mx+1+YSGga7NDOU7Q6isfsKnEPWOLC+DI4+x2o9FCc6pSCdIaoHJUbFMI8Vsuh1ADtx46ymU=~ $ \r\n```\r\nSo it redirects to another URL... which returns this:\r\n```\r\n~ $ curl -i 'https://dogsheep-photos-simon.s3-us-west-1.amazonaws.com/this_is_fine.jpg?AWSAccessKeyId=AKIAWXFXAIOZNZ3JFO7I&Signature=x1zrS4w4OTGAACd7yHp9mYqXvN8%3D&Expires=1587243398'\r\nHTTP/1.1 200 OK\r\nx-amz-id-2: XafOl6mswj3yz0GJC9+Ptot1ll5sROVwqsMc10CUUfgpaUANTdIx2GhnONb5d1GVFJ6wlS2j3UY=\r\nx-amz-request-id: 258387C180411AFE\r\nDate: Sat, 18 Apr 2020 20:47:52 GMT\r\nLast-Modified: Sat, 18 Apr 2020 20:37:35 GMT\r\nETag: \"ee04081c3182a44a1c6944e94012e977\"\r\nAccept-Ranges: bytes\r\nContent-Type: binary/octet-stream\r\nContent-Length: 53072\r\nServer: AmazonS3\r\n\r\n????JFIF??C\r\n```\r\nSo that worked! It did come back with `Content-Type: binary/octet-stream` though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615947229", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615947229, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk0NzIyOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T20:51:26Z", "updated_at": "2020-04-18T20:51:26Z", "author_association": "MEMBER", "body": "Running the upload again like this resulted in the correct content-type:\r\n```python\r\nclient.upload_file(\r\n \"/Users/simonw/Desktop/this_is_fine.jpg\",\r\n \"dogsheep-photos-simon\",\r\n \"this_is_fine.jpg\",\r\n ExtraArgs={\r\n \"ContentType\": \"image/jpeg\"\r\n }\r\n)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615947370", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615947370, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk0NzM3MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T20:52:13Z", "updated_at": "2020-04-18T20:52:13Z", "author_association": "MEMBER", "body": "This is great! I now have a key that can upload photos, and a separate key that can download photos OR generate signed URLs to access those photos.\r\n\r\nNext step: a script that starts uploading my photos.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615948102", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615948102, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk0ODEwMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T20:56:59Z", "updated_at": "2020-04-18T20:56:59Z", "author_association": "MEMBER", "body": "I'm going to start with this:\r\n\r\n`photos-to-sqlite upload photos.db ~/path/to/directory`\r\n\r\nThis will scan the provided directory (and all sub-directories) for image files. It will then:\r\n\r\n* Calculate a sha256 of the contents of that file\r\n* Upload the file to a key that's `sha256.jpg` or `.heic`\r\n* Upload a `sha256.json` file with the original path to the image\r\n* Add that image to a `uploads` table in `photos.db`\r\n\r\nStretch goal: grab the EXIF data and include that in the `.json` upload AND the `uploads` database table.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/5#issuecomment-615949574", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/5", "id": 615949574, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk0OTU3NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T21:06:07Z", "updated_at": "2020-04-18T21:06:07Z", "author_association": "MEMBER", "body": "```\r\n$ photos-to-sqlite s3-auth\r\nCreate S3 credentials and paste them here:\r\n\r\nAccess key ID: xxx\r\nSecret access key: yyy\r\n$ cat auth.json \r\n{\r\n \"access_key_id\": \"xxx\",\r\n \"secret_access_key\": \"yyy\"\r\n}\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602551638, "label": "photos-to-sqlite s3-auth command"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615957385", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615957385, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk1NzM4NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T21:56:16Z", "updated_at": "2020-04-18T21:58:11Z", "author_association": "MEMBER", "body": "Got this working! I'll do EXIF in a separate ticket #3.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/6#issuecomment-615979923", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/6", "id": 615979923, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk3OTkyMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T23:36:02Z", "updated_at": "2020-04-18T23:36:02Z", "author_association": "MEMBER", "body": "I'll use a Click progress bar. To do this I need to first calculate the sum number of bytes in the photos that are going to be uploaded, then run the upload.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602575575, "label": "Add progress bar to upload command"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/6#issuecomment-615983393", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/6", "id": 615983393, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk4MzM5Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T23:53:10Z", "updated_at": "2020-04-18T23:53:10Z", "author_association": "MEMBER", "body": "```\r\n$ photos-to-sqlite upload photos3.db ~/Pictures/Photos\\ Library.photoslibrary/Masters/2020\r\nUploading 2.09 GB [##----------------------------------] 6% 00:36:37\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602575575, "label": "Add progress bar to upload command"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/7#issuecomment-615993178", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/7", "id": 615993178, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk5MzE3OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-19T00:37:08Z", "updated_at": "2020-04-19T00:37:08Z", "author_association": "MEMBER", "body": "https://pypi.org/project/ImageHash/ Is one option.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602585497, "label": "Integrate image content hashing"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/45#issuecomment-616029262", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/45", "id": 616029262, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNjAyOTI2Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-19T04:39:21Z", "updated_at": "2020-04-19T04:39:21Z", "author_association": "MEMBER", "body": "![44714E00-8CC5-46CD-9E48-1F4DD148FCC8](https://user-images.githubusercontent.com/9599/79679696-09b6d300-81bd-11ea-80e4-0653d92e4f58.jpeg)\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602619330, "label": "Use raise_for_status() everywhere"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/30#issuecomment-616879753", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/30", "id": 616879753, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNjg3OTc1Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-21T00:29:29Z", "updated_at": "2020-04-21T00:29:29Z", "author_association": "MEMBER", "body": "`assignee` looks like it's the incorrect type too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 603618244, "label": "Issues milestone column is the wrong type"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/29#issuecomment-616883275", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/29", "id": 616883275, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNjg4MzI3NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-21T00:43:28Z", "updated_at": "2020-04-21T00:43:28Z", "author_association": "MEMBER", "body": "I'm copying repo from issue, which surprisingly is a string, not an integer ID.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 603617013, "label": "Milestones should have foreign key to creator and repo"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/30#issuecomment-616883726", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/30", "id": 616883726, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNjg4MzcyNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-21T00:45:23Z", "updated_at": "2020-04-21T00:45:23Z", "author_association": "MEMBER", "body": "Demo of fix: https://github-to-sqlite.dogsheep.net/github/issues?assignee__notblank=1&milestone__notblank=1\r\n\r\n![github__issues__4_rows_where_where_assignee_is_not_blank_and_milestone_is_not_blank_sorted_by_updated_at_descending](https://user-images.githubusercontent.com/9599/79812758-b49ace80-832e-11ea-81db-bdf993b872cc.png)\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 603618244, "label": "Issues milestone column is the wrong type"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/31#issuecomment-616884647", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/31", "id": 616884647, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNjg4NDY0Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-21T00:49:16Z", "updated_at": "2020-04-21T00:50:20Z", "author_association": "MEMBER", "body": "The API just gives us the `repository_url`: https://api.github.com/repos/simonw/datasette/issues\r\n\r\n![Mozilla_Firefox_and_Topic__Week_2__Discussion__Submit_your_six_story_points_here](https://user-images.githubusercontent.com/9599/79812950-283cdb80-832f-11ea-8759-9633087d1e7e.png)\r\n\r\nWe currently turn that into a `simonw/datasette` string here:\r\nhttps://github.com/dogsheep/github-to-sqlite/blob/e0e8d8caa9657b04bfb8a2cf16c9b580f38b1805/github_to_sqlite/utils.py#L43-L46", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 603624862, "label": "Issue and milestone should have foreign key to repo"}, "performed_via_github_app": null}