{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/51#issuecomment-1279224780", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/51", "id": 1279224780, "node_id": "IC_kwDODFdgUs5MP2vM", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2022-10-14T16:34:07Z", "updated_at": "2022-10-14T16:34:07Z", "author_association": "NONE", "body": "also, it says that authenticated requests have a much higher \"rate limit\". Unauthenticated requests only get 60 req/hour ?? seems more like a quota than a \"rate limit\" (although I guess that is semantic equivalence)\r\n\r\nYou would want to use `x-ratelimit-reset`\r\n\r\n```\r\ntime.sleep(r['x-ratelimit-reset'] + 1 - time.time())\r\n```\r\n\r\nBut a more complete solution would bring authenticated requests to the other subcommands. I'm surprised only `github-to-sqlite get` is using the `--auth=` CLI flag", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 703246031, "label": "github-to-sqlite should handle rate limits better"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/51#issuecomment-1208757153", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/51", "id": 1208757153, "node_id": "IC_kwDODFdgUs5IDCuh", "user": {"value": 9020979, "label": "hydrosquall"}, "created_at": "2022-08-09T00:29:44Z", "updated_at": "2022-08-09T00:29:44Z", "author_association": "NONE", "body": "I've been looking into how to to get this data out of Github (especially now there are \"secondary rate limits\" without an advertised allowance separate from the regular rate limits. \r\n\r\nI've had decent success with the Airbyte github extractor (aside from one data quality issue https://github.com/airbytehq/airbyte/pull/15420 ). Airbyte splits data extraction between the GraphQL and REST endpoints depending on the resource type, but they're very comprehensive. \r\n\r\nhttps://github.com/airbytehq/airbyte/blob/306a75ef5370728e0912cf52a1a898a530db0c90/airbyte-integrations/connectors/source-github/source_github/streams.py#L22-L122\r\n\r\nBefore this, I tried a few solutions in my own custom wrapper mentioned in this thread + its children https://github.com/PyGithub/PyGithub/issues/1989 , but they weren't working as expected.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 703246031, "label": "github-to-sqlite should handle rate limits better"}, "performed_via_github_app": null}