{"html_url": "https://github.com/dogsheep/pocket-to-sqlite/issues/1#issuecomment-538847446", "issue_url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/1", "id": 538847446, "node_id": "MDEyOklzc3VlQ29tbWVudDUzODg0NzQ0Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-07T05:41:17Z", "updated_at": "2019-10-07T05:41:17Z", "author_association": "MEMBER", "body": "Prototype code:\r\n```python\r\noffset = 0\r\nfetched = []\r\nsize = 500\r\nwhile True:\r\n    page = requests.get(\"https://getpocket.com/v3/get\", {\r\n        \"consumer_key\": consumer_key,\r\n        \"access_token\": access_token,\r\n        \"sort\": \"oldest\",\r\n        \"detailType\": \"complete\",\r\n        \"count\": size,\r\n        \"offset\": offset,\r\n    }).json()\r\n    print(offset)\r\n    fetched.append(page)\r\n    offset += size\r\n    if not len(page[\"list\"]):\r\n        break\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 503233021, "label": "Use better pagination (and implement progress bar)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/pocket-to-sqlite/issues/1#issuecomment-605316146", "issue_url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/1", "id": 605316146, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNTMxNjE0Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-27T21:09:15Z", "updated_at": "2020-03-27T21:09:22Z", "author_association": "MEMBER", "body": "For a progress bar I need to know how many total items there are.\r\n\r\nI found an undocumented API for this! `/v3/stats` which returns:\r\n```json\r\n{\r\n    \"count_list\": 7394,\r\n    \"count_read\": 1016,\r\n    \"count_unread\": 6378,\r\n    \"status\": 1\r\n}\r\n```\r\nI guessed this based on the documented v2 API: https://getpocket.com/api/v2_docs/#stats", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 503233021, "label": "Use better pagination (and implement progress bar)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/pocket-to-sqlite/issues/1#issuecomment-605325897", "issue_url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/1", "id": 605325897, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNTMyNTg5Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-27T21:37:26Z", "updated_at": "2020-03-27T21:38:37Z", "author_association": "MEMBER", "body": "I keep getting 503 errors even though I appear to be staying within the rate limit:\r\n```\r\n{'Date': 'Fri, 27 Mar 2020 21:35:57 GMT', 'Content-Type': 'application/json', 'Transfer-Encoding':\r\n'chunked', 'Connection': 'keep-alive', 'Server': 'Apache/2.4.25 (Debian)', 'Content-Location':\r\n'get.php', 'Vary': 'negotiate', 'TCN': 'choice', 'Set-Cookie': '...;  httponly',\r\n'X-Frame-Options': 'SAMEORIGIN', 'Status': '200 OK', 'X-Limit-Key-Limit': '10000',\r\n'X-Limit-Key-Remaining': '9960', 'X-Limit-Key-Reset': '282', 'X-Source': 'Pocket',\r\n'P3P': 'policyref=\"/w3c/p3p.xml\", CP=\"ALL CURa ADMa DEVa OUR IND UNI COM NAV INT STA PRE\"'}\r\n\r\n  [##----------------------------------]    6%  06:49:27\r\n\r\n{'Date': 'Fri, 27 Mar 2020 21:36:06 GMT', 'Content-Type': 'text/html; charset=UTF-8', 'Content-Length':\r\n'23', 'Connection': 'keep-alive', 'Server': 'Apache/2.4.25 (Debian)', 'Content-Location': 'get.php',\r\n'Vary': 'negotiate', 'TCN': 'choice', 'Set-Cookie': '...', 'X-Frame-Options': 'SAMEORIGIN',\r\n'X-Error': 'Pocket is currently under heavy load. Please wait a moment and try again.',\r\n'X-Error-Code': '199', 'Status': '503 Service Unavailable', 'X-Source': 'Pocket',\r\n'P3P': 'policyref=\"/w3c/p3p.xml\", CP=\"ALL CURa ADMa DEVa OUR IND UNI COM NAV INT STA PRE\"'}\r\n```\r\nI'm going to try doing a few automatic retries any time I see a 503 error.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 503233021, "label": "Use better pagination (and implement progress bar)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/pocket-to-sqlite/issues/1#issuecomment-605327655", "issue_url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/1", "id": 605327655, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNTMyNzY1NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-27T21:42:49Z", "updated_at": "2020-03-27T21:42:49Z", "author_association": "MEMBER", "body": "Or maybe it was because of the current Google Cloud outage? https://news.ycombinator.com/item?id=22706677", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 503233021, "label": "Use better pagination (and implement progress bar)"}, "performed_via_github_app": null}