html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app
https://github.com/dogsheep/twitter-to-sqlite/issues/50#issuecomment-690860653,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/50,690860653,MDEyOklzc3VlQ29tbWVudDY5MDg2MDY1Mw==,370930,2020-09-11T04:04:08Z,2020-09-11T04:04:08Z,CONTRIBUTOR,"There's probably a nicer way of doing (hence this is a comment rather than a PR), but this appears to fix it:
```diff
--- a/twitter_to_sqlite/utils.py
+++ b/twitter_to_sqlite/utils.py
@@ -181,6 +181,7 @@ def fetch_timeline(
args[""tweet_mode""] = ""extended""
min_seen_id = None
num_rate_limit_errors = 0
+ seen_count = 0
while True:
if min_seen_id is not None:
args[""max_id""] = min_seen_id - 1
@@ -208,6 +209,7 @@ def fetch_timeline(
yield tweet
min_seen_id = min(t[""id""] for t in tweets)
max_seen_id = max(t[""id""] for t in tweets)
+ seen_count += len(tweets)
if last_since_id is not None:
max_seen_id = max((last_since_id, max_seen_id))
last_since_id = max_seen_id
@@ -217,7 +219,9 @@ def fetch_timeline(
replace=True,
)
if stop_after is not None:
- break
+ if seen_count >= stop_after:
+ break
+ args[""count""] = min(args[""count""], stop_after - seen_count)
time.sleep(sleep)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",698791218,
https://github.com/simonw/datasette/issues/962#issuecomment-691250299,https://api.github.com/repos/simonw/datasette/issues/962,691250299,MDEyOklzc3VlQ29tbWVudDY5MTI1MDI5OQ==,9599,2020-09-11T18:33:50Z,2020-09-11T18:33:50Z,OWNER,Since this is purely a debugging option I'm going to allow myself not to write a unit test for it!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",699622046,
https://github.com/simonw/datasette/issues/947#issuecomment-691318133,https://api.github.com/repos/simonw/datasette/issues/947,691318133,MDEyOklzc3VlQ29tbWVudDY5MTMxODEzMw==,9599,2020-09-11T21:23:40Z,2020-09-11T21:23:40Z,OWNER,"I'm going to use exit code 1 for any errors, be they 500 or 404.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684111953,
https://github.com/simonw/datasette/issues/782#issuecomment-691323302,https://api.github.com/repos/simonw/datasette/issues/782,691323302,MDEyOklzc3VlQ29tbWVudDY5MTMyMzMwMg==,9599,2020-09-11T21:38:27Z,2020-09-11T21:40:04Z,OWNER,"Another idea: the default output could be the list of dicts:
```json
[
{
""pk1"": ""a"",
""pk2"": ""a"",
""pk3"": ""a"",
""content"": ""a-a-a""
},
...
]
```
BUT... I could include pagination information in the HTTP headers - as seen in the WordPress REST API or the GitHub API:
```
~ % curl -s -i 'https://api.github.com/repos/simonw/datasette/commits' | head -n 40
HTTP/1.1 200 OK
server: GitHub.com
date: Fri, 11 Sep 2020 21:37:46 GMT
content-type: application/json; charset=utf-8
status: 200 OK
cache-control: public, max-age=60, s-maxage=60
vary: Accept, Accept-Encoding, Accept, X-Requested-With
etag: W/""71c99379743513394e880c6306b66bf9""
last-modified: Fri, 11 Sep 2020 21:32:54 GMT
x-github-media-type: github.v3; format=json
link: ; rel=""next"", ; rel=""last""
access-control-expose-headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset
access-control-allow-origin: *
strict-transport-security: max-age=31536000; includeSubdomains; preload
x-frame-options: deny
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
referrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin
content-security-policy: default-src 'none'
X-Ratelimit-Limit: 60
X-Ratelimit-Remaining: 55
X-Ratelimit-Reset: 1599863850
X-Ratelimit-Used: 5
Accept-Ranges: bytes
Content-Length: 118240
X-GitHub-Request-Id: EC76:0EAD:313F40:5291A4:5F5BEE37
[
{
""sha"": ""d02f6151dae073135a22d0123e8abdc6cbef7c50"",
""node_id"": ""MDY6Q29tbWl0MTA3OTE0NDkzOmQwMmY2MTUxZGFlMDczMTM1YTIyZDAxMjNlOGFiZGM2Y2JlZjdjNTA="",
""commit"": {
```
Alternative shapes would provide the pagination information (and other extensions) in the JSON, e.g.:
`/squirrels/squirrels.json?_shape=paginated`
```json
{
""rows"": [
{
""pk1"": ""a"",
""pk2"": ""a"",
""pk3"": ""a"",
""content"": ""a-a-a""
}
],
""pagination"": {
""next"": ""234"",
""count"": 442
}
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,