issues

From #50 - right now it will crash with an error of it hits the rate limit. Since the rate limit information (including reset time) is available in the headers it could automatically sleep and try again instead.

Fetches paginated data from https://api.github.com/repos/simonw/datasette/tags

Description

Per your v2.2 release tweet I played with the demo, but the output did not match my expectations.

Expected Behavior

Expected a search query for "twitter" contained within the repo column to return non-zero results.

Actual Behavior

😭 0 rows where repo contains "twitter" sorted by starred_at descending

Best Explanation

Per the table schema (see appendix) repo is of type INTEGER which built from repo_id and does not expose the repo name in search.

Desired Behavior

Given that searching for "206156866" is less intuitive than "twitter", it would be great to support this via extending the search capabilities or by adding an additional column.

✅ 104 rows where repo contains "twitter" ❌ 104 rows where repo contains "206156866" sorted by starred_at descending

Appendix

CREATE TABLE [stars] ( [user] INTEGER REFERENCES [users]([id]), [repo] INTEGER REFERENCES [repos]([id]), [starred_at] TEXT, PRIMARY KEY ([user], [repo]) ); CREATE INDEX [idx_stars_repo] ON [stars] ([repo]); CREATE INDEX [idx_stars_user] ON [stars] ([user]);

Would you accept a PR that falls back to looking for an environment variable for the GitHub token? Specifically a change here: https://github.com/dogsheep/github-to-sqlite/blob/c34d5a18bfc41fa08755ba3d5cf9fe09ff204238/github_to_sqlite/cli.py#L271

I'd like to use github-to-sqlite in a GitHub Action workflow and this would be simpler than trying to fill out the prompt or generate a file with sensitive content.

Wanted to check first, I'm happy to submit a PR with tests and updates to the docs.

Exception running first command without an existing db or auth.

```py

mkdir ~/.github/coala /usr/bin/github-to-sqlite repos ~/.github/coala coala Traceback (most recent call last): File "/usr/bin/github-to-sqlite", line 11, in <module> load_entry_point('github-to-sqlite==0.6', 'console_scripts', 'github-to-sqlite')() File "/usr/lib/python3.7/site-packages/click/core.py", line 764, in call return self.main(args, kwargs) File "/usr/lib/python3.7/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/usr/lib/python3.7/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/lib/python3.7/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/lib/python3.7/site-packages/click/core.py", line 555, in invoke return callback(args, **kwargs) File "/usr/lib/python3.7/site-packages/github_to_sqlite/cli.py", line 163, in repos utils.save_repo(db, repo) File "/usr/lib/python3.7/site-packages/github_to_sqlite/utils.py", line 120, in save_repo to_save["owner"] = save_user(db, to_save["owner"]) File "/usr/lib/python3.7/site-packages/github_to_sqlite/utils.py", line 61, in save_user return db["users"].upsert(to_save, pk="id", alter=True).last_pk File "/usr/lib/python3.7/site-packages/sqlite_utils/db.py", line 1135, in upsert extracts=extracts, File "/usr/lib/python3.7/site-packages/sqlite_utils/db.py", line 1162, in upsert_all upsert=True, File "/usr/lib/python3.7/site-packages/sqlite_utils/db.py", line 1105, in insert_all row = list(self.rows_where("rowid = ?", [self.last_rowid]))[0] IndexError: list index out of range ```

The releases command extracts the releases table, but data about the individual assets are locked up in the JSON document in the assets field. My main interest is in individual and aggregate download counts. I was wondering if creating a new table with a record per asset may be useful? If so I'm happy to send a PR when I get a moment. Do you have opinions about that simply being part of the releases command or would you prefer a separate command as well?