issue_comments

I'm happy with this pattern: https://github.com/dogsheep/github-to-sqlite/blob/f78c4e9baaf0970ffab266ba780df7240aae9f32/github_to_sqlite/utils.py#L4-L18

I pointed https://github-to-sqlite.dogsheep.net/ at it. May take a few minutes for the new certificate to provision though.

Next step: use a metadata.json file to add some extras. And add the datasette-render-markdown plugin as soon as I ship https://github.com/simonw/datasette-render-markdown/issues/2 (GFM support).

Most recently updated issues across all Dogsheep repos, with faceting: https://github-to-sqlite-j7hipcg4aq-uc.a.run.app/github/issues?_facet=repo&_facet=user&_facet=state&_facet=author_association&_facet=type&_sort_desc=updated_at

Woohoo! https://github-to-sqlite-j7hipcg4aq-uc.a.run.app/

@simonw You're welcome - I was just trying it out back in December as I thought it should work. Now there's a pandemic to work on though.... so no time at all for more at the moment. BTW, I have datasette running on several protein and full (virus) genome databases I build, and it's great - thank you! Hi and best regards to you & Nat :-)

Thanks very much @terrycojones - I'll see if I can finish it up from here.

I just updated #652 to remove a merge conflict. I think it's an easy way to add this functionality. I don't have time to do more though, sorry!

This would also be useful for running Datasette in Jupyter notebooks on Binder. While you can use Jupyter-server-proxy to access Datasette on Binder, the links are broken.

Why run Datasette on Binder? I'm developing a range of Jupyter notebooks that are aimed at getting humanities researchers to explore data from libraries, archives, and museums. Many of them are aimed at researchers with limited digital skills, so being able to run examples in Binder without them installing anything is fantastic.

For example, there are a series of notebooks that help researchers harvest digitised historical newspaper articles from Trove. The metadata from this harvest is saved as a CSV file that users can download. I've also provided some extra notebooks that use Pandas etc to demonstrate ways of analysing and visualising the harvested data.

But it would be really nice if, after completing a harvest, the user could spin up Datasette for some initial exploration of their harvested data without ever leaving their browser.

On mybinder.org we allow access to arbitrary processes listening on a port inside the container via a reverse proxy.

This means we need support for a proxy prefix as the proxy ends up running at a URL like /something/random/proxy/datasette/...

An example that shows the problem is https://github.com/psychemedia/jupyterserverproxy-datasette-demo. Launch directly into a datasette instance on mybinder.org with https://mybinder.org/v2/gh/psychemedia/jupyterserverproxy-datasette-demo/master?urlpath=datasette then try to follow links inside the UI.

I'm going to do this now to help figure out the latest error in #13: Traceback (most recent call last): File "/opt/hostedtoolcache/Python/3.8.2/x64/bin/github-to-sqlite", line 11, in <module> load_entry_point('github-to-sqlite', 'console_scripts', 'github-to-sqlite')() File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py", line 237, in commits utils.save_commits(db, commits, repo_full["id"]) File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 345, in save_commits for commit in commits: File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 207, in fetch_commits if stop_when(commit): File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py", line 224, in stop_when db["commits"].get(commit["sha"]) TypeError: string indices must be integers

Urgh this is such a mess! I should have done this on a branch / pull request to avoid polluting my main master history, but never mind.

Following these instructions: https://simonwillison.net/2020/Jan/21/github-actions-cloud-run/

I'll run the commits and issues and issue-comments commands in addition to the releases command.

I'm going to do this with a scheduled GitHub Action in this repo.

So it turns out "author" and "committer" on the commit are null if the email address in the nested "commit" doesn't match an existing GitHub user.

Maybe I should be storing the nested data somewhere as well?

I implemented the raw_authors idea.

I could pull a pk-hashed version of the name/email into separate raw_author and raw_committer columns perhaps - against a commit_authors table. Could be interesting.

json [ { "sha": "a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0", "node_id": "MDY6Q29tbWl0OTMyMDk6YThkYzkxNDA4OWQzOTlkOWI1MjJlYmI1MWI2N2Y5YWMyZThhYTZiMA==", "commit": { "author": { "name": "Simon Willison", "email": "simon@...", "date": "2008-12-18T23:17:12Z" }, "committer": { "name": "Simon Willison", "email": "simon@...", "date": "2008-12-18T23:17:12Z" }, "message": "First commit", "tree": { "sha": "ac2dfb75e2592c59165c2880f3f7a16dafd452a1", "url": "https://api.github.com/repos/simonw/simonw.github.com/git/trees/ac2dfb75e2592c59165c2880f3f7a16dafd452a1" }, "url": "https://api.github.com/repos/simonw/simonw.github.com/git/commits/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0", "comment_count": 0, "verification": { "verified": false, "reason": "unsigned", "signature": null, "payload": null } }, "url": "https://api.github.com/repos/simonw/simonw.github.com/commits/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0", "html_url": "https://github.com/simonw/simonw.github.com/commit/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0", "comments_url": "https://api.github.com/repos/simonw/simonw.github.com/commits/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0/comments", "author": null, "committer": null, "parents": [] } ]

Found one: https://api.github.com/repos/simonw/simonw.github.com/commits

I need to find an example before I work on this.

Yeah alter=True means that this isn't necessary.

• commits: message
• issue_comments: body
• issues: title, body
• labels: name, description
• licenses: name
• milestones: title, description
• releases: name, body
• repos: name, description
• users: login, name (maybe company, bio)

Each asset looks like this: json { "url": "https://api.github.com/repos/instrumenta/conftest/releases/assets/11811946", "id": 11811946, "node_id": "MDEyOlJlbGVhc2VBc3NldDExODExOTQ2", "name": "checksums.txt", "label": "", "uploader": { "login": "garethr", "id": 2029, "node_id": "MDQ6VXNlcjIwMjk=", "avatar_url": "https://avatars2.githubusercontent.com/u/2029?v=4", "gravatar_id": "", "url": "https://api.github.com/users/garethr", "html_url": "https://github.com/garethr", "followers_url": "https://api.github.com/users/garethr/followers", "following_url": "https://api.github.com/users/garethr/following{/other_user}", "gists_url": "https://api.github.com/users/garethr/gists{/gist_id}", "starred_url": "https://api.github.com/users/garethr/starred{/owner}{/repo}", "subscriptions_url": "https://api.github.com/users/garethr/subscriptions", "organizations_url": "https://api.github.com/users/garethr/orgs", "repos_url": "https://api.github.com/users/garethr/repos", "events_url": "https://api.github.com/users/garethr/events{/privacy}", "received_events_url": "https://api.github.com/users/garethr/received_events", "type": "User", "site_admin": false }, "content_type": "text/plain; charset=utf-8", "state": "uploaded", "size": 600, "download_count": 2, "created_at": "2019-03-30T16:56:44Z", "updated_at": "2019-03-30T16:56:44Z", "browser_download_url": "https://github.com/instrumenta/conftest/releases/download/v0.1.0/checksums.txt" }

None of my own releases use assets (they are all pushed to PyPI instead) but I spotted that your project here uses assets, so I'll test against that: https://github.com/instrumenta/conftest/releases/tag/v0.18.0

github-to-sqlite releases releases.db instrumenta/conftest

Splitting assets out into a separate table totally makes sense to me. They can still be fetched as part of the releases command.

Current FTS code: https://github.com/dogsheep/github-to-sqlite/blob/e35eec4343aa560c58c1634cc228d0d46c442304/github_to_sqlite/utils.py#L136-L143