issues

Table actions

79 rows where repo = 207052882 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: user, comments, author_association, created_at (date), updated_at (date), closed_at (date)

type 2

state 2

repo 1

  • github-to-sqlite · 79 ✖
id node_id number title user state locked assignee milestone comments created_at updated_at ▲ closed_at author_association pull_request body repo type active_lock_reason performed_via_github_app reactions draft state_reason
1570375808 I_kwDODFdgUs5dmgiA 79 Deploy demo job is failing due to rate limit simonw 9599 open 0     2 2023-02-03T20:05:01Z 2023-12-08T14:50:15Z   MEMBER   github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/79/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1505411725 I_kwDODFdgUs5ZusKN 78 self-hosted or corp github enterprise ebdavison 549431 open 0     0 2022-12-20T22:51:45Z 2022-12-20T22:51:45Z   NONE  

We use github enterprise at work and I would like to use this tool to pull info from that site rather than the public github.com instance. Is there an option for this? If not, can one be added for a custom repo URL?

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/78/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
664485022 MDU6SXNzdWU2NjQ0ODUwMjI= 46 Feature: pull request reviews and comments bhrutledge 1326704 open 0     6 2020-07-23T13:43:45Z 2022-12-20T14:40:15Z   NONE  

Hi there! I saw your presentation at Boston Python. I'm already a light user of Datasette (thank you!), but wasn't aware of this project.

I've been working on a "pull request dashboard" to get a comprehensive view of the state of open PR's, esp. related to reviews (i.e., pending, approved, changes requested). Currently it's a CLI command, but I thought a Datasette UI might be fun.

I see that PR's are available from the issues command, but I don't see reviews anywhere. From the API docs, it looks like there are separate endpoints for those (as well as pull requests in general). What do you think about adding that? Would you accept a PR? Any sense of the level of effort?

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46/reactions",
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1410548368 I_kwDODFdgUs5UE0KQ 77 Feature: Support GitHub discussions frosencrantz 631242 open 0     0 2022-10-16T16:53:38Z 2022-10-16T16:53:38Z   CONTRIBUTOR  

Hi @simonw I've been a happy user of this tool. Thank you for writing it and sharing it.

I wanted to suggest a feature request to support Discussions. For example the VisiData project has discussions https://github.com/saulpw/visidata/discussions , and it would be useful if there was a way to pull that data into the database.

However, I'm not offering a pull request.

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/77/reactions",
    "total_count": 2,
    "+1": 2,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
703246031 MDU6SXNzdWU3MDMyNDYwMzE= 51 github-to-sqlite should handle rate limits better simonw 9599 open 0     4 2020-09-17T04:01:50Z 2022-10-14T16:34:07Z   MEMBER  

From #50 - right now it will crash with an error of it hits the rate limit. Since the rate limit information (including reset time) is available in the headers it could automatically sleep and try again instead.

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/51/reactions",
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
923270900 MDExOlB1bGxSZXF1ZXN0NjcyMDUzODEx 65 basic support for events khimaros 231498 open 0     2 2021-06-17T00:51:30Z 2022-10-03T22:35:03Z   FIRST_TIME_CONTRIBUTOR dogsheep/github-to-sqlite/pulls/65

a quick first pass at implementing the feature requested in https://github.com/dogsheep/github-to-sqlite/issues/64

testing instructions:

$ github-to-sqlite events events.db user/khimaros

if the specified user is the authenticated user, it will also include private events.

caveat: pagination appears to be broken (i don't see next in the response JSON from GitHub)

 github-to-sqlite 207052882 pull     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/65/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
1363280254 PR_kwDODFdgUs4-cIa_ 76 Add organization support to repos command OverkillGuy 2757699 open 0     1 2022-09-06T13:21:42Z 2022-09-06T13:59:08Z   FIRST_TIME_CONTRIBUTOR dogsheep/github-to-sqlite/pulls/76

New --organization flag to signify all given "usernames" are private orgs. Adapts API URL to the organization path instead.

Not the best implementation, but a first draft to talk around

Fixes #75 (badly, no tests, overly vague, untested)

 github-to-sqlite 207052882 pull     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/76/reactions",
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
1363244199 I_kwDODFdgUs5RQXSn 75 Fetch repos doesn't support organisations OverkillGuy 2757699 open 0     0 2022-09-06T12:55:06Z 2022-09-06T12:55:06Z   NONE  

Say I want to get all my Github Org's repos info, for data analysis. Not just the public repos, but also the private/internal repos.

The endpoints are different for organisation, and this tool doesn't take it into account: https://github.com/dogsheep/github-to-sqlite/blob/ace13ec3d98090d99bd71871c286a4a612c96a50/github_to_sqlite/utils.py#L453 https://github.com/dogsheep/github-to-sqlite/blob/ace13ec3d98090d99bd71871c286a4a612c96a50/github_to_sqlite/utils.py#L455

The endpoints for organisation repos is instead (source): url = "https://api.github.com/orgs/{}/repos".format(username)

Let's add support for organisations repo scraping.

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/75/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1308461063 I_kwDODFdgUs5N_YgH 74 500 error in github-to-sqlite demo simonw 9599 closed 0     5 2022-07-18T19:39:32Z 2022-07-18T21:16:18Z 2022-07-18T21:14:22Z MEMBER  

https://github-to-sqlite.dogsheep.net/github/issue_comments throws a 500:

cannot import name 'etree' from 'markdown.util' (/usr/local/lib/python3.8/site-packages/markdown/util.py)

https://console.cloud.google.com/run/detail/us-central1/github-to-sqlite/metrics?project=datasette-222320 suggests this started happening 3 days ago.

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/74/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1261884917 PR_kwDODFdgUs45K1L3 73 Fixing 'NoneType' object has no attribute 'items' empjustine 1224205 closed 0     1 2022-06-06T13:58:11Z 2022-07-18T19:40:12Z 2022-07-18T19:40:12Z CONTRIBUTOR dogsheep/github-to-sqlite/pulls/73

Under some conditions, GitHub caches removed starred repositories and ends up leaving dangling None user references.

Traceback (most recent call last):
  File "/home/dogsheep/dogsheep/github-to-sqlite/bin/github-to-sqlite", line 8, in <module>
    sys.exit(cli())
  File "/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/github_to_sqlite/cli.py", line 181, in starred
    utils.save_stars(db, user, stars)
  File "/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/github_to_sqlite/utils.py", line 494, in save_stars
    repo_id = save_repo(db, repo)
  File "/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/github_to_sqlite/utils.py", line 308, in save_repo
    to_save["owner"] = save_user(db, to_save["owner"])
  File "/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/github_to_sqlite/utils.py", line 229, in save_user
    for key, value in user.items()
AttributeError: 'NoneType' object has no attribute 'items'
github-to-sqlite 207052882 pull     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/73/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
1177059481 I_kwDODFdgUs5GKICZ 71 Store commit parents carltongibson 64686 closed 0     0 2022-03-22T17:06:48Z 2022-04-22T12:44:04Z 2022-04-22T12:44:04Z NONE  

Hi @simonw 👋

Currently, stored commit data doesn't quite give me the information I'm needing...

Committer date and author date are not 100% reliable for dividing a commit history up by release or branch. A PR created before a release but merged after can have earlier dates… — this can be quite frustrating if you're trying to pin down commits for a release: It should be there!, but then isn't. (This gets worse using release branches.)

Would you be open to adding the sha of a parent of a commit to the commit table? (As an FK? 🤔 — likely not feasible.)

It's part of the response body:

"parents": [ { "url": "https://api.github.com/repos/octocat/Hello-World/commits/6dcb09b5b57875f334f61aebed695e2e4193db5e", "sha": "6dcb09b5b57875f334f61aebed695e2e4193db5e" } ],

I think this list should only have a single entry. (🤔 — not sure why it's a list then...)

With this it would be possible to build/reconstruct a chain of commits from the history, that I don't think is available as yet (unless you know a better way).

It is certainly possible to get sequential lists of commits out of git directly, so the same would be possible combining tools, but wondering if a single tool could do it.

What do you think? Thanks! 🏅

github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/71/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1211283427 I_kwDODFdgUs5IMrfj 72 feature: display progress bar when downloading multi-page responses hydrosquall 9020979 open 0     1 2022-04-21T16:37:12Z 2022-04-21T17:29:31Z   NONE  

Motivation

For a long running command (longer than 1 minute) for a big table (like pull requests or commits), it can be tricky to know if the script is still running, or if a rate limit/error was encountered

We know how many pages there are, so it may be possible to indicate how many remain.

https://github.com/dogsheep/github-to-sqlite/blob/a6e237f75a4b86963d91dcb5c9582e3a1b3349d6/github_to_sqlite/utils.py#L367

Resources

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/72/reactions",
    "total_count": 3,
    "+1": 3,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1149402080 PR_kwDODFdgUs4zaUta 70 scrape-dependents: enable paging through package menu option if present stanbiryukov 36061055 open 0     0 2022-02-24T15:07:25Z 2022-02-24T15:07:25Z   FIRST_TIME_CONTRIBUTOR dogsheep/github-to-sqlite/pulls/70

Some repos organize network dependents by a Package toggle. This PR adds the ability to page through those options and scrape underlying dependents.

 github-to-sqlite 207052882 pull     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/70/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
1071071397 I_kwDODFdgUs4_10Cl 69 View that combines issues and issue comments simonw 9599 open 0     1 2021-12-04T00:34:33Z 2021-12-04T00:34:52Z   MEMBER  

I want to see a reverse chronologically ordered interface onto both issues and comments - essentially a unified log of comments and issues opened across one or multiple projects.

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/69/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1013506559 PR_kwDODFdgUs4skaNS 68 Add support for retrieving teams / members philwills 68329 open 0     0 2021-10-01T15:55:02Z 2021-10-01T15:59:53Z   FIRST_TIME_CONTRIBUTOR dogsheep/github-to-sqlite/pulls/68

Adds a method for retrieving all the teams within an organisation and all the members in those teams. The latter is stored as a join table team_members beteween teams and users.

 github-to-sqlite 207052882 pull     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/68/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
975161924 MDExOlB1bGxSZXF1ZXN0NzE2MzU3OTgy 66 Add --merged-by flag to pull-requests sub command sarcasticadmin 30531572 open 0     1 2021-08-20T00:57:55Z 2021-09-28T21:50:31Z   FIRST_TIME_CONTRIBUTOR dogsheep/github-to-sqlite/pulls/66

Description

Proposing a solution to the API limitation for merged_by in pull_requests. Specifically the following called out in the readme:

Note that the merged_by column on the pull_requests table will only be populated for pull requests that are loaded using the --pull-request option - the GitHub API does not return this field for pull requests that are loaded in bulk.

This approach might cause larger repos to hit rate limits called out in https://github.com/dogsheep/github-to-sqlite/issues/51 but seems to work well in the repos I tested and included below.

Old Behavior

  • Had to list out the pull-requests individually via multiple --pull-request flags

New Behavior

  • --merged-by flag for getting 'merge_by' information out of pull-requests without having to specify individual PR numbers.

Testing

Picking some repo that has more than one merger (datasette only has 1 😉 )

$ github-to-sqlite pull-requests ./github.db opnsense/tools --merged-by $ echo "select id, url, merged_by from pull_requests;" | sqlite3 ./github.db 83533612|https://github.com/opnsense/tools/pull/39|1915288 102632885|https://github.com/opnsense/tools/pull/43|1915288 149114810|https://github.com/opnsense/tools/pull/57|1915288 160394495|https://github.com/opnsense/tools/pull/64|1915288 163308408|https://github.com/opnsense/tools/pull/67|1915288 169723264|https://github.com/opnsense/tools/pull/69|1915288 171381422|https://github.com/opnsense/tools/pull/72|1915288 179938195|https://github.com/opnsense/tools/pull/77|1915288 196233824|https://github.com/opnsense/tools/pull/82|1915288 215289964|https://github.com/opnsense/tools/pull/93| 219696100|https://github.com/opnsense/tools/pull/97|1915288 223664843|https://github.com/opnsense/tools/pull/99| 228446172|https://github.com/opnsense/tools/pull/103|1915288 238930434|https://github.com/opnsense/tools/pull/110|1915288 255507110|https://github.com/opnsense/tools/pull/119|1915288 255980675|https://github.com/opnsense/tools/pull/120|1915288 261906770|https://github.com/opnsense/tools/pull/125| 263800503|https://github.com/opnsense/tools/pull/127|1915288 264038685|https://github.com/opnsense/tools/pull/128|1915288 264696704|https://github.com/opnsense/tools/pull/129|1915288 266660547|https://github.com/opnsense/tools/pull/130|1915288 273120409|https://github.com/opnsense/tools/pull/133|1915288 274370803|https://github.com/opnsense/tools/pull/135| 276600629|https://github.com/opnsense/tools/pull/139| 277303655|https://github.com/opnsense/tools/pull/141|1915288 293033714|https://github.com/opnsense/tools/pull/145| 294827649|https://github.com/opnsense/tools/pull/146| 295140008|https://github.com/opnsense/tools/pull/147|1915288 305690829|https://github.com/opnsense/tools/pull/150|9783985 307077931|https://github.com/opnsense/tools/pull/152|1915288 321782100|https://github.com/opnsense/tools/pull/155| 337265672|https://github.com/opnsense/tools/pull/160| 337267484|https://github.com/opnsense/tools/pull/161|1915288 368251763|https://github.com/opnsense/tools/pull/169| 428262505|https://github.com/opnsense/tools/pull/181| 437557011|https://github.com/opnsense/tools/pull/182|1915288 447079893|https://github.com/opnsense/tools/pull/185| 461822092|https://github.com/opnsense/tools/pull/191| 463290142|https://github.com/opnsense/tools/pull/193|1915288 470112962|https://github.com/opnsense/tools/pull/194|1915288 472644649|https://github.com/opnsense/tools/pull/195|1915288 488696898|https://github.com/opnsense/tools/pull/198| 513289902|https://github.com/opnsense/tools/pull/201| 522530265|https://github.com/opnsense/tools/pull/203| 564443347|https://github.com/opnsense/tools/pull/213| 597579516|https://github.com/opnsense/tools/pull/220|1915288 602860357|https://github.com/opnsense/tools/pull/221|1915288 608744738|https://github.com/opnsense/tools/pull/222|1915288 623279673|https://github.com/opnsense/tools/pull/228|1915288 664656182|https://github.com/opnsense/tools/pull/233| 664781786|https://github.com/opnsense/tools/pull/234|1915288 670683636|https://github.com/opnsense/tools/pull/235|1915288 683150764|https://github.com/opnsense/tools/pull/237| 685016233|https://github.com/opnsense/tools/pull/238| 687099825|https://github.com/opnsense/tools/pull/239|1915288 715705652|https://github.com/opnsense/tools/pull/244|1915288 715721248|https://github.com/opnsense/tools/pull/245|1915288 userid are now present for those PRs that were merged.

Without the flag the merged_by behavior remains missing as expected when get PRs bulk:

$ github-to-sqlite pull-requests ./github.db opnsense/tools $ echo "select id, url, merged_by from pull_requests;" | sqlite3 ./github.db 83533612|https://github.com/opnsense/tools/pull/39| 102632885|https://github.com/opnsense/tools/pull/43| 149114810|https://github.com/opnsense/tools/pull/57| 160394495|https://github.com/opnsense/tools/pull/64| 163308408|https://github.com/opnsense/tools/pull/67| 169723264|https://github.com/opnsense/tools/pull/69| 171381422|https://github.com/opnsense/tools/pull/72| 179938195|https://github.com/opnsense/tools/pull/77| 196233824|https://github.com/opnsense/tools/pull/82| 215289964|https://github.com/opnsense/tools/pull/93| 219696100|https://github.com/opnsense/tools/pull/97| 223664843|https://github.com/opnsense/tools/pull/99| 228446172|https://github.com/opnsense/tools/pull/103| 238930434|https://github.com/opnsense/tools/pull/110| 255507110|https://github.com/opnsense/tools/pull/119| 255980675|https://github.com/opnsense/tools/pull/120| 261906770|https://github.com/opnsense/tools/pull/125| 263800503|https://github.com/opnsense/tools/pull/127| 264038685|https://github.com/opnsense/tools/pull/128| 264696704|https://github.com/opnsense/tools/pull/129| 266660547|https://github.com/opnsense/tools/pull/130| 273120409|https://github.com/opnsense/tools/pull/133| 274370803|https://github.com/opnsense/tools/pull/135| 276600629|https://github.com/opnsense/tools/pull/139| 277303655|https://github.com/opnsense/tools/pull/141| 293033714|https://github.com/opnsense/tools/pull/145| 294827649|https://github.com/opnsense/tools/pull/146| 295140008|https://github.com/opnsense/tools/pull/147| 305690829|https://github.com/opnsense/tools/pull/150| 307077931|https://github.com/opnsense/tools/pull/152| 321782100|https://github.com/opnsense/tools/pull/155| 337265672|https://github.com/opnsense/tools/pull/160| 337267484|https://github.com/opnsense/tools/pull/161| 368251763|https://github.com/opnsense/tools/pull/169| 428262505|https://github.com/opnsense/tools/pull/181| 437557011|https://github.com/opnsense/tools/pull/182| 447079893|https://github.com/opnsense/tools/pull/185| 461822092|https://github.com/opnsense/tools/pull/191| 463290142|https://github.com/opnsense/tools/pull/193| 470112962|https://github.com/opnsense/tools/pull/194| 472644649|https://github.com/opnsense/tools/pull/195| 488696898|https://github.com/opnsense/tools/pull/198| 513289902|https://github.com/opnsense/tools/pull/201| 522530265|https://github.com/opnsense/tools/pull/203| 564443347|https://github.com/opnsense/tools/pull/213| 597579516|https://github.com/opnsense/tools/pull/220| 602860357|https://github.com/opnsense/tools/pull/221| 608744738|https://github.com/opnsense/tools/pull/222| 623279673|https://github.com/opnsense/tools/pull/228| 664656182|https://github.com/opnsense/tools/pull/233| 664781786|https://github.com/opnsense/tools/pull/234| 670683636|https://github.com/opnsense/tools/pull/235| 683150764|https://github.com/opnsense/tools/pull/237| 685016233|https://github.com/opnsense/tools/pull/238| 687099825|https://github.com/opnsense/tools/pull/239| 715705652|https://github.com/opnsense/tools/pull/244| 715721248|https://github.com/opnsense/tools/pull/245|

Individual PRs passed via --pull-request flag behaves as expected (unchanged):

$ github-to-sqlite pull-requests ./github.db opnsense/tools --pull-request 39 --pull-request 237 $ echo "select id, url, merged_by from pull_requests;" | sqlite3 ./github.db 83533612|https://github.com/opnsense/tools/pull/39|1915288 683150764|https://github.com/opnsense/tools/pull/237|

Picking 1 PR that has a merged_by (39) and one that does not (237)

 github-to-sqlite 207052882 pull     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/66/reactions",
    "total_count": 3,
    "+1": 2,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 1,
    "rocket": 0,
    "eyes": 0
}
 0  
981690086 MDExOlB1bGxSZXF1ZXN0NzIxNjg2NzIx 67 Replacing step ID key with step_id jshcmpbll 16374374 open 0     0 2021-08-28T01:26:41Z 2021-08-28T01:27:00Z   FIRST_TIME_CONTRIBUTOR dogsheep/github-to-sqlite/pulls/67

Workflows that have an id in any step result in the following error when running workflows:

e.g.github-to-sqlite workflows github.db nixos/nixpkgs

Traceback (most recent call last): File "/usr/local/bin/github-to-sqlite", line 8, in <module> sys.exit(cli()) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1137, in __call__ return self.main(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1062, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1668, in invokeTraceback (most recent call last): File "/usr/local/bin/github-to-sqlite", line 8, in <module> sys.exit(cli()) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1137, in call return self.main(args, kwargs) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1062, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1668, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 763, in invoke return __callback(args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/github_to_sqlite/cli.py", line 601, in workflows utils.save_workflow(db, repo_id, filename, content) File "/usr/local/lib/python3.8/dist-packages/github_to_sqlite/utils.py", line 865, in save_workflow db["steps"].insert_all( File "/usr/local/lib/python3.8/dist-packages/sqlite_utils/db.py", line 2596, in insert_all self.insert_chunk( File "/usr/local/lib/python3.8/dist-packages/sqlite_utils/db.py", line 2378, in insert_chunk result = self.db.execute(query, params) File "/usr/local/lib/python3.8/dist-packages/sqlite_utils/db.py", line 419, in execute return self.conn.execute(sql, parameters) sqlite3.IntegrityError: datatype mismatch ```

Changes

I'm proposing that the key for id in step is replaced with step_id so that it no longer interferes with the table id for tracking the record.

Special thanks to @sarcasticadmin @egiffen and @ruebenramirez for helping a bit on this 😄

github-to-sqlite 207052882 pull     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/67/reactions",
    "total_count": 1,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 1,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
920636216 MDU6SXNzdWU5MjA2MzYyMTY= 64 feature: support "events" khimaros 231498 open 0     5 2021-06-14T17:42:49Z 2021-06-15T00:48:37Z   NONE  

the GitHub API provides the ability to fetch all events for a given user, organization, or repository: https://docs.github.com/en/rest/reference/activity#list-events-for-the-authenticated-user

this would allow users to export all of the issue comments, new issues, etc. that they created. something which is currently missing from the GitHub takeout exports.

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
771872303 MDExOlB1bGxSZXF1ZXN0NTQzMjQ2NTM1 59 Remove unneeded exists=True for -a/--auth flag. frosencrantz 631242 closed 0     3 2020-12-21T06:03:55Z 2021-05-22T14:06:19Z 2021-05-19T16:08:12Z CONTRIBUTOR dogsheep/github-to-sqlite/pulls/59

The file does not need to exist when using an environment variable.

 github-to-sqlite 207052882 pull     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/59/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
897212458 MDU6SXNzdWU4OTcyMTI0NTg= 63 Ability to fetch commits from branches other than the default simonw 9599 open 0     0 2021-05-20T17:58:08Z 2021-05-20T17:58:08Z   MEMBER  

This tool is currently almost entirely ignorant of the concept of branches. One example: you can't retrieve commits from any branch other than the default (usually main).

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/63/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
797108702 MDExOlB1bGxSZXF1ZXN0NTY0MTcyMTQw 61 fixing typo in get cli help text daniel-butler 22578954 closed 0     1 2021-01-29T18:57:04Z 2021-05-19T16:07:09Z 2021-05-19T16:07:09Z CONTRIBUTOR dogsheep/github-to-sqlite/pulls/61
github-to-sqlite 207052882 pull     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/61/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
797097140 MDU6SXNzdWU3OTcwOTcxNDA= 60 Use Data from SQLite in other commands daniel-butler 22578954 open 0     3 2021-01-29T18:35:52Z 2021-02-12T18:29:43Z   CONTRIBUTOR  

As a total beginner here how could you access data from the sqlite table to run other commands.

What I am thinking is I want to get all the repos in an organization then using the repo list pull all the commit messages for each repo.

I love this project by the way!

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/60/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
797784080 MDU6SXNzdWU3OTc3ODQwODA= 62 Stargazers and workflows commands always require an auth file when using GITHUB_TOKEN frosencrantz 631242 open 0     0 2021-01-31T18:56:05Z 2021-01-31T18:56:05Z   CONTRIBUTOR  

Requested fix in https://github.com/dogsheep/github-to-sqlite/pull/59

The stargazers and workflows commands always require an auth file, even when using a GITHUB_TOKEN. Other commands don't require the auth file.

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/62/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
769150394 MDU6SXNzdWU3NjkxNTAzOTQ= 58 Readme HTML has broken internal links simonw 9599 closed 0     2 2020-12-16T17:58:11Z 2020-12-16T19:20:14Z 2020-12-16T19:20:14Z MEMBER   github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/58/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
758944006 MDU6SXNzdWU3NTg5NDQwMDY= 57 --readme throws 404 error if README does not exist in repo simonw 9599 closed 0     0 2020-12-07T23:58:49Z 2020-12-16T18:17:54Z 2020-12-16T18:17:54Z MEMBER  

It should fail silently (populate the column with a null) instead.

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/57/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
753122082 MDU6SXNzdWU3NTMxMjIwODI= 56 Link to example tables from the README simonw 9599 closed 0     0 2020-11-30T04:01:51Z 2020-11-30T04:10:27Z 2020-11-30T04:10:27Z MEMBER  

Would help demonstrate how the tool works.

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/56/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
753000405 MDU6SXNzdWU3NTMwMDA0MDU= 53 Command for fetching file contents simonw 9599 open 0     1 2020-11-29T20:31:04Z 2020-11-30T00:36:09Z   MEMBER  

Something like this:

github-to-sqlite files github.db simonw/datasette

This would fetch all files from the main branch into a files table.

Additional options could handle things like pulling files from a branch or tag, or just pulling files that match a specific glob or that exist in a specific directory.

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/53/reactions",
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
681228542 MDExOlB1bGxSZXF1ZXN0NDY5NjUxNzMy 48 Add pull requests adamjonas 755825 closed 0     2 2020-08-18T17:58:44Z 2020-11-29T23:51:09Z 2020-11-29T23:51:09Z CONTRIBUTOR dogsheep/github-to-sqlite/pulls/48

ref #46

Issues don't have merge information on them, which means that PRs need to be pulled separately.

Did my best to mimic the API of issues.

 github-to-sqlite 207052882 pull     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/48/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
753026388 MDU6SXNzdWU3NTMwMjYzODg= 55 github-to-sqlite workflows does not correctly replace existing records simonw 9599 closed 0     0 2020-11-29T21:58:43Z 2020-11-29T23:48:50Z 2020-11-29T23:48:50Z MEMBER   github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/55/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
753026003 MDU6SXNzdWU3NTMwMjYwMDM= 54 github-to-sqlite workflows command simonw 9599 closed 0     3 2020-11-29T21:56:42Z 2020-11-29T22:08:46Z 2020-11-29T21:57:17Z MEMBER  

A command that fetches the YAML workflows for different repos, parses them and stores them in relational tables would be really useful for maintaining larger numbers of workflows.

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/54/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
724264574 MDU6SXNzdWU3MjQyNjQ1NzQ= 52 Option to fetch README and/or HTML-rendered README for repos simonw 9599 closed 0     0 2020-10-19T05:10:24Z 2020-10-19T05:33:42Z 2020-10-19T05:33:42Z MEMBER  

I'm thinking:

github-to-sqlite repos ... --readme # Populates readme column with raw text
github-to-sqlite repos ... --readme-html # Populates readme_html column with raw HTML

https://developer.github.com/v3/repos/contents/#get-a-repository-readme

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/52/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
703218756 MDU6SXNzdWU3MDMyMTg3NTY= 50 Commands for making authenticated API calls simonw 9599 open 0     7 2020-09-17T02:39:07Z 2020-10-19T05:01:29Z   MEMBER  

Similar to twitter-to-sqlite fetch, see https://github.com/dogsheep/twitter-to-sqlite/issues/51

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
703216044 MDU6SXNzdWU3MDMyMTYwNDQ= 49 Feature: gists and starred gists simonw 9599 open 0     0 2020-09-17T02:30:52Z 2020-09-17T02:30:52Z   MEMBER   github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/49/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
681086659 MDU6SXNzdWU2ODEwODY2NTk= 47 emojis command simonw 9599 closed 0     1 2020-08-18T14:26:26Z 2020-08-18T14:52:13Z 2020-08-18T14:52:13Z MEMBER  

For fun - it can import https://api.github.com/emojis - maybe with an option to fetch the binary representations in addition to the URLs.

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/47/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
613777056 MDU6SXNzdWU2MTM3NzcwNTY= 39 issues foreign key to repo isn't working simonw 9599 closed 0     1 2020-05-07T05:11:48Z 2020-08-18T14:24:46Z 2020-08-18T14:23:56Z MEMBER  

https://github-to-sqlite.dogsheep.net/github/issues?_facet=repo

If the foreign key was working those would be repository names.

From the schema at the bottom of the page: [repo] TEXT, That's the wrong type and not a foreign key.

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/39/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
660355904 MDU6SXNzdWU2NjAzNTU5MDQ= 43 github-to-sqlite tags command for fetching tags simonw 9599 closed 0     4 2020-07-18T20:14:12Z 2020-07-18T23:05:56Z 2020-07-18T21:52:15Z MEMBER   github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
660429601 MDU6SXNzdWU2NjA0Mjk2MDE= 45 Fix the demo - it breaks because of the tags table change simonw 9599 closed 0     5 2020-07-18T22:49:32Z 2020-07-18T23:03:14Z 2020-07-18T23:03:13Z MEMBER  

https://github.com/dogsheep/github-to-sqlite/runs/885773677 File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 476, in save_tags db["tags"].insert_all( File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/sqlite_utils/db.py", line 1145, in insert_all result = self.db.conn.execute(query, params) sqlite3.OperationalError: table tags has no column named repo That's because I changed the name in #44. I thought this would be safe since no-one else could possibly be using this yet (it hadn't shipped in a release) but turns out I broke my demo!

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
660413281 MDU6SXNzdWU2NjA0MTMyODE= 44 Rename tags.repo_id column to tags.repo simonw 9599 closed 0     0 2020-07-18T22:13:46Z 2020-07-18T22:15:12Z 2020-07-18T22:15:12Z MEMBER  

For improved consistency with other tables. https://observablehq.com/@simonw/datasette-table-diagram

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/44/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
654405302 MDU6SXNzdWU2NTQ0MDUzMDI= 42 Option for importing just specific repos simonw 9599 closed 0     0 2020-07-09T23:20:15Z 2020-07-09T23:25:35Z 2020-07-09T23:25:35Z MEMBER  

For if you know which specific repos you care about, as opposed to loading everything owned by the authenticated user.

github-to-sqlite repos specific.db -r simonw/datasette -r simonw/github-contents
github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/42/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
651159727 MDU6SXNzdWU2NTExNTk3Mjc= 41 Demo is failing to deploy simonw 9599 closed 0     7 2020-07-05T22:40:33Z 2020-07-06T01:07:03Z 2020-07-06T01:07:02Z MEMBER  

https://github.com/dogsheep/github-to-sqlite/runs/837714622?check_suite_focus=true ``` Creating Revision.........................................................................................................................................failed Deployment failed ERROR: (gcloud.run.deploy) Cloud Run error: Container failed to start. Failed to start and then listen on the port defined by the PORT environment variable. Logs for this revision might contain more information. Traceback (most recent call last): File "/opt/hostedtoolcache/Python/3.8.3/x64/bin/datasette", line 8, in <module> sys.exit(cli()) File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 829, in call return self.main(args, kwargs) File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, ctx.params) File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 610, in invoke return callback(args, **kwargs) File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/datasette/publish/cloudrun.py", line 138, in cloudrun check_call( File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/subprocess.py", line 364, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'gcloud run deploy --allow-unauthenticated --platform=managed --image gcr.io/datasette-222320/datasette github-to-sqlite' returned non-zero exit status 1.

[error]Process completed with exit code 1.

```

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
637899539 MDU6SXNzdWU2Mzc4OTk1Mzk= 40 Demo deploy is broken simonw 9599 closed 0     2 2020-06-12T17:20:17Z 2020-06-12T18:06:48Z 2020-06-12T18:06:48Z MEMBER  

https://github.com/dogsheep/github-to-sqlite/runs/766180404?check_suite_focus=true

``` The following NEW packages will be installed: sqlite3 0 upgraded, 1 newly installed, 0 to remove and 11 not upgraded. Need to get 752 kB of archives. After this operation, 2482 kB of additional disk space will be used. Ign:1 http://azure.archive.ubuntu.com/ubuntu bionic-updates/main amd64 sqlite3 amd64 3.22.0-1ubuntu0.3 Err:1 http://security.ubuntu.com/ubuntu bionic-updates/main amd64 sqlite3 amd64 3.22.0-1ubuntu0.3 404 Not Found [IP: 52.177.174.250 80] E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/s/sqlite3/sqlite3_3.22.0-1ubuntu0.3_amd64.deb 404 Not Found [IP: 52.177.174.250 80] E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

[error]Process completed with exit code 100.

```

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/40/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
610408908 MDU6SXNzdWU2MTA0MDg5MDg= 34 Command for retrieving dependents for a repo simonw 9599 closed 0     6 2020-04-30T21:47:51Z 2020-05-03T15:53:01Z 2020-05-03T15:53:01Z MEMBER  

I really, really want to start grabbing this data: https://github.com/simonw/datasette/network/dependents

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
611284481 MDU6SXNzdWU2MTEyODQ0ODE= 38 [Feature Request] Support Repo Name in Search 🥺 zzeleznick 5779832 closed 0     4 2020-05-02T22:08:51Z 2020-05-03T02:34:32Z 2020-05-02T23:15:11Z NONE  

Description

Per your v2.2 release tweet I played with the demo, but the output did not match my expectations.

Expected Behavior

Expected a search query for "twitter" contained within the repo column to return non-zero results.

Actual Behavior

😭 0 rows where repo contains "twitter" sorted by starred_at descending

Best Explanation

Per the table schema (see appendix) repo is of type INTEGER which built from repo_id and does not expose the repo name in search.

Desired Behavior

Given that searching for "206156866" is less intuitive than "twitter", it would be great to support this via extending the search capabilities or by adding an additional column.

✅ 104 rows where repo contains "twitter" ❌ 104 rows where repo contains "206156866" sorted by starred_at descending

Appendix

CREATE TABLE [stars] ( [user] INTEGER REFERENCES [users]([id]), [repo] INTEGER REFERENCES [repos]([id]), [starred_at] TEXT, PRIMARY KEY ([user], [repo]) ); CREATE INDEX [idx_stars_repo] ON [stars] ([repo]); CREATE INDEX [idx_stars_user] ON [stars] ([user]);

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/38/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
493670730 MDU6SXNzdWU0OTM2NzA3MzA= 4 Command to fetch stargazers for one or more repos simonw 9599 closed 0     8 2019-09-14T21:58:22Z 2020-05-02T21:30:27Z 2020-05-02T21:30:27Z MEMBER  

Maybe this:

$ github-to-sqlite stargazers github.db simonw/datasette

It could accept more than one repos.

Maybe have options similar to --sql in twitter-to-sqlite so you can e.g. fetch all stargazers for all of the repos you have fetched into the database already (or all of the repos belonging to owner X)

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
516763727 MDExOlB1bGxSZXF1ZXN0MzM1OTgwMjQ2 8 stargazers command, refs #4 simonw 9599 closed 0     5 2019-11-03T00:37:36Z 2020-05-02T20:00:27Z 2020-05-02T20:00:26Z MEMBER dogsheep/github-to-sqlite/pulls/8

Needs tests. Refs #4.

 github-to-sqlite 207052882 pull     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/8/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
520756546 MDU6SXNzdWU1MjA3NTY1NDY= 12 Add this view for seeing new releases simonw 9599 closed 0     5 2019-11-11T06:00:12Z 2020-05-02T18:58:18Z 2020-05-02T18:58:17Z MEMBER  

sql CREATE VIEW recent_releases AS select json_object("label", repos.full_name, "href", repos.html_url) as repo, json_object( "href", releases.html_url, "label", releases.name ) as release, substr(releases.published_at, 0, 11) as date, releases.body as body_markdown, releases.published_at from releases join repos on repos.id = releases.repo order by releases.published_at desc

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
609950090 MDU6SXNzdWU2MDk5NTAwOTA= 33 Fall back to authentication via ENV garethr 2029 closed 0     4 2020-04-30T12:58:14Z 2020-05-02T18:46:10Z 2020-05-02T18:45:37Z NONE  

Would you accept a PR that falls back to looking for an environment variable for the GitHub token? Specifically a change here: https://github.com/dogsheep/github-to-sqlite/blob/c34d5a18bfc41fa08755ba3d5cf9fe09ff204238/github_to_sqlite/cli.py#L271

I'd like to use github-to-sqlite in a GitHub Action workflow and this would be simpler than trying to fill out the prompt or generate a file with sensitive content.

Wanted to check first, I'm happy to submit a PR with tests and updates to the docs.

github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
610511450 MDU6SXNzdWU2MTA1MTE0NTA= 35 Create index on issue_comments(user) and other foreign keys simonw 9599 closed 0     3 2020-05-01T02:06:56Z 2020-05-02T18:26:24Z 2020-05-02T18:26:24Z MEMBER  

create index issue_comments_user on issue_comments(user) I'm sure there are other user columns that could benefit from an index.

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/35/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
610842926 MDU6SXNzdWU2MTA4NDI5MjY= 36 Add view for better display of dependent repos simonw 9599 closed 0     2 2020-05-01T16:33:44Z 2020-05-02T16:50:31Z 2020-05-02T16:30:11Z MEMBER  

sql select repos.full_name as repo, 'https://github.com/' || repos2.full_name as dependent, repos2.created_at as dependent_repo_created, repos2.updated_at as dependent_repo_updated, repos2.stargazers_count as dependent_repo_stars, repos2.watchers_count as dependent_repo_watchers from dependents join repos as repos2 on dependents.dependent = repos2.id join repos on dependents.repo = repos.id order by repos2.created_at desc https://dogsheep.simonwillison.net/github?sql=select%0D%0A++repos.full_name+as+repo%2C%0D%0A++%27https%3A%2F%2Fgithub.com%2F%27+%7C%7C+repos2.full_name+as+dependent%2C%0D%0A++repos2.created_at+as+dependent_repo_created%2C%0D%0A++repos2.updated_at+as+dependent_repo_updated%2C%0D%0A++repos2.stargazers_count+as+dependent_repo_stars%2C%0D%0A++repos2.watchers_count+as+dependent_repo_watchers%0D%0Afrom%0D%0A++dependents%0D%0A++join+repos+as+repos2+on+dependents.dependent+%3D+repos2.id%0D%0A++join+repos+on+dependents.repo+%3D+repos.id%0D%0Aorder+by%0D%0A++repos2.created_at+desc

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/36/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
516967682 MDU6SXNzdWU1MTY5Njc2ODI= 10 Add this repos_starred view simonw 9599 closed 0     3 2019-11-04T05:44:38Z 2020-05-02T16:37:36Z 2020-05-02T16:37:36Z MEMBER  

sql create view repos_starred as select stars.starred_at, users.login, repos.* from repos join stars on repos.id = stars.repo join users on repos.owner = users.id order by starred_at desc;

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/10/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
610843136 MDU6SXNzdWU2MTA4NDMxMzY= 37 Mechanism for creating views if they don't yet exist simonw 9599 closed 0     3 2020-05-01T16:34:10Z 2020-05-02T16:19:47Z 2020-05-02T16:19:31Z MEMBER  

Needed for #36 #10 #12

github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/37/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
603624862 MDU6SXNzdWU2MDM2MjQ4NjI= 31 Issue and milestone should have foreign key to repo simonw 9599 closed 0     3 2020-04-21T00:46:24Z 2020-04-22T01:20:19Z 2020-04-22T01:20:19Z MEMBER  

Currently the repo column on those tables is a string simonw/datasette rather than an ID referencing a row in repos.

Originally posted by @simonw in https://github.com/dogsheep/github-to-sqlite/issues/29#issuecomment-616883275

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/31/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
604222295 MDU6SXNzdWU2MDQyMjIyOTU= 32 Issue comments don't appear to populate issues foreign key simonw 9599 closed 0     3 2020-04-21T19:17:32Z 2020-04-22T01:17:44Z 2020-04-22T01:17:44Z MEMBER   github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/32/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
603618244 MDU6SXNzdWU2MDM2MTgyNDQ= 30 Issues milestone column is the wrong type simonw 9599 closed 0     2 2020-04-21T00:24:34Z 2020-04-21T00:45:23Z 2020-04-21T00:36:22Z MEMBER  

https://github-to-sqlite.dogsheep.net/github/issues?milestone=2857392

It is TEXT when it should be an INTEGER - which is why the foreign key label is not correctly displayed.

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/30/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
603617013 MDU6SXNzdWU2MDM2MTcwMTM= 29 Milestones should have foreign key to creator and repo simonw 9599 closed 0     1 2020-04-21T00:20:44Z 2020-04-21T00:43:58Z 2020-04-21T00:43:58Z MEMBER  

https://github-to-sqlite.dogsheep.net/github/milestones

Creator is an integer but not a foreign key to users

Repo is missing entirely!

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/29/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
601333634 MDU6SXNzdWU2MDEzMzM2MzQ= 28 Pull repository contributors simonw 9599 closed 0     3 2020-04-16T18:46:40Z 2020-04-18T15:05:10Z 2020-04-18T15:05:10Z MEMBER  

https://developer.github.com/v3/repos/#list-contributors

GET /repos/:owner/:repo/contributors

Not sure if this should be a separate command or should be part of the existing repos command. I'm leaning towards a new contributors command.

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/28/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
601330277 MDU6SXNzdWU2MDEzMzAyNzc= 27 Repos have a big blob of JSON in the organization column simonw 9599 closed 0     5 2020-04-16T18:43:14Z 2020-04-18T00:19:16Z 2020-04-18T00:18:52Z MEMBER  

e.g. https://github-to-sqlite.dogsheep.net/github/repos

This appears to be obsolete because the owner column already links to that record, albeit in the users table with type set to Organization: https://github-to-sqlite.dogsheep.net/github/users/53015001

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
601265023 MDU6SXNzdWU2MDEyNjUwMjM= 25 Improvements to demo instance simonw 9599 closed 0     1 2020-04-16T17:26:55Z 2020-04-16T18:07:12Z 2020-04-16T18:07:12Z MEMBER  
  • [x] Demo should pull issue-comments as well
 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/25/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
601271612 MDU6SXNzdWU2MDEyNzE2MTI= 26 Topics are missing from repositories simonw 9599 closed 0     2 2020-04-16T17:36:32Z 2020-04-16T17:41:11Z 2020-04-16T17:41:11Z MEMBER  

I'm sure this used to work, but right now repositories are fetched without their topics.

https://developer.github.com/v3/repos/ says you need to send a custom Accept header of application/vnd.github.mercy-preview+json to get topics.

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/26/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
530491074 MDU6SXNzdWU1MzA0OTEwNzQ= 14 Command for importing events simonw 9599 open 0     3 2019-11-29T21:28:58Z 2020-04-14T19:38:34Z   MEMBER   github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
546051181 MDU6SXNzdWU1NDYwNTExODE= 16 Exception running first command: IndexError: list index out of range jayvdb 15092 closed 0     4 2020-01-07T03:01:58Z 2020-04-14T18:37:21Z 2020-04-14T18:37:21Z NONE  

Exception running first command without an existing db or auth.

```py

mkdir ~/.github/coala /usr/bin/github-to-sqlite repos ~/.github/coala coala Traceback (most recent call last): File "/usr/bin/github-to-sqlite", line 11, in <module> load_entry_point('github-to-sqlite==0.6', 'console_scripts', 'github-to-sqlite')() File "/usr/lib/python3.7/site-packages/click/core.py", line 764, in call return self.main(args, kwargs) File "/usr/lib/python3.7/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/usr/lib/python3.7/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/lib/python3.7/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/lib/python3.7/site-packages/click/core.py", line 555, in invoke return callback(args, **kwargs) File "/usr/lib/python3.7/site-packages/github_to_sqlite/cli.py", line 163, in repos utils.save_repo(db, repo) File "/usr/lib/python3.7/site-packages/github_to_sqlite/utils.py", line 120, in save_repo to_save["owner"] = save_user(db, to_save["owner"]) File "/usr/lib/python3.7/site-packages/github_to_sqlite/utils.py", line 61, in save_user return db["users"].upsert(to_save, pk="id", alter=True).last_pk File "/usr/lib/python3.7/site-packages/sqlite_utils/db.py", line 1135, in upsert extracts=extracts, File "/usr/lib/python3.7/site-packages/sqlite_utils/db.py", line 1162, in upsert_all upsert=True, File "/usr/lib/python3.7/site-packages/sqlite_utils/db.py", line 1105, in insert_all row = list(self.rows_where("rowid = ?", [self.last_rowid]))[0] IndexError: list index out of range ```

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/16/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
599776345 MDU6SXNzdWU1OTk3NzYzNDU= 24 Feature idea: github-to-sqlite everything ... simonw 9599 open 0     0 2020-04-14T18:34:00Z 2020-04-14T18:34:00Z   MEMBER  

At the moment if you want to pull all your repos, issues, issues comments etc you have to do it with a sequence of separate commands.

Consider adding a everything or all command which fetches everything that the tool knows how to fetch, and is designed to be run on a cron in a way that fetches just new stuff each time.

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/24/reactions",
    "total_count": 7,
    "+1": 7,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
544571092 MDU6SXNzdWU1NDQ1NzEwOTI= 15 Assets table with downloads garethr 2029 closed 0   1.0 5225818 4 2020-01-02T13:05:28Z 2020-03-28T12:17:01Z 2020-03-23T19:17:32Z NONE  

The releases command extracts the releases table, but data about the individual assets are locked up in the JSON document in the assets field. My main interest is in individual and aggregate download counts. I was wondering if creating a new table with a record per asset may be useful? If so I'm happy to send a PR when I get a moment. Do you have opinions about that simply being part of the releases command or would you prefer a separate command as well?

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
586595839 MDU6SXNzdWU1ODY1OTU4Mzk= 23 Release 1.0 simonw 9599 closed 0   1.0 5225818 1 2020-03-24T00:03:55Z 2020-03-24T00:15:50Z 2020-03-24T00:15:50Z MEMBER  

Need to compile release notes.

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/23/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
521275281 MDU6SXNzdWU1MjEyNzUyODE= 13 Set up a live demo Datasette instance simonw 9599 closed 0   1.0 5225818 9 2019-11-12T01:27:02Z 2020-03-24T00:03:26Z 2020-03-24T00:03:25Z MEMBER  

I deployed https://github-to-sqlite-releases-j7hipcg4aq-uc.a.run.app/ by running this: ```

!/bin/bash

Fetch repos for simonw and dogsheep

github-to-sqlite repos github.db simonw dogsheep -a auth.json

Fetch releases for the repos tagged 'datasette-io'

sqlite-utils github.db " select full_name from repos where rowid in ( select repos.rowid from repos, json_each(repos.topics) j where j.value = 'datasette-io' )" --csv --no-headers | while read repo; do github-to-sqlite releases \ github.db $(echo $repo | tr -d '\r') \ -a auth.json; sleep 2; done; And then deploying using this: $ datasette publish cloudrun github.db \ --title "github-to-sqlite releases demo" \ --about_url="https://github.com/simonw/github-to-sqlite" \ --about='github-to-sqlite' \ --install=datasette-render-markdown \ --install=datasette-json-html \ --service=github-to-sqlite-releases ``` This should happen automatically for every release. I can run it once a day in Circle CI to keep the demo database up-to-date.

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
586561727 MDU6SXNzdWU1ODY1NjE3Mjc= 21 Turn GitHub API errors into exceptions simonw 9599 closed 0   1.0 5225818 2 2020-03-23T22:37:24Z 2020-03-23T23:48:23Z 2020-03-23T23:48:22Z MEMBER  

This would have really helped in debugging the mess in #13. Running with this auth.json is a useful demo: json {"github_personal_token": ""}

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/21/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
586567379 MDU6SXNzdWU1ODY1NjczNzk= 22 Handle empty git repositories simonw 9599 closed 0     0 2020-03-23T22:49:48Z 2020-03-23T23:13:11Z 2020-03-23T23:13:11Z MEMBER  

Got this error: github_to_sqlite.utils.GitHubError: {'message': 'Git Repository is empty.', 'documentation_url': 'https://developer.github.com/v3/repos/commits/#list-commits-on-a-repository'} From https://api.github.com/repos/dogsheep/beta/commits

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/22/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
585411547 MDU6SXNzdWU1ODU0MTE1NDc= 18 Commits in GitHub API can have null author simonw 9599 closed 0   1.0 5225818 8 2020-03-21T02:20:56Z 2020-03-23T20:44:49Z 2020-03-23T20:44:26Z MEMBER  

Traceback (most recent call last): File "/home/ubuntu/datasette-venv/bin/github-to-sqlite", line 8, in <module> sys.exit(cli()) File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/click/core.py", line 764, in __call__ return self.main(*args, **kwargs) File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/github_to_sqlite/cli.py", line 235, in commits utils.save_commits(db, commits, repo_full["id"]) File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/github_to_sqlite/utils.py", line 290, in save_commits commit_to_insert["author"] = save_user(db, commit["author"]) File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/github_to_sqlite/utils.py", line 54, in save_user for key, value in user.items() AttributeError: 'NoneType' object has no attribute 'items' Got this running the commits command from cron.

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
493671014 MDU6SXNzdWU0OTM2NzEwMTQ= 5 Add "incomplete" boolean to users table for incomplete profiles simonw 9599 closed 0     2 2019-09-14T22:01:50Z 2020-03-23T19:23:31Z 2020-03-23T19:23:30Z MEMBER  

User profiles that are fetched from e.g. stargazers (#4) are incomplete - they have a login but they don't have name, company etc.

Add a incomplete boolean flag to the users table to record this. Then later I can add a backfill-users command which loops through and fetches missing data for those incomplete profiles.

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/5/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
586454513 MDU6SXNzdWU1ODY0NTQ1MTM= 20 Upgrade to sqlite-utils 2.x simonw 9599 closed 0   1.0 5225818 0 2020-03-23T19:17:58Z 2020-03-23T19:22:52Z 2020-03-23T19:22:52Z MEMBER  
github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/20/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
585850715 MDU6SXNzdWU1ODU4NTA3MTU= 19 Enable full-text search for more stuff (like commits, issues and issue_comments) simonw 9599 closed 0   1.0 5225818 2 2020-03-23T00:19:56Z 2020-03-23T19:06:39Z 2020-03-23T19:06:39Z MEMBER  

Currently FTS is only enabled for repos and releases.

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/19/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
578883725 MDU6SXNzdWU1Nzg4ODM3MjU= 17 Command for importing commits simonw 9599 closed 0     2 2020-03-10T21:55:12Z 2020-03-11T02:47:37Z 2020-03-11T02:47:37Z MEMBER   github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/17/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
516769276 MDU6SXNzdWU1MTY3NjkyNzY= 9 Commands do not work without an auth.json file simonw 9599 closed 0     0 2019-11-03T01:54:28Z 2019-11-11T05:30:48Z 2019-11-11T05:30:48Z MEMBER  

auth.json is meant to be optional. If it's not provided, the tool should make heavily rate-limited unauthenticated requests.

``` $ github-to-sqlite repos .data/repos.db simonw Usage: github-to-sqlite repos [OPTIONS] DB_PATH [USERNAME] Try "github-to-sqlite repos --help" for help.

Error: Invalid value for "-a" / "--auth": File "auth.json" does not exist. ```

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/9/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
520521843 MDU6SXNzdWU1MjA1MjE4NDM= 11 Command to fetch releases simonw 9599 closed 0     0 2019-11-09T22:23:30Z 2019-11-09T22:57:00Z 2019-11-09T22:57:00Z MEMBER   github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/11/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
506276893 MDU6SXNzdWU1MDYyNzY4OTM= 7 issue-comments command for importing issue comments simonw 9599 closed 0     1 2019-10-13T05:23:58Z 2019-10-14T14:44:12Z 2019-10-13T05:24:30Z MEMBER   github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/7/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
504238461 MDU6SXNzdWU1MDQyMzg0NjE= 6 sqlite3.OperationalError: table users has no column named bio dazzag24 1055831 closed 0     2 2019-10-08T19:39:52Z 2019-10-13T05:31:28Z 2019-10-13T05:30:19Z NONE  

``` $ github-to-sqlite repos github.db $ github-to-sqlite starred github.db dazzag24

Traceback (most recent call last): File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/bin/github-to-sqlite", line 10, in <module> sys.exit(cli()) File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py", line 764, in call return self.main(args, kwargs) File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, ctx.params) File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py", line 555, in invoke return callback(args, **kwargs) File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/github_to_sqlite/cli.py", line 106, in starred utils.save_stars(db, user, stars) File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/github_to_sqlite/utils.py", line 177, in save_stars user_id = save_user(db, user) File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/github_to_sqlite/utils.py", line 61, in save_user return db["users"].upsert(to_save, pk="id").last_pk File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/sqlite_utils/db.py", line 1067, in upsert extracts=extracts, File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/sqlite_utils/db.py", line 916, in insert extracts=extracts, File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/sqlite_utils/db.py", line 1024, in insert_all result = self.db.conn.execute(sql, values) sqlite3.OperationalError: table users has no column named bio

```

``` $ pipenv graph github-to-sqlite==0.4 - requests [required: Any, installed: 2.22.0] - certifi [required: >=2017.4.17, installed: 2019.9.11] - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4] - idna [required: >=2.5,<2.9, installed: 2.8] - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6] - sqlite-utils [required: ~=1.11, installed: 1.11] - click [required: Any, installed: 7.0] - click-default-group [required: Any, installed: 1.2.2] - click [required: Any, installed: 7.0] - tabulate [required: Any, installed: 0.8.5]

Python 3.6.8 ```

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/6/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
493670426 MDU6SXNzdWU0OTM2NzA0MjY= 3 Command to fetch all repos belonging to a user or organization simonw 9599 closed 0     2 2019-09-14T21:54:21Z 2019-09-17T00:17:53Z 2019-09-17T00:17:53Z MEMBER  

How about this:

$ github-to-sqlite repos simonw
github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/3/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
493668862 MDU6SXNzdWU0OTM2Njg4NjI= 2 Extract licenses from repos into a separate table simonw 9599 closed 0     0 2019-09-14T21:33:41Z 2019-09-14T21:46:58Z 2019-09-14T21:46:58Z MEMBER  

github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/2/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
493599818 MDU6SXNzdWU0OTM1OTk4MTg= 1 Command for fetching starred repos simonw 9599 closed 0     0 2019-09-14T08:36:29Z 2019-09-14T21:30:48Z 2019-09-14T21:30:48Z MEMBER  
github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/1/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issues] (
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [number] INTEGER,
   [title] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [state] TEXT,
   [locked] INTEGER,
   [assignee] INTEGER REFERENCES [users]([id]),
   [milestone] INTEGER REFERENCES [milestones]([id]),
   [comments] INTEGER,
   [created_at] TEXT,
   [updated_at] TEXT,
   [closed_at] TEXT,
   [author_association] TEXT,
   [pull_request] TEXT,
   [body] TEXT,
   [repo] INTEGER REFERENCES [repos]([id]),
   [type] TEXT
, [active_lock_reason] TEXT, [performed_via_github_app] TEXT, [reactions] TEXT, [draft] INTEGER, [state_reason] TEXT);
CREATE INDEX [idx_issues_repo]
                ON [issues] ([repo]);
CREATE INDEX [idx_issues_milestone]
                ON [issues] ([milestone]);
CREATE INDEX [idx_issues_assignee]
                ON [issues] ([assignee]);
CREATE INDEX [idx_issues_user]
                ON [issues] ([user]);