issues

Table actions

20 rows where comments = 0, repo = 207052882 and type = "issue" sorted by updated_at descending

✖
✖
✖
✖

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: user, author_association, body, created_at (date), updated_at (date), closed_at (date)

state 2

type 1

  • issue · 20 ✖

repo 1

  • github-to-sqlite · 20 ✖
id node_id number title user state locked assignee milestone comments created_at updated_at ▲ closed_at author_association pull_request body repo type active_lock_reason performed_via_github_app reactions draft state_reason
1505411725 I_kwDODFdgUs5ZusKN 78 self-hosted or corp github enterprise ebdavison 549431 open 0     0 2022-12-20T22:51:45Z 2022-12-20T22:51:45Z   NONE  

We use github enterprise at work and I would like to use this tool to pull info from that site rather than the public github.com instance. Is there an option for this? If not, can one be added for a custom repo URL?

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/78/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1410548368 I_kwDODFdgUs5UE0KQ 77 Feature: Support GitHub discussions frosencrantz 631242 open 0     0 2022-10-16T16:53:38Z 2022-10-16T16:53:38Z   CONTRIBUTOR  

Hi @simonw I've been a happy user of this tool. Thank you for writing it and sharing it.

I wanted to suggest a feature request to support Discussions. For example the VisiData project has discussions https://github.com/saulpw/visidata/discussions , and it would be useful if there was a way to pull that data into the database.

However, I'm not offering a pull request.

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/77/reactions",
    "total_count": 2,
    "+1": 2,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1363244199 I_kwDODFdgUs5RQXSn 75 Fetch repos doesn't support organisations OverkillGuy 2757699 open 0     0 2022-09-06T12:55:06Z 2022-09-06T12:55:06Z   NONE  

Say I want to get all my Github Org's repos info, for data analysis. Not just the public repos, but also the private/internal repos.

The endpoints are different for organisation, and this tool doesn't take it into account: https://github.com/dogsheep/github-to-sqlite/blob/ace13ec3d98090d99bd71871c286a4a612c96a50/github_to_sqlite/utils.py#L453 https://github.com/dogsheep/github-to-sqlite/blob/ace13ec3d98090d99bd71871c286a4a612c96a50/github_to_sqlite/utils.py#L455

The endpoints for organisation repos is instead (source): url = "https://api.github.com/orgs/{}/repos".format(username)

Let's add support for organisations repo scraping.

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/75/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1177059481 I_kwDODFdgUs5GKICZ 71 Store commit parents carltongibson 64686 closed 0     0 2022-03-22T17:06:48Z 2022-04-22T12:44:04Z 2022-04-22T12:44:04Z NONE  

Hi @simonw 👋

Currently, stored commit data doesn't quite give me the information I'm needing...

Committer date and author date are not 100% reliable for dividing a commit history up by release or branch. A PR created before a release but merged after can have earlier dates… — this can be quite frustrating if you're trying to pin down commits for a release: It should be there!, but then isn't. (This gets worse using release branches.)

Would you be open to adding the sha of a parent of a commit to the commit table? (As an FK? 🤔 — likely not feasible.)

It's part of the response body:

"parents": [ { "url": "https://api.github.com/repos/octocat/Hello-World/commits/6dcb09b5b57875f334f61aebed695e2e4193db5e", "sha": "6dcb09b5b57875f334f61aebed695e2e4193db5e" } ],

I think this list should only have a single entry. (🤔 — not sure why it's a list then...)

With this it would be possible to build/reconstruct a chain of commits from the history, that I don't think is available as yet (unless you know a better way).

It is certainly possible to get sequential lists of commits out of git directly, so the same would be possible combining tools, but wondering if a single tool could do it.

What do you think? Thanks! 🏅

github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/71/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
897212458 MDU6SXNzdWU4OTcyMTI0NTg= 63 Ability to fetch commits from branches other than the default simonw 9599 open 0     0 2021-05-20T17:58:08Z 2021-05-20T17:58:08Z   MEMBER  

This tool is currently almost entirely ignorant of the concept of branches. One example: you can't retrieve commits from any branch other than the default (usually main).

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/63/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
797784080 MDU6SXNzdWU3OTc3ODQwODA= 62 Stargazers and workflows commands always require an auth file when using GITHUB_TOKEN frosencrantz 631242 open 0     0 2021-01-31T18:56:05Z 2021-01-31T18:56:05Z   CONTRIBUTOR  

Requested fix in https://github.com/dogsheep/github-to-sqlite/pull/59

The stargazers and workflows commands always require an auth file, even when using a GITHUB_TOKEN. Other commands don't require the auth file.

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/62/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
758944006 MDU6SXNzdWU3NTg5NDQwMDY= 57 --readme throws 404 error if README does not exist in repo simonw 9599 closed 0     0 2020-12-07T23:58:49Z 2020-12-16T18:17:54Z 2020-12-16T18:17:54Z MEMBER  

It should fail silently (populate the column with a null) instead.

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/57/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
753122082 MDU6SXNzdWU3NTMxMjIwODI= 56 Link to example tables from the README simonw 9599 closed 0     0 2020-11-30T04:01:51Z 2020-11-30T04:10:27Z 2020-11-30T04:10:27Z MEMBER  

Would help demonstrate how the tool works.

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/56/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
753026388 MDU6SXNzdWU3NTMwMjYzODg= 55 github-to-sqlite workflows does not correctly replace existing records simonw 9599 closed 0     0 2020-11-29T21:58:43Z 2020-11-29T23:48:50Z 2020-11-29T23:48:50Z MEMBER   github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/55/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
724264574 MDU6SXNzdWU3MjQyNjQ1NzQ= 52 Option to fetch README and/or HTML-rendered README for repos simonw 9599 closed 0     0 2020-10-19T05:10:24Z 2020-10-19T05:33:42Z 2020-10-19T05:33:42Z MEMBER  

I'm thinking:

github-to-sqlite repos ... --readme # Populates readme column with raw text
github-to-sqlite repos ... --readme-html # Populates readme_html column with raw HTML

https://developer.github.com/v3/repos/contents/#get-a-repository-readme

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/52/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
703216044 MDU6SXNzdWU3MDMyMTYwNDQ= 49 Feature: gists and starred gists simonw 9599 open 0     0 2020-09-17T02:30:52Z 2020-09-17T02:30:52Z   MEMBER   github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/49/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
660413281 MDU6SXNzdWU2NjA0MTMyODE= 44 Rename tags.repo_id column to tags.repo simonw 9599 closed 0     0 2020-07-18T22:13:46Z 2020-07-18T22:15:12Z 2020-07-18T22:15:12Z MEMBER  

For improved consistency with other tables. https://observablehq.com/@simonw/datasette-table-diagram

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/44/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
654405302 MDU6SXNzdWU2NTQ0MDUzMDI= 42 Option for importing just specific repos simonw 9599 closed 0     0 2020-07-09T23:20:15Z 2020-07-09T23:25:35Z 2020-07-09T23:25:35Z MEMBER  

For if you know which specific repos you care about, as opposed to loading everything owned by the authenticated user.

github-to-sqlite repos specific.db -r simonw/datasette -r simonw/github-contents
github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/42/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
599776345 MDU6SXNzdWU1OTk3NzYzNDU= 24 Feature idea: github-to-sqlite everything ... simonw 9599 open 0     0 2020-04-14T18:34:00Z 2020-04-14T18:34:00Z   MEMBER  

At the moment if you want to pull all your repos, issues, issues comments etc you have to do it with a sequence of separate commands.

Consider adding a everything or all command which fetches everything that the tool knows how to fetch, and is designed to be run on a cron in a way that fetches just new stuff each time.

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/24/reactions",
    "total_count": 7,
    "+1": 7,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
586567379 MDU6SXNzdWU1ODY1NjczNzk= 22 Handle empty git repositories simonw 9599 closed 0     0 2020-03-23T22:49:48Z 2020-03-23T23:13:11Z 2020-03-23T23:13:11Z MEMBER  

Got this error: github_to_sqlite.utils.GitHubError: {'message': 'Git Repository is empty.', 'documentation_url': 'https://developer.github.com/v3/repos/commits/#list-commits-on-a-repository'} From https://api.github.com/repos/dogsheep/beta/commits

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/22/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
586454513 MDU6SXNzdWU1ODY0NTQ1MTM= 20 Upgrade to sqlite-utils 2.x simonw 9599 closed 0   1.0 5225818 0 2020-03-23T19:17:58Z 2020-03-23T19:22:52Z 2020-03-23T19:22:52Z MEMBER  
github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/20/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
516769276 MDU6SXNzdWU1MTY3NjkyNzY= 9 Commands do not work without an auth.json file simonw 9599 closed 0     0 2019-11-03T01:54:28Z 2019-11-11T05:30:48Z 2019-11-11T05:30:48Z MEMBER  

auth.json is meant to be optional. If it's not provided, the tool should make heavily rate-limited unauthenticated requests.

``` $ github-to-sqlite repos .data/repos.db simonw Usage: github-to-sqlite repos [OPTIONS] DB_PATH [USERNAME] Try "github-to-sqlite repos --help" for help.

Error: Invalid value for "-a" / "--auth": File "auth.json" does not exist. ```

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/9/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
520521843 MDU6SXNzdWU1MjA1MjE4NDM= 11 Command to fetch releases simonw 9599 closed 0     0 2019-11-09T22:23:30Z 2019-11-09T22:57:00Z 2019-11-09T22:57:00Z MEMBER   github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/11/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
493668862 MDU6SXNzdWU0OTM2Njg4NjI= 2 Extract licenses from repos into a separate table simonw 9599 closed 0     0 2019-09-14T21:33:41Z 2019-09-14T21:46:58Z 2019-09-14T21:46:58Z MEMBER  

github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/2/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
493599818 MDU6SXNzdWU0OTM1OTk4MTg= 1 Command for fetching starred repos simonw 9599 closed 0     0 2019-09-14T08:36:29Z 2019-09-14T21:30:48Z 2019-09-14T21:30:48Z MEMBER  
github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/1/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issues] (
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [number] INTEGER,
   [title] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [state] TEXT,
   [locked] INTEGER,
   [assignee] INTEGER REFERENCES [users]([id]),
   [milestone] INTEGER REFERENCES [milestones]([id]),
   [comments] INTEGER,
   [created_at] TEXT,
   [updated_at] TEXT,
   [closed_at] TEXT,
   [author_association] TEXT,
   [pull_request] TEXT,
   [body] TEXT,
   [repo] INTEGER REFERENCES [repos]([id]),
   [type] TEXT
, [active_lock_reason] TEXT, [performed_via_github_app] TEXT, [reactions] TEXT, [draft] INTEGER, [state_reason] TEXT);
CREATE INDEX [idx_issues_repo]
                ON [issues] ([repo]);
CREATE INDEX [idx_issues_milestone]
                ON [issues] ([milestone]);
CREATE INDEX [idx_issues_assignee]
                ON [issues] ([assignee]);
CREATE INDEX [idx_issues_user]
                ON [issues] ([user]);