issues

41 rows where repo = 207052882 sorted by updated_at descending

View and edit SQL

Suggested facets: user, comments, author_association, created_at (date), updated_at (date), closed_at (date)

type

state

repo

  • github-to-sqlite · 41
id node_id number title user state locked assignee milestone comments created_at updated_at ▲ closed_at author_association pull_request body repo type active_lock_reason
651159727 MDU6SXNzdWU2NTExNTk3Mjc= 41 Demo is failing to deploy simonw 9599 closed 0     7 2020-07-05T22:40:33Z 2020-07-06T01:07:03Z 2020-07-06T01:07:02Z MEMBER  

https://github.com/dogsheep/github-to-sqlite/runs/837714622?check_suite_focus=true

Creating Revision.........................................................................................................................................failed
Deployment failed
ERROR: (gcloud.run.deploy) Cloud Run error: Container failed to start. Failed to start and then listen on the port defined by the PORT environment variable. Logs for this revision might contain more information.
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.8.3/x64/bin/datasette", line 8, in <module>
    sys.exit(cli())
  File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/datasette/publish/cloudrun.py", line 138, in cloudrun
    check_call(
  File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'gcloud run deploy --allow-unauthenticated --platform=managed --image gcr.io/datasette-222320/datasette github-to-sqlite' returned non-zero exit status 1.
##[error]Process completed with exit code 1.
github-to-sqlite 207052882 issue  
637899539 MDU6SXNzdWU2Mzc4OTk1Mzk= 40 Demo deploy is broken simonw 9599 closed 0     2 2020-06-12T17:20:17Z 2020-06-12T18:06:48Z 2020-06-12T18:06:48Z MEMBER  

https://github.com/dogsheep/github-to-sqlite/runs/766180404?check_suite_focus=true

The following NEW packages will be installed:
  sqlite3
0 upgraded, 1 newly installed, 0 to remove and 11 not upgraded.
Need to get 752 kB of archives.
After this operation, 2482 kB of additional disk space will be used.
Ign:1 http://azure.archive.ubuntu.com/ubuntu bionic-updates/main amd64 sqlite3 amd64 3.22.0-1ubuntu0.3
Err:1 http://security.ubuntu.com/ubuntu bionic-updates/main amd64 sqlite3 amd64 3.22.0-1ubuntu0.3
  404  Not Found [IP: 52.177.174.250 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/s/sqlite3/sqlite3_3.22.0-1ubuntu0.3_amd64.deb  404  Not Found [IP: 52.177.174.250 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
##[error]Process completed with exit code 100.
github-to-sqlite 207052882 issue  
613777056 MDU6SXNzdWU2MTM3NzcwNTY= 39 issues foreign key to repo isn't working simonw 9599 open 0     0 2020-05-07T05:11:48Z 2020-05-07T05:11:48Z   MEMBER  

https://dogsheep.simonwillison.net/github/issues?_facet=repo

If the foreign key was working those would be repository names.

From the schema at the bottom of the page:

   [repo] TEXT,

That's the wrong type and not a foreign key.

github-to-sqlite 207052882 issue  
610408908 MDU6SXNzdWU2MTA0MDg5MDg= 34 Command for retrieving dependents for a repo simonw 9599 closed 0     6 2020-04-30T21:47:51Z 2020-05-03T15:53:01Z 2020-05-03T15:53:01Z MEMBER  

I really, really want to start grabbing this data: https://github.com/simonw/datasette/network/dependents

github-to-sqlite 207052882 issue  
611284481 MDU6SXNzdWU2MTEyODQ0ODE= 38 [Feature Request] Support Repo Name in Search 🥺 zzeleznick 5779832 closed 0     4 2020-05-02T22:08:51Z 2020-05-03T02:34:32Z 2020-05-02T23:15:11Z NONE  

Description

Per your v2.2 release tweet I played with the demo, but the output did not match my expectations.

Expected Behavior

Expected a search query for "twitter" contained within the repo column to return non-zero results.

Actual Behavior

😭 0 rows where repo contains "twitter" sorted by starred_at descending

Best Explanation

Per the table schema (see appendix) repo is of type INTEGER which built from repo_id and does not expose the repo name in search.

Desired Behavior

Given that searching for "206156866" is less intuitive than "twitter", it would be great to support this via extending the search capabilities or by adding an additional column.

✅ 104 rows where repo contains "twitter"
104 rows where repo contains "206156866" sorted by starred_at descending

Appendix

CREATE TABLE [stars] (
   [user] INTEGER REFERENCES [users]([id]),
   [repo] INTEGER REFERENCES [repos]([id]),
   [starred_at] TEXT,
   PRIMARY KEY ([user], [repo])
);
CREATE INDEX [idx_stars_repo]
                ON [stars] ([repo]);
CREATE INDEX [idx_stars_user]
                ON [stars] ([user]);
github-to-sqlite 207052882 issue  
493670730 MDU6SXNzdWU0OTM2NzA3MzA= 4 Command to fetch stargazers for one or more repos simonw 9599 closed 0     8 2019-09-14T21:58:22Z 2020-05-02T21:30:27Z 2020-05-02T21:30:27Z MEMBER  

Maybe this:

$ github-to-sqlite stargazers github.db simonw/datasette

It could accept more than one repos.

Maybe have options similar to --sql in twitter-to-sqlite so you can e.g. fetch all stargazers for all of the repos you have fetched into the database already (or all of the repos belonging to owner X)

github-to-sqlite 207052882 issue  
516763727 MDExOlB1bGxSZXF1ZXN0MzM1OTgwMjQ2 8 stargazers command, refs #4 simonw 9599 closed 0     5 2019-11-03T00:37:36Z 2020-05-02T20:00:27Z 2020-05-02T20:00:26Z MEMBER dogsheep/github-to-sqlite/pulls/8

Needs tests. Refs #4.

github-to-sqlite 207052882 pull  
520756546 MDU6SXNzdWU1MjA3NTY1NDY= 12 Add this view for seeing new releases simonw 9599 closed 0     5 2019-11-11T06:00:12Z 2020-05-02T18:58:18Z 2020-05-02T18:58:17Z MEMBER  
CREATE VIEW recent_releases AS select
  json_object("label", repos.full_name, "href", repos.html_url) as repo,
  json_object(
    "href",
    releases.html_url,
    "label",
    releases.name
  ) as release,
  substr(releases.published_at, 0, 11) as date,
  releases.body as body_markdown,
  releases.published_at
from
  releases
  join repos on repos.id = releases.repo
order by
  releases.published_at desc
github-to-sqlite 207052882 issue  
609950090 MDU6SXNzdWU2MDk5NTAwOTA= 33 Fall back to authentication via ENV garethr 2029 closed 0     4 2020-04-30T12:58:14Z 2020-05-02T18:46:10Z 2020-05-02T18:45:37Z NONE  

Would you accept a PR that falls back to looking for an environment variable for the GitHub token? Specifically a change here:
https://github.com/dogsheep/github-to-sqlite/blob/c34d5a18bfc41fa08755ba3d5cf9fe09ff204238/github_to_sqlite/cli.py#L271

I'd like to use github-to-sqlite in a GitHub Action workflow and this would be simpler than trying to fill out the prompt or generate a file with sensitive content.

Wanted to check first, I'm happy to submit a PR with tests and updates to the docs.

github-to-sqlite 207052882 issue  
610511450 MDU6SXNzdWU2MTA1MTE0NTA= 35 Create index on issue_comments(user) and other foreign keys simonw 9599 closed 0     3 2020-05-01T02:06:56Z 2020-05-02T18:26:24Z 2020-05-02T18:26:24Z MEMBER  
create index issue_comments_user on issue_comments(user)

I'm sure there are other user columns that could benefit from an index.

github-to-sqlite 207052882 issue  
610842926 MDU6SXNzdWU2MTA4NDI5MjY= 36 Add view for better display of dependent repos simonw 9599 closed 0     2 2020-05-01T16:33:44Z 2020-05-02T16:50:31Z 2020-05-02T16:30:11Z MEMBER  
select
  repos.full_name as repo,
  'https://github.com/' || repos2.full_name as dependent,
  repos2.created_at as dependent_repo_created,
  repos2.updated_at as dependent_repo_updated,
  repos2.stargazers_count as dependent_repo_stars,
  repos2.watchers_count as dependent_repo_watchers
from
  dependents
  join repos as repos2 on dependents.dependent = repos2.id
  join repos on dependents.repo = repos.id
order by
  repos2.created_at desc

https://dogsheep.simonwillison.net/github?sql=select%0D%0A++repos.full_name+as+repo%2C%0D%0A++%27https%3A%2F%2Fgithub.com%2F%27+%7C%7C+repos2.full_name+as+dependent%2C%0D%0A++repos2.created_at+as+dependent_repo_created%2C%0D%0A++repos2.updated_at+as+dependent_repo_updated%2C%0D%0A++repos2.stargazers_count+as+dependent_repo_stars%2C%0D%0A++repos2.watchers_count+as+dependent_repo_watchers%0D%0Afrom%0D%0A++dependents%0D%0A++join+repos+as+repos2+on+dependents.dependent+%3D+repos2.id%0D%0A++join+repos+on+dependents.repo+%3D+repos.id%0D%0Aorder+by%0D%0A++repos2.created_at+desc

github-to-sqlite 207052882 issue  
516967682 MDU6SXNzdWU1MTY5Njc2ODI= 10 Add this repos_starred view simonw 9599 closed 0     3 2019-11-04T05:44:38Z 2020-05-02T16:37:36Z 2020-05-02T16:37:36Z MEMBER  
create view repos_starred as select
  stars.starred_at,
  users.login,
  repos.*
from
  repos
  join stars on repos.id = stars.repo
  join users on repos.owner = users.id
order by
  starred_at desc;
github-to-sqlite 207052882 issue  
610843136 MDU6SXNzdWU2MTA4NDMxMzY= 37 Mechanism for creating views if they don't yet exist simonw 9599 closed 0     3 2020-05-01T16:34:10Z 2020-05-02T16:19:47Z 2020-05-02T16:19:31Z MEMBER  

Needed for #36 #10 #12

github-to-sqlite 207052882 issue  
603624862 MDU6SXNzdWU2MDM2MjQ4NjI= 31 Issue and milestone should have foreign key to repo simonw 9599 closed 0     3 2020-04-21T00:46:24Z 2020-04-22T01:20:19Z 2020-04-22T01:20:19Z MEMBER  

Currently the repo column on those tables is a string simonw/datasette rather than an ID referencing a row in repos.

Originally posted by @simonw in https://github.com/dogsheep/github-to-sqlite/issues/29#issuecomment-616883275

github-to-sqlite 207052882 issue  
604222295 MDU6SXNzdWU2MDQyMjIyOTU= 32 Issue comments don't appear to populate issues foreign key simonw 9599 closed 0     3 2020-04-21T19:17:32Z 2020-04-22T01:17:44Z 2020-04-22T01:17:44Z MEMBER  

https://github-to-sqlite.dogsheep.net/github?sql=select+html_url%2C+id%2C+issue+from+issue_comments+order+by+updated_at+desc+limit+101

github-to-sqlite 207052882 issue  
603618244 MDU6SXNzdWU2MDM2MTgyNDQ= 30 Issues milestone column is the wrong type simonw 9599 closed 0     2 2020-04-21T00:24:34Z 2020-04-21T00:45:23Z 2020-04-21T00:36:22Z MEMBER  

https://github-to-sqlite.dogsheep.net/github/issues?milestone=2857392

It is TEXT when it should be an INTEGER - which is why the foreign key label is not correctly displayed.

github-to-sqlite 207052882 issue  
603617013 MDU6SXNzdWU2MDM2MTcwMTM= 29 Milestones should have foreign key to creator and repo simonw 9599 closed 0     1 2020-04-21T00:20:44Z 2020-04-21T00:43:58Z 2020-04-21T00:43:58Z MEMBER  

https://github-to-sqlite.dogsheep.net/github/milestones

Creator is an integer but not a foreign key to users

Repo is missing entirely!

github-to-sqlite 207052882 issue  
601333634 MDU6SXNzdWU2MDEzMzM2MzQ= 28 Pull repository contributors simonw 9599 closed 0     3 2020-04-16T18:46:40Z 2020-04-18T15:05:10Z 2020-04-18T15:05:10Z MEMBER  

https://developer.github.com/v3/repos/#list-contributors

GET /repos/:owner/:repo/contributors

Not sure if this should be a separate command or should be part of the existing repos command. I'm leaning towards a new contributors command.

github-to-sqlite 207052882 issue  
601330277 MDU6SXNzdWU2MDEzMzAyNzc= 27 Repos have a big blob of JSON in the organization column simonw 9599 closed 0     5 2020-04-16T18:43:14Z 2020-04-18T00:19:16Z 2020-04-18T00:18:52Z MEMBER  

e.g. https://github-to-sqlite.dogsheep.net/github/repos

This appears to be obsolete because the owner column already links to that record, albeit in the users table with type set to Organization: https://github-to-sqlite.dogsheep.net/github/users/53015001

github-to-sqlite 207052882 issue  
601265023 MDU6SXNzdWU2MDEyNjUwMjM= 25 Improvements to demo instance simonw 9599 closed 0     1 2020-04-16T17:26:55Z 2020-04-16T18:07:12Z 2020-04-16T18:07:12Z MEMBER  
  • Demo should pull issue-comments as well
github-to-sqlite 207052882 issue  
601271612 MDU6SXNzdWU2MDEyNzE2MTI= 26 Topics are missing from repositories simonw 9599 closed 0     2 2020-04-16T17:36:32Z 2020-04-16T17:41:11Z 2020-04-16T17:41:11Z MEMBER  

I'm sure this used to work, but right now repositories are fetched without their topics.

https://developer.github.com/v3/repos/ says you need to send a custom Accept header of application/vnd.github.mercy-preview+json to get topics.

github-to-sqlite 207052882 issue  
530491074 MDU6SXNzdWU1MzA0OTEwNzQ= 14 Command for importing events simonw 9599 open 0     3 2019-11-29T21:28:58Z 2020-04-14T19:38:34Z   MEMBER  

Eg from https://api.github.com/users/simonw/events

Docs here: https://developer.github.com/v3/activity/events/#list-events-performed-by-a-user

github-to-sqlite 207052882 issue  
546051181 MDU6SXNzdWU1NDYwNTExODE= 16 Exception running first command: IndexError: list index out of range jayvdb 15092 closed 0     4 2020-01-07T03:01:58Z 2020-04-14T18:37:21Z 2020-04-14T18:37:21Z NONE  

Exception running first command without an existing db or auth.

> mkdir ~/.github/coala
> /usr/bin/github-to-sqlite repos ~/.github/coala coala
Traceback (most recent call last):
  File "/usr/bin/github-to-sqlite", line 11, in <module>
    load_entry_point('github-to-sqlite==0.6', 'console_scripts', 'github-to-sqlite')()
  File "/usr/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3.7/site-packages/github_to_sqlite/cli.py", line 163, in repos
    utils.save_repo(db, repo)
  File "/usr/lib/python3.7/site-packages/github_to_sqlite/utils.py", line 120, in save_repo
    to_save["owner"] = save_user(db, to_save["owner"])
  File "/usr/lib/python3.7/site-packages/github_to_sqlite/utils.py", line 61, in save_user
    return db["users"].upsert(to_save, pk="id", alter=True).last_pk
  File "/usr/lib/python3.7/site-packages/sqlite_utils/db.py", line 1135, in upsert
    extracts=extracts,
  File "/usr/lib/python3.7/site-packages/sqlite_utils/db.py", line 1162, in upsert_all
    upsert=True,
  File "/usr/lib/python3.7/site-packages/sqlite_utils/db.py", line 1105, in insert_all
    row = list(self.rows_where("rowid = ?", [self.last_rowid]))[0]
IndexError: list index out of range
github-to-sqlite 207052882 issue  
599776345 MDU6SXNzdWU1OTk3NzYzNDU= 24 Feature idea: github-to-sqlite everything ... simonw 9599 open 0     0 2020-04-14T18:34:00Z 2020-04-14T18:34:00Z   MEMBER  

At the moment if you want to pull all your repos, issues, issues comments etc you have to do it with a sequence of separate commands.

Consider adding a everything or all command which fetches everything that the tool knows how to fetch, and is designed to be run on a cron in a way that fetches just new stuff each time.

github-to-sqlite 207052882 issue  
544571092 MDU6SXNzdWU1NDQ1NzEwOTI= 15 Assets table with downloads garethr 2029 closed 0   1.0 5225818 4 2020-01-02T13:05:28Z 2020-03-28T12:17:01Z 2020-03-23T19:17:32Z NONE  

The releases command extracts the releases table, but data about the individual assets are locked up in the JSON document in the assets field. My main interest is in individual and aggregate download counts. I was wondering if creating a new table with a record per asset may be useful?
If so I'm happy to send a PR when I get a moment. Do you have opinions about that simply being part of the releases command or would you prefer a separate command as well?

github-to-sqlite 207052882 issue  
586595839 MDU6SXNzdWU1ODY1OTU4Mzk= 23 Release 1.0 simonw 9599 closed 0   1.0 5225818 1 2020-03-24T00:03:55Z 2020-03-24T00:15:50Z 2020-03-24T00:15:50Z MEMBER  

Need to compile release notes.

github-to-sqlite 207052882 issue  
521275281 MDU6SXNzdWU1MjEyNzUyODE= 13 Set up a live demo Datasette instance simonw 9599 closed 0   1.0 5225818 9 2019-11-12T01:27:02Z 2020-03-24T00:03:26Z 2020-03-24T00:03:25Z MEMBER  

I deployed https://github-to-sqlite-releases-j7hipcg4aq-uc.a.run.app/ by running this:

#!/bin/bash
# Fetch repos for simonw and dogsheep
github-to-sqlite repos github.db simonw dogsheep -a auth.json

# Fetch releases for the repos tagged 'datasette-io'
sqlite-utils github.db "
select full_name from repos where rowid in (
    select repos.rowid from repos, json_each(repos.topics) j
    where j.value = 'datasette-io'
)" --csv --no-headers | while read repo;
    do github-to-sqlite releases \
            github.db $(echo $repo | tr -d '\r') \
            -a auth.json;
        sleep 2;
    done;

And then deploying using this:

$ datasette publish cloudrun github.db \
  --title "github-to-sqlite releases demo" \
  --about_url="https://github.com/simonw/github-to-sqlite" \
  --about='github-to-sqlite' \
  --install=datasette-render-markdown \
  --install=datasette-json-html \
  --service=github-to-sqlite-releases

This should happen automatically for every release. I can run it once a day in Circle CI to keep the demo database up-to-date.

github-to-sqlite 207052882 issue  
586561727 MDU6SXNzdWU1ODY1NjE3Mjc= 21 Turn GitHub API errors into exceptions simonw 9599 closed 0   1.0 5225818 2 2020-03-23T22:37:24Z 2020-03-23T23:48:23Z 2020-03-23T23:48:22Z MEMBER  

This would have really helped in debugging the mess in #13. Running with this auth.json is a useful demo:

{"github_personal_token": ""}
github-to-sqlite 207052882 issue  
586567379 MDU6SXNzdWU1ODY1NjczNzk= 22 Handle empty git repositories simonw 9599 closed 0     0 2020-03-23T22:49:48Z 2020-03-23T23:13:11Z 2020-03-23T23:13:11Z MEMBER  

Got this error:

github_to_sqlite.utils.GitHubError: {'message': 'Git Repository is empty.', 'documentation_url': 'https://developer.github.com/v3/repos/commits/#list-commits-on-a-repository'}

From https://api.github.com/repos/dogsheep/beta/commits

github-to-sqlite 207052882 issue  
585411547 MDU6SXNzdWU1ODU0MTE1NDc= 18 Commits in GitHub API can have null author simonw 9599 closed 0   1.0 5225818 8 2020-03-21T02:20:56Z 2020-03-23T20:44:49Z 2020-03-23T20:44:26Z MEMBER  
Traceback (most recent call last):
  File "/home/ubuntu/datasette-venv/bin/github-to-sqlite", line 8, in <module>
    sys.exit(cli())
  File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/github_to_sqlite/cli.py", line 235, in commits
    utils.save_commits(db, commits, repo_full["id"])
  File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/github_to_sqlite/utils.py", line 290, in save_commits
    commit_to_insert["author"] = save_user(db, commit["author"])
  File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/github_to_sqlite/utils.py", line 54, in save_user
    for key, value in user.items()
AttributeError: 'NoneType' object has no attribute 'items'

Got this running the commits command from cron.

github-to-sqlite 207052882 issue  
493671014 MDU6SXNzdWU0OTM2NzEwMTQ= 5 Add "incomplete" boolean to users table for incomplete profiles simonw 9599 closed 0     2 2019-09-14T22:01:50Z 2020-03-23T19:23:31Z 2020-03-23T19:23:30Z MEMBER  

User profiles that are fetched from e.g. stargazers (#4) are incomplete - they have a login but they don't have name, company etc.

Add a incomplete boolean flag to the users table to record this. Then later I can add a backfill-users command which loops through and fetches missing data for those incomplete profiles.

github-to-sqlite 207052882 issue  
586454513 MDU6SXNzdWU1ODY0NTQ1MTM= 20 Upgrade to sqlite-utils 2.x simonw 9599 closed 0   1.0 5225818 0 2020-03-23T19:17:58Z 2020-03-23T19:22:52Z 2020-03-23T19:22:52Z MEMBER   github-to-sqlite 207052882 issue  
585850715 MDU6SXNzdWU1ODU4NTA3MTU= 19 Enable full-text search for more stuff (like commits, issues and issue_comments) simonw 9599 closed 0   1.0 5225818 2 2020-03-23T00:19:56Z 2020-03-23T19:06:39Z 2020-03-23T19:06:39Z MEMBER  

Currently FTS is only enabled for repos and releases.

github-to-sqlite 207052882 issue  
578883725 MDU6SXNzdWU1Nzg4ODM3MjU= 17 Command for importing commits simonw 9599 closed 0     2 2020-03-10T21:55:12Z 2020-03-11T02:47:37Z 2020-03-11T02:47:37Z MEMBER  

Using this API: https://api.github.com/repos/dogsheep/github-to-sqlite/commits

github-to-sqlite 207052882 issue  
516769276 MDU6SXNzdWU1MTY3NjkyNzY= 9 Commands do not work without an auth.json file simonw 9599 closed 0     0 2019-11-03T01:54:28Z 2019-11-11T05:30:48Z 2019-11-11T05:30:48Z MEMBER  

auth.json is meant to be optional. If it's not provided, the tool should make heavily rate-limited unauthenticated requests.

$ github-to-sqlite repos .data/repos.db simonw
Usage: github-to-sqlite repos [OPTIONS] DB_PATH [USERNAME]
Try "github-to-sqlite repos --help" for help.

Error: Invalid value for "-a" / "--auth": File "auth.json" does not exist.
github-to-sqlite 207052882 issue  
520521843 MDU6SXNzdWU1MjA1MjE4NDM= 11 Command to fetch releases simonw 9599 closed 0     0 2019-11-09T22:23:30Z 2019-11-09T22:57:00Z 2019-11-09T22:57:00Z MEMBER  

https://developer.github.com/v3/repos/releases/#list-releases-for-a-repository

GET /repos/:owner/:repo/releases

github-to-sqlite 207052882 issue  
506276893 MDU6SXNzdWU1MDYyNzY4OTM= 7 issue-comments command for importing issue comments simonw 9599 closed 0     1 2019-10-13T05:23:58Z 2019-10-14T14:44:12Z 2019-10-13T05:24:30Z MEMBER  

Using this API: https://developer.github.com/v3/issues/comments/

github-to-sqlite 207052882 issue  
504238461 MDU6SXNzdWU1MDQyMzg0NjE= 6 sqlite3.OperationalError: table users has no column named bio dazzag24 1055831 closed 0     2 2019-10-08T19:39:52Z 2019-10-13T05:31:28Z 2019-10-13T05:30:19Z NONE  
$ github-to-sqlite repos github.db
$ github-to-sqlite starred github.db dazzag24

Traceback (most recent call last):
  File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/bin/github-to-sqlite", line 10, in <module>
    sys.exit(cli())
  File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/github_to_sqlite/cli.py", line 106, in starred
    utils.save_stars(db, user, stars)
  File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/github_to_sqlite/utils.py", line 177, in save_stars
    user_id = save_user(db, user)
  File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/github_to_sqlite/utils.py", line 61, in save_user
    return db["users"].upsert(to_save, pk="id").last_pk
  File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/sqlite_utils/db.py", line 1067, in upsert
    extracts=extracts,
  File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/sqlite_utils/db.py", line 916, in insert
    extracts=extracts,
  File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/sqlite_utils/db.py", line 1024, in insert_all
    result = self.db.conn.execute(sql, values)
sqlite3.OperationalError: table users has no column named bio
$ pipenv graph
github-to-sqlite==0.4
  - requests [required: Any, installed: 2.22.0]
    - certifi [required: >=2017.4.17, installed: 2019.9.11]
    - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
    - idna [required: >=2.5,<2.9, installed: 2.8]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6]
  - sqlite-utils [required: ~=1.11, installed: 1.11]
    - click [required: Any, installed: 7.0]
    - click-default-group [required: Any, installed: 1.2.2]
      - click [required: Any, installed: 7.0]
    - tabulate [required: Any, installed: 0.8.5]

Python 3.6.8
github-to-sqlite 207052882 issue  
493670426 MDU6SXNzdWU0OTM2NzA0MjY= 3 Command to fetch all repos belonging to a user or organization simonw 9599 closed 0     2 2019-09-14T21:54:21Z 2019-09-17T00:17:53Z 2019-09-17T00:17:53Z MEMBER  

How about this:

$ github-to-sqlite repos simonw
github-to-sqlite 207052882 issue  
493668862 MDU6SXNzdWU0OTM2Njg4NjI= 2 Extract licenses from repos into a separate table simonw 9599 closed 0     0 2019-09-14T21:33:41Z 2019-09-14T21:46:58Z 2019-09-14T21:46:58Z MEMBER  

github-to-sqlite 207052882 issue  
493599818 MDU6SXNzdWU0OTM1OTk4MTg= 1 Command for fetching starred repos simonw 9599 closed 0     0 2019-09-14T08:36:29Z 2019-09-14T21:30:48Z 2019-09-14T21:30:48Z MEMBER   github-to-sqlite 207052882 issue  

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issues] (
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [number] INTEGER,
   [title] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [state] TEXT,
   [locked] INTEGER,
   [assignee] INTEGER REFERENCES [users]([id]),
   [milestone] INTEGER REFERENCES [milestones]([id]),
   [comments] INTEGER,
   [created_at] TEXT,
   [updated_at] TEXT,
   [closed_at] TEXT,
   [author_association] TEXT,
   [pull_request] TEXT,
   [body] TEXT,
   [repo] INTEGER REFERENCES [repos]([id]),
   [type] TEXT
, [active_lock_reason] TEXT);
CREATE INDEX [idx_issues_repo]
                ON [issues] ([repo]);
CREATE INDEX [idx_issues_milestone]
                ON [issues] ([milestone]);
CREATE INDEX [idx_issues_assignee]
                ON [issues] ([assignee]);
CREATE INDEX [idx_issues_user]
                ON [issues] ([user]);
Powered by Datasette · Query took 90.394ms · About: github-to-sqlite