issue_comments

Table actions

6 rows where author_association = "OWNER", "created_at" is on date 2022-09-15 and issue = 1366512990 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 1

issue 1

  • progressbar for inserts/upserts of all fileformats, closes #485 · 6 ✖

author_association 1

  • OWNER · 6 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
1248591268 https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248591268 https://api.github.com/repos/simonw/sqlite-utils/issues/486 IC_kwDOCGYnMM5Ka_2k simonw 9599 2022-09-15T20:36:02Z 2022-09-15T20:40:03Z OWNER

I had a big CSV file lying around, I converted it to other formats like this:

sqlite-utils insert /tmp/t.db t /tmp/en.openfoodfacts.org.products.csv --csv
sqlite-utils rows /tmp/t.db t --nl > /tmp/big.nl
sqlite-utils rows /tmp/t.db t > /tmp/big.json

Then tested the progress bar like this:

sqlite-utils insert /tmp/t2.db t /tmp/big.nl --nl

Output:

sqlite-utils insert /tmp/t2.db t /tmp/big.nl --nl [------------------------------------] 0% [#######-----------------------------] 20% 00:00:20 With --silent it is silent.

And for regular JSON:

sqlite-utils insert /tmp/t3.db t /tmp/big.json [####################################] 100% This is actually not doing the right thing. The problem is that sqlite-utils doesn't include a streaming JSON parser, so it instead reads that entire JSON file into memory first (exhausting the progress bar to 100% instantly) and then does the rest of the work in-memory while the bar sticks at 100%.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 progressbar for inserts/upserts of all fileformats, closes #485 1366512990  
1248593835 https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248593835 https://api.github.com/repos/simonw/sqlite-utils/issues/486 IC_kwDOCGYnMM5KbAer simonw 9599 2022-09-15T20:37:14Z 2022-09-15T20:37:14Z OWNER

I'm going to land this anyway. The lack of a streaming JSON parser is a separate issue, I don't think it should block landing this improvement.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 progressbar for inserts/upserts of all fileformats, closes #485 1366512990  
1248582147 https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248582147 https://api.github.com/repos/simonw/sqlite-utils/issues/486 IC_kwDOCGYnMM5Ka9oD simonw 9599 2022-09-15T20:29:17Z 2022-09-15T20:29:17Z OWNER

This looks good to me. I need to run some manual tests before merging (it's a good sign that the automated tests pass though).

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 progressbar for inserts/upserts of all fileformats, closes #485 1366512990  
1248568775 https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248568775 https://api.github.com/repos/simonw/sqlite-utils/issues/486 IC_kwDOCGYnMM5Ka6XH simonw 9599 2022-09-15T20:16:14Z 2022-09-15T20:16:14Z OWNER

https://github.com/actions/setup-python/blob/main/docs/advanced-usage.md#using-the-python-version-input says can set the full version:

- uses: actions/setup-python@v4 with: python-version: "3.10.6" I'll try that.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 progressbar for inserts/upserts of all fileformats, closes #485 1366512990  
1248567323 https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248567323 https://api.github.com/repos/simonw/sqlite-utils/issues/486 IC_kwDOCGYnMM5Ka6Ab simonw 9599 2022-09-15T20:14:45Z 2022-09-15T20:14:45Z OWNER

There's a fix for mypy that has landed but isn't out in a release yet: - https://github.com/python/mypy/issues/13385

For the moment looks like pinning to Python 3.10.6 could help. Need to figure out how to do that in GitHub Actions though.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 progressbar for inserts/upserts of all fileformats, closes #485 1366512990  
1248565396 https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248565396 https://api.github.com/repos/simonw/sqlite-utils/issues/486 IC_kwDOCGYnMM5Ka5iU simonw 9599 2022-09-15T20:12:50Z 2022-09-15T20:12:50Z OWNER

Annoying mypy test failure:

/Users/runner/hostedtoolcache/Python/3.10.7/x64/lib/python3.10/site-packages/numpy/__init__.pyi:636: error: Positional-only parameters are only supported in Python 3.8 and greater Looks like this: - https://github.com/python/mypy/issues/13627

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 progressbar for inserts/upserts of all fileformats, closes #485 1366512990  

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);