issue_comments

Table actions

7 rows where issue = 1366512990 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 2

author_association 2

issue 1

  • progressbar for inserts/upserts of all fileformats, closes #485 · 7 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
1248591268 https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248591268 https://api.github.com/repos/simonw/sqlite-utils/issues/486 IC_kwDOCGYnMM5Ka_2k simonw 9599 2022-09-15T20:36:02Z 2022-09-15T20:40:03Z OWNER

I had a big CSV file lying around, I converted it to other formats like this:

sqlite-utils insert /tmp/t.db t /tmp/en.openfoodfacts.org.products.csv --csv
sqlite-utils rows /tmp/t.db t --nl > /tmp/big.nl
sqlite-utils rows /tmp/t.db t > /tmp/big.json

Then tested the progress bar like this:

sqlite-utils insert /tmp/t2.db t /tmp/big.nl --nl

Output:

sqlite-utils insert /tmp/t2.db t /tmp/big.nl --nl [------------------------------------] 0% [#######-----------------------------] 20% 00:00:20 With --silent it is silent.

And for regular JSON:

sqlite-utils insert /tmp/t3.db t /tmp/big.json [####################################] 100% This is actually not doing the right thing. The problem is that sqlite-utils doesn't include a streaming JSON parser, so it instead reads that entire JSON file into memory first (exhausting the progress bar to 100% instantly) and then does the rest of the work in-memory while the bar sticks at 100%.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 progressbar for inserts/upserts of all fileformats, closes #485 1366512990  
1248593835 https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248593835 https://api.github.com/repos/simonw/sqlite-utils/issues/486 IC_kwDOCGYnMM5KbAer simonw 9599 2022-09-15T20:37:14Z 2022-09-15T20:37:14Z OWNER

I'm going to land this anyway. The lack of a streaming JSON parser is a separate issue, I don't think it should block landing this improvement.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 progressbar for inserts/upserts of all fileformats, closes #485 1366512990  
1248582147 https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248582147 https://api.github.com/repos/simonw/sqlite-utils/issues/486 IC_kwDOCGYnMM5Ka9oD simonw 9599 2022-09-15T20:29:17Z 2022-09-15T20:29:17Z OWNER

This looks good to me. I need to run some manual tests before merging (it's a good sign that the automated tests pass though).

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 progressbar for inserts/upserts of all fileformats, closes #485 1366512990  
1248527646 https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248527646 https://api.github.com/repos/simonw/sqlite-utils/issues/486 IC_kwDOCGYnMM5KawUe codecov[bot] 22429695 2022-09-15T19:34:59Z 2022-09-15T20:23:12Z NONE

Codecov Report

Base: 96.47% // Head: 96.52% // Increases project coverage by +0.04% :tada:

Coverage data is based on head (0acbc68) compared to base (d9b9e07). Patch coverage: 100.00% of modified lines in pull request are covered.

:exclamation: Current head 0acbc68 differs from pull request most recent head d5db749. Consider uploading reports for the commit d5db749 to get more accurate results

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #486 +/- ## ========================================== + Coverage 96.47% 96.52% +0.04% ========================================== Files 6 6 Lines 2642 2646 +4 ========================================== + Hits 2549 2554 +5 + Misses 93 92 -1 ``` | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/486?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/486/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.86% <100.00%> (ø)` | | | [sqlite\_utils/utils.py](https://codecov.io/gh/simonw/sqlite-utils/pull/486/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `94.98% <100.00%> (+0.47%)` | :arrow_up: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 progressbar for inserts/upserts of all fileformats, closes #485 1366512990  
1248568775 https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248568775 https://api.github.com/repos/simonw/sqlite-utils/issues/486 IC_kwDOCGYnMM5Ka6XH simonw 9599 2022-09-15T20:16:14Z 2022-09-15T20:16:14Z OWNER

https://github.com/actions/setup-python/blob/main/docs/advanced-usage.md#using-the-python-version-input says can set the full version:

- uses: actions/setup-python@v4 with: python-version: "3.10.6" I'll try that.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 progressbar for inserts/upserts of all fileformats, closes #485 1366512990  
1248567323 https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248567323 https://api.github.com/repos/simonw/sqlite-utils/issues/486 IC_kwDOCGYnMM5Ka6Ab simonw 9599 2022-09-15T20:14:45Z 2022-09-15T20:14:45Z OWNER

There's a fix for mypy that has landed but isn't out in a release yet: - https://github.com/python/mypy/issues/13385

For the moment looks like pinning to Python 3.10.6 could help. Need to figure out how to do that in GitHub Actions though.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 progressbar for inserts/upserts of all fileformats, closes #485 1366512990  
1248565396 https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248565396 https://api.github.com/repos/simonw/sqlite-utils/issues/486 IC_kwDOCGYnMM5Ka5iU simonw 9599 2022-09-15T20:12:50Z 2022-09-15T20:12:50Z OWNER

Annoying mypy test failure:

/Users/runner/hostedtoolcache/Python/3.10.7/x64/lib/python3.10/site-packages/numpy/__init__.pyi:636: error: Positional-only parameters are only supported in Python 3.8 and greater Looks like this: - https://github.com/python/mypy/issues/13627

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 progressbar for inserts/upserts of all fileformats, closes #485 1366512990  

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);