home / github

Menu
  • Search all tables
  • GraphQL API

issue_comments

Table actions
  • GraphQL API for issue_comments

10 rows where user = 4068 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: issue_url, reactions, created_at (date), updated_at (date)

issue 7

  • bool type not supported 2
  • Cannot set type JSON 2
  • illegal UTF-16 surrogate 2
  • Better default display of arrays of items 1
  • JSON export dumps JSON fields as TEXT 1
  • CLI eats my cursor 1
  • Misleading progress bar against utf-16-le CSV input 1

user 1

  • frafra · 10 ✖

author_association 1

  • NONE 10
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
1139426398 https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1139426398 https://api.github.com/repos/simonw/sqlite-utils/issues/439 IC_kwDOCGYnMM5D6kRe frafra 4068 2022-05-27T09:04:05Z 2022-05-27T10:44:54Z NONE

This code works:

python import csv import sqlite_utils db = sqlite_utils.Database("test.db") reader = csv.DictReader(open("csv", encoding="utf-16-le").read().split("\r\n"), delimiter=";") db["test"].insert_all(reader, pk="Id")

I used iconv to change the encoding; sqlite-utils can import the resulting file, even if it stops at 98 %:

sqlite-utils insert --csv test test.db clean [------------------------------------] 0% [###################################-] 98% 00:00:00

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Misleading progress bar against utf-16-le CSV input 1250495688  
1139484453 https://github.com/simonw/sqlite-utils/issues/433#issuecomment-1139484453 https://api.github.com/repos/simonw/sqlite-utils/issues/433 IC_kwDOCGYnMM5D6ycl frafra 4068 2022-05-27T10:20:08Z 2022-05-27T10:20:08Z NONE

I can confirm. This only happens with sqlite-utils. I am using gnome-terminal with bash.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
CLI eats my cursor 1239034903  
1139392769 https://github.com/simonw/sqlite-utils/issues/438#issuecomment-1139392769 https://api.github.com/repos/simonw/sqlite-utils/issues/438 IC_kwDOCGYnMM5D6cEB frafra 4068 2022-05-27T08:21:53Z 2022-05-27T08:21:53Z NONE

Argument were specified in the wrong order. PATH TABLE FILE can be misleading :)

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
illegal UTF-16 surrogate 1250161887  
1139379923 https://github.com/simonw/sqlite-utils/issues/438#issuecomment-1139379923 https://api.github.com/repos/simonw/sqlite-utils/issues/438 IC_kwDOCGYnMM5D6Y7T frafra 4068 2022-05-27T08:05:01Z 2022-05-27T08:05:01Z NONE

I tried to debug it using pdb, but it looks sqlite-utils catches the exception, so it is not quick to figure out where the failure is happening.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
illegal UTF-16 surrogate 1250161887  
862574390 https://github.com/simonw/sqlite-utils/issues/270#issuecomment-862574390 https://api.github.com/repos/simonw/sqlite-utils/issues/270 MDEyOklzc3VlQ29tbWVudDg2MjU3NDM5MA== frafra 4068 2021-06-16T17:34:49Z 2021-06-16T17:34:49Z NONE

Sorry, I got confused because SQLite has a JSON column type, even if it is treated as TEXT, and I though automatic facets were available for JSON arrays stored as JSON only :)

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Cannot set type JSON 919314806  
860548546 https://github.com/simonw/datasette/issues/1375#issuecomment-860548546 https://api.github.com/repos/simonw/datasette/issues/1375 MDEyOklzc3VlQ29tbWVudDg2MDU0ODU0Ng== frafra 4068 2021-06-14T09:41:59Z 2021-06-14T09:41:59Z NONE

There is a feature for this at the moment, but it's a little bit hidden: you can use ?_json=col to tell Datasette that you would like a specific column to be exported as nested JSON: https://docs.datasette.io/en/stable/json_api.html#special-json-arguments

Thanks :)

I considered trying to make this automatic - so it detects columns that appear to contain valid JSON and outputs them as nested objects - but the problem with that is that it can lead to inconsistent results - you might hit the API and find that not every column contains valid JSON (compared to the previous day) resulting in the API retuning string instead of the expected dictionary and breaking your code.

If a developer is not sure if the JSON fields are valid, but then retrieves and parse them, it should handle errors too. Handling inconsistent data is necessary due to the nature of SQLite. A global or dataset option to render the data as they have been defined (JSON, boolean, etc.) when requesting JSON could allow the user to download a regular JSON from the browser without having to rely on APIs. I would guess someone could just make a custom template with an extra JSON-parsed download button otherwise :)

{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
JSON export dumps JSON fields as TEXT 919508498  
860047794 https://github.com/simonw/datasette/issues/1286#issuecomment-860047794 https://api.github.com/repos/simonw/datasette/issues/1286 MDEyOklzc3VlQ29tbWVudDg2MDA0Nzc5NA== frafra 4068 2021-06-12T12:36:15Z 2021-06-12T12:36:15Z NONE

@mroswell That is a very nice solution. I wonder if custom classes, like col-columnName-value could be automatically added to cells when facets on such column are enabled, to allow custom styling without having to modify templates or add custom JavaScript code.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Better default display of arrays of items 849220154  
860031217 https://github.com/simonw/sqlite-utils/issues/269#issuecomment-860031217 https://api.github.com/repos/simonw/sqlite-utils/issues/269 MDEyOklzc3VlQ29tbWVudDg2MDAzMTIxNw== frafra 4068 2021-06-12T10:01:53Z 2021-06-12T10:01:53Z NONE

sqlite-utils transform does not allow setting the column type to boolean: Error: Invalid value for '--type': 'bool' is not one of 'INTEGER', 'TEXT', 'FLOAT', 'BLOB'.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
bool type not supported 919250621  
860031071 https://github.com/simonw/sqlite-utils/issues/270#issuecomment-860031071 https://api.github.com/repos/simonw/sqlite-utils/issues/270 MDEyOklzc3VlQ29tbWVudDg2MDAzMTA3MQ== frafra 4068 2021-06-12T10:00:24Z 2021-06-12T10:00:24Z NONE

Sure, I am sorry if my message hasn't been clear enough. I am also a new user :)

At the beginning, I just call sqlite-utils insert "$db" "$table" "$jsonfile" to create the database. sqlite-utils convert JSON values into TEXT, when it tries to determine the schema automatically. I then try to transform the table to set JSON as type:

``` sqlite-utils transform species.sqlite species --type criteria json Usage: sqlite-utils transform [OPTIONS] PATH TABLE Try 'sqlite-utils transform --help' for help.

Error: Invalid value for '--type': 'json' is not one of 'INTEGER', 'TEXT', 'FLOAT', 'BLOB'. ```

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Cannot set type JSON 919314806  
859940977 https://github.com/simonw/sqlite-utils/issues/269#issuecomment-859940977 https://api.github.com/repos/simonw/sqlite-utils/issues/269 MDEyOklzc3VlQ29tbWVudDg1OTk0MDk3Nw== frafra 4068 2021-06-11T22:33:08Z 2021-06-11T22:33:08Z NONE

true and false json values are cast to integer, which is not optimal.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
bool type not supported 919250621  

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);
Powered by Datasette · Queries took 1.2ms · About: github-to-sqlite