issue_comments

Table actions

6 rows where issue = 1426001541 and "updated_at" is on date 2022-10-27 sorted by updated_at descending

✖
✖
✖

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 1

issue 1

  • API for bulk inserting records into a table · 6 ✖

author_association 1

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
1293893789 https://github.com/simonw/datasette/issues/1866#issuecomment-1293893789 https://api.github.com/repos/simonw/datasette/issues/1866 IC_kwDOBm6k_c5NH0Cd simonw 9599 2022-10-27T18:13:00Z 2022-10-27T18:13:00Z OWNER

If people care about that kind of thing they could always push all of their inserts to a table called _tablename and then atomically rename that once they've uploaded all of the data (assuming I provide an atomic-rename-this-table mechanism).

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 API for bulk inserting records into a table 1426001541  
1293892818 https://github.com/simonw/datasette/issues/1866#issuecomment-1293892818 https://api.github.com/repos/simonw/datasette/issues/1866 IC_kwDOBm6k_c5NHzzS simonw 9599 2022-10-27T18:12:02Z 2022-10-27T18:12:02Z OWNER

There's one catch with batched inserts: if your CLI tool fails half way through you could end up with a partially populated table - since a bunch of batches will have succeeded first.

I think that's OK. In the future I may want to come up with a way to run multiple batches of inserts inside a single transaction, but I can ignore that for the first release of this feature.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 API for bulk inserting records into a table 1426001541  
1293891876 https://github.com/simonw/datasette/issues/1866#issuecomment-1293891876 https://api.github.com/repos/simonw/datasette/issues/1866 IC_kwDOBm6k_c5NHzkk simonw 9599 2022-10-27T18:11:05Z 2022-10-27T18:11:05Z OWNER

Likewise for newline-delimited JSON. While it's tempting to want to accept that as an ingest format (because it's nice to generate and stream) I think it's better to have a client application that can turn a stream of newline-delimited JSON into batched JSON inserts.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 API for bulk inserting records into a table 1426001541  
1293891191 https://github.com/simonw/datasette/issues/1866#issuecomment-1293891191 https://api.github.com/repos/simonw/datasette/issues/1866 IC_kwDOBm6k_c5NHzZ3 simonw 9599 2022-10-27T18:10:22Z 2022-10-27T18:10:22Z OWNER

So for the moment I'm just going to concentrate on the JSON API. I can consider CSV variants later on, or as plugins, or both.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 API for bulk inserting records into a table 1426001541  
1293890684 https://github.com/simonw/datasette/issues/1866#issuecomment-1293890684 https://api.github.com/repos/simonw/datasette/issues/1866 IC_kwDOBm6k_c5NHzR8 simonw 9599 2022-10-27T18:09:52Z 2022-10-27T18:09:52Z OWNER

Should this API accept CSV/TSV etc in addition to JSON?

I'm torn on this one. My initial instinct is that it should not - and there should instead be a Datasette client library / CLI tool you can use that knows how to turn CSV into batches of JSON calls for when you want to upload a CSV file.

I don't think the usability of curl https://datasette/db/table -F 'data=@path/to/file.csv' -H 'Authentication: Bearer xxx' is particularly great compared to something likedatasette client insert https://datasette/ db table file.csv --csv (where the command version could store API tokens for you too).

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 API for bulk inserting records into a table 1426001541  
1293887808 https://github.com/simonw/datasette/issues/1866#issuecomment-1293887808 https://api.github.com/repos/simonw/datasette/issues/1866 IC_kwDOBm6k_c5NHylA simonw 9599 2022-10-27T18:07:02Z 2022-10-27T18:07:02Z OWNER

Error handling is really important here.

What should happen if you submit 100 records and one of them has some kind of validation error? How should that error be reported back to you?

I'm inclined to say that it defaults to all-or-nothing in a transaction - but there should be a "continue_on_error": true option (or similar) which causes it to insert the ones that are valid while reporting back the ones that are invalid.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 API for bulk inserting records into a table 1426001541  

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);