issue_comments
20 rows where "created_at" is on date 2020-07-08 and "updated_at" is on date 2020-07-08 sorted by updated_at descending
This data as json, CSV (advanced)
Suggested facets: issue_url, reactions, created_at (date), updated_at (date)
id | html_url | issue_url | node_id | user | created_at | updated_at ▲ | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
655786374 | https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655786374 | https://api.github.com/repos/simonw/sqlite-utils/issues/114 | MDEyOklzc3VlQ29tbWVudDY1NTc4NjM3NA== | simonw 9599 | 2020-07-08T22:16:54Z | 2020-07-08T22:16:54Z | OWNER | According to https://www.sqlite.org/lang_altertable.html#making_other_kinds_of_table_schema_changes the hardest bits to consider are how to deal with existing foreign key relationships, triggers and views. I'm OK leaving views as an exercise for the caller - many of these transformations may not need any view changes at all. Foreign key relationships are important: it should handle these automatically as effectively as possible. Likewise trigger changes: need to think about what this means. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
table.transform() method for advanced alter table 621989740 | |
655785396 | https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655785396 | https://api.github.com/repos/simonw/sqlite-utils/issues/114 | MDEyOklzc3VlQ29tbWVudDY1NTc4NTM5Ng== | simonw 9599 | 2020-07-08T22:14:10Z | 2020-07-08T22:14:10Z | OWNER | Work in progress: not quite right yet, I need smarter logic for how renamed columns are reflected in the generated |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
table.transform() method for advanced alter table 621989740 | |
655783875 | https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655783875 | https://api.github.com/repos/simonw/sqlite-utils/issues/114 | MDEyOklzc3VlQ29tbWVudDY1NTc4Mzg3NQ== | simonw 9599 | 2020-07-08T22:09:51Z | 2020-07-08T22:10:16Z | OWNER | I can have a convenient |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
table.transform() method for advanced alter table 621989740 | |
655782477 | https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655782477 | https://api.github.com/repos/simonw/sqlite-utils/issues/114 | MDEyOklzc3VlQ29tbWVudDY1NTc4MjQ3Nw== | simonw 9599 | 2020-07-08T22:06:23Z | 2020-07-08T22:06:23Z | OWNER | Thinking about the method signature:
It would be useful if this was optional - if you could omit the columns and have it automatically use the previous columns. This would let you change things like the primary key or the column order using the other arguments. Even better: allow column renaming using an optional
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
table.transform() method for advanced alter table 621989740 | |
655778058 | https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655778058 | https://api.github.com/repos/simonw/sqlite-utils/issues/114 | MDEyOklzc3VlQ29tbWVudDY1NTc3ODA1OA== | simonw 9599 | 2020-07-08T21:54:30Z | 2020-07-08T21:54:30Z | OWNER | Don't forget this step:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
table.transform() method for advanced alter table 621989740 | |
655677909 | https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655677909 | https://api.github.com/repos/simonw/sqlite-utils/issues/114 | MDEyOklzc3VlQ29tbWVudDY1NTY3NzkwOQ== | simonw 9599 | 2020-07-08T18:16:39Z | 2020-07-08T18:16:39Z | OWNER | Since neither the term "transform" or "migrate" are used in the codebase at the moment, I think I'll go with |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
table.transform() method for advanced alter table 621989740 | |
655677396 | https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655677396 | https://api.github.com/repos/simonw/sqlite-utils/issues/114 | MDEyOklzc3VlQ29tbWVudDY1NTY3NzM5Ng== | simonw 9599 | 2020-07-08T18:15:39Z | 2020-07-08T18:15:39Z | OWNER | Alternative possible names:
- I'm torn between |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
table.transform() method for advanced alter table 621989740 | |
655677099 | https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655677099 | https://api.github.com/repos/simonw/sqlite-utils/issues/114 | MDEyOklzc3VlQ29tbWVudDY1NTY3NzA5OQ== | simonw 9599 | 2020-07-08T18:15:02Z | 2020-07-08T18:15:02Z | OWNER | I'm not so keen on that chained API - it's pretty complicated. Here's an idea for a much simpler interface. Essentially it lets you say "take table X and migrate its contents to a new table with this structure - then atomically rename the tables to switch them":
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
table.transform() method for advanced alter table 621989740 | |
655674910 | https://github.com/simonw/sqlite-utils/issues/119#issuecomment-655674910 | https://api.github.com/repos/simonw/sqlite-utils/issues/119 | MDEyOklzc3VlQ29tbWVudDY1NTY3NDkxMA== | simonw 9599 | 2020-07-08T18:10:18Z | 2020-07-08T18:10:18Z | OWNER | This will work similar to how Here's that code today - it could be adapted to include removal of foreign keys that we no longer want: |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Ability to remove a foreign key 652700770 | |
655673896 | https://github.com/simonw/sqlite-utils/issues/121#issuecomment-655673896 | https://api.github.com/repos/simonw/sqlite-utils/issues/121 | MDEyOklzc3VlQ29tbWVudDY1NTY3Mzg5Ng== | simonw 9599 | 2020-07-08T18:08:11Z | 2020-07-08T18:08:11Z | OWNER | I'm with you on most of this. Completely agreed that the CLI should do everything in a transaction. The one thing I'm not keen on is forcing calling code to explicitly start a transaction, for a couple of reasons:
So... how about this: IF you wrap your code in a That way existing code works as it does today, lazy people like me can call |
{ "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Improved (and better documented) support for transactions 652961907 | |
655653292 | https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655653292 | https://api.github.com/repos/simonw/sqlite-utils/issues/118 | MDEyOklzc3VlQ29tbWVudDY1NTY1MzI5Mg== | simonw 9599 | 2020-07-08T17:26:02Z | 2020-07-08T17:26:02Z | OWNER | Awesome, thank you very much. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add insert --truncate option 651844316 | |
655652679 | https://github.com/simonw/sqlite-utils/issues/121#issuecomment-655652679 | https://api.github.com/repos/simonw/sqlite-utils/issues/121 | MDEyOklzc3VlQ29tbWVudDY1NTY1MjY3OQ== | tsibley 79913 | 2020-07-08T17:24:46Z | 2020-07-08T17:24:46Z | CONTRIBUTOR | Better transaction handling would be really great. Some of my thoughts on implementing better transaction discipline are in https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655239728. My preferences:
```python db = sqlite_utils.Database(path) with db: # ← BEGIN issued here by Database.enter db.insert(…) db.create_view(…) ← COMMIT/ROLLBACK issue here by sqlite3.connection.exit``` |
{ "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Improved (and better documented) support for transactions 652961907 | |
655643078 | https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655643078 | https://api.github.com/repos/simonw/sqlite-utils/issues/118 | MDEyOklzc3VlQ29tbWVudDY1NTY0MzA3OA== | tsibley 79913 | 2020-07-08T17:05:59Z | 2020-07-08T17:05:59Z | CONTRIBUTOR |
Ah, yes, thanks for this reminder! I've repushed with doc bits added. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add insert --truncate option 651844316 | |
655290625 | https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655290625 | https://api.github.com/repos/simonw/sqlite-utils/issues/114 | MDEyOklzc3VlQ29tbWVudDY1NTI5MDYyNQ== | simonw 9599 | 2020-07-08T05:15:45Z | 2020-07-08T05:15:45Z | OWNER | Ideally this would all happen in a single transaction, such that other processes talking to the database would not see any inconsistent state while the table copy was taking place. Need to confirm that this is possible. Also refs transactions thoughts in #121. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
table.transform() method for advanced alter table 621989740 | |
655289686 | https://github.com/simonw/sqlite-utils/pull/120#issuecomment-655289686 | https://api.github.com/repos/simonw/sqlite-utils/issues/120 | MDEyOklzc3VlQ29tbWVudDY1NTI4OTY4Ng== | simonw 9599 | 2020-07-08T05:13:11Z | 2020-07-08T05:13:11Z | OWNER | This is an excellent fix, thanks! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Fix query command's support for DML 652816158 | |
655286864 | https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655286864 | https://api.github.com/repos/simonw/sqlite-utils/issues/118 | MDEyOklzc3VlQ29tbWVudDY1NTI4Njg2NA== | simonw 9599 | 2020-07-08T05:05:27Z | 2020-07-08T05:05:36Z | OWNER | The only thing missing from this PR is updates to the documentation. Those need to go in two places:
Here's an example of a previous commit that includes updates to both CLI and API documentation: https://github.com/simonw/sqlite-utils/commit/f9473ace14878212c1fa968b7bd2f51e4f064dba#diff-e3e2a9bfd88566b05001b02a3f51d286 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add insert --truncate option 651844316 | |
655284168 | https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655284168 | https://api.github.com/repos/simonw/sqlite-utils/issues/118 | MDEyOklzc3VlQ29tbWVudDY1NTI4NDE2OA== | simonw 9599 | 2020-07-08T04:58:00Z | 2020-07-08T04:58:00Z | OWNER | Oops didn't mean to click "close" there. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add insert --truncate option 651844316 | |
655284054 | https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655284054 | https://api.github.com/repos/simonw/sqlite-utils/issues/118 | MDEyOklzc3VlQ29tbWVudDY1NTI4NDA1NA== | simonw 9599 | 2020-07-08T04:57:38Z | 2020-07-08T04:57:38Z | OWNER | Thoughts on transactions would be much appreciated in #121 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add insert --truncate option 651844316 | |
655283393 | https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655283393 | https://api.github.com/repos/simonw/sqlite-utils/issues/118 | MDEyOklzc3VlQ29tbWVudDY1NTI4MzM5Mw== | simonw 9599 | 2020-07-08T04:55:18Z | 2020-07-08T04:55:18Z | OWNER | This is a really good idea - and thank you for the detailed discussion in the pull request. I'm keen to discuss how transactions can work better. I tend to use this pattern in my own code:
But it's not documented and I've not though very hard about it! I like having inserts that handle 10,000+ rows commit on every chunk so I can watch their progress from another process, but the library should absolutely support people who want to commit all of the rows in a single transaction - or combine changes with DML. Lots to discuss here. I'll start a new issue. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add insert --truncate option 651844316 | |
655239728 | https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655239728 | https://api.github.com/repos/simonw/sqlite-utils/issues/118 | MDEyOklzc3VlQ29tbWVudDY1NTIzOTcyOA== | tsibley 79913 | 2020-07-08T02:16:42Z | 2020-07-08T02:16:42Z | CONTRIBUTOR | I fixed my original oops by moving the I wanted to make the DELETE + INSERT happen all in the same transaction so it was robust, but that was more complicated than I expected. The transaction handling in the Database/Table classes isn't systematic, and this poses big hurdles to making For example, I wanted to do this (whitespace ignored in diff, so indentation change not highlighted): ```diff diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py index d6b9ecf..4107ceb 100644 --- a/sqlite_utils/db.py +++ b/sqlite_utils/db.py @@ -1028,6 +1028,11 @@ class Table(Queryable): batch_size = max(1, min(batch_size, SQLITE_MAX_VARS // num_columns)) self.last_rowid = None self.last_pk = None + with self.db.conn: + # Explicit BEGIN is necessary because Python's sqlite3 doesn't + # issue implicit BEGINs for DDL, only DML. We mix DDL and DML + # below and might execute DDL first, e.g. for table creation. + self.db.conn.execute("BEGIN") if truncate and self.exists(): self.db.conn.execute("DELETE FROM [{}];".format(self.name)) for chunk in chunks(itertools.chain([first_record], records), batch_size): @@ -1038,7 +1043,11 @@ class Table(Queryable): # Use the first batch to derive the table names column_types = suggest_column_types(chunk) column_types.update(columns or {}) - self.create( + # Not self.create() because that is wrapped in its own + # transaction and Python's sqlite3 doesn't support + # nested transactions. + self.db.create_table( + self.name, column_types, pk, foreign_keys, @@ -1139,7 +1148,6 @@ class Table(Queryable): flat_values = list(itertools.chain(*values)) queries_and_params = [(sql, flat_values)]
but that fails in tests because other methods call Stepping back, it would be nice to make the transaction handling systematic and predictable. One way to do this is to make the There is also the caveat that for each transaction, an explicit |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add insert --truncate option 651844316 |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issue_comments] ( [html_url] TEXT, [issue_url] TEXT, [id] INTEGER PRIMARY KEY, [node_id] TEXT, [user] INTEGER REFERENCES [users]([id]), [created_at] TEXT, [updated_at] TEXT, [author_association] TEXT, [body] TEXT, [reactions] TEXT, [issue] INTEGER REFERENCES [issues]([id]) , [performed_via_github_app] TEXT); CREATE INDEX [idx_issue_comments_issue] ON [issue_comments] ([issue]); CREATE INDEX [idx_issue_comments_user] ON [issue_comments] ([user]);
issue 5