{"id": 697030843, "node_id": "MDExOlB1bGxSZXF1ZXN0NDgzMDI3NTg3", "number": 156, "title": "Typos in tests", "user": {"value": 96218, "label": "simonwiles"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-09-09T18:00:58Z", "updated_at": "2020-09-09T18:24:50Z", "closed_at": "2020-09-09T18:21:23Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/156", "body": "One of these is my fault, and the other is one I just happened to come across. They're harmless, but might as well be fixed.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/156/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 688659182, "node_id": "MDU6SXNzdWU2ODg2NTkxODI=", "number": 145, "title": "Bug when first record contains fewer columns than subsequent records", "user": {"value": 96218, "label": "simonwiles"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-08-30T05:44:44Z", "updated_at": "2020-09-08T23:21:23Z", "closed_at": "2020-09-08T23:21:23Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "`insert_all()` selects the maximum batch size based on the number of fields in the first record. If the first record has fewer fields than subsequent records (and `alter=True` is passed), this can result in SQL statements with more than the maximum permitted number of host parameters. This situation is perhaps unlikely to occur, but could happen if the first record had, say, 10 columns, such that `batch_size` (based on `SQLITE_MAX_VARIABLE_NUMBER = 999`) would be 99. If the next 98 rows had 11 columns, the resulting SQL statement for the first batch would have `10 * 1 + 11 * 98 = 1088` host parameters (and subsequent batches, if the data were consistent from thereon out, would have `99 * 11 = 1089`).\r\n\r\nI suspect that this bug is masked somewhat by the fact that while:\r\n> [`SQLITE_MAX_VARIABLE_NUMBER`](https://www.sqlite.org/limits.html#max_variable_number) ... defaults to 999 for SQLite versions prior to 3.32.0 (2020-05-22) or 32766 for SQLite versions after 3.32.0.\r\n\r\nit is common that it is increased at compile time. Debian-based systems, for example, seem to ship with a version of sqlite compiled with `SQLITE_MAX_VARIABLE_NUMBER` set to 250,000, and I believe this is the case for homebrew installations too.\r\n\r\nA test for this issue might look like this:\r\n```python\r\ndef test_columns_not_in_first_record_should_not_cause_batch_to_be_too_large(fresh_db):\r\n # sqlite on homebrew and Debian/Ubuntu etc. is typically compiled with\r\n # SQLITE_MAX_VARIABLE_NUMBER set to 250,000, so we need to exceed this value to\r\n # trigger the error on these systems.\r\n THRESHOLD = 250000\r\n extra_columns = 1 + (THRESHOLD - 1) // 99\r\n records = [\r\n {\"c0\": \"first record\"}, # one column in first record -> batch_size = 100\r\n # fill out the batch with 99 records with enough columns to exceed THRESHOLD\r\n *[\r\n dict([(\"c{}\".format(i), j) for i in range(extra_columns)])\r\n for j in range(99)\r\n ]\r\n ]\r\n try:\r\n fresh_db[\"too_many_columns\"].insert_all(records, alter=True)\r\n except sqlite3.OperationalError:\r\n raise\r\n```\r\n\r\nThe best solution, I think, is simply to process all the records when determining columns, column types, and the batch size. In my tests this doesn't seem to be particularly costly at all, and cuts out a lot of complications (including obviating my implementation of #139 at #142). I'll raise a PR for your consideration.\r\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/145/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 688668680, "node_id": "MDExOlB1bGxSZXF1ZXN0NDc1ODc0NDkz", "number": 146, "title": "Handle case where subsequent records (after first batch) include extra columns", "user": {"value": 96218, "label": "simonwiles"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2020-08-30T07:13:58Z", "updated_at": "2020-09-08T23:20:37Z", "closed_at": "2020-09-08T23:20:37Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/146", "body": "Addresses #145.\r\n\r\nI think this should do the job. If it meets with your approval I'll update this PR to include an update to the documentation -- I came across this bug while preparing a PR to update the documentation around `batch_size` in any event.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/146/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 695441530, "node_id": "MDU6SXNzdWU2OTU0NDE1MzA=", "number": 154, "title": "OperationalError: cannot change into wal mode from within a transaction", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-09-07T23:42:44Z", "updated_at": "2020-09-07T23:47:10Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "I'm getting this error when running:\r\n\r\n sqlite-utils enable-wal beta.db\r\n\r\n`OperationalError: cannot change into wal mode from within a transaction`\r\n\r\nI'm worried that maybe that's because of this new code from #152:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/deb2eb013ff85bbc828ebc244a9654f0d9c3139e/sqlite_utils/db.py#L128-L129", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/154/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 695359607, "node_id": "MDU6SXNzdWU2OTUzNTk2MDc=", "number": 150, "title": "Feature for tracing SQL queries", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-09-07T19:43:08Z", "updated_at": "2020-09-07T21:57:01Z", "closed_at": "2020-09-07T21:57:01Z", "author_association": "OWNER", "pull_request": null, "body": "Debugging `sqlite-utils` when something weird happens (e.g. #149) can be a bit tricky since it runs a bunch of different SQL statements behind the scenes.\r\n\r\nAn optional \"tracing\" mechanism for seeing what SQL is being executed would be useful.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/150/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 695360889, "node_id": "MDExOlB1bGxSZXF1ZXN0NDgxNjE2NzA0", "number": 151, "title": "Tracer mechanism for seeing underlying SQL", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-09-07T19:46:43Z", "updated_at": "2020-09-07T21:57:00Z", "closed_at": "2020-09-07T21:57:00Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/151", "body": "Refs #150. Needs tests and documentation, including for the new `db.execute()` and `db.executescript()` methods.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/151/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 695376054, "node_id": "MDU6SXNzdWU2OTUzNzYwNTQ=", "number": 152, "title": "Turn on recursive_triggers by default", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-09-07T20:26:36Z", "updated_at": "2020-09-07T21:17:48Z", "closed_at": "2020-09-07T20:45:14Z", "author_association": "OWNER", "pull_request": null, "body": "https://www.sqlite.org/pragma.html#pragma_recursive_triggers says:\r\n\r\n> Prior to SQLite [version 3.6.18](https://www.sqlite.org/releaselog/3_6_18.html) (2009-09-11), recursive triggers were not supported. The behavior of SQLite was always as if this pragma was set to OFF. Support for recursive triggers was added in version 3.6.18 but was initially turned OFF by default, for compatibility. Recursive triggers may be turned on by default in future versions of SQLite.\r\n\r\nSo I think the fix for the complex issue in #149 is to turn on `recursive_triggers` globally by default for `sqlite-utils`.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688499924_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/152/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 695319258, "node_id": "MDU6SXNzdWU2OTUzMTkyNTg=", "number": 149, "title": "FTS table with 7 rows has _fts_docsize table with 9,141 rows", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 10, "created_at": "2020-09-07T18:06:16Z", "updated_at": "2020-09-07T21:16:34Z", "closed_at": "2020-09-07T21:16:34Z", "author_association": "OWNER", "pull_request": null, "body": "I'm seeing a weird issue with some of the SQLite databases that I am using with the FTS5 module.\r\n\r\nI have a database with a `licenses` table that contains 7 rows: \r\n\r\nThe FTS table also has 7 rows: \r\n\r\nSomehow the accompanying `licenses_fts_docsize` shadow table now has 9,141 rows in it! \r\n\r\nAnd `licenses_fts_data` has 41 rows - should I expect that to have 7 rows? \r\n\r\nI have a hunch that it might be a problem with the triggers. These are the triggers that are updating that FTS table: \r\n\r\n| type | name | tbl_name | rootpage | sql |\r\n| --- | --- | --- | --- | --- |\r\n| trigger | licenses_ai | licenses | 0 | `CREATE TRIGGER [licenses_ai] AFTER INSERT ON [licenses] BEGIN INSERT INTO [licenses_fts] (rowid, [name]) VALUES (new.rowid, new.[name]); END` |\r\n| trigger | licenses_ad | licenses | 0 | `CREATE TRIGGER [licenses_ad] AFTER DELETE ON [licenses] BEGIN INSERT INTO [licenses_fts] ([licenses_fts], rowid, [name]) VALUES('delete', old.rowid, old.[name]); END` |\r\n| trigger | licenses_au | licenses | 0 | `CREATE TRIGGER [licenses_au] AFTER UPDATE ON [licenses] BEGIN INSERT INTO [licenses_fts] ([licenses_fts], rowid, [name]) VALUES('delete', old.rowid, old.[name]); INSERT INTO [licenses_fts] (rowid, [name]) VALUES (new.rowid, new.[name]); END` |", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/149/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 695276328, "node_id": "MDU6SXNzdWU2OTUyNzYzMjg=", "number": 148, "title": "More attractive indentation of created FTS table schema", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-09-07T16:49:30Z", "updated_at": "2020-09-07T18:12:50Z", "closed_at": "2020-09-07T18:12:50Z", "author_association": "OWNER", "pull_request": null, "body": "On https://github-to-sqlite.dogsheep.net/github/licenses_fts the create table SQL is displayed as:\r\n```sql\r\nCREATE VIRTUAL TABLE [licenses_fts] USING FTS5 (\r\n [name],\r\n content=[licenses]\r\n );\r\n```\r\nIt would be more aesthetically pleasing if it looked like this:\r\n```sql\r\nCREATE VIRTUAL TABLE [licenses_fts] USING FTS5 (\r\n [name],\r\n content=[licenses]\r\n);\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/148/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 688386219, "node_id": "MDExOlB1bGxSZXF1ZXN0NDc1NjY1OTg0", "number": 142, "title": "insert_all(..., alter=True) should work for new columns introduced after the first 100 records", "user": {"value": 96218, "label": "simonwiles"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-08-28T22:22:57Z", "updated_at": "2020-08-30T07:28:23Z", "closed_at": "2020-08-28T22:30:14Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/142", "body": "Closes #139.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/142/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 688395275, "node_id": "MDU6SXNzdWU2ODgzOTUyNzU=", "number": 144, "title": "Run some tests against numpy", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-08-28T22:53:00Z", "updated_at": "2020-08-28T22:57:05Z", "closed_at": "2020-08-28T22:57:04Z", "author_association": "OWNER", "pull_request": null, "body": "Accidentally removed in #143:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/d7d3f962861ef32c5ead8f514c8756f5b6f7c4a0/.travis.yml#L18-L19", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/144/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 686978131, "node_id": "MDU6SXNzdWU2ODY5NzgxMzE=", "number": 139, "title": "insert_all(..., alter=True) should work for new columns introduced after the first 100 records", "user": {"value": 96218, "label": "simonwiles"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2020-08-27T06:25:25Z", "updated_at": "2020-08-28T22:48:51Z", "closed_at": "2020-08-28T22:30:14Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "Is there a way to make `.insert_all()` work properly when new columns are introduced outside the first 100 records (with or without the `alter=True` argument)?\r\n\r\nI'm using `.insert_all()` to bulk insert ~3-4k records at a time and it is common for records to need to introduce new columns. However, if new columns are introduced after the first 100 records, `sqlite_utils` doesn't even raise the `OperationalError: table ... has no column named ...` exception; it just silently drops the extra data and moves on.\r\n\r\nIt took me a while to find this little snippet in the [documentation for `.insert_all()`](https://sqlite-utils.readthedocs.io/en/stable/python-api.html#bulk-inserts) (it's not mentioned under [Adding columns automatically on insert/update](https://sqlite-utils.readthedocs.io/en/stable/python-api.html#bulk-inserts)):\r\n\r\n> The column types used in the CREATE TABLE statement are automatically derived from the types of data in that first batch of rows. **_Any additional or missing columns in subsequent batches will be ignored._**\r\n\r\nI tried changing the `batch_size` argument to the total number of records, but it seems only to effect the number of rows that are committed at a time, and has no influence on this problem.\r\n\r\nIs there a way around this that you would suggest? It seems like it should raise an exception at least.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/139/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 688389933, "node_id": "MDU6SXNzdWU2ODgzODk5MzM=", "number": 143, "title": "Move to GitHub Actions CI", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-08-28T22:34:11Z", "updated_at": "2020-08-28T22:41:35Z", "closed_at": "2020-08-28T22:41:35Z", "author_association": "OWNER", "pull_request": null, "body": "", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/143/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 683804172, "node_id": "MDU6SXNzdWU2ODM4MDQxNzI=", "number": 134, "title": "--load-extension option for sqlite-utils query", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-08-21T20:12:42Z", "updated_at": "2020-08-21T21:06:26Z", "closed_at": "2020-08-21T20:54:19Z", "author_association": "OWNER", "pull_request": null, "body": "I got this error:\r\n```\r\n% sqlite-utils calands.db 'create table superunits_with_maps_view_concrete as select * from superunits_with_maps_view'\r\nTraceback (most recent call last):\r\n...\r\n cursor = db.conn.execute(sql, dict(param))\r\nsqlite3.OperationalError: no such function: AsGeoJSON\r\n```\r\nA `--load-extension=/usr/local/lib/mod_spatialite.dylib` option (imitating the same option for Datasette) would help.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/134/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 677839979, "node_id": "MDU6SXNzdWU2Nzc4Mzk5Nzk=", "number": 133, "title": "Release a sdist to PyPI", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-08-12T16:55:09Z", "updated_at": "2020-08-12T17:05:06Z", "closed_at": "2020-08-12T17:05:06Z", "author_association": "OWNER", "pull_request": null, "body": "https://pypi.org/project/sqlite-utils/#files currently just has a wheel. I need this to package for homebrew: https://github.com/simonw/homebrew-datasette/issues/10", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/133/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 675839512, "node_id": "MDU6SXNzdWU2NzU4Mzk1MTI=", "number": 132, "title": "Features for enabling and disabling WAL mode", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2020-08-10T03:25:44Z", "updated_at": "2020-08-10T18:59:35Z", "closed_at": "2020-08-10T18:59:35Z", "author_association": "OWNER", "pull_request": null, "body": "I finally figured out how to enable WAL - turns out it's a property of the database file itself: https://github.com/simonw/til/blob/master/sqlite/enabling-wal-mode.md", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/132/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 671130371, "node_id": "MDU6SXNzdWU2NzExMzAzNzE=", "number": 130, "title": "Support tokenize option for FTS", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-08-01T19:27:22Z", "updated_at": "2020-08-01T20:51:28Z", "closed_at": "2020-08-01T20:51:14Z", "author_association": "OWNER", "pull_request": null, "body": "FTS5 supports things like porter stemming using a `tokenize=` option:\r\n\r\nhttps://www.sqlite.org/fts5.html#tokenizers\r\n\r\nSomething like this in code:\r\n```\r\n CREATE VIRTUAL TABLE [{table}_fts] USING {fts_version} (\r\n {columns},\r\n tokenize='porter',\r\n content=[{table}]\r\n );\r\n```\r\nI tried this out just now and it worked exactly as expected.\r\n\r\nSo... `db[table].enable_fts(...) should accept a 'tokenize=` argument, and `sqlite-utils enable-fts ...` should support a `--tokenize` option.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/130/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 665802405, "node_id": "MDU6SXNzdWU2NjU4MDI0MDU=", "number": 124, "title": "sqlite-utils query should support named parameters", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-07-26T15:25:10Z", "updated_at": "2020-07-30T22:57:51Z", "closed_at": "2020-07-27T03:53:58Z", "author_association": "OWNER", "pull_request": null, "body": "To help out with escaping - so you can run this:\r\n\r\n sqlite-utils query \"insert into foo (blah) values (:blah)\" --param blah `something here`", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/124/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 668308777, "node_id": "MDU6SXNzdWU2NjgzMDg3Nzc=", "number": 129, "title": "\"insert-files --sqlar\" for creating SQLite archives", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-07-30T02:28:29Z", "updated_at": "2020-07-30T22:41:01Z", "closed_at": "2020-07-30T22:40:55Z", "author_association": "OWNER", "pull_request": null, "body": "A `--sqlar` option could cause `insert-files` to behave in the same way as SQLite's own sqlar mechanism.\r\n\r\nhttps://www.sqlite.org/sqlar.html and https://sqlite.org/sqlar/doc/trunk/README.md", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/129/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 666040390, "node_id": "MDU6SXNzdWU2NjYwNDAzOTA=", "number": 127, "title": "Ability to insert files piped to insert-files stdin", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-07-27T07:09:33Z", "updated_at": "2020-07-30T03:08:52Z", "closed_at": "2020-07-30T03:08:18Z", "author_association": "OWNER", "pull_request": null, "body": "> Inserting files by piping them in should work - but since a filename cannot be derived this will need a `--name blah.gif` option.\r\n>\r\n> cat blah.gif | sqlite-utils insert-files files.db files - --name=blah.gif\r\n>\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664128071_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/127/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 666639051, "node_id": "MDU6SXNzdWU2NjY2MzkwNTE=", "number": 128, "title": "Support UUID and memoryview types", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-07-27T23:08:34Z", "updated_at": "2020-07-30T01:10:43Z", "closed_at": "2020-07-30T01:10:43Z", "author_association": "OWNER", "pull_request": null, "body": "`psycopg2` can return data from PostgreSQL as `uuid.UUID` or `memoryview` objects. These should to be supported by `sqlite-utils` - mainly for https://github.com/simonw/db-to-sqlite", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/128/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 665700495, "node_id": "MDU6SXNzdWU2NjU3MDA0OTU=", "number": 122, "title": "CLI utility for inserting binary files into SQLite", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 10, "created_at": "2020-07-26T03:27:39Z", "updated_at": "2020-07-27T07:10:41Z", "closed_at": "2020-07-27T07:09:03Z", "author_association": "OWNER", "pull_request": null, "body": "SQLite BLOB columns can store entire binary files. The challenge is inserting them, since they don't neatly fit into JSON objects.\r\n\r\nIt would be great if the `sqlite-utils` CLI had a trick for helping with this.\r\n\r\nInspired by https://github.com/simonw/datasette-media/issues/14", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/122/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 665819048, "node_id": "MDU6SXNzdWU2NjU4MTkwNDg=", "number": 126, "title": "Ability to insert binary data on the CLI using JSON", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-07-26T16:54:14Z", "updated_at": "2020-07-27T04:00:33Z", "closed_at": "2020-07-27T03:59:45Z", "author_association": "OWNER", "pull_request": null, "body": "> I could solve round tripping (at least a bit) by allowing insert to be run with a flag that says \"these columns are base64 encoded, store the decoded data in a BLOB\".\r\n>\r\n> That would solve inserting binary data using JSON too.\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/125#issuecomment-664012247_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/126/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 665817570, "node_id": "MDU6SXNzdWU2NjU4MTc1NzA=", "number": 125, "title": "Output binary columns in \"sqlite-utils query\" JSON", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-07-26T16:47:02Z", "updated_at": "2020-07-27T00:49:41Z", "closed_at": "2020-07-27T00:48:45Z", "author_association": "OWNER", "pull_request": null, "body": "You get an error if you try to run a query that returns data from a BLOB.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/125/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 665701216, "node_id": "MDU6SXNzdWU2NjU3MDEyMTY=", "number": 123, "title": "--raw option for outputting binary content", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-07-26T03:35:39Z", "updated_at": "2020-07-26T16:44:11Z", "closed_at": "2020-07-26T16:44:11Z", "author_association": "OWNER", "pull_request": null, "body": "Related to the `insert-files` work in #122 - it should be easy to get binary data back out of the database again.\r\n\r\nOne way to do that could be:\r\n\r\n sqlite-utils files.db \"select content from files where key = 'foo.jpg'\" --raw\r\n\r\nThe `--raw` option would cause just the contents of the first column to be output directly to stdout.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/123/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 651844316, "node_id": "MDExOlB1bGxSZXF1ZXN0NDQ1MDIzMzI2", "number": 118, "title": "Add insert --truncate option", "user": {"value": 79913, "label": "tsibley"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2020-07-06T21:58:40Z", "updated_at": "2020-07-08T17:26:21Z", "closed_at": "2020-07-08T17:26:21Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/118", "body": "\r\n\r\nDeletes all rows in the table (if it exists) before inserting new rows.\r\nSQLite doesn't implement a TRUNCATE TABLE statement but does optimize an\r\nunqualified DELETE FROM.\r\n\r\nThis can be handy if you want to refresh the entire contents of a table\r\nbut a) don't have a PK (so can't use --replace), b) don't want the table\r\nto disappear (even briefly) for other connections, and c) have to handle\r\nrecords that used to exist being deleted.\r\n\r\nIdeally the replacement of rows would appear instantaneous to other\r\nconnections by putting the DELETE + INSERT in a transaction, but this is\r\nvery difficult without breaking other code as the current transaction\r\nhandling is inconsistent and non-systematic. There exists the\r\npossibility for the DELETE to succeed but the INSERT to fail, leaving an\r\nempty table. This is not much worse, however, than the current\r\npossibility of one chunked INSERT succeeding and being committed while\r\nthe next chunked INSERT fails, leaving a partially complete operation.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/118/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 652816158, "node_id": "MDExOlB1bGxSZXF1ZXN0NDQ1ODMzOTA4", "number": 120, "title": "Fix query command's support for DML", "user": {"value": 79913, "label": "tsibley"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-07-08T01:36:34Z", "updated_at": "2020-07-08T05:14:04Z", "closed_at": "2020-07-08T05:14:04Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/120", "body": "See commit messages for details. I ran into this while investigating another feature/issue.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/120/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 644161221, "node_id": "MDU6SXNzdWU2NDQxNjEyMjE=", "number": 117, "title": "Support for compound (composite) foreign keys", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-06-23T21:33:42Z", "updated_at": "2020-06-23T21:40:31Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "It turns out SQLite supports composite foreign keys: https://www.sqlite.org/foreignkeys.html#fk_composite\r\n\r\nTheir example looks like this:\r\n```sql\r\nCREATE TABLE album(\r\n albumartist TEXT,\r\n albumname TEXT,\r\n albumcover BINARY,\r\n PRIMARY KEY(albumartist, albumname)\r\n);\r\n\r\nCREATE TABLE song(\r\n songid INTEGER,\r\n songartist TEXT,\r\n songalbum TEXT,\r\n songname TEXT,\r\n FOREIGN KEY(songartist, songalbum) REFERENCES album(albumartist, albumname)\r\n);\r\n```\r\n\r\nHere's what that looks like in sqlite-utils:\r\n\r\n```\r\nIn [1]: import sqlite_utils \r\n\r\nIn [2]: import sqlite3 \r\n\r\nIn [3]: conn = sqlite3.connect(\":memory:\") \r\n\r\nIn [4]: conn \r\nOut[4]: \r\n\r\nIn [5]: conn.executescript(\"\"\" \r\n ...: CREATE TABLE album( \r\n ...: albumartist TEXT, \r\n ...: albumname TEXT, \r\n ...: albumcover BINARY, \r\n ...: PRIMARY KEY(albumartist, albumname) \r\n ...: ); \r\n ...: \r\n ...: CREATE TABLE song( \r\n ...: songid INTEGER, \r\n ...: songartist TEXT, \r\n ...: songalbum TEXT, \r\n ...: songname TEXT, \r\n ...: FOREIGN KEY(songartist, songalbum) REFERENCES album(albumartist, albumname) \r\n ...: ); \r\n ...: \"\"\") \r\nOut[5]: \r\n\r\nIn [6]: db = sqlite_utils.Database(conn) \r\n\r\nIn [7]: db.tables \r\nOut[7]: \r\n[,\r\n
]\r\n\r\nIn [8]: db.tables[0].foreign_keys \r\nOut[8]: []\r\n\r\nIn [9]: db.tables[1].foreign_keys \r\nOut[9]: \r\n[ForeignKey(table='song', column='songartist', other_table='album', other_column='albumartist'),\r\n ForeignKey(table='song', column='songalbum', other_table='album', other_column='albumname')]\r\n```\r\nThe table appears to have two separate foreign keys, when actually it has a single compound composite foreign key.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/117/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 644122661, "node_id": "MDU6SXNzdWU2NDQxMjI2NjE=", "number": 116, "title": "Documentation for table.pks introspection property", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-06-23T20:27:24Z", "updated_at": "2020-06-23T21:21:33Z", "closed_at": "2020-06-23T21:03:14Z", "author_association": "OWNER", "pull_request": null, "body": "https://github.com/simonw/sqlite-utils/blob/4d9a3204361d956440307a57bd18c829a15861db/sqlite_utils/db.py#L535-L540", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/116/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 637889964, "node_id": "MDU6SXNzdWU2Mzc4ODk5NjQ=", "number": 115, "title": "Ability to execute insert/update statements with the CLI", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-06-12T17:01:17Z", "updated_at": "2020-06-12T17:51:11Z", "closed_at": "2020-06-12T17:41:10Z", "author_association": "OWNER", "pull_request": null, "body": "```\r\n$ sqlite-utils github.db \"update stars set starred_at = ''\"\r\nTraceback (most recent call last):\r\n File \"/Users/simon/.local/bin/sqlite-utils\", line 8, in \r\n sys.exit(cli())\r\n File \"/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/sqlite_utils/cli.py\", line 673, in query\r\n headers = [c[0] for c in cursor.description]\r\nTypeError: 'NoneType' object is not iterable\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/115/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 610517472, "node_id": "MDU6SXNzdWU2MTA1MTc0NzI=", "number": 103, "title": "sqlite3.OperationalError: too many SQL variables in insert_all when using rows with varying numbers of columns", "user": {"value": 32605365, "label": "b0b5h4rp13"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 8, "created_at": "2020-05-01T02:26:14Z", "updated_at": "2020-05-14T00:18:57Z", "closed_at": "2020-05-14T00:18:57Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "If using insert_all to put in 1000 rows of data with varying number of columns, it comes up with this message `sqlite3.OperationalError: too many SQL variables` if the number of columns is larger in later records (past the first row)\r\n\r\nI've reduced `SQLITE_MAX_VARS` by 100 to 899 at the top of `db.py` to add wiggle room, so that if the column count increases it wont go past SQLite's batch limit as calculated by this line of code based on the count of the first row's dict keys\r\n\r\n batch_size = max(1, min(batch_size, SQLITE_MAX_VARS // num_columns))", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/103/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 461215118, "node_id": "MDU6SXNzdWU0NjEyMTUxMTg=", "number": 30, "title": "Option to open database in read-only mode", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2019-06-26T22:50:38Z", "updated_at": "2020-05-11T19:17:17Z", "closed_at": "2020-05-11T19:17:17Z", "author_association": "OWNER", "pull_request": null, "body": "Would this make it 100% safe to run reads against a database file that is being written to by another process?", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/30/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 615477131, "node_id": "MDU6SXNzdWU2MTU0NzcxMzE=", "number": 111, "title": "sqlite-utils drop-table and drop-view commands", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-05-10T21:10:42Z", "updated_at": "2020-05-11T01:58:36Z", "closed_at": "2020-05-11T00:44:26Z", "author_association": "OWNER", "pull_request": null, "body": "Would be useful to be able to drop views and tables from the CLI.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/111/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 613755043, "node_id": "MDU6SXNzdWU2MTM3NTUwNDM=", "number": 110, "title": "Support decimal.Decimal type", "user": {"value": 134771, "label": "dvhthomas"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2020-05-07T03:57:19Z", "updated_at": "2020-05-11T01:58:20Z", "closed_at": "2020-05-11T01:50:11Z", "author_association": "NONE", "pull_request": null, "body": "Decimal types in Postgres cause a failure in db.py data type selection\r\n---\r\nI have a Django app using a MoneyField, which uses a `numeric(14,0)` data type in Postgres (https://www.postgresql.org/docs/9.3/datatype-numeric.html). When attempting to export that table I get the following error:\r\n\r\n```bash\r\n$ db-to-sqlite --table isaweb_proposal \"postgres://connection\" test.db\r\n....\r\n column_type=COLUMN_TYPE_MAPPING[column_type],\r\nKeyError: \r\n```\r\n\r\nLooking at `sql_utils.db.py` at 292-ish it's clear that there is no matching type for what I assume SQLAlchemy interprets as Python decimal.Decimal.\r\n\r\nFrom the [SQLite docs](https://www.sqlite.org/datatype3.html#affinity_name_examples) it looks like DECIMAL in other DBs are considered numeric.\r\n\r\nI'm not quite sure if it's as simple as adding a data type to that list or if there are repercussions beyond it.\r\n\r\nThanks for a great tool!", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/110/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 612658444, "node_id": "MDU6SXNzdWU2MTI2NTg0NDQ=", "number": 109, "title": "table.create_index(..., ignore=True)", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-05-05T14:44:21Z", "updated_at": "2020-05-05T14:46:53Z", "closed_at": "2020-05-05T14:46:53Z", "author_association": "OWNER", "pull_request": null, "body": "Option to silently do nothing if the index already exists.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/109/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 611222968, "node_id": "MDU6SXNzdWU2MTEyMjI5Njg=", "number": 107, "title": "sqlite-utils create-view CLI command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-05-02T16:15:13Z", "updated_at": "2020-05-03T15:36:58Z", "closed_at": "2020-05-03T15:36:37Z", "author_association": "OWNER", "pull_request": null, "body": "Can go with #27 - `sqlite-utils create-table`.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/107/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 455496504, "node_id": "MDU6SXNzdWU0NTU0OTY1MDQ=", "number": 27, "title": "sqlite-utils create-table command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 8, "created_at": "2019-06-13T01:43:30Z", "updated_at": "2020-05-03T15:26:15Z", "closed_at": "2020-05-03T15:26:15Z", "author_association": "OWNER", "pull_request": null, "body": "Spun off from #24 - it would be useful if CLI users could create new tables (with explicit column types, not null rules and defaults) without having to insert an example record.\r\n\r\n- [x] Get it working\r\n- [x] Support `--pk`\r\n- [x] Support `--not-null`\r\n- [x] Support `--default`\r\n- [x] Support `--fk colname othertable othercol`\r\n- [x] Support `--replace` and `--ignore`\r\n- [x] Documentation", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/27/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 611326701, "node_id": "MDU6SXNzdWU2MTEzMjY3MDE=", "number": 108, "title": "Documentation unit tests for CLI commands", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-05-03T03:58:42Z", "updated_at": "2020-05-03T04:13:57Z", "closed_at": "2020-05-03T04:13:57Z", "author_association": "OWNER", "pull_request": null, "body": "Have a test that ensures all CLI commands are documented.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/108/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 611216862, "node_id": "MDU6SXNzdWU2MTEyMTY4NjI=", "number": 106, "title": "create_view(..., ignore=True, replace=True) parameters", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-05-02T15:45:21Z", "updated_at": "2020-05-02T16:04:51Z", "closed_at": "2020-05-02T16:02:10Z", "author_association": "OWNER", "pull_request": null, "body": "Two new parameters which specify what should happen if the view already exists. I want this for https://github.com/dogsheep/github-to-sqlite/issues/37\r\n\r\nHere's the current `create_view()` implementation:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/b4d953d3ccef28bb81cea40ca165a647b59971fa/sqlite_utils/db.py#L325-L332\r\n\r\n`ignore=True` will not do anything if the view exists already.\r\n\r\n`replace=True` will drop and redefine the view - but only if its SQL definition differs, otherwise it will be left alone.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/106/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 602569315, "node_id": "MDU6SXNzdWU2MDI1NjkzMTU=", "number": 102, "title": "Can't store an array or dictionary containing a bytes value", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-04-18T22:49:21Z", "updated_at": "2020-05-01T20:45:45Z", "closed_at": "2020-05-01T20:45:45Z", "author_association": "OWNER", "pull_request": null, "body": "```\r\nIn [1]: import sqlite_utils \r\n\r\nIn [2]: db = sqlite_utils.Database(memory=True) \r\n\r\nIn [3]: db[\"t\"].insert({\"id\": 1, \"data\": {\"foo\": b\"bytes\"}}) \r\n---------------------------------------------------------------------------\r\nTypeError Traceback (most recent call last)\r\n in \r\n----> 1 db[\"t\"].insert({\"id\": 1, \"data\": {\"foo\": b\"bytes\"}})\r\n\r\n~/Dropbox/Development/sqlite-utils/sqlite_utils/db.py in insert(self, record, pk, foreign_keys, column_order, not_null, defaults, hash_id, alter, ignore, replace, extracts, conversions, columns)\r\n 950 extracts=extracts,\r\n 951 conversions=conversions,\r\n--> 952 columns=columns,\r\n 953 )\r\n 954 \r\n\r\n~/Dropbox/Development/sqlite-utils/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, ignore, replace, extracts, conversions, columns, upsert)\r\n 1052 for key in all_columns:\r\n 1053 value = jsonify_if_needed(\r\n-> 1054 record.get(key, None if key != hash_id else _hash(record))\r\n 1055 )\r\n 1056 if key in extracts:\r\n\r\n~/Dropbox/Development/sqlite-utils/sqlite_utils/db.py in jsonify_if_needed(value)\r\n 1318 def jsonify_if_needed(value):\r\n 1319 if isinstance(value, (dict, list, tuple)):\r\n-> 1320 return json.dumps(value)\r\n 1321 elif isinstance(value, (datetime.time, datetime.date, datetime.datetime)):\r\n 1322 return value.isoformat()\r\n\r\n/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py in dumps(obj, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, default, sort_keys, **kw)\r\n 229 cls is None and indent is None and separators is None and\r\n 230 default is None and not sort_keys and not kw):\r\n--> 231 return _default_encoder.encode(obj)\r\n 232 if cls is None:\r\n 233 cls = JSONEncoder\r\n\r\n/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py in encode(self, o)\r\n 197 # exceptions aren't as detailed. The list call should be roughly\r\n 198 # equivalent to the PySequence_Fast that ''.join() would do.\r\n--> 199 chunks = self.iterencode(o, _one_shot=True)\r\n 200 if not isinstance(chunks, (list, tuple)):\r\n 201 chunks = list(chunks)\r\n\r\n/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py in iterencode(self, o, _one_shot)\r\n 255 self.key_separator, self.item_separator, self.sort_keys,\r\n 256 self.skipkeys, _one_shot)\r\n--> 257 return _iterencode(o, 0)\r\n 258 \r\n 259 def _make_iterencode(markers, _default, _encoder, _indent, _floatstr,\r\n\r\n/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py in default(self, o)\r\n 177 \r\n 178 \"\"\"\r\n--> 179 raise TypeError(f'Object of type {o.__class__.__name__} '\r\n 180 f'is not JSON serializable')\r\n 181 \r\n\r\nTypeError: Object of type bytes is not JSON serializable\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/102/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 610853576, "node_id": "MDU6SXNzdWU2MTA4NTM1NzY=", "number": 105, "title": "\"sqlite-utils views\" command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-05-01T16:56:11Z", "updated_at": "2020-05-01T20:40:07Z", "closed_at": "2020-05-01T20:38:36Z", "author_association": "OWNER", "pull_request": null, "body": "Similar to `sqlite-utils tables`. See also #104.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/105/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 610853393, "node_id": "MDU6SXNzdWU2MTA4NTMzOTM=", "number": 104, "title": "--schema option to \"sqlite-utils tables\"", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-05-01T16:55:49Z", "updated_at": "2020-05-01T17:12:37Z", "closed_at": "2020-05-01T17:12:37Z", "author_association": "OWNER", "pull_request": null, "body": "Adds output showing the table schema.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/104/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 601392318, "node_id": "MDU6SXNzdWU2MDEzOTIzMTg=", "number": 101, "title": "README should include an example of CLI data insertion", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-04-16T19:45:37Z", "updated_at": "2020-04-17T23:59:49Z", "closed_at": "2020-04-17T23:59:49Z", "author_association": "OWNER", "pull_request": null, "body": "Maybe using `curl` from the GitHub API.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/101/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 601358649, "node_id": "MDU6SXNzdWU2MDEzNTg2NDk=", "number": 100, "title": "Mechanism for forcing column-type, over-riding auto-detection", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-04-16T19:12:52Z", "updated_at": "2020-04-17T23:53:32Z", "closed_at": "2020-04-17T23:53:32Z", "author_association": "OWNER", "pull_request": null, "body": "As seen in https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-614843406 - there's a problem where you insert a record with a `None` value for a column and that column is created as `TEXT` - but actually you intended it to be an `INT` (as later examples will demonstrate).\r\n\r\nSome kind of mechanism for over-riding the detected types of columns would be useful here.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/100/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 549287310, "node_id": "MDU6SXNzdWU1NDkyODczMTA=", "number": 76, "title": "order_by mechanism", "user": {"value": 10501166, "label": "metab0t"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-01-14T02:06:03Z", "updated_at": "2020-04-16T06:23:29Z", "closed_at": "2020-04-16T03:13:06Z", "author_association": "NONE", "pull_request": null, "body": "In some cases, I want to iterate rows in a table with `ORDER BY` clause. It would be nice to have a `rows_order_by` function similar to `rows_where`.\r\nIn a more general case, `rows_filter` function might be added to allow more customized filtering to iterate rows.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/76/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 593751293, "node_id": "MDU6SXNzdWU1OTM3NTEyOTM=", "number": 97, "title": "Adding a \"recreate\" flag to the `Database` constructor", "user": {"value": 1448859, "label": "betatim"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-04-04T05:41:10Z", "updated_at": "2020-04-15T14:29:31Z", "closed_at": "2020-04-13T03:52:29Z", "author_association": "NONE", "pull_request": null, "body": "I have a [script](https://github.com/betatim/binder-datasette/blob/master/create-db.ipynb) that imports data into a sqlite DB. When I re-run that script I'd like to remove the existing sqlite DB, instead of adding to it. The pragmatic answer is to add the check and file deletion to my script.\r\n\r\nHowever I thought it would be easy and useful for others to add a `recreate=True` flag to `db = sqlite_utils.Database(\"binder-launches.db\")`. After taking a look at the code for it I am not so sure any more. This is because the connection string could be a URL (or \"connection string\") like `\"file:///tmp/foo.db\"`. I don't know what the equivalent of `os.path.exists()` is for a connection string or how to detect that something is a connection string and raise an error \"can't use recreate=True and conn_string at the same time\".\r\n\r\nDoes anyone have an idea/suggestion where to start investigating?", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/97/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 598640234, "node_id": "MDU6SXNzdWU1OTg2NDAyMzQ=", "number": 99, "title": ".upsert_all() should maybe error if dictionaries passed to it do not have the same keys", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-04-13T03:02:25Z", "updated_at": "2020-04-13T03:05:20Z", "closed_at": "2020-04-13T03:05:04Z", "author_association": "OWNER", "pull_request": null, "body": "While investigating #98 I stumbled across this:\r\n```\r\n def test_upsert_compound_primary_key(fresh_db):\r\n table = fresh_db[\"table\"]\r\n table.upsert_all(\r\n [\r\n {\"species\": \"dog\", \"id\": 1, \"name\": \"Cleo\", \"age\": 4},\r\n {\"species\": \"cat\", \"id\": 1, \"name\": \"Catbag\"},\r\n ],\r\n pk=(\"species\", \"id\"),\r\n )\r\n table.upsert_all(\r\n [\r\n {\"species\": \"dog\", \"id\": 1, \"age\": 5},\r\n {\"species\": \"dog\", \"id\": 2, \"name\": \"New Dog\", \"age\": 1},\r\n ],\r\n pk=(\"species\", \"id\"),\r\n )\r\n> assert [\r\n {\"species\": \"dog\", \"id\": 1, \"name\": \"Cleo\", \"age\": 5},\r\n {\"species\": \"cat\", \"id\": 1, \"name\": \"Catbag\", \"age\": None},\r\n {\"species\": \"dog\", \"id\": 2, \"name\": \"New Dog\", \"age\": 1},\r\n ] == list(table.rows)\r\nE AssertionError: assert [{'age': 5, '...cies': 'dog'}] == [{'age': 5, '...cies': 'dog'}]\r\nE At index 0 diff: {'species': 'dog', 'id': 1, 'name': 'Cleo', 'age': 5} != {'species': 'dog', 'id': 1, 'name': None, 'age': 5}\r\nE Full diff:\r\nE - [{'age': 5, 'id': 1, 'name': 'Cleo', 'species': 'dog'},\r\nE ? ^^^ --\r\nE + [{'age': 5, 'id': 1, 'name': None, 'species': 'dog'},\r\nE ? ^^^\r\nE {'age': None, 'id': 1, 'name': 'Catbag', 'species': 'cat'},\r\nE {'age': 1, 'id': 2, 'name': 'New Dog', 'species': 'dog'}]\r\n```\r\nIf you run `.upsert_all()` with multiple dictionaries it doesn't quite have the effect you might expect.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/99/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 589801352, "node_id": "MDExOlB1bGxSZXF1ZXN0Mzk1MjU4Njg3", "number": 96, "title": "Add type conversion for Panda's Timestamp", "user": {"value": 32605365, "label": "b0b5h4rp13"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-03-29T14:13:09Z", "updated_at": "2020-03-31T04:40:49Z", "closed_at": "2020-03-31T04:40:48Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/96", "body": "Add type conversion for Panda's Timestamp, if Panda library is present in system\r\n(thanks for this project, I was about to do the same thing from scratch)", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/96/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 586486367, "node_id": "MDU6SXNzdWU1ODY0ODYzNjc=", "number": 95, "title": "Columns with only null values are no longer created in the database", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-03-23T20:07:42Z", "updated_at": "2020-03-23T20:31:15Z", "closed_at": "2020-03-23T20:31:15Z", "author_association": "OWNER", "pull_request": null, "body": "Bug introduced in #94, and released in `2.4.3`.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/95/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 586477757, "node_id": "MDU6SXNzdWU1ODY0Nzc3NTc=", "number": 94, "title": "If column data is a mixture of integers and nulls, detected type should be INTEGER", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-03-23T19:51:46Z", "updated_at": "2020-03-23T19:57:10Z", "closed_at": "2020-03-23T19:57:10Z", "author_association": "OWNER", "pull_request": null, "body": "It looks like detected type for that case is TEXT at the moment.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/94/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 581339961, "node_id": "MDU6SXNzdWU1ODEzMzk5NjE=", "number": 92, "title": ".columns_dict doesn't work for all possible column types", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2020-03-14T19:30:35Z", "updated_at": "2020-03-15T18:37:43Z", "closed_at": "2020-03-14T20:04:14Z", "author_association": "OWNER", "pull_request": null, "body": "Got this error:\r\n```\r\n File \".../python3.7/site-packages/sqlite_utils/db.py\", line 462, in \r\n for column in self.columns\r\nKeyError: 'REAL'\r\n```\r\n`.columns_dict` uses `REVERSE_COLUMN_TYPE_MAPPING`:\r\nhttps://github.com/simonw/sqlite-utils/blob/43f1c6ab4e3a6b76531fb6f5447adb83d26f3971/sqlite_utils/db.py#L457-L463\r\n`REVERSE_COLUMN_TYPE_MAPPING` defines `FLOAT` not `REAL`A\r\nhttps://github.com/simonw/sqlite-utils/blob/43f1c6ab4e3a6b76531fb6f5447adb83d26f3971/sqlite_utils/db.py#L68-L74", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/92/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 573740712, "node_id": "MDU6SXNzdWU1NzM3NDA3MTI=", "number": 90, "title": "Cannot .enable_fts() for columns with spaces in their names", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-03-02T06:06:03Z", "updated_at": "2020-03-02T06:10:49Z", "closed_at": "2020-03-02T06:10:49Z", "author_association": "OWNER", "pull_request": null, "body": "```\r\nimport sqlite_utils\r\ndb = sqlite_utils.Database(memory=True) \r\ndb[\"test\"].insert({\"space in name\": \"hello\"}) \r\ndb[\"test\"].enable_fts([\"space in name\"]) \r\n---------------------------------------------------------------------------\r\nOperationalError Traceback (most recent call last)\r\n in \r\n----> 1 db['test'].enable_fts([\"space in name\"])\r\n\r\n/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in enable_fts(self, columns, fts_version, create_triggers)\r\n 755 )\r\n 756 self.db.conn.executescript(sql)\r\n--> 757 self.populate_fts(columns)\r\n 758 \r\n 759 if create_triggers:\r\n\r\n/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in populate_fts(self, columns)\r\n 787 table=self.name, columns=\", \".join(columns)\r\n 788 )\r\n--> 789 self.db.conn.executescript(sql)\r\n 790 return self\r\n 791 \r\n\r\nOperationalError: near \"in\": syntax error\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/90/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 471780443, "node_id": "MDU6SXNzdWU0NzE3ODA0NDM=", "number": 46, "title": "extracts= option for insert/update/etc", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2019-07-23T15:55:46Z", "updated_at": "2020-03-01T16:53:40Z", "closed_at": "2019-07-23T17:00:44Z", "author_association": "OWNER", "pull_request": null, "body": "Relates to #42 and #44. I want the ability to extract values out into lookup tables during bulk insert/upsert operations.\r\n\r\n`db.insert_all(rows, extracts=[\"species\"])`\r\n\r\n- creates species table for values in the species column\r\n\r\n`db.insert_all(rows, extracts={\"species\": \"Species\"})`\r\n\r\n- as above but the new table is called `Species`.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/46/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 571805300, "node_id": "MDU6SXNzdWU1NzE4MDUzMDA=", "number": 88, "title": "table.disable_fts() method and \"sqlite-utils disable-fts ...\" command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2020-02-27T04:00:50Z", "updated_at": "2020-02-27T04:40:44Z", "closed_at": "2020-02-27T04:40:44Z", "author_association": "OWNER", "pull_request": null, "body": "This would make it easier to iterate on the FTS configuration for a database without having to wipe and recreate the database each time.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/88/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 539204432, "node_id": "MDU6SXNzdWU1MzkyMDQ0MzI=", "number": 70, "title": "Implement ON DELETE and ON UPDATE actions for foreign keys", "user": {"value": 26292069, "label": "LucasElArruda"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-12-17T17:19:10Z", "updated_at": "2020-02-27T04:18:53Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Hi! I did not find any mention on the library about ON DELETE and ON UPDATE actions for foreign keys. Are those expected to be implemented? If not, it would be a nice thing to include!", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/70/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 559197745, "node_id": "MDU6SXNzdWU1NTkxOTc3NDU=", "number": 82, "title": "Tutorial command no longer works", "user": {"value": 10350886, "label": "petey284"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-02-03T16:36:11Z", "updated_at": "2020-02-27T04:16:43Z", "closed_at": "2020-02-27T04:16:30Z", "author_association": "NONE", "pull_request": null, "body": "Issue with command on [tutorial](https://simonwillison.net/2019/Feb/25/sqlite-utils/) on Simon's site.\r\n\r\nThe following command no longer works, and breaks with the previous too many variables error: #50\r\n\r\n``` cmd\r\n> curl \"https://data.nasa.gov/resource/y77d-th95.json\" | \\\r\n sqlite-utils insert meteorites.db meteorites - --pk=id\r\n```\r\n\r\nOutput:\r\n``` cmd\r\nTraceback (most recent call last):\r\n File \"continuum\\miniconda3\\envs\\main\\lib\\runpy.py\", line 193, in _run_module_as_main\r\n \"__main__\", mod_spec)\r\n File \"continuum\\miniconda3\\envs\\main\\lib\\runpy.py\", line 85, in _run_code\r\n exec(code, run_globals)\r\n File \"Continuum\\miniconda3\\envs\\main\\Scripts\\sqlite-utils.exe\\__main__.py\", line 9, in \r\n File \"continuum\\miniconda3\\envs\\main\\lib\\site-packages\\click\\core.py\", line 764, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"continuum\\miniconda3\\envs\\main\\lib\\site-packages\\click\\core.py\", line 717, in main\r\n rv = self.invoke(ctx)\r\n File \"continuum\\miniconda3\\envs\\main\\lib\\site-packages\\click\\core.py\", line 1137, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"continuum\\miniconda3\\envs\\main\\lib\\site-packages\\click\\core.py\", line 956, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"continuum\\miniconda3\\envs\\main\\lib\\site-packages\\click\\core.py\", line 555, in invoke\r\n return callback(*args, **kwargs)\r\n File \"continuum\\miniconda3\\envs\\main\\lib\\site-packages\\sqlite_utils\\cli.py\", line 434, in insert\r\n default=default,\r\n File \"continuum\\miniconda3\\envs\\main\\lib\\site-packages\\sqlite_utils\\cli.py\", line 384, in insert_upsert_implementation\r\n docs, pk=pk, batch_size=batch_size, alter=alter, **extra_kwargs\r\n File \"continuum\\miniconda3\\envs\\main\\lib\\site-packages\\sqlite_utils\\db.py\", line 1081, in insert_all\r\n result = self.db.conn.execute(query, params)\r\nsqlite3.OperationalError: too many SQL variables\r\n```\r\n\r\nMy thought is that maybe the dataset grew over the last few years and so didn't run into this issue before.\r\n\r\nNo error when I reduce the count of entries to 83. Once the number of entries hits 84 the command fails.\r\n\r\n// This passes\r\n``` cmd\r\ntype meteorite_83.txt | sqlite-utils insert meteorites.db meteorites - --pk=id\r\n```\r\n\r\n// But this fails\r\n``` cmd\r\ntype meteorite_84.txt | sqlite-utils insert meteorites.db meteorites - --pk=id\r\n```\r\n\r\nA potential fix might be to chunk the incoming data? I can work on a PR if pointed in right direction.\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/82/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 564579430, "node_id": "MDU6SXNzdWU1NjQ1Nzk0MzA=", "number": 86, "title": "Problem with square bracket in CSV column name", "user": {"value": 8149512, "label": "foscoj"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2020-02-13T10:19:57Z", "updated_at": "2020-02-27T04:16:08Z", "closed_at": "2020-02-27T04:16:07Z", "author_association": "NONE", "pull_request": null, "body": "testing some data from european power information (entsoe.eu), the title of the csv contains square brackets.\r\nas I am playing with glitch, sqlite-utils are used for creating the db.\r\n\r\nTraceback (most recent call last):\r\n\r\n File \"/app/.local/bin/sqlite-utils\", line 8, in \r\n\r\n sys.exit(cli())\r\n\r\n File \"/app/.local/lib/python3.7/site-packages/click/core.py\", line 764, in __call__\r\n\r\n return self.main(*args, **kwargs)\r\n\r\n File \"/app/.local/lib/python3.7/site-packages/click/core.py\", line 717, in main\r\n\r\n rv = self.invoke(ctx)\r\n\r\n File \"/app/.local/lib/python3.7/site-packages/click/core.py\", line 1137, in invoke\r\n\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n\r\n File \"/app/.local/lib/python3.7/site-packages/click/core.py\", line 956, in invoke\r\n\r\n return ctx.invoke(self.callback, **ctx.params)\r\n\r\n File \"/app/.local/lib/python3.7/site-packages/click/core.py\", line 555, in invoke\r\n\r\n return callback(*args, **kwargs)\r\n\r\n File \"/app/.local/lib/python3.7/site-packages/sqlite_utils/cli.py\", line 434, in insert\r\n\r\n default=default,\r\n\r\n File \"/app/.local/lib/python3.7/site-packages/sqlite_utils/cli.py\", line 384, in insert_upsert_implementation\r\n\r\n docs, pk=pk, batch_size=batch_size, alter=alter, **extra_kwargs\r\n\r\n File \"/app/.local/lib/python3.7/site-packages/sqlite_utils/db.py\", line 997, in insert_all\r\n\r\n extracts=extracts,\r\n\r\n File \"/app/.local/lib/python3.7/site-packages/sqlite_utils/db.py\", line 618, in create\r\n\r\n extracts=extracts,\r\n\r\n File \"/app/.local/lib/python3.7/site-packages/sqlite_utils/db.py\", line 310, in create_table\r\n\r\n self.conn.execute(sql)\r\n\r\nsqlite3.OperationalError: unrecognized token: \"]\"\r\n\r\nentsoe_2016.csv\r\n\r\nrenamed to txt for uploading compatibility\r\n\r\n[entsoe_2016.txt](https://github.com/simonw/sqlite-utils/files/4197688/entsoe_2016.txt)\r\n\r\ncode is remixed directly from your https://glitch.com/edit/#!/datasette-csvs repo\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/86/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 565837965, "node_id": "MDU6SXNzdWU1NjU4Mzc5NjU=", "number": 87, "title": "Should detect collections.OrderedDict as a regular dictionary", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-02-16T02:06:34Z", "updated_at": "2020-02-16T02:20:59Z", "closed_at": "2020-02-16T02:20:59Z", "author_association": "OWNER", "pull_request": null, "body": "```\r\n File \"...python3.7/site-packages/sqlite_utils/db.py\", line 292, in create_table\r\n column_type=COLUMN_TYPE_MAPPING[column_type],\r\nKeyError: \r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/87/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 562911863, "node_id": "MDU6SXNzdWU1NjI5MTE4NjM=", "number": 85, "title": "Create index doesn't work for columns containing spaces", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-02-11T00:34:46Z", "updated_at": "2020-02-11T05:13:20Z", "closed_at": "2020-02-11T05:13:20Z", "author_association": "OWNER", "pull_request": null, "body": "", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/85/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 559374410, "node_id": "MDU6SXNzdWU1NTkzNzQ0MTA=", "number": 83, "title": "Make db[\"table\"].exists a documented API", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-02-03T22:31:44Z", "updated_at": "2020-02-08T23:58:35Z", "closed_at": "2020-02-08T23:56:23Z", "author_association": "OWNER", "pull_request": null, "body": "Right now it's a static thing which might get out-of-sync with the database. It should probably be a live check. Maybe call it `.exists()` instead?", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/83/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 561460274, "node_id": "MDU6SXNzdWU1NjE0NjAyNzQ=", "number": 84, "title": ".upsert() with hash_id throws error", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-02-07T07:08:19Z", "updated_at": "2020-02-07T07:17:11Z", "closed_at": "2020-02-07T07:17:11Z", "author_association": "OWNER", "pull_request": null, "body": "```python\r\ndb[table_name].upsert_all(rows, hash_id=\"pk\")\r\n```\r\nThis throws an error: `PrimaryKeyRequired('upsert() requires a pk')`\r\n\r\nThe problem is, if you try this:\r\n\r\n```python\r\ndb[table_name].upsert_all(rows, hash_id=\"pk\", pk=\"pk\")\r\n```\r\nYou get this error: `AssertionError('Use either pk= or hash_id=')`\r\n\r\n`hash_id=` should imply that `pk=` that column.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/84/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 558600274, "node_id": "MDU6SXNzdWU1NTg2MDAyNzQ=", "number": 81, "title": "Remove .detect_column_types() from table, make it a documented API", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-02-01T21:25:54Z", "updated_at": "2020-02-01T21:55:35Z", "closed_at": "2020-02-01T21:55:35Z", "author_association": "OWNER", "pull_request": null, "body": "I used it in `geojson-to-sqlite` here: https://github.com/simonw/geojson-to-sqlite/blob/f10e44264712dd59ae7dfa2e6fd5a904b682fb33/geojson_to_sqlite/utils.py#L45-L50\r\n\r\nIt would make more sense for this method to live on the Database rather than the Table - or even to exist as a separate utility method entirely.\r\n\r\nThen it should be documented.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/81/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 545407916, "node_id": "MDU6SXNzdWU1NDU0MDc5MTY=", "number": 73, "title": "upsert_all() throws issue when upserting to empty table", "user": {"value": 82988, "label": "psychemedia"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2020-01-05T11:58:57Z", "updated_at": "2020-01-31T14:21:09Z", "closed_at": "2020-01-05T17:20:18Z", "author_association": "NONE", "pull_request": null, "body": "If I try to add a list of `dict`s to an empty table using `upsert_all`, I get an error:\r\n\r\n```python\r\nimport sqlite3\r\nfrom sqlite_utils import Database\r\nimport pandas as pd\r\n\r\nconx = sqlite3.connect(':memory')\r\ncx = conx.cursor()\r\ncx.executescript('CREATE TABLE \"test\" (\"Col1\" TEXT);')\r\n\r\nq=\"SELECT * FROM test;\"\r\npd.read_sql(q, conx) #shows empty table\r\n\r\ndb = Database(conx)\r\ndb['test'].upsert_all([{'Col1':'a'},{'Col1':'b'}])\r\n\r\n---------------------------------------------------------------------------\r\nTypeError Traceback (most recent call last)\r\n in \r\n 1 db = Database(conx)\r\n----> 2 db['test'].upsert_all([{'Col1':'a'},{'Col1':'b'}])\r\n\r\n/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in upsert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, extracts)\r\n 1157 alter=alter,\r\n 1158 extracts=extracts,\r\n-> 1159 upsert=True,\r\n 1160 )\r\n 1161 \r\n\r\n/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, ignore, replace, extracts, upsert)\r\n 1040 sql = \"INSERT OR IGNORE INTO [{table}]({pks}) VALUES({pk_placeholders});\".format(\r\n 1041 table=self.name,\r\n-> 1042 pks=\", \".join([\"[{}]\".format(p) for p in pks]),\r\n 1043 pk_placeholders=\", \".join([\"?\" for p in pks]),\r\n 1044 )\r\n\r\nTypeError: 'NoneType' object is not iterable\r\n\r\n```\r\n\r\nA hacky workaround in use is:\r\n\r\n```python\r\ntry:\r\n db['test'].upsert_all([{'Col1':'a'},{'Col1':'b'}])\r\nexcept:\r\n db['test'].insert_all([{'Col1':'a'},{'Col1':'b'}])\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/73/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 557892819, "node_id": "MDExOlB1bGxSZXF1ZXN0MzY5Mzk0MDQz", "number": 80, "title": "on_create mechanism for after table creation", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2020-01-31T03:38:48Z", "updated_at": "2020-01-31T05:08:04Z", "closed_at": "2020-01-31T05:08:04Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/80", "body": "I need this for `geojson-to-sqlite`, in particular https://github.com/simonw/geojson-to-sqlite/issues/6", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/80/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 546078359, "node_id": "MDExOlB1bGxSZXF1ZXN0MzU5ODIyNzcz", "number": 75, "title": "Explicitly include tests and docs in sdist", "user": {"value": 15092, "label": "jayvdb"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-01-07T04:53:20Z", "updated_at": "2020-01-31T00:21:27Z", "closed_at": "2020-01-31T00:21:27Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/75", "body": "Also exclude 'tests' from runtime installation.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/75/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 546073980, "node_id": "MDU6SXNzdWU1NDYwNzM5ODA=", "number": 74, "title": "Test failures on openSUSE 15.1: AssertionError: Explicit other_table and other_column", "user": {"value": 15092, "label": "jayvdb"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-01-07T04:35:50Z", "updated_at": "2020-01-12T07:21:17Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "openSUSE 15.1 is using python 3.6.5 and click-7.0 , however it has test failures while openSUSE Tumbleweed on py37 passes.\r\n\r\nMost fail on the cli exit code like\r\n```py\r\n[ 74s] =================================== FAILURES ===================================\r\n[ 74s] _________________________________ test_tables __________________________________\r\n[ 74s] \r\n[ 74s] db_path = '/tmp/pytest-of-abuild/pytest-0/test_tables0/test.db'\r\n[ 74s] \r\n[ 74s] def test_tables(db_path):\r\n[ 74s] result = CliRunner().invoke(cli.cli, [\"tables\", db_path])\r\n[ 74s] > assert '[{\"table\": \"Gosh\"},\\n {\"table\": \"Gosh2\"}]' == result.output.strip()\r\n[ 74s] E assert '[{\"table\": \"...e\": \"Gosh2\"}]' == ''\r\n[ 74s] E - [{\"table\": \"Gosh\"},\r\n[ 74s] E - {\"table\": \"Gosh2\"}]\r\n[ 74s] \r\n[ 74s] tests/test_cli.py:28: AssertionError\r\n```\r\n\r\npackaging project at https://build.opensuse.org/package/show/home:jayvdb:py-new/python-sqlite-utils\r\n\r\nI'll keep digging into this after I have github-to-sqlite working on Tumbleweed, as I'll need openSUSE Leap 15.1 working before I can submit this into the main python repo.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/74/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 521868864, "node_id": "MDU6SXNzdWU1MjE4Njg4NjQ=", "number": 66, "title": "The \".upsert()\" method is misnamed", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 15, "created_at": "2019-11-12T23:48:28Z", "updated_at": "2019-12-31T01:30:21Z", "closed_at": "2019-12-31T01:30:20Z", "author_association": "OWNER", "pull_request": null, "body": "This thread here is illuminating: https://stackoverflow.com/questions/3634984/insert-if-not-exists-else-update\r\n\r\nThe term `UPSERT` in SQLite has a specific meaning as-of 3.24.0 (2018-06-04): https://www.sqlite.org/lang_UPSERT.html\r\n\r\nIt means \"behave as an UPDATE or a no-op if the INSERT would violate a uniqueness constraint\". The syntax in 3.24.0+ looks like this (confusingly it does not use the term \"upsert\"):\r\n```sql\r\nINSERT INTO phonebook(name,phonenumber) VALUES('Alice','704-555-1212')\r\n ON CONFLICT(name) DO UPDATE SET phonenumber=excluded.phonenumber\r\n```\r\nHere's the problem: the `sqlite-utils` `.upsert()` and `.upsert_all()` methods don't do this. They use the following SQL:\r\n\r\n```sql\r\nINSERT OR REPLACE INTO [{table}] ({columns}) VALUES {rows};\r\n```\r\n\r\nIf the record already exists, it will be entirely replaced by a new record - as opposed to updating any specified fields but leaving existing fields as they are (the behaviour of \"upsert\" in SQLite itself).", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/66/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 529376481, "node_id": "MDExOlB1bGxSZXF1ZXN0MzQ2MjY0OTI2", "number": 67, "title": "Run tests against 3.5 too", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-11-27T14:20:35Z", "updated_at": "2019-12-31T01:29:44Z", "closed_at": "2019-12-31T01:29:43Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/67", "body": "", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/67/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 543738004, "node_id": "MDExOlB1bGxSZXF1ZXN0MzU3OTkyNTg4", "number": 72, "title": "Fixed implementation of upsert", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-12-30T05:08:05Z", "updated_at": "2019-12-30T05:29:24Z", "closed_at": "2019-12-30T05:29:24Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/72", "body": "Refs #66", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/72/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 542814756, "node_id": "MDU6SXNzdWU1NDI4MTQ3NTY=", "number": 71, "title": "Tests are failing due to missing FTS5", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2019-12-27T09:41:16Z", "updated_at": "2019-12-27T09:49:37Z", "closed_at": "2019-12-27T09:49:37Z", "author_association": "OWNER", "pull_request": null, "body": "https://travis-ci.com/simonw/sqlite-utils/jobs/268436167\r\n\r\nThis is a recent change: 2 months ago they worked fine.\r\n\r\nI'm not sure what changed here. Maybe something to do with https://launchpad.net/~jonathonf/+archive/ubuntu/backports ?", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/71/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 519039316, "node_id": "MDExOlB1bGxSZXF1ZXN0MzM3ODUzMzk0", "number": 65, "title": "Release 1.12.1", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-11-07T04:51:29Z", "updated_at": "2019-11-07T04:58:48Z", "closed_at": "2019-11-07T04:58:47Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/65", "body": "", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/65/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 476413293, "node_id": "MDU6SXNzdWU0NzY0MTMyOTM=", "number": 52, "title": "Throws error if .insert_all() / .upsert_all() called with empty list", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2019-08-03T04:09:00Z", "updated_at": "2019-11-07T04:32:39Z", "closed_at": "2019-11-07T04:32:39Z", "author_association": "OWNER", "pull_request": null, "body": "See also https://github.com/simonw/db-to-sqlite/issues/18", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/52/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 519032008, "node_id": "MDExOlB1bGxSZXF1ZXN0MzM3ODQ3NTcz", "number": 64, "title": "test_insert_upsert_all_empty_list", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-11-07T04:24:45Z", "updated_at": "2019-11-07T04:32:38Z", "closed_at": "2019-11-07T04:32:38Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/64", "body": "", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/64/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 500783373, "node_id": "MDU6SXNzdWU1MDA3ODMzNzM=", "number": 62, "title": "[enhancement] Method to delete a row in python", "user": {"value": 4454869, "label": "Sergeileduc"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2019-10-01T09:45:47Z", "updated_at": "2019-11-04T16:30:34Z", "closed_at": "2019-11-04T16:18:18Z", "author_association": "NONE", "pull_request": null, "body": "Hi !\r\nThanks for the lib !\r\n\r\nObviously, every possible sql queries won't have a dedicated method.\r\n\r\nBut I was thinking : a method to delete a row (I'm terrible with names, maybe `delete_where()` or something, would be useful.\r\n\r\nI have a Database, with primary key.\r\n\r\nFor the moment, I use :\r\n\r\n```Python3\r\ndb.conn.execute(f\"DELETE FROM table WHERE key = {key_id}\")\r\ndb.conn.commit()\r\n```\r\nto delete a row I don't need anymore, giving his primary key.\r\n\r\nWorks like a charm.\r\n\r\nJust an idea :\r\n\r\n```Python3\r\ntable.delete_where_pkey({'key': key_id})\r\n```\r\nor something (I know, I'm terrible at naming methods...).\r\n\r\nPros : well, no need to write SQL query.\r\n\r\nCons : WHERE normally allows to do many more things (operators =, <>, >, <, BETWEEN), not to mention AND, OR, etc...\r\nMethod is maybe to specific, and/or a pain to render more flexible.\r\n\r\nAgain, just a thought. Writing his own sql works too, so...\r\n\r\nThanks again.\r\nSee yah.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/62/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 491219910, "node_id": "MDU6SXNzdWU0OTEyMTk5MTA=", "number": 61, "title": "importing CSV to SQLite as library", "user": {"value": 17739, "label": "witeshadow"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-09-09T17:12:40Z", "updated_at": "2019-11-04T16:25:01Z", "closed_at": "2019-11-04T16:25:01Z", "author_association": "NONE", "pull_request": null, "body": "CSV can be imported to SQLite when used CLI, but I don't see documentation for when using as library. ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/61/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 517241040, "node_id": "MDU6SXNzdWU1MTcyNDEwNDA=", "number": 63, "title": "ensure_index() method", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2019-11-04T15:51:22Z", "updated_at": "2019-11-04T16:20:36Z", "closed_at": "2019-11-04T16:20:35Z", "author_association": "OWNER", "pull_request": null, "body": "```python\r\ndb[\"table\"].ensure_index([\"col1\", \"col2\"])\r\n```\r\n\r\nThis will do the following:\r\n- if the specified table or column does not exist, do nothing\r\n- if they exist and already have an index, do nothing\r\n- otherwise, create the index\r\n\r\nI want this for tools like [twitter-to-sqlite search](https://github.com/dogsheep/twitter-to-sqlite/blob/801c0c2daf17d8abce9dcb5d8d610410e7e25dbe/README.md#running-searches) where the `search_runs` table may or not have been created yet but, if it IS created, I want to put an index on the `hash` column.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/63/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 488338965, "node_id": "MDU6SXNzdWU0ODgzMzg5NjU=", "number": 59, "title": "Ability to introspect triggers", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-09-02T23:47:16Z", "updated_at": "2019-09-03T01:52:36Z", "closed_at": "2019-09-03T00:09:42Z", "author_association": "OWNER", "pull_request": null, "body": "Now that we're creating triggers (thanks to @amjith in #57) it would be neat if we could introspect them too.\r\n\r\nI'm thinking:\r\n\r\n`db.triggers` - lists all triggers for the database\r\n`db[\"tablename\"].triggers` - lists triggers for that table\r\n\r\nThe underlying query for this is `select * from sqlite_master where type = 'trigger'`\r\n\r\nI'll return the trigger information in a new namedtuple, similar to how Indexes and ForeignKeys work.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/59/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 487987958, "node_id": "MDExOlB1bGxSZXF1ZXN0MzEzMTA1NjM0", "number": 57, "title": "Add triggers while enabling FTS", "user": {"value": 49260, "label": "amjith"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2019-09-02T04:23:40Z", "updated_at": "2019-09-03T01:03:59Z", "closed_at": "2019-09-02T23:42:29Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/57", "body": "This adds the option for a user to set up triggers in the database to keep their FTS table in sync with the parent table. \r\n\r\nRef: https://sqlite.org/fts5.html#external_content_and_contentless_tables\r\n\r\nI would prefer to make the creation of triggers the default behavior, but that will break existing usage where people have been calling `populate_fts` after inserting new rows.\r\n\r\nI am happy to make changes to the PR as you see fit. ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/57/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 488341021, "node_id": "MDExOlB1bGxSZXF1ZXN0MzEzMzgzMzE3", "number": 60, "title": "db.triggers and table.triggers introspection", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-09-03T00:04:32Z", "updated_at": "2019-09-03T00:09:42Z", "closed_at": "2019-09-03T00:09:42Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/60", "body": "Closes #59", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/60/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 487847945, "node_id": "MDExOlB1bGxSZXF1ZXN0MzEzMDA3NDgz", "number": 56, "title": "Escape the table name in populate_fts and search.", "user": {"value": 49260, "label": "amjith"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-09-01T06:29:05Z", "updated_at": "2019-09-02T17:23:21Z", "closed_at": "2019-09-02T17:23:21Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/56", "body": "The table names weren't escaped using double quotes in the populate_fts method. \r\n\r\nReproducible case: \r\n```\r\n>>> import sqlite_utils\r\n>>> db = sqlite_utils.Database(\"abc.db\")\r\n>>> db[\"http://example.com\"].insert_all([\r\n... {\"id\": 1, \"age\": 4, \"name\": \"Cleo\"},\r\n... {\"id\": 2, \"age\": 2, \"name\": \"Pancakes\"}\r\n... ], pk=\"id\")\r\n
\r\n>>> db[\"http://example.com\"].enable_fts([\"name\"])\r\nTraceback (most recent call last):\r\n File \"\", line 1, in \r\n db[\"http://example.com\"].enable_fts([\"name\"])\r\n File \"/home/amjith/.virtualenvs/itsysearch/lib/python3.7/site-packages/sqlite_utils/db.py\", l\r\nine 705, in enable_fts\r\n self.populate_fts(columns)\r\n File \"/home/amjith/.virtualenvs/itsysearch/lib/python3.7/site-packages/sqlite_utils/db.py\", l\r\nine 715, in populate_fts\r\n self.db.conn.executescript(sql)\r\nsqlite3.OperationalError: unrecognized token: \":\"\r\n>>> \r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/56/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 480961330, "node_id": "MDU6SXNzdWU0ODA5NjEzMzA=", "number": 54, "title": "Ability to list views, and to access db[\"view_name\"].rows / rows_where / etc", "user": {"value": 20264, "label": "ftrain"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2019-08-15T02:00:28Z", "updated_at": "2019-08-23T12:41:09Z", "closed_at": "2019-08-23T12:20:15Z", "author_association": "NONE", "pull_request": null, "body": "The docs show me how to create a view via `db.create_view()` but I can't seem to get back to that view post-creation; if I query it as a table it returns `None`, and it doesn't appear in the table listing, even though querying the view works fine from inside the sqlite3 command-line.\r\n\r\nIt'd be great to have the view as a pseudo-table, or if the python/sqlite3 module makes that hard to pull off (I couldn't figure it out), to have that edge-case documented next to the `db.create_view()` docs.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/54/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 481887482, "node_id": "MDExOlB1bGxSZXF1ZXN0MzA4MjkyNDQ3", "number": 55, "title": "Ability to introspect and run queries against views", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2019-08-17T13:40:56Z", "updated_at": "2019-08-23T12:19:42Z", "closed_at": "2019-08-23T12:19:42Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/55", "body": "See #54 ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/55/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 449565204, "node_id": "MDU6SXNzdWU0NDk1NjUyMDQ=", "number": 23, "title": "Syntactic sugar for creating m2m records", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 10, "created_at": "2019-05-29T02:17:48Z", "updated_at": "2019-08-04T03:54:58Z", "closed_at": "2019-08-04T03:37:34Z", "author_association": "OWNER", "pull_request": null, "body": "Python library only. What would be a syntactically pleasant way of creating a m2m record?", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/23/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 476436920, "node_id": "MDExOlB1bGxSZXF1ZXN0MzAzOTkwNjgz", "number": 53, "title": "Work in progress: m2m() method for creating many-to-many records", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-08-03T10:03:56Z", "updated_at": "2019-08-04T03:38:10Z", "closed_at": "2019-08-04T03:37:33Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/53", "body": "- [x] `table.insert({\"name\": \"Barry\"}).m2m(\"tags\", lookup={\"tag\": \"Coworker\"})`\r\n- [x] Explicit table name `.m2m(\"humans\", ..., m2m_table=\"relationships\")`\r\n- [x] Automatically use an existing m2m table if a single obvious candidate exists (a table with two foreign keys in the correct directions)\r\n- [x] Require the explicit `m2m_table=` argument if multiple candidates for the m2m table exist\r\n- [x] Documentation\r\n\r\nRefs #23", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/53/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 462430920, "node_id": "MDU6SXNzdWU0NjI0MzA5MjA=", "number": 35, "title": "table.update(...) method", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-06-30T18:06:15Z", "updated_at": "2019-07-28T15:43:52Z", "closed_at": "2019-07-28T15:43:52Z", "author_association": "OWNER", "pull_request": null, "body": "Spun off from #23 - this method will allow a user to update a specific row.\r\n\r\nCurrently the only way to do that it is to call `.upsert({full record})` with the primary key field matching an existing record - but this does not support partial updates.\r\n```python\r\ndb[\"events\"].update(3, {\"name\": \"Renamed\"})\r\n```\r\nThis method only works on an existing table, so there's no need for a `pk=\"id\"` specifier - it can detect the primary key by looking at the table.\r\n\r\nIf the primary key is compound the first argument can be a tuple:\r\n```python\r\ndb[\"events_venues\"].update((3, 2), {\"custom_label\": \"Label\"})\r\n```\r\nThe method can be called without the second dictionary argument. Doing this selects the row specified by the primary key (throwing an error if it does not exist) and remembers it so that chained operations can be carried out - see proposal in https://github.com/simonw/sqlite-utils/issues/23#issuecomment-507055345\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/35/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 467862459, "node_id": "MDExOlB1bGxSZXF1ZXN0Mjk3NDEyNDY0", "number": 38, "title": "table.update() method", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-07-14T17:03:49Z", "updated_at": "2019-07-28T15:43:51Z", "closed_at": "2019-07-28T15:43:51Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/38", "body": "Refs #35\r\n\r\nStill to do:\r\n\r\n- [x] Unit tests\r\n- [x] Switch to using `.get()`\r\n- [x] Better exceptions, plus unit tests for what happens if pk does not exist\r\n- [x] Documentation\r\n- [x] Ensure compound primary keys work properly\r\n- [x] `alter=True` support", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/38/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 473733752, "node_id": "MDExOlB1bGxSZXF1ZXN0MzAxODI0MDk3", "number": 51, "title": "Fix for too many SQL variables, closes #50", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2019-07-28T11:30:30Z", "updated_at": "2019-07-28T11:59:32Z", "closed_at": "2019-07-28T11:59:32Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/51", "body": "", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/51/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 471797101, "node_id": "MDExOlB1bGxSZXF1ZXN0MzAwMzc3NTk5", "number": 47, "title": "extracts= table parameter", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-07-23T16:30:29Z", "updated_at": "2019-07-23T17:00:43Z", "closed_at": "2019-07-23T17:00:43Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/47", "body": "Still needs docs. Refs #46", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/47/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 470691999, "node_id": "MDU6SXNzdWU0NzA2OTE5OTk=", "number": 43, "title": ".add_column() doesn't match indentation of initial creation", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2019-07-20T16:33:10Z", "updated_at": "2019-07-23T13:09:11Z", "closed_at": "2019-07-23T13:09:05Z", "author_association": "OWNER", "pull_request": null, "body": "I spotted a table which was created once and then had columns added to it and the formatted SQL looks like this:\r\n\r\n```sql\r\nCREATE TABLE [records] (\r\n [type] TEXT,\r\n [sourceName] TEXT,\r\n [sourceVersion] TEXT,\r\n [unit] TEXT,\r\n [creationDate] TEXT,\r\n [startDate] TEXT,\r\n [endDate] TEXT,\r\n [value] TEXT,\r\n [metadata_Health Mate App Version] TEXT,\r\n [metadata_Withings User Identifier] TEXT,\r\n [metadata_Modified Date] TEXT,\r\n [metadata_Withings Link] TEXT,\r\n [metadata_HKWasUserEntered] TEXT\r\n, [device] TEXT, [metadata_HKMetadataKeyHeartRateMotionContext] TEXT, [metadata_HKDeviceManufacturerName] TEXT, [metadata_HKMetadataKeySyncVersion] TEXT, [metadata_HKMetadataKeySyncIdentifier] TEXT, [metadata_HKSwimmingStrokeStyle] TEXT, [metadata_HKVO2MaxTestType] TEXT, [metadata_HKTimeZone] TEXT, [metadata_Average HR] TEXT, [metadata_Recharge] TEXT, [metadata_Lights] TEXT, [metadata_Asleep] TEXT, [metadata_Rating] TEXT, [metadata_Energy Threshold] TEXT, [metadata_Deep Sleep] TEXT, [metadata_Nap] TEXT, [metadata_Edit Slots] TEXT, [metadata_Tags] TEXT, [metadata_Daytime HR] TEXT)\r\n```\r\n\r\nIt would be nice if the columns that were added later matched the indentation of the initial columns.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/43/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 471628483, "node_id": "MDU6SXNzdWU0NzE2Mjg0ODM=", "number": 44, "title": "Utilities for building lookup tables", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-07-23T10:59:58Z", "updated_at": "2019-07-23T13:07:01Z", "closed_at": "2019-07-23T13:07:01Z", "author_association": "OWNER", "pull_request": null, "body": "While building https://github.com/dogsheep/healthkit-to-sqlite I found a need for a neat mechanism for easily building lookup tables - tables where each unique value in a column is replaced by a foreign key to a separate table.\r\n\r\ncsvs-to-sqlite currently creates those with its \"extract\" mechanism - but that's written as custom code against Pandas. I'd like to eventually replace Pandas with sqlite-utils there.\r\n\r\nSee also #42 ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/44/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 471684708, "node_id": "MDExOlB1bGxSZXF1ZXN0MzAwMjg2NTM1", "number": 45, "title": "Implemented table.lookup(...), closes #44", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-07-23T13:03:30Z", "updated_at": "2019-07-23T13:07:00Z", "closed_at": "2019-07-23T13:07:00Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/45", "body": "", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/45/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 351845423, "node_id": "MDU6SXNzdWUzNTE4NDU0MjM=", "number": 3, "title": "Experiment with contentless FTS tables", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2018-08-18T19:31:01Z", "updated_at": "2019-07-22T20:58:55Z", "closed_at": "2019-07-22T20:58:55Z", "author_association": "OWNER", "pull_request": null, "body": "Could greatly reduce size of resulting database for large datasets: http://cocoamine.net/blog/2015/09/07/contentless-fts4-for-large-immutable-documents/", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/3/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 470131537, "node_id": "MDU6SXNzdWU0NzAxMzE1Mzc=", "number": 41, "title": "sqlite-utils insert --tsv option", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-07-19T04:27:21Z", "updated_at": "2019-07-19T04:50:47Z", "closed_at": "2019-07-19T04:50:47Z", "author_association": "OWNER", "pull_request": null, "body": "Right now we only support ingesting CSV, but sometimes interesting data is released as TSV.\r\n\r\nhttps://www.washingtonpost.com/national/2019/07/18/how-download-use-dea-pain-pills-database/ for example.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/41/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 467928674, "node_id": "MDExOlB1bGxSZXF1ZXN0Mjk3NDU5Nzk3", "number": 40, "title": ".get() method plus support for compound primary keys", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2019-07-15T03:43:13Z", "updated_at": "2019-07-15T04:28:57Z", "closed_at": "2019-07-15T04:28:52Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/40", "body": "- [x] Tests for the `NotFoundError` exception\r\n- [x] Documentation for `.get()` method\r\n- [x] Support `--pk` multiple times to define CLI compound primary keys\r\n- [x] Documentation for compound primary keys", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/40/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 467864071, "node_id": "MDU6SXNzdWU0Njc4NjQwNzE=", "number": 39, "title": "table.get(...) method", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-07-14T17:20:51Z", "updated_at": "2019-07-15T04:28:53Z", "closed_at": "2019-07-15T04:28:53Z", "author_association": "OWNER", "pull_request": null, "body": "Utility method for fetching a record by its primary key.\r\n\r\nAccepts a single value (for primary key / rowid tables) or a list/tuple of values (for compound primary keys, refs #36).\r\n\r\nRaises a `NotFoundError` if the record cannot be found.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/39/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 462817589, "node_id": "MDU6SXNzdWU0NjI4MTc1ODk=", "number": 36, "title": "Support compound primary keys", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-07-01T17:00:07Z", "updated_at": "2019-07-15T04:28:52Z", "closed_at": "2019-07-15T04:28:52Z", "author_association": "OWNER", "pull_request": null, "body": "This should work:\r\n```python\r\ntable = db[\"dog_breeds\"].insert({\r\n \"dog_id\": 1,\r\n \"breed_id\": 2\r\n}, pk=(\"dog_id\", \"breed_id\"))\r\n```\r\nNeeded for m2m work in #23", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/36/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 455996809, "node_id": "MDU6SXNzdWU0NTU5OTY4MDk=", "number": 28, "title": "Rearrange the docs by area, not CLI vs Python", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2019-06-13T23:33:35Z", "updated_at": "2019-07-15T02:37:20Z", "closed_at": "2019-07-15T02:37:20Z", "author_association": "OWNER", "pull_request": null, "body": "The docs for eg inserting data should live on the same page, rather than being split across the API and CLI pages.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/28/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 462423839, "node_id": "MDU6SXNzdWU0NjI0MjM4Mzk=", "number": 33, "title": "index_foreign_keys / index-foreign-keys utilities", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-06-30T16:42:03Z", "updated_at": "2019-06-30T23:54:11Z", "closed_at": "2019-06-30T23:50:55Z", "author_association": "OWNER", "pull_request": null, "body": "Sometimes it's good to have indices on all columns that are foreign keys, to allow for efficient reverse lookups.\r\n\r\nThis would be a useful utility:\r\n\r\n $ sqlite-utils index-foreign-keys database.db\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/33/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 462423972, "node_id": "MDExOlB1bGxSZXF1ZXN0MjkzMTE3MTgz", "number": 34, "title": "sqlite-utils index-foreign-keys / db.index_foreign_keys()", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-06-30T16:43:40Z", "updated_at": "2019-06-30T23:50:55Z", "closed_at": "2019-06-30T23:50:55Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/34", "body": "Refs #33\r\n\r\n- [x] `sqlite-utils index-foreign-keys` command\r\n- [x] `db.index_foreign_keys()` method\r\n- [x] unit tests\r\n- [x] documentation", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/34/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 461237618, "node_id": "MDU6SXNzdWU0NjEyMzc2MTg=", "number": 31, "title": "Mechanism for adding multiple foreign key constraints at once", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-06-27T00:04:30Z", "updated_at": "2019-06-29T06:27:40Z", "closed_at": "2019-06-29T06:27:40Z", "author_association": "OWNER", "pull_request": null, "body": "Needed by [db-to-sqlite](https://github.com/simonw/db-to-sqlite). It currently works by collecting all of the foreign key relationships it can find and then applying them at the end of the process.\r\n\r\nThe problem is, the `add_foreign_key()` method looks like this:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/86bd2bba689e25f09551d611ccfbee1e069e5b66/sqlite_utils/db.py#L498-L516\r\n\r\nThat means it's doing a full `VACUUM` for every single relationship it sets up - and if you have hundreds of foreign key relationships in your database this can take hours.\r\n\r\nI think the right solution is to have a `.add_foreign_keys(list_of_args)` method which does the bulk operation and then a single `VACUUM`. `.add_foreign_key(...)` can then call the bulk action with a single list item.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/31/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}