{"html_url": "https://github.com/simonw/sqlite-utils/issues/99#issuecomment-612727400", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/99", "id": 612727400, "node_id": "MDEyOklzc3VlQ29tbWVudDYxMjcyNzQwMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-13T03:03:09Z", "updated_at": "2020-04-13T03:03:09Z", "author_association": "OWNER", "body": "I think I'm going to leave this as intended behaviour. Or maybe passing multiple dictionaries to `.upsert_all()` with different numbers of keys should raise an error?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 598640234, "label": ".upsert_all() should maybe error if dictionaries passed to it do not have the same keys"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/99#issuecomment-612727814", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/99", "id": 612727814, "node_id": "MDEyOklzc3VlQ29tbWVudDYxMjcyNzgxNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-13T03:05:04Z", "updated_at": "2020-04-13T03:05:04Z", "author_association": "OWNER", "body": "Bit trick from an implementation point of view this, since we want to be able to handle input that is a generator - so we can't scan through the input to validate that every dictionary has the same exact keys without consuming the entire iterator.\r\n\r\nThe alternative would be to raise an error the first time we spot a dictionary with keys that differ... but that's weird because we commit changes in batches, so we may end up only applying half of the changes before exiting with the error.\r\n\r\nOn that basis, I'm going to leave this as-is and mark this as wontfix.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 598640234, "label": ".upsert_all() should maybe error if dictionaries passed to it do not have the same keys"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612173156", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/98", "id": 612173156, "node_id": "MDEyOklzc3VlQ29tbWVudDYxMjE3MzE1Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-10T19:03:32Z", "updated_at": "2020-04-10T23:08:28Z", "author_association": "OWNER", "body": "Investigate this traceback:\r\n```\r\nTraceback (most recent call last):\r\n File \"fetch_projects.py\", line 60, in \r\n fetch_projects(db, token)\r\n File \"fetch_projects.py\", line 41, in fetch_projects\r\n db[\"projects\"].upsert(project, pk=\"id\")\r\n File \"/Users/simonw/.local/share/virtualenvs/big-local-datasette-2jT6nJCT/lib/python3.7/site-packages/sqlite_utils/db.py\", line 1139, in upsert\r\n conversions=conversions,\r\n File \"/Users/simonw/.local/share/virtualenvs/big-local-datasette-2jT6nJCT/lib/python3.7/site-packages/sqlite_utils/db.py\", line 1168, in upsert_all\r\n upsert=True,\r\n File \"/Users/simonw/.local/share/virtualenvs/big-local-datasette-2jT6nJCT/lib/python3.7/site-packages/sqlite_utils/db.py\", line 1107, in insert_all\r\n row = list(self.rows_where(\"rowid = ?\", [self.last_rowid]))[0]\r\nIndexError: list index out of range\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 597671518, "label": "Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612258687", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/98", "id": 612258687, "node_id": "MDEyOklzc3VlQ29tbWVudDYxMjI1ODY4Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-10T23:08:48Z", "updated_at": "2020-04-10T23:08:48Z", "author_association": "OWNER", "body": "I need a test that reproduces this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 597671518, "label": "Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612707293", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/98", "id": 612707293, "node_id": "MDEyOklzc3VlQ29tbWVudDYxMjcwNzI5Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-13T01:21:22Z", "updated_at": "2020-04-13T01:21:22Z", "author_association": "OWNER", "body": "I have a hunch that the root of the problem here is that accessing `result.lastrowid` during my version of an `.upsert()` doesn't actually make sense:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/6161ebf4de44411b3f33feeacaf4501e803d1116/sqlite_utils/db.py#L1102-L1113\r\n\r\nIn the bug I'm seeing (which I still haven't reduced to a reproducible test) the debugger shows me this at that point:\r\n\r\n```\r\n(Pdb) query\r\n'UPDATE [files] SET [createdAt] = ?, [ext] = ?, [updatedAt] = ?, [uri] = ?, [uriType] = ? WHERE [project] = ? AND [name] = ?'\r\n(Pdb) params\r\n['2020-03-04T04:04:40.152000+00:00', 'csv', '2020-03-04T04:04:40.152000+00:00', 'https://storage.googleapis.com/bln_prod/...', 'download', 'UHJvamVjdDo4MTgyMjU2Ny01ZjI0LTQxM2ItYWZmNi05NTlmNGY3MjExMjI=', 'loans_to_documentation.csv']\r\n(Pdb) result.lastrowid\r\n100\r\n```\r\nBut here's the weird thing... there's no row in the table with a rowid of 100!\r\n```\r\n(Pdb) [r['rowid'] for r in self.db.execute_returning_dicts('select rowid, * from files')]\r\n[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27,\r\n28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50,\r\n51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73,\r\n74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99]\r\n```\r\nSo what the heck is going on?\r\n\r\nThe last SQL statement I executed here was an `UPDATE`. The `lastrowid` docs say: https://kite.com/python/docs/sqlite3.Cursor.lastrowid\r\n\r\n> This read-only attribute provides the rowid of the last modified row. It is only set if you issued a INSERT statement using the execute() method. For operations other than INSERT or when executemany() is called, lastrowid is set to None.\r\n\r\nSo where did that `100` come from? It should be `None`!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 597671518, "label": "Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612707828", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/98", "id": 612707828, "node_id": "MDEyOklzc3VlQ29tbWVudDYxMjcwNzgyOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-13T01:24:05Z", "updated_at": "2020-04-13T01:24:16Z", "author_association": "OWNER", "body": "Why do I even care about `lastrowid` here?\r\n\r\nI'm trying to ensure that after you insert or upsert a row you can use `table.last_pk` to start doing things like building additional foreign key relationships.\r\n\r\nSo maybe it doesn't make sense to make `.last_pk` available _at all_ for cases where you called `.upsert_all()` or `.insert_all()` - it should just be populated for `.upsert()` and `.insert()`.\r\n\r\nThe documentation doesn't say it should work for `.upsert_all()` - it's only documented for the single actions.\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/6161ebf4de44411b3f33feeacaf4501e803d1116/sqlite_utils/db.py#L1113-L1124", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 597671518, "label": "Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612708274", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/98", "id": 612708274, "node_id": "MDEyOklzc3VlQ29tbWVudDYxMjcwODI3NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-13T01:25:59Z", "updated_at": "2020-04-13T01:26:11Z", "author_association": "OWNER", "body": "In mucking around with `sqlite3` it looks like `result.lastrowid` is indeed populated for `UPDATE` - in this case with the last inserted rowid in the table. This differs from the documented behaviour I linked to above.\r\n\r\n```\r\nIn [1]: import sqlite3 \r\n\r\nIn [2]: c = sqlite3.connect(\":memory:\") \r\n\r\nIn [3]: c \r\nOut[3]: \r\n\r\nIn [4]: c.execute('create table foo (bar integer);') \r\nOut[4]: \r\n\r\nIn [5]: c.execute('insert into foo (bar) values (1)') \r\nOut[5]: \r\n\r\nIn [6]: c.execute('select * from foo').fetchall() \r\nOut[6]: [(1,)]\r\n\r\nIn [7]: c.execute('insert into foo (bar) values (1)') \r\nOut[7]: \r\n\r\nIn [8]: c.execute('select * from foo').fetchall() \r\nOut[8]: [(1,), (1,)]\r\n\r\nIn [9]: c.execute('insert into foo (bar) values (1)').lastrowid \r\nOut[9]: 3\r\n\r\nIn [10]: c.execute('select * from foo').fetchall() \r\nOut[10]: [(1,), (1,), (1,)]\r\n\r\nIn [11]: c.execute('select rowid, bar from foo').fetchall() \r\nOut[11]: [(1, 1), (2, 1), (3, 1)]\r\n\r\nIn [12]: c.execute('insert into foo (bar) values (1)').lastrowid \r\nOut[12]: 4\r\n\r\nIn [13]: c.execute('select rowid, bar from foo').fetchall() \r\nOut[13]: [(1, 1), (2, 1), (3, 1), (4, 1)]\r\n\r\nIn [14]: r = c.execute('update foo set bar =2 where rowid = 1') \r\n\r\nIn [15]: r.lastrowid \r\nOut[15]: 4\r\n\r\nIn [16]: c.execute('select rowid, bar from foo').fetchall() \r\nOut[16]: [(1, 2), (2, 1), (3, 1), (4, 1)]\r\n\r\nIn [17]: r = c.execute('select rowid, bar from foo') \r\n\r\nIn [18]: r.fetchall() \r\nOut[18]: [(1, 2), (2, 1), (3, 1), (4, 1)]\r\n\r\nIn [19]: r.lastrowid \r\nOut[19]: 4\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 597671518, "label": "Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612728047", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/98", "id": 612728047, "node_id": "MDEyOklzc3VlQ29tbWVudDYxMjcyODA0Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-13T03:06:10Z", "updated_at": "2020-04-13T03:06:10Z", "author_association": "OWNER", "body": "Implementation plan: `.insert_all()` and `.upsert_all()` should only set `.last_rowid` and `last_pk` if they were called with a single item.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 597671518, "label": "Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/98#issuecomment-928790381", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/98", "id": 928790381, "node_id": "IC_kwDOCGYnMM43XDdt", "user": {"value": 36834097, "label": "patricktrainer"}, "created_at": "2021-09-28T04:38:44Z", "updated_at": "2021-09-28T04:38:44Z", "author_association": "NONE", "body": "Hi @simonw - wondering if you might be able to shed some light here. I've seemed to reproduce this issue. \r\nHere's the stacktrace: \r\n\r\n```\r\n... db[\"potholes\"].insert(pothole, pk='id', alter=True, replace=True)\r\n... \r\n\r\nTraceback (most recent call last):\r\n File \"\", line 3, in \r\n File \"/Users/patricktrainer/.pyenv/versions/3.9.0/lib/python3.9/site-packages/sqlite_utils/db.py\", line 2481, in insert\r\n return self.insert_all(\r\n File \"/Users/patricktrainer/.pyenv/versions/3.9.0/lib/python3.9/site-packages/sqlite_utils/db.py\", line 2596, in insert_all\r\n self.insert_chunk(\r\n File \"/Users/patricktrainer/.pyenv/versions/3.9.0/lib/python3.9/site-packages/sqlite_utils/db.py\", line 2424, in insert_chunk\r\n row = list(self.rows_where(\"rowid = ?\", [self.last_rowid]))[0]\r\nIndexError: list index out of range\r\n```\r\n\r\nInteresting enough, I found that omitting the `pk` param does not throw the error.\r\n\r\nLet me know how I can help out! \r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 597671518, "label": "Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/97#issuecomment-612732129", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/97", "id": 612732129, "node_id": "MDEyOklzc3VlQ29tbWVudDYxMjczMjEyOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-13T03:25:29Z", "updated_at": "2020-04-13T03:25:29Z", "author_association": "OWNER", "body": "Interesting thought. I've run into this myself a lot - many of my scripts intend to create the database from scratch, so I end up running `!rm /tmp/blah.db` in Jupyter and occasionally getting errors if the file doesn't exist.\r\n\r\nI think adding `recreate=True` could make sense. It could throw an error if you attempt to use it after passing in something other than a path to a file on disk.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 593751293, "label": "Adding a \"recreate\" flag to the `Database` constructor"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/97#issuecomment-612732453", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/97", "id": 612732453, "node_id": "MDEyOklzc3VlQ29tbWVudDYxMjczMjQ1Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-13T03:26:46Z", "updated_at": "2020-04-13T03:26:46Z", "author_association": "OWNER", "body": "I wonder if it should delete an recreate the file or if it would be safer to drop every table instead?\r\n\r\nDropping tables gets messy: then you need to drop triggers and views, and you need to run `vacuum` to clean up the space.\r\n\r\nMy worry with deleting and recreating the file is that it could trigger errors in other processes that are currently attached to that database file. But... if you know that's going to be likely, maybe you shouldn't use the `recreate=True` feature?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 593751293, "label": "Adding a \"recreate\" flag to the `Database` constructor"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/97#issuecomment-612738311", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/97", "id": 612738311, "node_id": "MDEyOklzc3VlQ29tbWVudDYxMjczODMxMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-13T03:55:11Z", "updated_at": "2020-04-13T03:55:11Z", "author_association": "OWNER", "body": "Shipped in 2.5 - documentation is here: https://sqlite-utils.readthedocs.io/en/stable/python-api.html#connecting-to-or-creating-a-database", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 593751293, "label": "Adding a \"recreate\" flag to the `Database` constructor"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/97#issuecomment-614073859", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/97", "id": 614073859, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNDA3Mzg1OQ==", "user": {"value": 1448859, "label": "betatim"}, "created_at": "2020-04-15T14:29:30Z", "updated_at": "2020-04-15T14:29:30Z", "author_association": "NONE", "body": "Woah! Thanks a lot. Next time I will add a more obvious/explicit \"if you like this idea let me know I'd love to work on it to get my feet wet here\" :D", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 593751293, "label": "Adding a \"recreate\" flag to the `Database` constructor"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/96#issuecomment-606394349", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/96", "id": 606394349, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjM5NDM0OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-31T04:37:16Z", "updated_at": "2020-03-31T04:37:16Z", "author_association": "OWNER", "body": "Test failure was just a Black formatting issue.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 589801352, "label": "Add type conversion for Panda's Timestamp"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/96#issuecomment-606394619", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/96", "id": 606394619, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjM5NDYxOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-31T04:38:17Z", "updated_at": "2020-03-31T04:40:23Z", "author_association": "OWNER", "body": "I wonder if there are any other Pandas conversions we should be doing? https://pandas.pydata.org/pandas-docs/stable/getting_started/basics.html#dtypes ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 589801352, "label": "Add type conversion for Panda's Timestamp"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599125455", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/92", "id": 599125455, "node_id": "MDEyOklzc3VlQ29tbWVudDU5OTEyNTQ1NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-14T19:34:35Z", "updated_at": "2020-03-14T19:34:35Z", "author_association": "OWNER", "body": "From https://www.sqlite.org/datatype3.html it looks like `FLOAT` is a supported keyword for creating tables but `REAL` is the correct keyword.\r\n\r\nSo actually `sqlite-utils` gets this wrong, because when we create a table we turn Python `float` values into a `FLOAT` column. Looks like the correct behaviour would be to turn them into a `REAL` column.\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/43f1c6ab4e3a6b76531fb6f5447adb83d26f3971/sqlite_utils/db.py#L28-L48", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 581339961, "label": ".columns_dict doesn't work for all possible column types"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599125557", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/92", "id": 599125557, "node_id": "MDEyOklzc3VlQ29tbWVudDU5OTEyNTU1Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-14T19:35:29Z", "updated_at": "2020-03-14T19:35:29Z", "author_association": "OWNER", "body": "Fixing that would technically constitute a breaking change for library consumers, so it should be a major version release.\r\n\r\nI'm not inclined to release `3.0` just for this one issue, so I'm going to hold back on fixing that and address the smaller issue in this bug as a dot release instead for the moment.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 581339961, "label": ".columns_dict doesn't work for all possible column types"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599126831", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/92", "id": 599126831, "node_id": "MDEyOklzc3VlQ29tbWVudDU5OTEyNjgzMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-14T19:45:28Z", "updated_at": "2020-03-14T19:45:28Z", "author_association": "OWNER", "body": "Turns out there are a TON of valid column definitions that aren't being considered yet - https://www.sqlite.org/datatype3.html#affinity_name_examples - stuff like `VARYING CHARACTER(255)` and `DECIMAL(10,5)`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 581339961, "label": ".columns_dict doesn't work for all possible column types"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599127197", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/92", "id": 599127197, "node_id": "MDEyOklzc3VlQ29tbWVudDU5OTEyNzE5Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-14T19:48:06Z", "updated_at": "2020-03-14T19:48:06Z", "author_association": "OWNER", "body": "Actually it looks like I should implement the exact rules described in https://www.sqlite.org/datatype3.html#determination_of_column_affinity\r\n\r\n> The affinity of a column is determined by the declared type of the column, according to the following rules in the order shown:\r\n> \r\n> 1. If the declared type contains the string \"INT\" then it is assigned INTEGER affinity.\r\n> 2. If the declared type of the column contains any of the strings \"CHAR\", \"CLOB\", or \"TEXT\" then that column has TEXT affinity. Notice that the type VARCHAR contains the string \"CHAR\" and is thus assigned TEXT affinity.\r\n> 3. If the declared type for a column contains the string \"BLOB\" or if no type is specified then the column has affinity BLOB.\r\n> 4. If the declared type for a column contains any of the strings \"REAL\", \"FLOA\", or \"DOUB\" then the column has REAL affinity.\r\n> 5. Otherwise, the affinity is NUMERIC.\r\n> \r\n> Note that the order of the rules for determining column affinity is important. A column whose declared type is \"CHARINT\" will match both rules 1 and 2 but the first rule takes precedence and so the column affinity will be INTEGER.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 581339961, "label": ".columns_dict doesn't work for all possible column types"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599127453", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/92", "id": 599127453, "node_id": "MDEyOklzc3VlQ29tbWVudDU5OTEyNzQ1Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-14T19:50:08Z", "updated_at": "2020-03-14T19:50:08Z", "author_association": "OWNER", "body": "> If the declared type for a column contains the string \"BLOB\" or if no type is specified then the column has affinity BLOB\r\n\r\nI currently treat those as `str` - it sounds like I should treat them as `bytes`:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/43f1c6ab4e3a6b76531fb6f5447adb83d26f3971/sqlite_utils/db.py#L68-L69\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 581339961, "label": ".columns_dict doesn't work for all possible column types"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599128891", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/92", "id": 599128891, "node_id": "MDEyOklzc3VlQ29tbWVudDU5OTEyODg5MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-14T20:03:45Z", "updated_at": "2020-03-14T20:03:45Z", "author_association": "OWNER", "body": "I'm going to keep treating them as `str`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 581339961, "label": ".columns_dict doesn't work for all possible column types"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599247833", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/92", "id": 599247833, "node_id": "MDEyOklzc3VlQ29tbWVudDU5OTI0NzgzMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-15T18:37:28Z", "updated_at": "2020-03-15T18:37:43Z", "author_association": "OWNER", "body": "Released in 2.4.2.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 581339961, "label": ".columns_dict doesn't work for all possible column types"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/91#issuecomment-723350956", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/91", "id": 723350956, "node_id": "MDEyOklzc3VlQ29tbWVudDcyMzM1MDk1Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-11-06T23:53:25Z", "updated_at": "2020-11-06T23:53:25Z", "author_association": "OWNER", "body": "This is now possible, for both FTS4 and FTS5 - see #197.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 577302229, "label": "Enable ordering FTS results by rank"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/89#issuecomment-593122605", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/89", "id": 593122605, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MzEyMjYwNQ==", "user": {"value": 35075, "label": "chrishas35"}, "created_at": "2020-03-01T17:33:11Z", "updated_at": "2020-03-01T17:33:11Z", "author_association": "NONE", "body": "If you're happy with the proposed implementation, I have code & tests written that I'll get ready for a PR.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 573578548, "label": "Ability to customize columns used by extracts= feature"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/89#issuecomment-615515867", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/89", "id": 615515867, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTUxNTg2Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T00:00:41Z", "updated_at": "2020-04-18T00:00:41Z", "author_association": "OWNER", "body": "Yes pleas, I'd love to see that pull request!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 573578548, "label": "Ability to customize columns used by extracts= feature"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/89#issuecomment-710460242", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/89", "id": 710460242, "node_id": "MDEyOklzc3VlQ29tbWVudDcxMDQ2MDI0Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-16T19:17:27Z", "updated_at": "2020-10-16T19:17:50Z", "author_association": "OWNER", "body": "I came up with potential syntax for that here: https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710393550 - based on how `table.extract(...)` works:\r\n```python\r\nfresh_db.table(\"tree\", extracts=[Extract(\r\n columns=(\"CommonName\", \"LatinName\"),\r\n table=\"Species\",\r\n fk_column=\"species_id\",\r\n rename={\"CommonName\": \"name\", \"LatinName\": \"latin\"}\r\n)])\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 573578548, "label": "Ability to customize columns used by extracts= feature"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/88#issuecomment-591768604", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/88", "id": 591768604, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MTc2ODYwNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-27T04:03:03Z", "updated_at": "2020-02-27T04:03:03Z", "author_association": "OWNER", "body": "`drop table resources_fts` drops the FTS table and the other ones that it created (resources_fts_data, resources_fts_idx, resources_fts_docsize, resources_fts_config) - but keeps the triggers.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 571805300, "label": "table.disable_fts() method and \"sqlite-utils disable-fts ...\" command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/88#issuecomment-591769046", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/88", "id": 591769046, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MTc2OTA0Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-27T04:05:15Z", "updated_at": "2020-02-27T04:05:15Z", "author_association": "OWNER", "body": "I can reliably get the list of triggers to delete from `select name from sqlite_master where type = 'trigger' and tbl_name = 'resources';`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 571805300, "label": "table.disable_fts() method and \"sqlite-utils disable-fts ...\" command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/88#issuecomment-591769171", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/88", "id": 591769171, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MTc2OTE3MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-27T04:05:58Z", "updated_at": "2020-02-27T04:26:31Z", "author_association": "OWNER", "body": "Strange - https://www.sqlite.org/lang_droptrigger.html says \"Note that triggers are automatically dropped when the associated table is dropped\" but that doesn't seem to be true in my experimenting.\r\n\r\nUPDATE: no that makes sense - the triggers are on `resources` which still exists, it was `resources_fts` that was dropped.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 571805300, "label": "table.disable_fts() method and \"sqlite-utils disable-fts ...\" command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/88#issuecomment-591769373", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/88", "id": 591769373, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MTc2OTM3Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-27T04:06:47Z", "updated_at": "2020-02-27T04:06:47Z", "author_association": "OWNER", "body": "Looks like safest option is to loop through those trigger names and run `DROP TRIGGER IF EXISTS foo` on each one.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 571805300, "label": "table.disable_fts() method and \"sqlite-utils disable-fts ...\" command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/88#issuecomment-591769759", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/88", "id": 591769759, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MTc2OTc1OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-27T04:08:29Z", "updated_at": "2020-02-27T04:08:29Z", "author_association": "OWNER", "body": "I think the method should be called `table.disable_fts()` - the opposite of `table.enable_fts(...)`.\r\n\r\nThere should be a `sqlite-utils disable-fts database.db tablename` command to match it.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 571805300, "label": "table.disable_fts() method and \"sqlite-utils disable-fts ...\" command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/87#issuecomment-586661250", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/87", "id": 586661250, "node_id": "MDEyOklzc3VlQ29tbWVudDU4NjY2MTI1MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-16T02:19:33Z", "updated_at": "2020-02-16T02:19:33Z", "author_association": "OWNER", "body": "Here's the code: \r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/5e0000609f9be6efafea1b96f610988eb18d6d89/sqlite_utils/utils.py#L18-L24", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 565837965, "label": "Should detect collections.OrderedDict as a regular dictionary"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/87#issuecomment-586661276", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/87", "id": 586661276, "node_id": "MDEyOklzc3VlQ29tbWVudDU4NjY2MTI3Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-16T02:20:14Z", "updated_at": "2020-02-16T02:20:14Z", "author_association": "OWNER", "body": "`OrderedDict` is actually a subclass of `dict` - so a smart fix would be for this logic to check and see if the type `t` is a subclass of one of `list`, `tuple` or `dict`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 565837965, "label": "Should detect collections.OrderedDict as a regular dictionary"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586661934", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/86", "id": 586661934, "node_id": "MDEyOklzc3VlQ29tbWVudDU4NjY2MTkzNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-16T02:33:07Z", "updated_at": "2020-02-16T02:33:07Z", "author_association": "OWNER", "body": "Thanks for the example file - looks like it can be trimmed down to just these two lines to replicate the bug:\r\n```csv\r\n\"MTU (CET)\",\"Day-ahead Price [EUR/MWh]\"\r\n\"01.01.2016 00:00 - 01.01.2016 01:00\",\"23.86\"\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 564579430, "label": "Problem with square bracket in CSV column name"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586662404", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/86", "id": 586662404, "node_id": "MDEyOklzc3VlQ29tbWVudDU4NjY2MjQwNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-16T02:43:12Z", "updated_at": "2020-02-16T02:43:12Z", "author_association": "OWNER", "body": "https://stackoverflow.com/a/22694438 looks like the answer:\r\n> When using square brackets, it is not possible to have these characters in the identifier.\r\n> \r\n> When using double quotes, you can escape them in the name by doubling them:\r\n> \r\n> `CREATE TABLE \"hello \"\"world\"\"\"(key INTEGER PRIMARY KEY);`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 564579430, "label": "Problem with square bracket in CSV column name"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586676640", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/86", "id": 586676640, "node_id": "MDEyOklzc3VlQ29tbWVudDU4NjY3NjY0MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-16T07:16:31Z", "updated_at": "2020-02-16T07:16:31Z", "author_association": "OWNER", "body": "There's something weird about this. I created a test database file like so:\r\n```\r\nsqlite3 /tmp/demo.db < .schema\r\nCREATE TABLE IF NOT EXISTS \"data\" (\r\n \"MTU (CET)\" TEXT,\r\n \"Day-ahead Price [EUR/MWh]\" TEXT\r\n );\r\nsqlite> .headers on\r\nsqlite> select * from data;\r\nMTU (CET)|Day-ahead Price [EUR/MWh]\r\n01.01.2016 00:00 - 01.01.2016 01:00|23.86\r\nsqlite> \r\n```\r\nBUT... if I open the same database in Python, something weird happens:\r\n```\r\nIn [1]: import sqlite3 \r\nIn [2]: conn = sqlite3.connect(\"/tmp/demo.db\") \r\nIn [3]: cursor = conn.cursor() \r\nIn [4]: cursor.execute(\"select * from data\") \r\nOut[4]: \r\nIn [5]: cursor.fetchall() \r\nOut[5]: [('01.01.2016 00:00 - 01.01.2016 01:00', '23.86')]\r\nIn [6]: cursor.description \r\nOut[6]: \r\n(('MTU (CET)', None, None, None, None, None, None),\r\n ('Day-ahead Price', None, None, None, None, None, None))\r\nIn [7]: conn.row_factory = sqlite3.Row \r\nIn [8]: cursor = conn.cursor() \r\nIn [9]: cursor.execute(\"select * from data\") \r\nOut[9]: \r\nIn [10]: row = cursor.fetchall() \r\nIn [12]: row \r\nOut[12]: \r\nIn [15]: row.keys() \r\nOut[15]: ['MTU (CET)', 'Day-ahead Price']\r\n```\r\nNote that in `cursor.description` AND in `row.keys()` above the second column is displayed as `'Day-ahead Price'` - when we would expect it to be displayed as `Day-ahead Price [EUR/MWh]`\r\n\r\nSo.... it looks like there may be a bug in Python's `sqlite3` module where columns with square braces in them have that portion of the name stripped out!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 564579430, "label": "Problem with square bracket in CSV column name"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586676856", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/86", "id": 586676856, "node_id": "MDEyOklzc3VlQ29tbWVudDU4NjY3Njg1Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-16T07:20:34Z", "updated_at": "2020-02-16T07:20:34Z", "author_association": "OWNER", "body": "I'm not sure what to do about this one.\r\n\r\nI can't fix it: this bug in Python's `sqlite3` module means that even if I write a database out with column names that include `[]` I won't be able to read them back again.\r\n\r\nSo... I could do one of the following:\r\n\r\n- Throw an error if a column name includes those characters. That's my preferred option I think.\r\n- Automatically replace `[` in column names with `(` and `]` with `)`\r\n- Do the automatic replacement but show a user-visible warning when I do it\r\n- Throw an error, but give the user an option to run with e.g. `--fix-column-names` which applies that automatic fix.\r\n\r\nSince this is likely to be an incredibly rare edge-case I think I'd rather minimize the amount of code that deals with it, so my preferred option is to just throw that error and stop.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 564579430, "label": "Problem with square bracket in CSV column name"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586683572", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/86", "id": 586683572, "node_id": "MDEyOklzc3VlQ29tbWVudDU4NjY4MzU3Mg==", "user": {"value": 8149512, "label": "foscoj"}, "created_at": "2020-02-16T09:03:54Z", "updated_at": "2020-02-16T09:03:54Z", "author_association": "NONE", "body": "Probably the best option to just throw the error.\r\nIs there any active dev chan where we could post the issue to python sqlite3?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 564579430, "label": "Problem with square bracket in CSV column name"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586729798", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/86", "id": 586729798, "node_id": "MDEyOklzc3VlQ29tbWVudDU4NjcyOTc5OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-16T17:11:02Z", "updated_at": "2020-02-16T17:11:02Z", "author_association": "OWNER", "body": "I filed a bug in the Python issue tracker here: https://bugs.python.org/issue39652", "reactions": "{\"total_count\": 2, \"+1\": 2, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 564579430, "label": "Problem with square bracket in CSV column name"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/86#issuecomment-591770623", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/86", "id": 591770623, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MTc3MDYyMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-27T04:12:39Z", "updated_at": "2020-02-27T04:12:39Z", "author_association": "OWNER", "body": "I pushed a branch with my experiment in it, but I'm going to fix this by throwing an error on `[` or `]` in a column name instead - I won't implement the changes from that branch.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 564579430, "label": "Problem with square bracket in CSV column name"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/85#issuecomment-584426938", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/85", "id": 584426938, "node_id": "MDEyOklzc3VlQ29tbWVudDU4NDQyNjkzOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-11T00:35:09Z", "updated_at": "2020-02-11T00:35:09Z", "author_association": "OWNER", "body": "Here's why:\r\nhttps://github.com/simonw/sqlite-utils/blob/0c2451e0690c5f4e6463a2f339b0a280e30ed806/sqlite_utils/db.py#L627-L636", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 562911863, "label": "Create index doesn't work for columns containing spaces"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/83#issuecomment-583789015", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/83", "id": 583789015, "node_id": "MDEyOklzc3VlQ29tbWVudDU4Mzc4OTAxNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-08T23:58:35Z", "updated_at": "2020-02-08T23:58:35Z", "author_association": "OWNER", "body": "Shipped as 2.3", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 559374410, "label": "Make db[\"table\"].exists a documented API"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/82#issuecomment-581651409", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/82", "id": 581651409, "node_id": "MDEyOklzc3VlQ29tbWVudDU4MTY1MTQwOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-03T22:32:41Z", "updated_at": "2020-02-03T22:32:41Z", "author_association": "OWNER", "body": "This should work - the data should be chunked automatically. It looks like this is a bug.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 559197745, "label": "Tutorial command no longer works"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/82#issuecomment-581652388", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/82", "id": 581652388, "node_id": "MDEyOklzc3VlQ29tbWVudDU4MTY1MjM4OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-03T22:35:44Z", "updated_at": "2020-02-03T22:35:44Z", "author_association": "OWNER", "body": "I can't replicate this problem:\r\n```\r\n/tmp $ sqlite-utils --version\r\nsqlite-utils, version 2.2\r\n/tmp $ curl \"https://data.nasa.gov/resource/y77d-th95.json\" | sqlite-utils insert meteorites.db meteorites - --pk=id\r\n % Total % Received % Xferd Average Speed Time Time Time Current\r\n Dload Upload Total Spent Left Speed\r\n100 240k 0 240k 0 0 185k 0 --:--:-- 0:00:01 --:--:-- 185k\r\n```\r\nCould you run `sqlite-utils --version` and tell me what you get?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 559197745, "label": "Tutorial command no longer works"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/82#issuecomment-591771532", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/82", "id": 591771532, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MTc3MTUzMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-27T04:16:30Z", "updated_at": "2020-02-27T04:16:30Z", "author_association": "OWNER", "body": "Closing as can't reproduce.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 559197745, "label": "Tutorial command no longer works"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/81#issuecomment-581071010", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/81", "id": 581071010, "node_id": "MDEyOklzc3VlQ29tbWVudDU4MTA3MTAxMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-01T21:27:00Z", "updated_at": "2020-02-01T21:27:00Z", "author_association": "OWNER", "body": "Here's the current method:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/f7289174e66ae4d91d57de94bbd9d09fabf7aff4/sqlite_utils/db.py#L823-L845\r\n\r\nIf I make it a utility function instead of a class method I could ensure it is directly importable like so:\r\n\r\n```python\r\nfrom sqlite_utils import detect_column_types\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 558600274, "label": "Remove .detect_column_types() from table, make it a documented API"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/81#issuecomment-581071116", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/81", "id": 581071116, "node_id": "MDEyOklzc3VlQ29tbWVudDU4MTA3MTExNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-01T21:28:35Z", "updated_at": "2020-02-01T21:28:53Z", "author_association": "OWNER", "body": "Should I keep `table.detect_column_types()` working so as not to break existing code?\r\n\r\nIf it was part of the documented API then I wouldn't break that without bumping to 3.x. Since it's undocumented I'm going to make it as a breaking change instead (and bump the `geojson-to-sqlite` dependency version).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 558600274, "label": "Remove .detect_column_types() from table, make it a documented API"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/81#issuecomment-581071235", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/81", "id": 581071235, "node_id": "MDEyOklzc3VlQ29tbWVudDU4MTA3MTIzNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-01T21:30:09Z", "updated_at": "2020-02-01T21:30:09Z", "author_association": "OWNER", "body": "Actually I'll put it in the `utils.py` module.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 558600274, "label": "Remove .detect_column_types() from table, make it a documented API"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/81#issuecomment-581071434", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/81", "id": 581071434, "node_id": "MDEyOklzc3VlQ29tbWVudDU4MTA3MTQzNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-01T21:32:34Z", "updated_at": "2020-02-01T21:32:34Z", "author_association": "OWNER", "body": "While I'm at it I think I'll rename it to `suggest_column_types` - it's not really detecting them since the input is just a list of dictionaries.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 558600274, "label": "Remove .detect_column_types() from table, make it a documented API"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/80#issuecomment-580567505", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/80", "id": 580567505, "node_id": "MDEyOklzc3VlQ29tbWVudDU4MDU2NzUwNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-01-31T03:39:19Z", "updated_at": "2020-01-31T03:39:19Z", "author_association": "OWNER", "body": "Still needs documentation and tests. Also I'm not certain that this should be an argument you can pass to the `.table()` constructor, need to think that over.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 557892819, "label": "on_create mechanism for after table creation"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/80#issuecomment-580567604", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/80", "id": 580567604, "node_id": "MDEyOklzc3VlQ29tbWVudDU4MDU2NzYwNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-01-31T03:39:58Z", "updated_at": "2020-01-31T03:39:58Z", "author_association": "OWNER", "body": "Perhaps this should be called `after_create` instead of `on_create`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 557892819, "label": "on_create mechanism for after table creation"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/80#issuecomment-580567886", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/80", "id": 580567886, "node_id": "MDEyOklzc3VlQ29tbWVudDU4MDU2Nzg4Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-01-31T03:41:31Z", "updated_at": "2020-01-31T03:41:31Z", "author_association": "OWNER", "body": "I think it does make sense to be able to pass it to the `.table()` constructor.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 557892819, "label": "on_create mechanism for after table creation"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/80#issuecomment-580569059", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/80", "id": 580569059, "node_id": "MDEyOklzc3VlQ29tbWVudDU4MDU2OTA1OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-01-31T03:48:41Z", "updated_at": "2020-01-31T03:48:41Z", "author_association": "OWNER", "body": "This may not be the right feature after all, see https://github.com/simonw/geojson-to-sqlite/issues/6#issuecomment-580569002", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 557892819, "label": "on_create mechanism for after table creation"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/80#issuecomment-580584269", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/80", "id": 580584269, "node_id": "MDEyOklzc3VlQ29tbWVudDU4MDU4NDI2OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-01-31T05:08:04Z", "updated_at": "2020-01-31T05:08:04Z", "author_association": "OWNER", "body": "Ditching this since it won't actually solve my problem.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 557892819, "label": "on_create mechanism for after table creation"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/8#issuecomment-464341721", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/8", "id": 464341721, "node_id": "MDEyOklzc3VlQ29tbWVudDQ2NDM0MTcyMQ==", "user": {"value": 82988, "label": "psychemedia"}, "created_at": "2019-02-16T12:08:41Z", "updated_at": "2019-02-16T12:08:41Z", "author_association": "NONE", "body": "We also get an error if a column name contains a `.`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 403922644, "label": "Problems handling column names containing spaces or - "}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/8#issuecomment-466695500", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/8", "id": 466695500, "node_id": "MDEyOklzc3VlQ29tbWVudDQ2NjY5NTUwMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-02-23T21:09:03Z", "updated_at": "2019-02-23T21:09:03Z", "author_association": "OWNER", "body": "Fixed in https://github.com/simonw/sqlite-utils/commit/228d595f7d10994f34e948888093c2cd290267c4\r\n\r\n\"screen\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 403922644, "label": "Problems handling column names containing spaces or - "}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/8#issuecomment-482994231", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/8", "id": 482994231, "node_id": "MDEyOklzc3VlQ29tbWVudDQ4Mjk5NDIzMQ==", "user": {"value": 82988, "label": "psychemedia"}, "created_at": "2019-04-14T15:04:07Z", "updated_at": "2019-04-14T15:29:33Z", "author_association": "NONE", "body": "\r\n\r\nPLEASE IGNORE THE BELOW... I did a package update and rebuilt the kernel I was working in... may just have been an old version of sqlite_utils, seems to be working now. (Too many containers / too many environments!)\r\n\r\n\r\nHas an issue been reintroduced here with FTS? eg I'm getting an error thrown by spaces in column names here:\r\n\r\n```\r\n/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, upsert, batch_size, column_order)\r\n\r\ndef enable_fts(self, columns, fts_version=\"FTS5\"):\r\n--> 329 \"Enables FTS on the specified columns\"\r\n 330 sql = \"\"\"\r\n 331 CREATE VIRTUAL TABLE \"{table}_fts\" USING {fts_version} (\r\n```\r\n\r\nwhen trying an `insert_all`.\r\n\r\nAlso, if a col has a `.` in it, I seem to get:\r\n\r\n```\r\n/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, upsert, batch_size, column_order)\r\n 327 jsonify_if_needed(record.get(key, None)) for key in all_columns\r\n 328 )\r\n--> 329 result = self.db.conn.execute(sql, values)\r\n 330 self.db.conn.commit()\r\n 331 self.last_id = result.lastrowid\r\n\r\nOperationalError: near \".\": syntax error\r\n```\r\n\r\n(Can't post a worked minimal example right now; racing trying to build something against a live timing screen that will stop until next weekend in an hour or two...)\r\n\r\nPS Hmmm I did a test and they seem to work; I must be messing up s/where else...\r\n\r\n```\r\nimport sqlite3\r\nfrom sqlite_utils import Database\r\n\r\ndbname='testingDB_sqlite_utils.db'\r\n\r\n#!rm $dbname\r\nconn = sqlite3.connect(dbname, timeout=10)\r\n\r\n\r\n#Setup database tables\r\nc = conn.cursor()\r\n\r\nsetup='''\r\nCREATE TABLE IF NOT EXISTS \"test1\" (\r\n \"NO\" INTEGER,\r\n \"NAME\" TEXT\r\n);\r\n\r\nCREATE TABLE IF NOT EXISTS \"test2\" (\r\n \"NO\" INTEGER,\r\n `TIME OF DAY` TEXT\r\n);\r\n\r\nCREATE TABLE IF NOT EXISTS \"test3\" (\r\n \"NO\" INTEGER,\r\n `AVG. SPEED (MPH)` FLOAT\r\n);\r\n'''\r\n\r\nc.executescript(setup)\r\n\r\n\r\nDB = Database(conn)\r\n\r\nimport pandas as pd\r\n\r\ndf1 = pd.DataFrame({'NO':[1,2],'NAME':['a','b']})\r\nDB['test1'].insert_all(df1.to_dict(orient='records'))\r\n\r\ndf2 = pd.DataFrame({'NO':[1,2],'TIME OF DAY':['early on','late']})\r\nDB['test2'].insert_all(df2.to_dict(orient='records'))\r\n\r\ndf3 = pd.DataFrame({'NO':[1,2],'AVG. SPEED (MPH)':['123.3','123.4']})\r\nDB['test3'].insert_all(df3.to_dict(orient='records'))\r\n```\r\n\r\nall seem to work ok. I'm still getting errors in my set up though, which is not too different to the text cases?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 403922644, "label": "Problems handling column names containing spaces or - "}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012158895", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/79", "id": 1012158895, "node_id": "IC_kwDOCGYnMM48VFGv", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2022-01-13T13:55:59Z", "updated_at": "2022-01-13T13:55:59Z", "author_association": "CONTRIBUTOR", "body": "Came here to add this. I might pick it up.\r\n\r\nWould also add a utility to create (and update and delete?) a spatial index. It's not much code but I have to look it up every time.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 557842245, "label": "Helper methods for working with SpatiaLite"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012230212", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/79", "id": 1012230212, "node_id": "IC_kwDOCGYnMM48VWhE", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2022-01-13T15:15:13Z", "updated_at": "2022-01-13T15:15:13Z", "author_association": "CONTRIBUTOR", "body": "Some proposals I'd add to sqlite-utils:\r\n\r\nSome version of this, from [geojson-to-sqlite](https://github.com/simonw/geojson-to-sqlite/blob/main/geojson_to_sqlite/utils.py#L124-L130):\r\n\r\n```python\r\ndef init_spatialite(db, lib):\r\n db.conn.enable_load_extension(True)\r\n db.conn.load_extension(lib)\r\n # Initialize SpatiaLite if not yet initialized\r\n if \"spatial_ref_sys\" in db.table_names():\r\n return\r\n db.conn.execute(\"select InitSpatialMetadata(1)\")\r\n```\r\n\r\nAlso a function for creating a spatial index:\r\n\r\n```python\r\ndb.conn.execute(\"select CreateSpatialIndex(?, ?)\", [table, \"geometry\"])\r\n```\r\n\r\nI don't know the nuances of updating a spatial index, or checking if one already exists. This could be a CLI method like:\r\n\r\n```sh\r\nsqlite-utils spatial-index spatial.db table-name column-name\r\n```\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 557842245, "label": "Helper methods for working with SpatiaLite"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012253198", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/79", "id": 1012253198, "node_id": "IC_kwDOCGYnMM48VcIO", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2022-01-13T15:39:14Z", "updated_at": "2022-01-13T15:39:14Z", "author_association": "CONTRIBUTOR", "body": "Other thing: If there get to be enough utils, I think it's worth moving all the spatialite stuff into its own file (`gis.py` or something) just so it's easier to find later.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 557842245, "label": "Helper methods for working with SpatiaLite"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012413729", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/79", "id": 1012413729, "node_id": "IC_kwDOCGYnMM48WDUh", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2022-01-13T18:50:00Z", "updated_at": "2022-01-13T18:50:00Z", "author_association": "CONTRIBUTOR", "body": "One more thing I'm going to add: A method to add a geometry column, which I'll need to do to create a spatial index on a table.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 557842245, "label": "Helper methods for working with SpatiaLite"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1013698557", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/79", "id": 1013698557, "node_id": "IC_kwDOCGYnMM48a8_9", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2022-01-15T15:15:22Z", "updated_at": "2022-01-15T15:15:22Z", "author_association": "CONTRIBUTOR", "body": "@simonw I have a PR here https://github.com/simonw/sqlite-utils/pull/385 that adds Spatialite helpers on the Python side. Please let me know how it looks.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 557842245, "label": "Helper methods for working with SpatiaLite"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1029317527", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/79", "id": 1029317527, "node_id": "IC_kwDOCGYnMM49WiOX", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2022-02-03T19:18:02Z", "updated_at": "2022-02-03T19:18:02Z", "author_association": "CONTRIBUTOR", "body": "Taking part of the conversation from #385 here.\r\n\r\n> Would sqlite-utils add-geometry-column ... be a good CLI enhancement. for example?\r\n\r\nYes. And also `sqlite-utils create-spatial-index` would be great to have. My plan would be to add those once the Python API is settled.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 557842245, "label": "Helper methods for working with SpatiaLite"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1029683977", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/79", "id": 1029683977, "node_id": "IC_kwDOCGYnMM49X7sJ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-04T05:58:15Z", "updated_at": "2022-02-04T05:58:15Z", "author_association": "OWNER", "body": "Documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#spatialite-helpers", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 557842245, "label": "Helper methods for working with SpatiaLite"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1029703503", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/79", "id": 1029703503, "node_id": "IC_kwDOCGYnMM49YAdP", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-04T06:46:32Z", "updated_at": "2022-02-04T06:46:32Z", "author_association": "OWNER", "body": "Shipped in 3.23: https://sqlite-utils.datasette.io/en/stable/changelog.html#v3-23", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 557842245, "label": "Helper methods for working with SpatiaLite"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/77#issuecomment-580515506", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/77", "id": 580515506, "node_id": "MDEyOklzc3VlQ29tbWVudDU4MDUxNTUwNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-01-30T23:48:41Z", "updated_at": "2020-01-30T23:48:41Z", "author_association": "OWNER", "body": "Potential design: a `conversions={}` option. Used like this:\r\n```python\r\ndb[table].insert(record, conversions={\"geom\": \"GeomFromText(?, 4326)\"})\r\n```\r\nThe `conversions=` key would be supported on `.insert()`, `.insert_all()`, `.upsert()` etc. It could also be passed to the `db.table()` constructor function:\r\n```python\r\ntable = db.table(\r\n \"features\",\r\n pk=\"id\",\r\n conversions={\r\n \"geom\": \"GeomFromText(?, 4326)\"\r\n }\r\n)\r\n# Then used like this:\r\ntable.insert(record)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 557825032, "label": "Ability to insert data that is transformed by a SQL function"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/77#issuecomment-580527238", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/77", "id": 580527238, "node_id": "MDEyOklzc3VlQ29tbWVudDU4MDUyNzIzOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-01-31T00:34:02Z", "updated_at": "2020-01-31T00:34:02Z", "author_association": "OWNER", "body": "Documentation: https://sqlite-utils.readthedocs.io/en/stable/python-api.html#python-api-conversions", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 557825032, "label": "Ability to insert data that is transformed by a SQL function"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/76#issuecomment-614354219", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/76", "id": 614354219, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNDM1NDIxOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-16T01:01:34Z", "updated_at": "2020-04-16T01:01:34Z", "author_association": "OWNER", "body": "I think a neat way to do this would be with an optional argument for `.rows_where()`:\r\n\r\n```python\r\nrows = db[\"table\"].rows_where(\"age > 10\", order_by=\"age desc\")\r\n```\r\n\r\nIf you want everything you can use this:\r\n\r\n```python\r\nrows = db[\"table\"].rows_where(order_by=\"age desc\")\r\n```\r\n\r\nIt's a tiny bit weird calling `.rows_where()` without a where clause, but I think it makes sense here - especially since `.rows` is a property that can't take any arguments - though under the hood it actually does this:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/ad6ac19470a67867b96cb4c086450b8e4e46bf02/sqlite_utils/db.py#L436-L443", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 549287310, "label": "order_by mechanism"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/76#issuecomment-614400454", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/76", "id": 614400454, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNDQwMDQ1NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-16T03:51:01Z", "updated_at": "2020-04-16T03:51:01Z", "author_association": "OWNER", "body": "Released in 2.6", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 549287310, "label": "order_by mechanism"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/76#issuecomment-614400533", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/76", "id": 614400533, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNDQwMDUzMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-16T03:51:26Z", "updated_at": "2020-04-16T03:51:26Z", "author_association": "OWNER", "body": "Documentation here: https://sqlite-utils.readthedocs.io/en/stable/python-api.html#listing-rows", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 549287310, "label": "order_by mechanism"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/76#issuecomment-614440032", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/76", "id": 614440032, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNDQ0MDAzMg==", "user": {"value": 10501166, "label": "metab0t"}, "created_at": "2020-04-16T06:23:29Z", "updated_at": "2020-04-16T06:23:29Z", "author_association": "NONE", "body": "Thanks for your hard work!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 549287310, "label": "order_by mechanism"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/75#issuecomment-580523995", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/75", "id": 580523995, "node_id": "MDEyOklzc3VlQ29tbWVudDU4MDUyMzk5NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-01-31T00:21:11Z", "updated_at": "2020-01-31T00:21:11Z", "author_association": "OWNER", "body": "This makes sense, thanks!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 546078359, "label": "Explicitly include tests and docs in sdist"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/74#issuecomment-572871797", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/74", "id": 572871797, "node_id": "MDEyOklzc3VlQ29tbWVudDU3Mjg3MTc5Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-01-10T04:47:55Z", "updated_at": "2020-01-10T04:47:55Z", "author_association": "OWNER", "body": "This is odd. I'd love to see more about that result object.\r\n\r\nCould you try running `pytest --pdb` and then `result.exit_code, result.exception` in the PDB prompt, something like this?\r\n\r\n```\r\n$ pytest --pdb\r\n========================================================= test session starts =========================================================\r\nplatform darwin -- Python 3.7.4, pytest-5.2.2, py-1.8.0, pluggy-0.13.0\r\nrootdir: /Users/simonw/Dropbox/Development/sqlite-utils\r\nplugins: cov-2.8.1\r\ncollected 216 items \r\n\r\ntests/test_black.py s [ 0%]\r\ntests/test_cli.py F\r\n>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>\r\n\r\ndb_path = '/private/var/folders/bl/5x847xbj2yb7xmp7f2tz7l280000gn/T/pytest-of-simonw/pytest-3/test_tables0/test.db'\r\n\r\n def test_tables(db_path):\r\n result = CliRunner().invoke(cli.cli, [\"tables1\", db_path])\r\n> assert '[{\"table\": \"Gosh\"},\\n {\"table\": \"Gosh2\"}]' == result.output.strip()\r\nE assert '[{\"table\": \"...e\": \"Gosh2\"}]' == ''\r\nE - [{\"table\": \"Gosh\"},\r\nE - {\"table\": \"Gosh2\"}]\r\n\r\ntests/test_cli.py:28: AssertionError\r\n>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>\r\n\r\n>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>\r\n> /Users/simonw/Dropbox/Development/sqlite-utils/tests/test_cli.py(28)test_tables()\r\n-> assert '[{\"table\": \"Gosh\"},\\n {\"table\": \"Gosh2\"}]' == result.output.strip()\r\n(Pdb) result.exit_code, result.exception\r\n(1, OperationalError('near \"/\": syntax error'))\r\n```\r\n\r\nThat should show the exception that caused the script to fail to run.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 546073980, "label": "Test failures on openSUSE 15.1: AssertionError: Explicit other_table and other_column"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/74#issuecomment-573388052", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/74", "id": 573388052, "node_id": "MDEyOklzc3VlQ29tbWVudDU3MzM4ODA1Mg==", "user": {"value": 15092, "label": "jayvdb"}, "created_at": "2020-01-12T06:51:30Z", "updated_at": "2020-01-12T06:51:30Z", "author_association": "CONTRIBUTOR", "body": "Thanks. That showed me that there was a click cli runner error, and setting `export LANG=en_US.UTF-8` fixed it.\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 546073980, "label": "Test failures on openSUSE 15.1: AssertionError: Explicit other_table and other_column"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/74#issuecomment-573389669", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/74", "id": 573389669, "node_id": "MDEyOklzc3VlQ29tbWVudDU3MzM4OTY2OQ==", "user": {"value": 15092, "label": "jayvdb"}, "created_at": "2020-01-12T07:21:17Z", "updated_at": "2020-01-12T07:21:17Z", "author_association": "CONTRIBUTOR", "body": "I guess there is some extra flag for ` CliRunner.invoke` to check exitcode and raise the exception, or that should be an extra assert added.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 546073980, "label": "Test failures on openSUSE 15.1: AssertionError: Explicit other_table and other_column"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/73#issuecomment-570930239", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/73", "id": 570930239, "node_id": "MDEyOklzc3VlQ29tbWVudDU3MDkzMDIzOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-01-05T17:15:18Z", "updated_at": "2020-01-05T17:15:18Z", "author_association": "OWNER", "body": "I think this is because you forgot to include a `pk=` argument. I'll change the code to throw a more useful error in this case.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 545407916, "label": "upsert_all() throws issue when upserting to empty table"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/73#issuecomment-570931650", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/73", "id": 570931650, "node_id": "MDEyOklzc3VlQ29tbWVudDU3MDkzMTY1MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-01-05T17:34:33Z", "updated_at": "2020-01-05T17:34:33Z", "author_association": "OWNER", "body": "Released as 2.0.1 https://github.com/simonw/sqlite-utils/releases/tag/2.0.1", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 545407916, "label": "upsert_all() throws issue when upserting to empty table"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/73#issuecomment-571138093", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/73", "id": 571138093, "node_id": "MDEyOklzc3VlQ29tbWVudDU3MTEzODA5Mw==", "user": {"value": 82988, "label": "psychemedia"}, "created_at": "2020-01-06T13:28:31Z", "updated_at": "2020-01-06T13:28:31Z", "author_association": "NONE", "body": "I think I actually had several issues in play...\r\n\r\nThe missing key was one, but I think there is also an issue as per below.\r\n\r\nFor example, in the following:\r\n\r\n```python\r\ndef init_testdb(dbname='test.db'):\r\n \r\n if os.path.exists(dbname):\r\n os.remove(dbname)\r\n\r\n conn = sqlite3.connect(dbname)\r\n db = Database(conn)\r\n \r\n return conn, db\r\n\r\nconn, db = init_testdb()\r\n\r\nc = conn.cursor()\r\nc.executescript('CREATE TABLE \"test1\" (\"Col1\" TEXT, \"Col2\" TEXT, PRIMARY KEY (\"Col1\"));')\r\nc.executescript('CREATE TABLE \"test2\" (\"Col1\" TEXT, \"Col2\" TEXT, PRIMARY KEY (\"Col1\"));')\r\n\r\nprint('Test 1...')\r\nfor i in range(3):\r\n db['test1'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1'))\r\n db['test2'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1'))\r\n\r\nprint('Test 2...')\r\nfor i in range(3):\r\n db['test1'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1'))\r\n db['test2'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'},\r\n {'Col1':'c','Col2':'x'}], pk=('Col1'))\r\nprint('Done...')\r\n\r\n---------------------------------------------------------------------------\r\nTest 1...\r\nTest 2...\r\n IndexError: list index out of range \r\n---------------------------------------------------------------------------\r\nIndexError Traceback (most recent call last)\r\n in \r\n 22 print('Test 2...')\r\n 23 for i in range(3):\r\n---> 24 db['test1'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1'))\r\n 25 db['test2'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'},\r\n 26 {'Col1':'c','Col2':'x'}], pk=('Col1'))\r\n\r\n/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in upsert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, extracts)\r\n 1157 alter=alter,\r\n 1158 extracts=extracts,\r\n-> 1159 upsert=True,\r\n 1160 )\r\n 1161 \r\n\r\n/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, ignore, replace, extracts, upsert)\r\n 1097 # self.last_rowid will be 0 if a \"INSERT OR IGNORE\" happened\r\n 1098 if (hash_id or pk) and self.last_rowid:\r\n-> 1099 row = list(self.rows_where(\"rowid = ?\", [self.last_rowid]))[0]\r\n 1100 if hash_id:\r\n 1101 self.last_pk = row[hash_id]\r\n\r\nIndexError: list index out of range\r\n```\r\n\r\nthe first test works but the second fails. Is the length of the list of items being upserted leaking somewhere?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 545407916, "label": "upsert_all() throws issue when upserting to empty table"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/73#issuecomment-572870032", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/73", "id": 572870032, "node_id": "MDEyOklzc3VlQ29tbWVudDU3Mjg3MDAzMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-01-10T04:38:41Z", "updated_at": "2020-01-10T04:38:41Z", "author_association": "OWNER", "body": "Odd.. I'm not able to replicate that error. Here's what I got:\r\n\"Tony_Bug\"\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 545407916, "label": "upsert_all() throws issue when upserting to empty table"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/73#issuecomment-573047321", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/73", "id": 573047321, "node_id": "MDEyOklzc3VlQ29tbWVudDU3MzA0NzMyMQ==", "user": {"value": 82988, "label": "psychemedia"}, "created_at": "2020-01-10T14:02:56Z", "updated_at": "2020-01-10T14:09:23Z", "author_association": "NONE", "body": "Hmmm... just tried with installs from pip and the repo (v2.0.0 and v2.0.1) and I get the error each time (start of second run through the second loop).\r\n\r\nCould it be sqlite3? I'm on 3.30.1.\r\n\r\nUPDATE: just tried it on jupyter.org/try and I get the error there, too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 545407916, "label": "upsert_all() throws issue when upserting to empty table"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/73#issuecomment-580745213", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/73", "id": 580745213, "node_id": "MDEyOklzc3VlQ29tbWVudDU4MDc0NTIxMw==", "user": {"value": 82988, "label": "psychemedia"}, "created_at": "2020-01-31T14:02:38Z", "updated_at": "2020-01-31T14:21:09Z", "author_association": "NONE", "body": "So the conundrum continues.. The simple test case above now runs, but if I upsert a large number of new records (successfully) and then try to upsert a fewer number of new records to a different table, I get the same error.\r\n\r\nIf I run the same upserts again (which in the first case means there are no new records to add, because they were already added), the second upsert works correctly.\r\n\r\nIt feels as if the number of items added via an upsert >> the number of items I try to add in an upsert immediately after, I get the error.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 545407916, "label": "upsert_all() throws issue when upserting to empty table"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/71#issuecomment-569233996", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/71", "id": 569233996, "node_id": "MDEyOklzc3VlQ29tbWVudDU2OTIzMzk5Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-12-27T09:45:17Z", "updated_at": "2019-12-27T09:45:17Z", "author_association": "OWNER", "body": "It looks like those backports no longer include sqlite3 - Google Searches still find it but when you click through to launchpad you get 404s: https://launchpad.net/~jonathonf/+archive/ubuntu/codelite/+build/10511920\r\n\r\nMaybe Travis have a newer Ubuntu I can use that ships with FTS5 in its SQLite?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 542814756, "label": "Tests are failing due to missing FTS5"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/71#issuecomment-569234096", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/71", "id": 569234096, "node_id": "MDEyOklzc3VlQ29tbWVudDU2OTIzNDA5Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-12-27T09:45:52Z", "updated_at": "2019-12-27T09:45:52Z", "author_association": "OWNER", "body": "I'll try `bionic`: https://docs.travis-ci.com/user/reference/bionic/", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 542814756, "label": "Tests are failing due to missing FTS5"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/71#issuecomment-569234571", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/71", "id": 569234571, "node_id": "MDEyOklzc3VlQ29tbWVudDU2OTIzNDU3MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-12-27T09:48:48Z", "updated_at": "2019-12-27T09:48:48Z", "author_association": "OWNER", "body": "That fixed it: https://travis-ci.com/simonw/sqlite-utils/builds/142443259", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 542814756, "label": "Tests are failing due to missing FTS5"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/70#issuecomment-569130037", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/70", "id": 569130037, "node_id": "MDEyOklzc3VlQ29tbWVudDU2OTEzMDAzNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-12-26T20:39:04Z", "updated_at": "2019-12-26T20:39:04Z", "author_association": "OWNER", "body": "I hadn't thought about those at all.\r\n\r\nAre you suggesting a utility mechanism in the library for setting it up so that, for a specific foreign key, rows are deleted from other tables if the row they are pointing at is deleted?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 539204432, "label": "Implement ON DELETE and ON UPDATE actions for foreign keys"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/70#issuecomment-575799104", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/70", "id": 575799104, "node_id": "MDEyOklzc3VlQ29tbWVudDU3NTc5OTEwNA==", "user": {"value": 26292069, "label": "LucasElArruda"}, "created_at": "2020-01-17T21:20:17Z", "updated_at": "2020-01-17T21:20:17Z", "author_association": "NONE", "body": "Omg sorry I took so long to reply!\r\n\r\nOn SQL we can say how the foreign key behaves when it is deleted or updated on the parent table (see https://www.sqlitetutorial.net/sqlite-foreign-key/ for more details).\r\n\r\nI did not see clearly how to create tables with this feature on sqlite-utils library.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 539204432, "label": "Implement ON DELETE and ON UPDATE actions for foreign keys"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/7#issuecomment-457980966", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/7", "id": 457980966, "node_id": "MDEyOklzc3VlQ29tbWVudDQ1Nzk4MDk2Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-01-28T02:29:32Z", "updated_at": "2019-01-28T02:29:32Z", "author_association": "OWNER", "body": "Remember to remove this TODO (and turn the `[]` into `()` on this line) as part of this task:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/5309c5c7755818323a0f5353bad0de98ecc866be/sqlite_utils/cli.py#L78-L80", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 403625674, "label": ".insert_all() should accept a generator and process it efficiently"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/7#issuecomment-458011885", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/7", "id": 458011885, "node_id": "MDEyOklzc3VlQ29tbWVudDQ1ODAxMTg4NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-01-28T06:25:48Z", "updated_at": "2019-01-28T06:25:48Z", "author_association": "OWNER", "body": "Re-opening for the second bit involving the cli tool.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 403625674, "label": ".insert_all() should accept a generator and process it efficiently"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/7#issuecomment-458011906", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/7", "id": 458011906, "node_id": "MDEyOklzc3VlQ29tbWVudDQ1ODAxMTkwNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-01-28T06:25:55Z", "updated_at": "2019-01-28T06:25:55Z", "author_association": "OWNER", "body": "I tested this with a script called `churn_em_out.py`\r\n```\r\ni = 0\r\nwhile True:\r\n i += 1\r\n print(\r\n '{\"id\": I, \"another\": \"row\", \"number\": J}'.replace(\"I\", str(i)).replace(\r\n \"J\", str(i + 1)\r\n )\r\n )\r\n```\r\n\r\nThen I ran this:\r\n```\r\npython churn_em_out.py | \\\r\n sqlite-utils insert /tmp/getbig.db stats - \\\r\n --nl --batch-size=10000\r\n```\r\nAnd used `watch 'ls -lah /tmp/getbig.db'` to watch the file growing as it had 10,000 lines of junk committed in batches. The memory used by the process never grew about around 50MB.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 403625674, "label": ".insert_all() should accept a generator and process it efficiently"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/69#issuecomment-569130196", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/69", "id": 569130196, "node_id": "MDEyOklzc3VlQ29tbWVudDU2OTEzMDE5Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-12-26T20:40:21Z", "updated_at": "2019-12-26T20:40:21Z", "author_association": "OWNER", "body": "This is a good idea. Datasette has this in the form of the `--load-extension` CLI argument, e.g. for SpatiaLite here: https://datasette.readthedocs.io/en/stable/spatialite.html#installation\r\n\r\nHaving that available for `sqlite-utils` definitely makes sense.", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 1}", "issue": {"value": 534507142, "label": "Feature request: enable extensions loading"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/69#issuecomment-710405658", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/69", "id": 710405658, "node_id": "MDEyOklzc3VlQ29tbWVudDcxMDQwNTY1OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-16T18:42:48Z", "updated_at": "2020-10-16T18:42:48Z", "author_association": "OWNER", "body": "Did some work on this for #134 in 7e9aad7e1c09d1cf80d0b4d17d6157212a4b857d\r\n\r\nI still need to add `--load-extension` to other CLI methods, see #137. Closing this issue in favour of that one.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 534507142, "label": "Feature request: enable extensions loading"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/69#issuecomment-710768396", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/69", "id": 710768396, "node_id": "MDEyOklzc3VlQ29tbWVudDcxMDc2ODM5Ng==", "user": {"value": 30607, "label": "aborruso"}, "created_at": "2020-10-17T07:46:59Z", "updated_at": "2020-10-17T07:46:59Z", "author_association": "NONE", "body": "Great @simonw thank you very much", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 534507142, "label": "Feature request: enable extensions loading"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/68#issuecomment-695695776", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/68", "id": 695695776, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NTY5NTc3Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-20T04:25:47Z", "updated_at": "2020-09-20T04:25:47Z", "author_association": "OWNER", "body": "This is a dupe of #130 ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 531583658, "label": "Add support for porter stemming in FTS"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/67#issuecomment-559108591", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/67", "id": 559108591, "node_id": "MDEyOklzc3VlQ29tbWVudDU1OTEwODU5MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-27T14:24:59Z", "updated_at": "2019-11-27T14:24:59Z", "author_association": "OWNER", "body": "Failed due to black testing dependency: https://travis-ci.com/simonw/sqlite-utils/jobs/260995814", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 529376481, "label": "Run tests against 3.5 too"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/67#issuecomment-569844320", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/67", "id": 569844320, "node_id": "MDEyOklzc3VlQ29tbWVudDU2OTg0NDMyMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-12-31T01:29:43Z", "updated_at": "2019-12-31T01:29:43Z", "author_association": "OWNER", "body": "I don't really care about 3.5 any more.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 529376481, "label": "Run tests against 3.5 too"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553170650", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/66", "id": 553170650, "node_id": "MDEyOklzc3VlQ29tbWVudDU1MzE3MDY1MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-12T23:49:29Z", "updated_at": "2019-11-12T23:49:29Z", "author_association": "OWNER", "body": "This relates to this bug: https://github.com/dogsheep/github-to-sqlite/pull/8#issuecomment-549233778", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521868864, "label": "The \".upsert()\" method is misnamed"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553171011", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/66", "id": 553171011, "node_id": "MDEyOklzc3VlQ29tbWVudDU1MzE3MTAxMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-12T23:50:52Z", "updated_at": "2019-11-12T23:50:52Z", "author_association": "OWNER", "body": "Fixing this is going to be a real pain. There's lots of code out there that uses `sqlite-utils` with the expectation that `upsert()` behaves as it currently does.\r\n\r\nMaybe I need to introduce new terms for both of these different patterns and deprecate the existing `.upsert()` and `.upsert_all()` since their behaviour can't be changed?\r\n\r\nOr maybe I fix this and ship `sqlite-utils 2.0` with a breaking change?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521868864, "label": "The \".upsert()\" method is misnamed"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553171414", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/66", "id": 553171414, "node_id": "MDEyOklzc3VlQ29tbWVudDU1MzE3MTQxNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-12T23:52:35Z", "updated_at": "2019-11-12T23:52:35Z", "author_association": "OWNER", "body": "If I do implement the correct definition of `.upsert()` I think I'll use this pattern, since it works in versions of SQLite prior to 3.24:\r\n```sql\r\nINSERT OR IGNORE INTO book(id) VALUES(1001);\r\nUPDATE book SET name = 'Programming' WHERE id = 1001;\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521868864, "label": "The \".upsert()\" method is misnamed"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553526685", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/66", "id": 553526685, "node_id": "MDEyOklzc3VlQ29tbWVudDU1MzUyNjY4NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-13T17:58:59Z", "updated_at": "2019-11-13T17:58:59Z", "author_association": "OWNER", "body": "This warrants making a backwards compatible change, which means I'll need to bump the major version number and release 2.0.\r\n\r\nI'm going to rename the existing `upsert()` and `upsert_all()` methods to `replace()` and `replace_all()` - then write new `upsert()` and `upsert_all()` methods that implement the correct behavior.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521868864, "label": "The \".upsert()\" method is misnamed"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553527384", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/66", "id": 553527384, "node_id": "MDEyOklzc3VlQ29tbWVudDU1MzUyNzM4NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-13T18:00:41Z", "updated_at": "2019-11-13T18:00:41Z", "author_association": "OWNER", "body": "Is `replace()` a good name here? It doesn't really convey the idea that a brand new record will be created if there isn't an existing one to replace.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521868864, "label": "The \".upsert()\" method is misnamed"}, "performed_via_github_app": null}