{"html_url": "https://github.com/simonw/sqlite-utils/issues/186#issuecomment-897600677", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/186", "id": 897600677, "node_id": "IC_kwDOCGYnMM41gEyl", "user": {"value": 9308268, "label": "rayvoelker"}, "created_at": "2021-08-12T12:32:14Z", "updated_at": "2021-08-12T12:32:14Z", "author_association": "NONE", "body": "Actually, I forgot to include the `bib_pub_year` in the extract ... \r\n\r\nBut also, I tried again with empty string values instead of `NULL` values and it seems to place the foreign key properly / correctly... \r\n\r\n```python3\r\nsql = \"\"\"\\\r\nINSERT INTO \"circulation_info\" (\"item_id\", \"bib_title\", \"bib_creator\", \"bib_format\", \"bib_pub_year\", \"checkout_date\")\r\nVALUES\r\n(1, \"title one\", \"creator one\", \"Book\", 2018, \"2021-08-12 00:01\"),\r\n(2, \"title two\", \"creator one\", \"Book\", 2019, \"2021-08-12 00:02\"),\r\n(3, \"title three\", \"\", \"DVD\", 2020, \"2021-08-12 00:03\"),\r\n(4, \"title four\", \"\", \"DVD\", \"\", \"2021-08-12 00:04\"),\r\n(5, \"title five\", \"\", \"DVD\", \"\", \"2021-08-12 00:05\")\r\n\"\"\"\r\n\r\nwith sqlite3.connect('test_bib_2.db') as con:\r\n    con.execute(sql)\r\n```\r\n\r\n```python3\r\ndb[\"circulation_info\"].extract(\r\n    [\r\n        \"bib_title\",\r\n        \"bib_creator\",\r\n        \"bib_format\",\r\n        \"bib_pub_year\"\r\n    ],\r\n    table=\"bib_info\", \r\n    fk_column=\"bib_info_id\"\r\n)\r\n```\r\n\r\n```\r\n{'id': 1, 'item_id': 1, 'bib_info_id': 1, 'bib_pub_year': 2018, 'checkout_date': '2021-08-12 00:01'}\r\n{'id': 2, 'item_id': 2, 'bib_info_id': 2, 'bib_pub_year': 2019, 'checkout_date': '2021-08-12 00:02'}\r\n{'id': 3, 'item_id': 3, 'bib_info_id': 3, 'bib_pub_year': 2020, 'checkout_date': '2021-08-12 00:03'}\r\n{'id': 4, 'item_id': 4, 'bib_info_id': 4, 'bib_pub_year': '', 'checkout_date': '2021-08-12 00:04'}\r\n{'id': 5, 'item_id': 5, 'bib_info_id': 5, 'bib_pub_year': '', 'checkout_date': '2021-08-12 00:05'}\r\n\r\n---\r\n\r\n{'id': 1, 'bib_title': 'title one', 'bib_creator': 'creator one', 'bib_format': 'Book'}\r\n{'id': 2, 'bib_title': 'title two', 'bib_creator': 'creator one', 'bib_format': 'Book'}\r\n{'id': 3, 'bib_title': 'title three', 'bib_creator': '', 'bib_format': 'DVD'}\r\n{'id': 4, 'bib_title': 'title four', 'bib_creator': '', 'bib_format': 'DVD'}\r\n{'id': 5, 'bib_title': 'title five', 'bib_creator': '', 'bib_format': 'DVD'}\r\n```\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 722816436, "label": ".extract() shouldn't extract null values"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/186#issuecomment-897588624", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/186", "id": 897588624, "node_id": "IC_kwDOCGYnMM41gB2Q", "user": {"value": 9308268, "label": "rayvoelker"}, "created_at": "2021-08-12T12:13:25Z", "updated_at": "2021-08-12T12:13:25Z", "author_association": "NONE", "body": "I think I ran into an issue that's perhaps related with `extract()`\r\n\r\nI have a case where I want to create a lookup table for all the related title data where there are possibly multiple null values in the related columns ....\r\n```python3\r\nsql = \"\"\"\\\r\nINSERT INTO \"circulation_info\" (\"item_id\", \"bib_title\", \"bib_creator\", \"bib_format\", \"bib_pub_year\", \"checkout_date\")\r\nVALUES\r\n(1, \"title one\", \"creator one\", \"Book\", 2018, \"2021-08-12 00:01\"),\r\n(2, \"title two\", \"creator one\", \"Book\", 2019, \"2021-08-12 00:02\"),\r\n(3, \"title three\", NULL, \"DVD\", 2020, \"2021-08-12 00:03\"),\r\n(4, \"title four\", NULL, \"DVD\", NULL, \"2021-08-12 00:04\"),\r\n(5, \"title five\", NULL, \"DVD\", NULL, \"2021-08-12 00:05\")\r\n\"\"\"\r\n\r\nwith sqlite3.connect('test_bib.db') as con:\r\n    con.execute(sql)\r\n```\r\n\r\nwhen I run the `extract()` method ... \r\n\r\n```python3\r\ndb[\"circulation_info\"].extract(\r\n    [\r\n        \"bib_title\",\r\n        \"bib_creator\",\r\n        \"bib_format\"        \r\n    ],\r\n    table=\"bib_info\", \r\n    fk_column=\"bib_info_id\"\r\n)\r\n\r\ndb = sqlite_utils.Database(\"test_bib.db\")\r\n\r\nfor row in db[\"circulation_info\"].rows:\r\n    print(row)\r\n\r\nprint(\"\\n---\\n\")\r\n\r\nfor row in db[\"bib_info\"].rows:\r\n    print(row)\r\n```\r\n\r\nresults in this .. \r\n```\r\n{'id': 1, 'item_id': 1, 'bib_info_id': 1, 'bib_pub_year': 2018, 'checkout_date': '2021-08-12 00:01'}\r\n{'id': 2, 'item_id': 2, 'bib_info_id': 2, 'bib_pub_year': 2019, 'checkout_date': '2021-08-12 00:02'}\r\n{'id': 3, 'item_id': 3, 'bib_info_id': None, 'bib_pub_year': 2020, 'checkout_date': '2021-08-12 00:03'}\r\n{'id': 4, 'item_id': 4, 'bib_info_id': None, 'bib_pub_year': None, 'checkout_date': '2021-08-12 00:04'}\r\n{'id': 5, 'item_id': 5, 'bib_info_id': None, 'bib_pub_year': None, 'checkout_date': '2021-08-12 00:05'}\r\n\r\n---\r\n\r\n{'id': 1, 'bib_title': 'title one', 'bib_creator': 'creator one', 'bib_format': 'Book'}\r\n{'id': 2, 'bib_title': 'title two', 'bib_creator': 'creator one', 'bib_format': 'Book'}\r\n{'id': 3, 'bib_title': 'title three', 'bib_creator': None, 'bib_format': 'DVD'}\r\n{'id': 4, 'bib_title': 'title four', 'bib_creator': None, 'bib_format': 'DVD'}\r\n{'id': 5, 'bib_title': 'title five', 'bib_creator': None, 'bib_format': 'DVD'}\r\n```\r\n\r\nSeems like it's correctly generating the row data for those lookups, but it's not correctly updating the foreign key back to the primary table? Looks like it just results in a `NULL` value in that original table.\r\n\r\nAny ideas on why? Thanks again!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 722816436, "label": ".extract() shouldn't extract null values"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/186#issuecomment-710198162", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/186", "id": 710198162, "node_id": "MDEyOklzc3VlQ29tbWVudDcxMDE5ODE2Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-16T16:41:00Z", "updated_at": "2020-10-16T16:41:00Z", "author_association": "OWNER", "body": "Failing test:\r\n```python\r\ndef test_extract_null_values(fresh_db):\r\n    fresh_db[\"species\"].insert({\"id\": 1, \"species\": \"Wolf\"}, pk=\"id\")\r\n    fresh_db[\"individuals\"].insert_all(\r\n        [\r\n            {\"id\": 10, \"name\": \"Terriana\", \"species\": \"Fox\"},\r\n            {\"id\": 11, \"name\": \"Spenidorm\", \"species\": None},\r\n            {\"id\": 12, \"name\": \"Grantheim\", \"species\": \"Wolf\"},\r\n            {\"id\": 13, \"name\": \"Turnutopia\", \"species\": None},\r\n            {\"id\": 14, \"name\": \"Wargal\", \"species\": \"Wolf\"},\r\n        ],\r\n        pk=\"id\",\r\n    )\r\n    fresh_db[\"individuals\"].extract(\"species\")\r\n    assert fresh_db[\"species\"].schema == (\r\n        \"CREATE TABLE [species] (\\n\"\r\n        \"   [id] INTEGER PRIMARY KEY,\\n\"\r\n        \"   [species] TEXT\\n\"\r\n        \")\"\r\n    )\r\n    assert fresh_db[\"individuals\"].schema == (\r\n        'CREATE TABLE \"individuals\" (\\n'\r\n        \"   [id] INTEGER PRIMARY KEY,\\n\"\r\n        \"   [name] TEXT,\\n\"\r\n        \"   [species_id] INTEGER,\\n\"\r\n        \"   FOREIGN KEY(species_id) REFERENCES species(id)\\n\"\r\n        \")\"\r\n    )\r\n    assert list(fresh_db[\"species\"].rows) == [\r\n        {\"id\": 1, \"species\": \"Wolf\"},\r\n        {\"id\": 2, \"species\": \"Fox\"},\r\n    ]\r\n    assert list(fresh_db[\"individuals\"].rows) == [\r\n        {\"id\": 10, \"name\": \"Terriana\", \"species_id\": 2},\r\n        {\"id\": 11, \"name\": \"Spenidorm\", \"species_id\": None},\r\n        {\"id\": 12, \"name\": \"Grantheim\", \"species_id\": 1},\r\n        {\"id\": 13, \"name\": \"Turnutopia\", \"species_id\": None},\r\n        {\"id\": 14, \"name\": \"Wargal\", \"species_id\": 1},\r\n    ]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 722816436, "label": ".extract() shouldn't extract null values"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/186#issuecomment-709706260", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/186", "id": 709706260, "node_id": "MDEyOklzc3VlQ29tbWVudDcwOTcwNjI2MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-16T03:17:02Z", "updated_at": "2020-10-16T03:17:17Z", "author_association": "OWNER", "body": "Actually I think this should be an option to `.extract()` which controls if nulls are extracted or left alone. Maybe called `extract_nulls=True/False`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 722816436, "label": ".extract() shouldn't extract null values"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/186#issuecomment-709706065", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/186", "id": 709706065, "node_id": "MDEyOklzc3VlQ29tbWVudDcwOTcwNjA2NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-16T03:16:22Z", "updated_at": "2020-10-16T03:16:22Z", "author_association": "OWNER", "body": "Either way I think I'm going to need to add some SQL which uses `where a = b or (a is null and b is null)`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 722816436, "label": ".extract() shouldn't extract null values"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/186#issuecomment-709705885", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/186", "id": 709705885, "node_id": "MDEyOklzc3VlQ29tbWVudDcwOTcwNTg4NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-16T03:15:39Z", "updated_at": "2020-10-16T03:15:39Z", "author_association": "OWNER", "body": "The alternative solution here would be that a single `null` value DOES get extracted. To implement this I would need to add some logic that uses `is null` instead of `=`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 722816436, "label": ".extract() shouldn't extract null values"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/186#issuecomment-709705624", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/186", "id": 709705624, "node_id": "MDEyOklzc3VlQ29tbWVudDcwOTcwNTYyNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-16T03:14:39Z", "updated_at": "2020-10-16T03:14:39Z", "author_association": "OWNER", "body": "How should this work with extractions covering multiple columns?\r\n\r\nIf there's a single column then it makes sense that a `null` value would not be extracted into the lookup table, but would instead become stay as `null`.\r\n\r\nFor a multiple column extraction, provided at least one of those columns is not null It should map to a record in the lookup table. Only if ALL of the extracted columns are null should the lookup value stay null.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 722816436, "label": ".extract() shouldn't extract null values"}, "performed_via_github_app": null}