{"html_url": "https://github.com/simonw/sqlite-utils/issues/239#issuecomment-1236214402", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/239", "id": 1236214402, "node_id": "IC_kwDOCGYnMM5JryKC", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-03T23:46:02Z", "updated_at": "2022-09-03T23:46:02Z", "author_association": "OWNER", "body": "Yeah having a version of this that can setup m2m relationships would definitely be interesting.", "reactions": "{\"total_count\": 2, \"+1\": 2, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 816526538, "label": "sqlite-utils extract could handle nested objects"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/239#issuecomment-1236200834", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/239", "id": 1236200834, "node_id": "IC_kwDOCGYnMM5Jru2C", "user": {"value": 14294, "label": "hubgit"}, "created_at": "2022-09-03T21:26:32Z", "updated_at": "2022-09-03T21:26:32Z", "author_association": "NONE", "body": "I was looking for something like this today, for extracting columns containing objects (and arrays of objects) into separate tables. \r\n\r\nWould it make sense (especially for the fields containing arrays of objects) to create a one-to-many relationship, where each row of the newly created table would contain the id of the row that originally contained it?\r\n\r\nIf the extracted objects have a unique id and are repeated, it could even create a many-to-many relationship, with a third table for the joins.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 816526538, "label": "sqlite-utils extract could handle nested objects"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/239#issuecomment-960295228", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/239", "id": 960295228, "node_id": "IC_kwDOCGYnMM45PPE8", "user": {"value": 350038, "label": "tmaier"}, "created_at": "2021-11-03T23:35:37Z", "updated_at": "2021-11-03T23:36:50Z", "author_association": "NONE", "body": "I think I only wonder how I would parse the JSON `value` within such a lambda...\r\n\r\nMy naive approach would have been `$ sqlite-utils convert demo.db statuses statuses 'return value' --multi`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 816526538, "label": "sqlite-utils extract could handle nested objects"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/239#issuecomment-960292442", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/239", "id": 960292442, "node_id": "IC_kwDOCGYnMM45POZa", "user": {"value": 350038, "label": "tmaier"}, "created_at": "2021-11-03T23:28:55Z", "updated_at": "2021-11-03T23:28:55Z", "author_association": "NONE", "body": "I am super interested in this feature.\r\n\r\nAfter reading the other issues you referenced, I think the right way would be to use the current extract feature and then to use `sqlite-utils convert` to extract the json object into individual columns", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 816526538, "label": "sqlite-utils extract could handle nested objects"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/239#issuecomment-786830832", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/239", "id": 786830832, "node_id": "MDEyOklzc3VlQ29tbWVudDc4NjgzMDgzMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-02-26T18:52:40Z", "updated_at": "2021-02-26T18:52:40Z", "author_association": "OWNER", "body": "Could this handle lists of objects too? That would be pretty amazing - if the column has a `[{...}, {...}]` list in it could turn that into a many-to-many.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 816526538, "label": "sqlite-utils extract could handle nested objects"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/239#issuecomment-786795132", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/239", "id": 786795132, "node_id": "MDEyOklzc3VlQ29tbWVudDc4Njc5NTEzMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-02-26T17:45:53Z", "updated_at": "2021-02-26T17:45:53Z", "author_association": "OWNER", "body": "If there's no primary key in the JSON could use the `hash_id` mechanism.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 816526538, "label": "sqlite-utils extract could handle nested objects"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/239#issuecomment-786794435", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/239", "id": 786794435, "node_id": "MDEyOklzc3VlQ29tbWVudDc4Njc5NDQzNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-02-26T17:44:38Z", "updated_at": "2021-02-26T17:44:38Z", "author_association": "OWNER", "body": "This came up in office hours!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 816526538, "label": "sqlite-utils extract could handle nested objects"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/239#issuecomment-786035142", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/239", "id": 786035142, "node_id": "MDEyOklzc3VlQ29tbWVudDc4NjAzNTE0Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-02-25T16:36:17Z", "updated_at": "2021-02-25T16:36:17Z", "author_association": "OWNER", "body": "WIP in a pull request.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 816526538, "label": "sqlite-utils extract could handle nested objects"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785992158", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/239", "id": 785992158, "node_id": "MDEyOklzc3VlQ29tbWVudDc4NTk5MjE1OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-02-25T15:37:04Z", "updated_at": "2021-02-25T15:37:04Z", "author_association": "OWNER", "body": "Here's the current implementation of `.extract()`: https://github.com/simonw/sqlite-utils/blob/806c21044ac8d31da35f4c90600e98115aade7c6/sqlite_utils/db.py#L1049-L1074\r\n\r\nTricky detail here: I create the lookup table first, based on the types of the columns that are being extracted.\r\n\r\nI need to do this because extraction currently uses unique tuples of values, so the table has to be created in advance.\r\n\r\nBut if I'm using these new expand functions to figure out what's going to be extracted, I don't know the names of the columns and their types in advance. I'm only going to find those out during the transformation.\r\n\r\nThis may turn out to be incompatible with how `.extract()` works at the moment. I may need a new method, `.extract_expand()` perhaps? It could be simpler - work only against a single column for example.\r\n\r\nI can still use the existing `sqlite-utils extract` CLI command though, with a `--json` flag and a rule that you can't run it against multiple columns.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 816526538, "label": "sqlite-utils extract could handle nested objects"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785983837", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/239", "id": 785983837, "node_id": "MDEyOklzc3VlQ29tbWVudDc4NTk4MzgzNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-02-25T15:25:21Z", "updated_at": "2021-02-25T15:28:57Z", "author_association": "OWNER", "body": "Problem with calling this argument `transform=` is that the term \"transform\" already means something else in this library.\r\n\r\nI could use `convert=` instead.\r\n\r\n... but that doesn't instantly make me think of turning a value into multiple columns.\r\n\r\nHow about `expand=`? I've not used that term anywhere yet.\r\n\r\n    db[\"Reports\"].extract([\"Reported by\"], expand={\"Reported by\": json.loads})\r\n\r\nI think that works. You're expanding a single value into several columns of information.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 816526538, "label": "sqlite-utils extract could handle nested objects"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785983070", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/239", "id": 785983070, "node_id": "MDEyOklzc3VlQ29tbWVudDc4NTk4MzA3MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-02-25T15:24:17Z", "updated_at": "2021-02-25T15:24:17Z", "author_association": "OWNER", "body": "I'm going to go with last-wins - so if multiple transform functions return the same key the last one will over-write the others.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 816526538, "label": "sqlite-utils extract could handle nested objects"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785980813", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/239", "id": 785980813, "node_id": "MDEyOklzc3VlQ29tbWVudDc4NTk4MDgxMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-02-25T15:21:02Z", "updated_at": "2021-02-25T15:23:47Z", "author_association": "OWNER", "body": "Maybe the Python version takes an optional dictionary mapping column names to transformation functions? It could then merge all of those results together - and maybe throw an error if the same key is produced by more than one column.\r\n\r\n```python\r\n    db[\"Reports\"].extract([\"Reported by\"], transform={\"Reported by\": json.loads})\r\n```\r\nOr it could have an option for different strategies if keys collide: first wins, last wins, throw exception, add a prefix to the new column name. That feels a bit too complex for an edge-case though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 816526538, "label": "sqlite-utils extract could handle nested objects"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785980083", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/239", "id": 785980083, "node_id": "MDEyOklzc3VlQ29tbWVudDc4NTk4MDA4Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-02-25T15:20:02Z", "updated_at": "2021-02-25T15:20:02Z", "author_association": "OWNER", "body": "It would be OK if the CLI version only allows you to specify a single column if you are using the `--json` option.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 816526538, "label": "sqlite-utils extract could handle nested objects"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785979769", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/239", "id": 785979769, "node_id": "MDEyOklzc3VlQ29tbWVudDc4NTk3OTc2OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-02-25T15:19:37Z", "updated_at": "2021-02-25T15:19:37Z", "author_association": "OWNER", "body": "For the Python version I'd like to be able to provide a transformation callback function - which can be `json.loads` but could also be anything else which accepts the value of the current column and returns a Python dictionary of columns and their values to use in the new table.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 816526538, "label": "sqlite-utils extract could handle nested objects"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785979192", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/239", "id": 785979192, "node_id": "MDEyOklzc3VlQ29tbWVudDc4NTk3OTE5Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-02-25T15:18:46Z", "updated_at": "2021-02-25T15:18:46Z", "author_association": "OWNER", "body": "Likewise the `sqlite-utils extract` command takes one or more columns:\r\n```\r\nUsage: sqlite-utils extract [OPTIONS] PATH TABLE COLUMNS...\r\n\r\n  Extract one or more columns into a separate table\r\n\r\nOptions:\r\n  --table TEXT             Name of the other table to extract columns to\r\n  --fk-column TEXT         Name of the foreign key column to add to the table\r\n  --rename <TEXT TEXT>...  Rename this column in extracted table\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 816526538, "label": "sqlite-utils extract could handle nested objects"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785978689", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/239", "id": 785978689, "node_id": "MDEyOklzc3VlQ29tbWVudDc4NTk3ODY4OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-02-25T15:18:03Z", "updated_at": "2021-02-25T15:18:03Z", "author_association": "OWNER", "body": "The Python `.extract()` method currently starts like this:\r\n```python\r\ndef extract(self, columns, table=None, fk_column=None, rename=None):\r\n        rename = rename or {}\r\n        if isinstance(columns, str):\r\n            columns = [columns]\r\n        if not set(columns).issubset(self.columns_dict.keys()):\r\n            raise InvalidColumns(\r\n                \"Invalid columns {} for table with columns {}\".format(\r\n                    columns, list(self.columns_dict.keys())\r\n                )\r\n            )\r\n        ...\r\n```\r\nNote that it takes a list of columns (and treats a string as a single item list). That's because it can be called with a list of columns and it will use them to populate another table of unique tuples of those column values.\r\n\r\nSo a new mechanism that can instead read JSON values from a single column needs to be compatible with that existing design.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 816526538, "label": "sqlite-utils extract could handle nested objects"}, "performed_via_github_app": null}