{"html_url": "https://github.com/simonw/sqlite-utils/issues/356#issuecomment-1006318007", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/356", "id": 1006318007, "node_id": "IC_kwDOCGYnMM47-zG3", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-06T06:28:53Z", "updated_at": "2022-01-06T06:28:53Z", "author_association": "OWNER", "body": "Implemented in #361.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1077431957, "label": "`sqlite-utils insert --convert` option"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/356#issuecomment-1006318443", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/356", "id": 1006318443, "node_id": "IC_kwDOCGYnMM47-zNr", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-06T06:30:13Z", "updated_at": "2022-01-06T06:30:13Z", "author_association": "OWNER", "body": "Documentation:\r\n\r\n- https://sqlite-utils.datasette.io/en/latest/cli.html#inserting-unstructured-data-with-lines-and-text\r\n- https://sqlite-utils.datasette.io/en/latest/cli.html#applying-conversions-while-inserting-data", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1077431957, "label": "`sqlite-utils insert --convert` option"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/356#issuecomment-991517209", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/356", "id": 991517209, "node_id": "IC_kwDOCGYnMM47GVoZ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-11T07:46:41Z", "updated_at": "2021-12-11T07:46:41Z", "author_association": "OWNER", "body": "By default this will accept single lines, but maybe there could be a `--all` option which instead grabs all of stdin into a single string against which the conversion function runs - like `git-history file`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1077431957, "label": "`sqlite-utils insert --convert` option"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997485361", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/356", "id": 997485361, "node_id": "IC_kwDOCGYnMM47dGsx", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-19T23:45:30Z", "updated_at": "2021-12-19T23:45:30Z", "author_association": "OWNER", "body": "Really interesting example input for this: https://blog.timac.org/2021/1219-state-of-swift-and-swiftui-ios15/iOS13.txt - see https://blog.timac.org/2021/1219-state-of-swift-and-swiftui-ios15/", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1077431957, "label": "`sqlite-utils insert --convert` option"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997486156", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/356", "id": 997486156, "node_id": "IC_kwDOCGYnMM47dG5M", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-19T23:51:02Z", "updated_at": "2021-12-19T23:51:02Z", "author_association": "OWNER", "body": "This is going to need a `--import` multi option too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1077431957, "label": "`sqlite-utils insert --convert` option"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997492872", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/356", "id": 997492872, "node_id": "IC_kwDOCGYnMM47dIiI", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-20T00:23:31Z", "updated_at": "2021-12-20T00:23:31Z", "author_association": "OWNER", "body": "I think this should work on JSON, or CSV, or individual lines, or the entire content at once.\r\n\r\nSo I'll require `--lines --convert ...` to import individual lines, or `--all --convert` to run the conversion against the entire input at once.\r\n\r\nWhat would `--lines` or `--all` do without `--convert`? Maybe insert records as `{\"line\": \"line of text\"}` or `{\"all\": \"whole input}`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1077431957, "label": "`sqlite-utils insert --convert` option"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997496626", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/356", "id": 997496626, "node_id": "IC_kwDOCGYnMM47dJcy", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-20T00:38:15Z", "updated_at": "2022-01-06T01:29:03Z", "author_association": "OWNER", "body": "The implementation of this gets a tiny bit complicated.\r\n\r\nIgnoring `--convert`, the `--lines` option can internally produce `{\"line\": ...}` records and the `--all` option can produce `{\"all\": ...}` records.\r\n\r\nBut... when `--convert` is used, what should the code run against?\r\n\r\nIt could run against those already-converted records but that's a little bit strange, since you'd have to do this:\r\n\r\n    sqlite-utils insert blah.db blah myfile.txt --all --convert '{\"item\": s for s in value[\"all\"].split(\"-\")}'\r\n\r\nHaving to use `value[\"all\"]` there is unintuitive. It would be nicer to have a `all` variable to work against.\r\n\r\nBut then for `--lines` should the local variable be called `line`? And how best to summarize these different names for local variables in the inline help for the feature?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1077431957, "label": "`sqlite-utils insert --convert` option"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997496931", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/356", "id": 997496931, "node_id": "IC_kwDOCGYnMM47dJhj", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-20T00:39:14Z", "updated_at": "2021-12-20T00:39:52Z", "author_association": "OWNER", "body": "```\r\n% sqlite-utils insert --help\r\nUsage: sqlite-utils insert [OPTIONS] PATH TABLE JSON_FILE\r\n\r\n  Insert records from JSON file into a table, creating the table if it does\r\n  not already exist.\r\n\r\n  Input should be a JSON array of objects, unless --nl or --csv is used.\r\n\r\nOptions:\r\n  --pk TEXT                 Columns to use as the primary key, e.g. id\r\n  --nl                      Expect newline-delimited JSON\r\n  --flatten                 Flatten nested JSON objects\r\n  -c, --csv                 Expect CSV\r\n  --tsv                     Expect TSV\r\n  --convert TEXT            Python code to convert each item\r\n  --import TEXT             Python modules to import\r\n  --delimiter TEXT          Delimiter to use for CSV files\r\n  --quotechar TEXT          Quote character to use for CSV/TSV\r\n  --sniff                   Detect delimiter and quote character\r\n  --no-headers              CSV file has no header row\r\n  --batch-size INTEGER      Commit every X records\r\n  --alter                   Alter existing table to add any missing columns\r\n  --not-null TEXT           Columns that should be created as NOT NULL\r\n  --default <TEXT TEXT>...  Default value that should be set for a column\r\n  --encoding TEXT           Character encoding for input, defaults to utf-8\r\n  -d, --detect-types        Detect types for columns in CSV/TSV data\r\n  --load-extension TEXT     SQLite extensions to load\r\n  --silent                  Do not show progress bar\r\n  --ignore                  Ignore records if pk already exists\r\n  --replace                 Replace records if pk already exists\r\n  --truncate                Truncate table before inserting records, if table\r\n                            already exists\r\n  -h, --help                Show this message and exit.\r\n```\r\nI can add a bunch of extra help at the top there to explain all of this stuff. That \"Input should be a JSON array of objects\" bit could be expanded to several paragraphs.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1077431957, "label": "`sqlite-utils insert --convert` option"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997497262", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/356", "id": 997497262, "node_id": "IC_kwDOCGYnMM47dJmu", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-20T00:40:15Z", "updated_at": "2021-12-20T00:40:15Z", "author_association": "OWNER", "body": "`--flatten` could do with a better description too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1077431957, "label": "`sqlite-utils insert --convert` option"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997507074", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/356", "id": 997507074, "node_id": "IC_kwDOCGYnMM47dMAC", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-20T01:10:06Z", "updated_at": "2021-12-20T01:16:11Z", "author_association": "OWNER", "body": "Work-in-progress improved help:\r\n```\r\nUsage: sqlite-utils insert [OPTIONS] PATH TABLE FILE\r\n\r\n  Insert records from FILE into a table, creating the table if it does not\r\n  already exist.\r\n\r\n  By default the input is expected to be a JSON array of objects. Or:\r\n\r\n  - Use --nl for newline-delimited JSON objects\r\n  - Use --csv or --tsv for comma-separated or tab-separated input\r\n  - Use --lines to write each incoming line to a column called \"line\"\r\n  - Use --all to write the entire input to a column called \"all\"\r\n\r\n  You can also use --convert to pass a fragment of Python code that will be\r\n  used to convert each input.\r\n\r\n  Your Python code will be passed a \"row\" variable representing the imported\r\n  row, and can return a modified row.\r\n\r\n  If you are using --lines your code will be passed a \"line\" variable, and for\r\n  --all an \"all\" variable.\r\n\r\nOptions:\r\n  --pk TEXT                 Columns to use as the primary key, e.g. id\r\n  --flatten                 Flatten nested JSON objects, so {\"a\": {\"b\": 1}}\r\n                            becomes {\"a_b\": 1}\r\n  --nl                      Expect newline-delimited JSON\r\n  -c, --csv                 Expect CSV input\r\n  --tsv                     Expect TSV input\r\n  --lines                   Treat each line as a single value called 'line'\r\n  --all                     Treat input as a single value called 'all'\r\n  --convert TEXT            Python code to convert each item\r\n  --import TEXT             Python modules to import\r\n  --delimiter TEXT          Delimiter to use for CSV files\r\n  --quotechar TEXT          Quote character to use for CSV/TSV\r\n  --sniff                   Detect delimiter and quote character\r\n  --no-headers              CSV file has no header row\r\n  --batch-size INTEGER      Commit every X records\r\n  --alter                   Alter existing table to add any missing columns\r\n  --not-null TEXT           Columns that should be created as NOT NULL\r\n  --default <TEXT TEXT>...  Default value that should be set for a column\r\n  --encoding TEXT           Character encoding for input, defaults to utf-8\r\n  -d, --detect-types        Detect types for columns in CSV/TSV data\r\n  --load-extension TEXT     SQLite extensions to load\r\n  --silent                  Do not show progress bar\r\n  --ignore                  Ignore records if pk already exists\r\n  --replace                 Replace records if pk already exists\r\n  --truncate                Truncate table before inserting records, if table\r\n                            already exists\r\n  -h, --help                Show this message and exit.\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1077431957, "label": "`sqlite-utils insert --convert` option"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997508728", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/356", "id": 997508728, "node_id": "IC_kwDOCGYnMM47dMZ4", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-20T01:14:43Z", "updated_at": "2021-12-20T01:14:43Z", "author_association": "OWNER", "body": "(This makes me want `--extract` from #352 even more.)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1077431957, "label": "`sqlite-utils insert --convert` option"}, "performed_via_github_app": null}