issue_comments

Abandoning this since it didn't work how I wanted.

OK, this should hopefully fix that for you:

pip install https://github.com/simonw/datasette/archive/f36e010b3b69ada104b79d83c7685caf9359049e.zip

@wragge thanks, that's a bug! Working on that in #1575.

Looks like specname is new in Pluggy 1.0: https://github.com/pytest-dev/pluggy/blob/main/CHANGELOG.rst#pluggy-100-2021-08-25

Work-in-progress improved help: ``` Usage: sqlite-utils insert [OPTIONS] PATH TABLE FILE

Insert records from FILE into a table, creating the table if it does not already exist.

By default the input is expected to be a JSON array of objects. Or:

• Use --nl for newline-delimited JSON objects
• Use --csv or --tsv for comma-separated or tab-separated input
• Use --lines to write each incoming line to a column called "line"
• Use --all to write the entire input to a column called "all"

You can also use --convert to pass a fragment of Python code that will be used to convert each input.

Your Python code will be passed a "row" variable representing the imported row, and can return a modified row.

If you are using --lines your code will be passed a "line" variable, and for --all an "all" variable.

Options: --pk TEXT Columns to use as the primary key, e.g. id --flatten Flatten nested JSON objects, so {"a": {"b": 1}} becomes {"a_b": 1} --nl Expect newline-delimited JSON -c, --csv Expect CSV input --tsv Expect TSV input --lines Treat each line as a single value called 'line' --all Treat input as a single value called 'all' --convert TEXT Python code to convert each item --import TEXT Python modules to import --delimiter TEXT Delimiter to use for CSV files --quotechar TEXT Quote character to use for CSV/TSV --sniff Detect delimiter and quote character --no-headers CSV file has no header row --batch-size INTEGER Commit every X records --alter Alter existing table to add any missing columns --not-null TEXT Columns that should be created as NOT NULL --default <TEXT TEXT>... Default value that should be set for a column --encoding TEXT Character encoding for input, defaults to utf-8 -d, --detect-types Detect types for columns in CSV/TSV data --load-extension TEXT SQLite extensions to load --silent Do not show progress bar --ignore Ignore records if pk already exists --replace Replace records if pk already exists --truncate Truncate table before inserting records, if table already exists -h, --help Show this message and exit. ```

(This makes me want --extract from #352 even more.)

Maybe sqlite-utils should absorb all of the functionality from sqlite-transform - having two separate tools doesn't necessarily make sense.

I implemented that in: - #251

--flatten could do with a better description too.

``` % sqlite-utils insert --help Usage: sqlite-utils insert [OPTIONS] PATH TABLE JSON_FILE

Insert records from JSON file into a table, creating the table if it does not already exist.

Input should be a JSON array of objects, unless --nl or --csv is used.

Options: --pk TEXT Columns to use as the primary key, e.g. id --nl Expect newline-delimited JSON --flatten Flatten nested JSON objects -c, --csv Expect CSV --tsv Expect TSV --convert TEXT Python code to convert each item --import TEXT Python modules to import --delimiter TEXT Delimiter to use for CSV files --quotechar TEXT Quote character to use for CSV/TSV --sniff Detect delimiter and quote character --no-headers CSV file has no header row --batch-size INTEGER Commit every X records --alter Alter existing table to add any missing columns --not-null TEXT Columns that should be created as NOT NULL --default <TEXT TEXT>... Default value that should be set for a column --encoding TEXT Character encoding for input, defaults to utf-8 -d, --detect-types Detect types for columns in CSV/TSV data --load-extension TEXT SQLite extensions to load --silent Do not show progress bar --ignore Ignore records if pk already exists --replace Replace records if pk already exists --truncate Truncate table before inserting records, if table already exists -h, --help Show this message and exit. ``` I can add a bunch of extra help at the top there to explain all of this stuff. That "Input should be a JSON array of objects" bit could be expanded to several paragraphs.

I think this should work on JSON, or CSV, or individual lines, or the entire content at once.

So I'll require --lines --convert ... to import individual lines, or --all --convert to run the conversion against the entire input at once.

What would --lines or --all do without --convert? Maybe insert records as {"line": "line of text"} or {"all": "whole input}.