issues

Folder structure:

bibliography/ bibliography/static-files bibliography/static-files/styles.css bibliography/bibliography.db bibliography/metadata.json bibliography/settings.json

$ cat bibliography/settings.json { "suggest_facets": false, "truncate_cells_html": 1000, "static": "assets:static-files/" }

File /assets/styles.css is not found (HTTP 404, Database not found: assets).

Using datasette revision d0737e4de51ce178e556fc011ccb8cc46bbb6359.

The program crashes without any error. wget "https://artsdatabanken.no/Fab2018/api/export/csv" sqlite-utils create-database test.db sqlite-utils insert --csv --delimiter ";" --encoding "utf-16-le" test test.db csv [------------------------------------] 0% [#################-------------------] 49% 00:00:01 I would like to highlight various issues: 1. sqlite-utils catches exceptions without printing the stacktrace and/or reraising the exception, so there is no easy way to use pdb or similar to debug the program, solution: add a debug option 2. Silent crash: this is related to (1.), and it happens when there is a catch-all mechanism; solution: let the program fail.

Original title: csv.DictReader can have None as key

In some cases, csv.DictReader can have None as key for unnamed columns, and a list of values as value. sqlite_utils.utils.rows_from_file cannot handle that:

```python url="https://artsdatabanken.no/Fab2018/api/export/csv" db = sqlite_utils.Database(":memory")

with urlopen(url) as fab: reader, _ = sqlite_utils.utils.rows_from_file(fab, encoding="utf-16le")
db["fab2018"].insert_all(reader, pk="Id") ```

Result: Traceback (most recent call last): File "<stdin>", line 3, in <module> File "/home/user/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/sqlite_utils/db.py", line 2924, in insert_all chunk = list(chunk) File "/home/user/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/sqlite_utils/db.py", line 3454, in fix_square_braces if any("[" in key or "]" in key for key in record.keys()): File "/home/user/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/sqlite_utils/db.py", line 3454, in <genexpr> if any("[" in key or "]" in key for key in record.keys()): TypeError: argument of type 'NoneType' is not iterable

Code: https://github.com/simonw/sqlite-utils/blob/59be60c471fd7a2c4be7f75e8911163e618ff5ca/sqlite_utils/db.py#L3454

sqlite-utils insert from command line is not affected by this issue.

I am trying to insert https://artsdatabanken.no/Fab2018/api/export/csv into a SQLite database, but I have an error when using sqlite-utils:

``` sqlite-utils insert --csv --delimiter ";" --encoding="utf-16-le" --pk "Id" csv fremmedart test.db [------------------------------------] 0% Error: 'utf-16-le' codec can't decode bytes in position 98-99: illegal UTF-16 surrogate

The input you provided uses a character encoding other than utf-8.

You can fix this by passing the --encoding= option with the encoding of the file.

If you do not know the encoding, running 'file filename.csv' may tell you.

It's often worth trying: --encoding=latin-1 ```

I tried to convert the file using iconv -f "utf-16le" -t "utf-8", but I still get a similar error (slightly different position):

``` sqlite-utils insert --csv --delimiter ";" --encoding=utf-8 --pk "Id" csv_utf8 fremmedart test.db [------------------------------------] 0% Error: 'utf-8' codec can't decode byte 0xd9 in position 99: invalid continuation byte

The input you provided uses a character encoding other than utf-8.

You can fix this by passing the --encoding= option with the encoding of the file.

If you do not know the encoding, running 'file filename.csv' may tell you.

It's often worth trying: --encoding=latin-1 ```

I have no issues reading such file using this Python code: python content = open('csv', encoding='utf-16-le').read())

in2csv works too.

It would be great if the column type could be set to JSON. That would not be different from handling a regular string. It would be something like repr(value) and it would work with both JSON and CSV inputs, no matter if value is a real list or just a string representing a list.

Hi! Thank you for sharing this very nice tool :) It would be nice to have support for more types, like bool: it is not possible to convert to boolean at the moment. My suggestion would be to handle it as bool(int(value)), like csvkit does.

Hi! When a user tries to export data as JSON, I would expect to see the value of JSON columns represented as JSON instead of being rendered as a string. What do you think?