issues: 1250161887

This data as json

id node_id number title user state locked assignee milestone comments created_at updated_at closed_at author_association pull_request body repo type active_lock_reason performed_via_github_app reactions draft state_reason
1250161887 I_kwDOCGYnMM5Kg_Tf 438 illegal UTF-16 surrogate 4068 closed 0     2 2022-05-26T22:49:52Z 2022-05-27T08:21:53Z 2022-05-27T08:21:53Z NONE  

I am trying to insert https://artsdatabanken.no/Fab2018/api/export/csv into a SQLite database, but I have an error when using sqlite-utils:

sqlite-utils insert --csv --delimiter ";" --encoding="utf-16-le" --pk "Id" csv fremmedart test.db
  [------------------------------------]    0%
Error: 'utf-16-le' codec can't decode bytes in position 98-99: illegal UTF-16 surrogate

The input you provided uses a character encoding other than utf-8.

You can fix this by passing the --encoding= option with the encoding of the file.

If you do not know the encoding, running 'file filename.csv' may tell you.

It's often worth trying: --encoding=latin-1

I tried to convert the file using iconv -f "utf-16le" -t "utf-8", but I still get a similar error (slightly different position):

sqlite-utils insert --csv --delimiter ";" --encoding=utf-8 --pk "Id" csv_utf8 fremmedart test.db
  [------------------------------------]    0%
Error: 'utf-8' codec can't decode byte 0xd9 in position 99: invalid continuation byte

The input you provided uses a character encoding other than utf-8.

You can fix this by passing the --encoding= option with the encoding of the file.

If you do not know the encoding, running 'file filename.csv' may tell you.

It's often worth trying: --encoding=latin-1

I have no issues reading such file using this Python code:

content = open('csv', encoding='utf-16-le').read())

in2csv works too.

 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/438/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed

Links from other tables