issue_comments

Documentation: https://sqlite-utils.readthedocs.io/en/latest/cli.html#inserting-csv-or-tsv-data

The file is opened for me by click.File(), which also handles things like - for stdin. But i neee to be able to switch the encoding used to read from that based on the --encoding option.

I think the way to do that is to open the file in binary mode and then wrap it in a codec reader: python fp = codecs.getreader(encoding)(fp)

For the moment I'm going to add a --encoding option and some code that catches UnicodeDecodeError and shows an error message that suggests using --encoding.

That error message could detect if the file command is available and, if it is, suggest running file filename.txt to detect the character encoding.

I could use https://github.com/chardet/chardet to help here, though I'd rather not add it as a dependency (sqlite-utils has very few dependencies at the moment). I could add it as an optional dependency though.

It looks like http://maps.natalian.org/data.txt is encoded as latin-1, but sqlite-utils assumes utf-8 and hence breaks.

It would be worth improving the error message here. I could also add a --encoding latin-1 option to sqlite-utils insert to help in consuming files that are stored in charsets other than utf-8.