issue_comments

Updated documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#inserting-data-from-files

I'm going to skip this test on windows.

https://discuss.python.org/t/pep-597-use-utf-8-for-default-text-file-encoding/1819 says:

Currently, TextIOWrapper uses locale.getpreferredencoding(False) (hereinafter called “locale encoding”) when encoding is not specified. ... Package authors using macOS or Linux may forget that the default encoding is not always UTF-8.

Hah, the error here is actually: ```

      assert result.exit_code == 1, result.output

E AssertionError: E
E assert 0 == 1 E + where 0 = <Result okay>.exit_code ``` So I was expecting an error, but instead the command worked.

I suspect this is because on Windows the default character set may not be UTF-8?

Huh, tests are failing but only on Windows!

Oh, I misread. Yes some files will not be valid UTF-8, I'd throw a warning and continue (not adding that file) but if you want to get more elaborate you could allow to define a policy on what to do. Not adding the file, index binary content or use a conversion policy like the ones available on Python's decode.

I thought about supporting those different policies (with something like --errors ignore) but I feel like that's getting a little bit too deep into the weeds. Right now if you try to import an invalid file the behaviour is the same as for the sqlite-utils insert command (I added the same detailed error message):

``` Error: Could not read file '/Users/simon/Dropbox/Development/sqlite-utils/data.txt' as text

'utf-8' codec can't decode byte 0xe3 in position 83: invalid continuation byte

The input you provided uses a character encoding other than utf-8.

You can fix this by passing the --encoding= option with the encoding of the file.

If you do not know the encoding, running 'file filename.csv' may tell you.

It's often worth trying: --encoding=latin-1 `` If someone has data that can't be translated to valid text using a known encoding, I'm happy leaving them to have to insert it into aBLOB` column instead.

I had a few doubts about the design just now. Since content_text is supported as a special argument, an alternative way of handling the above would be:

sqlite-utils insert-files /tmp/text.db files *.txt -c path -c content_text -c size

This does exactly the same thing as just using --text and not specifying any columns, because the actual implementation of --text is as follows:

https://github.com/simonw/sqlite-utils/blob/0c796cd945b146b7395ff5f553861400be503867/sqlite_utils/cli.py#L1851-L1855

But actually I think that's OK - --text is a useful shorthand that avoids you having to remember how to manually specify those columns with -c. So I'm going to leave the design as is.

Here's the error message I have working for invalid unicode: ``` sqlite-utils insert-files /tmp/text.db files *.txt --text [------------------------------------] 0% Error: Could not read file '/Users/simon/Dropbox/Development/sqlite-utils/data.txt' as text

'utf-8' codec can't decode byte 0xe3 in position 83: invalid continuation byte ```

I'm going to assume utf-8 but allow --encoding to be used to specify something different, since that option is already supported by other commands.

Will need to re-title this section of the documentation: https://sqlite-utils.datasette.io/en/3.16/cli.html#inserting-binary-data-from-files - "Inserting binary data from files" will become "Inserting data from files"

I'm OK with keeping the default as BLOB but I could add a --text option which stores the content as text instead.

If the text can't be stored as utf-8 I'll probably raise an error.

This is a good idea.

Pasting that CSS into the styles editor in the developer tools on https://latest.datasette.io/ has the desired effect: footer at the bottom of the window unless the page is too long, in which case the footer is at the bottom of the scroll.

OK, now the following optional CSS gives us a sticky footer: css html, body { height: 100%; } body { display: flex; flex-direction: column; } .not-footer { flex: 1 0 auto; } footer { flex-shrink: 0; }