{"html_url": "https://github.com/simonw/sqlite-utils/issues/182#issuecomment-710258736", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/182", "id": 710258736, "node_id": "MDEyOklzc3VlQ29tbWVudDcxMDI1ODczNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-16T17:20:41Z", "updated_at": "2020-10-16T17:20:41Z", "author_association": "OWNER", "body": "Documentation: https://sqlite-utils.readthedocs.io/en/latest/cli.html#inserting-csv-or-tsv-data", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 711649325, "label": "Better handling of encodings other than utf-8 for \"sqlite-utils insert\""}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/182#issuecomment-710178871", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/182", "id": 710178871, "node_id": "MDEyOklzc3VlQ29tbWVudDcxMDE3ODg3MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-16T16:27:39Z", "updated_at": "2020-10-16T16:28:14Z", "author_association": "OWNER", "body": "The file is opened for me by `click.File()`, which also handles things like `-` for stdin. But i neee to be able to switch the encoding used to read from that based on the `--encoding` option.\r\n\r\nI think the way to do that is to open the file in binary mode and then wrap it in a codec reader:\r\n```python\r\nfp = codecs.getreader(encoding)(fp)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 711649325, "label": "Better handling of encodings other than utf-8 for \"sqlite-utils insert\""}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/182#issuecomment-708695761", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/182", "id": 708695761, "node_id": "MDEyOklzc3VlQ29tbWVudDcwODY5NTc2MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-14T22:33:55Z", "updated_at": "2020-10-14T22:35:17Z", "author_association": "OWNER", "body": "For the moment I'm going to add a `--encoding` option and some code that catches `UnicodeDecodeError` and shows an error message that suggests using `--encoding`.\r\n\r\nThat error message could detect if the `file` command is available and, if it is, suggest running `file filename.txt` to detect the character encoding.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 711649325, "label": "Better handling of encodings other than utf-8 for \"sqlite-utils insert\""}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/182#issuecomment-708693133", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/182", "id": 708693133, "node_id": "MDEyOklzc3VlQ29tbWVudDcwODY5MzEzMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-14T22:26:26Z", "updated_at": "2020-10-14T22:26:26Z", "author_association": "OWNER", "body": "I could use https://github.com/chardet/chardet to help here, though I'd rather not add it as a dependency (`sqlite-utils` has very few dependencies at the moment). I could add it as an optional dependency though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 711649325, "label": "Better handling of encodings other than utf-8 for \"sqlite-utils insert\""}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/182#issuecomment-701626134", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/182", "id": 701626134, "node_id": "MDEyOklzc3VlQ29tbWVudDcwMTYyNjEzNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-30T20:27:09Z", "updated_at": "2020-09-30T20:27:42Z", "author_association": "OWNER", "body": "It looks like http://maps.natalian.org/data.txt is encoded as `latin-1`, but `sqlite-utils` assumes `utf-8` and hence breaks.\r\n\r\nIt would be worth improving the error message here. I could also add a `--encoding latin-1` option to `sqlite-utils insert` to help in consuming files that are stored in charsets other than `utf-8`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 711649325, "label": "Better handling of encodings other than utf-8 for \"sqlite-utils insert\""}, "performed_via_github_app": null}