github
html_url | issue_url | id | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154475454 | https://api.github.com/repos/simonw/sqlite-utils/issues/440 | 1154475454 | IC_kwDOCGYnMM5Ez-W- | 9599 | 2022-06-13T21:52:03Z | 2022-06-13T21:52:03Z | OWNER | The exception will be called `RowError`. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1250629388 | |
https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154474482 | https://api.github.com/repos/simonw/sqlite-utils/issues/440 | 1154474482 | IC_kwDOCGYnMM5Ez-Hy | 9599 | 2022-06-13T21:50:59Z | 2022-06-13T21:51:24Z | OWNER | Decision: I'm going to default to raising an exception if a row has too many values in it. You'll be able to pass `ignore_extras=True` to ignore those extra values, or pass `restkey="the_rest"` to stick them in a list in the `restkey` column. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1250629388 | |
https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154457893 | https://api.github.com/repos/simonw/sqlite-utils/issues/440 | 1154457893 | IC_kwDOCGYnMM5Ez6El | 9599 | 2022-06-13T21:29:02Z | 2022-06-13T21:29:02Z | OWNER | Here's the current function signature for `rows_from_file()`: https://github.com/simonw/sqlite-utils/blob/26e6d2622c57460a24ffdd0128bbaac051d51a5f/sqlite_utils/utils.py#L174-L179 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1250629388 | |
https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154457028 | https://api.github.com/repos/simonw/sqlite-utils/issues/440 | 1154457028 | IC_kwDOCGYnMM5Ez53E | 9599 | 2022-06-13T21:28:03Z | 2022-06-13T21:28:03Z | OWNER | Whatever I decide, I can implement it in `rows_from_file()`, maybe as an optional parameter - then decide how to call it from the `sqlite-utils insert` CLI (perhaps with a new option there too). | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1250629388 | |
https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154456183 | https://api.github.com/repos/simonw/sqlite-utils/issues/440 | 1154456183 | IC_kwDOCGYnMM5Ez5p3 | 9599 | 2022-06-13T21:26:55Z | 2022-06-13T21:26:55Z | OWNER | So I need to make a design decision here: what should `sqlite-utils` do with CSV files that have rows with more values than there are headings? Some options: - Ignore those extra fields entirely - silently drop that data. I'm not keen on this. - Throw an error. The library does this already, but the error is incomprehensible - it could turn into a useful, human-readable error instead. - Put the data in a JSON list in a column with a known name (`None` is not a valid column name, so not that). This could be something like `_restkey` or `_values_with_no_heading`. This feels like a better option, but I'd need to carefully pick a name for it - and come up with an answer for the question of what to do if the CSV file being important already uses that heading name for something else. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1250629388 | |
https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154454127 | https://api.github.com/repos/simonw/sqlite-utils/issues/440 | 1154454127 | IC_kwDOCGYnMM5Ez5Jv | 9599 | 2022-06-13T21:24:18Z | 2022-06-13T21:24:18Z | OWNER | That weird behaviour is documented here: https://docs.python.org/3/library/csv.html#csv.DictReader > If a row has more fields than fieldnames, the remaining data is put in a list and stored with the fieldname specified by *restkey* (which defaults to `None`). If a non-blank row has fewer fields than fieldnames, the missing values are filled-in with the value of *restval* (which defaults to `None`). | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1250629388 | |
https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154453319 | https://api.github.com/repos/simonw/sqlite-utils/issues/440 | 1154453319 | IC_kwDOCGYnMM5Ez49H | 9599 | 2022-06-13T21:23:16Z | 2022-06-13T21:23:16Z | OWNER | Aha! I think I see what's happening here. Here's what `DictReader` does if one of the lines has too many items in it: ```pycon >>> import csv, io >>> list(csv.DictReader(io.StringIO("id,name\n1,Cleo,nohead\n2,Barry"))) [{'id': '1', 'name': 'Cleo', None: ['nohead']}, {'id': '2', 'name': 'Barry'}] ``` See how that row with too many items gets this: `[{'id': '1', 'name': 'Cleo', None: ['nohead']}` That's a `None` for the key and (weirdly) a list containing the single item for the value! | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1250629388 | |
https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154449442 | https://api.github.com/repos/simonw/sqlite-utils/issues/440 | 1154449442 | IC_kwDOCGYnMM5Ez4Ai | 9599 | 2022-06-13T21:18:26Z | 2022-06-13T21:20:12Z | OWNER | Here are full steps to replicate the bug: ```python from urllib.request import urlopen import sqlite_utils db = sqlite_utils.Database(memory=True) with urlopen("https://artsdatabanken.no/Fab2018/api/export/csv") as fab: reader, other = sqlite_utils.utils.rows_from_file(fab, encoding="utf-16le") db["fab2018"].insert_all(reader, pk="Id") ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1250629388 | |
https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154396400 | https://api.github.com/repos/simonw/sqlite-utils/issues/440 | 1154396400 | IC_kwDOCGYnMM5EzrDw | 9599 | 2022-06-13T20:28:25Z | 2022-06-13T20:28:25Z | OWNER | Fixing that `key` thing (to ignore any key that is `None`) revealed a new bug: ``` File ~/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py:376, in hash_record(record, keys) 373 if keys is not None: 374 to_hash = {key: record[key] for key in keys} 375 return hashlib.sha1( --> 376 json.dumps(to_hash, separators=(",", ":"), sort_keys=True, default=repr).encode( 377 "utf8" 378 ) 379 ).hexdigest() File ~/.pyenv/versions/3.8.2/lib/python3.8/json/__init__.py:234, in dumps(obj, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, default, sort_keys, **kw) 232 if cls is None: 233 cls = JSONEncoder --> 234 return cls( 235 skipkeys=skipkeys, ensure_ascii=ensure_ascii, 236 check_circular=check_circular, allow_nan=allow_nan, indent=indent, 237 separators=separators, default=default, sort_keys=sort_keys, 238 **kw).encode(obj) File ~/.pyenv/versions/3.8.2/lib/python3.8/json/encoder.py:199, in JSONEncoder.encode(self, o) 195 return encode_basestring(o) 196 # This doesn't pass the iterator directly to ''.join() because the 197 # exceptions aren't as detailed. The list call should be roughly 198 # equivalent to the PySequence_Fast that ''.join() would do. --> 199 chunks = self.iterencode(o, _one_shot=True) 200 if not isinstance(chunks, (list, tuple)): 201 chunks = list(chunks) File ~/.pyenv/versions/3.8.2/lib/python3.8/json/encoder.py:257, in JSONEncoder.iterencode(self, o, _one_shot) 252 else: 253 _iterencode = _make_iterencode( 254 markers, self.default, _encoder, self.indent, floatstr, 255 self.key_separator, self.item_separator, self.sort_keys, 256 self.skipkeys, _one_shot) --> 257 return _iterencode(o, 0) TypeError: '<' not supported between instances of 'NoneType' and 'str' ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1250629388 | |
https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154387591 | https://api.github.com/repos/simonw/sqlite-utils/issues/440 | 1154387591 | IC_kwDOCGYnMM5Ezo6H | 9599 | 2022-06-13T20:17:51Z | 2022-06-13T20:17:51Z | OWNER | I don't understand why that works but calling `insert_all()` does not. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1250629388 | |
https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154386795 | https://api.github.com/repos/simonw/sqlite-utils/issues/440 | 1154386795 | IC_kwDOCGYnMM5Ezotr | 9599 | 2022-06-13T20:16:53Z | 2022-06-13T20:16:53Z | OWNER | Steps to demonstrate that `sqlite-utils insert` is not affected: ```bash curl -o artsdatabanken.csv https://artsdatabanken.no/Fab2018/api/export/csv sqlite-utils insert arts.db artsdatabanken artsdatabanken.csv --sniff --csv --encoding utf-16le ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1250629388 | |
https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154385916 | https://api.github.com/repos/simonw/sqlite-utils/issues/440 | 1154385916 | IC_kwDOCGYnMM5Ezof8 | 9599 | 2022-06-13T20:15:49Z | 2022-06-13T20:15:49Z | OWNER | `rows_from_file()` isn't part of the documented API but maybe it should be! | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1250629388 |