github
html_url | issue_url | id | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864208476 | https://api.github.com/repos/simonw/sqlite-utils/issues/279 | 864208476 | MDEyOklzc3VlQ29tbWVudDg2NDIwODQ3Ng== | 9599 | 2021-06-18T18:30:08Z | 2021-06-18T23:30:19Z | OWNER | So maybe this is a function which can either be told the format or, if none is provided, it detects one for itself. ```python def rows_from_file(fp, format=None): # ... yield from rows ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
924990677 | |
https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864207841 | https://api.github.com/repos/simonw/sqlite-utils/issues/279 | 864207841 | MDEyOklzc3VlQ29tbWVudDg2NDIwNzg0MQ== | 9599 | 2021-06-18T18:28:40Z | 2021-06-18T18:28:46Z | OWNER | ```python def detect_format(fp): # ... return "csv", fp, dialect # or return "json", fp, parsed_data # or return "json-nl", fp, docs ``` The mixed return types here are ugly. In all of these cases what we really want is to return a generator of `{...}` objects. So maybe it returns that instead. ```python def filepointer_to_documents(fp): # ... yield from documents ``` I can refactor `sqlite-utils insert` to use this new code too. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
924990677 | |
https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864206308 | https://api.github.com/repos/simonw/sqlite-utils/issues/279 | 864206308 | MDEyOklzc3VlQ29tbWVudDg2NDIwNjMwOA== | 9599 | 2021-06-18T18:25:04Z | 2021-06-18T18:25:04Z | OWNER | Or... since I'm not using a streaming JSON parser at the moment, if I think something is JSON I can load the entire thing into memory to validate it. I still need to detect newline-delimited JSON. For that I can consume the first line of the input to see if it's a valid JSON object, then maybe sniff the second line too? This does mean that if the input is a single line of GIANT JSON it will all be consumed into memory at once, but that's going to happen anyway. So I need a function which, given a file pointer, consumes from it, detects the type, then returns that type AND a file pointer to the beginning of the file again. I can use `io.BufferedReader` for this. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
924990677 | |
https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864129273 | https://api.github.com/repos/simonw/sqlite-utils/issues/279 | 864129273 | MDEyOklzc3VlQ29tbWVudDg2NDEyOTI3Mw== | 9599 | 2021-06-18T15:47:47Z | 2021-06-18T15:47:47Z | OWNER | Detecting valid JSON is tricky - just because a stream starts with `[` or `{` doesn't mean the entire stream is valid JSON. You need to parse the entire stream to determine that for sure. One way to solve this would be with a custom state machine. Another would be to use the `ijson` streaming parser - annoyingly it throws the same exception class for invalid JSON for different reasons, but the `e.args[0]` for that exception includes human-readable text about the error - if it's anything other than `parse error: premature EOF` then it probably means the JSON was invalid. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
924990677 | |
https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864103005 | https://api.github.com/repos/simonw/sqlite-utils/issues/279 | 864103005 | MDEyOklzc3VlQ29tbWVudDg2NDEwMzAwNQ== | 9599 | 2021-06-18T15:04:15Z | 2021-06-18T15:04:15Z | OWNER | To detect JSON, check to see if the stream starts with `[` or `{` - maybe do something more sophisticated than that. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
924990677 |