issue_comments

So maybe this is a function which can either be told the format or, if none is provided, it detects one for itself. python def rows_from_file(fp, format=None): # ... yield from rows

python def detect_format(fp): # ... return "csv", fp, dialect # or return "json", fp, parsed_data # or return "json-nl", fp, docs The mixed return types here are ugly. In all of these cases what we really want is to return a generator of {...} objects. So maybe it returns that instead. python def filepointer_to_documents(fp): # ... yield from documents I can refactor sqlite-utils insert to use this new code too.

Or... since I'm not using a streaming JSON parser at the moment, if I think something is JSON I can load the entire thing into memory to validate it.

I still need to detect newline-delimited JSON. For that I can consume the first line of the input to see if it's a valid JSON object, then maybe sniff the second line too?

This does mean that if the input is a single line of GIANT JSON it will all be consumed into memory at once, but that's going to happen anyway.

So I need a function which, given a file pointer, consumes from it, detects the type, then returns that type AND a file pointer to the beginning of the file again. I can use io.BufferedReader for this.

Detecting valid JSON is tricky - just because a stream starts with [ or { doesn't mean the entire stream is valid JSON. You need to parse the entire stream to determine that for sure.

One way to solve this would be with a custom state machine. Another would be to use the ijson streaming parser - annoyingly it throws the same exception class for invalid JSON for different reasons, but the e.args[0] for that exception includes human-readable text about the error - if it's anything other than parse error: premature EOF then it probably means the JSON was invalid.

To detect JSON, check to see if the stream starts with [ or { - maybe do something more sophisticated than that.