Note that datasette-cluster-map also seems to be limited to 998 displayed points:
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811054000,
https://github.com/simonw/datasette/issues/283#issuecomment-781077127,https://api.github.com/repos/simonw/datasette/issues/283,781077127,MDEyOklzc3VlQ29tbWVudDc4MTA3NzEyNw==,9599,2021-02-18T05:56:30Z,2021-02-18T05:57:34Z,OWNER,I'm going to to try prototyping the `--crossdb` option that causes `/_memory` to connect to all databases as a starting point and see how well that works.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 1, ""eyes"": 0}",325958506,
https://github.com/simonw/datasette/issues/283#issuecomment-780991910,https://api.github.com/repos/simonw/datasette/issues/283,780991910,MDEyOklzc3VlQ29tbWVudDc4MDk5MTkxMA==,9308268,2021-02-18T02:13:56Z,2021-02-18T02:13:56Z,NONE,"I was going ask you about this issue when we talk during your office-hours schedule this Friday, but was there any support ever added for doing this cross-database joining?
I have a use-case where could be pretty neat to do analysis using this tool on time-specific databases from snapshots
https://ilsweb.cincinnatilibrary.org/collection-analysis/
and thanks again for such an amazing tool!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",325958506,
https://github.com/dogsheep/google-takeout-to-sqlite/issues/4#issuecomment-780817596,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4,780817596,MDEyOklzc3VlQ29tbWVudDc4MDgxNzU5Ng==,306240,2021-02-17T20:01:35Z,2021-02-17T20:01:35Z,NONE,I've got this almost working. Just needs some polish,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",778380836,
https://github.com/simonw/sqlite-utils/issues/227#issuecomment-779785638,https://api.github.com/repos/simonw/sqlite-utils/issues/227,779785638,MDEyOklzc3VlQ29tbWVudDc3OTc4NTYzOA==,295329,2021-02-16T11:48:03Z,2021-02-16T11:48:03Z,NONE,Thank you @simonw ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807174161,
https://github.com/simonw/datasette/issues/1226#issuecomment-779467451,https://api.github.com/repos/simonw/datasette/issues/1226,779467451,MDEyOklzc3VlQ29tbWVudDc3OTQ2NzQ1MQ==,9599,2021-02-15T22:02:46Z,2021-02-15T22:02:46Z,OWNER,"I'm OK with the current error message shown if you try to use too low a port:
```
datasette fivethirtyeight.db -p 800
INFO: Started server process [45511]
INFO: Waiting for application startup.
INFO: Application startup complete.
ERROR: [Errno 13] error while attempting to bind on address ('127.0.0.1', 800): permission denied
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",808843401,
https://github.com/simonw/datasette/issues/1226#issuecomment-779467160,https://api.github.com/repos/simonw/datasette/issues/1226,779467160,MDEyOklzc3VlQ29tbWVudDc3OTQ2NzE2MA==,9599,2021-02-15T22:01:53Z,2021-02-15T22:01:53Z,OWNER,"This check needs to happen in two places:
https://github.com/simonw/datasette/blob/9603d893b9b72653895318c9104d754229fdb146/datasette/cli.py#L222-L227
https://github.com/simonw/datasette/blob/9603d893b9b72653895318c9104d754229fdb146/datasette/cli.py#L328-L333","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",808843401,
https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779416619,https://api.github.com/repos/simonw/sqlite-utils/issues/147,779416619,MDEyOklzc3VlQ29tbWVudDc3OTQxNjYxOQ==,9599,2021-02-15T19:40:57Z,2021-02-15T21:27:55Z,OWNER,"Tried this experiment (not proper binary search, it only searches downwards):
```python
import sqlite3
db = sqlite3.connect("":memory:"")
def tryit(n):
sql = ""select 1 where 1 in ({})"".format("", "".join(""?"" for i in range(n)))
db.execute(sql, [0 for i in range(n)])
def find_limit(min=0, max=5_000_000):
value = max
while True:
print('Trying', value)
try:
tryit(value)
return value
except:
value = value // 2
```
Running `find_limit()` with those default parameters takes about 1.47s on my laptop:
```
In [9]: %timeit find_limit()
Trying 5000000
Trying 2500000...
1.47 s ± 28 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```
Interestingly the value it suggested was 156250 - suggesting that the macOS `sqlite3` binary with a 500,000 limit isn't the same as whatever my Python is using here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688670158,
https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779448912,https://api.github.com/repos/simonw/sqlite-utils/issues/147,779448912,MDEyOklzc3VlQ29tbWVudDc3OTQ0ODkxMg==,9599,2021-02-15T21:09:50Z,2021-02-15T21:09:50Z,OWNER,"I fiddled around and replaced that line with `batch_size = SQLITE_MAX_VARS // num_columns` - which evaluated to `10416` for this particular file. That got me this:
40.71s user 1.81s system 98% cpu 43.081 total
43s is definitely better than 56s, but it's still not as big as the ~26.5s to ~3.5s improvement described by @simonwiles at the top of this issue. I wonder what I'm missing here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688670158,
https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779446652,https://api.github.com/repos/simonw/sqlite-utils/issues/147,779446652,MDEyOklzc3VlQ29tbWVudDc3OTQ0NjY1Mg==,9599,2021-02-15T21:04:19Z,2021-02-15T21:04:19Z,OWNER,"... but it looks like `batch_size` is hard-coded to 100, rather than `None` - which means it's not being calculated using that value:
https://github.com/simonw/sqlite-utils/blob/1f49f32814a942fa076cfe5f504d1621188097ed/sqlite_utils/db.py#L704
And
https://github.com/simonw/sqlite-utils/blob/1f49f32814a942fa076cfe5f504d1621188097ed/sqlite_utils/db.py#L1877","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688670158,
https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779445423,https://api.github.com/repos/simonw/sqlite-utils/issues/147,779445423,MDEyOklzc3VlQ29tbWVudDc3OTQ0NTQyMw==,9599,2021-02-15T21:00:44Z,2021-02-15T21:01:09Z,OWNER,"I tried changing the hard-coded value from 999 to 156_250 and running `sqlite-utils insert` against a 500MB CSV file, with these results:
```
(sqlite-utils) sqlite-utils % time sqlite-utils insert slow-ethos.db ethos ../ethos-datasette/ethos.csv --no-headers
[###################################-] 99% 00:00:00sqlite-utils insert slow-ethos.db ethos ../ethos-datasette/ethos.csv
44.74s user 7.61s system 92% cpu 56.601 total
# Increased the setting here
(sqlite-utils) sqlite-utils % time sqlite-utils insert fast-ethos.db ethos ../ethos-datasette/ethos.csv --no-headers
[###################################-] 99% 00:00:00sqlite-utils insert fast-ethos.db ethos ../ethos-datasette/ethos.csv
39.40s user 5.15s system 96% cpu 46.320 total
```
Not as big a difference as I was expecting.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688670158,
https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779417723,https://api.github.com/repos/simonw/sqlite-utils/issues/147,779417723,MDEyOklzc3VlQ29tbWVudDc3OTQxNzcyMw==,9599,2021-02-15T19:44:02Z,2021-02-15T19:47:00Z,OWNER,"`%timeit find_limit(max=1_000_000)` took 378ms on my laptop
`%timeit find_limit(max=500_000)` took 197ms
`%timeit find_limit(max=200_000)` reported 53ms per loop
`%timeit find_limit(max=100_000)` reported 26.8ms per loop.
All of these are still slow enough that I'm not comfortable running this search for every time the library is imported. Allowing users to opt-in to this as a performance enhancement might be better.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688670158,
https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779409770,https://api.github.com/repos/simonw/sqlite-utils/issues/147,779409770,MDEyOklzc3VlQ29tbWVudDc3OTQwOTc3MA==,9599,2021-02-15T19:23:11Z,2021-02-15T19:23:11Z,OWNER,"On my Mac right now I'm seeing a limit of 500,000:
```
% sqlite3 -cmd "".limits variable_number""
variable_number 500000
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688670158,
https://github.com/simonw/sqlite-utils/issues/227#issuecomment-778854808,https://api.github.com/repos/simonw/sqlite-utils/issues/227,778854808,MDEyOklzc3VlQ29tbWVudDc3ODg1NDgwOA==,9599,2021-02-14T22:46:54Z,2021-02-14T22:46:54Z,OWNER,Fix is released in 3.5.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807174161,
https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778851721,https://api.github.com/repos/simonw/sqlite-utils/issues/228,778851721,MDEyOklzc3VlQ29tbWVudDc3ODg1MTcyMQ==,9599,2021-02-14T22:23:46Z,2021-02-14T22:23:46Z,OWNER,I called this `--no-headers` for consistency with the existing output option: https://github.com/simonw/sqlite-utils/blob/427dace184c7da57f4a04df07b1e84cdae3261e8/sqlite_utils/cli.py#L61-L64,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807437089,
https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778849394,https://api.github.com/repos/simonw/sqlite-utils/issues/228,778849394,MDEyOklzc3VlQ29tbWVudDc3ODg0OTM5NA==,9599,2021-02-14T22:06:53Z,2021-02-14T22:06:53Z,OWNER,"For the moment I think just adding `--no-header` - which causes column names ""unknown1,unknown2,..."" to be used - should be enough.
Users can import with that option, then use `sqlite-utils transform --rename` to rename them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807437089,
https://github.com/simonw/sqlite-utils/issues/229#issuecomment-778844016,https://api.github.com/repos/simonw/sqlite-utils/issues/229,778844016,MDEyOklzc3VlQ29tbWVudDc3ODg0NDAxNg==,9599,2021-02-14T21:22:45Z,2021-02-14T21:22:45Z,OWNER,"I'm going to use this pattern from https://stackoverflow.com/a/15063941
```python
import sys
import csv
maxInt = sys.maxsize
while True:
# decrease the maxInt value by factor 10
# as long as the OverflowError occurs.
try:
csv.field_size_limit(maxInt)
break
except OverflowError:
maxInt = int(maxInt/10)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807817197,
https://github.com/simonw/sqlite-utils/issues/229#issuecomment-778843503,https://api.github.com/repos/simonw/sqlite-utils/issues/229,778843503,MDEyOklzc3VlQ29tbWVudDc3ODg0MzUwMw==,9599,2021-02-14T21:18:51Z,2021-02-14T21:18:51Z,OWNER,"I want to set this to the maximum allowed limit, which seems to be surprisingly hard! That StackOverflow thread is full of ideas for that, many of them involving `ctypes`. I'm a bit loathe to add a dependency on `ctypes` though - even though it's in the Python standard library I worry that it might not be available on some architectures.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807817197,
https://github.com/simonw/sqlite-utils/issues/229#issuecomment-778843362,https://api.github.com/repos/simonw/sqlite-utils/issues/229,778843362,MDEyOklzc3VlQ29tbWVudDc3ODg0MzM2Mg==,9599,2021-02-14T21:17:53Z,2021-02-14T21:17:53Z,OWNER,Same issue as #227.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807817197,
https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778811746,https://api.github.com/repos/simonw/sqlite-utils/issues/228,778811746,MDEyOklzc3VlQ29tbWVudDc3ODgxMTc0Ng==,9599,2021-02-14T17:39:30Z,2021-02-14T21:16:54Z,OWNER,"I'm going to detach this from the #131 column types idea.
The three things I need to handle here are:
- The CSV file doesn't have a header row at all, so I need to specify what the column names should be
- The CSV file DOES have a header row but I want to ignore it and use alternative column names
- The CSV doesn't have a header row at all and I want to automatically use `unknown1,unknown2...` so I can start exploring it as quickly as possible.
Here's a potential design that covers the first two:
`--replace-header=""foo,bar,baz""` - ignore whatever is in the first row and pretend it was this instead
`--add-header=""foo,bar,baz""` - add a first row with these details, to use as the header
It doesn't cover the ""give me unknown column names"" case though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807437089,
https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778843086,https://api.github.com/repos/simonw/sqlite-utils/issues/228,778843086,MDEyOklzc3VlQ29tbWVudDc3ODg0MzA4Ng==,9599,2021-02-14T21:15:43Z,2021-02-14T21:15:43Z,OWNER,"I'm not convinced the `.has_header()` rules are useful for the kind of CSV files I work with: https://github.com/python/cpython/blob/63298930fb531ba2bb4f23bc3b915dbf1e17e9e1/Lib/csv.py#L383
```python
def has_header(self, sample):
# Creates a dictionary of types of data in each column. If any
# column is of a single type (say, integers), *except* for the first
# row, then the first row is presumed to be labels. If the type
# can't be determined, it is assumed to be a string in which case
# the length of the string is the determining factor: if all of the
# rows except for the first are the same length, it's a header.
# Finally, a 'vote' is taken at the end for each column, adding or
# subtracting from the likelihood of the first row being a header.
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807437089,
https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778842982,https://api.github.com/repos/simonw/sqlite-utils/issues/228,778842982,MDEyOklzc3VlQ29tbWVudDc3ODg0Mjk4Mg==,9599,2021-02-14T21:15:11Z,2021-02-14T21:15:11Z,OWNER,"Implementation tip: I have code that reads the first row and uses it as headers here: https://github.com/simonw/sqlite-utils/blob/8f042ae1fd323995d966a94e8e6df85cc843b938/sqlite_utils/cli.py#L689-L691
So If I want to use `unknown1,unknown2...` I can do that by reading the first row, counting the number of columns, generating headers based on that range and then continuing to build that generator (maybe with `itertools.chain()` to replay the record we already read).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807437089,
https://github.com/simonw/sqlite-utils/issues/227#issuecomment-778841704,https://api.github.com/repos/simonw/sqlite-utils/issues/227,778841704,MDEyOklzc3VlQ29tbWVudDc3ODg0MTcwNA==,9599,2021-02-14T21:05:20Z,2021-02-14T21:05:20Z,OWNER,This has also been reported in #229.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807174161,
https://github.com/simonw/sqlite-utils/pull/225#issuecomment-778841547,https://api.github.com/repos/simonw/sqlite-utils/issues/225,778841547,MDEyOklzc3VlQ29tbWVudDc3ODg0MTU0Nw==,9599,2021-02-14T21:04:13Z,2021-02-14T21:04:13Z,OWNER,I added a test and fixed this in #234 - thanks for the fix.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",797159961,
https://github.com/simonw/sqlite-utils/issues/234#issuecomment-778841278,https://api.github.com/repos/simonw/sqlite-utils/issues/234,778841278,MDEyOklzc3VlQ29tbWVudDc3ODg0MTI3OA==,9599,2021-02-14T21:02:11Z,2021-02-14T21:02:11Z,OWNER,"I managed to replicate this in a test:
```python
def test_insert_all_with_extra_columns_in_later_chunks(fresh_db):
chunk = [
{""record"": ""Record 1""},
{""record"": ""Record 2""},
{""record"": ""Record 3""},
{""record"": ""Record 4"", ""extra"": 1},
]
fresh_db[""t""].insert_all(chunk, batch_size=2, alter=True)
assert list(fresh_db[""t""].rows) == [
{""record"": ""Record 1"", ""extra"": None},
{""record"": ""Record 2"", ""extra"": None},
{""record"": ""Record 3"", ""extra"": None},
{""record"": ""Record 4"", ""extra"": 1},
]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",808046597,
https://github.com/simonw/sqlite-utils/pull/225#issuecomment-778834504,https://api.github.com/repos/simonw/sqlite-utils/issues/225,778834504,MDEyOklzc3VlQ29tbWVudDc3ODgzNDUwNA==,9599,2021-02-14T20:09:30Z,2021-02-14T20:09:30Z,OWNER,Thanks for this. I'm going to try and get the test suite to run in Windows on GitHub Actions.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",797159961,
https://github.com/simonw/sqlite-utils/issues/231#issuecomment-778829456,https://api.github.com/repos/simonw/sqlite-utils/issues/231,778829456,MDEyOklzc3VlQ29tbWVudDc3ODgyOTQ1Ng==,9599,2021-02-14T19:37:52Z,2021-02-14T19:37:52Z,OWNER,"I'm going to add `limit` and `offset` to the following methods:
- `rows_where()`
- `search_sql()`
- `search()`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",808028757,
https://github.com/simonw/sqlite-utils/issues/231#issuecomment-778828758,https://api.github.com/repos/simonw/sqlite-utils/issues/231,778828758,MDEyOklzc3VlQ29tbWVudDc3ODgyODc1OA==,9599,2021-02-14T19:33:14Z,2021-02-14T19:33:14Z,OWNER,The `limit=` parameter is currently only available on the `.search()` method - it would make sense to add this to other methods as well.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",808028757,
https://github.com/simonw/sqlite-utils/pull/224#issuecomment-778828495,https://api.github.com/repos/simonw/sqlite-utils/issues/224,778828495,MDEyOklzc3VlQ29tbWVudDc3ODgyODQ5NQ==,9599,2021-02-14T19:31:06Z,2021-02-14T19:31:06Z,OWNER,I'm going to add a `offset=` parameter to support this case. Thanks for the suggestion!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",792297010,
https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778827570,https://api.github.com/repos/simonw/sqlite-utils/issues/230,778827570,MDEyOklzc3VlQ29tbWVudDc3ODgyNzU3MA==,9599,2021-02-14T19:24:20Z,2021-02-14T19:24:20Z,OWNER,Here's the implementation in Python: https://github.com/python/cpython/blob/63298930fb531ba2bb4f23bc3b915dbf1e17e9e1/Lib/csv.py#L204-L225,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",808008305,
https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778824361,https://api.github.com/repos/simonw/sqlite-utils/issues/230,778824361,MDEyOklzc3VlQ29tbWVudDc3ODgyNDM2MQ==,9599,2021-02-14T18:59:22Z,2021-02-14T18:59:22Z,OWNER,"I think I've got it. I can use `io.BufferedReader()` to get an object I can run `.peek(2048)` on, then wrap THAT in `io.TextIOWrapper`:
```python
encoding = encoding or ""utf-8""
buffered = io.BufferedReader(json_file, buffer_size=4096)
decoded = io.TextIOWrapper(buffered, encoding=encoding, line_buffering=True)
if pk and len(pk) == 1:
pk = pk[0]
if csv or tsv:
if sniff:
# Read first 2048 bytes and use that to detect
first_bytes = buffered.peek(2048)
print('first_bytes', first_bytes)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",808008305,
https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778821403,https://api.github.com/repos/simonw/sqlite-utils/issues/230,778821403,MDEyOklzc3VlQ29tbWVudDc3ODgyMTQwMw==,9599,2021-02-14T18:38:16Z,2021-02-14T18:38:16Z,OWNER,"There are two code paths here that matter:
- For a regular file, can read the first 2048 bytes, then `.seek(0)` before continuing. That's easy.
- `stdin` is harder. I need to read and buffer the first 2048 bytes, then pass an object to `csv.reader()` which will replay that chunk and then play the rest of stdin.
I'm a bit stuck on the second one. Ideally I could use something like `itertools.chain()` but I can't find an alternative for file-like objects.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",808008305,
https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778818639,https://api.github.com/repos/simonw/sqlite-utils/issues/230,778818639,MDEyOklzc3VlQ29tbWVudDc3ODgxODYzOQ==,9599,2021-02-14T18:22:38Z,2021-02-14T18:22:38Z,OWNER,Maybe I shouldn't be using `StreamReader` at all - https://www.python.org/dev/peps/pep-0400/ suggests that it should be deprecated in favour of `io.TextIOWrapper`. I'm using `StreamReader` due to this line: https://github.com/simonw/sqlite-utils/blob/726219c3503e77440975cd15b74d006639feb0f8/sqlite_utils/cli.py#L667-L668,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",808008305,
https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778817494,https://api.github.com/repos/simonw/sqlite-utils/issues/230,778817494,MDEyOklzc3VlQ29tbWVudDc3ODgxNzQ5NA==,9599,2021-02-14T18:16:06Z,2021-02-14T18:16:06Z,OWNER,"Types involved:
```
(Pdb) type(json_file.raw)
This note includes two images.