github
html_url | issue_url | id | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
https://github.com/simonw/datasette/issues/283#issuecomment-781077127 | https://api.github.com/repos/simonw/datasette/issues/283 | 781077127 | MDEyOklzc3VlQ29tbWVudDc4MTA3NzEyNw== | 9599 | 2021-02-18T05:56:30Z | 2021-02-18T05:57:34Z | OWNER | I'm going to to try prototyping the `--crossdb` option that causes `/_memory` to connect to all databases as a starting point and see how well that works. | { "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 1, "eyes": 0 } |
325958506 | |
https://github.com/simonw/datasette/issues/1226#issuecomment-779467451 | https://api.github.com/repos/simonw/datasette/issues/1226 | 779467451 | MDEyOklzc3VlQ29tbWVudDc3OTQ2NzQ1MQ== | 9599 | 2021-02-15T22:02:46Z | 2021-02-15T22:02:46Z | OWNER | I'm OK with the current error message shown if you try to use too low a port: ``` datasette fivethirtyeight.db -p 800 INFO: Started server process [45511] INFO: Waiting for application startup. INFO: Application startup complete. ERROR: [Errno 13] error while attempting to bind on address ('127.0.0.1', 800): permission denied INFO: Waiting for application shutdown. INFO: Application shutdown complete. ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
808843401 | |
https://github.com/simonw/datasette/issues/1226#issuecomment-779467160 | https://api.github.com/repos/simonw/datasette/issues/1226 | 779467160 | MDEyOklzc3VlQ29tbWVudDc3OTQ2NzE2MA== | 9599 | 2021-02-15T22:01:53Z | 2021-02-15T22:01:53Z | OWNER | This check needs to happen in two places: https://github.com/simonw/datasette/blob/9603d893b9b72653895318c9104d754229fdb146/datasette/cli.py#L222-L227 https://github.com/simonw/datasette/blob/9603d893b9b72653895318c9104d754229fdb146/datasette/cli.py#L328-L333 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
808843401 | |
https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779416619 | https://api.github.com/repos/simonw/sqlite-utils/issues/147 | 779416619 | MDEyOklzc3VlQ29tbWVudDc3OTQxNjYxOQ== | 9599 | 2021-02-15T19:40:57Z | 2021-02-15T21:27:55Z | OWNER | Tried this experiment (not proper binary search, it only searches downwards): ```python import sqlite3 db = sqlite3.connect(":memory:") def tryit(n): sql = "select 1 where 1 in ({})".format(", ".join("?" for i in range(n))) db.execute(sql, [0 for i in range(n)]) def find_limit(min=0, max=5_000_000): value = max while True: print('Trying', value) try: tryit(value) return value except: value = value // 2 ``` Running `find_limit()` with those default parameters takes about 1.47s on my laptop: ``` In [9]: %timeit find_limit() Trying 5000000 Trying 2500000... 1.47 s ± 28 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) ``` Interestingly the value it suggested was 156250 - suggesting that the macOS `sqlite3` binary with a 500,000 limit isn't the same as whatever my Python is using here. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
688670158 | |
https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779448912 | https://api.github.com/repos/simonw/sqlite-utils/issues/147 | 779448912 | MDEyOklzc3VlQ29tbWVudDc3OTQ0ODkxMg== | 9599 | 2021-02-15T21:09:50Z | 2021-02-15T21:09:50Z | OWNER | I fiddled around and replaced that line with `batch_size = SQLITE_MAX_VARS // num_columns` - which evaluated to `10416` for this particular file. That got me this: 40.71s user 1.81s system 98% cpu 43.081 total 43s is definitely better than 56s, but it's still not as big as the ~26.5s to ~3.5s improvement described by @simonwiles at the top of this issue. I wonder what I'm missing here. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
688670158 | |
https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779446652 | https://api.github.com/repos/simonw/sqlite-utils/issues/147 | 779446652 | MDEyOklzc3VlQ29tbWVudDc3OTQ0NjY1Mg== | 9599 | 2021-02-15T21:04:19Z | 2021-02-15T21:04:19Z | OWNER | ... but it looks like `batch_size` is hard-coded to 100, rather than `None` - which means it's not being calculated using that value: https://github.com/simonw/sqlite-utils/blob/1f49f32814a942fa076cfe5f504d1621188097ed/sqlite_utils/db.py#L704 And https://github.com/simonw/sqlite-utils/blob/1f49f32814a942fa076cfe5f504d1621188097ed/sqlite_utils/db.py#L1877 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
688670158 | |
https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779445423 | https://api.github.com/repos/simonw/sqlite-utils/issues/147 | 779445423 | MDEyOklzc3VlQ29tbWVudDc3OTQ0NTQyMw== | 9599 | 2021-02-15T21:00:44Z | 2021-02-15T21:01:09Z | OWNER | I tried changing the hard-coded value from 999 to 156_250 and running `sqlite-utils insert` against a 500MB CSV file, with these results: ``` (sqlite-utils) sqlite-utils % time sqlite-utils insert slow-ethos.db ethos ../ethos-datasette/ethos.csv --no-headers [###################################-] 99% 00:00:00sqlite-utils insert slow-ethos.db ethos ../ethos-datasette/ethos.csv 44.74s user 7.61s system 92% cpu 56.601 total # Increased the setting here (sqlite-utils) sqlite-utils % time sqlite-utils insert fast-ethos.db ethos ../ethos-datasette/ethos.csv --no-headers [###################################-] 99% 00:00:00sqlite-utils insert fast-ethos.db ethos ../ethos-datasette/ethos.csv 39.40s user 5.15s system 96% cpu 46.320 total ``` Not as big a difference as I was expecting. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
688670158 | |
https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779417723 | https://api.github.com/repos/simonw/sqlite-utils/issues/147 | 779417723 | MDEyOklzc3VlQ29tbWVudDc3OTQxNzcyMw== | 9599 | 2021-02-15T19:44:02Z | 2021-02-15T19:47:00Z | OWNER | `%timeit find_limit(max=1_000_000)` took 378ms on my laptop `%timeit find_limit(max=500_000)` took 197ms `%timeit find_limit(max=200_000)` reported 53ms per loop `%timeit find_limit(max=100_000)` reported 26.8ms per loop. All of these are still slow enough that I'm not comfortable running this search for every time the library is imported. Allowing users to opt-in to this as a performance enhancement might be better. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
688670158 | |
https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779409770 | https://api.github.com/repos/simonw/sqlite-utils/issues/147 | 779409770 | MDEyOklzc3VlQ29tbWVudDc3OTQwOTc3MA== | 9599 | 2021-02-15T19:23:11Z | 2021-02-15T19:23:11Z | OWNER | On my Mac right now I'm seeing a limit of 500,000: ``` % sqlite3 -cmd ".limits variable_number" variable_number 500000 ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
688670158 | |
https://github.com/simonw/sqlite-utils/issues/227#issuecomment-778854808 | https://api.github.com/repos/simonw/sqlite-utils/issues/227 | 778854808 | MDEyOklzc3VlQ29tbWVudDc3ODg1NDgwOA== | 9599 | 2021-02-14T22:46:54Z | 2021-02-14T22:46:54Z | OWNER | Fix is released in 3.5. | { "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 1, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
807174161 | |
https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778851721 | https://api.github.com/repos/simonw/sqlite-utils/issues/228 | 778851721 | MDEyOklzc3VlQ29tbWVudDc3ODg1MTcyMQ== | 9599 | 2021-02-14T22:23:46Z | 2021-02-14T22:23:46Z | OWNER | I called this `--no-headers` for consistency with the existing output option: https://github.com/simonw/sqlite-utils/blob/427dace184c7da57f4a04df07b1e84cdae3261e8/sqlite_utils/cli.py#L61-L64 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
807437089 | |
https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778849394 | https://api.github.com/repos/simonw/sqlite-utils/issues/228 | 778849394 | MDEyOklzc3VlQ29tbWVudDc3ODg0OTM5NA== | 9599 | 2021-02-14T22:06:53Z | 2021-02-14T22:06:53Z | OWNER | For the moment I think just adding `--no-header` - which causes column names "unknown1,unknown2,..." to be used - should be enough. Users can import with that option, then use `sqlite-utils transform --rename` to rename them. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
807437089 | |
https://github.com/simonw/sqlite-utils/issues/229#issuecomment-778844016 | https://api.github.com/repos/simonw/sqlite-utils/issues/229 | 778844016 | MDEyOklzc3VlQ29tbWVudDc3ODg0NDAxNg== | 9599 | 2021-02-14T21:22:45Z | 2021-02-14T21:22:45Z | OWNER | I'm going to use this pattern from https://stackoverflow.com/a/15063941 ```python import sys import csv maxInt = sys.maxsize while True: # decrease the maxInt value by factor 10 # as long as the OverflowError occurs. try: csv.field_size_limit(maxInt) break except OverflowError: maxInt = int(maxInt/10) ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
807817197 | |
https://github.com/simonw/sqlite-utils/issues/229#issuecomment-778843503 | https://api.github.com/repos/simonw/sqlite-utils/issues/229 | 778843503 | MDEyOklzc3VlQ29tbWVudDc3ODg0MzUwMw== | 9599 | 2021-02-14T21:18:51Z | 2021-02-14T21:18:51Z | OWNER | I want to set this to the maximum allowed limit, which seems to be surprisingly hard! That StackOverflow thread is full of ideas for that, many of them involving `ctypes`. I'm a bit loathe to add a dependency on `ctypes` though - even though it's in the Python standard library I worry that it might not be available on some architectures. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
807817197 | |
https://github.com/simonw/sqlite-utils/issues/229#issuecomment-778843362 | https://api.github.com/repos/simonw/sqlite-utils/issues/229 | 778843362 | MDEyOklzc3VlQ29tbWVudDc3ODg0MzM2Mg== | 9599 | 2021-02-14T21:17:53Z | 2021-02-14T21:17:53Z | OWNER | Same issue as #227. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
807817197 | |
https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778811746 | https://api.github.com/repos/simonw/sqlite-utils/issues/228 | 778811746 | MDEyOklzc3VlQ29tbWVudDc3ODgxMTc0Ng== | 9599 | 2021-02-14T17:39:30Z | 2021-02-14T21:16:54Z | OWNER | I'm going to detach this from the #131 column types idea. The three things I need to handle here are: - The CSV file doesn't have a header row at all, so I need to specify what the column names should be - The CSV file DOES have a header row but I want to ignore it and use alternative column names - The CSV doesn't have a header row at all and I want to automatically use `unknown1,unknown2...` so I can start exploring it as quickly as possible. Here's a potential design that covers the first two: `--replace-header="foo,bar,baz"` - ignore whatever is in the first row and pretend it was this instead `--add-header="foo,bar,baz"` - add a first row with these details, to use as the header It doesn't cover the "give me unknown column names" case though. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
807437089 | |
https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778843086 | https://api.github.com/repos/simonw/sqlite-utils/issues/228 | 778843086 | MDEyOklzc3VlQ29tbWVudDc3ODg0MzA4Ng== | 9599 | 2021-02-14T21:15:43Z | 2021-02-14T21:15:43Z | OWNER | I'm not convinced the `.has_header()` rules are useful for the kind of CSV files I work with: https://github.com/python/cpython/blob/63298930fb531ba2bb4f23bc3b915dbf1e17e9e1/Lib/csv.py#L383 ```python def has_header(self, sample): # Creates a dictionary of types of data in each column. If any # column is of a single type (say, integers), *except* for the first # row, then the first row is presumed to be labels. If the type # can't be determined, it is assumed to be a string in which case # the length of the string is the determining factor: if all of the # rows except for the first are the same length, it's a header. # Finally, a 'vote' is taken at the end for each column, adding or # subtracting from the likelihood of the first row being a header. ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
807437089 | |
https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778842982 | https://api.github.com/repos/simonw/sqlite-utils/issues/228 | 778842982 | MDEyOklzc3VlQ29tbWVudDc3ODg0Mjk4Mg== | 9599 | 2021-02-14T21:15:11Z | 2021-02-14T21:15:11Z | OWNER | Implementation tip: I have code that reads the first row and uses it as headers here: https://github.com/simonw/sqlite-utils/blob/8f042ae1fd323995d966a94e8e6df85cc843b938/sqlite_utils/cli.py#L689-L691 So If I want to use `unknown1,unknown2...` I can do that by reading the first row, counting the number of columns, generating headers based on that range and then continuing to build that generator (maybe with `itertools.chain()` to replay the record we already read). | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
807437089 | |
https://github.com/simonw/sqlite-utils/issues/227#issuecomment-778841704 | https://api.github.com/repos/simonw/sqlite-utils/issues/227 | 778841704 | MDEyOklzc3VlQ29tbWVudDc3ODg0MTcwNA== | 9599 | 2021-02-14T21:05:20Z | 2021-02-14T21:05:20Z | OWNER | This has also been reported in #229. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
807174161 | |
https://github.com/simonw/sqlite-utils/pull/225#issuecomment-778841547 | https://api.github.com/repos/simonw/sqlite-utils/issues/225 | 778841547 | MDEyOklzc3VlQ29tbWVudDc3ODg0MTU0Nw== | 9599 | 2021-02-14T21:04:13Z | 2021-02-14T21:04:13Z | OWNER | I added a test and fixed this in #234 - thanks for the fix. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
797159961 | |
https://github.com/simonw/sqlite-utils/issues/234#issuecomment-778841278 | https://api.github.com/repos/simonw/sqlite-utils/issues/234 | 778841278 | MDEyOklzc3VlQ29tbWVudDc3ODg0MTI3OA== | 9599 | 2021-02-14T21:02:11Z | 2021-02-14T21:02:11Z | OWNER | I managed to replicate this in a test: ```python def test_insert_all_with_extra_columns_in_later_chunks(fresh_db): chunk = [ {"record": "Record 1"}, {"record": "Record 2"}, {"record": "Record 3"}, {"record": "Record 4", "extra": 1}, ] fresh_db["t"].insert_all(chunk, batch_size=2, alter=True) assert list(fresh_db["t"].rows) == [ {"record": "Record 1", "extra": None}, {"record": "Record 2", "extra": None}, {"record": "Record 3", "extra": None}, {"record": "Record 4", "extra": 1}, ] ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
808046597 | |
https://github.com/simonw/sqlite-utils/pull/225#issuecomment-778834504 | https://api.github.com/repos/simonw/sqlite-utils/issues/225 | 778834504 | MDEyOklzc3VlQ29tbWVudDc3ODgzNDUwNA== | 9599 | 2021-02-14T20:09:30Z | 2021-02-14T20:09:30Z | OWNER | Thanks for this. I'm going to try and get the test suite to run in Windows on GitHub Actions. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
797159961 | |
https://github.com/simonw/sqlite-utils/issues/231#issuecomment-778829456 | https://api.github.com/repos/simonw/sqlite-utils/issues/231 | 778829456 | MDEyOklzc3VlQ29tbWVudDc3ODgyOTQ1Ng== | 9599 | 2021-02-14T19:37:52Z | 2021-02-14T19:37:52Z | OWNER | I'm going to add `limit` and `offset` to the following methods: - `rows_where()` - `search_sql()` - `search()` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
808028757 | |
https://github.com/simonw/sqlite-utils/issues/231#issuecomment-778828758 | https://api.github.com/repos/simonw/sqlite-utils/issues/231 | 778828758 | MDEyOklzc3VlQ29tbWVudDc3ODgyODc1OA== | 9599 | 2021-02-14T19:33:14Z | 2021-02-14T19:33:14Z | OWNER | The `limit=` parameter is currently only available on the `.search()` method - it would make sense to add this to other methods as well. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
808028757 | |
https://github.com/simonw/sqlite-utils/pull/224#issuecomment-778828495 | https://api.github.com/repos/simonw/sqlite-utils/issues/224 | 778828495 | MDEyOklzc3VlQ29tbWVudDc3ODgyODQ5NQ== | 9599 | 2021-02-14T19:31:06Z | 2021-02-14T19:31:06Z | OWNER | I'm going to add a `offset=` parameter to support this case. Thanks for the suggestion! | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
792297010 | |
https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778827570 | https://api.github.com/repos/simonw/sqlite-utils/issues/230 | 778827570 | MDEyOklzc3VlQ29tbWVudDc3ODgyNzU3MA== | 9599 | 2021-02-14T19:24:20Z | 2021-02-14T19:24:20Z | OWNER | Here's the implementation in Python: https://github.com/python/cpython/blob/63298930fb531ba2bb4f23bc3b915dbf1e17e9e1/Lib/csv.py#L204-L225 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
808008305 | |
https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778824361 | https://api.github.com/repos/simonw/sqlite-utils/issues/230 | 778824361 | MDEyOklzc3VlQ29tbWVudDc3ODgyNDM2MQ== | 9599 | 2021-02-14T18:59:22Z | 2021-02-14T18:59:22Z | OWNER | I think I've got it. I can use `io.BufferedReader()` to get an object I can run `.peek(2048)` on, then wrap THAT in `io.TextIOWrapper`: ```python encoding = encoding or "utf-8" buffered = io.BufferedReader(json_file, buffer_size=4096) decoded = io.TextIOWrapper(buffered, encoding=encoding, line_buffering=True) if pk and len(pk) == 1: pk = pk[0] if csv or tsv: if sniff: # Read first 2048 bytes and use that to detect first_bytes = buffered.peek(2048) print('first_bytes', first_bytes) ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
808008305 | |
https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778821403 | https://api.github.com/repos/simonw/sqlite-utils/issues/230 | 778821403 | MDEyOklzc3VlQ29tbWVudDc3ODgyMTQwMw== | 9599 | 2021-02-14T18:38:16Z | 2021-02-14T18:38:16Z | OWNER | There are two code paths here that matter: - For a regular file, can read the first 2048 bytes, then `.seek(0)` before continuing. That's easy. - `stdin` is harder. I need to read and buffer the first 2048 bytes, then pass an object to `csv.reader()` which will replay that chunk and then play the rest of stdin. I'm a bit stuck on the second one. Ideally I could use something like `itertools.chain()` but I can't find an alternative for file-like objects. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
808008305 | |
https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778818639 | https://api.github.com/repos/simonw/sqlite-utils/issues/230 | 778818639 | MDEyOklzc3VlQ29tbWVudDc3ODgxODYzOQ== | 9599 | 2021-02-14T18:22:38Z | 2021-02-14T18:22:38Z | OWNER | Maybe I shouldn't be using `StreamReader` at all - https://www.python.org/dev/peps/pep-0400/ suggests that it should be deprecated in favour of `io.TextIOWrapper`. I'm using `StreamReader` due to this line: https://github.com/simonw/sqlite-utils/blob/726219c3503e77440975cd15b74d006639feb0f8/sqlite_utils/cli.py#L667-L668 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
808008305 | |
https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778817494 | https://api.github.com/repos/simonw/sqlite-utils/issues/230 | 778817494 | MDEyOklzc3VlQ29tbWVudDc3ODgxNzQ5NA== | 9599 | 2021-02-14T18:16:06Z | 2021-02-14T18:16:06Z | OWNER | Types involved: ``` (Pdb) type(json_file.raw) <class '_io.FileIO'> (Pdb) type(json_file) <class 'encodings.utf_8.StreamReader'> ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
808008305 | |
https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778816333 | https://api.github.com/repos/simonw/sqlite-utils/issues/230 | 778816333 | MDEyOklzc3VlQ29tbWVudDc3ODgxNjMzMw== | 9599 | 2021-02-14T18:08:44Z | 2021-02-14T18:08:44Z | OWNER | No, you can't `.seek(0)` on stdin: ``` File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py", line 678, in insert_upsert_implementation json_file.raw.seek(0) OSError: [Errno 29] Illegal seek ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
808008305 | |
https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778815740 | https://api.github.com/repos/simonw/sqlite-utils/issues/230 | 778815740 | MDEyOklzc3VlQ29tbWVudDc3ODgxNTc0MA== | 9599 | 2021-02-14T18:05:03Z | 2021-02-14T18:05:03Z | OWNER | The challenge here is how to read the first 2048 bytes and then reset the incoming file. The Python docs example looks like this: ```python with open('example.csv', newline='') as csvfile: dialect = csv.Sniffer().sniff(csvfile.read(1024)) csvfile.seek(0) reader = csv.reader(csvfile, dialect) ``` Here's the relevant code in `sqlite-utils`: https://github.com/simonw/sqlite-utils/blob/726219c3503e77440975cd15b74d006639feb0f8/sqlite_utils/cli.py#L671-L679 The challenge is going to be having the `--sniff` option work with the progress bar. Here's how `file_progress()` works: https://github.com/simonw/sqlite-utils/blob/726219c3503e77440975cd15b74d006639feb0f8/sqlite_utils/utils.py#L106-L113 If `file.raw` is `stdin` can I do the equivalent of `csvfile.seek(0)` on it? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
808008305 | |
https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778812684 | https://api.github.com/repos/simonw/sqlite-utils/issues/230 | 778812684 | MDEyOklzc3VlQ29tbWVudDc3ODgxMjY4NA== | 9599 | 2021-02-14T17:45:16Z | 2021-02-14T17:45:16Z | OWNER | Running this could take any CSV (or TSV) file and automatically detect the delimiter. If no header row is detected it could add `unknown1,unknown2` headers: sqlite-utils insert db.db data file.csv --sniff (Using `--sniff` would imply `--csv`) This could be called `--sniffer` instead but I like `--sniff` better. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
808008305 | |
https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778812050 | https://api.github.com/repos/simonw/sqlite-utils/issues/228 | 778812050 | MDEyOklzc3VlQ29tbWVudDc3ODgxMjA1MA== | 9599 | 2021-02-14T17:41:30Z | 2021-02-14T17:41:30Z | OWNER | I just spotted that `csv.Sniffer` in the Python standard library has a `.has_header(sample)` method which detects if the first row appears to be a header or not, which is interesting. https://docs.python.org/3/library/csv.html#csv.Sniffer | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
807437089 | |
https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778811934 | https://api.github.com/repos/simonw/sqlite-utils/issues/228 | 778811934 | MDEyOklzc3VlQ29tbWVudDc3ODgxMTkzNA== | 9599 | 2021-02-14T17:40:48Z | 2021-02-14T17:40:48Z | OWNER | Another pattern that might be useful is to generate a header that is just "unknown1,unknown2,unknown3" for each of the columns in the rest of the file. This makes it easy to e.g. facet-explore within Datasette to figure out the correct names, then use `sqlite-utils transform --rename` to rename the columns. I needed to do that for the https://bl.iro.bl.uk/work/ns/3037474a-761c-456d-a00c-9ef3c6773f4c example. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
807437089 | |
https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778511347 | https://api.github.com/repos/simonw/sqlite-utils/issues/228 | 778511347 | MDEyOklzc3VlQ29tbWVudDc3ODUxMTM0Nw== | 9599 | 2021-02-12T23:27:50Z | 2021-02-12T23:27:50Z | OWNER | For the moment, a workaround can be to `cat` an additional row onto the start of the file. echo "name,url,description" | cat - missing_headings.csv | sqlite-utils insert blah.db table - --csv | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
807437089 | |
https://github.com/simonw/sqlite-utils/issues/131#issuecomment-778510528 | https://api.github.com/repos/simonw/sqlite-utils/issues/131 | 778510528 | MDEyOklzc3VlQ29tbWVudDc3ODUxMDUyOA== | 9599 | 2021-02-12T23:25:06Z | 2021-02-12T23:25:06Z | OWNER | If `-c` isn't available, maybe `-t` or `--type` would work for specifying column types: ``` sqlite-utils insert db.db images images.tsv \ --tsv \ --type id int \ --type score float ``` or ``` sqlite-utils insert db.db images images.tsv \ --tsv \ -t id int \ -t score float ``` | { "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
675753042 | |
https://github.com/simonw/sqlite-utils/issues/131#issuecomment-778508887 | https://api.github.com/repos/simonw/sqlite-utils/issues/131 | 778508887 | MDEyOklzc3VlQ29tbWVudDc3ODUwODg4Nw== | 9599 | 2021-02-12T23:20:11Z | 2021-02-12T23:20:11Z | OWNER | Annoyingly `-c` is currently a shortcut for `--csv` - so I'd have to do a major version bump to use that. https://github.com/simonw/sqlite-utils/blob/726219c3503e77440975cd15b74d006639feb0f8/sqlite_utils/cli.py#L601-L603 Particularly annoying because I attempted to remove the `-c` shortcut in https://github.com/simonw/sqlite-utils/commit/2c00567aac6d9c79087cfff0d054f64922b1473d#diff-76294b3d4afeb27e74e738daa01c26dd4dc9ccb6f4477451483a2ece1095902eL48 but forgot to remove it from the input options (I removed it from the output options). | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
675753042 | |
https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778349672 | https://api.github.com/repos/simonw/sqlite-utils/issues/228 | 778349672 | MDEyOklzc3VlQ29tbWVudDc3ODM0OTY3Mg== | 9599 | 2021-02-12T18:00:43Z | 2021-02-12T18:00:43Z | OWNER | I could combine this with #131 to allow types to be specified in addition to column names. Probably need an option that means "ignore the existing heading row and use this one instead". | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
807437089 | |
https://github.com/simonw/sqlite-utils/issues/227#issuecomment-778349142 | https://api.github.com/repos/simonw/sqlite-utils/issues/227 | 778349142 | MDEyOklzc3VlQ29tbWVudDc3ODM0OTE0Mg== | 9599 | 2021-02-12T17:59:35Z | 2021-02-12T17:59:35Z | OWNER | It looks like I can at least bump this size limit up to the maximum allowed by Python - I'll take a look at that. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
807174161 | |
https://github.com/simonw/datasette/issues/1221#issuecomment-777901052 | https://api.github.com/repos/simonw/datasette/issues/1221 | 777901052 | MDEyOklzc3VlQ29tbWVudDc3NzkwMTA1Mg== | 9599 | 2021-02-12T01:09:54Z | 2021-02-12T01:09:54Z | OWNER | I also tested this manually. I generated certificate files like so: cd /tmp python -m trustme This created `/tmp/server.pem`, `/tmp/client.pem` and `/tmp/server.key` Then I started Datasette like this: datasette --memory --ssl-keyfile=/tmp/server.key --ssl-certfile=/tmp/server.pem And exercise it using `curl` like so: /tmp % curl --cacert /tmp/client.pem 'https://localhost:8001/_memory.json' {"database": "_memory", "path": "/_memory", "size": 0, "tables": [], "hidden_count": 0, "views": [], "queries": [], "private": false, "allow_execute_sql": true, "query_ms": 0.8843200000114848} Note that without the `--cacert` option I get an error: ``` /tmp % curl 'https://localhost:8001/_memory.json' curl: (60) SSL certificate problem: Invalid certificate chain More details here: https://curl.haxx.se/docs/sslcerts.html curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
806849424 | |
https://github.com/simonw/datasette/issues/1221#issuecomment-777887190 | https://api.github.com/repos/simonw/datasette/issues/1221 | 777887190 | MDEyOklzc3VlQ29tbWVudDc3Nzg4NzE5MA== | 9599 | 2021-02-12T00:29:18Z | 2021-02-12T00:29:18Z | OWNER | I can use this recipe to start a `datasette` server in a sub-process during the pytest run and exercise it with real HTTP requests: https://til.simonwillison.net/pytest/subprocess-server | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
806849424 | |
https://github.com/simonw/datasette/issues/1221#issuecomment-777883452 | https://api.github.com/repos/simonw/datasette/issues/1221 | 777883452 | MDEyOklzc3VlQ29tbWVudDc3Nzg4MzQ1Mg== | 9599 | 2021-02-12T00:19:30Z | 2021-02-12T00:19:40Z | OWNER | Uvicorn supports these options: https://www.uvicorn.org/#command-line-options ``` --ssl-keyfile TEXT SSL key file --ssl-certfile TEXT SSL certificate file --ssl-keyfile-password TEXT SSL keyfile password --ssl-version INTEGER SSL version to use (see stdlib ssl module's) [default: 2] --ssl-cert-reqs INTEGER Whether client certificate is required (see stdlib ssl module's) [default: 0] --ssl-ca-certs TEXT CA certificates file --ssl-ciphers TEXT Ciphers to use (see stdlib ssl module's) [default: TLSv1] ``` For the moment I'm going to support just `--ssl-keyfile` and `--ssl-certfile` as arguments to `datasette serve`. I'll add other options if people ask for them. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
806849424 | |
https://github.com/dogsheep/evernote-to-sqlite/pull/10#issuecomment-777839351 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/10 | 777839351 | MDEyOklzc3VlQ29tbWVudDc3NzgzOTM1MQ== | 9599 | 2021-02-11T22:37:55Z | 2021-02-11T22:37:55Z | MEMBER | I've merged these changes by hand now, thanks! | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
770712149 | |
https://github.com/dogsheep/evernote-to-sqlite/issues/7#issuecomment-777827396 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/7 | 777827396 | MDEyOklzc3VlQ29tbWVudDc3NzgyNzM5Ng== | 9599 | 2021-02-11T22:13:14Z | 2021-02-11T22:13:14Z | MEMBER | My best guess is that you have an older version of `sqlite-utils` installed here - the `replace=True` argument was added in version 2.0. I've bumped the dependency in `setup.py`. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
743297582 | |
https://github.com/dogsheep/evernote-to-sqlite/issues/9#issuecomment-777821383 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/9 | 777821383 | MDEyOklzc3VlQ29tbWVudDc3NzgyMTM4Mw== | 9599 | 2021-02-11T22:01:28Z | 2021-02-11T22:01:28Z | MEMBER | Aha! I think I've figured out what's going on here. The CData blocks containing the notes look like this: `<![CDATA[<!DOCTYPE en-note SYSTEM "http://xml.evernote.com/pub/enml2.dtd"><en-note><div>This note includes two images.</div><div><br /></div>...` The DTD at http://xml.evernote.com/pub/enml2.dtd includes some entities: ``` <!--=========== External character mnemonic entities ===================--> <!ENTITY % HTMLlat1 PUBLIC "-//W3C//ENTITIES Latin 1 for XHTML//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml-lat1.ent"> %HTMLlat1; <!ENTITY % HTMLsymbol PUBLIC "-//W3C//ENTITIES Symbols for XHTML//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml-symbol.ent"> %HTMLsymbol; <!ENTITY % HTMLspecial PUBLIC "-//W3C//ENTITIES Special for XHTML//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml-special.ent"> %HTMLspecial; ``` So I need to be able to handle all of those different entities. I think I can do that using `html.entities.entitydefs` from the Python standard library, which looks a bit like this: ```python {'Aacute': 'Á', 'aacute': 'á', 'Aacute;': 'Á', 'aacute;': 'á', 'Abreve;': 'Ă', 'abreve;': 'ă', 'ac;': '∾', 'acd;': '∿', # ... } ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
748372469 | |
https://github.com/dogsheep/evernote-to-sqlite/issues/11#issuecomment-777798330 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/11 | 777798330 | MDEyOklzc3VlQ29tbWVudDc3Nzc5ODMzMA== | 9599 | 2021-02-11T21:18:58Z | 2021-02-11T21:18:58Z | MEMBER | Thanks for the fix! | { "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
792851444 | |
https://github.com/simonw/datasette/issues/1200#issuecomment-777178728 | https://api.github.com/repos/simonw/datasette/issues/1200 | 777178728 | MDEyOklzc3VlQ29tbWVudDc3NzE3ODcyOA== | 9599 | 2021-02-11T03:13:59Z | 2021-02-11T03:13:59Z | OWNER | I came up with the need for this while playing with this tool: https://calands.datasettes.com/calands?sql=select%0D%0A++AsGeoJSON(geometry)%2C+*%0D%0Afrom%0D%0A++CPAD_2020a_SuperUnits%0D%0Awhere%0D%0A++PARK_NAME+like+'%25mini%25'+and%0D%0A++Intersects(GeomFromGeoJSON(%3Afreedraw)%2C+geometry)+%3D+1%0D%0A++and+CPAD_2020a_SuperUnits.rowid+in+(%0D%0A++++select%0D%0A++++++rowid%0D%0A++++from%0D%0A++++++SpatialIndex%0D%0A++++where%0D%0A++++++f_table_name+%3D+'CPAD_2020a_SuperUnits'%0D%0A++++++and+search_frame+%3D+GeomFromGeoJSON(%3Afreedraw)%0D%0A++)&freedraw={"type"%3A"MultiPolygon"%2C"coordinates"%3A[[[[-122.42202758789064%2C37.82280243352759]%2C[-122.39868164062501%2C37.823887203271454]%2C[-122.38220214843751%2C37.81846319511331]%2C[-122.35061645507814%2C37.77071473849611]%2C[-122.34924316406251%2C37.74465712069939]%2C[-122.37258911132814%2C37.703380457832374]%2C[-122.39044189453125%2C37.690340943717715]%2C[-122.41241455078126%2C37.680559803205135]%2C[-122.44262695312501%2C37.67295135774715]%2C[-122.47283935546876%2C37.67295135774715]%2C[-122.52502441406251%2C37.68382032669382]%2C[-122.53463745117189%2C37.6892542140253]%2C[-122.54699707031251%2C37.690340943717715]%2C[-122.55798339843751%2C37.72945260537781]%2C[-122.54287719726564%2C37.77831314799672]%2C[-122.49893188476564%2C37.81303878836991]%2C[-122.46185302734376%2C37.82822612280363]%2C[-122.42889404296876%2C37.82822612280363]%2C[-122.42202758789064%2C37.82280243352759]]]]} - before I fixed https://github.com/simonw/datasette-leaflet-geojson/issues/16 it was loading a LOT of maps, which felt bad. I wanted to be able to link people to that page with a hard limit on the number of rows displayed on that page. It's mainly to guard against unexpected behaviour from limit-less queries though. It's not a very high priority feature! | { "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
792890765 | |
https://github.com/simonw/datasette/issues/1219#issuecomment-775442039 | https://api.github.com/repos/simonw/datasette/issues/1219 | 775442039 | MDEyOklzc3VlQ29tbWVudDc3NTQ0MjAzOQ== | 9599 | 2021-02-08T20:39:52Z | 2021-02-08T22:13:00Z | OWNER | This comment helped me find a pattern for running Scalene against the Datasette test suite: https://github.com/emeryberger/scalene/issues/70#issuecomment-755245858 ``` pip install scalene ``` Then I created a file called `run_tests.py` with the following contents: ```python if __name__ == "__main__": import sys, pytest pytest.main(sys.argv) ``` Then I ran this: ``` scalene --profile-all run_tests.py -sv -x . ``` But... it quit with a segmentation fault! ``` (datasette) datasette % scalene --profile-all run_tests.py -sv -x . ======================================================================== test session starts ======================================================================== platform darwin -- Python 3.8.6, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- python cachedir: .pytest_cache rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini plugins: asyncio-0.14.0, timeout-1.4.2 collecting ... Fatal Python error: Segmentation fault Current thread 0x0000000110c1edc0 (most recent call first): File "/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py", line 553 in detect_json1 File "/Users/simon/Dropbox/Development/datasette/datasette/filters.py", line 168 in Filters File "/Users/simon/Dropbox/Development/datasette/datasette/filters.py", line 94 in <module> File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed File "<frozen importlib._bootstrap_external>", line 783 in exec_module File "<frozen importlib._bootstrap>", line 671 in _load_unlocked File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 991 in _find_and_load File "/Users/simon/Dropbox/Development/datasette/datasette/views/table.py", line 27 in <module> File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed File "<frozen importlib._bootstrap_external>", line 783 in exec_module File "<frozen importlib._bootstrap>", line 671 in _load_unlocked … | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
803929694 | |
https://github.com/simonw/datasette/issues/1219#issuecomment-775497449 | https://api.github.com/repos/simonw/datasette/issues/1219 | 775497449 | MDEyOklzc3VlQ29tbWVudDc3NTQ5NzQ0OQ== | 9599 | 2021-02-08T22:11:34Z | 2021-02-08T22:11:34Z | OWNER | https://github.com/emeryberger/scalene/issues/110 reports a "received signal SIGSEGV" error that was fixed by upgrading to the latest Scalene version, but I'm running that already. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
803929694 | |
https://github.com/simonw/sqlite-utils/issues/223#issuecomment-774373829 | https://api.github.com/repos/simonw/sqlite-utils/issues/223 | 774373829 | MDEyOklzc3VlQ29tbWVudDc3NDM3MzgyOQ== | 9599 | 2021-02-06T01:39:47Z | 2021-02-06T01:39:47Z | OWNER | Documentation: https://sqlite-utils.datasette.io/en/stable/cli.html#cli-insert-csv-tsv-delimiter | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
788527932 | |
https://github.com/simonw/datasette/issues/1216#issuecomment-772796111 | https://api.github.com/repos/simonw/datasette/issues/1216 | 772796111 | MDEyOklzc3VlQ29tbWVudDc3Mjc5NjExMQ== | 9599 | 2021-02-03T20:20:48Z | 2021-02-03T20:20:48Z | OWNER | Relevant code: https://github.com/simonw/datasette/blob/1600d2a3ec3ada1f6fb5b1eb73bdaeccb5f80530/datasette/app.py#L620-L632 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
800669347 | |
https://github.com/simonw/datasette/issues/1214#issuecomment-772001787 | https://api.github.com/repos/simonw/datasette/issues/1214 | 772001787 | MDEyOklzc3VlQ29tbWVudDc3MjAwMTc4Nw== | 9599 | 2021-02-02T21:28:53Z | 2021-02-02T21:28:53Z | OWNER | Fix is now live on https://latest.datasette.io/fixtures/searchable?_search=terry - clearing "terry" and re-submitting the form now works as expected. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
799693777 | |
https://github.com/simonw/datasette/issues/1214#issuecomment-771992628 | https://api.github.com/repos/simonw/datasette/issues/1214 | 771992628 | MDEyOklzc3VlQ29tbWVudDc3MTk5MjYyOA== | 9599 | 2021-02-02T21:15:18Z | 2021-02-02T21:15:18Z | OWNER | The cause of this bug is form fields which begin with `_` but ARE displayed as form inputs on the page - hence should not be duplicated in an `<input type="hidden">` element. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
799693777 | |
https://github.com/simonw/datasette/issues/1214#issuecomment-771992025 | https://api.github.com/repos/simonw/datasette/issues/1214 | 771992025 | MDEyOklzc3VlQ29tbWVudDc3MTk5MjAyNQ== | 9599 | 2021-02-02T21:14:16Z | 2021-02-02T21:14:16Z | OWNER | As a result, navigating to https://github-to-sqlite.dogsheep.net/github/labels?_search=help and clearing out the `_search` field then submitting the form does NOT clear the search term. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
799693777 | |
https://github.com/simonw/datasette/issues/1212#issuecomment-771976561 | https://api.github.com/repos/simonw/datasette/issues/1212 | 771976561 | MDEyOklzc3VlQ29tbWVudDc3MTk3NjU2MQ== | 9599 | 2021-02-02T20:53:27Z | 2021-02-02T20:53:27Z | OWNER | It would be great if we could get `python-xdist` to run too - I tried it in the past and gave up when I ran into those race conditions, but I've not done any further digging to see if there's a way to fix that. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
797651831 | |
https://github.com/simonw/datasette/issues/1212#issuecomment-771975941 | https://api.github.com/repos/simonw/datasette/issues/1212 | 771975941 | MDEyOklzc3VlQ29tbWVudDc3MTk3NTk0MQ== | 9599 | 2021-02-02T20:52:36Z | 2021-02-02T20:52:36Z | OWNER | 37 minutes, wow! They're a little slow for me (4-5 minutes perhaps) but not nearly that bad. Thanks for running that profile. I think you're right: figuring out how to use more session scopes would definitely help. The `:memory:` idea is interesting too. The new `memory_name=` feature added in #1151 (released in Datasette 0.54) could help a lot here, since it allows Datasette instances to share the same in-memory database across multiple HTTP requests and connections. Note that `memory_name=` also persists within test runs themselves, independently of any `scope=` options on the fixtures. That might actually help us here! I'd be delighted if you explored this issue further, especially the option of using `memory_name=` for the fixtures databases used by the tests. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
797651831 | |
https://github.com/simonw/datasette/issues/1213#issuecomment-771968675 | https://api.github.com/repos/simonw/datasette/issues/1213 | 771968675 | MDEyOklzc3VlQ29tbWVudDc3MTk2ODY3NQ== | 9599 | 2021-02-02T20:41:55Z | 2021-02-02T20:41:55Z | OWNER | So maybe I could a special response header which ASGI middleware can pick up that means "Don't attempt to gzip this, just stream it through". | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
799663959 | |
https://github.com/simonw/datasette/issues/1213#issuecomment-771968177 | https://api.github.com/repos/simonw/datasette/issues/1213 | 771968177 | MDEyOklzc3VlQ29tbWVudDc3MTk2ODE3Nw== | 9599 | 2021-02-02T20:41:13Z | 2021-02-02T20:41:13Z | OWNER | Starlette accumulates the full response body in a `body` variable and then does this: ```python elif message_type == "http.response.body": # Remaining body in streaming GZip response. body = message.get("body", b"") more_body = message.get("more_body", False) self.gzip_file.write(body) if not more_body: self.gzip_file.close() message["body"] = self.gzip_buffer.getvalue() self.gzip_buffer.seek(0) self.gzip_buffer.truncate() await self.send(message) ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
799663959 | |
https://github.com/simonw/datasette/issues/1213#issuecomment-771965281 | https://api.github.com/repos/simonw/datasette/issues/1213 | 771965281 | MDEyOklzc3VlQ29tbWVudDc3MTk2NTI4MQ== | 9599 | 2021-02-02T20:37:08Z | 2021-02-02T20:39:24Z | OWNER | Starlette's gzip middleware implementation is here: https://github.com/encode/starlette/blob/0.14.2/starlette/middleware/gzip.py | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
799663959 | |
https://github.com/dogsheep/github-to-sqlite/issues/60#issuecomment-770071568 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/60 | 770071568 | MDEyOklzc3VlQ29tbWVudDc3MDA3MTU2OA== | 9599 | 2021-01-29T21:56:15Z | 2021-01-29T21:56:15Z | MEMBER | I really like the way you're using pipes here - really smart. It's similar to how I build the demo database in this GitHub Actions workflow: https://github.com/dogsheep/github-to-sqlite/blob/62dfd3bc4014b108200001ef4bc746feb6f33b45/.github/workflows/deploy-demo.yml#L52-L82 `twitter-to-sqlite` actually has a mechanism for doing this kind of thing, documented at https://github.com/dogsheep/twitter-to-sqlite#providing-input-from-a-sql-query-with---sql-and---attach It lets you do things like: ``` $ twitter-to-sqlite users-lookup my.db --sql="select follower_id from following" --ids ``` Maybe I should add something similar to `github-to-sqlite`? Feels like it could be really useful. | { "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
797097140 | |
https://github.com/dogsheep/twitter-to-sqlite/issues/56#issuecomment-769957751 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/56 | 769957751 | MDEyOklzc3VlQ29tbWVudDc2OTk1Nzc1MQ== | 9599 | 2021-01-29T17:59:40Z | 2021-01-29T17:59:40Z | MEMBER | This is interesting - how did you create that initial table? Was this using the `twitter-to-sqlite import archive.db ~/Downloads/twitter-2019-06-25-b31f2.zip` command, or something else? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
796736607 | |
https://github.com/simonw/datasette/issues/1207#issuecomment-769534187 | https://api.github.com/repos/simonw/datasette/issues/1207 | 769534187 | MDEyOklzc3VlQ29tbWVudDc2OTUzNDE4Nw== | 9599 | 2021-01-29T02:37:19Z | 2021-01-29T02:37:19Z | OWNER | https://docs.datasette.io/en/latest/testing_plugins.html#using-pdb-for-errors-thrown-inside-datasette | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
793881756 | |
https://github.com/simonw/datasette/issues/1209#issuecomment-769455370 | https://api.github.com/repos/simonw/datasette/issues/1209 | 769455370 | MDEyOklzc3VlQ29tbWVudDc2OTQ1NTM3MA== | 9599 | 2021-01-28T23:00:21Z | 2021-01-28T23:00:21Z | OWNER | Good catch on the workaround here. The root problem is that `datasette-template-sql` looks for the first available databsae if you don't provide it with a `database=` argument, and in Datasette 0.54 the first available database changed to being the new `_internal` database. Is this a bug? I think it is - because the documented behaviour on https://docs.datasette.io/en/stable/internals.html#get-database-name is this: > `name` - string, optional > > The name to be used for this database - this will be used in the URL path, e.g. `/dbname`. If not specified Datasette will pick one based on the filename or memory name. Since the new behaviour differs from what was in the documentation I'm going to treat this as a bug and fix it. | { "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
795367402 | |
https://github.com/simonw/datasette/issues/1205#issuecomment-769453074 | https://api.github.com/repos/simonw/datasette/issues/1205 | 769453074 | MDEyOklzc3VlQ29tbWVudDc2OTQ1MzA3NA== | 9599 | 2021-01-28T22:54:49Z | 2021-01-28T22:55:02Z | OWNER | I also checked that the following works: echo '{"foo": "bar"}' | sqlite-utils insert _memory.db demo - datasette _memory.db --memory Sure enough, it results in the following Datasette homepage - thanks to #509 <img width="274" alt="Datasette___memory___memory_2" src="https://user-images.githubusercontent.com/9599/106208790-c8564980-6178-11eb-8b8b-053a9f1d0193.png"> | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
793027837 | |
https://github.com/simonw/datasette/issues/1205#issuecomment-769452084 | https://api.github.com/repos/simonw/datasette/issues/1205 | 769452084 | MDEyOklzc3VlQ29tbWVudDc2OTQ1MjA4NA== | 9599 | 2021-01-28T22:52:23Z | 2021-01-28T22:52:23Z | OWNER | Here are the redirect tests: https://github.com/simonw/datasette/blob/1600d2a3ec3ada1f6fb5b1eb73bdaeccb5f80530/tests/test_api.py#L635-L648 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
793027837 | |
https://github.com/simonw/datasette/issues/1205#issuecomment-769442165 | https://api.github.com/repos/simonw/datasette/issues/1205 | 769442165 | MDEyOklzc3VlQ29tbWVudDc2OTQ0MjE2NQ== | 9599 | 2021-01-28T22:30:16Z | 2021-01-28T22:30:27Z | OWNER | I'm going to do this, with redirects from `/:memory:*`. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
793027837 | |
https://github.com/simonw/datasette/issues/1210#issuecomment-769274591 | https://api.github.com/repos/simonw/datasette/issues/1210 | 769274591 | MDEyOklzc3VlQ29tbWVudDc2OTI3NDU5MQ== | 9599 | 2021-01-28T18:10:02Z | 2021-01-28T18:10:02Z | OWNER | That definitely sounds like a bug! Can you provide a copy of your `metadata.JSON` and the command-line you are using to launch Datasette? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
796234313 | |
https://github.com/simonw/datasette/issues/1208#issuecomment-767823684 | https://api.github.com/repos/simonw/datasette/issues/1208 | 767823684 | MDEyOklzc3VlQ29tbWVudDc2NzgyMzY4NA== | 9599 | 2021-01-26T20:58:51Z | 2021-01-26T20:58:51Z | OWNER | This is a good catch - I've been lazy about this, but you're right that it's an issue that needs cleaning up. Would be very happy to apply a PR, thanks! | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
794554881 | |
https://github.com/simonw/datasette/issues/1151#issuecomment-767762551 | https://api.github.com/repos/simonw/datasette/issues/1151 | 767762551 | MDEyOklzc3VlQ29tbWVudDc2Nzc2MjU1MQ== | 9599 | 2021-01-26T19:07:44Z | 2021-01-26T19:07:44Z | OWNER | Mentioned in https://simonwillison.net/2021/Jan/25/datasette/ | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
770448622 | |
https://github.com/simonw/datasette/issues/991#issuecomment-767761155 | https://api.github.com/repos/simonw/datasette/issues/991 | 767761155 | MDEyOklzc3VlQ29tbWVudDc2Nzc2MTE1NQ== | 9599 | 2021-01-26T19:05:21Z | 2021-01-26T19:06:36Z | OWNER | Idea: implement this using the existing table view, with a custom template called `table-internal-bb0ec0-tables.html` - that's the custom template listed in the HTML comments at the bottom of https://latest.datasette.io/_internal/tables | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
714377268 | |
https://github.com/simonw/datasette/issues/1201#issuecomment-766991680 | https://api.github.com/repos/simonw/datasette/issues/1201 | 766991680 | MDEyOklzc3VlQ29tbWVudDc2Njk5MTY4MA== | 9599 | 2021-01-25T17:42:21Z | 2021-01-25T17:42:21Z | OWNER | https://docs.datasette.io/en/stable/changelog.html#v0-54 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
792904595 | |
https://github.com/simonw/datasette/pull/1206#issuecomment-766588371 | https://api.github.com/repos/simonw/datasette/issues/1206 | 766588371 | MDEyOklzc3VlQ29tbWVudDc2NjU4ODM3MQ== | 9599 | 2021-01-25T06:49:06Z | 2021-01-25T06:49:06Z | OWNER | Last thing to do: write up the annotated version of these release notes, assign it a URL on my blog and link to it from the release notes here so I can publish them simultaneously. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
793086333 | |
https://github.com/simonw/datasette/pull/1206#issuecomment-766588020 | https://api.github.com/repos/simonw/datasette/issues/1206 | 766588020 | MDEyOklzc3VlQ29tbWVudDc2NjU4ODAyMA== | 9599 | 2021-01-25T06:48:20Z | 2021-01-25T06:48:20Z | OWNER | Issues to reference in the commit message: #509, #1091, #1150, #1151, #1166, #1167, #1178, #1181, #1182, #1184, #1185, #1186, #1187, #1194, #1198 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
793086333 | |
https://github.com/simonw/datasette/issues/1201#issuecomment-766586151 | https://api.github.com/repos/simonw/datasette/issues/1201 | 766586151 | MDEyOklzc3VlQ29tbWVudDc2NjU4NjE1MQ== | 9599 | 2021-01-25T06:44:43Z | 2021-01-25T06:44:43Z | OWNER | OK, release notes are ready to merge from that branch. I'll ship the release in the morning, to give me time to write the accompanying annotated release notes. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
792904595 | |
https://github.com/simonw/datasette/issues/1201#issuecomment-766545604 | https://api.github.com/repos/simonw/datasette/issues/1201 | 766545604 | MDEyOklzc3VlQ29tbWVudDc2NjU0NTYwNA== | 9599 | 2021-01-25T05:14:31Z | 2021-01-25T05:14:31Z | OWNER | The two big ticket items are `<script type="module">` support and the new `_internal` mechanism. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
792904595 | |
https://github.com/simonw/datasette/issues/1201#issuecomment-766545442 | https://api.github.com/repos/simonw/datasette/issues/1201 | 766545442 | MDEyOklzc3VlQ29tbWVudDc2NjU0NTQ0Mg== | 9599 | 2021-01-25T05:13:59Z | 2021-01-25T05:13:59Z | OWNER | The big stuff: - Database(memory_name=) for shared in-memory databases, closes #1151 - The `_internal` database - #1150 - script type=module support, closes #1186 , #1187 - Improved design for the `.add_database()` method 8919f99c2f7f245aca7f94bd53d5ac9d04aa42b5 - which means databases with the same stem can now be opened, #509 - Adopted Prettier #1166 Smaller: - force_https_urls on for publish cloudrun, refs #1178 - Fixed bug in example nginx config, refs #1091 - Shrunk ecosystem docs in favour of datasette.io, closes #1182 - request.full_path property, closes #1184 - Better PRAGMA error message, closes #1185 - publish heroku now uses python-3.8.7 - Plugin testing documentation on using pytest-httpx Closes #1198 - Contributing docs for Black and Prettier, closes #1167 - All ?_ parameters now copied to hidden form fields, closes #1194 - Fixed bug loading database called 'test-database (1).sqlite' - Closes #1181. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
792904595 | |
https://github.com/simonw/datasette/issues/1201#issuecomment-766543387 | https://api.github.com/repos/simonw/datasette/issues/1201 | 766543387 | MDEyOklzc3VlQ29tbWVudDc2NjU0MzM4Nw== | 9599 | 2021-01-25T05:07:40Z | 2021-01-25T05:13:29Z | OWNER | Changes: https://github.com/simonw/datasette/compare/0.53...a5ede3cdd455e2bb1a1fb2f4e1b5a9855caf5179 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
792904595 | |
https://github.com/simonw/datasette/issues/983#issuecomment-766536076 | https://api.github.com/repos/simonw/datasette/issues/983 | 766536076 | MDEyOklzc3VlQ29tbWVudDc2NjUzNjA3Ng== | 9599 | 2021-01-25T04:43:53Z | 2021-01-25T04:43:53Z | OWNER | ... actually not going to include this in 0.54, I need to write a couple of plugins myself using it before I even make it available in preview. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
712260429 | |
https://github.com/simonw/datasette/issues/1195#issuecomment-766535046 | https://api.github.com/repos/simonw/datasette/issues/1195 | 766535046 | MDEyOklzc3VlQ29tbWVudDc2NjUzNTA0Ng== | 9599 | 2021-01-25T04:40:08Z | 2021-01-25T04:40:08Z | OWNER | Also: should the view name really be the same for both of these pages? - https://latest.datasette.io/fixtures?sql=select+*+from+facetable - https://latest.datasette.io/fixtures/neighborhood_search Where one of them is a canned query and the other is an arbitrary query? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
789336592 | |
https://github.com/simonw/datasette/issues/1195#issuecomment-766534748 | https://api.github.com/repos/simonw/datasette/issues/1195 | 766534748 | MDEyOklzc3VlQ29tbWVudDc2NjUzNDc0OA== | 9599 | 2021-01-25T04:38:56Z | 2021-01-25T04:38:56Z | OWNER | Here's a diff showing how far I got before I abandoned this particular effort: ```diff diff --git a/datasette/views/base.py b/datasette/views/base.py index a21b929..04e4aa9 100644 --- a/datasette/views/base.py +++ b/datasette/views/base.py @@ -120,7 +120,7 @@ class BaseView: handler = getattr(self, request.method.lower(), None) return await handler(request, *args, **kwargs) - async def render(self, templates, request, context=None): + async def render(self, templates, request, context=None, view_name=None): context = context or {} template = self.ds.jinja_env.select_template(templates) template_context = { @@ -135,7 +135,7 @@ class BaseView: } return Response.html( await self.ds.render_template( - template, template_context, request=request, view_name=self.name + template, template_context, request=request, view_name=view_name ) ) @@ -155,7 +155,7 @@ class BaseView: class DataView(BaseView): - name = "" + view_name = "no-view-name" re_named_parameter = re.compile(":([a-zA-Z0-9_]+)") async def options(self, request, *args, **kwargs): @@ -414,6 +414,10 @@ class DataView(BaseView): args["table"] = urllib.parse.unquote_plus(args["table"]) return _format, args + async def get_view_name(self, request, database, hash, **kwargs): + # Defaults to self.view_name, but can be over-ridden by subclasses + return self.view_name + async def view_get(self, request, database, hash, correct_hash_provided, **kwargs): _format, kwargs = await self.get_format(request, database, kwargs) @@ -424,6 +428,8 @@ class DataView(BaseView): # HTML views default to expanding all foreign key labels kwargs["default_labels"] = True + view_name = await self.get_view_name(request, database, hash, **kwargs) + extra_template_data = {} … | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
789336592 | |
https://github.com/simonw/datasette/issues/1195#issuecomment-766534634 | https://api.github.com/repos/simonw/datasette/issues/1195 | 766534634 | MDEyOklzc3VlQ29tbWVudDc2NjUzNDYzNA== | 9599 | 2021-01-25T04:38:30Z | 2021-01-25T04:38:30Z | OWNER | This has proved surprisingly difficult to implement, due to the weird way the QueryView is actually called. The class itself isn't used like other view classes - instead, the `.data()` methods of both `DatabaseView` and `TableView` dispatch out to `QueryView.data()` when they need to: https://github.com/simonw/datasette/blob/07e163561592c743e4117f72102fcd350a600909/datasette/views/table.py#L259-L270 https://github.com/simonw/datasette/blob/07e163561592c743e4117f72102fcd350a600909/datasette/views/table.py#L290-L294 https://github.com/simonw/datasette/blob/07e163561592c743e4117f72102fcd350a600909/datasette/views/database.py#L39-L44 It turns out this is a bad pattern because it makes changes like this one WAY harder than they should be. I think I should clean this up as part of #878. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
789336592 | |
https://github.com/simonw/datasette/pull/1204#issuecomment-766525337 | https://api.github.com/repos/simonw/datasette/issues/1204 | 766525337 | MDEyOklzc3VlQ29tbWVudDc2NjUyNTMzNw== | 9599 | 2021-01-25T04:04:28Z | 2021-01-25T04:04:28Z | OWNER | Writing the tests will be a bit tricky since we need to confirm that the `include_table_top(datasette, database, actor, table)` arguments were all passed correctly but the only thing we get back from the plugin is a list of templates. Maybe encode those values into the template names somehow? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
793002853 | |
https://github.com/simonw/datasette/issues/987#issuecomment-766524141 | https://api.github.com/repos/simonw/datasette/issues/987 | 766524141 | MDEyOklzc3VlQ29tbWVudDc2NjUyNDE0MQ== | 9599 | 2021-01-25T03:59:55Z | 2021-01-25T03:59:55Z | OWNER | This is joined with #1191 now, which I've bumped from 0.54. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
712984738 | |
https://github.com/simonw/datasette/issues/1191#issuecomment-766524016 | https://api.github.com/repos/simonw/datasette/issues/1191 | 766524016 | MDEyOklzc3VlQ29tbWVudDc2NjUyNDAxNg== | 9599 | 2021-01-25T03:59:17Z | 2021-01-25T03:59:17Z | OWNER | More work can happen in the PR: #1204 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
787098345 | |
https://github.com/simonw/datasette/issues/1191#issuecomment-766523866 | https://api.github.com/repos/simonw/datasette/issues/1191 | 766523866 | MDEyOklzc3VlQ29tbWVudDc2NjUyMzg2Ng== | 9599 | 2021-01-25T03:58:34Z | 2021-01-25T03:58:34Z | OWNER | I've got a good prototype working now, but I'm dropping this from the Datasette 0.54 milestone because it requires a bunch of additional work to make sure it is really well tested and documented. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
787098345 | |
https://github.com/simonw/datasette/issues/1194#issuecomment-766491911 | https://api.github.com/repos/simonw/datasette/issues/1194 | 766491911 | MDEyOklzc3VlQ29tbWVudDc2NjQ5MTkxMQ== | 9599 | 2021-01-25T02:02:15Z | 2021-01-25T02:02:15Z | OWNER | I'm going to copy across anything starting with an underscore. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
788447787 | |
https://github.com/simonw/datasette/issues/1167#issuecomment-766491566 | https://api.github.com/repos/simonw/datasette/issues/1167 | 766491566 | MDEyOklzc3VlQ29tbWVudDc2NjQ5MTU2Ng== | 9599 | 2021-01-25T02:01:19Z | 2021-01-25T02:01:19Z | OWNER | New documentation section here (I documented Black as well): https://docs.datasette.io/en/latest/contributing.html#code-formatting | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
777145954 | |
https://github.com/simonw/datasette/issues/1167#issuecomment-766487520 | https://api.github.com/repos/simonw/datasette/issues/1167 | 766487520 | MDEyOklzc3VlQ29tbWVudDc2NjQ4NzUyMA== | 9599 | 2021-01-25T01:44:43Z | 2021-01-25T01:44:43Z | OWNER | Thanks @benpickles, I just merged that in. I'll use it in the documentation. # To check code is conformant npm run prettier -- --check # To fix it if it isn't npm run fix | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
777145954 | |
https://github.com/simonw/datasette/issues/983#issuecomment-766484915 | https://api.github.com/repos/simonw/datasette/issues/983 | 766484915 | MDEyOklzc3VlQ29tbWVudDc2NjQ4NDkxNQ== | 9599 | 2021-01-25T01:33:29Z | 2021-01-25T01:33:29Z | OWNER | I'm going to ship a version of this in Datasette 0.54 with a warning that the interface should be considered unstable (see #1202) so that we can start trying this out. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
712260429 | |
https://github.com/simonw/datasette/issues/1179#issuecomment-766484435 | https://api.github.com/repos/simonw/datasette/issues/1179 | 766484435 | MDEyOklzc3VlQ29tbWVudDc2NjQ4NDQzNQ== | 9599 | 2021-01-25T01:31:36Z | 2021-01-25T01:31:36Z | OWNER | Relevant existing tests: https://github.com/simonw/datasette/blob/461670a0b87efa953141b449a9a261919864ceb3/tests/test_utils.py#L365-L398 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
780278550 | |
https://github.com/simonw/datasette/issues/1179#issuecomment-766484257 | https://api.github.com/repos/simonw/datasette/issues/1179 | 766484257 | MDEyOklzc3VlQ29tbWVudDc2NjQ4NDI1Nw== | 9599 | 2021-01-25T01:30:57Z | 2021-01-25T01:30:57Z | OWNER | The challenge here is figuring out what the original path, without the `.format`, actually was - while taking into account that Datasette has a special case for tables that themselves end in a `.something`. The `path_with_format()` function nearly does what we need here: https://github.com/simonw/datasette/blob/b6a7b58fa01af0cd5a5e94bd17d686d283a46819/datasette/utils/__init__.py#L710-L729 It can be called with `replace_format="csv"` to REMOVE the `.csv` format and replace it with something else. Problem is, we want to use it to get rid of the format entirely. We could update `path_with_format()` to accept `format=''` to mean "remove the format entirely", but it's a bit messy. It may be better to reconsider the design of `path_with_format()` and related utility functions entirely. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
780278550 | |
https://github.com/simonw/datasette/issues/1191#issuecomment-766466030 | https://api.github.com/repos/simonw/datasette/issues/1191 | 766466030 | MDEyOklzc3VlQ29tbWVudDc2NjQ2NjAzMA== | 9599 | 2021-01-25T00:11:04Z | 2021-01-25T00:11:04Z | OWNER | I can combine this with #987 - each of these areas of the page can be wrapped in a `<div>` with a class that matches the name of the plugin hook, that way JavaScript plugins can append their content in the same place as Python plugins. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
787098345 | |
https://github.com/simonw/datasette/issues/1154#issuecomment-766465719 | https://api.github.com/repos/simonw/datasette/issues/1154 | 766465719 | MDEyOklzc3VlQ29tbWVudDc2NjQ2NTcxOQ== | 9599 | 2021-01-25T00:09:22Z | 2021-01-25T00:09:22Z | OWNER | https://docs.datasette.io/en/latest/internals.html#the-internal-database <img width="738" alt="Internals_for_plugins_—_Datasette_documentation" src="https://user-images.githubusercontent.com/9599/105648087-76eb4900-5e5e-11eb-9c73-c00ba37b7b10.png"> | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
771208009 | |
https://github.com/simonw/datasette/issues/1202#issuecomment-766464136 | https://api.github.com/repos/simonw/datasette/issues/1202 | 766464136 | MDEyOklzc3VlQ29tbWVudDc2NjQ2NDEzNg== | 9599 | 2021-01-25T00:01:02Z | 2021-01-25T00:01:02Z | OWNER | I'm going to use the existing `.. warning::` pattern for this. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
792931244 | |
https://github.com/simonw/datasette/issues/1090#issuecomment-766463496 | https://api.github.com/repos/simonw/datasette/issues/1090 | 766463496 | MDEyOklzc3VlQ29tbWVudDc2NjQ2MzQ5Ng== | 9599 | 2021-01-24T23:57:00Z | 2021-01-24T23:57:00Z | OWNER | Related: I built [datasette-leaflet-freedraw](https://datasette.io/plugins/datasette-leaflet-freedraw) which turns any canned query field called `freedraw` or `something_freedraw` into an interactive map that you can draw on to create a GeoJSON MultiPolygon. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
741862364 | |
https://github.com/simonw/datasette/issues/1202#issuecomment-766462475 | https://api.github.com/repos/simonw/datasette/issues/1202 | 766462475 | MDEyOklzc3VlQ29tbWVudDc2NjQ2MjQ3NQ== | 9599 | 2021-01-24T23:49:28Z | 2021-01-24T23:50:33Z | OWNER | Can use an "admonition" similar to this: ```sphinx .. warning:: Restricting access to tables and views in this way will NOT prevent users from querying them using arbitrary SQL queries, `like this <https://latest.datasette.io/fixtures?sql=select+*+from+facetable>`__ for example. ``` As seen on https://docs.datasette.io/en/stable/authentication.html#controlling-access-to-specific-tables-and-views Documentation: https://docutils.sourceforge.io/docs/ref/rst/directives.html#specific-admonitions | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
792931244 | |
https://github.com/simonw/datasette/issues/1154#issuecomment-766462197 | https://api.github.com/repos/simonw/datasette/issues/1154 | 766462197 | MDEyOklzc3VlQ29tbWVudDc2NjQ2MjE5Nw== | 9599 | 2021-01-24T23:47:06Z | 2021-01-24T23:47:06Z | OWNER | I'm going to document this but mark it as unstable, using a new documentation convention for marking unstable APIs. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
771208009 | |
https://github.com/simonw/datasette/issues/1179#issuecomment-766434629 | https://api.github.com/repos/simonw/datasette/issues/1179 | 766434629 | MDEyOklzc3VlQ29tbWVudDc2NjQzNDYyOQ== | 9599 | 2021-01-24T21:23:47Z | 2021-01-24T21:23:47Z | OWNER | I'm just going to do `path` and `full_path` (which includes the querystring)`. The `datasette.absolute_url()` method can be used by plugins that need the full URL. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
780278550 | |
https://github.com/simonw/datasette/issues/1190#issuecomment-766433153 | https://api.github.com/repos/simonw/datasette/issues/1190 | 766433153 | MDEyOklzc3VlQ29tbWVudDc2NjQzMzE1Mw== | 9599 | 2021-01-24T21:13:25Z | 2021-01-24T21:13:25Z | OWNER | This ties in to a bunch of other ideas that are in flight at the moment. If you're publishing databases by uploading them, how do you attach metadata? Ideally by baking it into the database file itself, using the mechanism from #1169. How could this interact with the `datasette insert` concept from #1163? Could you pass a CSV file to the `upload` command and have that converted and uploaded for you, or would you create the database file locally using `datasette insert` and then upload it as a separate `datasette upload` step? Lots to think about here. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
787098146 | |
https://github.com/simonw/datasette/issues/1190#issuecomment-766430644 | https://api.github.com/repos/simonw/datasette/issues/1190 | 766430644 | MDEyOklzc3VlQ29tbWVudDc2NjQzMDY0NA== | 9599 | 2021-01-24T20:57:03Z | 2021-01-24T20:57:03Z | OWNER | I really like this idea. It feels like an opportunity for a plugin that adds two things: an API endpoint to Datasette for accepting uploaded databases, and a `datasette publish upload` subcommand which can upload files to that endpoint (with some kind of authentication mechanism). | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
787098146 |