github
html_url | issue_url | id | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
https://github.com/simonw/datasette/issues/1362#issuecomment-855427396 | https://api.github.com/repos/simonw/datasette/issues/1362 | 855427396 | MDEyOklzc3VlQ29tbWVudDg1NTQyNzM5Ng== | 9599 | 2021-06-06T16:46:17Z | 2021-06-06T16:46:17Z | OWNER | Mind you, since that plugin hook looks like this: ```python @hookimpl def extra_body_script(): return { "module": True, "script": "console.log('Your JavaScript goes here...')" } ``` Having it calculate a sha256 hash wouldn't be difficult. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
912864936 | |
https://github.com/simonw/datasette/issues/1362#issuecomment-855426750 | https://api.github.com/repos/simonw/datasette/issues/1362 | 855426750 | MDEyOklzc3VlQ29tbWVudDg1NTQyNjc1MA== | 9599 | 2021-06-06T16:41:30Z | 2021-06-06T16:44:49Z | OWNER | This is from the current `base.html` template: https://github.com/simonw/datasette/blob/030deb4b25cda842ff7129ab7c18550c44dd8379/datasette/templates/base.html#L62-L66 Which includes this: https://github.com/simonw/datasette/blob/030deb4b25cda842ff7129ab7c18550c44dd8379/datasette/templates/_close_open_menus.html#L1-L16 The `body_scripts` bit is for this `extra_body_script` plugin hook, which is the thing that will be the most affected by implementing CSP: https://docs.datasette.io/en/stable/plugin_hooks.html#extra-body-script-template-database-table-columns-view-name-request-datasette | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
912864936 | |
https://github.com/simonw/datasette/issues/1362#issuecomment-855426516 | https://api.github.com/repos/simonw/datasette/issues/1362 | 855426516 | MDEyOklzc3VlQ29tbWVudDg1NTQyNjUxNg== | 9599 | 2021-06-06T16:39:34Z | 2021-06-06T16:39:34Z | OWNER | The reason Datasette uses small inline scripts right now is to avoid the overhead of an extra HTTP request for a JavaScript file - but these are both inherently cachable and perform much better under HTTP/2 so that's likely a false optimization. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
912864936 | |
https://github.com/simonw/datasette/issues/1362#issuecomment-855426314 | https://api.github.com/repos/simonw/datasette/issues/1362 | 855426314 | MDEyOklzc3VlQ29tbWVudDg1NTQyNjMxNA== | 9599 | 2021-06-06T16:38:04Z | 2021-06-06T16:38:04Z | OWNER | The other option for inline scripts is the CSP nonce: Content-Security-Policy: script-src 'nonce-2726c7f26c' Then: <script nonce="2726c7f26c"> var inline = 1; </script> Since an attacker can't guess what the nonce will be it prevents them from injecting their own script block - this seems easier to make available to plugins than a full hashing mechanism, just make `{{ csp_nonce() }}` available to the template. That template function can then be smart enough to set a flag which Datasette uses to decide if the `script-src 'nonce-2726c7f26c'` policy should be sent or not. Presumably this would also require adding `Content-Security-Policy` to the `Vary` header though, which will have a nasty effect on Cloudflare and Fastly and such like. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
912864936 | |
https://github.com/simonw/datasette/issues/1362#issuecomment-855418899 | https://api.github.com/repos/simonw/datasette/issues/1362 | 855418899 | MDEyOklzc3VlQ29tbWVudDg1NTQxODg5OQ== | 9599 | 2021-06-06T15:42:55Z | 2021-06-06T15:42:55Z | OWNER | Another consideration: testing that this works correctly could require adoption of a real browser test environment (probably Cypress or maybe Playwright) to execute tests that will fail if CSP is violated. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
912864936 | |
https://github.com/simonw/datasette/issues/1362#issuecomment-855418698 | https://api.github.com/repos/simonw/datasette/issues/1362 | 855418698 | MDEyOklzc3VlQ29tbWVudDg1NTQxODY5OA== | 9599 | 2021-06-06T15:41:24Z | 2021-06-06T15:41:24Z | OWNER | I think the best way to answer these questions is with some prototyping - of both Datasette and some of the existing JavaScript plugins. I can start with a `datasette-experimental-csp` plugin that sets the header (and could even run an optional report URI mechanism). | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
912864936 | |
https://github.com/simonw/datasette/issues/1362#issuecomment-855418401 | https://api.github.com/repos/simonw/datasette/issues/1362 | 855418401 | MDEyOklzc3VlQ29tbWVudDg1NTQxODQwMQ== | 9599 | 2021-06-06T15:39:38Z | 2021-06-06T15:39:38Z | OWNER | The security benefit of forcing all JavaScript plugins to be written as CSP-friendly external scripts is very compelling though. Other plugin-heavy ecosystems such as WordPress have suffered greatly from insecurely written plugins - could this be a huge security win for the Datasette ecosystem generally? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
912864936 | |
https://github.com/simonw/datasette/issues/1362#issuecomment-855418065 | https://api.github.com/repos/simonw/datasette/issues/1362 | 855418065 | MDEyOklzc3VlQ29tbWVudDg1NTQxODA2NQ== | 9599 | 2021-06-06T15:37:11Z | 2021-06-06T15:37:11Z | OWNER | The easiest way to apply CSP is to remove all inline `<script>` blocks (Datasette has a few) and instead serve JavaScript as separate linked files. It's possible to keep inline script blocks by calculating a hash of their content and adding a `Content-Security-Policy: script-src 'sha256-B2yPHKaXnvFWtRChIbabYmUBFZdVfKKXHbWtWidDVF8='` to the policy. https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy/script-src This could be achieved with some Django template tricks, but it feels very risky - and done carelessly could end up calculating a hash of a reflected XSS attack! The biggest challenge I see around here involves plugins and custom templates. Adopting CSP would require plugins to avoid using any inline scripts, instead keeping their entire implementations in `.js` files. That's maybe not a bad thing, but it represents a big commitment. It would need to be adopted before Datasette 1.0. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
912864936 | |
https://github.com/simonw/datasette/issues/283#issuecomment-855369819 | https://api.github.com/repos/simonw/datasette/issues/283 | 855369819 | MDEyOklzc3VlQ29tbWVudDg1NTM2OTgxOQ== | 9599 | 2021-06-06T09:40:18Z | 2021-06-06T09:40:18Z | OWNER | > One note on using this pragma I got an error on starting datasette `no such table: pragma_database_list`. > > I diagnosed this to an older version of sqlite3 (3.14.2) and upgrading to a newer version (3.34.2) fixed the issue. That issue is fixed in #1276. | { "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 1, "rocket": 0, "eyes": 0 } |
325958506 | |
https://github.com/simonw/datasette/issues/1361#issuecomment-855308811 | https://api.github.com/repos/simonw/datasette/issues/1361 | 855308811 | MDEyOklzc3VlQ29tbWVudDg1NTMwODgxMQ== | 9599 | 2021-06-05T23:16:21Z | 2021-06-05T23:16:21Z | OWNER | That seems to have fixed it. I'd love to get rid of this `restore_working_directory` hack entirely. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
912485040 | |
https://github.com/simonw/datasette/issues/1361#issuecomment-855307292 | https://api.github.com/repos/simonw/datasette/issues/1361 | 855307292 | MDEyOklzc3VlQ29tbWVudDg1NTMwNzI5Mg== | 9599 | 2021-06-05T22:59:35Z | 2021-06-05T22:59:35Z | OWNER | That broke things. Here's how `pytest-cov` fixed a similar issue: https://github.com/pytest-dev/pytest-cov/commit/7ccb1783bf8290447df58deeb41eae74296a6d9b See also https://github.com/nedbat/coveragepy/issues/881 and https://github.com/pytest-dev/pytest-cov/issues/306 and https://github.com/nedbat/coveragepy/issues/824 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
912485040 | |
https://github.com/simonw/datasette/issues/1361#issuecomment-855306497 | https://api.github.com/repos/simonw/datasette/issues/1361 | 855306497 | MDEyOklzc3VlQ29tbWVudDg1NTMwNjQ5Nw== | 9599 | 2021-06-05T22:51:01Z | 2021-06-05T22:51:01Z | OWNER | I'm going to try removing that `restore_working_directory` fixture entirely. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
912485040 | |
https://github.com/simonw/datasette/issues/1361#issuecomment-855306347 | https://api.github.com/repos/simonw/datasette/issues/1361 | 855306347 | MDEyOklzc3VlQ29tbWVudDg1NTMwNjM0Nw== | 9599 | 2021-06-05T22:49:30Z | 2021-06-05T22:49:30Z | OWNER | Stack Overflow: https://stackoverflow.com/a/49367679/6083 > The answer was that `os.chdir()` had been set to the deleted directory by accident. The directory was missing, but the error happened (it seems) at the attempt to get it with `os.getcwd()`. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
912485040 | |
https://github.com/simonw/datasette/issues/1360#issuecomment-855303776 | https://api.github.com/repos/simonw/datasette/issues/1360 | 855303776 | MDEyOklzc3VlQ29tbWVudDg1NTMwMzc3Ng== | 9599 | 2021-06-05T22:23:23Z | 2021-06-05T22:23:23Z | OWNER | Worth noting that I found this issue myself, and to my knowledge it has not been uncovered by anyone else prior to the patch being released. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
912464443 | |
https://github.com/simonw/datasette/issues/1360#issuecomment-855303649 | https://api.github.com/repos/simonw/datasette/issues/1360 | 855303649 | MDEyOklzc3VlQ29tbWVudDg1NTMwMzY0OQ== | 9599 | 2021-06-05T22:22:06Z | 2021-06-05T22:22:06Z | OWNER | I've released fixes in both 0.56.1 and 0.57. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
912464443 | |
https://github.com/simonw/datasette/issues/1358#issuecomment-855302339 | https://api.github.com/repos/simonw/datasette/issues/1358 | 855302339 | MDEyOklzc3VlQ29tbWVudDg1NTMwMjMzOQ== | 9599 | 2021-06-05T22:08:16Z | 2021-06-05T22:08:16Z | OWNER | Release notes are in, pushing the release now. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
912418094 | |
https://github.com/simonw/datasette/issues/1358#issuecomment-855302320 | https://api.github.com/repos/simonw/datasette/issues/1358 | 855302320 | MDEyOklzc3VlQ29tbWVudDg1NTMwMjMyMA== | 9599 | 2021-06-05T22:08:06Z | 2021-06-05T22:08:06Z | OWNER | See #1360 for the security fix. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
912418094 | |
https://github.com/simonw/datasette/issues/1358#issuecomment-855288228 | https://api.github.com/repos/simonw/datasette/issues/1358 | 855288228 | MDEyOklzc3VlQ29tbWVudDg1NTI4ODIyOA== | 9599 | 2021-06-05T19:57:18Z | 2021-06-05T19:57:18Z | OWNER | There's also a security fix I need to make, so I'm not going to block this on wrapping up the work on the new Docker image testing from #1344 before putting out this release. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
912418094 | |
https://github.com/simonw/datasette/pull/1291#issuecomment-855287200 | https://api.github.com/repos/simonw/datasette/issues/1291 | 855287200 | MDEyOklzc3VlQ29tbWVudDg1NTI4NzIwMA== | 9599 | 2021-06-05T19:48:36Z | 2021-06-05T19:48:36Z | OWNER | This is great, thank you. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
849582643 | |
https://github.com/simonw/datasette/issues/1356#issuecomment-855282466 | https://api.github.com/repos/simonw/datasette/issues/1356 | 855282466 | MDEyOklzc3VlQ29tbWVudDg1NTI4MjQ2Ng== | 9599 | 2021-06-05T19:05:06Z | 2021-06-05T19:05:06Z | OWNER | Yeah that's a good point. I avoided making them sub-commands because `datasette serve` already supports the multitude of other arguments they also need... but actually that was just me being lazy - I can easily share arguments between multiple functions like I do in `sqlite-utils` itself: https://github.com/simonw/sqlite-utils/blob/d1a372b3006e6cf7d2017b3ddc484bf5c033e45d/sqlite_utils/cli.py#L46 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
910092577 | |
https://github.com/simonw/datasette/issues/1238#issuecomment-855278998 | https://api.github.com/repos/simonw/datasette/issues/1238 | 855278998 | MDEyOklzc3VlQ29tbWVudDg1NTI3ODk5OA== | 9599 | 2021-06-05T18:37:16Z | 2021-06-05T18:37:16Z | OWNER | Alternative idea: populate `request.scope` with a new `route_path` which is the base-url-stripped version, which we then use for other routing operations. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
813899472 | |
https://github.com/simonw/datasette/issues/1238#issuecomment-855278540 | https://api.github.com/repos/simonw/datasette/issues/1238 | 855278540 | MDEyOklzc3VlQ29tbWVudDg1NTI3ODU0MA== | 9599 | 2021-06-05T18:33:25Z | 2021-06-05T18:33:25Z | OWNER | Got the test to pass by ensuring the tests don't accidentally double-rewrite the path. Ran into a new problem: ``` @pytest.mark.asyncio @pytest.mark.parametrize( "prefix,expected_path", [(None, "/asgi-scope"), ("/prefix/", "/prefix/asgi-scope")] ) async def test_client_path(datasette, prefix, expected_path): original_base_url = datasette._settings["base_url"] try: if prefix is not None: datasette._settings["base_url"] = prefix response = await datasette.client.get("/asgi-scope") path = response.json()["path"] > assert path == expected_path E AssertionError: assert '/asgi-scope' == '/prefix/asgi-scope' E - /prefix/asgi-scope E ? ------- E + /asgi-scope ``` That test confirms that messing around with the `base_url` doesn't modify the ASGI scope... but the fix I'm using for this issue DOES modify the ASGI scope. The question raised here is: should the ASGI scope stay unmodified when `base_url` is used? I think it should. It doesn't make sense to obscure the "real" path just to get custom pages to work properly. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
813899472 | |
https://github.com/simonw/datasette/issues/1238#issuecomment-855272693 | https://api.github.com/repos/simonw/datasette/issues/1238 | 855272693 | MDEyOklzc3VlQ29tbWVudDg1NTI3MjY5Mw== | 9599 | 2021-06-05T17:45:42Z | 2021-06-05T17:45:42Z | OWNER | Applying this fix worked when I manually tested it: ```diff base_url = self.ds.setting("base_url") if base_url != "/" and path.startswith(base_url): path = "/" + path[len(base_url) :] + scope = dict(scope, path=path, raw_path=path.encode("utf-8")) request = Request(scope, receive) ``` But... the test I wrote still failed. My hunch is that this is because deep within the test framework requests go through `ds.client` which may be applying its own changes relevant to `base_url`: https://github.com/simonw/datasette/blob/6e9b07be92905011211d8df7a872fb7c1f2737b2/datasette/utils/testing.py#L139 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
813899472 | |
https://github.com/simonw/datasette/issues/1238#issuecomment-855270917 | https://api.github.com/repos/simonw/datasette/issues/1238 | 855270917 | MDEyOklzc3VlQ29tbWVudDg1NTI3MDkxNw== | 9599 | 2021-06-05T17:32:29Z | 2021-06-05T17:32:29Z | OWNER | This looks like the cause: https://github.com/simonw/datasette/blob/6e9b07be92905011211d8df7a872fb7c1f2737b2/datasette/app.py#L1087-L1092 Note how `path` is modified... but then we create a new `Request()` that uses the old scope, which has unmodified `scope["path"]` - and then the code later on looks at `request.scope["path"]` when deciding if the request matches: https://github.com/simonw/datasette/blob/afed51b1e36cf275c39e71c7cb262d6c5bdbaa31/datasette/app.py#L1154-L1155 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
813899472 | |
https://github.com/simonw/sqlite-utils/issues/264#issuecomment-853567861 | https://api.github.com/repos/simonw/sqlite-utils/issues/264 | 853567861 | MDEyOklzc3VlQ29tbWVudDg1MzU2Nzg2MQ== | 9599 | 2021-06-03T05:12:21Z | 2021-06-03T05:12:21Z | OWNER | I think this is more likely to happen in Datasette than in sqlite-utils - see https://github.com/simonw/datasette/issues/1356 for thoughts on this. | { "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
907642546 | |
https://github.com/simonw/datasette/issues/526#issuecomment-853567413 | https://api.github.com/repos/simonw/datasette/issues/526 | 853567413 | MDEyOklzc3VlQ29tbWVudDg1MzU2NzQxMw== | 9599 | 2021-06-03T05:11:27Z | 2021-06-03T05:11:27Z | OWNER | Another potential way to implement this would be to hold the SQLite connection open and execute the full query there. I've avoided this in the past due to concerns of resource exhaustion - if multiple requests attempt this at the same time all of the connections in the pool will become tied up and the site will be unable to respond to further requests. But... now that Datasette has authentication there's the possibility of making this feature only available to specific authenticated users - the `--root` user for example. Which avoids the danger while unlocking a super-useful feature. Not to mention people who are running Datasette privately on their own laptop, or the proposed `--query` CLI feature in #1356. | { "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
459882902 | |
https://github.com/simonw/datasette/issues/1356#issuecomment-853566337 | https://api.github.com/repos/simonw/datasette/issues/1356 | 853566337 | MDEyOklzc3VlQ29tbWVudDg1MzU2NjMzNw== | 9599 | 2021-06-03T05:08:32Z | 2021-06-03T05:08:32Z | OWNER | Also relevant: CSV streaming for canned queries in #526 - even better if we could stream ANY SQL query. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
910092577 | |
https://github.com/simonw/datasette/issues/1062#issuecomment-853566011 | https://api.github.com/repos/simonw/datasette/issues/1062 | 853566011 | MDEyOklzc3VlQ29tbWVudDg1MzU2NjAxMQ== | 9599 | 2021-06-03T05:07:42Z | 2021-06-03T05:07:42Z | OWNER | Implementing this would make #1356 a whole lot more interesting. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
732674148 | |
https://github.com/simonw/datasette/issues/1356#issuecomment-853565850 | https://api.github.com/repos/simonw/datasette/issues/1356 | 853565850 | MDEyOklzc3VlQ29tbWVudDg1MzU2NTg1MA== | 9599 | 2021-06-03T05:07:21Z | 2021-06-03T05:07:21Z | OWNER | A problem with this is that if you're using `--query` you likely want ALL of the results - at the moment the only Datasette output type that can stream everything is `.csv` and plugin formats can't handle full streams, see #1062 and #1177. So there's not much point implementing this unless we first make plugins able to add custom streaming formats. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
910092577 | |
https://github.com/simonw/datasette/issues/1356#issuecomment-853562891 | https://api.github.com/repos/simonw/datasette/issues/1356 | 853562891 | MDEyOklzc3VlQ29tbWVudDg1MzU2Mjg5MQ== | 9599 | 2021-06-03T04:59:40Z | 2021-06-03T04:59:40Z | OWNER | It's weird that `--get` is documented on this page right now: https://docs.datasette.io/en/stable/getting_started.html#datasette-get If I implement this I should build a new "Datasette on the command-line" documentation page to cover both `--get` and `--query`. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
910092577 | |
https://github.com/simonw/datasette/issues/1356#issuecomment-853560870 | https://api.github.com/repos/simonw/datasette/issues/1356 | 853560870 | MDEyOklzc3VlQ29tbWVudDg1MzU2MDg3MA== | 9599 | 2021-06-03T04:53:47Z | 2021-06-03T04:53:56Z | OWNER | This is also interesting when used in conjunction with the proposed `datasette insert` command from #1163 - Datasette would become a plugin-driven CLI tool for both ingesting and outputting data, as a side-effect of its web features. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
910092577 | |
https://github.com/simonw/datasette/issues/1356#issuecomment-853560389 | https://api.github.com/repos/simonw/datasette/issues/1356 | 853560389 | MDEyOklzc3VlQ29tbWVudDg1MzU2MDM4OQ== | 9599 | 2021-06-03T04:52:13Z | 2021-06-03T04:52:13Z | OWNER | I should implement #1355 for more efficient `--csv` streaming as part of this. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
910092577 | |
https://github.com/simonw/datasette/issues/1356#issuecomment-853560237 | https://api.github.com/repos/simonw/datasette/issues/1356 | 853560237 | MDEyOklzc3VlQ29tbWVudDg1MzU2MDIzNw== | 9599 | 2021-06-03T04:51:49Z | 2021-06-03T04:51:49Z | OWNER | This feels like a relatively simple feature to implement that unlocks a whole new set of possible uses for Datasette - as described by @eyeseast in https://github.com/simonw/sqlite-utils/issues/264#issue-907642546. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
910092577 | |
https://github.com/simonw/datasette/issues/1356#issuecomment-853559915 | https://api.github.com/repos/simonw/datasette/issues/1356 | 853559915 | MDEyOklzc3VlQ29tbWVudDg1MzU1OTkxNQ== | 9599 | 2021-06-03T04:50:52Z | 2021-06-03T04:50:52Z | OWNER | What happens if you pass multiple databases? The `--query` would be executed against the first one. And if you pass `--crossdb` it would be executed against the `/_memory` database and would support cross-database joins. Key thing here is that output plugins are supported (also plugins that add new SQL functions), making many Datasette plugins usable from the command-line. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
910092577 | |
https://github.com/simonw/sqlite-utils/issues/264#issuecomment-853558741 | https://api.github.com/repos/simonw/sqlite-utils/issues/264 | 853558741 | MDEyOklzc3VlQ29tbWVudDg1MzU1ODc0MQ== | 9599 | 2021-06-03T04:47:19Z | 2021-06-03T04:47:19Z | OWNER | This inspired me to re-examine how `--get` works, hence this issue: https://github.com/simonw/datasette/issues/1355 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
907642546 | |
https://github.com/simonw/datasette/issues/1355#issuecomment-853557439 | https://api.github.com/repos/simonw/datasette/issues/1355 | 853557439 | MDEyOklzc3VlQ29tbWVudDg1MzU1NzQzOQ== | 9599 | 2021-06-03T04:43:14Z | 2021-06-03T04:43:14Z | OWNER | It's using `TestClient` at the moment which is a wrapper around `httpx` (as of ) that uses the `@async_to_sync` decorator to hide the async nature. https://github.com/simonw/datasette/blob/f78ebdc04537a6102316d6dbbf6c887565806078/datasette/utils/testing.py#L102-L156 Maybe the fix here is to switch the `--get` implementation to using `httpx` directly with https://www.python-httpx.org/async/#streaming-responses | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
910088936 | |
https://github.com/simonw/sqlite-utils/issues/264#issuecomment-853553754 | https://api.github.com/repos/simonw/sqlite-utils/issues/264 | 853553754 | MDEyOklzc3VlQ29tbWVudDg1MzU1Mzc1NA== | 9599 | 2021-06-03T04:32:42Z | 2021-06-03T04:36:36Z | OWNER | This is a really interesting thought. I've so far resisted the temptation to add plugins to `sqlite-utils`, partly to avoid overlap with Datasette - but I'm open to discussing it. There's actually a way for you to do what you're describing using `datasette` on the command-line, though it's a little obscure - also Datasette doesn't yet have a GeoJSON output extension, though it really should have one. Here's an example using [datasette-yaml](https://datasette.io/plugins/datasette-yaml): ``` datasette /tmp/covid.db --get='/covid/ny_times_us_counties.yaml' - rowid: 1 date: '2020-01-21' county: Snohomish state: Washington fips: 53061 cases: 1 deaths: 0 - rowid: 2 date: '2020-01-22' county: Snohomish state: Washington fips: 53061 cases: 1 deaths: 0 ``` It even works with arbitrary SQL queries, though you might have to apply URL encoding to the `--get` string (this seems to work though): ``` datasette /tmp/covid.db --get='/covid.yaml?sql=select * from ny_times_us_counties limit 2' - date: '2020-01-21' county: Snohomish state: Washington fips: 53061 cases: 1 deaths: 0 - date: '2020-01-22' county: Snohomish state: Washington fips: 53061 cases: 1 deaths: 0 ``` Here's the documentation for `--get`: https://docs.datasette.io/en/latest/getting_started.html#datasette-get | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
907642546 | |
https://github.com/simonw/sqlite-utils/issues/263#issuecomment-853554550 | https://api.github.com/repos/simonw/sqlite-utils/issues/263 | 853554550 | MDEyOklzc3VlQ29tbWVudDg1MzU1NDU1MA== | 9599 | 2021-06-03T04:34:38Z | 2021-06-03T04:34:38Z | OWNER | Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#listing-indexes | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
906356331 | |
https://github.com/simonw/sqlite-utils/issues/263#issuecomment-853548442 | https://api.github.com/repos/simonw/sqlite-utils/issues/263 | 853548442 | MDEyOklzc3VlQ29tbWVudDg1MzU0ODQ0Mg== | 9599 | 2021-06-03T04:16:49Z | 2021-06-03T04:16:49Z | OWNER | Needs to show the table each index applies to. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
906356331 | |
https://github.com/simonw/sqlite-utils/issues/263#issuecomment-853547681 | https://api.github.com/repos/simonw/sqlite-utils/issues/263 | 853547681 | MDEyOklzc3VlQ29tbWVudDg1MzU0NzY4MQ== | 9599 | 2021-06-03T04:14:28Z | 2021-06-03T04:14:28Z | OWNER | This looks good: ``` % sqlite-utils indexes /tmp/covid.db -t index_name seqno cid name desc coll key ------------------------------------------------------ ------- ----- ----------------- ------ ------ ----- idx_johns_hopkins_csse_daily_reports_combined_key 0 12 combined_key 0 BINARY 1 idx_johns_hopkins_csse_daily_reports_country_or_region 0 1 country_or_region 0 BINARY 1 idx_johns_hopkins_csse_daily_reports_province_or_state 0 2 province_or_state 0 BINARY 1 idx_johns_hopkins_csse_daily_reports_day 0 0 day 0 BINARY 1 idx_ny_times_us_counties_date 0 0 date 1 BINARY 1 idx_ny_times_us_counties_fips 0 3 fips 0 BINARY 1 idx_ny_times_us_counties_county 0 1 county 0 BINARY 1 idx_ny_times_us_counties_state 0 2 state 0 BINARY 1 % sqlite-utils indexes /tmp/covid.db -t --aux index_name seqno cid name desc coll key ------------------------------------------------------ ------- ----- ----------------- ------ ------ ----- idx_johns_hopkins_csse_daily_reports_combined_key 0 12 combined_key 0 BINARY 1 idx_johns_hopkins_csse_daily_reports_combined_key 1 -1 0 BINARY 0 idx_johns_hopkins_csse_daily_reports_country_or_region 0 1 country_or_region 0 BINARY 1 idx_johns_hopkins_csse_daily_reports_country_or_region 1 -1 0 BINARY 0 idx_johns_hopkins_csse_daily_reports_province_or_state 0 2 province_or_state 0 BINARY… | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
906356331 | |
https://github.com/simonw/sqlite-utils/issues/263#issuecomment-853546818 | https://api.github.com/repos/simonw/sqlite-utils/issues/263 | 853546818 | MDEyOklzc3VlQ29tbWVudDg1MzU0NjgxOA== | 9599 | 2021-06-03T04:11:46Z | 2021-06-03T04:11:46Z | OWNER | By default I won't return auxiliary columns, but I'll offer a `--aux` option to return them. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
906356331 | |
https://github.com/simonw/sqlite-utils/issues/263#issuecomment-853545743 | https://api.github.com/repos/simonw/sqlite-utils/issues/263 | 853545743 | MDEyOklzc3VlQ29tbWVudDg1MzU0NTc0Mw== | 9599 | 2021-06-03T04:08:04Z | 2021-06-03T04:08:04Z | OWNER | Figuring out the right queries: https://covid-19.datasettes.com/covid?sql=select+sqlite_master.name%2C+i.*+from+sqlite_master%0D%0Ajoin+pragma_index_list%28sqlite_master.name%29+i%0D%0Awhere+type+%3D+%27table%27 This query shows all columns across all indexes across all tables: ```sql select i.name as index_name, xinfo.* from sqlite_master join pragma_index_list(sqlite_master.name) i join pragma_index_xinfo(index_name) xinfo where sqlite_master.type = 'table' ``` https://covid-19.datasettes.com/covid?sql=select+i.name+as+index_name%2C+xinfo.*+from+sqlite_master%0D%0Ajoin+pragma_index_list%28sqlite_master.name%29+i%0D%0Ajoin+pragma_index_xinfo%28index_name%29+xinfo%0D%0Awhere+sqlite_master.type+%3D+%27table%27 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
906356331 | |
https://github.com/simonw/sqlite-utils/issues/263#issuecomment-853544493 | https://api.github.com/repos/simonw/sqlite-utils/issues/263 | 853544493 | MDEyOklzc3VlQ29tbWVudDg1MzU0NDQ5Mw== | 9599 | 2021-06-03T04:03:59Z | 2021-06-03T04:03:59Z | OWNER | Here's how `sqlite-utils triggers` works: https://github.com/simonw/sqlite-utils/blob/9c67cb925253cd5ef54a1fe0496e0ff9caeacfd6/sqlite_utils/cli.py#L1266-L1277 Running it from a SQL query makes it easy to support modifiers like `--csv` and `-t`. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
906356331 | |
https://github.com/simonw/sqlite-utils/issues/261#issuecomment-853541869 | https://api.github.com/repos/simonw/sqlite-utils/issues/261 | 853541869 | MDEyOklzc3VlQ29tbWVudDg1MzU0MTg2OQ== | 9599 | 2021-06-03T03:54:14Z | 2021-06-03T03:54:14Z | OWNER | Documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#xindexes | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
906345899 | |
https://github.com/simonw/sqlite-utils/issues/261#issuecomment-853535559 | https://api.github.com/repos/simonw/sqlite-utils/issues/261 | 853535559 | MDEyOklzc3VlQ29tbWVudDg1MzUzNTU1OQ== | 9599 | 2021-06-03T03:32:47Z | 2021-06-03T03:32:47Z | OWNER | New design: ```python def test_xindexes(fresh_db): fresh_db.executescript( """ create table Gosh (c1 text, c2 text, c3 text); create index Gosh_c1 on Gosh(c1); create index Gosh_c2c3 on Gosh(c2, c3 desc); """ ) assert fresh_db["Gosh"].xindexes == [ XIndex( name="Gosh_c2c3", columns=[ XIndexColumn(seqno=0, cid=1, name="c2", desc=0, coll="BINARY", key=1), XIndexColumn(seqno=1, cid=2, name="c3", desc=1, coll="BINARY", key=1), XIndexColumn(seqno=2, cid=-1, name=None, desc=0, coll="BINARY", key=0), ], ), XIndex( name="Gosh_c1", columns=[ XIndexColumn(seqno=0, cid=0, name="c1", desc=0, coll="BINARY", key=1), XIndexColumn(seqno=1, cid=-1, name=None, desc=0, coll="BINARY", key=0), ], ), ] ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
906345899 | |
https://github.com/simonw/sqlite-utils/issues/261#issuecomment-853534732 | https://api.github.com/repos/simonw/sqlite-utils/issues/261 | 853534732 | MDEyOklzc3VlQ29tbWVudDg1MzUzNDczMg== | 9599 | 2021-06-03T03:30:10Z | 2021-06-03T03:30:10Z | OWNER | I'm going to return `XIndex(name, columns)` - where `columns` is a list of `XIndexColumn`. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
906345899 | |
https://github.com/simonw/sqlite-utils/issues/261#issuecomment-853530348 | https://api.github.com/repos/simonw/sqlite-utils/issues/261 | 853530348 | MDEyOklzc3VlQ29tbWVudDg1MzUzMDM0OA== | 9599 | 2021-06-03T03:16:33Z | 2021-06-03T03:16:33Z | OWNER | In prototyping this out I realize that I actually want to get back the name of each index, then for each of them the detailed list of index columns. Here's the test from my initial prototype: ```python def test_xindexes(fresh_db): fresh_db.executescript( """ create table Gosh (c1 text, c2 text, c3 text); create index Gosh_c1 on Gosh(c1); create index Gosh_c2c3 on Gosh(c2, c3 desc); """ ) assert fresh_db["Gosh"].xindexes == [ ( "Gosh_c2c3", [ XIndex(seqno=0, cid=1, name="c2", desc=0, coll="BINARY", key=1), XIndex(seqno=1, cid=2, name="c3", desc=1, coll="BINARY", key=1), XIndex(seqno=2, cid=-1, name=None, desc=0, coll="BINARY", key=0), ], ), ( "Gosh_c1", [ XIndex(seqno=0, cid=0, name="c1", desc=0, coll="BINARY", key=1), XIndex(seqno=1, cid=-1, name=None, desc=0, coll="BINARY", key=0), ], ), ] ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
906345899 | |
https://github.com/simonw/sqlite-utils/issues/261#issuecomment-853525036 | https://api.github.com/repos/simonw/sqlite-utils/issues/261 | 853525036 | MDEyOklzc3VlQ29tbWVudDg1MzUyNTAzNg== | 9599 | 2021-06-03T03:02:22Z | 2021-06-03T03:02:22Z | OWNER | This would be a breaking change - and the fact that it returns auxiliary columns isn't particularly useful for most cases - "Auxiliary columns are additional columns needed to locate the table entry that corresponds to each index entry". Instead, I'm going to add a new property `table.xindexes` which exposes this. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
906345899 | |
https://github.com/simonw/datasette/issues/893#issuecomment-852712469 | https://api.github.com/repos/simonw/datasette/issues/893 | 852712469 | MDEyOklzc3VlQ29tbWVudDg1MjcxMjQ2OQ== | 9599 | 2021-06-02T04:29:56Z | 2021-06-02T04:29:56Z | OWNER | I think I fixed this a while ago, though I can't find the commit now. Please re-open if the bug still happens! | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
656959584 | |
https://github.com/simonw/datasette/issues/1127#issuecomment-852712106 | https://api.github.com/repos/simonw/datasette/issues/1127 | 852712106 | MDEyOklzc3VlQ29tbWVudDg1MjcxMjEwNg== | 9599 | 2021-06-02T04:28:55Z | 2021-06-02T04:28:55Z | OWNER | This became resizable in #1236. | { "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 1, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
756818250 | |
https://github.com/simonw/datasette/pull/1306#issuecomment-852710117 | https://api.github.com/repos/simonw/datasette/issues/1306 | 852710117 | MDEyOklzc3VlQ29tbWVudDg1MjcxMDExNw== | 9599 | 2021-06-02T04:24:33Z | 2021-06-02T04:24:33Z | OWNER | I have a test - I'll merge this and then add that test. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
864979486 | |
https://github.com/simonw/datasette/pull/1306#issuecomment-852708574 | https://api.github.com/repos/simonw/datasette/issues/1306 | 852708574 | MDEyOklzc3VlQ29tbWVudDg1MjcwODU3NA== | 9599 | 2021-06-02T04:20:18Z | 2021-06-02T04:20:18Z | OWNER | https://github.com/simonw/datasette/blob/115332ce76c0e867d9936406aaf4bcee6b1ef3cb/datasette/views/index.py#L77 helps here - looks like we can pass `?_sort=relationships` to trigger the bug. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
864979486 | |
https://github.com/simonw/datasette/pull/1306#issuecomment-852707676 | https://api.github.com/repos/simonw/datasette/issues/1306 | 852707676 | MDEyOklzc3VlQ29tbWVudDg1MjcwNzY3Ng== | 9599 | 2021-06-02T04:17:41Z | 2021-06-02T04:17:41Z | OWNER | We need to add a test that illustrates the bug in #1305 so we can prove that this definitely fixes it. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
864979486 | |
https://github.com/simonw/datasette/issues/619#issuecomment-852706970 | https://api.github.com/repos/simonw/datasette/issues/619 | 852706970 | MDEyOklzc3VlQ29tbWVudDg1MjcwNjk3MA== | 9599 | 2021-06-02T04:15:54Z | 2021-06-02T04:15:54Z | OWNER | <img width="1009" alt="fixtures__select___from__foo__and_Facets_should_not_execute_for__shape_array_object_·_Issue__263_·_simonw_datasette" src="https://user-images.githubusercontent.com/9599/120423073-8a338700-c31e-11eb-8c8f-affc24708b32.png"> | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
520655983 | |
https://github.com/simonw/datasette/issues/619#issuecomment-852703357 | https://api.github.com/repos/simonw/datasette/issues/619 | 852703357 | MDEyOklzc3VlQ29tbWVudDg1MjcwMzM1Nw== | 9599 | 2021-06-02T04:08:03Z | 2021-06-02T04:08:03Z | OWNER | A SQL error now looks like this: https://latest.datasette.io/fixtures?sql=select%0D%0A++*%0D%0Afrom%0D%0A++%5Bfoo%5D <img width="1023" alt="fixtures__select___from__foo_" src="https://user-images.githubusercontent.com/9599/120422579-691e6680-c31d-11eb-99ea-655bdc2a401d.png"> I'm going to get rid of that "0 results" message if an error is shown. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
520655983 | |
https://github.com/simonw/datasette/issues/1257#issuecomment-852686827 | https://api.github.com/repos/simonw/datasette/issues/1257 | 852686827 | MDEyOklzc3VlQ29tbWVudDg1MjY4NjgyNw== | 9599 | 2021-06-02T03:26:27Z | 2021-06-02T03:26:27Z | OWNER | I tried and failed to get this fix working for tables with double quotes in their name - I couldn't figure out what the double-quote-in-a-table-name version of this code would look like: https://github.com/simonw/datasette/blob/807de378d08752a0f05bb1b980a0a62620a70520/datasette/utils/__init__.py#L538-L548 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
828811618 | |
https://github.com/simonw/datasette/issues/1257#issuecomment-852681622 | https://api.github.com/repos/simonw/datasette/issues/1257 | 852681622 | MDEyOklzc3VlQ29tbWVudDg1MjY4MTYyMg== | 9599 | 2021-06-02T03:12:18Z | 2021-06-02T03:12:18Z | OWNER | Created a test database like this: ``` % sqlite-utils create-table quote-in-name.db "this'hasquoteinname" id integer name text --pk id % datasette quote-in-name.db -p 8025 --pdb INFO: Started server process [86046] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://127.0.0.1:8025 (Press CTRL+C to quit) > /Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py(530)detect_fts() -> rows = conn.execute(detect_fts_sql(table)).fetchall() (Pdb) c Traceback (most recent call last): File "/Users/simon/Dropbox/Development/datasette/datasette/app.py", line 1124, in route_path response = await view(request, send) File "/Users/simon/Dropbox/Development/datasette/datasette/views/base.py", line 147, in view return await self.dispatch_request( File "/Users/simon/Dropbox/Development/datasette/datasette/views/base.py", line 122, in dispatch_request return await handler(request, *args, **kwargs) File "/Users/simon/Dropbox/Development/datasette/datasette/views/index.py", line 72, in get "fts_table": await db.fts_table(table), File "/Users/simon/Dropbox/Development/datasette/datasette/database.py", line 279, in fts_table return await self.execute_fn(lambda conn: detect_fts(conn, table)) File "/Users/simon/Dropbox/Development/datasette/datasette/database.py", line 155, in execute_fn return await asyncio.get_event_loop().run_in_executor( File "/Users/simon/.pyenv/versions/3.8.2/lib/python3.8/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/Users/simon/Dropbox/Development/datasette/datasette/database.py", line 153, in in_thread return fn(conn) File "/Users/simon/Dropbox/Development/datasette/datasette/database.py", line 279, in <lambda> return await self.execute_fn(lambda conn: detect_fts(conn, table)) File "/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py", line 530, in detect_fts rows = con… | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
828811618 | |
https://github.com/simonw/datasette/pull/1352#issuecomment-852673695 | https://api.github.com/repos/simonw/datasette/issues/1352 | 852673695 | MDEyOklzc3VlQ29tbWVudDg1MjY3MzY5NQ== | 9599 | 2021-06-02T02:52:26Z | 2021-06-02T02:52:26Z | OWNER | @dependabot recreate | { "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
908276134 | |
https://github.com/simonw/datasette/issues/263#issuecomment-852256784 | https://api.github.com/repos/simonw/datasette/issues/263 | 852256784 | MDEyOklzc3VlQ29tbWVudDg1MjI1Njc4NA== | 9599 | 2021-06-01T16:20:09Z | 2021-06-01T16:20:09Z | OWNER | This is a lot easier to implement now that we have a `?_nofacet=1` option from #1350. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
323671577 | |
https://github.com/simonw/datasette/issues/1350#issuecomment-852256454 | https://api.github.com/repos/simonw/datasette/issues/1350 | 852256454 | MDEyOklzc3VlQ29tbWVudDg1MjI1NjQ1NA== | 9599 | 2021-06-01T16:19:38Z | 2021-06-01T16:19:38Z | OWNER | I renamed this to `?_nofacet=1` in #1353. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
906977719 | |
https://github.com/simonw/datasette/issues/1353#issuecomment-852255951 | https://api.github.com/repos/simonw/datasette/issues/1353 | 852255951 | MDEyOklzc3VlQ29tbWVudDg1MjI1NTk1MQ== | 9599 | 2021-06-01T16:18:54Z | 2021-06-01T16:18:54Z | OWNER | Documented here: https://docs.datasette.io/en/latest/json_api.html#special-table-arguments | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
908446997 | |
https://github.com/simonw/datasette/issues/1354#issuecomment-852254298 | https://api.github.com/repos/simonw/datasette/issues/1354 | 852254298 | MDEyOklzc3VlQ29tbWVudDg1MjI1NDI5OA== | 9599 | 2021-06-01T16:16:32Z | 2021-06-01T16:16:32Z | OWNER | Running `python update-docs-help.py` helps fix this. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
908465747 | |
https://github.com/simonw/datasette/issues/1353#issuecomment-852238201 | https://api.github.com/repos/simonw/datasette/issues/1353 | 852238201 | MDEyOklzc3VlQ29tbWVudDg1MjIzODIwMQ== | 9599 | 2021-06-01T15:53:57Z | 2021-06-01T15:53:57Z | OWNER | I'm going to rename `?_nofacets=1` to `?_nofacet=1` to keep it consistent with the new `?_nocount=1` option (and because I don't like `?_nocounts=1`). | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
908446997 | |
https://github.com/simonw/datasette/issues/1349#issuecomment-852237347 | https://api.github.com/repos/simonw/datasette/issues/1349 | 852237347 | MDEyOklzc3VlQ29tbWVudDg1MjIzNzM0Nw== | 9599 | 2021-06-01T15:52:50Z | 2021-06-01T15:52:50Z | OWNER | Fixed in https://github.com/simonw/datasette/commit/d1d06ace49606da790a765689b4fbffa4c6deecb | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
906385991 | |
https://github.com/simonw/datasette/issues/1351#issuecomment-851133471 | https://api.github.com/repos/simonw/datasette/issues/1351 | 851133471 | MDEyOklzc3VlQ29tbWVudDg1MTEzMzQ3MQ== | 9599 | 2021-05-31T03:02:59Z | 2021-05-31T03:02:59Z | OWNER | Since traces only work with `text/html` and JSON at the moment, the easiest way to do this will be to wrap generated CSV in a HTML page in a textarea if the user specified `?_trace=1`: https://github.com/simonw/datasette/blob/c5ae1197a208e1b034c88882e3ac865813a40980/datasette/tracer.py#L125-L134 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
906993731 | |
https://github.com/simonw/datasette/issues/1349#issuecomment-851133125 | https://api.github.com/repos/simonw/datasette/issues/1349 | 851133125 | MDEyOklzc3VlQ29tbWVudDg1MTEzMzEyNQ== | 9599 | 2021-05-31T03:01:48Z | 2021-05-31T03:01:48Z | OWNER | I think it's worth getting `?_trace=1` to work with streaming CSV - this would have helped me spot this issue a long time ago. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
906385991 | |
https://github.com/simonw/datasette/issues/1349#issuecomment-851131999 | https://api.github.com/repos/simonw/datasette/issues/1349 | 851131999 | MDEyOklzc3VlQ29tbWVudDg1MTEzMTk5OQ== | 9599 | 2021-05-31T02:57:25Z | 2021-05-31T02:57:25Z | OWNER | I'm having a really hard time figuring out how to unit test this - ideally I'd monitor which SQL queries are executed using the tracing mechanism, but that's not set up to work with anything other than HTML or JSON outputs: https://github.com/simonw/datasette/blob/c5ae1197a208e1b034c88882e3ac865813a40980/datasette/tracer.py#L125-L134 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
906385991 | |
https://github.com/simonw/datasette/issues/1349#issuecomment-851129464 | https://api.github.com/repos/simonw/datasette/issues/1349 | 851129464 | MDEyOklzc3VlQ29tbWVudDg1MTEyOTQ2NA== | 9599 | 2021-05-31T02:48:06Z | 2021-05-31T02:48:06Z | OWNER | Actually there is precedent for swapping out `request.scope` for a new scope, as seen here in the routing code: https://github.com/simonw/datasette/blob/c5ae1197a208e1b034c88882e3ac865813a40980/datasette/app.py#L1117-L1122 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
906385991 | |
https://github.com/simonw/datasette/issues/1349#issuecomment-851129019 | https://api.github.com/repos/simonw/datasette/issues/1349 | 851129019 | MDEyOklzc3VlQ29tbWVudDg1MTEyOTAxOQ== | 9599 | 2021-05-31T02:46:38Z | 2021-05-31T02:46:38Z | OWNER | I think the right way to do this is to construct a new `Request` with a modified ASGI scope: https://github.com/simonw/datasette/blob/c5ae1197a208e1b034c88882e3ac865813a40980/datasette/utils/asgi.py#L88-L99 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
906385991 | |
https://github.com/simonw/datasette/issues/1349#issuecomment-851128524 | https://api.github.com/repos/simonw/datasette/issues/1349 | 851128524 | MDEyOklzc3VlQ29tbWVudDg1MTEyODUyNA== | 9599 | 2021-05-31T02:44:44Z | 2021-05-31T02:44:44Z | OWNER | Now that I have `?_nofacets=1` I can use that to fix this. The challenge is that in this block of code I need to modify the incoming request's query string arguments, which isn't something I've done before: https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/views/base.py#L263-L270 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
906385991 | |
https://github.com/simonw/datasette/issues/1345#issuecomment-851127314 | https://api.github.com/repos/simonw/datasette/issues/1345 | 851127314 | MDEyOklzc3VlQ29tbWVudDg1MTEyNzMxNA== | 9599 | 2021-05-31T02:40:26Z | 2021-05-31T02:40:44Z | OWNER | Demo: https://latest.datasette.io/fixtures/facetable?_facet=state&_nocol=state - the state column is not selected but facet by state still works: <img width="1134" alt="fixtures__facetable__15_rows_and_Why_Russians_do_not_smile" src="https://user-images.githubusercontent.com/9599/120132273-110a2780-c198-11eb-91a5-cafb77a78c97.png"> | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
904071938 | |
https://github.com/simonw/datasette/issues/1350#issuecomment-851122934 | https://api.github.com/repos/simonw/datasette/issues/1350 | 851122934 | MDEyOklzc3VlQ29tbWVudDg1MTEyMjkzNA== | 9599 | 2021-05-31T02:23:47Z | 2021-05-31T02:23:47Z | OWNER | I think `?_facets_off=1` is a good design for this. I considered `?_facet=` with an empty string but that's a bit weird, and it's not clear what that would do if used like `?_facet=state&_facet=`. So it definitely needs to be a separate named option. Actually, since I have `?_nocol=` I'm going to call this `?_nofacets=1` instead as that's a little more consistent. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
906977719 | |
https://github.com/simonw/datasette/issues/1345#issuecomment-851120306 | https://api.github.com/repos/simonw/datasette/issues/1345 | 851120306 | MDEyOklzc3VlQ29tbWVudDg1MTEyMDMwNg== | 9599 | 2021-05-31T02:14:36Z | 2021-05-31T02:14:36Z | OWNER | Yes! This was easier than I thought. I'm going with that solution - where facets are calculated against all columns, ignoring `?_col=` and `?_nocol=` entirely. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
904071938 | |
https://github.com/simonw/datasette/issues/1345#issuecomment-851117233 | https://api.github.com/repos/simonw/datasette/issues/1345 | 851117233 | MDEyOklzc3VlQ29tbWVudDg1MTExNzIzMw== | 9599 | 2021-05-31T02:04:35Z | 2021-05-31T02:04:35Z | OWNER | That long-term solution may not be too difficult. The facets are calculated against `sql_no_limit` which is constructed here: https://github.com/simonw/datasette/blob/7b106e106000713bbee31b34d694b3dadbd4818c/datasette/views/table.py#L659-L665 And used here: https://github.com/simonw/datasette/blob/7b106e106000713bbee31b34d694b3dadbd4818c/datasette/views/table.py#L706-L718 Crucially, `sql_no_limit` is ONLY used for faceting - nothing else uses it anywhere. So constructing it before constructing `sql` and taking `?_col=` and `?_nocol=` into account may not be a complex change. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
904071938 | |
https://github.com/simonw/datasette/issues/1345#issuecomment-851116105 | https://api.github.com/repos/simonw/datasette/issues/1345 | 851116105 | MDEyOklzc3VlQ29tbWVudDg1MTExNjEwNQ== | 9599 | 2021-05-31T02:00:44Z | 2021-05-31T02:00:44Z | OWNER | Maybe there's a short-term and longer-term solution for this - where the long-term solution is to use different columns in the faceting selects, while the short-term solution is to disable "Hide this column" for certain things. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
904071938 | |
https://github.com/simonw/datasette/issues/1345#issuecomment-851115805 | https://api.github.com/repos/simonw/datasette/issues/1345 | 851115805 | MDEyOklzc3VlQ29tbWVudDg1MTExNTgwNQ== | 9599 | 2021-05-31T01:59:39Z | 2021-05-31T01:59:39Z | OWNER | Maybe there's a concept here of the columns that are required by a selected facet? Those can then be included as `data-` attributes on the page, which will then impact which "Hide this column" options are available. I can also use them to provide a better error message than "no such column: state" - I can verify that `?_nocol` and `?_col` have not been used to disable the required columns. There is one other option here: I could still include the columns that are known to be needed for faceting in the faceting SQL queries, but leave them out of the query that is used to return the results! That's actually a pretty tempting (albeit more complex) option. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
904071938 | |
https://github.com/simonw/datasette/issues/1345#issuecomment-851114806 | https://api.github.com/repos/simonw/datasette/issues/1345 | 851114806 | MDEyOklzc3VlQ29tbWVudDg1MTExNDgwNg== | 9599 | 2021-05-31T01:55:56Z | 2021-05-31T01:55:56Z | OWNER | Two options here: - Don't provide users with options that will lead to this situation - so no "Hide this column" option on pages that are already faceted by that column - Ignore facet selections for columns which are no longer visible I think I like the first option more. I could partially implement that in the `table.js` JavaScript by looking at the `?_facet=` parameters... but that won't cover the case where the facet is happening because of default facets configured in `metadata.yml`. Instead the JavaScript should look for evidence in the DOM that specific facets are enabled. This could also help me cover other types of faceting, such as `?_facet_array=` or even custom facets provided by plugins. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
904071938 | |
https://github.com/simonw/datasette/issues/1345#issuecomment-851113886 | https://api.github.com/repos/simonw/datasette/issues/1345 | 851113886 | MDEyOklzc3VlQ29tbWVudDg1MTExMzg4Ng== | 9599 | 2021-05-31T01:52:27Z | 2021-05-31T01:52:27Z | OWNER | Related issue: visit https://latest.datasette.io/fixtures/facetable?_facet=state and click "Hide this column" on the "state" cog menu and you get https://latest.datasette.io/fixtures/facetable?_facet=state&_nocol=state which shows an error: > ### Invalid SQL > no such column: state | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
904071938 | |
https://github.com/simonw/datasette/issues/1349#issuecomment-850778311 | https://api.github.com/repos/simonw/datasette/issues/1349 | 850778311 | MDEyOklzc3VlQ29tbWVudDg1MDc3ODMxMQ== | 9599 | 2021-05-29T06:12:21Z | 2021-05-29T06:12:21Z | OWNER | It's not just facets, I think it's trying to execute suggested facets too! | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
906385991 | |
https://github.com/simonw/datasette/issues/1349#issuecomment-850778273 | https://api.github.com/repos/simonw/datasette/issues/1349 | 850778273 | MDEyOklzc3VlQ29tbWVudDg1MDc3ODI3Mw== | 9599 | 2021-05-29T06:11:59Z | 2021-05-29T06:11:59Z | OWNER | Related issue: https://github.com/simonw/datasette/issues/263 - "Facets should not execute for ?shape=array|object" | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
906385991 | |
https://github.com/simonw/datasette/issues/1349#issuecomment-850778230 | https://api.github.com/repos/simonw/datasette/issues/1349 | 850778230 | MDEyOklzc3VlQ29tbWVudDg1MDc3ODIzMA== | 9599 | 2021-05-29T06:11:28Z | 2021-05-29T06:11:28Z | OWNER | Relevant code: https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/views/base.py#L263-L315 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
906385991 | |
https://github.com/simonw/sqlite-utils/issues/250#issuecomment-850772428 | https://api.github.com/repos/simonw/sqlite-utils/issues/250 | 850772428 | MDEyOklzc3VlQ29tbWVudDg1MDc3MjQyOA== | 9599 | 2021-05-29T05:16:02Z | 2021-05-29T05:28:57Z | OWNER | I needed to find some CSV files on my computer with a BOM at the beginning - I figured out this recipe: ``` % rg -U -E none '^(?-u:\xEF\xBB\xBF)' --glob '*.csv' . ``` TIL here: https://til.simonwillison.net/bash/finding-bom-csv-files-with-ripgrep | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
838148087 | |
https://github.com/simonw/sqlite-utils/issues/250#issuecomment-850771264 | https://api.github.com/repos/simonw/sqlite-utils/issues/250 | 850771264 | MDEyOklzc3VlQ29tbWVudDg1MDc3MTI2NA== | 9599 | 2021-05-29T05:06:13Z | 2021-05-29T05:06:13Z | OWNER | The other option is to check if the file starts with `codecs.BOM_UTF8` - which is `b'\xef\xbb\xbf'`. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
838148087 | |
https://github.com/simonw/sqlite-utils/issues/250#issuecomment-850771017 | https://api.github.com/repos/simonw/sqlite-utils/issues/250 | 850771017 | MDEyOklzc3VlQ29tbWVudDg1MDc3MTAxNw== | 9599 | 2021-05-29T05:04:28Z | 2021-05-29T05:04:28Z | OWNER | https://stackoverflow.com/a/44573867/6083 says: > There is no reason to check if a BOM exists or not, utf-8-sig manages that for you and behaves exactly as utf-8 if the BOM does not exist | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
838148087 | |
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850769067 | https://api.github.com/repos/simonw/sqlite-utils/issues/260 | 850769067 | MDEyOklzc3VlQ29tbWVudDg1MDc2OTA2Nw== | 9599 | 2021-05-29T04:48:10Z | 2021-05-29T04:48:10Z | OWNER | I confirmed and it's possible to have a SQLite column with a hyphen at the start, confirmed using: ``` % sqlite-utils create-table demo.db demo -- id integer name text -blah integer % sqlite-utils tables --schema demo.db -t table schema ------- --------------------- demo CREATE TABLE [demo] ( [id] INTEGER, [name] TEXT, [-blah] INTEGER ) ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
906330187 | |
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850768315 | https://api.github.com/repos/simonw/sqlite-utils/issues/260 | 850768315 | MDEyOklzc3VlQ29tbWVudDg1MDc2ODMxNQ== | 9599 | 2021-05-29T04:39:33Z | 2021-05-29T04:39:33Z | OWNER | This doesn't work: ``` sqlite-utils create-index mydb.db dogs "-age" name ``` But this does: ``` sqlite-utils create-index mydb.db dogs -- -age name ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
906330187 | |
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850767210 | https://api.github.com/repos/simonw/sqlite-utils/issues/260 | 850767210 | MDEyOklzc3VlQ29tbWVudDg1MDc2NzIxMA== | 9599 | 2021-05-29T04:26:26Z | 2021-05-29T04:28:31Z | OWNER | It's weird having to use `Database.DescIndex` - I'm going to put `DescIndex` in `sqlite_utils.db` directly and let people import it. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
906330187 | |
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850766552 | https://api.github.com/repos/simonw/sqlite-utils/issues/260 | 850766552 | MDEyOklzc3VlQ29tbWVudDg1MDc2NjU1Mg== | 9599 | 2021-05-29T04:20:40Z | 2021-05-29T04:24:01Z | OWNER | `PRAGMA index_xinfo(table)` DOES return that data: ``` (Pdb) [c[0] for c in fresh_db.execute("PRAGMA index_xinfo('idx_dogs_age_name')").description] ['seqno', 'cid', 'name', 'desc', 'coll', 'key'] (Pdb) fresh_db.execute("PRAGMA index_xinfo('idx_dogs_age_name')").fetchall() [(0, 2, 'age', 1, 'BINARY', 1), (1, 0, 'name', 0, 'BINARY', 1), (2, -1, None, 0, 'BINARY', 0)] ``` See https://sqlite.org/pragma.html#pragma_index_xinfo Example output: https://covid-19.datasettes.com/covid?sql=select+*+from+pragma_index_xinfo%28%27idx_ny_times_us_counties_date%27%29 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
906330187 | |
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850766335 | https://api.github.com/repos/simonw/sqlite-utils/issues/260 | 850766335 | MDEyOklzc3VlQ29tbWVudDg1MDc2NjMzNQ== | 9599 | 2021-05-29T04:18:19Z | 2021-05-29T04:18:19Z | OWNER | Annoyingly the `table.indexes` property won't indicate if an index is in regular or reverse order - because the SQLite `PRAGMA index_info(table)` statement doesn't indicate that either. You have to look at the `sqlite_master` index definition to tell if any of the columns are in reverse order: ``` (Pdb) fresh_db.execute("select * from sqlite_master where type = 'index'").fetchall() [('index', 'idx_dogs_age_name', 'dogs', 3, 'CREATE INDEX [idx_dogs_age_name]\n ON [dogs] ([age] desc, [name])')] (Pdb) fresh_db.execute("PRAGMA index_info('idx_dogs_age_name')").fetchall() [(0, 2, 'age'), (1, 0, 'name')] (Pdb) fresh_db.execute("PRAGMA index_info('idx_dogs_age_name')").description (('seqno', None, None, None, None, None, None), ('cid', None, None, None, None, None, None), ('name', None, None, None, None, None, None)) (Pdb) dogs.indexes [Index(seq=0, name='idx_dogs_age_name', unique=0, origin='c', partial=0, columns=['age', 'name'])] ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
906330187 | |
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850765450 | https://api.github.com/repos/simonw/sqlite-utils/issues/260 | 850765450 | MDEyOklzc3VlQ29tbWVudDg1MDc2NTQ1MA== | 9599 | 2021-05-29T04:09:13Z | 2021-05-29T04:09:13Z | OWNER | Decisions: for the Python API I'm going with `db.DescIndex("column")` as the way to do this. For the CLI I'm going to do the "-age" thing. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
906330187 | |
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850765291 | https://api.github.com/repos/simonw/sqlite-utils/issues/260 | 850765291 | MDEyOklzc3VlQ29tbWVudDg1MDc2NTI5MQ== | 9599 | 2021-05-29T04:07:48Z | 2021-05-29T04:08:21Z | OWNER | For the CLI version I could say that you can use a `-` prefix to specify reverse direction: ```sh sqlite-utils create-index mydb.db dogs -age name ``` No, that doesn't work - it could get confused with a command-line flag. I guess you could do this: ``` sqlite-utils create-index mydb.db dogs "-age" name ``` This does mean that if any of your column names begin with a hyphen you can't use the CLI to add indexes to them. Is that an acceptable limitation? Users can always use `sqlite-utils mydb.db "create index ..."` in that case. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
906330187 | |
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850765050 | https://api.github.com/repos/simonw/sqlite-utils/issues/260 | 850765050 | MDEyOklzc3VlQ29tbWVudDg1MDc2NTA1MA== | 9599 | 2021-05-29T04:05:24Z | 2021-05-29T04:05:40Z | OWNER | Need to solve this for the CLI tool too. Currently that works like this: https://sqlite-utils.datasette.io/en/stable/cli.html#creating-indexes ```sh sqlite-utils create-index mydb.db mytable col1 [col2...] ``` Even harder to decide how to add a descending option to this. Maybe like this? ```sh sqlite-utils create-index mydb.db mytable --direction col1 asc --direction col2 desc ``` It's a bit gross though! We're saying here that if a single one of the columns you are creating an index for is in reverse direction you have to use `--direction` to specify each end every other index. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
906330187 | |
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850764700 | https://api.github.com/repos/simonw/sqlite-utils/issues/260 | 850764700 | MDEyOklzc3VlQ29tbWVudDg1MDc2NDcwMA== | 9599 | 2021-05-29T04:02:10Z | 2021-05-29T04:02:10Z | OWNER | I could use `db.desc_index("age")` to match SQLite SQL syntax, which uses `desc` and not `descending`. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
906330187 | |
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850764655 | https://api.github.com/repos/simonw/sqlite-utils/issues/260 | 850764655 | MDEyOklzc3VlQ29tbWVudDg1MDc2NDY1NQ== | 9599 | 2021-05-29T04:01:41Z | 2021-05-29T04:01:41Z | OWNER | Maybe: ```python db["dogs"].create_index([db.descending_index("age"), "name"]) ``` It's a little verbose but it's for a relatively rare activity and it does make it very clear what is going on. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
906330187 | |
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850764594 | https://api.github.com/repos/simonw/sqlite-utils/issues/260 | 850764594 | MDEyOklzc3VlQ29tbWVudDg1MDc2NDU5NA== | 9599 | 2021-05-29T04:00:54Z | 2021-05-29T04:00:54Z | OWNER | A few options: ```python db["dogs"].create_index([("age", "desc"), "name"]) db["dogs"].create_index([desc("age"), "name"]) db["dogs"].create_index([db.desc("age"), "name"]) ``` The first option uses an optional tuple. The second two use a `desc()` function - the question is where should that live? `sqlite_utils.desc(column)` or `db.desc(column)` are both options. I don't like using the term `desc()` for "descending index" though - it feels like it should mean something more broad. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
906330187 | |
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850764253 | https://api.github.com/repos/simonw/sqlite-utils/issues/260 | 850764253 | MDEyOklzc3VlQ29tbWVudDg1MDc2NDI1Mw== | 9599 | 2021-05-29T03:57:54Z | 2021-05-29T03:57:54Z | OWNER | The problem here is differentiating between a column with the name `date desc` and wanting to create a descending index on a column called `date`. This won't work: ```python db["ny_times_us_counties"].create_index(["date desc"], desc=True) ``` Because we need to be able to create compound indexes with columns with different directions - for example: ```sql create index idx_age_desc_name on dogs (age desc, name) ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
906330187 | |
https://github.com/simonw/datasette/pull/1296#issuecomment-850583584 | https://api.github.com/repos/simonw/datasette/issues/1296 | 850583584 | MDEyOklzc3VlQ29tbWVudDg1MDU4MzU4NA== | 9599 | 2021-05-28T18:06:11Z | 2021-05-28T18:06:11Z | OWNER | > As a bonus, the Docker image becomes smaller That's a huge surprise to me! And most welcome. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
855446829 | |
https://github.com/simonw/datasette/pull/1346#issuecomment-850059328 | https://api.github.com/repos/simonw/datasette/issues/1346 | 850059328 | MDEyOklzc3VlQ29tbWVudDg1MDA1OTMyOA== | 9599 | 2021-05-28T02:08:10Z | 2021-05-28T02:09:25Z | OWNER | This is the current test failure, but it actually another problem that we don't have tests in place for errors with different formats and shapes: ``` def test_magic_parameters_cannot_be_used_in_arbitrary_queries(magic_parameters_client): response = magic_parameters_client.get( "/data.json?sql=select+:_header_host&_shape=array" ) assert 400 == response.status > assert "You did not supply a value for binding 1." == response.json["error"] E TypeError: list indices must be integers or slices, not str ``` The test fails because `response.json` here is the empty list `[]`. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
904537568 | |
https://github.com/simonw/datasette/issues/619#issuecomment-850059607 | https://api.github.com/repos/simonw/datasette/issues/619 | 850059607 | MDEyOklzc3VlQ29tbWVudDg1MDA1OTYwNw== | 9599 | 2021-05-28T02:08:58Z | 2021-05-28T02:08:58Z | OWNER | Remaining work will happen in #1346 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
520655983 | |
https://github.com/simonw/datasette/pull/1346#issuecomment-850058851 | https://api.github.com/repos/simonw/datasette/issues/1346 | 850058851 | MDEyOklzc3VlQ29tbWVudDg1MDA1ODg1MQ== | 9599 | 2021-05-28T02:06:46Z | 2021-05-28T02:06:46Z | OWNER | The one test failure here actually illustrates a larger problem: if the user specifies `.json?_shape=array` but an error occurs, what should we do? Prior to this change we return the following JSON: ```json { "ok": false, "error": "You did not supply a value for binding 1.", "status": 500, "title": null } ``` But this comes from the `handle_500` higher level code here: https://github.com/simonw/datasette/blob/eae3084b46e2c3931db12cdef79093ad0e644bce/datasette/app.py#L1251-L1263 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
904537568 | |
https://github.com/simonw/datasette/issues/619#issuecomment-850057694 | https://api.github.com/repos/simonw/datasette/issues/619 | 850057694 | MDEyOklzc3VlQ29tbWVudDg1MDA1NzY5NA== | 9599 | 2021-05-28T02:03:05Z | 2021-05-28T02:03:05Z | OWNER | I nearly got this working, but I ran into one last problem: the code path for when an error is raised but the user specified `?_shape=array`. I'll open a draft PR with where I've got to so far. | { "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 1, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
520655983 |