github
html_url | issue_url | id | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
https://github.com/simonw/datasette/issues/1875#issuecomment-1314491150 | https://api.github.com/repos/simonw/datasette/issues/1875 | 1314491150 | IC_kwDOBm6k_c5OWYsO | 9599 | 2022-11-14T22:25:20Z | 2022-11-14T22:25:20Z | OWNER | That's using JSON Pointer: https://www.rfc-editor.org/rfc/rfc6901 There's a Python library for that here https://github.com/stefankoegl/python-json-pointer/blob/master/jsonpointer.py - which looks simple and clean and well maintained and documented, but it only handles the "what is at this pointer within this JSON object" case - I need to generate the correct JSON pointer to explain where my error is. So I think I'll end up hand-rolling this. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1430797211 | |
https://github.com/simonw/datasette/issues/1875#issuecomment-1314488010 | https://api.github.com/repos/simonw/datasette/issues/1875 | 1314488010 | IC_kwDOBm6k_c5OWX7K | 9599 | 2022-11-14T22:21:43Z | 2022-11-14T22:21:43Z | OWNER | Here's the most relevant example from the RFC spec: ``` POST /details HTTP/1.1 Host: account.example.com Accept: application/json ``` ```json { "age": 42.3, "profile": { "color": "yellow" } } ``` ``` HTTP/1.1 400 Bad Request Content-Type: application/problem+json Content-Language: en ``` ```json { "type": "https://example.net/validation-error", "title": "Your request is not valid.", "errors": [ { "detail": "must be a positive integer", "pointer": "#/age" }, { "detail": "must be 'green', 'red' or 'blue'", "pointer": "#/profile/color" } ] } ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1430797211 | |
https://github.com/simonw/datasette/issues/1884#issuecomment-1314054300 | https://api.github.com/repos/simonw/datasette/issues/1884 | 1314054300 | IC_kwDOBm6k_c5OUuCc | 9599 | 2022-11-14T16:40:06Z | 2022-11-14T16:40:06Z | OWNER | I wonder if there are any reasons that inspect SHOULD try to count virtual tables? Like are there any likely uses for a cirial table where the count is both interesting and likely to be accessed often enough that it's worth caching? I have an issue open to add a setting to disable table counts entirely: - #1818 Maybe that should be expanded to automatically disable row counts for virtual tables entirely? Which would mean no count would be shown for them in the UI. If you desperately wanted a count you would then have to run a count(*) query against them explicitly. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1439009231 | |
https://github.com/simonw/datasette/issues/1850#issuecomment-1313156167 | https://api.github.com/repos/simonw/datasette/issues/1850 | 1313156167 | IC_kwDOBm6k_c5ORSxH | 9599 | 2022-11-14T06:23:39Z | 2022-11-14T06:23:39Z | OWNER | The API explorer is now live here: https://latest-1-0-dev.datasette.io/-/api | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1421529723 | |
https://github.com/simonw/datasette/issues/1850#issuecomment-1313155712 | https://api.github.com/repos/simonw/datasette/issues/1850 | 1313155712 | IC_kwDOBm6k_c5ORSqA | 9599 | 2022-11-14T06:22:57Z | 2022-11-14T06:22:57Z | OWNER | I think the ability to create tokens should be protected by a `create-tokens` permission, not just a global setting. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1421529723 | |
https://github.com/simonw/datasette/issues/1888#issuecomment-1313139657 | https://api.github.com/repos/simonw/datasette/issues/1888 | 1313139657 | IC_kwDOBm6k_c5OROvJ | 9599 | 2022-11-14T06:04:48Z | 2022-11-14T06:04:48Z | OWNER | Demo: https://latest-1-0-dev.datasette.io/-/api | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1447439985 | |
https://github.com/simonw/datasette/issues/1866#issuecomment-1313128913 | https://api.github.com/repos/simonw/datasette/issues/1866 | 1313128913 | IC_kwDOBm6k_c5ORMHR | 9599 | 2022-11-14T05:48:22Z | 2022-11-14T05:48:22Z | OWNER | I changed my mind about the `"return_rows": true` option - I'm going to rename it to `"return": true`. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1426001541 | |
https://github.com/simonw/datasette/issues/1874#issuecomment-1313127054 | https://api.github.com/repos/simonw/datasette/issues/1874 | 1313127054 | IC_kwDOBm6k_c5ORLqO | 9599 | 2022-11-14T05:45:00Z | 2022-11-14T05:45:00Z | OWNER | Demo: https://latest-1-0-dev.datasette.io/-/api#path=%2Ffixtures%2Ffacetable%2F-%2Fdrop&json=&method=POST <img width="974" alt="image" src="https://user-images.githubusercontent.com/9599/201584446-fd9d3978-d37d-4e81-8179-cd587027df46.png"> | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1429030341 | |
https://github.com/simonw/datasette/issues/1871#issuecomment-1313125870 | https://api.github.com/repos/simonw/datasette/issues/1871 | 1313125870 | IC_kwDOBm6k_c5ORLXu | 9599 | 2022-11-14T05:42:50Z | 2022-11-14T05:42:50Z | OWNER | Demo: https://latest-1-0-dev.datasette.io/-/api#path=%2Ffixtures%2Ffacetable%2F-%2Fdrop&json=%7B%22confirm%22%3A+true%7D&method=POST | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1427293909 | |
https://github.com/simonw/datasette/issues/1874#issuecomment-1313125123 | https://api.github.com/repos/simonw/datasette/issues/1874 | 1313125123 | IC_kwDOBm6k_c5ORLMD | 9599 | 2022-11-14T05:41:20Z | 2022-11-14T05:42:23Z | OWNER | I also changed the confirmation JSON returned by this endpoint to add the `database` and `table` like so: ```json { "ok": true, "database": "data", "table": "docs", "row_count": 1, "message": "Pass \"confirm\": true to confirm" } ``` Updated docs: https://docs.datasette.io/en/1.0-dev/json_api.html#dropping-tables | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1429030341 | |
https://github.com/simonw/datasette/issues/1874#issuecomment-1313119558 | https://api.github.com/repos/simonw/datasette/issues/1874 | 1313119558 | IC_kwDOBm6k_c5ORJ1G | 9599 | 2022-11-14T05:30:27Z | 2022-11-14T05:30:27Z | OWNER | Found a bug: you get a 500 error if you try this against an immutable database. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1429030341 | |
https://github.com/simonw/datasette/issues/1850#issuecomment-1313115059 | https://api.github.com/repos/simonw/datasette/issues/1850 | 1313115059 | IC_kwDOBm6k_c5ORIuz | 9599 | 2022-11-14T05:21:30Z | 2022-11-14T05:21:30Z | OWNER | New documentation for these features currently lives here: https://docs.datasette.io/en/1.0-dev/json_api.html#the-json-write-api | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1421529723 | |
https://github.com/simonw/datasette/issues/1875#issuecomment-1313114283 | https://api.github.com/repos/simonw/datasette/issues/1875 | 1313114283 | IC_kwDOBm6k_c5ORIir | 9599 | 2022-11-14T05:20:00Z | 2022-11-14T05:20:00Z | OWNER | I started a conversation about JSON error standards on Mastodon here: https://fedi.simonwillison.net/web/@simon/109338725610487457 Quite a few people pointed to this RFC independently. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1430797211 | |
https://github.com/simonw/datasette/issues/1887#issuecomment-1313113642 | https://api.github.com/repos/simonw/datasette/issues/1887 | 1313113642 | IC_kwDOBm6k_c5ORIYq | 9599 | 2022-11-14T05:18:51Z | 2022-11-14T05:18:51Z | OWNER | Updated docs: https://docs.datasette.io/en/1.0-dev/json_api.html#dropping-tables | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1447388809 | |
https://github.com/simonw/datasette/issues/1887#issuecomment-1313097713 | https://api.github.com/repos/simonw/datasette/issues/1887 | 1313097713 | IC_kwDOBm6k_c5OREfx | 9599 | 2022-11-14T05:00:54Z | 2022-11-14T05:00:54Z | OWNER | I'm going to add a `"confirm": true` option to the API. Without that, it returns a note about how many rows will be deleted. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1447388809 | |
https://github.com/simonw/datasette/issues/1871#issuecomment-1313097057 | https://api.github.com/repos/simonw/datasette/issues/1871 | 1313097057 | IC_kwDOBm6k_c5OREVh | 9599 | 2022-11-14T04:59:28Z | 2022-11-14T04:59:28Z | OWNER | In playing with the API explorer just now I realized it's way too easy to accidentally drop a table using it. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1427293909 | |
https://github.com/simonw/datasette/issues/1871#issuecomment-1313072900 | https://api.github.com/repos/simonw/datasette/issues/1871 | 1313072900 | IC_kwDOBm6k_c5OQ-cE | 9599 | 2022-11-14T04:15:50Z | 2022-11-14T04:15:50Z | OWNER | For the example links - I'm going to have these at the bottom of the page so you don't have to scroll past them. Ideally these would take the user's permissions into account. This could make the page expensive to load, but I'm going to risk it for the moment. Something like this then: > - data > - /data/-/create - create table > - /data/table1/-/insert - insert into table1 > - /data/table1/-/drop - drop table1 I won't bother with per-row demo links (for update and delete) because there could be thousands of them for each table. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1427293909 | |
https://github.com/simonw/datasette/issues/1871#issuecomment-1313062699 | https://api.github.com/repos/simonw/datasette/issues/1871 | 1313062699 | IC_kwDOBm6k_c5OQ78r | 9599 | 2022-11-14T04:03:29Z | 2022-11-14T04:12:41Z | OWNER | Two things left before I close this issue: - [x] I want to preserve the state of the forms in the URL - probably after a `#` - [ ] Instead of hard-coding the current examples, I want to provide a list of links which populate the forms | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1427293909 | |
https://github.com/simonw/datasette/issues/1886#issuecomment-1313052863 | https://api.github.com/repos/simonw/datasette/issues/1886 | 1313052863 | IC_kwDOBm6k_c5OQ5i_ | 9599 | 2022-11-14T03:40:50Z | 2022-11-14T03:40:50Z | OWNER | Tim Sherratt on Twitter: https://twitter.com/wragge/status/1591930345469153282 > Where do I start? The [#GLAMWorkbench](https://twitter.com/hashtag/GLAMWorkbench?src=hashtag_click) now includes a number of examples where GLAM data is harvested, processed, and then made available for exploration via Datasette. > > https://glam-workbench.net/ > > For example the GLAM Name Index Search brings together 10+ million entries from 240 indexes and provides an aggregated search using the Datasette search-all plugin: > > https://glam-workbench.net/name-search/ > > Most recently I converted PDFs of the Tasmanian Postal Directories to a big Datasette instance: https://updates.timsherratt.org/2022/09/15/from-pdfs-to.html the process is documented and reusable. | { "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 1, "rocket": 0, "eyes": 0 } |
1447050738 | |
https://github.com/simonw/datasette/issues/1871#issuecomment-1312822353 | https://api.github.com/repos/simonw/datasette/issues/1871 | 1312822353 | IC_kwDOBm6k_c5OQBRR | 9599 | 2022-11-13T21:07:40Z | 2022-11-13T21:07:40Z | OWNER | I'm going to need extra code to toggle POST closed when GET opens and vice-versa. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1427293909 | |
https://github.com/simonw/datasette/issues/1871#issuecomment-1312821031 | https://api.github.com/repos/simonw/datasette/issues/1871 | 1312821031 | IC_kwDOBm6k_c5OQA8n | 9599 | 2022-11-13T21:02:06Z | 2022-11-13T21:03:11Z | OWNER | Actually no, I'm going to add a class of `details-menu` to the other details elements that SHOULD be closed. That way custom templates using `<details>` won't close in a surprising way. | { "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1427293909 | |
https://github.com/simonw/datasette/issues/1871#issuecomment-1312816451 | https://api.github.com/repos/simonw/datasette/issues/1871 | 1312816451 | IC_kwDOBm6k_c5OP_1D | 9599 | 2022-11-13T20:39:26Z | 2022-11-13T20:39:34Z | OWNER | I'm going to add a special `no-auto-close` class to these and teach that code not to close them. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1427293909 | |
https://github.com/simonw/datasette/issues/1871#issuecomment-1312816292 | https://api.github.com/repos/simonw/datasette/issues/1871 | 1312816292 | IC_kwDOBm6k_c5OP_yk | 9599 | 2022-11-13T20:38:42Z | 2022-11-13T20:38:42Z | OWNER | The current API explorer uses details/summary elements for the GET and POST dialogs. I only want one of these to be open at a time, to reflect that you can make either a GET or a POST. I just noticed that clicking anywhere else on the page closes both elements, which isn't what I want to happen. Turns out that's because of this code I added as part of Datasette's menu implementation! https://github.com/simonw/datasette/blob/9f54f00a50a4d950cfd69a0ff3526ae82c858826/datasette/templates/_close_open_menus.html#L2-L15 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1427293909 | |
https://github.com/simonw/datasette/issues/1882#issuecomment-1312582512 | https://api.github.com/repos/simonw/datasette/issues/1882 | 1312582512 | IC_kwDOBm6k_c5OPGtw | 9599 | 2022-11-12T22:11:18Z | 2022-11-12T22:11:18Z | OWNER | I like this: <img width="1080" alt="image" src="https://user-images.githubusercontent.com/9599/201496508-2fe102e8-f0de-4dd8-87c5-40b6aae97413.png"> ```json { "ok": true, "database": "data", "table": "agai2n", "table_url": "http://127.0.0.1:8001/data/agai2n", "schema": "CREATE TABLE [agai2n] (\n [hello] INTEGER\n)" } ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1435294468 | |
https://github.com/simonw/datasette/issues/1882#issuecomment-1312581121 | https://api.github.com/repos/simonw/datasette/issues/1882 | 1312581121 | IC_kwDOBm6k_c5OPGYB | 9599 | 2022-11-12T22:01:32Z | 2022-11-12T22:01:32Z | OWNER | I'm going to change it to `table` in the output AND the input. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1435294468 | |
https://github.com/simonw/datasette/issues/1882#issuecomment-1312581008 | https://api.github.com/repos/simonw/datasette/issues/1882 | 1312581008 | IC_kwDOBm6k_c5OPGWQ | 9599 | 2022-11-12T22:00:52Z | 2022-11-12T22:00:52Z | OWNER | Tried out my prototype in the API explorer: <img width="1229" alt="image" src="https://user-images.githubusercontent.com/9599/201496194-a6ebcefe-2be8-47d7-8f15-8c6345cba7cf.png"> The `"name"` on the output is bothering me a bit - should it be `table_name` or `table` instead? Problem is I really like `name` for the input, so should it be consistent to have the same name on the output here, or should I aim for consistency with other endpoints that might return `table` rather than the ambiguous `name` elsewhere? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1435294468 | |
https://github.com/simonw/datasette/issues/1882#issuecomment-1312580348 | https://api.github.com/repos/simonw/datasette/issues/1882 | 1312580348 | IC_kwDOBm6k_c5OPGL8 | 9599 | 2022-11-12T21:55:54Z | 2022-11-12T21:56:45Z | OWNER | What should this API return? I think the name of the table (`name`), the URL to that table (`table_url` - for consistency with how faceting API works already) and the schema of the table (`schema`). | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1435294468 | |
https://github.com/simonw/datasette/issues/1882#issuecomment-1312575048 | https://api.github.com/repos/simonw/datasette/issues/1882 | 1312575048 | IC_kwDOBm6k_c5OPE5I | 9599 | 2022-11-12T21:22:58Z | 2022-11-12T21:22:58Z | OWNER | Need to validate the table name. SQLite supports almost any table name - but they can't contain a newline character and cannot start with `sqlite_` - according to https://stackoverflow.com/questions/3694276/what-are-valid-table-names-in-sqlite/43049720#43049720 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1435294468 | |
https://github.com/simonw/datasette/issues/1882#issuecomment-1312556044 | https://api.github.com/repos/simonw/datasette/issues/1882 | 1312556044 | IC_kwDOBm6k_c5OPAQM | 9599 | 2022-11-12T19:29:11Z | 2022-11-12T19:29:11Z | OWNER | Thought of an edge-case: with `sqlite-utils` one feature I really like is that I can pipe data into it without caring if the table already exists or not: cat data.json | sqlite-utils insert my.db mytable - How could this new API support that? I thought about adding a `"create": true` option to `/db/table/-/insert` which creates the table if it doesn't already exist, but if I do that I'll need to start adding other options to that endpoint - to set the primary key, add foreign keys and suchlike - which would be ignored except for the cases where the table was being created from scratch. This doesn't feel right to me - I want to keep those options here, on `/db/-/create`. One idea I had was to implement it such that you can call `/db/-/create` multiple times for the same table, but only if you are using the `"rows"` option. If so, and if the rows can be safely inserted, it would let you do that. But instead, I'm going to outsource this to the CLI tool I plan to write that feeds data into this API. I'm already planning to use that tool for CSV inserts (so the API doesn't need to accept CSV directly). I think it's a good place for other usability enhancements like "insert this, creating the table if it does not exist" as well. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1435294468 | |
https://github.com/simonw/datasette/issues/1883#issuecomment-1311314981 | https://api.github.com/repos/simonw/datasette/issues/1883 | 1311314981 | IC_kwDOBm6k_c5OKRQl | 9599 | 2022-11-11T07:15:48Z | 2022-11-11T07:15:48Z | OWNER | I released that fix in Datasette 0.63.1: https://docs.datasette.io/en/stable/changelog.html#v0-63-1 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1435917503 | |
https://github.com/simonw/datasette/issues/1883#issuecomment-1311299535 | https://api.github.com/repos/simonw/datasette/issues/1883 | 1311299535 | IC_kwDOBm6k_c5OKNfP | 9599 | 2022-11-11T06:54:58Z | 2022-11-11T06:54:58Z | OWNER | This time deployed with: ``` cd demos/apache-proxy fly deploy --build-arg DATASETTE_REF=8d9a957c6329d26cc1e417b5d6911640d74765eb ``` To ensure the exact commit with the fix. And that fixed it! ``` % curl -i 'https://datasette-apache-proxy-demo.datasette.io/prefix/fixtures/binary_data?_filter_column=rowid&_filter_op=exact&_filter_value=1&_sort=rowid' HTTP/2 302 date: Fri, 11 Nov 2022 06:54:45 GMT server: Fly/b1863e2e7 (2022-11-09) location: /prefix/fixtures/binary_data?_sort=rowid&rowid__exact=1 link: </prefix/fixtures/binary_data?_sort=rowid&rowid__exact=1>; rel=preload content-type: text/plain x-proxied-by: Apache2 Debian via: 2 fly.io fly-request-id: 01GHJQGBSXBR7E53TY0EKMQ9PA-sjc ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1435917503 | |
https://github.com/simonw/datasette/issues/1883#issuecomment-1311292463 | https://api.github.com/repos/simonw/datasette/issues/1883 | 1311292463 | IC_kwDOBm6k_c5OKLwv | 9599 | 2022-11-11T06:44:24Z | 2022-11-11T06:44:24Z | OWNER | Modifying that test to the following does indeed cause a failure: ```python def test_base_url_affects_filter_redirects(app_client_base_url_prefix): response = app_client_base_url_prefix.get( "/fixtures/binary_data?_filter_column=rowid&_filter_op=exact&_filter_value=1&_sort=rowid" ) assert response.status == 302 assert ( response.headers["location"] == "/prefix/fixtures/binary_data?_sort=rowid&rowid__exact=1" ) ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1435917503 | |
https://github.com/simonw/datasette/issues/1883#issuecomment-1311291632 | https://api.github.com/repos/simonw/datasette/issues/1883 | 1311291632 | IC_kwDOBm6k_c5OKLjw | 9599 | 2022-11-11T06:43:00Z | 2022-11-11T06:43:00Z | OWNER | https://datasette-apache-proxy-demo.datasette.io/prefix/-/asgi-scope is useful: It confirms that `/prefix/` is nowhere to be seen in the incoming request data: ``` 'path': '/-/asgi-scope', 'query_string': b'', 'raw_path': b'/-/asgi-scope', ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1435917503 | |
https://github.com/simonw/datasette/issues/1883#issuecomment-1311290115 | https://api.github.com/repos/simonw/datasette/issues/1883 | 1311290115 | IC_kwDOBm6k_c5OKLMD | 9599 | 2022-11-11T06:40:14Z | 2022-11-11T06:41:56Z | OWNER | I modified that config file to have this line instead: ``` ProxyPass /prefix/ http://127.0.0.1:8001/ nocanon ``` And then deployed it by running: flyctl deploy --build-arg DATASETTE_REF=main This does NOT seem to have fixed the bug: ``` ~ % curl -i 'https://datasette-apache-proxy-demo.datasette.io/prefix/fixtures/binary_data?_filter_column=rowid&_filter_op=exact&_filter_value=1&_sort=rowid' HTTP/2 302 date: Fri, 11 Nov 2022 06:40:01 GMT server: Fly/b1863e2e7 (2022-11-09) location: /fixtures/binary_data?_sort=rowid&rowid__exact=1 link: </fixtures/binary_data?_sort=rowid&rowid__exact=1>; rel=preload content-type: text/plain x-proxied-by: Apache2 Debian via: 2 fly.io fly-request-id: 01GHJPNCF51CJ626EWZEHK2CH9-sjc ``` https://datasette-apache-proxy-demo.datasette.io/prefix/-/versions seems to confirm that this is the latest deployed version (0.63), so it looks like the deploy worked. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1435917503 | |
https://github.com/simonw/datasette/issues/1883#issuecomment-1311286593 | https://api.github.com/repos/simonw/datasette/issues/1883 | 1311286593 | IC_kwDOBm6k_c5OKKVB | 9599 | 2022-11-11T06:34:09Z | 2022-11-11T06:34:09Z | OWNER | https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypass includes this note: > Normally, mod_proxy will canonicalise ProxyPassed URLs. But this may be incompatible with some backends, particularly those that make use of *PATH_INFO*. The optional *nocanon* keyword suppresses this and passes the URL path "raw" to the backend. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1435917503 | |
https://github.com/simonw/datasette/issues/1883#issuecomment-1311284537 | https://api.github.com/repos/simonw/datasette/issues/1883 | 1311284537 | IC_kwDOBm6k_c5OKJ05 | 9599 | 2022-11-11T06:30:38Z | 2022-11-11T06:30:38Z | OWNER | Is there a chance that it's Apache that's messing with that `location:` header here, not Datasette? https://github.com/simonw/datasette/blob/bbaab3b38ec2ce5944239ffbe2dd53328df40fff/demos/apache-proxy/000-default.conf#L7-L13 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1435917503 | |
https://github.com/simonw/datasette/issues/1883#issuecomment-1311283301 | https://api.github.com/repos/simonw/datasette/issues/1883 | 1311283301 | IC_kwDOBm6k_c5OKJhl | 9599 | 2022-11-11T06:28:38Z | 2022-11-11T06:29:33Z | OWNER | `path_with_added_args(request, redirect_params)` should be preserving the current path from the request. https://github.com/simonw/datasette/blob/bbaab3b38ec2ce5944239ffbe2dd53328df40fff/datasette/utils/__init__.py#L273-L286 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1435917503 | |
https://github.com/simonw/datasette/issues/1883#issuecomment-1311282970 | https://api.github.com/repos/simonw/datasette/issues/1883 | 1311282970 | IC_kwDOBm6k_c5OKJca | 9599 | 2022-11-11T06:28:05Z | 2022-11-11T06:28:05Z | OWNER | Relevant code: https://github.com/simonw/datasette/blob/bbaab3b38ec2ce5944239ffbe2dd53328df40fff/datasette/views/table.py#L227-L249 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1435917503 | |
https://github.com/simonw/datasette/issues/1883#issuecomment-1311280709 | https://api.github.com/repos/simonw/datasette/issues/1883 | 1311280709 | IC_kwDOBm6k_c5OKI5F | 9599 | 2022-11-11T06:25:27Z | 2022-11-11T06:25:27Z | OWNER | I tried adding this test but it passed! I expected it to fail: ```python def test_base_url_affects_filter_redirects(app_client_base_url_prefix): response = app_client_base_url_prefix.get( "/prefix/fixtures/binary_data?_filter_column=rowid&_filter_op=exact&_filter_value=1&_sort=rowid" ) assert response.status == 302 assert ( response.headers["location"] == "/prefix/fixtures/binary_data?_sort=rowid&rowid__exact=1" ) ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1435917503 | |
https://github.com/simonw/datasette/issues/1883#issuecomment-1311278678 | https://api.github.com/repos/simonw/datasette/issues/1883 | 1311278678 | IC_kwDOBm6k_c5OKIZW | 9599 | 2022-11-11T06:22:37Z | 2022-11-11T06:22:37Z | OWNER | If you view source on that page the HTML looks correct: ```html <form class="filters" action="/prefix/fixtures/binary_data" method="get"> ``` (I just added a test that confirms this too.) But... it looks like the bug is in the redirection code. https://datasette-apache-proxy-demo.datasette.io/prefix/fixtures/binary_data?_filter_column=rowid&_filter_op=exact&_filter_value=1&_sort=rowid returns the following: location: /fixtures/binary_data?_sort=rowid&rowid__exact=1 Which is incorrect. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1435917503 | |
https://github.com/simonw/datasette/issues/1883#issuecomment-1311273461 | https://api.github.com/repos/simonw/datasette/issues/1883 | 1311273461 | IC_kwDOBm6k_c5OKHH1 | 9599 | 2022-11-11T06:16:08Z | 2022-11-11T06:16:08Z | OWNER | Great catch, thanks! | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1435917503 | |
https://github.com/simonw/datasette/issues/1880#issuecomment-1311273063 | https://api.github.com/repos/simonw/datasette/issues/1880 | 1311273063 | IC_kwDOBm6k_c5OKHBn | 9599 | 2022-11-11T06:15:28Z | 2022-11-11T06:15:28Z | OWNER | The `_internal` database is intended to help Datasette handle much larger attached databases. Right now Datasette attempts to show every database on the https://latest.datasette.io/ index page and every table on the https://latest.datasette.io/fixtures database index page - but these are not paginated. If you had a database containing 1,000 tables the database index page would get pretty slow. So I want to be able to paginate (and search) those. But to paginate them it's useful to have them in a database table itself, since then I can paginate using SQL. My plan for `_internal` is to use it to implement those advanced browsing features. I've not completed this work yet though. See this issue for more details on that: - #417 | { "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1433576351 | |
https://github.com/simonw/datasette/issues/1880#issuecomment-1311271298 | https://api.github.com/repos/simonw/datasette/issues/1880 | 1311271298 | IC_kwDOBm6k_c5OKGmC | 9599 | 2022-11-11T06:12:29Z | 2022-11-11T06:12:29Z | OWNER | I think you may have misunderstood this feature. This is talking about the `_internal` in-memory database, which maintains a set of tables that list the databases and tables that are attached to Datasette. They're not a copy of the data itself - just a list of table names, column names and database names. You can see what that database looks like by signing in as root - running `datasette --root` and clicking the link. Or you can see an example here: - Click the button on https://latest.datasette.io/login-as-root - Now visit https://latest.datasette.io/_internal For the example instance that looks like this: <img width="697" alt="image" src="https://user-images.githubusercontent.com/9599/201275230-b6375574-17f0-4cd8-b363-0c69a5907080.png"> The two most interesting tables in there are these ones: <img width="1703" alt="image" src="https://user-images.githubusercontent.com/9599/201275343-589eb6d7-6a9f-403c-b84a-07455d936a85.png"> <img width="1670" alt="CleanShot 2022-11-10 at 22 11 23@2x" src="https://user-images.githubusercontent.com/9599/201275287-04e6c57d-3ef9-4987-9a8b-5d421a48c452.png"> As you can see, it's just the table schema itself and the columns that make up the tables. Even if you have hundreds of databases connected each with hundreds of tables this should still only add up to a few MB of RAM. | { "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1433576351 | |
https://github.com/simonw/datasette/issues/1884#issuecomment-1311269045 | https://api.github.com/repos/simonw/datasette/issues/1884 | 1311269045 | IC_kwDOBm6k_c5OKGC1 | 9599 | 2022-11-11T06:08:28Z | 2022-11-11T06:08:28Z | OWNER | Does that work if you add `--load-extension spatialite`? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1439009231 | |
https://github.com/simonw/datasette/issues/1882#issuecomment-1302818784 | https://api.github.com/repos/simonw/datasette/issues/1882 | 1302818784 | IC_kwDOBm6k_c5Np2_g | 9599 | 2022-11-04T00:25:18Z | 2022-11-04T16:12:39Z | OWNER | On that basis I think the core API design should change to this: ``` POST /db/-/create Authorization: Bearer xxx Content-Type: application/json { "name": "my new table", "columns": [ { "name": "id", "type": "integer" }, { "name": "title", "type": "text" } ] "pk": "id" } ``` This leaves room for a `"rows": []` key at the root too. Having that as a child of `"table"` felt unintuitive to me, and I didn't like the way this looked either: ```json { "table": { "name": "my_new_table" }, "rows": [ {"id": 1, "title": "Title"} ] } ``` Weird to have the table `name` nested inside `table` when `rows` wasn't. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1435294468 | |
https://github.com/simonw/datasette/issues/1882#issuecomment-1302818153 | https://api.github.com/repos/simonw/datasette/issues/1882 | 1302818153 | IC_kwDOBm6k_c5Np21p | 9599 | 2022-11-04T00:23:58Z | 2022-11-04T00:23:58Z | OWNER | I made a decision here that this endpoint should also accept an optional `"rows": [...]` list which is used to automatically create the table using a schema derived from those example rows (which then get inserted): - https://github.com/simonw/datasette/issues/1862#issuecomment-1302817807 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1435294468 | |
https://github.com/simonw/datasette/issues/1862#issuecomment-1302817807 | https://api.github.com/repos/simonw/datasette/issues/1862 | 1302817807 | IC_kwDOBm6k_c5Np2wP | 9599 | 2022-11-04T00:23:13Z | 2022-11-04T00:23:13Z | OWNER | I don't like this on `/db/table/-/insert` - I think it makes more sense to optionally pass a `"rows"` key to the `/db/-/create` endpoint instead. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1425011030 | |
https://github.com/simonw/datasette/issues/1862#issuecomment-1302817500 | https://api.github.com/repos/simonw/datasette/issues/1862 | 1302817500 | IC_kwDOBm6k_c5Np2rc | 9599 | 2022-11-04T00:22:31Z | 2022-11-04T00:22:31Z | OWNER | Maybe this is a feature added to the existing `/db/table/-/insert` endpoint? Bit weird that you can call that endpoint for a table that doesn't exist yet, but it fits the `sqlite-utils` way of creating tables which I've found very pleasant over the past few years. So perhaps the API looks like this: ``` POST /<database>/<table>/-/insert Content-Type: application/json Authorization: Bearer dstok_<rest-of-token> { "create_table": true, "rows": [ { "column1": "value1", "column2": "value2" }, { "column1": "value3", "column2": "value4" } ] } ``` The `create_table` option will cause the table to be created if it doesn't already exist. That means I probably also need a `"pk": "..."` column for setting a primary key if the table is being created ... and maybe other options that I invent for this other feature too? - #1882 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1425011030 | |
https://github.com/simonw/datasette/issues/1871#issuecomment-1302815105 | https://api.github.com/repos/simonw/datasette/issues/1871 | 1302815105 | IC_kwDOBm6k_c5Np2GB | 9599 | 2022-11-04T00:17:23Z | 2022-11-04T00:17:23Z | OWNER | I'll probably enhance it a bit more though, I want to provide a UI that lists all the tables you can explore and lets you click to pre-fill the forms with them. Though at that point what should I do about the other endpoints? Probably list those too. Gets a bit complex, especially with the row-level update and delete endpoints. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1427293909 | |
https://github.com/simonw/datasette/issues/1871#issuecomment-1302814693 | https://api.github.com/repos/simonw/datasette/issues/1871 | 1302814693 | IC_kwDOBm6k_c5Np1_l | 9599 | 2022-11-04T00:16:36Z | 2022-11-04T00:16:36Z | OWNER | I can close this issue once I fix it so it no longer hard-codes a potentially invalid example endpoint: https://github.com/simonw/datasette/blob/bcc781f4c50a8870e3389c4e60acb625c34b0317/datasette/templates/api_explorer.html#L24-L26 https://github.com/simonw/datasette/blob/bcc781f4c50a8870e3389c4e60acb625c34b0317/datasette/templates/api_explorer.html#L34-L35 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1427293909 | |
https://github.com/simonw/datasette/issues/1881#issuecomment-1302813449 | https://api.github.com/repos/simonw/datasette/issues/1881 | 1302813449 | IC_kwDOBm6k_c5Np1sJ | 9599 | 2022-11-04T00:14:07Z | 2022-11-04T00:14:07Z | OWNER | Tool is now live here: https://latest-1-0-dev.datasette.io/-/permissions Needs root perms, so access this first: https://latest-1-0-dev.datasette.io/login-as-root | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1434094365 | |
https://github.com/simonw/datasette/issues/1881#issuecomment-1302812918 | https://api.github.com/repos/simonw/datasette/issues/1881 | 1302812918 | IC_kwDOBm6k_c5Np1j2 | 9599 | 2022-11-04T00:13:05Z | 2022-11-04T00:13:05Z | OWNER | Has tests now. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1434094365 | |
https://github.com/simonw/datasette/issues/1863#issuecomment-1302790013 | https://api.github.com/repos/simonw/datasette/issues/1863 | 1302790013 | IC_kwDOBm6k_c5Npv99 | 9599 | 2022-11-03T23:32:30Z | 2022-11-03T23:32:30Z | OWNER | I'm not going to allow updates to primary keys. If you need to do that, you can instead delete the record and then insert a new one with the new primary keys you wanted - or maybe use a custom SQL query. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1425029242 | |
https://github.com/simonw/datasette/issues/1851#issuecomment-1294224185 | https://api.github.com/repos/simonw/datasette/issues/1851 | 1294224185 | IC_kwDOBm6k_c5NJEs5 | 9599 | 2022-10-27T23:18:24Z | 2022-11-03T23:26:05Z | OWNER | So new API design is: ``` POST /db/table/-/insert Authorization: Bearer xxx Content-Type: application/json { "row": { "id": 1, "name": "New record" } } ``` Returns: ``` 201 Created { "row": [{ "id": 1, "name": "New record" }] } ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1421544654 | |
https://github.com/simonw/datasette/issues/1863#issuecomment-1302785086 | https://api.github.com/repos/simonw/datasette/issues/1863 | 1302785086 | IC_kwDOBm6k_c5Npuw- | 9599 | 2022-11-03T23:24:33Z | 2022-11-03T23:24:56Z | OWNER | Thinking more about validation: I'm considering if this should validate that columns which are defined as SQLite foreign keys are being updated to values that exist in those other tables. I like the sound of this. It seems like a sensible default behaviour for Datasette. And it fits with the fact that Datasette treats foreign keys specially elsewhere in the interface. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1425029242 | |
https://github.com/simonw/datasette/issues/1863#issuecomment-1302760549 | https://api.github.com/repos/simonw/datasette/issues/1863 | 1302760549 | IC_kwDOBm6k_c5Npoxl | 9599 | 2022-11-03T22:43:04Z | 2022-11-03T23:21:31Z | OWNER | The `id=(int, ...)` thing is weird, but is apparently Pydantic syntax for a required field? https://cs.github.com/starlite-api/starlite/blob/28ddc847c4cb072f0d5d21a9ecd5259711f12ec9/docs/usage/11-data-transfer-objects.md#L161 confirms: > 1. For required fields use a tuple of type + ellipsis, for example `(str, ...)`. > 2. For optional fields use a tuple of type + `None`, for example `(str, None)` > 3. To set a default value use a tuple of type + default value, for example `(str, "Hello World")` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1425029242 | |
https://github.com/simonw/datasette/issues/1863#issuecomment-1302760382 | https://api.github.com/repos/simonw/datasette/issues/1863 | 1302760382 | IC_kwDOBm6k_c5Npou- | 9599 | 2022-11-03T22:42:47Z | 2022-11-03T22:42:47Z | OWNER | ```python print(create_model('document', id=(int, ...), title=(str, None)).schema_json(indent=2)) ``` ```json { "title": "document", "type": "object", "properties": { "id": { "title": "Id", "type": "integer" }, "title": { "title": "Title", "type": "string" } }, "required": [ "id" ] } ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1425029242 | |
https://github.com/simonw/datasette/issues/1863#issuecomment-1302759174 | https://api.github.com/repos/simonw/datasette/issues/1863 | 1302759174 | IC_kwDOBm6k_c5NpocG | 9599 | 2022-11-03T22:40:47Z | 2022-11-03T22:40:47Z | OWNER | I'm considering Pydantic for this, see: - https://github.com/simonw/datasette/issues/1882#issuecomment-1302716350 In particular the `create_model()` method: https://pydantic-docs.helpmanual.io/usage/models/#dynamic-model-creation This would give me good validation. It would also, weirdly, give me the ability to output JSON schema. Maybe I could have this as the JSON schema for a row? `/db/table/-/json-schema` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1425029242 | |
https://github.com/simonw/datasette/issues/1882#issuecomment-1302716350 | https://api.github.com/repos/simonw/datasette/issues/1882 | 1302716350 | IC_kwDOBm6k_c5Npd-- | 9599 | 2022-11-03T21:51:14Z | 2022-11-03T22:35:54Z | OWNER | Validating this JSON object is getting a tiny bit complex. I'm tempted to adopt https://pydantic-docs.helpmanual.io/ at this point. The `create_model` example on https://stackoverflow.com/questions/66168517/generate-dynamic-model-using-pydantic/66168682#66168682 is particularly relevant, especially when I work on this issue: - #1863 ```python from pydantic import create_model d = {"strategy": {"name": "test_strat2", "periods": 10}} Strategy = create_model("Strategy", **d["strategy"]) print(Strategy.schema_json(indent=2)) ``` `create_model()`: https://pydantic-docs.helpmanual.io/usage/models/#dynamic-model-creation | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1435294468 | |
https://github.com/simonw/datasette/issues/1882#issuecomment-1302721916 | https://api.github.com/repos/simonw/datasette/issues/1882 | 1302721916 | IC_kwDOBm6k_c5NpfV8 | 9599 | 2022-11-03T21:58:50Z | 2022-11-03T21:59:17Z | OWNER | Mocked up a quick HTML+JavaScript form for creating that JSON structure using some iteration against Copilot prompts: ```html <pre> /* JSON format: { "table": { "name": "my new table", "columns": [ { "name": "id", "type": "integer" }, { "name": "title", "type": "text" } ] "pk": "id" } } HTML form with Javascript for creating this JSON: */</pre> <form id="create-table-form"> <label for="table-name">Table name</label> <input type="text" id="table-name" name="table-name" required><br> <label for="table-pk">Primary key</label> <input type="text" id="table-pk" name="table-pk" required><br> <label for="column-name">Column name</label> <input type="text" id="column-name" name="column-name" required> <label for="column-type">Column type</label> <input type="text" id="column-type" name="column-type" required> <button type="button" id="add-column">Add column</button> <p>Current columns:</p> <ul id="columns"></ul> <button type="button" id="create-table">Create table</button> </form> <script> var form = document.getElementById('create-table-form'); var tableName = document.getElementById('table-name'); var tablePk = document.getElementById('table-pk'); var columnName = document.getElementById('column-name'); var columnType = document.getElementById('column-type'); var addColumn = document.getElementById('add-column'); var createTable = document.getElementById('create-table'); var columns = []; addColumn.addEventListener('click', () => { columns.push({ name: columnName.value, type: columnType.value }); var li = document.createElement('li'); li.textContent = columnName.value + ' (' + columnType.value + ')'; // Add a delete button to each column var deleteButton = document.createElement('button'); deleteButton.textContent = 'Delete'; deleteButton.addEventListener('click', () => { columns.splice(colu… | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1435294468 | |
https://github.com/simonw/datasette/issues/1882#issuecomment-1302715662 | https://api.github.com/repos/simonw/datasette/issues/1882 | 1302715662 | IC_kwDOBm6k_c5Npd0O | 9599 | 2022-11-03T21:50:27Z | 2022-11-03T21:50:27Z | OWNER | API design for this: ``` POST /db/-/create Authorization: Bearer xxx Content-Type: application/json { "table": { "name": "my new table", "columns": [ { "name": "id", "type": "integer" }, { "name": "title", "type": "text" } ] "pk": "id" } } ``` Supported column types are: - `integer` - `text` - `float` (even though SQLite calls it a "real") - `blob` This matches my design for `sqlite-utils`: https://sqlite-utils.datasette.io/en/stable/cli.html#cli-create-table | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1435294468 | |
https://github.com/simonw/datasette/issues/1843#issuecomment-1302679026 | https://api.github.com/repos/simonw/datasette/issues/1843 | 1302679026 | IC_kwDOBm6k_c5NpU3y | 9599 | 2022-11-03T21:22:42Z | 2022-11-03T21:22:42Z | OWNER | Docs for the new `db.close()` method: https://docs.datasette.io/en/latest/internals.html#db-close | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1408757705 | |
https://github.com/simonw/datasette/issues/1843#issuecomment-1302678384 | https://api.github.com/repos/simonw/datasette/issues/1843 | 1302678384 | IC_kwDOBm6k_c5NpUtw | 9599 | 2022-11-03T21:21:59Z | 2022-11-03T21:21:59Z | OWNER | I added extra debug info to `/-/threads` to see this for myself: ```diff diff --git a/datasette/app.py b/datasette/app.py index 02bd38f1..16579e28 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -969,6 +969,13 @@ class Datasette: "threads": [ {"name": t.name, "ident": t.ident, "daemon": t.daemon} for t in threads ], + "file_connections": { + db.name: [ + [dict(r) for r in conn.execute("pragma database_list").fetchall()] + for conn in db._all_file_connections + ] + for db in self.databases.values() + }, } # Only available in Python 3.7+ if hasattr(asyncio, "all_tasks"): ``` Output after hitting refresh on a few `/fixtures` tables to ensure more threads started: ``` "file_connections": { "_internal": [], "fixtures": [ [ { "seq": 0, "name": "main", "file": "/Users/simon/Dropbox/Development/datasette/fixtures.db" } ], [ { "seq": 0, "name": "main", "file": "/Users/simon/Dropbox/Development/datasette/fixtures.db" } ], [ { "seq": 0, "name": "main", "file": "/Users/simon/Dropbox/Development/datasette/fixtures.db" } ] ] }, ``` I decided not to ship this feature though as it leaks the names of internal database files. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1408757705 | |
https://github.com/simonw/datasette/issues/1843#issuecomment-1302634332 | https://api.github.com/repos/simonw/datasette/issues/1843 | 1302634332 | IC_kwDOBm6k_c5NpJ9c | 9599 | 2022-11-03T20:34:56Z | 2022-11-03T20:34:56Z | OWNER | Confirmed that calling `conn.close()` on each SQLite file-based connection is the way to fix this problem. I'm adding a `db.close()` method (sync, not async - I tried async first but it was really hard to cause every thread in the pool to close its threadlocal database connection) which loops through all known open file-based connections and closes them. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1408757705 | |
https://github.com/simonw/datasette/issues/1843#issuecomment-1302574330 | https://api.github.com/repos/simonw/datasette/issues/1843 | 1302574330 | IC_kwDOBm6k_c5No7T6 | 9599 | 2022-11-03T19:30:22Z | 2022-11-03T19:30:22Z | OWNER | This is affecting me a lot at the moment, on my laptop (runs fine in CI). Here's a change to `conftest.py` which highlights the problem - it cause a failure the moment there are more than 5 open files according to `psutil`: ```diff diff --git a/tests/conftest.py b/tests/conftest.py index f4638a14..21d433c1 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -1,6 +1,7 @@ import httpx import os import pathlib +import psutil import pytest import re import subprocess @@ -192,3 +193,8 @@ def ds_unix_domain_socket_server(tmp_path_factory): yield ds_proc, uds # Shut it down at the end of the pytest session ds_proc.terminate() + + +def pytest_runtest_teardown(item: pytest.Item) -> None: + open_files = psutil.Process().open_files() + assert len(open_files) < 5 ``` The first error I get from this with `pytest --pdb -x` is here: ``` tests/test_api.py ............E >>>>> traceback >>>>> item = <Function test_sql_time_limit> def pytest_runtest_teardown(item: pytest.Item) -> None: open_files = psutil.Process().open_files() > assert len(open_files) < 5 E AssertionError: assert 5 < 5 E + where 5 = len([popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfglrt4p2/fixtures.db', fd=14), popenfile(... fd=19), popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmphdi5b250/fixtures.dot.db', fd=20)]) /Users/simon/Dropbox/Development/datasette/tests/conftest.py:200: AssertionError >>>>> entering PDB >>>>> >>>>> PDB post_mortem (IO-capturing turned off) >>>>> > /Users/simon/Dropbox/Development/datasette/tests/conftest.py(200)pytest_runtest_teardown() -> assert len(open_files) < 5 ``` That's this test: https://github.com/simonw/datasette/blob/2ec5583629005b32cb0877786f9681c5d43ca33f/tests/test_api.py#L656-L673 Which uses this fixture: https://github.com/simonw/datasette/blob/2ec5583629005b32cb0877786f9681c5d43ca33f/tests/fixtures.py#L228-L231 Which calls this func… | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1408757705 | |
https://github.com/simonw/datasette/issues/1855#issuecomment-1301646670 | https://api.github.com/repos/simonw/datasette/issues/1855 | 1301646670 | IC_kwDOBm6k_c5NlY1O | 9599 | 2022-11-03T05:11:26Z | 2022-11-03T05:11:26Z | OWNER | That still needs comprehensive tests before I land it. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1423336089 | |
https://github.com/simonw/datasette/issues/1855#issuecomment-1301646493 | https://api.github.com/repos/simonw/datasette/issues/1855 | 1301646493 | IC_kwDOBm6k_c5NlYyd | 9599 | 2022-11-03T05:11:06Z | 2022-11-03T05:11:06Z | OWNER | Built a prototype of the above: ```diff diff --git a/datasette/default_permissions.py b/datasette/default_permissions.py index 32b0c758..f68aa38f 100644 --- a/datasette/default_permissions.py +++ b/datasette/default_permissions.py @@ -6,8 +6,8 @@ import json import time -@hookimpl(tryfirst=True) -def permission_allowed(datasette, actor, action, resource): +@hookimpl(tryfirst=True, specname="permission_allowed") +def permission_allowed_default(datasette, actor, action, resource): async def inner(): if action in ( "permissions-debug", @@ -57,6 +57,44 @@ def permission_allowed(datasette, actor, action, resource): return inner +@hookimpl(specname="permission_allowed") +def permission_allowed_actor_restrictions(actor, action, resource): + if actor is None: + return None + _r = actor.get("_r") + if not _r: + # No restrictions, so we have no opinion + return None + action_initials = "".join([word[0] for word in action.split("-")]) + # If _r is defined then we use those to further restrict the actor + # Crucially, we only use this to say NO (return False) - we never + # use it to return YES (True) because that might over-ride other + # restrictions placed on this actor + all_allowed = _r.get("a") + if all_allowed is not None: + assert isinstance(all_allowed, list) + if action_initials in all_allowed: + return None + # How about for the current database? + if action in ("view-database", "view-database-download", "execute-sql"): + database_allowed = _r.get("d", {}).get(resource) + if database_allowed is not None: + assert isinstance(database_allowed, list) + if action_initials in database_allowed: + return None + # Or the current table? That's any time the resource is (database, table) + if not isinstance(resource, str) and len(resource) == 2: + database, table = resource + table_al… | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1423336089 | |
https://github.com/simonw/datasette/issues/1881#issuecomment-1301639741 | https://api.github.com/repos/simonw/datasette/issues/1881 | 1301639741 | IC_kwDOBm6k_c5NlXI9 | 9599 | 2022-11-03T04:58:21Z | 2022-11-03T04:58:21Z | OWNER | The whole `database_name` or `(database_name, table_name)` tuple for resource is a bit of a code smell. Maybe this is a chance to tidy that up too? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1434094365 | |
https://github.com/simonw/datasette/issues/1881#issuecomment-1301639370 | https://api.github.com/repos/simonw/datasette/issues/1881 | 1301639370 | IC_kwDOBm6k_c5NlXDK | 9599 | 2022-11-03T04:57:21Z | 2022-11-03T04:57:21Z | OWNER | The plugin hook would be called `register_permissions()`, for consistency with `register_routes()` and `register_commands()`. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1434094365 | |
https://github.com/simonw/datasette/issues/1881#issuecomment-1301638918 | https://api.github.com/repos/simonw/datasette/issues/1881 | 1301638918 | IC_kwDOBm6k_c5NlW8G | 9599 | 2022-11-03T04:56:06Z | 2022-11-03T04:56:06Z | OWNER | I've also introduced a new concept of a permission abbreviation, which like the permission name needs to be globally unique. That's a problem for plugins - they might just be able to guarantee that their permission long-form name is unique among other plugins (through sensible naming conventions) but the thing where they declare a initial-letters-only abbreviation is far more risky. I think abbreviations are optional - they are provided for core permissions but plugins are advised not to use them. Also Datasette could check that the installed plugins do not provide conflicting permissions on startup and refuse to start if they do. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1434094365 | |
https://github.com/simonw/datasette/issues/1881#issuecomment-1301638156 | https://api.github.com/repos/simonw/datasette/issues/1881 | 1301638156 | IC_kwDOBm6k_c5NlWwM | 9599 | 2022-11-03T04:54:00Z | 2022-11-03T04:54:00Z | OWNER | If I have the permissions defined like this: ```python PERMISSIONS = ( Permission("view-instance", "vi", False, False, True), Permission("view-database", "vd", True, False, True), Permission("view-database-download", "vdd", True, False, True), Permission("view-table", "vt", True, True, True), Permission("view-query", "vq", True, True, True), Permission("insert-row", "ir", True, True, False), Permission("delete-row", "dr", True, True, False), Permission("drop-table", "dt", True, True, False), Permission("execute-sql", "es", True, False, True), Permission("permissions-debug", "pd", False, False, False), Permission("debug-menu", "dm", False, False, False), ) ``` Instead of just calling them by their undeclared names in places like this: ```python await self.ds.permission_allowed( request.actor, "execute-sql", database, default=True ) ``` On the one hand I can ditch that confusing `default=True` option - whether a permission is on by default becomes a characteristic of that `Permission()` itself, which feels much neater. On the other hand though, plugins that introduce their own permissions - like https://datasette.io/plugins/datasette-edit-schema - will need a way to register those permissions with Datasette core. Probably another plugin hook. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1434094365 | |
https://github.com/simonw/datasette/issues/1881#issuecomment-1301635906 | https://api.github.com/repos/simonw/datasette/issues/1881 | 1301635906 | IC_kwDOBm6k_c5NlWNC | 9599 | 2022-11-03T04:48:09Z | 2022-11-03T04:48:09Z | OWNER | I built this prototype on the http://127.0.0.1:8001/-/allow-debug page, which is open to anyone to visit. But... I just realized that using this tool can leak information - you can use it to guess the names of invisible databases and tables and run theoretical permission checks against them. Using the tool also pollutes the list of permission checks that show up on the root-anlo `/-/permissions` page. So.... I'm going to restrict the usage of this tool to users with access to `/-/permissions` and put it on that page instead. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1434094365 | |
https://github.com/simonw/datasette/issues/1881#issuecomment-1301635340 | https://api.github.com/repos/simonw/datasette/issues/1881 | 1301635340 | IC_kwDOBm6k_c5NlWEM | 9599 | 2022-11-03T04:46:41Z | 2022-11-03T04:46:41Z | OWNER | Built this prototype: ![prototype](https://user-images.githubusercontent.com/9599/199649219-f146e43b-bfb5-45e6-9777-956f21a79887.gif) In building it I realized I needed to know which permissions took a table, a database, both or neither. So I had to bake that into the code. Here's the prototype so far (which includes a prototype of the logic for the `_r` field on actor, see #1855): ```diff diff --git a/datasette/default_permissions.py b/datasette/default_permissions.py index 32b0c758..f68aa38f 100644 --- a/datasette/default_permissions.py +++ b/datasette/default_permissions.py @@ -6,8 +6,8 @@ import json import time -@hookimpl(tryfirst=True) -def permission_allowed(datasette, actor, action, resource): +@hookimpl(tryfirst=True, specname="permission_allowed") +def permission_allowed_default(datasette, actor, action, resource): async def inner(): if action in ( "permissions-debug", @@ -57,6 +57,44 @@ def permission_allowed(datasette, actor, action, resource): return inner +@hookimpl(specname="permission_allowed") +def permission_allowed_actor_restrictions(actor, action, resource): + if actor is None: + return None + _r = actor.get("_r") + if not _r: + # No restrictions, so we have no opinion + return None + action_initials = "".join([word[0] for word in action.split("-")]) + # If _r is defined then we use those to further restrict the actor + # Crucially, we only use this to say NO (return False) - we never + # use it to return YES (True) because that might over-ride other + # restrictions placed on this actor + all_allowed = _r.get("a") + if all_allowed is not None: + assert isinstance(all_allowed, list) + if action_initials in all_allowed: + return None + # How about for the current database? + if action in ("view-database", "view-database-download", "execute-sql"): + database_allowed = _r.get("d", {}).get(resource) + if databa… | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1434094365 | |
https://github.com/simonw/datasette/issues/1855#issuecomment-1301594495 | https://api.github.com/repos/simonw/datasette/issues/1855 | 1301594495 | IC_kwDOBm6k_c5NlMF_ | 9599 | 2022-11-03T03:11:17Z | 2022-11-03T03:11:17Z | OWNER | Maybe the way to do this is through a new standard mechanism on the actor: a set of additional restrictions, e.g.: ``` { "id": "root", "_r": { "a": ["ir", "ur", "dr"], "d": { "fixtures": ["ir", "ur", "dr"] }, "t": { "fixtures": { "searchable": ["ir"] } } } ``` `"a"` is "all permissions" - these apply to everything. `"d"` permissions only apply to the specified database `"t"` permissions only apply to the specified table The way this works is there's a default [permission_allowed(datasette, actor, action, resource)](https://docs.datasette.io/en/stable/plugin_hooks.html#id25) hook which only consults these, and crucially just says NO if those rules do not match. In this way it would apply as an extra layer of permission rules over the defaults (which for this `root` instance would all return yes). | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1423336089 | |
https://github.com/simonw/datasette/issues/1871#issuecomment-1299607082 | https://api.github.com/repos/simonw/datasette/issues/1871 | 1299607082 | IC_kwDOBm6k_c5Ndm4q | 9599 | 2022-11-02T05:45:31Z | 2022-11-02T05:45:31Z | OWNER | I'm going to add a link to the Datasette API docs for the current running version of Datasette, e.g. to https://docs.datasette.io/en/0.63/json_api.html | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1427293909 | |
https://github.com/simonw/datasette/issues/1871#issuecomment-1299600257 | https://api.github.com/repos/simonw/datasette/issues/1871 | 1299600257 | IC_kwDOBm6k_c5NdlOB | 9599 | 2022-11-02T05:36:40Z | 2022-11-02T05:36:40Z | OWNER | The API Explorer should definitely link to the `/-/create-token` page for users who have permission though. And it should probably go in the Datasette application menu? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1427293909 | |
https://github.com/simonw/datasette/issues/1871#issuecomment-1299599461 | https://api.github.com/repos/simonw/datasette/issues/1871 | 1299599461 | IC_kwDOBm6k_c5NdlBl | 9599 | 2022-11-02T05:35:36Z | 2022-11-02T05:36:15Z | OWNER | Here's a slightly wild idea: what if there was a button on `/-/api` that you could click to turn on "API explorer mode" for the rest of the Datasette interface - which sets a cookie, and that cookie means you then see "API explorer" links in all sorts of other relevant places in the Datasette UI (maybe tucked away in cog menus). Only reason I don't want to show these to everyone is that I don't think this is a very user-friendly feature: if you don't know what an API is I don't want to expose you to it unnecessarily. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1427293909 | |
https://github.com/simonw/datasette/issues/1871#issuecomment-1299598570 | https://api.github.com/repos/simonw/datasette/issues/1871 | 1299598570 | IC_kwDOBm6k_c5Ndkzq | 9599 | 2022-11-02T05:34:28Z | 2022-11-02T05:34:28Z | OWNER | This is pretty useful now. Two features I still want to add: - The ability to link to the API explorer such that the form is pre-filled with material from the URL. Need to guard against clickjacking first though, so no-one can link to it in an invisible iframe and trick the user into hitting POST. - Some kind of list of endpoints so people can click links to start using the API explorer. A list of every table the user can write to with each of their `/db/table/-/insert` endpoints for example. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1427293909 | |
https://github.com/simonw/datasette/issues/1871#issuecomment-1299597066 | https://api.github.com/repos/simonw/datasette/issues/1871 | 1299597066 | IC_kwDOBm6k_c5NdkcK | 9599 | 2022-11-02T05:32:22Z | 2022-11-02T05:32:22Z | OWNER | Demo of the latest API explorer: ![explorer](https://user-images.githubusercontent.com/9599/199406184-1292df42-25ea-4daf-8b54-ca26170ec1ea.gif) | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1427293909 | |
https://github.com/simonw/datasette/issues/1871#issuecomment-1299388341 | https://api.github.com/repos/simonw/datasette/issues/1871 | 1299388341 | IC_kwDOBm6k_c5Ncxe1 | 9599 | 2022-11-02T00:24:28Z | 2022-11-02T00:25:00Z | OWNER | I want JSON syntax highlighting. https://github.com/luyilin/json-format-highlight is an MIT licensed tiny highlighter that looks decent for this. https://unpkg.com/json-format-highlight@1.0.1/dist/json-format-highlight.js | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1427293909 | |
https://github.com/simonw/datasette/issues/1871#issuecomment-1299349741 | https://api.github.com/repos/simonw/datasette/issues/1871 | 1299349741 | IC_kwDOBm6k_c5NcoDt | 9599 | 2022-11-01T23:22:55Z | 2022-11-01T23:22:55Z | OWNER | It's weird that the API explorer only lets you explore POST APIs. It should probably also let you explore GET APIs, or be renamed. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1427293909 | |
https://github.com/simonw/datasette/issues/1879#issuecomment-1299098458 | https://api.github.com/repos/simonw/datasette/issues/1879 | 1299098458 | IC_kwDOBm6k_c5Nbqta | 9599 | 2022-11-01T20:27:40Z | 2022-11-01T20:33:52Z | OWNER | https://github.com/simonw/datasette-x-forwarded-host/blob/main/datasette_x_forwarded_host/__init__.py could happen in core controlled by: `--setting trust_forwarded_host 1` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1432037325 | |
https://github.com/simonw/datasette/issues/1879#issuecomment-1299102108 | https://api.github.com/repos/simonw/datasette/issues/1879 | 1299102108 | IC_kwDOBm6k_c5Nbrmc | 9599 | 2022-11-01T20:30:54Z | 2022-11-01T20:33:06Z | OWNER | One idea: add a `/-/debug` page (or `/-/tips` or `/-/checks`) which shows the incoming requests headers and could even detect if there's an `x-forwarded-host` header that isn't being repeated and show a tip on how to fix that. | { "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1432037325 | |
https://github.com/simonw/datasette/issues/1879#issuecomment-1299102755 | https://api.github.com/repos/simonw/datasette/issues/1879 | 1299102755 | IC_kwDOBm6k_c5Nbrwj | 9599 | 2022-11-01T20:31:37Z | 2022-11-01T20:31:37Z | OWNER | And some JavaScript that can spot if Datasette thinks it is being served over HTTP when it's actually being served over HTTPS. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1432037325 | |
https://github.com/simonw/datasette/issues/1879#issuecomment-1299096850 | https://api.github.com/repos/simonw/datasette/issues/1879 | 1299096850 | IC_kwDOBm6k_c5NbqUS | 9599 | 2022-11-01T20:26:12Z | 2022-11-01T20:26:12Z | OWNER | The other relevant plugin here is https://datasette.io/plugins/datasette-x-forwarded-host Maybe that should be rolled into core too? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1432037325 | |
https://github.com/simonw/datasette/issues/1879#issuecomment-1299090678 | https://api.github.com/repos/simonw/datasette/issues/1879 | 1299090678 | IC_kwDOBm6k_c5Nboz2 | 9599 | 2022-11-01T20:20:28Z | 2022-11-01T20:20:28Z | OWNER | My first step in debugging these is to install https://datasette.io/plugins/datasette-debug-asgi - but now I'm thinking maybe something like that should be part of core. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1432037325 | |
https://github.com/simonw/datasette/issues/1862#issuecomment-1299073433 | https://api.github.com/repos/simonw/datasette/issues/1862 | 1299073433 | IC_kwDOBm6k_c5NbkmZ | 9599 | 2022-11-01T20:04:31Z | 2022-11-01T20:04:31Z | OWNER | It really feels like this should be accompanied by a `/db/-/create` API for creating tables. I had to add that to `sqlite-utils` eventually (initially it only supported creating by passing in an example document): https://sqlite-utils.datasette.io/en/stable/cli.html#cli-create-table | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1425011030 | |
https://github.com/simonw/datasette/issues/1878#issuecomment-1299071456 | https://api.github.com/repos/simonw/datasette/issues/1878 | 1299071456 | IC_kwDOBm6k_c5NbkHg | 9599 | 2022-11-01T20:02:43Z | 2022-11-01T20:02:43Z | OWNER | Note that "update" is partially covered by the `replace` option to `/-/insert`, added here: - https://github.com/simonw/datasette/issues/1873#issuecomment-1298885451 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1432013704 | |
https://github.com/simonw/datasette/issues/1873#issuecomment-1298919552 | https://api.github.com/repos/simonw/datasette/issues/1873 | 1298919552 | IC_kwDOBm6k_c5Na_CA | 9599 | 2022-11-01T18:11:27Z | 2022-11-01T18:11:27Z | OWNER | I forgot to document `ignore` and `replace`. Also I need to add tests that cover: - Forgetting to include a primary key on a non-autoincrement table - Compound primary keys - Rowid only tables with and without rowid specified I think my validation logic here will get caught out by the fact that `rowid` does not show up as a valid column name: https://github.com/simonw/datasette/blob/9bec7c38eb93cde5afb16df9bdd96aea2a5b0459/datasette/views/table.py#L1151-L1160 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1428630253 | |
https://github.com/simonw/datasette/issues/1873#issuecomment-1298905135 | https://api.github.com/repos/simonw/datasette/issues/1873 | 1298905135 | IC_kwDOBm6k_c5Na7gv | 9599 | 2022-11-01T17:59:59Z | 2022-11-01T17:59:59Z | OWNER | It's a bit surprising that you can send `"ignore": true, "return_rows": true` and the returned `"inserted"` key will list rows that were NOT inserted (since they were ignored). Three options: 1. Ignore that and document it 2. Fix it so `"inserted"` only returns rows that were actually inserted (bit tricky) 3. Change the name of `"inserted"` to something else I'm picking 3 - I'm going to change it to be called `"rows"` instead. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1428630253 | |
https://github.com/simonw/datasette/issues/1873#issuecomment-1298885451 | https://api.github.com/repos/simonw/datasette/issues/1873 | 1298885451 | IC_kwDOBm6k_c5Na2tL | 9599 | 2022-11-01T17:42:20Z | 2022-11-01T17:42:20Z | OWNER | Design decision: ```json { "rows": [{"id": 1, "title": "The title"}], "ignore": true } ``` Or `"replace": true`. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1428630253 | |
https://github.com/simonw/sqlite-utils/issues/506#issuecomment-1298879701 | https://api.github.com/repos/simonw/sqlite-utils/issues/506 | 1298879701 | IC_kwDOCGYnMM5Na1TV | 9599 | 2022-11-01T17:37:13Z | 2022-11-01T17:37:13Z | OWNER | The question I was originally trying to answer here was this: how many rows were actually inserted by that call to `.insert_all()`? I don't know that `.rowcount` would ever be useful here, since the "correct" answer depends on other factors - had I determined to ignore or replace records with a primary key that matches an existing record for example? So I think if people need `rowcount` they can get it by using a `cursor` directly. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1429029604 | |
https://github.com/simonw/sqlite-utils/issues/506#issuecomment-1298877872 | https://api.github.com/repos/simonw/sqlite-utils/issues/506 | 1298877872 | IC_kwDOCGYnMM5Na02w | 9599 | 2022-11-01T17:35:30Z | 2022-11-01T17:35:30Z | OWNER | This may not make sense. First, `.last_rowid` is a property on table - but that doesn't make sense for `rowcount` since it should clearly be a property on the database itself (you can run a query directly using `db.execute()` without going through a `Table` object). So I tried this prototype: ```diff diff --git a/docs/python-api.rst b/docs/python-api.rst index 206e5e6..78d3a8d 100644 --- a/docs/python-api.rst +++ b/docs/python-api.rst @@ -186,6 +186,15 @@ The ``db.query(sql)`` function executes a SQL query and returns an iterator over # {'name': 'Cleo'} # {'name': 'Pancakes'} +After executing a query the ``db.rowcount`` property on that database instance will reflect the number of rows affected by any insert, update or delete operations performed by that query: + +.. code-block:: python + + db = Database(memory=True) + db["dogs"].insert_all([{"name": "Cleo"}, {"name": "Pancakes"}]) + print(db.rowcount) + # Outputs: 2 + .. _python_api_execute: db.execute(sql, params) diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py index a06f4b7..c19c2dd 100644 --- a/sqlite_utils/db.py +++ b/sqlite_utils/db.py @@ -294,6 +294,8 @@ class Database: _counts_table_name = "_counts" use_counts_table = False + # Number of rows inserted, updated or deleted + rowcount: Optional[int] = None def __init__( self, @@ -480,9 +482,11 @@ class Database: if self._tracer: self._tracer(sql, parameters) if parameters is not None: - return self.conn.execute(sql, parameters) + cursor = self.conn.execute(sql, parameters) else: - return self.conn.execute(sql) + cursor = self.conn.execute(sql) + self.rowcount = cursor.rowcount + return cursor def executescript(self, sql: str) -> sqlite3.Cursor: """ ``` But this happens: ```pycon >>> from sqlite_utils import Database >>> db = Database(memory=True) >>> db["dogs"].insert_a… | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1429029604 | |
https://github.com/simonw/datasette/issues/1876#issuecomment-1298856054 | https://api.github.com/repos/simonw/datasette/issues/1876 | 1298856054 | IC_kwDOBm6k_c5Navh2 | 9599 | 2022-11-01T17:16:01Z | 2022-11-01T17:16:01Z | OWNER | `ta.style.height = ta.scrollHeight + 'px'` is an easy way to do that. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1431786951 | |
https://github.com/simonw/datasette/issues/1876#issuecomment-1298854321 | https://api.github.com/repos/simonw/datasette/issues/1876 | 1298854321 | IC_kwDOBm6k_c5NavGx | 9599 | 2022-11-01T17:14:33Z | 2022-11-01T17:14:33Z | OWNER | I could use a `textarea` here (would need to figure out a neat pattern to expand it to fit the query): <img width="426" alt="image" src="https://user-images.githubusercontent.com/9599/199295041-25abe0b9-f825-43a2-ae5a-face622e08bc.png"> | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1431786951 | |
https://github.com/simonw/datasette/issues/1864#issuecomment-1296403316 | https://api.github.com/repos/simonw/datasette/issues/1864 | 1296403316 | IC_kwDOBm6k_c5NRYt0 | 9599 | 2022-10-31T00:39:43Z | 2022-10-31T00:39:43Z | OWNER | It looks like SQLite has features for this already: https://www.sqlite.org/foreignkeys.html#fk_actions > Foreign key ON DELETE and ON UPDATE clauses are used to configure actions that take place when deleting rows from the parent table (ON DELETE), or modifying the parent key values of existing rows (ON UPDATE). A single foreign key constraint may have different actions configured for ON DELETE and ON UPDATE. Foreign key actions are similar to triggers in many ways. On that basis, I'm not going to implement anything additional in the `.../-/delete` endpoint relating to foreign keys. Developers who want special treatment of them can do that with a combination of a plugin (maybe I'll build a `datasette-enable-foreign-keys` plugin) and tables created using those `ON DELETE` clauses. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1425029275 | |
https://github.com/simonw/datasette/issues/1864#issuecomment-1296402071 | https://api.github.com/repos/simonw/datasette/issues/1864 | 1296402071 | IC_kwDOBm6k_c5NRYaX | 9599 | 2022-10-31T00:37:09Z | 2022-10-31T00:37:09Z | OWNER | I need to think about what happens if you delete a row that is the target of a foreign key from another row. https://www.sqlite.org/foreignkeys.html#fk_enable shows that SQLite will only actively enforce these relationships (e.g. throw an error if you try to delete a row that is referenced by another row) if you first run `PRAGMA foreign_keys = ON;` against the connection. > Foreign key constraints are disabled by default (for backwards compatibility), so must be enabled separately for each [database connection](https://www.sqlite.org/c3ref/sqlite3.html). (Note, however, that future releases of SQLite might change so that foreign key constraints enabled by default. Careful developers will not make any assumptions about whether or not foreign keys are enabled by default but will instead enable or disable them as necessary.) I don't actually believe that the SQLite maintainers will ever make that the default though. Datasette doesn't turn these on at the moment, but it could be turned on by a `prepare_connection()` plugin. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1425029275 | |
https://github.com/simonw/datasette/issues/1864#issuecomment-1296375536 | https://api.github.com/repos/simonw/datasette/issues/1864 | 1296375536 | IC_kwDOBm6k_c5NRR7w | 9599 | 2022-10-30T23:17:11Z | 2022-10-30T23:17:11Z | OWNER | I'm a bit nervous about calling `.delete()` with the `pk_values` - can I be sure they are in the correct order? https://github.com/simonw/datasette/blob/00632ded30e7cf9f0cf9478680645d1dabe269ae/datasette/views/row.py#L188-L190 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1425029275 | |
https://github.com/simonw/datasette/issues/1864#issuecomment-1296375310 | https://api.github.com/repos/simonw/datasette/issues/1864 | 1296375310 | IC_kwDOBm6k_c5NRR4O | 9599 | 2022-10-30T23:16:19Z | 2022-10-30T23:16:19Z | OWNER | Still needs tests that cover compound primary keys and rowid tables. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1425029275 | |
https://github.com/simonw/datasette/issues/1874#issuecomment-1296363981 | https://api.github.com/repos/simonw/datasette/issues/1874 | 1296363981 | IC_kwDOBm6k_c5NRPHN | 9599 | 2022-10-30T22:19:47Z | 2022-10-30T22:19:47Z | OWNER | Documentation: https://docs.datasette.io/en/1.0-dev/json_api.html#dropping-tables | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1429030341 | |
https://github.com/simonw/sqlite-utils/issues/506#issuecomment-1296358636 | https://api.github.com/repos/simonw/sqlite-utils/issues/506 | 1296358636 | IC_kwDOCGYnMM5NRNzs | 9599 | 2022-10-30T21:52:11Z | 2022-10-30T21:52:11Z | OWNER | This could work in a similar way to `db.insert(...).last_rowid`. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1429029604 |