github
html_url | issue_url | id | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
https://github.com/simonw/datasette/pull/2152#issuecomment-1691767797 | https://api.github.com/repos/simonw/datasette/issues/2152 | 1691767797 | IC_kwDOBm6k_c5k1lP1 | 9599 | 2023-08-24T14:15:10Z | 2023-08-24T14:15:10Z | OWNER | This is broken because Sphinx no longer supports Python 3.8. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1865174661 | |
https://github.com/simonw/datasette/issues/2153#issuecomment-1691763427 | https://api.github.com/repos/simonw/datasette/issues/2153 | 1691763427 | IC_kwDOBm6k_c5k1kLj | 9599 | 2023-08-24T14:12:43Z | 2023-08-24T14:12:43Z | OWNER | Annoying that `datasette client ...` makes a great name both for a plugin that executes simulated queries against a local database (thanks to its similarity to the existing `datasette.client` Python API) but is also the ideal name for a command for running commands as a client of an external Datasette instance! | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1865232341 | |
https://github.com/simonw/datasette/issues/2153#issuecomment-1691761685 | https://api.github.com/repos/simonw/datasette/issues/2153 | 1691761685 | IC_kwDOBm6k_c5k1jwV | 9599 | 2023-08-24T14:11:41Z | 2023-08-24T14:11:41Z | OWNER | Another option: implement this as a plugin, providing a new command like `datasette get ...` Or implement `datasette client get ...` as core commands or a plugin - except that clashes with the `datasette client` command that https://github.com/simonw/dclient adds (which is a tool for hitting remote Datasette instances, not running simulated queries through a local one). | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1865232341 | |
https://github.com/simonw/datasette/issues/2102#issuecomment-1691758168 | https://api.github.com/repos/simonw/datasette/issues/2102 | 1691758168 | IC_kwDOBm6k_c5k1i5Y | 9599 | 2023-08-24T14:09:45Z | 2023-08-24T14:09:45Z | OWNER | I'm going to implement this in a branch to make it easier to test out. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1805076818 | |
https://github.com/simonw/datasette/issues/2153#issuecomment-1691753489 | https://api.github.com/repos/simonw/datasette/issues/2153 | 1691753489 | IC_kwDOBm6k_c5k1hwR | 9599 | 2023-08-24T14:07:25Z | 2023-08-24T14:09:16Z | OWNER | Building that `"_r"` array is the main reason this would be useful, but it's also fiddly to get right. `datasette create-token` has a design for that already: https://docs.datasette.io/en/1.0a4/authentication.html#datasette-create-token ``` datasette create-token root \ --secret mysecret \ --all view-instance \ --all view-table \ --database docs view-query \ --resource docs documents insert-row \ --resource docs documents update-row ``` Adding imitations of those options (excluding `--secret`, not needed here) to `datasette serve` would add a LOT of extra options, but it would also make it really convenient to attempt a request with a specific set of restrictions. Not sure if that would be worth the extra `--help` output or not. I feel like the names would have to have a common prefix though. Maybe something like this: ```bash datasette serve data.db --get `/data/mytable.json' \ --actor-id root \ --r-all view-instance \ --r-database data view-query \ --r-resource data documents update-row ``` Other options could be the longer `--restrict-all/--restrict-database/--restrict-resource`. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1865232341 | |
https://github.com/simonw/datasette/issues/2102#issuecomment-1691045051 | https://api.github.com/repos/simonw/datasette/issues/2102 | 1691045051 | IC_kwDOBm6k_c5ky0y7 | 9599 | 2023-08-24T05:51:59Z | 2023-08-24T05:51:59Z | OWNER | With that fix in place, this works: ```bash datasette fixtures.db --get '/fixtures/facetable.json' --actor '{ "_r": { "r": { "fixtures": { "facetable": [ "vt" ] } } }, "a": "user" }' ``` But this fails, because it's for a table not explicitly listed: ```bash datasette fixtures.db --get '/fixtures/searchable.json' --actor '{ "_r": { "r": { "fixtures": { "facetable": [ "vt" ] } } }, "a": "user" }' ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1805076818 | |
https://github.com/simonw/datasette/issues/2102#issuecomment-1691044283 | https://api.github.com/repos/simonw/datasette/issues/2102 | 1691044283 | IC_kwDOBm6k_c5ky0m7 | 9599 | 2023-08-24T05:51:02Z | 2023-08-24T05:51:02Z | OWNER | Also need to confirm that permissions like `insert-row`, `delete-row`, `create-table` etc don't also need special cases to ensure they get through the `view-instance` etc checks, if those exist for those actions. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1805076818 | |
https://github.com/simonw/datasette/issues/2102#issuecomment-1691043475 | https://api.github.com/repos/simonw/datasette/issues/2102 | 1691043475 | IC_kwDOBm6k_c5ky0aT | 9599 | 2023-08-24T05:50:04Z | 2023-08-24T05:50:04Z | OWNER | On first test this seems to work! ```diff diff --git a/datasette/default_permissions.py b/datasette/default_permissions.py index 63a66c3c..9303dac8 100644 --- a/datasette/default_permissions.py +++ b/datasette/default_permissions.py @@ -187,6 +187,30 @@ def permission_allowed_actor_restrictions(datasette, actor, action, resource): return None _r = actor.get("_r") + # Special case for view-instance: it's allowed if there are any view-database + # or view-table permissions defined + if action == "view-instance": + database_rules = _r.get("d") or {} + for rules in database_rules.values(): + if "vd" in rules or "view-database" in rules: + return None + # Now check resources + resource_rules = _r.get("r") or {} + for _database, resources in resource_rules.items(): + for rules in resources.values(): + if "vt" in rules or "view-table" in rules: + return None + + # Special case for view-database: it's allowed if there are any view-table permissions + # defined within that database + if action == "view-database": + database_name = resource + resource_rules = _r.get("r") or {} + resources_in_database = resource_rules.get(database_name) or {} + for rules in resources_in_database.values(): + if "vt" in rules or "view-table" in rules: + return None + # Does this action have an abbreviation? to_check = {action} permission = datasette.permissions.get(action) ``` Needs a LOT of testing to make sure what it's doing is sensible though. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1805076818 | |
https://github.com/simonw/datasette/issues/2102#issuecomment-1691037971 | https://api.github.com/repos/simonw/datasette/issues/2102 | 1691037971 | IC_kwDOBm6k_c5kyzET | 9599 | 2023-08-24T05:42:47Z | 2023-08-24T05:42:47Z | OWNER | I applied a fun trick to help test this out: ```diff diff --git a/datasette/cli.py b/datasette/cli.py index 58f89c1c..830f47ef 100644 --- a/datasette/cli.py +++ b/datasette/cli.py @@ -445,6 +445,10 @@ def uninstall(packages, yes): "--token", help="API token to send with --get requests", ) +@click.option( + "--actor", + help="Actor to use for --get requests", +) @click.option("--version-note", help="Additional note to show on /-/versions") @click.option("--help-settings", is_flag=True, help="Show available settings") @click.option("--pdb", is_flag=True, help="Launch debugger on any errors") @@ -499,6 +503,7 @@ def serve( root, get, token, + actor, version_note, help_settings, pdb, @@ -611,7 +616,10 @@ def serve( headers = {} if token: headers["Authorization"] = "Bearer {}".format(token) - response = client.get(get, headers=headers) + cookies = {} + if actor: + cookies["ds_actor"] = client.actor_cookie(json.loads(actor)) + response = client.get(get, headers=headers, cookies=cookies) click.echo(response.text) exit_code = 0 if response.status == 200 else 1 sys.exit(exit_code) ``` This adds a `--actor` option to `datasette ... --get /path` which makes it easy to test an API endpoint using a fake actor with a set of `_r` restrictions. With that in place I can try this, with a token that has view-instance and view-database and view-table: ```bash datasette fixtures.db --get '/fixtures/facetable.json' --actor '{ "_r": { "a": [ "vi" ], "d": { "fixtures": [ "vd" ] }, "r": { "fixtures": { "facetable": [ "vt" ] } } }, "a": "user" }' ``` Or this, with a token that just has view-table but is missing the view-database and view-instan… | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1805076818 | |
https://github.com/simonw/datasette/issues/2102#issuecomment-1691036559 | https://api.github.com/repos/simonw/datasette/issues/2102 | 1691036559 | IC_kwDOBm6k_c5kyyuP | 9599 | 2023-08-24T05:40:53Z | 2023-08-24T05:40:53Z | OWNER | There might be an easier way to solve this. Here's some permission checks that run when hitting `/fixtures/facetable.json`: ``` permission_allowed: action=view-table, resource=('fixtures', 'facetable'), actor={'_r': {'a': ['vi'], 'd': {'fixtures': ['vd']}, 'r': {'fixtures': {'facetable': ['vt']}}}, 'a': 'user'} File "/datasette/views/table.py", line 727, in table_view_traced view_data = await table_view_data( File "/datasette/views/table.py", line 875, in table_view_data visible, private = await datasette.check_visibility( File "/datasette/app.py", line 890, in check_visibility await self.ensure_permissions(actor, permissions) permission_allowed: action=view-database, resource=fixtures, actor={'_r': {'a': ['vi'], 'd': {'fixtures': ['vd']}, 'r': {'fixtures': {'facetable': ['vt']}}}, 'a': 'user'} File "/datasette/views/table.py", line 727, in table_view_traced view_data = await table_view_data( File "/datasette/views/table.py", line 875, in table_view_data visible, private = await datasette.check_visibility( File "/datasette/app.py", line 890, in check_visibility await self.ensure_permissions(actor, permissions) permission_allowed: action=view-instance, resource=<None>, actor={'_r': {'a': ['vi'], 'd': {'fixtures': ['vd']}, 'r': {'fixtures': {'facetable': ['vt']}}}, 'a': 'user'} File "/datasette/views/table.py", line 727, in table_view_traced view_data = await table_view_data( File "/datasette/views/table.py", line 875, in table_view_data visible, private = await datasette.check_visibility( File "/datasette/app.py", line 890, in check_visibility await self.ensure_permissions(actor, permissions) ``` That's with a token that has the view instance, view database and view table permissions required. But... what if the restrictions logic said that if you have view-table you automatically also get view-database and view-instance? Would that actually let people do anything they shouldn't be able to do? I don't think … | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1805076818 | |
https://github.com/simonw/datasette/issues/2143#issuecomment-1690800119 | https://api.github.com/repos/simonw/datasette/issues/2143 | 1690800119 | IC_kwDOBm6k_c5kx4_3 | 9599 | 2023-08-24T00:10:32Z | 2023-08-24T00:39:00Z | OWNER | Something notable about this design is that, because the values in the key-value pairs are treated as JSON first and then strings only if they don't parse cleanly as JSON, it's possible to represent any structure (including nesting structures) using this syntax. You can do things like this if you need to (settings for an imaginary plugin): ```bash datasette data.db \ -s plugins.datasette-complex-plugin.configs '{"foo": [1,2,3], "bar": "baz"}' ``` Which would be equivalent to: ```yaml plugins: datasette-complex-plugin: configs: foo: - 1 - 2 - 3 bar: baz ``` This is a bit different from a previous attempt I made at the same problem: https://github.com/simonw/json-flatten - that used syntax like `foo.bar.[0]$int = 1` to specify an integer as the first item of an array, which is much more complex. That previous design was meant to support round-trips, so you could take any nested JSON object and turn it into an HTMl form or query string where every value can have its own form field, then turn the result back again. For the `datasette -s key value` feature we don't need round-tripping with individual values each editable on their own, so we can go with something much simpler. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1855885427 | |
https://github.com/simonw/datasette/issues/2143#issuecomment-1690800641 | https://api.github.com/repos/simonw/datasette/issues/2143 | 1690800641 | IC_kwDOBm6k_c5kx5IB | 9599 | 2023-08-24T00:11:16Z | 2023-08-24T00:11:16Z | OWNER | > @simonw, FWIW, I do exactly the same thing for one of my projects (both to allow multiple configuration files to be passed on the command line and setting individual values) and it works quite well for me and my users. I even use the same parameter name for both (https://studio.zerobrane.com/doc-configuration#configuration-via-command-line), but I understand why you may want to use different ones for files and individual values. There is one small difference that I accept code snippets, but I don't think it matters much in this case. That's a neat example thanks! | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1855885427 | |
https://github.com/simonw/datasette/issues/2143#issuecomment-1690792514 | https://api.github.com/repos/simonw/datasette/issues/2143 | 1690792514 | IC_kwDOBm6k_c5kx3JC | 9599 | 2023-08-24T00:00:16Z | 2023-08-24T00:02:55Z | OWNER | I've been thinking about what it might look like to allow command-line arguments to be used to define _any_ of the configuration options in `datasette.yml`, as alternative and more convenient syntax. Here's what I've come up with: ``` datasette \ -s settings.sql_time_limit_ms 1000 \ -s plugins.datasette-auth-tokens.manage_tokens true \ -s plugins.datasette-auth-tokens.manage_tokens_database tokens \ -s plugins.datasette-ripgrep.path "/home/simon/code-to-search" \ -s databases.mydatabase.tables.example_table.sort created \ mydatabase.db tokens.db ``` Which would be equivalent to `datasette.yml` containing this: ```yaml plugins: datasette-auth-tokens: manage_tokens: true manage_tokens_database: tokens datasette-ripgrep: path: /home/simon/code-to-search databases: mydatabase: tables: example_table: sort: created settings: sql_time_limit_ms: 1000 ``` Here's a prototype implementation of this: ```python import json from typing import Any, List, Tuple def _handle_pair(key: str, value: str) -> dict: """ Turn a key-value pair into a nested dictionary. foo, bar => {'foo': 'bar'} foo.bar, baz => {'foo': {'bar': 'baz'}} foo.bar, [1, 2, 3] => {'foo': {'bar': [1, 2, 3]}} foo.bar, "baz" => {'foo': {'bar': 'baz'}} foo.bar, '{"baz": "qux"}' => {'foo': {'bar': "{'baz': 'qux'}"}} """ try: value = json.loads(value) except json.JSONDecodeError: # If it doesn't parse as JSON, treat it as a string pass keys = key.split('.') result = current_dict = {} for k in keys[:-1]: current_dict[k] = {} current_dict = current_dict[k] current_dict[keys[-1]] = value return result def _combine(base: dict, update: dict) -> dict: """ Recursively merge two dictionaries. """ for key, value in update.items(): if isinstance(value, dict) and key in base and isinstance(base[key], dict): base[key] … | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1855885427 | |
https://github.com/simonw/datasette/issues/2143#issuecomment-1690787394 | https://api.github.com/repos/simonw/datasette/issues/2143 | 1690787394 | IC_kwDOBm6k_c5kx15C | 9599 | 2023-08-23T23:52:02Z | 2023-08-23T23:52:02Z | OWNER | > This also makes it simple to separate out secrets. > > `datasette --config settings.yaml --config secrets.yaml --config db-docs.yaml --config db-fixtures.yaml` Having multiple configs that combine in that way is a really interesting direction. > To chime in from a poweruser perspective: I'm worried that this is an overengineering trap. Yes, the current solution is somewhat messy. But there are datasette-wide settings, there are database-scope settings, there are table-scope settings etc, but then there are database-scope metadata and table-scope metadata. Trying to cleanly separate "settings" from "configuration" is, I believe, an uphill fight. I'm very keen on separating out the "metadata" - where metadata is the slimmest possible set of things, effectively the data license and the source and the column and table descriptions - from everything else, mainly because I want metadata to be able to travel with the data. One idea that's been discussed before is having an optional mechanism for storing metadata in the SQLite database file itself - potentially in a `_datasette_metadata` table. That way you could distribute a DB file and anyone who opened it in Datasette would also see the correct metadata about it. That's why I'm so keen on splitting out metadata from all of the other stuff - settings and plugin configuration and authentication rules. So really it becomes "true metadata" v.s. "all of the other junk that's accumulated in metadata and `settings.json`". | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1855885427 | |
https://github.com/simonw/datasette/issues/2102#issuecomment-1690705243 | https://api.github.com/repos/simonw/datasette/issues/2102 | 1690705243 | IC_kwDOBm6k_c5kxh1b | 9599 | 2023-08-23T22:03:54Z | 2023-08-23T22:03:54Z | OWNER | Idea: `datasette-permissions-debug` plugin which simply prints out a stacktrace for every permission check so you can see where in the code they are. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1805076818 | |
https://github.com/simonw/datasette/issues/2102#issuecomment-1690703764 | https://api.github.com/repos/simonw/datasette/issues/2102 | 1690703764 | IC_kwDOBm6k_c5kxheU | 9599 | 2023-08-23T22:02:14Z | 2023-08-23T22:02:14Z | OWNER | Built this new test: ```python @pytest.mark.asyncio async def test_view_table_token_can_access_table(perms_ds): actor = { "id": "restricted-token", "token": "dstok", # Restricted to just view-table on perms_ds_two/t1 "_r": {"r": {"perms_ds_two": {"t1": ["vt"]}}}, } cookies = {"ds_actor": perms_ds.client.actor_cookie(actor)} response = await perms_ds.client.get("/perms_ds_two/t1.json", cookies=cookies) assert response.status_code == 200 ``` The test fails. Running it with `pytest --pdb` let me do this: ``` (Pdb) from pprint import pprint (Pdb) pprint(perms_ds._permission_checks) deque([{'action': 'view-table', 'actor': {'_r': {'r': {'perms_ds_two': {'t1': ['vt']}}}, 'id': 'restricted-token', 'token': 'dstok'}, 'resource': ('perms_ds_two', 't1'), 'result': None, 'used_default': True, 'when': '2023-08-23T21:59:45.117155'}, {'action': 'view-database', 'actor': {'_r': {'r': {'perms_ds_two': {'t1': ['vt']}}}, 'id': 'restricted-token', 'token': 'dstok'}, 'resource': 'perms_ds_two', 'result': False, 'used_default': False, 'when': '2023-08-23T21:59:45.117189'}, {'action': 'view-instance', 'actor': {'_r': {'r': {'perms_ds_two': {'t1': ['vt']}}}, 'id': 'restricted-token', 'token': 'dstok'}, 'resource': None, 'result': False, 'used_default': False, 'when': '2023-08-23T21:59:45.126751'}, {'action': 'debug-menu', 'actor': {'_r': {'r': {'perms_ds_two': {'t1': ['vt']}}}, 'id': 'restricted-token', 'token': 'dstok'}, 'resource': None, 'result': False, 'used_default': False, 'when': '2023-08-23T21:59:45.126777'}], maxlen=200) ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1805076818 | |
https://github.com/simonw/datasette/issues/2102#issuecomment-1690693830 | https://api.github.com/repos/simonw/datasette/issues/2102 | 1690693830 | IC_kwDOBm6k_c5kxfDG | 9599 | 2023-08-23T21:51:52Z | 2023-08-23T21:52:58Z | OWNER | This is the hook in question: https://github.com/simonw/datasette/blob/bdf59eb7db42559e538a637bacfe86d39e5d17ca/datasette/hookspecs.py#L108-L110 - `True` means they are allowed to access it. You only need a single`True` from a plugin to allow it. - `False` means they are not, and just one `False` from a plugin will deny it (even if another one returned `True` I think) - `None` means that the plugin has no opinion on this question. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1805076818 | |
https://github.com/simonw/datasette/issues/950#issuecomment-680262437 | https://api.github.com/repos/simonw/datasette/issues/950 | 680262437 | MDEyOklzc3VlQ29tbWVudDY4MDI2MjQzNw== | 9599 | 2020-08-25T20:49:24Z | 2023-08-23T21:34:09Z | OWNER | The alternative to this would be to use regular databases and control access to them using [Authentication and permissions](https://docs.datasette.io/en/stable/authentication.html). My concern there is that it's just too easy for someone to mess up their configuration, which would be really bad. I like the idea of a much stronger defense mechanism specifically designed for secrets that should not be exposed. Outside of secrets, passwords and tokens this mechanism could also be useful for the use-case of using Datasette to power websites - as seen on https://www.niche-museums.com/ and https://www.rockybeaches.com/ - maybe those sites don't want to expose their data through their API but still want to use `datasette-template-sql` and the `graphql()` template tag in `datasette-graphql` to render data. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
685806511 | |
https://github.com/simonw/datasette/issues/2150#issuecomment-1690438270 | https://api.github.com/repos/simonw/datasette/issues/2150 | 1690438270 | IC_kwDOBm6k_c5kwgp- | 9599 | 2023-08-23T18:27:47Z | 2023-08-23T18:27:47Z | OWNER | I added `outline: 3px dotted pink` to that `form label` block to help spot where else it's being used. Oh interesting, looks like it's over-ridden here too: <img width="950" alt="CleanShot 2023-08-23 at 11 27 15@2x" src="https://github.com/simonw/datasette/assets/9599/f3f3e995-2d6c-4afe-8370-01e0b5141a64"> | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1863810783 | |
https://github.com/simonw/datasette/issues/2150#issuecomment-1690435866 | https://api.github.com/repos/simonw/datasette/issues/2150 | 1690435866 | IC_kwDOBm6k_c5kwgEa | 9599 | 2023-08-23T18:25:51Z | 2023-08-23T18:25:51Z | OWNER | Looks like that affects a few forms: <img width="701" alt="CleanShot 2023-08-23 at 11 24 46@2x" src="https://github.com/simonw/datasette/assets/9599/7145aec1-832e-4eea-84a6-a735df4dfd6e"> The search form on the table page over-rides it already: <img width="1243" alt="CleanShot 2023-08-23 at 11 25 26@2x" src="https://github.com/simonw/datasette/assets/9599/e002089e-e992-4cc5-bb6b-c54cf614ec95"> | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1863810783 | |
https://github.com/simonw/datasette/issues/2150#issuecomment-1690432928 | https://api.github.com/repos/simonw/datasette/issues/2150 | 1690432928 | IC_kwDOBm6k_c5kwfWg | 9599 | 2023-08-23T18:23:26Z | 2023-08-23T18:23:26Z | OWNER | That should be scoped just to the query filters form on the table page. I'll fix it in `main` but I'm still going to ship a fix for those plugins separately so they work well before people upgrade to the next Datasette release with this change in it. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1863810783 | |
https://github.com/simonw/datasette/issues/2150#issuecomment-1690431509 | https://api.github.com/repos/simonw/datasette/issues/2150 | 1690431509 | IC_kwDOBm6k_c5kwfAV | 9599 | 2023-08-23T18:22:47Z | 2023-08-23T18:22:47Z | OWNER | https://github.com/simonw/datasette/blob/64fd1d788eeed2624f107ac699f2370590ae1aa3/datasette/static/app.css#L485-L489 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1863810783 | |
https://github.com/simonw/datasette/issues/2091#issuecomment-1690423878 | https://api.github.com/repos/simonw/datasette/issues/2091 | 1690423878 | IC_kwDOBm6k_c5kwdJG | 9599 | 2023-08-23T18:18:18Z | 2023-08-23T18:18:18Z | OWNER | Dupe of: - #2097 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1781022369 | |
https://github.com/simonw/datasette/issues/2123#issuecomment-1689207309 | https://api.github.com/repos/simonw/datasette/issues/2123 | 1689207309 | IC_kwDOBm6k_c5kr0IN | 9599 | 2023-08-23T03:07:27Z | 2023-08-23T03:07:27Z | OWNER | > I'm happy to debug and land a patch if it's welcome. Yes please! What an odd bug. | { "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1825007061 | |
https://github.com/simonw/datasette/issues/2147#issuecomment-1689206768 | https://api.github.com/repos/simonw/datasette/issues/2147 | 1689206768 | IC_kwDOBm6k_c5krz_w | 9599 | 2023-08-23T03:06:32Z | 2023-08-23T03:06:32Z | OWNER | I'm less convinced by the "rewrite the query in some way" optional idea. What kind of use-cases can you imagine for that? My hunch is that it's much more likely to cause weird breakages than it is to allow for useful plugin extensions, but I'm willing to be convinced otherwise. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1858228057 | |
https://github.com/simonw/datasette/issues/2147#issuecomment-1689206170 | https://api.github.com/repos/simonw/datasette/issues/2147 | 1689206170 | IC_kwDOBm6k_c5krz2a | 9599 | 2023-08-23T03:05:32Z | 2023-08-23T03:05:32Z | OWNER | Interestingly enough there's actually a mechanism that looks like that a bit already: https://github.com/simonw/datasette/blob/64fd1d788eeed2624f107ac699f2370590ae1aa3/datasette/views/database.py#L496-L508 That `validate_sql_select()` function is defined here: https://github.com/simonw/datasette/blob/64fd1d788eeed2624f107ac699f2370590ae1aa3/datasette/utils/__init__.py#L256-L265 Against these constants: https://github.com/simonw/datasette/blob/64fd1d788eeed2624f107ac699f2370590ae1aa3/datasette/utils/__init__.py#L223-L253 Which isn't a million miles away from your suggestion to have a hook that can say if the query should be executed or not. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1858228057 | |
https://github.com/simonw/datasette/pull/2148#issuecomment-1689198368 | https://api.github.com/repos/simonw/datasette/issues/2148 | 1689198368 | IC_kwDOBm6k_c5krx8g | 9599 | 2023-08-23T02:57:53Z | 2023-08-23T02:57:53Z | OWNER | @dependabot rebase | { "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1859415334 | |
https://github.com/simonw/datasette/pull/2148#issuecomment-1689177556 | https://api.github.com/repos/simonw/datasette/issues/2148 | 1689177556 | IC_kwDOBm6k_c5krs3U | 9599 | 2023-08-23T02:44:29Z | 2023-08-23T02:44:29Z | OWNER | Simplest possible solution is to only run the `pip install .[docs]` bit under Python 3.9+, ditto for the docs tests. I think I'll try that. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1859415334 | |
https://github.com/simonw/datasette/pull/2148#issuecomment-1689175062 | https://api.github.com/repos/simonw/datasette/issues/2148 | 1689175062 | IC_kwDOBm6k_c5krsQW | 9599 | 2023-08-23T02:40:46Z | 2023-08-23T02:40:46Z | OWNER | Here's why the tests are failing: https://github.com/simonw/datasette/blob/2ce7872e3ba8d07248c194ef554bbdc1df510f32/.github/workflows/test.yml#L30-L46 It looks like those tests don't actually need Sphinx installed - they install `pip install -e '.[test,docs]'` to get the other docs dependencies: https://github.com/simonw/datasette/blob/2ce7872e3ba8d07248c194ef554bbdc1df510f32/setup.py#L70-L80 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1859415334 | |
https://github.com/simonw/datasette/pull/2148#issuecomment-1689173748 | https://api.github.com/repos/simonw/datasette/issues/2148 | 1689173748 | IC_kwDOBm6k_c5krr70 | 9599 | 2023-08-23T02:38:31Z | 2023-08-23T02:38:31Z | OWNER | Sphinx dropped support for Python 3.8 in their version 7.2.0: - https://www.sphinx-doc.org/en/master/changes.html#release-7-2-0-released-aug-17-2023 - https://www.sphinx-doc.org/en/master/internals/release-process.html#python-version-support-policy - https://github.com/sphinx-doc/sphinx/pull/11511 - | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1859415334 | |
https://github.com/simonw/datasette/issues/516#issuecomment-1689159200 | https://api.github.com/repos/simonw/datasette/issues/516 | 1689159200 | IC_kwDOBm6k_c5kroYg | 9599 | 2023-08-23T02:15:36Z | 2023-08-23T02:15:36Z | OWNER | This could play havoc with unmerged PRs. I should merge any big ones (like the JavaScript plugins work) first. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
459509126 | |
https://github.com/simonw/datasette/issues/516#issuecomment-1689158712 | https://api.github.com/repos/simonw/datasette/issues/516 | 1689158712 | IC_kwDOBm6k_c5kroQ4 | 9599 | 2023-08-23T02:14:45Z | 2023-08-23T02:14:45Z | OWNER | Thinking about this again today. Posted about it on Discord: https://discord.com/channels/823971286308356157/823971286941302908/1143729300349132933 I won't enforce it in a `pytest` test, I'll enforce it with a CI lint check that's also in the `Justfile` here instead: https://github.com/simonw/datasette/blob/17ec309e14f9c2e90035ba33f2f38ecc5afba2fa/Justfile#L19-L23 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
459509126 | |
https://github.com/simonw/datasette/issues/516#issuecomment-1689154837 | https://api.github.com/repos/simonw/datasette/issues/516 | 1689154837 | IC_kwDOBm6k_c5krnUV | 9599 | 2023-08-23T02:08:33Z | 2023-08-23T02:08:50Z | OWNER | Browse this commit to see the result: https://github.com/simonw/datasette/tree/59a5d336bd4336bc53103922ada4bf726f4336c9 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
459509126 | |
https://github.com/simonw/datasette/issues/516#issuecomment-1689153446 | https://api.github.com/repos/simonw/datasette/issues/516 | 1689153446 | IC_kwDOBm6k_c5krm-m | 9599 | 2023-08-23T02:06:35Z | 2023-08-23T02:06:35Z | OWNER | I just tried this again today by dropping this into `.isort.cfg`: ```ini [settings] multi_line_output=3 include_trailing_comma=True force_grid_wrap=0 use_parentheses=True line_length=88 known_first_party=datasette ``` And running this in the root of the project: ``` isort . ``` It produced a huge diff, but when I ran the tests: ```bash pytest -n auto ``` The tests all passed. I'll push a PR. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
459509126 | |
https://github.com/simonw/datasette/pull/524#issuecomment-1689130061 | https://api.github.com/repos/simonw/datasette/issues/524 | 1689130061 | IC_kwDOBm6k_c5krhRN | 9599 | 2023-08-23T01:31:08Z | 2023-08-23T01:31:08Z | OWNER | This branch is WAY out of date. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
459689615 | |
https://github.com/simonw/datasette/issues/493#issuecomment-1689128911 | https://api.github.com/repos/simonw/datasette/issues/493 | 1689128911 | IC_kwDOBm6k_c5krg_P | 9599 | 2023-08-23T01:29:20Z | 2023-08-23T01:29:20Z | OWNER | It's going to be called `datasette.json` and the concept of metadata will be split out separately. See: - #2149 | { "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
449886319 | |
https://github.com/simonw/datasette/issues/275#issuecomment-1689128553 | https://api.github.com/repos/simonw/datasette/issues/275 | 1689128553 | IC_kwDOBm6k_c5krg5p | 9599 | 2023-08-23T01:28:37Z | 2023-08-23T01:28:37Z | OWNER | This is obsoleted by the work happening here: - #2093 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
324720095 | |
https://github.com/simonw/datasette/pull/2148#issuecomment-1689127479 | https://api.github.com/repos/simonw/datasette/issues/2148 | 1689127479 | IC_kwDOBm6k_c5krgo3 | 9599 | 2023-08-23T01:26:53Z | 2023-08-23T01:26:53Z | OWNER | @dependabot recreate | { "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1859415334 | |
https://github.com/simonw/datasette/pull/2149#issuecomment-1689125244 | https://api.github.com/repos/simonw/datasette/issues/2149 | 1689125244 | IC_kwDOBm6k_c5krgF8 | 9599 | 2023-08-23T01:23:27Z | 2023-08-23T01:23:27Z | OWNER | This is a really great start - tests pass, implementation looks clean, the new stubbed documentation page makes sense. Let's land it in `main` and iterate on it further in future PRs. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1861812208 | |
https://github.com/simonw/datasette/issues/2147#issuecomment-1686747420 | https://api.github.com/repos/simonw/datasette/issues/2147 | 1686747420 | IC_kwDOBm6k_c5kibkc | 9599 | 2023-08-21T17:31:42Z | 2023-08-21T17:34:19Z | OWNER | Are you talking just about queries submitted to `/database?sql=` using the interface on https://latest.datasette.io/fixtures?sql=select+*+from+facetable or are you interested in queries that are run to power other pages like https://latest.datasette.io/fixtures/facetable as well? I'll assume the former. There are a few ways you could solve this at the moment. The easiest would be with a piece of ASGI middleware that looks for URLs matching `/dbname?sql=...` and logs those. I played with a version of that a few years ago: https://simonwillison.net/2019/Dec/16/logging-sqlite-asgi-middleware/ - see also https://github.com/simonw/asgi-log-to-sqlite Then you can load that middleware from a plugin using https://docs.datasette.io/en/stable/plugin_hooks.html#asgi-wrapper-datasette That feels a bit delicate because it's relying on the URL design not changing, but I'm happy to confirm that URL is going to stay the same for Datasette 1.0 and I have no plans to change it ever. There's also a tracing mechanism built into Datasette itself that you could hook into. The internals of that are documented here: https://docs.datasette.io/en/stable/internals.html#datasette-tracer - but I don't yet consider it a 100% stable API. I don't plan to change it but I won't promise not to either. I used that mechanism in this plugin: https://datasette.io/plugins/datasette-pretty-traces - demonstrated here: https://latest-with-plugins.datasette.io/github/commits?_trace=1 The hackiest way to do this would be to patch Datasette itself and try to replace the `query_view`. This definitely isn't a documented, stable API though and would be very likely to break at arbitrary points in the future. So my recommendation for the moment is the ASGI middleware option. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1858228057 | |
https://github.com/simonw/datasette/issues/2147#issuecomment-1686749342 | https://api.github.com/repos/simonw/datasette/issues/2147 | 1686749342 | IC_kwDOBm6k_c5kicCe | 9599 | 2023-08-21T17:33:11Z | 2023-08-21T17:33:11Z | OWNER | I'm definitely open to suggestions for plugin hooks that might make this kind of thing easier. One idea I've been mulling is whether there should be a plugin hook that files on arbitrary views - similar to Django's `process_view` mechanism: https://docs.djangoproject.com/en/4.2/topics/http/middleware/#process-view That would allow people to setup code that runs before or after any of the default views in Datasette. I'm not yet 100% sold on the idea, because I worry about implementing it in a way that guarantees plugins won't break on future releases. But I'm open to considering it. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1858228057 | |
https://github.com/simonw/datasette/issues/2145#issuecomment-1686683596 | https://api.github.com/repos/simonw/datasette/issues/2145 | 1686683596 | IC_kwDOBm6k_c5kiL_M | 9599 | 2023-08-21T16:49:12Z | 2023-08-21T16:49:12Z | OWNER | Suggestion from @asg017 is that we say that if your row has a null primary key you don't get a link to a row page for that row. Which has some precedent, because our SQL view display doesn't link to row pages at all (since they don't make sense for views): https://latest.datasette.io/fixtures/simple_view | { "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1857234285 | |
https://github.com/simonw/sqlite-utils/issues/587#issuecomment-1685096381 | https://api.github.com/repos/simonw/sqlite-utils/issues/587 | 1685096381 | IC_kwDOCGYnMM5kcIe9 | 9599 | 2023-08-19T20:04:32Z | 2023-08-19T20:04:32Z | OWNER | I'm inclined to say this isn't a bug in `sqlite-utils` though - it's a bug in the code that calls it. So I'm not going to fix it here. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1857851384 | |
https://github.com/simonw/sqlite-utils/issues/587#issuecomment-1685096284 | https://api.github.com/repos/simonw/sqlite-utils/issues/587 | 1685096284 | IC_kwDOCGYnMM5kcIdc | 9599 | 2023-08-19T20:03:59Z | 2023-08-19T20:03:59Z | OWNER | Although this is revealing a problem in the underlying code (that schema is invalid), it also represents a regression: `sqlite-utils 3.34` ran this just fine, but it fails on `sqlite-utils 3.35` due to the change made in: - #577 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1857851384 | |
https://github.com/simonw/sqlite-utils/issues/587#issuecomment-1685096129 | https://api.github.com/repos/simonw/sqlite-utils/issues/587 | 1685096129 | IC_kwDOCGYnMM5kcIbB | 9599 | 2023-08-19T20:03:00Z | 2023-08-19T20:03:00Z | OWNER | Simplest possible recreation of the bug: ```bash python -c ' import sqlite_utils db = sqlite_utils.Database(memory=True) db.execute(""" CREATE TABLE "logs" ( [id] INTEGER PRIMARY KEY, [chat_id] INTEGER REFERENCES [log]([id]), [reply_to_id] INTEGER ); """) db["logs"].add_foreign_key("reply_to_id", "logs", "id") ' ``` That `chat_id` line is the line that causes the problem - because it is defining a reference to a table that no longer exists! | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1857851384 | |
https://github.com/simonw/datasette/issues/2145#issuecomment-1684530060 | https://api.github.com/repos/simonw/datasette/issues/2145 | 1684530060 | IC_kwDOBm6k_c5kZ-OM | 9599 | 2023-08-18T23:09:03Z | 2023-08-18T23:09:14Z | OWNER | Ran a quick benchmark on ChatGPT Code Interpreter: https://chat.openai.com/share/8357dc01-a97e-48ae-b35a-f06249935124 Conclusion from there is that this query returns fast no matter how much the table grows: ```sql SELECT EXISTS(SELECT 1 FROM "nasty" WHERE "id" IS NULL) ``` So detecting if a table contains any null primary keys is definitely feasible without a performance hit. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1857234285 | |
https://github.com/simonw/datasette/issues/2145#issuecomment-1684526447 | https://api.github.com/repos/simonw/datasette/issues/2145 | 1684526447 | IC_kwDOBm6k_c5kZ9Vv | 9599 | 2023-08-18T23:05:02Z | 2023-08-18T23:05:02Z | OWNER | How expensive is it to detect if a SQLite table contains at least one `null` primary key? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1857234285 | |
https://github.com/simonw/datasette/issues/2145#issuecomment-1684525943 | https://api.github.com/repos/simonw/datasette/issues/2145 | 1684525943 | IC_kwDOBm6k_c5kZ9N3 | 9599 | 2023-08-18T23:04:14Z | 2023-08-18T23:04:14Z | OWNER | This is hard. I tried this: ```python def path_from_row_pks(row, pks, use_rowid, quote=True): """Generate an optionally tilde-encoded unique identifier for a row from its primary keys.""" if use_rowid or any(row[pk] is None for pk in pks): bits = [row["rowid"]] else: bits = [ row[pk]["value"] if isinstance(row[pk], dict) else row[pk] for pk in pks ] if quote: bits = [tilde_encode(str(bit)) for bit in bits] else: bits = [str(bit) for bit in bits] return ",".join(bits) ``` The ` if use_rowid or any(row[pk] is None for pk in pks)` bit is new. But I got this error on http://127.0.0.1:8003/nulls/nasty : ``` File "/Users/simon/Dropbox/Development/datasette/datasette/views/table.py", line 1364, in run_display_columns_and_rows display_columns, display_rows = await display_columns_and_rows( File "/Users/simon/Dropbox/Development/datasette/datasette/views/table.py", line 186, in display_columns_and_rows pk_path = path_from_row_pks(row, pks, not pks, False) File "/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py", line 124, in path_from_row_pks bits = [row["rowid"]] IndexError: No item with that key ``` Because the SQL query I ran to populate the page didn't know that it would need to select `rowid` as well. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1857234285 | |
https://github.com/simonw/datasette/issues/2145#issuecomment-1684525054 | https://api.github.com/repos/simonw/datasette/issues/2145 | 1684525054 | IC_kwDOBm6k_c5kZ8_- | 9599 | 2023-08-18T23:02:26Z | 2023-08-18T23:02:26Z | OWNER | Creating a quick test database: ```bash sqlite-utils create-table nulls.db nasty id text --pk id sqlite-utils nulls.db 'insert into nasty (id) values (null)' ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1857234285 | |
https://github.com/simonw/datasette/issues/2145#issuecomment-1684523322 | https://api.github.com/repos/simonw/datasette/issues/2145 | 1684523322 | IC_kwDOBm6k_c5kZ8k6 | 9599 | 2023-08-18T22:59:14Z | 2023-08-18T22:59:14Z | OWNER | Except it looks like the Links from other tables section is broken: <img width="649" alt="CleanShot 2023-08-18 at 15 58 54@2x" src="https://github.com/simonw/datasette/assets/9599/ddebf0a7-9855-4cf0-b283-f06f8c25414d"> | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1857234285 | |
https://github.com/simonw/datasette/issues/2145#issuecomment-1684522567 | https://api.github.com/repos/simonw/datasette/issues/2145 | 1684522567 | IC_kwDOBm6k_c5kZ8ZH | 9599 | 2023-08-18T22:58:07Z | 2023-08-18T22:58:07Z | OWNER | Here's a prototype of that: ```diff diff --git a/datasette/app.py b/datasette/app.py index b2644ace..acc55249 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -1386,7 +1386,7 @@ class Datasette: ) add_route( RowView.as_view(self), - r"/(?P<database>[^\/\.]+)/(?P<table>[^/]+?)/(?P<pks>[^/]+?)(\.(?P<format>\w+))?$", + r"/(?P<database>[^\/\.]+)/(?P<table>[^/]+?)/(?P<pks>[A-Za-z0-9\_\-\~]+|\.\d+)(\.(?P<format>\w+))?$", ) add_route( TableInsertView.as_view(self), @@ -1440,7 +1440,15 @@ class Datasette: async def resolve_row(self, request): db, table_name, _ = await self.resolve_table(request) pk_values = urlsafe_components(request.url_vars["pks"]) - sql, params, pks = await row_sql_params_pks(db, table_name, pk_values) + + if len(pk_values) == 1 and pk_values[0].startswith("."): + # It's a special .rowid value + pk_values = (pk_values[0][1:],) + sql, params, pks = await row_sql_params_pks( + db, table_name, pk_values, rowid=True + ) + else: + sql, params, pks = await row_sql_params_pks(db, table_name, pk_values) results = await db.execute(sql, params, truncate=True) row = results.first() if row is None: diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py index c388673d..96669281 100644 --- a/datasette/utils/__init__.py +++ b/datasette/utils/__init__.py @@ -1206,9 +1206,12 @@ def truncate_url(url, length): return url[: length - 1] + "…" -async def row_sql_params_pks(db, table, pk_values): +async def row_sql_params_pks(db, table, pk_values, rowid=False): pks = await db.primary_keys(table) - use_rowid = not pks + if rowid: + use_rowid = True + else: + use_rowid = not pks select = "*" if use_rowid: select = "rowid, *" ``` It works: <img width="649" alt="CleanShot 202… | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1857234285 | |
https://github.com/simonw/datasette/issues/2145#issuecomment-1684505071 | https://api.github.com/repos/simonw/datasette/issues/2145 | 1684505071 | IC_kwDOBm6k_c5kZ4Hv | 9599 | 2023-08-18T22:44:35Z | 2023-08-18T22:44:35Z | OWNER | Also relevant: https://github.com/simonw/datasette/blob/943df09dcca93c3b9861b8c96277a01320db8662/datasette/utils/__init__.py#L1147-L1153 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1857234285 | |
https://github.com/simonw/datasette/issues/2145#issuecomment-1684504398 | https://api.github.com/repos/simonw/datasette/issues/2145 | 1684504398 | IC_kwDOBm6k_c5kZ39O | 9599 | 2023-08-18T22:43:31Z | 2023-08-18T22:43:46Z | OWNER | `(?P<pks>[^/]+?)` could instead be a regex that is restricted to the tilde-encoded set of characters, or `\.\d+`. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1857234285 | |
https://github.com/simonw/datasette/issues/2145#issuecomment-1684504051 | https://api.github.com/repos/simonw/datasette/issues/2145 | 1684504051 | IC_kwDOBm6k_c5kZ33z | 9599 | 2023-08-18T22:43:06Z | 2023-08-18T22:43:06Z | OWNER | Here's the regex in question at the moment: https://github.com/simonw/datasette/blob/943df09dcca93c3b9861b8c96277a01320db8662/datasette/app.py#L1387-L1390 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1857234285 | |
https://github.com/simonw/datasette/issues/2145#issuecomment-1684503587 | https://api.github.com/repos/simonw/datasette/issues/2145 | 1684503587 | IC_kwDOBm6k_c5kZ3wj | 9599 | 2023-08-18T22:42:28Z | 2023-08-18T22:42:39Z | OWNER | I could set a rule that extensions (including custom render extensions set by plugins) must not be valid integers, and teach Datasette that `/\.\d+` is the indication of a `rowid`. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1857234285 | |
https://github.com/simonw/datasette/issues/2145#issuecomment-1684503189 | https://api.github.com/repos/simonw/datasette/issues/2145 | 1684503189 | IC_kwDOBm6k_c5kZ3qV | 9599 | 2023-08-18T22:41:51Z | 2023-08-18T22:41:51Z | OWNER | ```pycon >>> tilde_encode("~") '~7E' >>> tilde_encode(".") '~2E' >>> tilde_encode("-") '-' ``` I think `.` might be the way to do this: /database/table/.4 But... I worry about that colliding with my URL routing code that spots the difference between these: /database/table/.4 /database/table/.4.json /database/table/.4.csv etc. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1857234285 | |
https://github.com/simonw/datasette/issues/2145#issuecomment-1684502278 | https://api.github.com/repos/simonw/datasette/issues/2145 | 1684502278 | IC_kwDOBm6k_c5kZ3cG | 9599 | 2023-08-18T22:40:20Z | 2023-08-18T22:40:20Z | OWNER | From reviewing https://simonwillison.net/2022/Mar/19/weeknotes/ unreserved = ALPHA / DIGIT / "-" / "." / "_" / "~" That's how I chose the tilde character - but it also suggests that I could use `-` or `.` or `_` for my new `rowid` encoding. So maybe `/database/table/_4` could indicate "the row with `rowid` of 4". No, that doesn't work: ```pycon >>> from datasette.utils import tilde_encode >>> tilde_encode("_") '_' ``` I need a character which tilde-encoding does indeed encode. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1857234285 | |
https://github.com/simonw/datasette/issues/2145#issuecomment-1684500540 | https://api.github.com/repos/simonw/datasette/issues/2145 | 1684500540 | IC_kwDOBm6k_c5kZ3A8 | 9599 | 2023-08-18T22:37:37Z | 2023-08-18T22:37:37Z | OWNER | I just found this and panicked, thinking maybe tilde encoding is a bad idea after all! https://jkorpela.fi/tilde.html But... "Date of last update: 1999-08-27" - I think I'm OK. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1857234285 | |
https://github.com/simonw/datasette/issues/2145#issuecomment-1684500172 | https://api.github.com/repos/simonw/datasette/issues/2145 | 1684500172 | IC_kwDOBm6k_c5kZ27M | 9599 | 2023-08-18T22:37:04Z | 2023-08-18T22:37:04Z | OWNER | Looking at the way these URLs work: because the components themselves in `a~2Fb,~2Ec-d` are tilde-encoded, any character that's "safe" in tilde-encoding could be used to indicate "this is actually a rowid". | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1857234285 | |
https://github.com/simonw/datasette/issues/2145#issuecomment-1684498947 | https://api.github.com/repos/simonw/datasette/issues/2145 | 1684498947 | IC_kwDOBm6k_c5kZ2oD | 9599 | 2023-08-18T22:35:04Z | 2023-08-18T22:35:04Z | OWNER | The most interesting row URL in the fixtures database right now is this one: https://latest.datasette.io/fixtures/compound_primary_key/a~2Fb,~2Ec-d <img width="163" alt="CleanShot 2023-08-18 at 15 34 52@2x" src="https://github.com/simonw/datasette/assets/9599/dd79b0b6-e1c3-4e30-b64b-973b062bfe70"> | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1857234285 | |
https://github.com/simonw/datasette/issues/2145#issuecomment-1684497642 | https://api.github.com/repos/simonw/datasette/issues/2145 | 1684497642 | IC_kwDOBm6k_c5kZ2Tq | 9599 | 2023-08-18T22:32:53Z | 2023-08-18T22:32:53Z | OWNER | Here's a potential solution: make it so ALL `rowid` tables in SQLite can be optionally addressed by their `rowid` instead of by their primary key. Then teach the code that outputs the URL to a row page to spot if there are `null` primary keys and switch to that alternative addressing mechanism instead. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1857234285 | |
https://github.com/simonw/datasette/issues/2145#issuecomment-1684497000 | https://api.github.com/repos/simonw/datasette/issues/2145 | 1684497000 | IC_kwDOBm6k_c5kZ2Jo | 9599 | 2023-08-18T22:31:53Z | 2023-08-18T22:31:53Z | OWNER | So it sounds like SQLite does ensure that a `rowid` before it allows a primary key to be null. So one solution here would be to detect a null primary key and switch that table over to using `rowid` URLs instead. The key problem we're trying to solve here after all is how to link to a row: https://latest.datasette.io/fixtures/infinity/1 But when would we run that check? And does every row in the table get a new `/rowid/` URL just because someone messed up and inserted a `null` by mistake? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1857234285 | |
https://github.com/simonw/datasette/issues/2145#issuecomment-1684495674 | https://api.github.com/repos/simonw/datasette/issues/2145 | 1684495674 | IC_kwDOBm6k_c5kZ106 | 9599 | 2023-08-18T22:29:47Z | 2023-08-18T22:29:47Z | OWNER | https://www.sqlite.org/lang_createtable.html#the_primary_key says: >According to the SQL standard, PRIMARY KEY should always imply NOT NULL. Unfortunately, due to a bug in some early versions, this is not the case in SQLite. Unless the column is an [INTEGER PRIMARY KEY](https://www.sqlite.org/lang_createtable.html#rowid) or the table is a [WITHOUT ROWID](https://www.sqlite.org/withoutrowid.html) table or a [STRICT](https://www.sqlite.org/stricttables.html) table or the column is declared NOT NULL, SQLite allows NULL values in a PRIMARY KEY column. SQLite could be fixed to conform to the standard, but doing so might break legacy applications. Hence, it has been decided to merely document the fact that SQLite allows NULLs in most PRIMARY KEY columns. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1857234285 | |
https://github.com/simonw/datasette/issues/2145#issuecomment-1684494464 | https://api.github.com/repos/simonw/datasette/issues/2145 | 1684494464 | IC_kwDOBm6k_c5kZ1iA | 9599 | 2023-08-18T22:27:51Z | 2023-08-18T22:28:40Z | OWNER | Oh wow, null primary keys are bad news... SQLite lets you insert multiple rows with the same `null` value! ```pycon >>> import sqlite_utils >>> db = sqlite_utils.Database(memory=True) >>> db["foo"].insert({"id": None, "name": "No ID"}, pk="id") <Table foo (id, name)> >>> db.schema 'CREATE TABLE [foo] (\n [id] TEXT PRIMARY KEY,\n [name] TEXT\n);' >>> db["foo"].insert({"id": None, "name": "No ID"}, pk="id") <Table foo (id, name)> >>> db.schema 'CREATE TABLE [foo] (\n [id] TEXT PRIMARY KEY,\n [name] TEXT\n);' >>> list(db["foo"].rows) [{'id': None, 'name': 'No ID'}, {'id': None, 'name': 'No ID'}] >>> list(db.query('select * from foo where id = null')) [] >>> list(db.query('select * from foo where id is null')) [{'id': None, 'name': 'No ID'}, {'id': None, 'name': 'No ID'}] ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1857234285 | |
https://github.com/simonw/datasette/issues/2143#issuecomment-1684488526 | https://api.github.com/repos/simonw/datasette/issues/2143 | 1684488526 | IC_kwDOBm6k_c5kZ0FO | 9599 | 2023-08-18T22:18:39Z | 2023-08-18T22:18:39Z | OWNER | > Another option would be, instead of flat `datasette.json`/`datasette.yaml` files, we could instead use a Python file, like `datasette_config.py`. That way one could dynamically generate config (ex dev vs prod, auto-discover credentials, etc.). Kinda like Django settings. > Another option would be, instead of flat `datasette.json`/`datasette.yaml` files, we could instead use a Python file, like `datasette_config.py`. That way one could dynamically generate config (ex dev vs prod, auto-discover credentials, etc.). Kinda like Django settings. I'm not a fan of that. I feel like software history is full of examples of projects that implemented configuration-as-code and then later regretted it - the most recent example is `setup.py` in Python turning into `pyproject.yaml`, but I feel like I've seen that pattern play out elsewhere too. I don't think having people dynamically generate JSON/YAML for their configuration is a big burden. I'd have to see some very compelling use-cases to convince me otherwise. That said, I do really like a bias towards settings that can be changed at runtime. Datasette has suffered a bit from some settings that can't be easily changed at runtime already - hence my gnarly https://github.com/simonw/datasette-remote-metadata plugin. For things like Datasette Cloud for example the more people can configure without rebooting their container the better! I don't think live reconfiguration at runtime is incompatible with JSON/YAML configuration though. Caddy is one of my favourite examples of software that can be entirely re-configured at runtime by POSTING a big blob of JSON to it: https://caddyserver.com/docs/quick-starts/api | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1855885427 | |
https://github.com/simonw/datasette/issues/2143#issuecomment-1684485591 | https://api.github.com/repos/simonw/datasette/issues/2143 | 1684485591 | IC_kwDOBm6k_c5kZzXX | 9599 | 2023-08-18T22:14:35Z | 2023-08-18T22:14:35Z | OWNER | Actually there is one thing that I'm not comfortable about with respect to the existing design: the way the database / tables stuff is nested. They assume that the user will attach the database to Datasette using a fixed name - `docs.db` or whatever. But what if we want to support users downloading databases from each other and attaching them to Datasette where those DBs might carry some of their own configuration? Moving metadata into the databases makes sense there, but what about database-specific settings like the default sort order for a table, or configured canned queries? Having those tied to the filename of the database itself feels unpleasant to me. But how else could we handle this? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1855885427 | |
https://github.com/simonw/datasette/issues/2143#issuecomment-1684484426 | https://api.github.com/repos/simonw/datasette/issues/2143 | 1684484426 | IC_kwDOBm6k_c5kZzFK | 9599 | 2023-08-18T22:12:52Z | 2023-08-18T22:12:52Z | OWNER | Yeah, I'm convinced by that. There's not point in having both `settings.json` and `datasette.json`. I like `datasette.json` ( / `datasette.yml`) as a name. That can be the file that lives in your config directory too, so if you run `datasette .` in a folder containing `datasette.yml` all of those settings get picked up. Here's a thought for how it could look - I'll go with the YAML format because I expect that to be the default most people use, just because it supports multi-line strings better. I based this on the big example at https://docs.datasette.io/en/1.0a3/metadata.html#using-yaml-for-metadata - and combined some bits from https://docs.datasette.io/en/1.0a3/authentication.html as well. ```yaml title: Demonstrating Metadata from YAML description_html: |- <p>This description includes a long HTML string</p> <ul> <li>YAML is better for embedding HTML strings than JSON!</li> </ul> settings: default_page_size: 10 max_returned_rows: 3000 sql_time_limit_ms": 8000 databases: docs: permissions: create-table: id: editor fixtures: tables: no_primary_key: hidden: true queries: neighborhood_search: sql: |- select neighborhood, facet_cities.name, state from facetable join facet_cities on facetable.city_id = facet_cities.id where neighborhood like '%' || :text || '%' order by neighborhood; title: Search neighborhoods description_html: |- <p>This demonstrates <em>basic</em> LIKE search permissions: debug-menu: id: '*' plugins: datasette-ripgrep: path: /usr/local/lib/python3.11/site-packages ``` I'm inclined to say we try to be a super-set of the existing `metadata.yml` format, at least where it makes sense to do so. That way the upgrade path is smooth for people. Also, I don't think the format itself is terrible - it's the name that's the big problem. In this example I've mixed in one extra concept: that `settings:` block wi… | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1855885427 | |
https://github.com/simonw/datasette/issues/2145#issuecomment-1684384750 | https://api.github.com/repos/simonw/datasette/issues/2145 | 1684384750 | IC_kwDOBm6k_c5kZavu | 9599 | 2023-08-18T20:07:18Z | 2023-08-18T20:07:18Z | OWNER | The big challenge here is what the URL to that row page should look like. How can I encode a `None` in a form that can be encoded and decoded without clashing with primary keys that are the string `None` or `null`? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1857234285 | |
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1684235760 | https://api.github.com/repos/simonw/sqlite-utils/issues/577 | 1684235760 | IC_kwDOCGYnMM5kY2Xw | 9599 | 2023-08-18T17:43:11Z | 2023-08-18T17:43:11Z | OWNER | Here's a new plugin that brings back the `sqlite_master` modifying version, for those that can use it: https://github.com/simonw/sqlite-utils-fast-fks | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1817289521 | |
https://github.com/simonw/datasette/issues/2143#issuecomment-1683429959 | https://api.github.com/repos/simonw/datasette/issues/2143 | 1683429959 | IC_kwDOBm6k_c5kVxpH | 9599 | 2023-08-18T06:43:33Z | 2023-08-18T15:19:07Z | OWNER | The single biggest design challenge I've had with metadata relates to how it should or should not be inherited. If you apply a license to a Datasette instance, it feels like that should flow down to cover all of the databases and all of the tables within those databases. If the license is at the database level, it should cover all tables. But... should source do the same thing? I made it behave the same way as license, but it's presumably common for one database to have a single license but multiple different sources of data. Then there's title - should that inherit? It feels like title should apply to only one level - you may want a title that applies to the instance, then a different custom title for databases and tables. Here's the current state of play for metadata: https://docs.datasette.io/en/1.0a3/metadata.html So there's `title` and `description` - and I'll be honest, I'm not 100% sure even I understand how those should be inherited down by tables/etc. There's `description_html` which over-rides the `description` if it is set. It's a useful customization hack, but a bit surprising. Then there are these six: - `license` - `license_url` - `source` - `source_url` - `about` - `about_url` I added `about` later than the others, because I realized that plenty of my own projects needed a link to an article explaining them somewhere - e.g. https://scotrail.datasette.io/ Tables can also have column descriptions - just a string for each column. There's a demo of those here: https://latest.datasette.io/fixtures/roadside_attractions And then there's all of the other stuff, most of which feels much more like "settings" than "metadata": - `sort: created` - the custom sort order - `size: 10` for a custom page size for a specific table - `sortable_columns` to set which columns can be used to sort - `hidden: true` to hide a table - `label_column: title` is an interesting one - it lets you hint to Datasette which column should be displayed when there is a foreign key relations… | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1855885427 | |
https://github.com/simonw/datasette/issues/2143#issuecomment-1683420879 | https://api.github.com/repos/simonw/datasette/issues/2143 | 1683420879 | IC_kwDOBm6k_c5kVvbP | 9599 | 2023-08-18T06:33:24Z | 2023-08-18T15:15:34Z | OWNER | I completely agree: metadata is a mess, and it deserves our attention. > 1. Metadata cannot be updated without re-starting the entire Datasette instance. That's not completely true - there are hacks around that. I have a plugin that applies one set of gnarly hacks for that here: https://github.com/simonw/datasette-remote-metadata - it's pretty grim though! > 2. The `metadata.json`/`metadata.yaml` has become a kitchen sink of unrelated (imo) features like plugin config, authentication config, canned queries 100% this: it's a complete mess. Datasette used to have a `datasette --config foo:bar` mechanism, which I deprecated in favour of `datasette --setting foo bar` partly because I wanted to free up `--config` for pointing at a real config file, so we could stop dropping everything in `--metadata metadata.yml`. > 3. The Python APIs for defining extra metadata are a bit awkward (the `datasette.metadata()` class, `get_metadata()` hook, etc.) Yes, they're not pretty at all. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1855885427 | |
https://github.com/simonw/datasette/issues/2143#issuecomment-1683443891 | https://api.github.com/repos/simonw/datasette/issues/2143 | 1683443891 | IC_kwDOBm6k_c5kV1Cz | 9599 | 2023-08-18T06:58:15Z | 2023-08-18T06:58:15Z | OWNER | Hah, that `--plugin-secret` thing was a messy solution I came up with to the problem that all metadata is visible at `/-/metadata` - so if you need to stash a secret you need a way to keep it not-visible in there! Hence the whole `$env` mess: https://docs.datasette.io/en/stable/plugins.html#secret-configuration-values ```json { "plugins": { "datasette-auth-github": { "client_secret": { "$env": "GITHUB_CLIENT_SECRET" } } } } ``` If configuration and metadata were separate we could ditch that whole messy situation - configuration can stay hidden, metadata can stay public. Though I have been thinking that Datasette might benefit from a "secrets" mechanism that's separate from configuration and metadata... kind of like what LLM has: https://llm.datasette.io/en/stable/help.html#llm-keys-help | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1855885427 | |
https://github.com/simonw/datasette/issues/2143#issuecomment-1683440597 | https://api.github.com/repos/simonw/datasette/issues/2143 | 1683440597 | IC_kwDOBm6k_c5kV0PV | 9599 | 2023-08-18T06:54:49Z | 2023-08-18T06:54:49Z | OWNER | A related point that I've been considering a lot recently: it turns out that sometimes I really want to define settings on the CLI instead of in a file, purely for convenience. It's pretty annoying when I want to try out a new plugin but I have to create a dedicated `metadata.yml` file for it just to setup a single option - I'd love to have the option to be able to run this instead: ```bash datasette data.db --plugin-setting datasette-upload-csvs default-database data ``` So maybe there's a world in which all of the settings can be applied in a `datasette.yml` file OR with command-line options. That gets trickier when you need to pass a nested structure or similar, but we could always support those as JSON: ```bash datasette data.db --plugin-setting datasette-emoji-reactions emoji '["😼", "🐺"]' ``` Note that we kind of have precedent for this in `datasette publish`: https://docs.datasette.io/en/stable/publish.html#custom-metadata-and-plugins ```bash datasette publish heroku my_database.db \ --name my-heroku-app-demo \ --install=datasette-auth-github \ --plugin-secret datasette-auth-github client_id your_client_id \ --plugin-secret datasette-auth-github client_secret your_client_secret ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1855885427 | |
https://github.com/simonw/datasette/issues/2143#issuecomment-1683435579 | https://api.github.com/repos/simonw/datasette/issues/2143 | 1683435579 | IC_kwDOBm6k_c5kVzA7 | 9599 | 2023-08-18T06:49:39Z | 2023-08-18T06:49:39Z | OWNER | My ideal situation then would be something like this: - Metadata itself is VERY clearly described, including sensible rules for metadata inheritance where it makes sense. There is a `datasette.X` method for accessing it which is much more intuitive than `datasette.metadata()`. - It's possible that method should be an `async` method, because that would support things like plugins that lookup metadata in database tables better. - All templates etc switch to the new, clean, intuitive metadata mechanism before 1.0. - I'm interested in the option of metadata being able to live in a `_datasette_metadata` table in the databases themselves - either as a plugin or as a core feature. I think it makes a lot of sense for metadata to optionally live with the data that it describes. - Configuration gets split from metadata. The stuff that configures Datasette no longer lives in the `metadata.yml` file - it lives in `config.yml` (or even `datasette.yml`). Currently we have three types of things: - Metadata - information about the data - Configuration - stuff like "these columns should be sortable" and "this is configured as `fts_table`" and suchlike - Settings - the stuff that you pass to `datasette --setting x y` on server start. Should settings and configuration be separate? I'm not 100% sure that they should - maybe those two concepts should be combined somehow. Configuration directory mode needs to be considered too: https://docs.datasette.io/en/stable/settings.html#configuration-directory-mode - interestingly it already has a thing where it can pick up settings from a `settings.json` file - where settings are things like `datasette --setting sql_time_limit_ms 4000`. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1855885427 | |
https://github.com/simonw/sqlite-utils/issues/586#issuecomment-1683404978 | https://api.github.com/repos/simonw/sqlite-utils/issues/586 | 1683404978 | IC_kwDOCGYnMM5kVriy | 9599 | 2023-08-18T06:13:46Z | 2023-08-18T06:13:46Z | OWNER | I shipped the view recreating fix in `datasette-edit-schema`, so at least I can start exercising that fix and see if it has any weird issues. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1856075668 | |
https://github.com/simonw/sqlite-utils/issues/586#issuecomment-1683398866 | https://api.github.com/repos/simonw/sqlite-utils/issues/586 | 1683398866 | IC_kwDOCGYnMM5kVqDS | 9599 | 2023-08-18T06:05:50Z | 2023-08-18T06:06:42Z | OWNER | Options: - Provide a `recreate_views: bool` parameter to `table.transform()` controlling if views that might reference this table are stashed and dropped and recreated within a transaction as part of the operation. But should that be `True` or `False` by default? - Read that `PRAGMA` and automatically do that view workaround if it's turned on - Toggle that `PRAGMA` off for the duration of the `.transform()` operation and on again at the end. Does it only affect the current connection? - Try the `transform()` in a transaction, detect the `"error in view"`, `"no such table"`error, if spotted then do the VIEW workaround and try again I'm on the fence as to which of these I like the most. I'm tempted to go with the one which just drops VIEWS and recreates them all the time, because it feels simpler. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1856075668 | |
https://github.com/simonw/sqlite-utils/issues/586#issuecomment-1683396150 | https://api.github.com/repos/simonw/sqlite-utils/issues/586 | 1683396150 | IC_kwDOCGYnMM5kVpY2 | 9599 | 2023-08-18T06:02:18Z | 2023-08-18T06:06:31Z | OWNER | More notes in here: - https://github.com/simonw/datasette-edit-schema/issues/35#issuecomment-1683392873 Not all Python/SQLite installations exhibit this problem by default! It turns out this is controlled by the `legacy_alter_table` pragma: https://sqlite.org/pragma.html#pragma_legacy_alter_table If that PRAGMA is turned on (default in newer SQLites) then `alter table` will error if you try to rename a table that is referenced in a view. Here's a one-liner to test if it is on or not: ```bash python -c 'import sqlite3; print(sqlite3.connect(":memory:").execute("PRAGMA legacy_alter_table").fetchall())' ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1856075668 | |
https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683217284 | https://api.github.com/repos/simonw/sqlite-utils/issues/585 | 1683217284 | IC_kwDOCGYnMM5kU9uE | 9599 | 2023-08-18T01:50:21Z | 2023-08-18T01:50:21Z | OWNER | And a test of the `--sql` option: ```bash sqlite-utils create-table /tmp/t.db places id integer name text country integer city integer continent integer --pk id sqlite-utils create-table /tmp/t.db country id integer name text sqlite-utils create-table /tmp/t.db city id integer name text sqlite-utils create-table /tmp/t.db continent id integer name text sqlite-utils transform /tmp/t.db places --add-foreign-key country country id --add-foreign-key continent continent id --sql ``` Outputs: ```sql CREATE TABLE [places_new_6a705d2f5a13] ( [id] INTEGER PRIMARY KEY, [name] TEXT, [country] INTEGER REFERENCES [country]([id]), [city] INTEGER, [continent] INTEGER REFERENCES [continent]([id]) ); INSERT INTO [places_new_6a705d2f5a13] ([id], [name], [country], [city], [continent]) SELECT [id], [name], [country], [city], [continent] FROM [places]; DROP TABLE [places]; ALTER TABLE [places_new_6a705d2f5a13] RENAME TO [places]; ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1855894222 | |
https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683212074 | https://api.github.com/repos/simonw/sqlite-utils/issues/585 | 1683212074 | IC_kwDOCGYnMM5kU8cq | 9599 | 2023-08-18T01:43:54Z | 2023-08-18T01:43:54Z | OWNER | Some manual testing: ```bash sqlite-utils create-table /tmp/t.db places id integer name text country integer city integer continent integer --pk id sqlite-utils schema /tmp/t.db ``` ```sql CREATE TABLE [places] ( [id] INTEGER PRIMARY KEY, [name] TEXT, [country] INTEGER, [city] INTEGER, [continent] INTEGER ); ``` ```bash sqlite-utils create-table /tmp/t.db country id integer name text sqlite-utils create-table /tmp/t.db city id integer name text sqlite-utils create-table /tmp/t.db continent id integer name text sqlite-utils schema /tmp/t.db ``` ```sql CREATE TABLE [places] ( [id] INTEGER PRIMARY KEY, [name] TEXT, [country] INTEGER, [city] INTEGER, [continent] INTEGER ); CREATE TABLE [country] ( [id] INTEGER, [name] TEXT ); CREATE TABLE [city] ( [id] INTEGER, [name] TEXT ); CREATE TABLE [continent] ( [id] INTEGER, [name] TEXT ); ``` ```bash sqlite-utils transform /tmp/t.db places --add-foreign-key country country id --add-foreign-key continent continent id sqlite-utils schema /tmp/t.db ``` ```sql CREATE TABLE [country] ( [id] INTEGER, [name] TEXT ); CREATE TABLE [city] ( [id] INTEGER, [name] TEXT ); CREATE TABLE [continent] ( [id] INTEGER, [name] TEXT ); CREATE TABLE "places" ( [id] INTEGER PRIMARY KEY, [name] TEXT, [country] INTEGER REFERENCES [country]([id]), [city] INTEGER, [continent] INTEGER REFERENCES [continent]([id]) ); ``` ```bash sqlite-utils transform /tmp/t.db places --drop-foreign-key country sqlite-utils schema /tmp/t.db places ``` ```sql CREATE TABLE "places" ( [id] INTEGER PRIMARY KEY, [name] TEXT, [country] INTEGER, [city] INTEGER, [continent] INTEGER REFERENCES [continent]([id]) ) ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1855894222 | |
https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683201239 | https://api.github.com/repos/simonw/sqlite-utils/issues/585 | 1683201239 | IC_kwDOCGYnMM5kU5zX | 9599 | 2023-08-18T01:30:46Z | 2023-08-18T01:30:46Z | OWNER | Help can now look like this: ``` --drop-foreign-key TEXT Drop foreign key constraint for this column --add-foreign-key <TEXT TEXT TEXT>... Add a foreign key constraint from a column to another table with another column ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1855894222 | |
https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683200128 | https://api.github.com/repos/simonw/sqlite-utils/issues/585 | 1683200128 | IC_kwDOCGYnMM5kU5iA | 9599 | 2023-08-18T01:29:00Z | 2023-08-18T01:29:00Z | OWNER | I'm not going to implement the `foreign_keys=` option that entirely replaces existing foreign keys - I'll just do a `--add-foreign-key` multi-option. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1855894222 | |
https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683198740 | https://api.github.com/repos/simonw/sqlite-utils/issues/585 | 1683198740 | IC_kwDOCGYnMM5kU5MU | 9599 | 2023-08-18T01:26:47Z | 2023-08-18T01:26:47Z | OWNER | The only CLI feature that supports providing just the column name appears to be this: ```bash sqlite-utils add-foreign-key --help ``` ``` Usage: sqlite-utils add-foreign-key [OPTIONS] PATH TABLE COLUMN [OTHER_TABLE] [OTHER_COLUMN] Add a new foreign key constraint to an existing table Example: sqlite-utils add-foreign-key my.db books author_id authors id WARNING: Could corrupt your database! Back up your database file first. ``` I can drop that WARNING now since I'm not writing to `sqlite_master` any more. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1855894222 | |
https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683197882 | https://api.github.com/repos/simonw/sqlite-utils/issues/585 | 1683197882 | IC_kwDOCGYnMM5kU4-6 | 9599 | 2023-08-18T01:25:53Z | 2023-08-18T01:25:53Z | OWNER | Probably most relevant here is this snippet from: ```bash sqlite-utils create-table --help ``` ``` --default <TEXT TEXT>... Default value that should be set for a column --fk <TEXT TEXT TEXT>... Column, other table, other column to set as a foreign key ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1855894222 | |
https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683195669 | https://api.github.com/repos/simonw/sqlite-utils/issues/585 | 1683195669 | IC_kwDOCGYnMM5kU4cV | 9599 | 2023-08-18T01:24:57Z | 2023-08-18T01:24:57Z | OWNER | Currently: ```bash sqlite-utils transform --help ``` ``` Usage: sqlite-utils transform [OPTIONS] PATH TABLE Transform a table beyond the capabilities of ALTER TABLE Example: sqlite-utils transform mydb.db mytable \ --drop column1 \ --rename column2 column_renamed Options: --type <TEXT CHOICE>... Change column type to INTEGER, TEXT, FLOAT or BLOB --drop TEXT Drop this column --rename <TEXT TEXT>... Rename this column to X -o, --column-order TEXT Reorder columns --not-null TEXT Set this column to NOT NULL --not-null-false TEXT Remove NOT NULL from this column --pk TEXT Make this column the primary key --pk-none Remove primary key (convert to rowid table) --default <TEXT TEXT>... Set default value for this column --default-none TEXT Remove default from this column --drop-foreign-key TEXT Drop foreign key constraint for this column --sql Output SQL without executing it --load-extension TEXT Path to SQLite extension, with optional :entrypoint -h, --help Show this message and exit. ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1855894222 | |
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683164661 | https://api.github.com/repos/simonw/sqlite-utils/issues/584 | 1683164661 | IC_kwDOCGYnMM5kUw31 | 9599 | 2023-08-18T00:45:53Z | 2023-08-18T00:45:53Z | OWNER | More updated documentation: - https://sqlite-utils--584.org.readthedocs.build/en/584/reference.html#sqlite_utils.db.Table.transform - https://sqlite-utils--584.org.readthedocs.build/en/584/python-api.html#python-api-transform-add-foreign-key-constraints | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1855838223 | |
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683145819 | https://api.github.com/repos/simonw/sqlite-utils/issues/584 | 1683145819 | IC_kwDOCGYnMM5kUsRb | 9599 | 2023-08-18T00:17:26Z | 2023-08-18T00:17:26Z | OWNER | Updated documentation: https://sqlite-utils--584.org.readthedocs.build/en/584/python-api.html#adding-foreign-key-constraints | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1855838223 | |
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683145110 | https://api.github.com/repos/simonw/sqlite-utils/issues/584 | 1683145110 | IC_kwDOCGYnMM5kUsGW | 9599 | 2023-08-18T00:16:28Z | 2023-08-18T00:16:48Z | OWNER | One last piece of documentation: need to document the new option to `table.transform()` and `table.transform_sql()`: https://github.com/simonw/sqlite-utils/blob/0771ac61fe5c2aca74075b20b1a99b9bd4c65661/sqlite_utils/db.py#L1706-L1708 I should write tests for them too. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1855838223 | |
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683143723 | https://api.github.com/repos/simonw/sqlite-utils/issues/584 | 1683143723 | IC_kwDOCGYnMM5kUrwr | 9599 | 2023-08-18T00:14:52Z | 2023-08-18T00:14:52Z | OWNER | Another docs update: this bit in here https://sqlite-utils.datasette.io/en/3.34/python-api.html#adding-multiple-foreign-key-constraints-at-once talks about how `.add_foreign_keys()` is a performance optimization to avoid having to run VACUUM a bunch of separate times: > The final step in adding a new foreign key to a SQLite database is to run `VACUUM`, to ensure the new foreign key is available in future introspection queries. > > `VACUUM` against a large (multi-GB) database can take several minutes or longer. If you are adding multiple foreign keys using `table.add_foreign_key(...)` these can quickly add up. > > Instead, you can use `db.add_foreign_keys(...)` to add multiple foreign keys within a single transaction. This method takes a list of four-tuples, each one specifying a `table`, `column`, `other_table` and `other_column`. That doesn't apply any more - the new mechanism using `.transform()` works completely differently, so this issue around running VACUUM no longer applies. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1855838223 | |
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683139304 | https://api.github.com/repos/simonw/sqlite-utils/issues/584 | 1683139304 | IC_kwDOCGYnMM5kUqro | 9599 | 2023-08-18T00:09:56Z | 2023-08-18T00:09:56Z | OWNER | Upgrading `flake8` locally replicated the error: ``` pip install -U flake8 flake8 ``` ``` ./tests/test_recipes.py:99:9: F811 redefinition of unused 'fn' from line 96 ./tests/test_recipes.py:127:9: F811 redefinition of unused 'fn' from line 124 ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1855838223 | |
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683138953 | https://api.github.com/repos/simonw/sqlite-utils/issues/584 | 1683138953 | IC_kwDOCGYnMM5kUqmJ | 9599 | 2023-08-18T00:09:20Z | 2023-08-18T00:09:20Z | OWNER | Weird, I'm getting a `flake8` problem in CI which doesn't occur on my laptop: ``` ./tests/test_recipes.py:99:9: F811 redefinition of unused 'fn' from line 96 ./tests/test_recipes.py:127:9: F811 redefinition of unused 'fn' from line 124 ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1855838223 | |
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683137259 | https://api.github.com/repos/simonw/sqlite-utils/issues/584 | 1683137259 | IC_kwDOCGYnMM5kUqLr | 9599 | 2023-08-18T00:06:59Z | 2023-08-18T00:06:59Z | OWNER | The docs still describe the old trick, I need to update that: https://sqlite-utils.datasette.io/en/3.34/python-api.html#adding-foreign-key-constraints | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1855838223 | |
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683122767 | https://api.github.com/repos/simonw/sqlite-utils/issues/584 | 1683122767 | IC_kwDOCGYnMM5kUmpP | 9599 | 2023-08-17T23:46:09Z | 2023-08-17T23:46:09Z | OWNER | Oops, `mypy` failures: ``` sqlite_utils/db.py:781: error: Incompatible types in assignment (expression has type "Tuple[Any, ...]", variable has type "Union[str, ForeignKey, Tuple[str, str], Tuple[str, str, str], Tuple[str, str, str, str]]") [assignment] sqlite_utils/db.py:1164: error: Need type annotation for "by_table" (hint: "by_table: Dict[<type>, <type>] = ...") [var-annotated] sqlite_utils/db.py:1169: error: Item "View" of "Union[Table, View]" has no attribute "transform" [union-attr] sqlite_utils/db.py:1813: error: Argument 1 to "append" of "list" has incompatible type "ForeignKey"; expected <nothing> [arg-type] sqlite_utils/db.py:1824: error: Argument 1 to "append" of "list" has incompatible type "ForeignKey"; expected <nothing> [arg-type] Found 5 errors in 1 file (checked 56 source files) ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1855838223 | |
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683118376 | https://api.github.com/repos/simonw/sqlite-utils/issues/584 | 1683118376 | IC_kwDOCGYnMM5kUlko | 9599 | 2023-08-17T23:41:10Z | 2023-08-17T23:41:10Z | OWNER | The problem here is that the table created by this line: ```python fresh_db.create_table("breeds", {"name": str}) ``` Has this schema: ```sql CREATE TABLE [breeds] ( [name] TEXT ); ``` SQLite creates an invisible `rowid` column for it automatically. On the `main` branch with the old implementation that table ends up looking like this after the foreign key has been added: ```sql (Pdb) print(fresh_db.schema) CREATE TABLE [dogs] ( [name] TEXT , [breed_id] INTEGER, FOREIGN KEY([breed_id]) REFERENCES [breeds]([rowid]) ); CREATE TABLE [breeds] ( [name] TEXT ); ``` But I think this validation check is failing now: https://github.com/simonw/sqlite-utils/blob/842b61321fc6a9f0bdb913ab138e39d71bf42e00/sqlite_utils/db.py#L875-L884 Here's what the debugger reveals about this code: ```python for fk in foreign_keys: if fk.other_table == name and columns.get(fk.other_column): continue if not any( c for c in self[fk.other_table].columns if c.name == fk.other_column ): raise AlterError( "No such column: {}.{}".format(fk.other_table, fk.other_column) ) ``` ``` (Pdb) fk ForeignKey(table='dogs', column='breed_id', other_table='breeds', other_column='rowid') (Pdb) self[fk.other_table].columns [Column(cid=0, name='name', type='TEXT', notnull=0, default_value=None, is_pk=0)] ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1855838223 | |
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683114719 | https://api.github.com/repos/simonw/sqlite-utils/issues/584 | 1683114719 | IC_kwDOCGYnMM5kUkrf | 9599 | 2023-08-17T23:36:02Z | 2023-08-17T23:36:02Z | OWNER | Just these three lines recreate the problem: ```python from sqlite_utils import Database fresh_db = Database(memory=True) fresh_db.create_table("dogs", {"name": str}) fresh_db.create_table("breeds", {"name": str}) fresh_db["dogs"].add_column("breed_id", fk="breeds") ``` Traceback: ``` Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py", line 2170, in add_column self.add_foreign_key(col_name, fk, fk_col) File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py", line 2273, in add_foreign_key self.db.add_foreign_keys([(self.name, column, other_table, other_column)]) File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py", line 1169, in add_foreign_keys self[table].transform(add_foreign_keys=fks) File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py", line 1728, in transform sqls = self.transform_sql( ^^^^^^^^^^^^^^^^^^^ File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py", line 1896, in transform_sql self.db.create_table_sql( File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py", line 882, in create_table_sql raise AlterError( sqlite_utils.db.AlterError: No such column: breeds.rowid ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1855838223 | |
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683112857 | https://api.github.com/repos/simonw/sqlite-utils/issues/584 | 1683112857 | IC_kwDOCGYnMM5kUkOZ | 9599 | 2023-08-17T23:33:58Z | 2023-08-17T23:33:58Z | OWNER | Full test: https://github.com/simonw/sqlite-utils/blob/842b61321fc6a9f0bdb913ab138e39d71bf42e00/tests/test_create.py#L468-L484 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1855838223 | |
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683112298 | https://api.github.com/repos/simonw/sqlite-utils/issues/584 | 1683112298 | IC_kwDOCGYnMM5kUkFq | 9599 | 2023-08-17T23:33:14Z | 2023-08-17T23:33:14Z | OWNER | Just one failing test left: ``` # Soundness check foreign_keys point to existing tables for fk in foreign_keys: if fk.other_table == name and columns.get(fk.other_column): continue if not any( c for c in self[fk.other_table].columns if c.name == fk.other_column ): > raise AlterError( "No such column: {}.{}".format(fk.other_table, fk.other_column) ) E sqlite_utils.db.AlterError: No such column: breeds.rowid sqlite_utils/db.py:882: AlterError ==== short test summary info ==== FAILED tests/test_create.py::test_add_column_foreign_key - sqlite_utils.db.AlterError: No such column: breeds.rowid ==== 1 failed, 378 deselected in 0.49s ==== ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1855838223 | |
https://github.com/simonw/sqlite-utils/issues/583#issuecomment-1683110636 | https://api.github.com/repos/simonw/sqlite-utils/issues/583 | 1683110636 | IC_kwDOCGYnMM5kUjrs | 9599 | 2023-08-17T23:31:27Z | 2023-08-17T23:31:27Z | OWNER | Spotted this while working on: - #577 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1855836914 | |
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683098094 | https://api.github.com/repos/simonw/sqlite-utils/issues/577 | 1683098094 | IC_kwDOCGYnMM5kUgnu | 9599 | 2023-08-17T23:15:36Z | 2023-08-17T23:15:36Z | OWNER | An interesting side-effect of this change is that it does result in a slightly different schema - e.g. this test: https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/tests/test_extract.py#L118-L133 Needs updating like so: ```diff diff --git a/tests/test_extract.py b/tests/test_extract.py index 70ad0cf..fd52534 100644 --- a/tests/test_extract.py +++ b/tests/test_extract.py @@ -127,8 +127,7 @@ def test_extract_rowid_table(fresh_db): assert fresh_db["tree"].schema == ( 'CREATE TABLE "tree" (\n' " [name] TEXT,\n" - " [common_name_latin_name_id] INTEGER,\n" - " FOREIGN KEY([common_name_latin_name_id]) REFERENCES [common_name_latin_name]([id])\n" + " [common_name_latin_name_id] INTEGER REFERENCES [common_name_latin_name]([id])\n" ")" ) assert ( ``` Unfortunately this means it may break other test suites that depend on `sqlite-utils` that have schema tests like this baked in. I don't think this should count as a breaking change release though, but it's still worth noting. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1817289521 | |
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683076325 | https://api.github.com/repos/simonw/sqlite-utils/issues/577 | 1683076325 | IC_kwDOCGYnMM5kUbTl | 9599 | 2023-08-17T22:48:36Z | 2023-08-17T22:48:36Z | OWNER | I'm inclined to just go with the `.transform()` method and not attempt to keep around the method that involves updating `sqlite_master` and then add code to detect if that's possible (or catch if it fails) and fall back on the other mechanism. It would be nice to drop some code complexity, plus I don't yet have a way of running automated tests against Python + SQLite versions that exhibit the problem. | { "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 1, "rocket": 0, "eyes": 0 } |
1817289521 | |
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683074009 | https://api.github.com/repos/simonw/sqlite-utils/issues/577 | 1683074009 | IC_kwDOCGYnMM5kUavZ | 9599 | 2023-08-17T22:45:29Z | 2023-08-17T22:47:08Z | OWNER | Actually I think `table.transform()` might get the following optional arguments: ```python def transform( self, *, # ... # This one exists already: drop_foreign_keys: Optional[Iterable] = None, # These two are new. This one specifies keys to add: add_foreign_keys: Optional[ForeignKeysType] = None, # Or this one causes them all to be replaced with the new definitions: foreign_keys: Optional[ForeignKeysType] = None, ``` There should be validation that forbids you from using `foreign_keys=` at the same time as either `drop_foreign_keys=` or `add_foreign_keys=` because the point of `foreign_keys=` is to define the keys for the new table all in one go. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1817289521 | |
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683074857 | https://api.github.com/repos/simonw/sqlite-utils/issues/577 | 1683074857 | IC_kwDOCGYnMM5kUa8p | 9599 | 2023-08-17T22:46:40Z | 2023-08-17T22:46:40Z | OWNER | As a reminder: https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/sqlite_utils/db.py#L159-L165 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1817289521 |