html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app https://github.com/simonw/datasette/issues/2145#issuecomment-1686683596,https://api.github.com/repos/simonw/datasette/issues/2145,1686683596,IC_kwDOBm6k_c5kiL_M,9599,simonw,2023-08-21T16:49:12Z,2023-08-21T16:49:12Z,OWNER,"Suggestion from @asg017 is that we say that if your row has a null primary key you don't get a link to a row page for that row. Which has some precedent, because our SQL view display doesn't link to row pages at all (since they don't make sense for views): https://latest.datasette.io/fixtures/simple_view","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684530060,https://api.github.com/repos/simonw/datasette/issues/2145,1684530060,IC_kwDOBm6k_c5kZ-OM,9599,simonw,2023-08-18T23:09:03Z,2023-08-18T23:09:14Z,OWNER,"Ran a quick benchmark on ChatGPT Code Interpreter: https://chat.openai.com/share/8357dc01-a97e-48ae-b35a-f06249935124 Conclusion from there is that this query returns fast no matter how much the table grows: ```sql SELECT EXISTS(SELECT 1 FROM ""nasty"" WHERE ""id"" IS NULL) ``` So detecting if a table contains any null primary keys is definitely feasible without a performance hit.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684526447,https://api.github.com/repos/simonw/datasette/issues/2145,1684526447,IC_kwDOBm6k_c5kZ9Vv,9599,simonw,2023-08-18T23:05:02Z,2023-08-18T23:05:02Z,OWNER,How expensive is it to detect if a SQLite table contains at least one `null` primary key?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684525943,https://api.github.com/repos/simonw/datasette/issues/2145,1684525943,IC_kwDOBm6k_c5kZ9N3,9599,simonw,2023-08-18T23:04:14Z,2023-08-18T23:04:14Z,OWNER,"This is hard. I tried this: ```python def path_from_row_pks(row, pks, use_rowid, quote=True): """"""Generate an optionally tilde-encoded unique identifier for a row from its primary keys."""""" if use_rowid or any(row[pk] is None for pk in pks): bits = [row[""rowid""]] else: bits = [ row[pk][""value""] if isinstance(row[pk], dict) else row[pk] for pk in pks ] if quote: bits = [tilde_encode(str(bit)) for bit in bits] else: bits = [str(bit) for bit in bits] return "","".join(bits) ``` The ` if use_rowid or any(row[pk] is None for pk in pks)` bit is new. But I got this error on http://127.0.0.1:8003/nulls/nasty : ``` File ""/Users/simon/Dropbox/Development/datasette/datasette/views/table.py"", line 1364, in run_display_columns_and_rows display_columns, display_rows = await display_columns_and_rows( File ""/Users/simon/Dropbox/Development/datasette/datasette/views/table.py"", line 186, in display_columns_and_rows pk_path = path_from_row_pks(row, pks, not pks, False) File ""/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py"", line 124, in path_from_row_pks bits = [row[""rowid""]] IndexError: No item with that key ``` Because the SQL query I ran to populate the page didn't know that it would need to select `rowid` as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684525054,https://api.github.com/repos/simonw/datasette/issues/2145,1684525054,IC_kwDOBm6k_c5kZ8_-,9599,simonw,2023-08-18T23:02:26Z,2023-08-18T23:02:26Z,OWNER,"Creating a quick test database: ```bash sqlite-utils create-table nulls.db nasty id text --pk id sqlite-utils nulls.db 'insert into nasty (id) values (null)' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684523322,https://api.github.com/repos/simonw/datasette/issues/2145,1684523322,IC_kwDOBm6k_c5kZ8k6,9599,simonw,2023-08-18T22:59:14Z,2023-08-18T22:59:14Z,OWNER,"Except it looks like the Links from other tables section is broken: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684522567,https://api.github.com/repos/simonw/datasette/issues/2145,1684522567,IC_kwDOBm6k_c5kZ8ZH,9599,simonw,2023-08-18T22:58:07Z,2023-08-18T22:58:07Z,OWNER,"Here's a prototype of that: ```diff diff --git a/datasette/app.py b/datasette/app.py index b2644ace..acc55249 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -1386,7 +1386,7 @@ class Datasette: ) add_route( RowView.as_view(self), - r""/(?P[^\/\.]+)/(?P[^/]+?)/(?P[^/]+?)(\.(?P\w+))?$"", + r""/(?P[^\/\.]+)/(?P
[^/]+?)/(?P[A-Za-z0-9\_\-\~]+|\.\d+)(\.(?P\w+))?$"", ) add_route( TableInsertView.as_view(self), @@ -1440,7 +1440,15 @@ class Datasette: async def resolve_row(self, request): db, table_name, _ = await self.resolve_table(request) pk_values = urlsafe_components(request.url_vars[""pks""]) - sql, params, pks = await row_sql_params_pks(db, table_name, pk_values) + + if len(pk_values) == 1 and pk_values[0].startswith("".""): + # It's a special .rowid value + pk_values = (pk_values[0][1:],) + sql, params, pks = await row_sql_params_pks( + db, table_name, pk_values, rowid=True + ) + else: + sql, params, pks = await row_sql_params_pks(db, table_name, pk_values) results = await db.execute(sql, params, truncate=True) row = results.first() if row is None: diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py index c388673d..96669281 100644 --- a/datasette/utils/__init__.py +++ b/datasette/utils/__init__.py @@ -1206,9 +1206,12 @@ def truncate_url(url, length): return url[: length - 1] + ""…"" -async def row_sql_params_pks(db, table, pk_values): +async def row_sql_params_pks(db, table, pk_values, rowid=False): pks = await db.primary_keys(table) - use_rowid = not pks + if rowid: + use_rowid = True + else: + use_rowid = not pks select = ""*"" if use_rowid: select = ""rowid, *"" ``` It works: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684505071,https://api.github.com/repos/simonw/datasette/issues/2145,1684505071,IC_kwDOBm6k_c5kZ4Hv,9599,simonw,2023-08-18T22:44:35Z,2023-08-18T22:44:35Z,OWNER,"Also relevant: https://github.com/simonw/datasette/blob/943df09dcca93c3b9861b8c96277a01320db8662/datasette/utils/__init__.py#L1147-L1153","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684504398,https://api.github.com/repos/simonw/datasette/issues/2145,1684504398,IC_kwDOBm6k_c5kZ39O,9599,simonw,2023-08-18T22:43:31Z,2023-08-18T22:43:46Z,OWNER,"`(?P[^/]+?)` could instead be a regex that is restricted to the tilde-encoded set of characters, or `\.\d+`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684504051,https://api.github.com/repos/simonw/datasette/issues/2145,1684504051,IC_kwDOBm6k_c5kZ33z,9599,simonw,2023-08-18T22:43:06Z,2023-08-18T22:43:06Z,OWNER,Here's the regex in question at the moment: https://github.com/simonw/datasette/blob/943df09dcca93c3b9861b8c96277a01320db8662/datasette/app.py#L1387-L1390,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684503587,https://api.github.com/repos/simonw/datasette/issues/2145,1684503587,IC_kwDOBm6k_c5kZ3wj,9599,simonw,2023-08-18T22:42:28Z,2023-08-18T22:42:39Z,OWNER,"I could set a rule that extensions (including custom render extensions set by plugins) must not be valid integers, and teach Datasette that `/\.\d+` is the indication of a `rowid`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684503189,https://api.github.com/repos/simonw/datasette/issues/2145,1684503189,IC_kwDOBm6k_c5kZ3qV,9599,simonw,2023-08-18T22:41:51Z,2023-08-18T22:41:51Z,OWNER,"```pycon >>> tilde_encode(""~"") '~7E' >>> tilde_encode(""."") '~2E' >>> tilde_encode(""-"") '-' ``` I think `.` might be the way to do this: /database/table/.4 But... I worry about that colliding with my URL routing code that spots the difference between these: /database/table/.4 /database/table/.4.json /database/table/.4.csv etc.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684502278,https://api.github.com/repos/simonw/datasette/issues/2145,1684502278,IC_kwDOBm6k_c5kZ3cG,9599,simonw,2023-08-18T22:40:20Z,2023-08-18T22:40:20Z,OWNER,"From reviewing https://simonwillison.net/2022/Mar/19/weeknotes/ unreserved = ALPHA / DIGIT / ""-"" / ""."" / ""_"" / ""~"" That's how I chose the tilde character - but it also suggests that I could use `-` or `.` or `_` for my new `rowid` encoding. So maybe `/database/table/_4` could indicate ""the row with `rowid` of 4"". No, that doesn't work: ```pycon >>> from datasette.utils import tilde_encode >>> tilde_encode(""_"") '_' ``` I need a character which tilde-encoding does indeed encode.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684500540,https://api.github.com/repos/simonw/datasette/issues/2145,1684500540,IC_kwDOBm6k_c5kZ3A8,9599,simonw,2023-08-18T22:37:37Z,2023-08-18T22:37:37Z,OWNER,"I just found this and panicked, thinking maybe tilde encoding is a bad idea after all! https://jkorpela.fi/tilde.html But... ""Date of last update: 1999-08-27"" - I think I'm OK.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684500172,https://api.github.com/repos/simonw/datasette/issues/2145,1684500172,IC_kwDOBm6k_c5kZ27M,9599,simonw,2023-08-18T22:37:04Z,2023-08-18T22:37:04Z,OWNER,"Looking at the way these URLs work: because the components themselves in `a~2Fb,~2Ec-d` are tilde-encoded, any character that's ""safe"" in tilde-encoding could be used to indicate ""this is actually a rowid"". ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684498947,https://api.github.com/repos/simonw/datasette/issues/2145,1684498947,IC_kwDOBm6k_c5kZ2oD,9599,simonw,2023-08-18T22:35:04Z,2023-08-18T22:35:04Z,OWNER,"The most interesting row URL in the fixtures database right now is this one: https://latest.datasette.io/fixtures/compound_primary_key/a~2Fb,~2Ec-d ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684497642,https://api.github.com/repos/simonw/datasette/issues/2145,1684497642,IC_kwDOBm6k_c5kZ2Tq,9599,simonw,2023-08-18T22:32:53Z,2023-08-18T22:32:53Z,OWNER,"Here's a potential solution: make it so ALL `rowid` tables in SQLite can be optionally addressed by their `rowid` instead of by their primary key. Then teach the code that outputs the URL to a row page to spot if there are `null` primary keys and switch to that alternative addressing mechanism instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684497000,https://api.github.com/repos/simonw/datasette/issues/2145,1684497000,IC_kwDOBm6k_c5kZ2Jo,9599,simonw,2023-08-18T22:31:53Z,2023-08-18T22:31:53Z,OWNER,"So it sounds like SQLite does ensure that a `rowid` before it allows a primary key to be null. So one solution here would be to detect a null primary key and switch that table over to using `rowid` URLs instead. The key problem we're trying to solve here after all is how to link to a row: https://latest.datasette.io/fixtures/infinity/1 But when would we run that check? And does every row in the table get a new `/rowid/` URL just because someone messed up and inserted a `null` by mistake?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684495674,https://api.github.com/repos/simonw/datasette/issues/2145,1684495674,IC_kwDOBm6k_c5kZ106,9599,simonw,2023-08-18T22:29:47Z,2023-08-18T22:29:47Z,OWNER,"https://www.sqlite.org/lang_createtable.html#the_primary_key says: >According to the SQL standard, PRIMARY KEY should always imply NOT NULL. Unfortunately, due to a bug in some early versions, this is not the case in SQLite. Unless the column is an [INTEGER PRIMARY KEY](https://www.sqlite.org/lang_createtable.html#rowid) or the table is a [WITHOUT ROWID](https://www.sqlite.org/withoutrowid.html) table or a [STRICT](https://www.sqlite.org/stricttables.html) table or the column is declared NOT NULL, SQLite allows NULL values in a PRIMARY KEY column. SQLite could be fixed to conform to the standard, but doing so might break legacy applications. Hence, it has been decided to merely document the fact that SQLite allows NULLs in most PRIMARY KEY columns.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684494464,https://api.github.com/repos/simonw/datasette/issues/2145,1684494464,IC_kwDOBm6k_c5kZ1iA,9599,simonw,2023-08-18T22:27:51Z,2023-08-18T22:28:40Z,OWNER,"Oh wow, null primary keys are bad news... SQLite lets you insert multiple rows with the same `null` value! ```pycon >>> import sqlite_utils >>> db = sqlite_utils.Database(memory=True) >>> db[""foo""].insert({""id"": None, ""name"": ""No ID""}, pk=""id"")
>>> db.schema 'CREATE TABLE [foo] (\n [id] TEXT PRIMARY KEY,\n [name] TEXT\n);' >>> db[""foo""].insert({""id"": None, ""name"": ""No ID""}, pk=""id"")
>>> db.schema 'CREATE TABLE [foo] (\n [id] TEXT PRIMARY KEY,\n [name] TEXT\n);' >>> list(db[""foo""].rows) [{'id': None, 'name': 'No ID'}, {'id': None, 'name': 'No ID'}] >>> list(db.query('select * from foo where id = null')) [] >>> list(db.query('select * from foo where id is null')) [{'id': None, 'name': 'No ID'}, {'id': None, 'name': 'No ID'}] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684384750,https://api.github.com/repos/simonw/datasette/issues/2145,1684384750,IC_kwDOBm6k_c5kZavu,9599,simonw,2023-08-18T20:07:18Z,2023-08-18T20:07:18Z,OWNER,The big challenge here is what the URL to that row page should look like. How can I encode a `None` in a form that can be encoded and decoded without clashing with primary keys that are the string `None` or `null`?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break,