... Column, other table, other column to set as a
foreign key
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855894222,
https://github.com/simonw/datasette/issues/2145#issuecomment-1684530060,https://api.github.com/repos/simonw/datasette/issues/2145,1684530060,IC_kwDOBm6k_c5kZ-OM,9599,2023-08-18T23:09:03Z,2023-08-18T23:09:14Z,OWNER,"Ran a quick benchmark on ChatGPT Code Interpreter: https://chat.openai.com/share/8357dc01-a97e-48ae-b35a-f06249935124
Conclusion from there is that this query returns fast no matter how much the table grows:
```sql
SELECT EXISTS(SELECT 1 FROM ""nasty"" WHERE ""id"" IS NULL)
```
So detecting if a table contains any null primary keys is definitely feasible without a performance hit.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,
https://github.com/simonw/datasette/issues/2145#issuecomment-1684497000,https://api.github.com/repos/simonw/datasette/issues/2145,1684497000,IC_kwDOBm6k_c5kZ2Jo,9599,2023-08-18T22:31:53Z,2023-08-18T22:31:53Z,OWNER,"So it sounds like SQLite does ensure that a `rowid` before it allows a primary key to be null.
So one solution here would be to detect a null primary key and switch that table over to using `rowid` URLs instead. The key problem we're trying to solve here after all is how to link to a row:
https://latest.datasette.io/fixtures/infinity/1
But when would we run that check? And does every row in the table get a new `/rowid/` URL just because someone messed up and inserted a `null` by mistake?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,
https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683212074,https://api.github.com/repos/simonw/sqlite-utils/issues/585,1683212074,IC_kwDOCGYnMM5kU8cq,9599,2023-08-18T01:43:54Z,2023-08-18T01:43:54Z,OWNER,"Some manual testing:
```bash
sqlite-utils create-table /tmp/t.db places id integer name text country integer city integer continent integer --pk id
sqlite-utils schema /tmp/t.db
```
```sql
CREATE TABLE [places] (
[id] INTEGER PRIMARY KEY,
[name] TEXT,
[country] INTEGER,
[city] INTEGER,
[continent] INTEGER
);
```
```bash
sqlite-utils create-table /tmp/t.db country id integer name text
sqlite-utils create-table /tmp/t.db city id integer name text
sqlite-utils create-table /tmp/t.db continent id integer name text
sqlite-utils schema /tmp/t.db
```
```sql
CREATE TABLE [places] (
[id] INTEGER PRIMARY KEY,
[name] TEXT,
[country] INTEGER,
[city] INTEGER,
[continent] INTEGER
);
CREATE TABLE [country] (
[id] INTEGER,
[name] TEXT
);
CREATE TABLE [city] (
[id] INTEGER,
[name] TEXT
);
CREATE TABLE [continent] (
[id] INTEGER,
[name] TEXT
);
```
```bash
sqlite-utils transform /tmp/t.db places --add-foreign-key country country id --add-foreign-key continent continent id
sqlite-utils schema /tmp/t.db
```
```sql
CREATE TABLE [country] (
[id] INTEGER,
[name] TEXT
);
CREATE TABLE [city] (
[id] INTEGER,
[name] TEXT
);
CREATE TABLE [continent] (
[id] INTEGER,
[name] TEXT
);
CREATE TABLE ""places"" (
[id] INTEGER PRIMARY KEY,
[name] TEXT,
[country] INTEGER REFERENCES [country]([id]),
[city] INTEGER,
[continent] INTEGER REFERENCES [continent]([id])
);
```
```bash
sqlite-utils transform /tmp/t.db places --drop-foreign-key country
sqlite-utils schema /tmp/t.db places
```
```sql
CREATE TABLE ""places"" (
[id] INTEGER PRIMARY KEY,
[name] TEXT,
[country] INTEGER,
[city] INTEGER,
[continent] INTEGER REFERENCES [continent]([id])
)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855894222,
https://github.com/simonw/datasette/issues/2145#issuecomment-1684384750,https://api.github.com/repos/simonw/datasette/issues/2145,1684384750,IC_kwDOBm6k_c5kZavu,9599,2023-08-18T20:07:18Z,2023-08-18T20:07:18Z,OWNER,The big challenge here is what the URL to that row page should look like. How can I encode a `None` in a form that can be encoded and decoded without clashing with primary keys that are the string `None` or `null`?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683137259,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683137259,IC_kwDOCGYnMM5kUqLr,9599,2023-08-18T00:06:59Z,2023-08-18T00:06:59Z,OWNER,"The docs still describe the old trick, I need to update that: https://sqlite-utils.datasette.io/en/3.34/python-api.html#adding-foreign-key-constraints","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,
https://github.com/simonw/datasette/issues/2145#issuecomment-1684498947,https://api.github.com/repos/simonw/datasette/issues/2145,1684498947,IC_kwDOBm6k_c5kZ2oD,9599,2023-08-18T22:35:04Z,2023-08-18T22:35:04Z,OWNER,"The most interesting row URL in the fixtures database right now is this one:
https://latest.datasette.io/fixtures/compound_primary_key/a~2Fb,~2Ec-d
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,
https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683198740,https://api.github.com/repos/simonw/sqlite-utils/issues/585,1683198740,IC_kwDOCGYnMM5kU5MU,9599,2023-08-18T01:26:47Z,2023-08-18T01:26:47Z,OWNER,"The only CLI feature that supports providing just the column name appears to be this:
```bash
sqlite-utils add-foreign-key --help
```
```
Usage: sqlite-utils add-foreign-key [OPTIONS] PATH TABLE COLUMN [OTHER_TABLE]
[OTHER_COLUMN]
Add a new foreign key constraint to an existing table
Example:
sqlite-utils add-foreign-key my.db books author_id authors id
WARNING: Could corrupt your database! Back up your database file first.
```
I can drop that WARNING now since I'm not writing to `sqlite_master` any more.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855894222,
https://github.com/simonw/datasette/issues/2143#issuecomment-1683429959,https://api.github.com/repos/simonw/datasette/issues/2143,1683429959,IC_kwDOBm6k_c5kVxpH,9599,2023-08-18T06:43:33Z,2023-08-18T15:19:07Z,OWNER,"The single biggest design challenge I've had with metadata relates to how it should or should not be inherited.
If you apply a license to a Datasette instance, it feels like that should flow down to cover all of the databases and all of the tables within those databases.
If the license is at the database level, it should cover all tables.
But... should source do the same thing? I made it behave the same way as license, but it's presumably common for one database to have a single license but multiple different sources of data.
Then there's title - should that inherit? It feels like title should apply to only one level - you may want a title that applies to the instance, then a different custom title for databases and tables.
Here's the current state of play for metadata: https://docs.datasette.io/en/1.0a3/metadata.html
So there's `title` and `description` - and I'll be honest, I'm not 100% sure even I understand how those should be inherited down by tables/etc.
There's `description_html` which over-rides the `description` if it is set. It's a useful customization hack, but a bit surprising.
Then there are these six:
- `license`
- `license_url`
- `source`
- `source_url`
- `about`
- `about_url`
I added `about` later than the others, because I realized that plenty of my own projects needed a link to an article explaining them somewhere - e.g. https://scotrail.datasette.io/
Tables can also have column descriptions - just a string for each column. There's a demo of those here: https://latest.datasette.io/fixtures/roadside_attractions
And then there's all of the other stuff, most of which feels much more like ""settings"" than ""metadata"":
- `sort: created` - the custom sort order
- `size: 10` for a custom page size for a specific table
- `sortable_columns` to set which columns can be used to sort
- `hidden: true` to hide a table
- `label_column: title` is an interesting one - it lets you hint to Datasette which column should be displayed when there is a foreign key relationship. It's sort-of-metadata and sort-of-a-setting.
- `facets` sets default facets, see https://docs.datasette.io/en/1.0a3/facets.html#facets-in-metadata
- `facet_size` sets the number of facets to display
- `fts_table` and `fts_pk` can be used to configure FTS, especially for views: https://docs.datasette.io/en/1.0a3/full_text_search.html
And the authentication stuff! `allow` and `allow_sql` blocks: https://docs.datasette.io/en/1.0a3/authentication.html#defining-permissions-with-allow-blocks
And the new `permissions` key in the 1.0 alphas: https://docs.datasette.io/en/1.0a3/authentication.html#other-permissions-in-metadata
I think that might be everything (excluding the `plugins` settings stuff, which is also a bad fit for metadata.)
And to make things even more confusing... I believe you can add arbitrary key/value pairs to your metadata and then use them in your templates! I think I've heard from at least one person who uses that ability.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,
https://github.com/simonw/datasette/issues/2145#issuecomment-1684525943,https://api.github.com/repos/simonw/datasette/issues/2145,1684525943,IC_kwDOBm6k_c5kZ9N3,9599,2023-08-18T23:04:14Z,2023-08-18T23:04:14Z,OWNER,"This is hard. I tried this:
```python
def path_from_row_pks(row, pks, use_rowid, quote=True):
""""""Generate an optionally tilde-encoded unique identifier
for a row from its primary keys.""""""
if use_rowid or any(row[pk] is None for pk in pks):
bits = [row[""rowid""]]
else:
bits = [
row[pk][""value""] if isinstance(row[pk], dict) else row[pk] for pk in pks
]
if quote:
bits = [tilde_encode(str(bit)) for bit in bits]
else:
bits = [str(bit) for bit in bits]
return "","".join(bits)
```
The ` if use_rowid or any(row[pk] is None for pk in pks)` bit is new.
But I got this error on http://127.0.0.1:8003/nulls/nasty :
```
File ""/Users/simon/Dropbox/Development/datasette/datasette/views/table.py"", line 1364, in run_display_columns_and_rows
display_columns, display_rows = await display_columns_and_rows(
File ""/Users/simon/Dropbox/Development/datasette/datasette/views/table.py"", line 186, in display_columns_and_rows
pk_path = path_from_row_pks(row, pks, not pks, False)
File ""/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py"", line 124, in path_from_row_pks
bits = [row[""rowid""]]
IndexError: No item with that key
```
Because the SQL query I ran to populate the page didn't know that it would need to select `rowid` as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683145819,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683145819,IC_kwDOCGYnMM5kUsRb,9599,2023-08-18T00:17:26Z,2023-08-18T00:17:26Z,OWNER,Updated documentation: https://sqlite-utils--584.org.readthedocs.build/en/584/python-api.html#adding-foreign-key-constraints,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683139304,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683139304,IC_kwDOCGYnMM5kUqro,9599,2023-08-18T00:09:56Z,2023-08-18T00:09:56Z,OWNER,"Upgrading `flake8` locally replicated the error:
```
pip install -U flake8
flake8
```
```
./tests/test_recipes.py:99:9: F811 redefinition of unused 'fn' from line 96
./tests/test_recipes.py:127:9: F811 redefinition of unused 'fn' from line 124
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683138953,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683138953,IC_kwDOCGYnMM5kUqmJ,9599,2023-08-18T00:09:20Z,2023-08-18T00:09:20Z,OWNER,"Weird, I'm getting a `flake8` problem in CI which doesn't occur on my laptop:
```
./tests/test_recipes.py:99:9: F811 redefinition of unused 'fn' from line 96
./tests/test_recipes.py:127:9: F811 redefinition of unused 'fn' from line 124
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,
https://github.com/simonw/datasette/issues/2143#issuecomment-1684484426,https://api.github.com/repos/simonw/datasette/issues/2143,1684484426,IC_kwDOBm6k_c5kZzFK,9599,2023-08-18T22:12:52Z,2023-08-18T22:12:52Z,OWNER,"Yeah, I'm convinced by that. There's not point in having both `settings.json` and `datasette.json`.
I like `datasette.json` ( / `datasette.yml`) as a name. That can be the file that lives in your config directory too, so if you run `datasette .` in a folder containing `datasette.yml` all of those settings get picked up.
Here's a thought for how it could look - I'll go with the YAML format because I expect that to be the default most people use, just because it supports multi-line strings better.
I based this on the big example at https://docs.datasette.io/en/1.0a3/metadata.html#using-yaml-for-metadata - and combined some bits from https://docs.datasette.io/en/1.0a3/authentication.html as well.
```yaml
title: Demonstrating Metadata from YAML
description_html: |-
This description includes a long HTML string
- YAML is better for embedding HTML strings than JSON!
settings:
default_page_size: 10
max_returned_rows: 3000
sql_time_limit_ms"": 8000
databases:
docs:
permissions:
create-table:
id: editor
fixtures:
tables:
no_primary_key:
hidden: true
queries:
neighborhood_search:
sql: |-
select neighborhood, facet_cities.name, state
from facetable join facet_cities on facetable.city_id = facet_cities.id
where neighborhood like '%' || :text || '%' order by neighborhood;
title: Search neighborhoods
description_html: |-
This demonstrates basic LIKE search
permissions:
debug-menu:
id: '*'
plugins:
datasette-ripgrep:
path: /usr/local/lib/python3.11/site-packages
```
I'm inclined to say we try to be a super-set of the existing `metadata.yml` format, at least where it makes sense to do so. That way the upgrade path is smooth for people. Also, I don't think the format itself is terrible - it's the name that's the big problem.
In this example I've mixed in one extra concept: that `settings:` block with a bunch of settings in it.
There are some things in there that look a little bit like metadata - the `title` and `description_html` fields.
But _are they_ metadata? The title and description of the overall instance feels like it could be described as general configuration. The stuff for the `query` should live where the query itself is defined.
Note that queries can be defined by a plugin hook too: https://docs.datasette.io/en/1.0a3/plugin_hooks.html#canned-queries-datasette-database-actor
What do you think? Is this the right direction, or are you thinking there's a more radical redesign that would make sense here?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,
https://github.com/simonw/datasette/issues/2145#issuecomment-1684504398,https://api.github.com/repos/simonw/datasette/issues/2145,1684504398,IC_kwDOBm6k_c5kZ39O,9599,2023-08-18T22:43:31Z,2023-08-18T22:43:46Z,OWNER,"`(?P[^/]+?)` could instead be a regex that is restricted to the tilde-encoded set of characters, or `\.\d+`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,
https://github.com/simonw/datasette/issues/2145#issuecomment-1684503189,https://api.github.com/repos/simonw/datasette/issues/2145,1684503189,IC_kwDOBm6k_c5kZ3qV,9599,2023-08-18T22:41:51Z,2023-08-18T22:41:51Z,OWNER,"```pycon
>>> tilde_encode(""~"")
'~7E'
>>> tilde_encode(""."")
'~2E'
>>> tilde_encode(""-"")
'-'
```
I think `.` might be the way to do this:
/database/table/.4
But... I worry about that colliding with my URL routing code that spots the difference between these:
/database/table/.4
/database/table/.4.json
/database/table/.4.csv
etc.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,
https://github.com/simonw/datasette/issues/2145#issuecomment-1684495674,https://api.github.com/repos/simonw/datasette/issues/2145,1684495674,IC_kwDOBm6k_c5kZ106,9599,2023-08-18T22:29:47Z,2023-08-18T22:29:47Z,OWNER,"https://www.sqlite.org/lang_createtable.html#the_primary_key says:
>According to the SQL standard, PRIMARY KEY should always imply NOT NULL. Unfortunately, due to a bug in some early versions, this is not the case in SQLite. Unless the column is an [INTEGER PRIMARY KEY](https://www.sqlite.org/lang_createtable.html#rowid) or the table is a [WITHOUT ROWID](https://www.sqlite.org/withoutrowid.html) table or a [STRICT](https://www.sqlite.org/stricttables.html) table or the column is declared NOT NULL, SQLite allows NULL values in a PRIMARY KEY column. SQLite could be fixed to conform to the standard, but doing so might break legacy applications. Hence, it has been decided to merely document the fact that SQLite allows NULLs in most PRIMARY KEY columns.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,