Also need to update the datasette.urls methods that construct the URL to a database/table/row - they take the database name but they need to know to look for the route.

Need to add tests that check the links in the HTML and can confirm this is working correctly.

Need to update documentation in a few places - e.g. https://docs.datasette.io/en/stable/internals.html#remove-database-name

This removes a database that has been previously added. name= is the unique name of that database, used in its URL path.

I have this issue, too, with a fresh export. None of my Workout entries in export.xml have an id key, though the sample export.xml in the tests folder doesn’t either, so I don’t think this is the culprit. Indeed, it seems @simonw is using the hash_id function from sqlite_utils, which creates a column (id, in this case) based on a hash of the row’s contents.

When I run the script, a workouts table is created, with one entry: my first workout. No workout_points table is created, as I’d expect from utils.py. I then get essentially the same error as noted in this thread:

Importing from HealthKit [###################################-] 98% 00:00:01 Traceback (most recent call last): File "/Users/lchski/.pyenv/versions/3.10.3/bin/healthkit-to-sqlite", line 8, in <module> sys.exit(cli()) File "/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/click/core.py", line 1128, in __call__ return self.main(*args, **kwargs) File "/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/click/core.py", line 1053, in main rv = self.invoke(ctx) File "/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/click/core.py", line 1395, in invoke return ctx.invoke(self.callback, **ctx.params) File "/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/click/core.py", line 754, in invoke return __callback(*args, **kwargs) File "/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/healthkit_to_sqlite/cli.py", line 57, in cli convert_xml_to_sqlite(fp, db, progress_callback=bar.update, zipfile=zf) File "/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/healthkit_to_sqlite/utils.py", line 34, in convert_xml_to_sqlite workout_to_db(el, db, zipfile) File "/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/healthkit_to_sqlite/utils.py", line 57, in workout_to_db pk = db["workouts"].insert(record, alter=True, hash_id="id").last_pk File "/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/sqlite_utils/db.py", line 2822, in insert return self.insert_all( File "/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/sqlite_utils/db.py", line 2950, in insert_all self.insert_chunk( File "/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/sqlite_utils/db.py", line 2715, in insert_chunk result = self.db.execute(query, params) File "/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/sqlite_utils/db.py", line 458, in execute return self.conn.execute(sql, parameters) sqlite3.IntegrityError: UNIQUE constraint failed: workouts.id

Are there maybe duplicate workouts in the data, which’d cause multiple rows to share the same id? It’s strange, though, that no workout_points is created at all. Export created from iOS 15.3.1.

I think I've got this working but I need to write a test for it that covers the rare case when the route is not the same thing as the database name.

I'll do that with a new test.

I'm trying to think if there's any reason not to use route for this. Would I possibly want to use that noun for something else in the future? I like it more than route_path because it has no underscore.

Decision made: I'm going with route.

I can now read format from request.url_vars and delete this code entirely: https://github.com/simonw/datasette/blob/b9c2b1cfc8692b9700416db98721fa3ec982f6be/datasette/views/base.py#L375-L381

I'm going to keep path as the path to the file on disk. I'll pick a new name for what is currently path in that undocumented JSON API.

Pretty sure changing it will break some existing plugins though, including likely Datasette Desktop.

The docs do currently describe path as the filesystem path here: https://docs.datasette.io/en/stable/internals.html#database-class

Good thing I'm not at 1.0 yet so I can change that!

I'm inclined to redefine ds.path to ds.file_path to fix this. Or ds.filepath.

The output of /.json DOES use path to mean the URL path, not the path to the file on disk:

{ "fixtures.dot": { "name": "fixtures.dot", "hash": null, "color": "631f11", "path": "/fixtures~2Edot", So that's a problem already: having db.path refer to something different from that JSON is inconsistent.

Right now if a database has a . in its name e.g. fixtures.dot the URL to that database is:

/fixtures~2Edot

But the output on /-/databases doesn't reflect that, it still shows the name with the dot.

This blocks: - #1668

Marking this as blocked until #1660 is done.

Implementing this is a little tricky because there's a whole lot of code that expects the database captured by the URL routing to be the name used to look up the database in datasette.databases - or via .get_database().

The DataView.get() method is a good example of the trickyness here. It even has code that dispatches out to plugin hooks that take database as a parameter.

https://github.com/simonw/datasette/blob/61419388c134001118aaf7dfb913562d467d7913/datasette/views/base.py#L383-L555

All the more reason to get rid of that BaseView -> DataView -> TableView hierarchy entirely: - #1660

Options: - route_path - url_path - route

I like route_path, or maybe route.

Worth noting that the name right now is picked automatically to avoid conflicts:

https://github.com/simonw/datasette/blob/61419388c134001118aaf7dfb913562d467d7913/datasette/app.py#L397-L413

The Database class already has a path property but it means something else - it's the path to the .db file on disk:

https://github.com/simonw/datasette/blob/61419388c134001118aaf7dfb913562d467d7913/datasette/database.py#L29-L50

So need a different name for the path-that-is-used-in-the-URL.

OK, I've made this more consistent - I still need to address the fact that format can be .json or json or not used at all before I close this issue.

https://github.com/simonw/datasette/blob/61419388c134001118aaf7dfb913562d467d7913/tests/test_routes.py#L15-L35

I called it as_format to avoid clashing with the Python built-in format() function when these things were turned into keyword arguments, but now that they're not I can use format instead.

I think I'm going to go with database, table, format and pks.

This does make it more interesting - it also highlights how inconsistent the way the capturing works is. Especially as_format which can be None or "" or .json or json or not used at all in the case of TableView.

https://github.com/simonw/datasette/blob/764738dfcb16cd98b0987d443f59d5baa9d3c332/tests/test_routes.py#L12-L36

This is more interesting if it also asserts against the captured matches from the pattern.

On revisiting https://gist.github.com/simonw/281eac9c73b062c3469607ad86470eb2 a few months later I'm having second thoughts about using @inject on the main() method.

But I still like the pattern as a way to resolve more complex cases like "to generate GeoJSON of the expanded view with labels, the label expansion code needs to run once at some before the GeoJSON formatting code does".

So I'm going to stick with it a tiny bit longer, but maybe try to make it a lot more explicit when it's going to happen rather than having the main view methods themselves also use async DI.

... unless your data had a column called n?

Exactly, that's highly likely even though I can't double check from this computer just now. Thanks!

I ended up moving hashed URL mode out to a plugin in: - #647

If you're still interested in using it with _memory please open an issue in that repo here: https://github.com/simonw/datasette-hashed-urls

Tests so far: https://github.com/simonw/datasette/blob/711767bcd3c1e76a0861fe7f24069ff1c8efc97a/tests/test_routes.py#L12-L34

Demo: https://latest.datasette.io/fixtures/facetable - which now has a column called n.

Yes! That's the problem. I was able to replicate it like so: echo '[{ "n": "one", "abc": 1 }, { "n": "one", "abc": 2 }, { "n": "two", "abc": 3 }]' | sqlite-utils insert column-called-n.db t -

... unless your data had a column called n?

It looks to me like something is causing the faceting query here to return a string when it was expected to return a number:

https://github.com/simonw/datasette/blob/32963018e7edfab1233de7c7076c428d0e5c7813/datasette/facets.py#L153-L170

I can't think of any way that a count(*) as n would turn into a string though!

Had a thought about the implementation of this: it could make a really neat plugin.

Something like datasette-export which adds a export command using https://docs.datasette.io/en/stable/plugin_hooks.html#register-commands-cli - then you could run:

datasette export my-export-dir mydatabase.db -m metadata.json --template-dir templates/

And the command would then:

• Create a Datasette() instance with those databases/metadata/etc
• Executeawait datasette.client.get("/") to get the homepage HTML
• Parse the HTML using BeautifulSoup to find all a[href], link[href], script[src], img[src] elements that reference a relative path as opposed to one that starts with http://
• Write out the homepage to my-export-dir/index.html
• Recursively fetch and dump all of the other pages and assets that it found too

All of that HTML parsing may be over-complicating things. It could alternatively accept options for which pages you want to export:

datasette export my-export-dir \ mydatabase.db -m metadata.json --template-dir templates/ \ --path / \ --path /mydatabase ...

Or a really wild option: it could allow you to define the paths you want to export using a SQL query:

datasette export my-export-dir \ mydatabase.db -m metadata.json --template-dir templates/ \ --sql " select '/' as path, 'index.html' as filename union all select '/mydatabase/articles/' || id as path, 'article-' || id || '.html' as filename from articles union all select '/mydatabase/tags/' || tag as path, 'tag-' || tag || '.html' as filename from tags " Which would save these files: - index.html as the content of / - article-1.html (and more) as the content of /mydatabase/articles/1 - tag-python.html (and more) as the content of /mydatabase/tags/python

Those client-side SQLite tricks are really neat.

datasette publish defaults to configuring it so the raw SQLite database can be downloaded from /fixtures.db - and this issue updated it to be served with a CORS header that would allow client-side scripts to load the file:

• 1057

If you're not going to run any server-side code at all you don't need Datasette for this - you can upload the SQLite database file to any static hosting with CORS headers and load it into the client that way.

In terms of static publishing, I do think there's something interesting about using Datasette to generate static sites. There's an issue discussing options for that over here:

• 1605

Updated docs: https://docs.datasette.io/en/latest/performance.html#datasette-hashed-urls

I can remove the default_cache_ttl_hashed setting too.

I'm going to land this and handle those in separate commits.

Still need to remove it from the documentation and do something about that hash_urls setting.