If you attempt to use the old setting:

datasette mydatabase.db --setting hash_urls 1

It should error with a message saying that the feature has been moved to a plugin.

I'll do this with a deprecated_settings mechanism so the error can be detected even though datasette --help-settings will no longer return the setting.

https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/cli.py#L479-L489

This is going to simplify the code in the various view classes substantially: - #1660

The plugin works. I'm going to implement one last feature for it:

• https://github.com/simonw/datasette-hashed-urls/issues/3

Then I can remove hashed URL mode in a separate issue.

I shipped the first version of this: https://github.com/simonw/datasette-hashed-urls

Next step: test it with a live demo: - https://github.com/simonw/datasette-hashed-urls/issues/2

I'm happy with this now that I've landed Tilde encoding in #1657.

Dupe of #1510.

... maybe Datasette itself should include interactive API documentation, in addition to documenting it in the manual?

/dbname/table/-/apidocs could return documentation about the specific table, taking into account columns and types.

Idea: I could add Pydantic https://pydantic-docs.helpmanual.io/usage/schema/ as an optional test dependency and use it to generate JSON schemas and run validation against examples in the API documentation.

Maybe generate API documentation from it too?

If I set a rule that everything available in the template context MUST also be available via the JSON API (maybe through an extras mechanism) I can combine this with API documentation and solve both at once.

This would be a lot easier if I had extracted out the hash logic to a plugin, see: - #647

Also related: - #878 - #1512 - #1518 - #870

DataView is used as the base class for:

• DatabaseView
• DatabaseDownload (just so the permissions checks can be called)
• QueryView - which isn't routed to directly, it's called from DatabaseView if ?sql= is available and TableView for canned queries
• RowTableShared which is the base class for TableView and RowView

This is one of the worst bits - the get_format() method on the DataView base class actually modifies args, including removing keys! Really confusing: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L454-L482

Then BaseView has some surprising responsibilities. It has a utility helper for checking multiple permissions at once:

https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L81-L105

And its own render method that adds extra stuff to the template context and handles the rel: alternate header:

https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L131-L157

Then DataView does all sorts of weird stuff - from handling database hashes (which I want to remove, see #647):

https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L206-L219

To streaming CSV responses: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L286-L308

To handling SQLite exceptions: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L514-L526

And a ton more. It' s a big mess.

I can get regular .json to stream too, using the pattern described in this TIL: https://til.simonwillison.net/python/output-json-array-streaming

Done:

• https://latest.datasette.io/fixtures/table~2Fwith~2Fslashes~2Ecsv
• https://latest.datasette.io/fixtures/table~2Fwith~2Fslashes~2Ecsv.csv
• https://latest.datasette.io/fixtures/table~2Fwith~2Fslashes~2Ecsv.json

Demo:

• https://latest.datasette.io/fixtures/table~2Fwith~2Fslashes~2Ecsv
• https://latest.datasette.io/fixtures/table~2Fwith~2Fslashes~2Ecsv.csv
• https://latest.datasette.io/fixtures/table~2Fwith~2Fslashes~2Ecsv.json

Now live here: https://fivethirtyeight.datasettes.com/fivethirtyeight/august-senate-polls~2Faugust_senate_polls

Documentation: https://docs.datasette.io/en/latest/internals.html#tilde-encoding

Codecov Report

Merging #1659 (85dde28) into main (c10cd48) will increase coverage by 0.03%. The diff coverage is 100.00%.

:exclamation: Current head 85dde28 differs from pull request most recent head 99b8263. Consider uploading reports for the commit 99b8263 to get more accurate results

```diff @@ Coverage Diff @@

main #1659 +/-

========================================== + Coverage 92.06% 92.10% +0.03%
========================================== Files 34 34
Lines 4576 4584 +8
========================================== + Hits 4213 4222 +9
+ Misses 363 362 -1
```

| Impacted Files | Coverage Δ | | |---|---|---| | datasette/app.py | 94.36% <100.00%> (ø) | | | datasette/url_builder.py | 100.00% <100.00%> (ø) | | | datasette/utils/__init__.py | 94.84% <100.00%> (-0.13%) | :arrow_down: | | datasette/views/base.py | 96.07% <100.00%> (+0.58%) | :arrow_up: | | datasette/views/table.py | 96.21% <100.00%> (+0.01%) | :arrow_up: |

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update c10cd48...99b8263. Read the comment docs.

Moving this to a PR.

Codecov Report

Merging #1656 (5d9883f) into main (c10cd48) will not change coverage. The diff coverage is n/a.

```diff @@ Coverage Diff @@

main #1656 +/-

======================================= Coverage 92.06% 92.06%
======================================= Files 34 34
Lines 4576 4576
======================================= Hits 4213 4213
Misses 363 363
```

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update c10cd48...5d9883f. Read the comment docs.

The thing that broke everything was this change:

I'm going to bring back the horrible get_format() method for the moment, with its weird mutations of the args object, then try and get rid of it again later.

Easiest way to do this was with three reverts, then cherry-pick back the code of conduct.

The state I had got to prior to that revert is in https://github.com/simonw/datasette/tree/issue-1657-wip

I've made a real mess of this. I'm going to revert Datasettemain back to the last commit that passed the tests and try this again in a branch.

Just ran into this issue last night. I have a big table that's mostly numbers, but also a zip code column in a state where ZIP codes start with 0. Would be great to run something like this:

sh sqlite-utils insert data.db places file.csv --csv --detect-types --type zipcode text

Maybe I'll take a crack at this one.