Codecov Report

Base: 92.55% // Head: 92.55% // No change to project coverage :thumbsup:

Coverage data is based on head (4faa4fd) compared to base (bf00b0b). Patch has no changes to coverable lines.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Relevant sqlite forum thread: https://www.sqlite.org/forum/forumpost/02f7bda329f41e30451472421cf9ce7f715b768ce3db02797db1768e47950d48

API design: (**UPDATE: this was later changed to POST /db/table/-/insert)

POST /db/table Authorization: Bearer xxx Content-Type: application/json { "row": { "id": 1, "name": "New record" } } Returns: 201 Created { "row": { "id": 1, "name": "New record" } } You can omit optional fields in the input, including the ID field. The returned object will always include all fields - and will even include rowid if your object doesn't have a primary key of its own.

I decided to use "row" as the key in both request and response, to preserve space for other future keys - one that tells you that the table has been created, for example.

https://docs.datasette.io/en/stable/changelog.html#v0-63

Annotated release notes: https://simonwillison.net/2022/Oct/27/datasette-0-63/

Demo in Datasette Lite: https://lite.datasette.io/#/fixtures?sql=select%0A++pk1%2C%0A++pk2%2C%0A++content%2C%0A++sortable%2C%0A++sortable_with_nulls%2C%0A++sortable_with_nulls_2%2C%0A++text%0Afrom%0A++sortable%0Aorder+by%0A++pk1%2C%0A++pk2%0Alimit%0A++101

I'm going to do annotated release notes for this one.

Those release notes as markdown:

Features

• Now tested against Python 3.11. Docker containers used by datasette publish and datasette package both now use that version of Python. (#1853)
• --load-extension option now supports entrypoints. Thanks, Alex Garcia. (#1789)
• Facet size can now be set per-table with the new facet_size table metadata option. (#1804)
• The truncate_cells_html setting now also affects long URLs in columns. (#1805)
• The non-JavaScript SQL editor textarea now increases height to fit the SQL query. (#1786)
• Facets are now displayed with better line-breaks in long values. Thanks, Daniel Rech. (#1794)
• The settings.json file used in Configuration directory mode is now validated on startup. (#1816)
• SQL queries can now include leading SQL comments, using /* ... */ or -- ... syntax. Thanks, Charles Nepote. (#1860)
• SQL query is now re-displayed when terminated with a time limit error. (#1819)
• The inspect data mechanism is now used to speed up server startup - thanks, Forest Gregg. (#1834)
• In Configuration directory mode databases with filenames ending in .sqlite or .sqlite3 are now automatically added to the Datasette instance. (#1646)
• Breadcrumb navigation display now respects the current user's permissions. (#1831)

Plugin hooks and internals

• The prepare_jinja2_environment(env, datasette) plugin hook now accepts an optional datasette argument. Hook implementations can also now return an async function which will be awaited automatically. (#1809)
• Database(is_mutable=) now defaults to True. (#1808)
• The datasette.check_visibility() method now accepts an optional permissions= list, allowing it to take multiple permissions into account at once when deciding if something should be shown as public or private. This has been used to correctly display padlock icons in more places in the Datasette interface. (#1829)
• Datasette no longer enforces upper bounds on its dependencies. (#1800)

Documentation

• New tutorial: Cleaning data with sqlite-utils and Datasette.
• Screenshots in the documentation are now maintained using shot-scraper, as described in Automating screenshots for the Datasette documentation using shot-scraper. (#1844)
• More detailed command descriptions on the CLI reference page. (#1787)
• New documentation on Running Datasette using OpenRC - thanks, Adam Simpson. (#1825)

See also: - https://github.com/simonw/datasette/pull/1837

Yeah this is better, thanks!

@dependabot rebase

And the response to "inserted": [{...}] - it will be the same for bulk inserts.

I'm going to change the incoming JSON back to {"row": {...}} - no need to POST {"insert": ...} to something with /-/insert in the URL already.

Need to implement the new URL design from: - #1868

This is now going to be /db/table/-/insert - and it will eventually handle bulk inserts as well as single inserts.

I'm happy with this /db/table/-/action design for the moment. Will review it once I've built it to see if I still like it!

I'm going to stick with one /-/insert endpoint which handles both single row inserts AND multiple row inserts I think - partly because I don't want to build both /-/upsert and /-/upsert-many, I'd rather just have /-/upsert.

So given this scheme, the URL design would look like this:

• POST /db/table/-/insert - insert a single row
• POST /db/table/-/insert-many - insert multiple rows (might just keep that on /-/insert with a JSON array rather than object though)
• POST /db/table/-/drop - drop a table
• POST /db/table/-/alter - alter a table
• POST /db/table/-/upsert - upsert, https://sqlite-utils.datasette.io/en/stable/python-api.html#upserting-data
• POST /db/table/-/create - could be an endpoint for explicitly creating a table, or should that live at /db/-/create instead?

And for rows (pks here since compound primary keys are supported):

• POST /db/table/pks/-/update - update row
• POST /db/table/pks/-/delete - delete row

The other option here would be to lean into custom HTTP verbs like DELETE and PATCH. I'm not sold on those: they've never given me any convincing wins over just using POST for the many times I've encountered them in my career to date.

The problem with the above design is that I want to support a bunch of different actions that can be taken against a table: - insert a single row - insert multiple rows - bulk update rows - rename table - alter table - drop table

I could have ALL of those be a POST /db/table with different JSON root keys ({"drop": true} for example, but this raises two problems:

1. Server logs that only show POST /db/table will be less useful, they won't reveal what action was performed
2. What happens if you send {"insert": {"title": "New record"}, "drop": true}? Does that return an error, or does it perform both of those actions?

This is already slightly confusing in that POST /db/name-of-query is the existing API for executing a writable canned query: https://docs.datasette.io/en/stable/sql_queries.html#json-api-for-writable-canned-queries

So I'm ready to consider other design options.

Initial thoughts on possible designs (for the single row insert case, but could be expanded to cover other verbs):

• POST /db/table?action=insert
• POST /db/table?nsert
• POST /db/table/-/insert

I quite like that third one: it feels consistent with the existing /-/actor etc pages that Datasette serves already.

There's one slight confusion here in that it overlaps with the URL for a row with a primary key of "-" - which is currently at /db/table/- - but that might be OK.

Especially if I say that child pages of rows must theselves use the /-/ pattern. So to update or delet a row you would use:

• POST /db/table/row/-/update
• POST /db/table/row/-/delete

So a row with primary key - would end up as /db/table/row/-/-/update - which I think is OK.

Updated docs: https://docs.datasette.io/en/1.0-dev/json_api.html#inserting-a-single-row

I'm going to change the design of this to: { "insert": { "title" :"..." } } Renaming "row" to "insert".

This will be consistent with adding "drop": true for dropping a table, and maybe other verbs like for modifying the schema.

The API response will look like this:

json { "inserted_row": { "id": 1, "title": "..." } }

The new code is now live at https://latest.datasette.io/fixtures

I think mine has a better pattern for handling /* ... anything in here that isn't */ ... */

Here's yours on Debuggex: https://www.debuggex.com/r/HjdJryTy9ezGsuWK

Hah, I just came up with this one - we were clearly working on this at the same time!

^\s*((?:\-\-.*?\n\s*)|(?:\/\*((?!\*\/)[\s\S])*\*\/)\s*)*\s*select\b

https://www.debuggex.com/r/Rbw-UWD9PdOU2GyO

Here is my suggestion:

^\s*((?:\-\-.*?\n\s*)|(?:/\*.*?(?=\*/)\*/\s*))*select\b

See the following test: https://regex101.com/r/Doeqqa/1

And here I played all your tests: https://regexr.com/713ir

If people care about that kind of thing they could always push all of their inserts to a table called _tablename and then atomically rename that once they've uploaded all of the data (assuming I provide an atomic-rename-this-table mechanism).

There's one catch with batched inserts: if your CLI tool fails half way through you could end up with a partially populated table - since a bunch of batches will have succeeded first.

I think that's OK. In the future I may want to come up with a way to run multiple batches of inserts inside a single transaction, but I can ignore that for the first release of this feature.

Likewise for newline-delimited JSON. While it's tempting to want to accept that as an ingest format (because it's nice to generate and stream) I think it's better to have a client application that can turn a stream of newline-delimited JSON into batched JSON inserts.

So for the moment I'm just going to concentrate on the JSON API. I can consider CSV variants later on, or as plugins, or both.

Should this API accept CSV/TSV etc in addition to JSON?

I'm torn on this one. My initial instinct is that it should not - and there should instead be a Datasette client library / CLI tool you can use that knows how to turn CSV into batches of JSON calls for when you want to upload a CSV file.

I don't think the usability of curl https://datasette/db/table -F 'data=@path/to/file.csv' -H 'Authentication: Bearer xxx' is particularly great compared to something likedatasette client insert https://datasette/ db table file.csv --csv (where the command version could store API tokens for you too).

Error handling is really important here.

What should happen if you submit 100 records and one of them has some kind of validation error? How should that error be reported back to you?

I'm inclined to say that it defaults to all-or-nothing in a transaction - but there should be a "continue_on_error": true option (or similar) which causes it to insert the ones that are valid while reporting back the ones that are invalid.

Sorry I forgot the -- comments like that.

I'm afraid there is an issue in your regexp, see: https://regex101.com/r/pyubJf/1

I guess I can fix it.

Strongly related to: - #1866

Here's the issue where I started doing this: - #849

I mentioned this issue here: https://simonwillison.net/2020/Jun/26/weeknotes-plugins-sqlite-generate/

Repositories created by following the README in https://github.com/simonw/datasette-template and https://github.com/simonw/click-app have a main branch instead of master so I have a few examples live now. https://github.com/simonw/datasette-saved-queries is one example.

I should probably refactor this to use sqlite-utils, since I'm going to want to use that later for the feature that automatically creates tables.

Might make it easier to solve the rowid issues too.

Also need a test for invalid JSON (currently triggers a 500 HTML error).

My hunch on this is that anyone with that level of complex permissions requirements needs to be using a custom authentication plugin which includes much more concrete token rules, rather than the default signed stateless token implementation that ships with Datasette core.

Here's an interesting edge-case to consider: what if a user creates themselves a token for a specific table, then deletes that table, and waits for another user to create a table of the same name... and then uses their previously created token to write to the table that someone else created?

Not sure if this is a threat I need to actively consider, but it's worth thinking a little bit about the implications of such a thing - since there will be APIs that allow users to create tables, and there may be cases where people want to have a concept of users "owning" specific tables.

This is probably something that could be left for plugins to solve, but it still needs to be understood and potentially documented.

There may even be a world in which tracking the timestamp at which a table was created becomes useful - because that could then be baked into API tokens, such that a token created BEFORE the table was created does not grant access to that table.

And come up with a whole bunch of tests for weird table shapes, surprising column names, different types etc.

Also need to think about transactions - it should use them!

Documentation for this first draft of the API: https://docs.datasette.io/en/1.0-dev/json_api.html#inserting-a-single-row

It currently returns errors as HTML - it needs to return errors as JSON. Also the errors need comprehensive test coverage.

I'm also worried about what happens if you use it on a table that doesn't use an integer primary key - need to check that. I think this code may break:

https://github.com/simonw/datasette/blob/51c436fed29205721dcf17fa31d7e7090d34ebb8/datasette/views/table.py#L155-L171

Plus will rowid tables without an explicit primary key return the rowid column? They should.

Working on that first "insert row" implementation:

• https://github.com/simonw/datasette/issues/1851

Has made it very clear to me that I should go the whole hog and build the basic form-based interface for this as well.