TIL: https://til.simonwillison.net/json/json-pointer

Spec looks pretty simple:

A JSON Pointer is a Unicode string (see [RFC4627], Section 3) containing a sequence of zero or more reference tokens, each prefixed by a / (%x2F) character.

Because the characters ~ (%x7E) and / (%x2F) have special meanings in JSON Pointer, ~ needs to be encoded as ~0 and / needs to be encoded as ~1 when these characters appear in a reference token.

That's using JSON Pointer: https://www.rfc-editor.org/rfc/rfc6901

There's a Python library for that here https://github.com/stefankoegl/python-json-pointer/blob/master/jsonpointer.py - which looks simple and clean and well maintained and documented, but it only handles the "what is at this pointer within this JSON object" case - I need to generate the correct JSON pointer to explain where my error is.

So I think I'll end up hand-rolling this.

Here's the most relevant example from the RFC spec: POST /details HTTP/1.1 Host: account.example.com Accept: application/json json { "age": 42.3, "profile": { "color": "yellow" } } HTTP/1.1 400 Bad Request Content-Type: application/problem+json Content-Language: en json { "type": "https://example.net/validation-error", "title": "Your request is not valid.", "errors": [ { "detail": "must be a positive integer", "pointer": "#/age" }, { "detail": "must be 'green', 'red' or 'blue'", "pointer": "#/profile/color" } ] }

Happy Birthday Datasette!

I am a librarian at the Université du Québec à Montréal (UQAM) and I've been using Datasette to publish excerpts of our library data. There are several use cases I'm working with as a proof of concept : 1. New titles list : based on reports of recent acquisitions by subject, discipline, etc. 2. List of all UQAM theses and dissertations : based on an extract of bibliographic records 3. List of all publications by UQAM Authors : based on an extract of bibliographic records

See our prototype under construction here : https://datasette-bib.uqam.ca/ (some bits and pieces have been translated into French)

Datasette is amazing, there is so much potential here for libraries. Thanks to Simon and all the contributors for this outstanding effort. Also sqlite-utils deserves special mention as incredibly handy and useful.

This probably counts as a case study: https://github.com/eyeseast/spatial-data-cooking-show. Even has video.

Seriously, though, this workflow has become integral to my work with reporters and editors across USA TODAY Network. Very often, I get sent a folder of data in mixed formats, with a vague ask of how we should communicate some part of it to users. Datasette and its constellation of tools makes it easy to get a quick look at that data, run exploratory queries, map it and ask questions to figure out what's important to show. And then I export a version of the data that's exactly what I need for display.

I use Datasette to analyze blocklists by using csv-to-sqlite to pull their contents into a database and Datasette to look around through them. I also use its REST API to query said database as part of filtering out garbage from domains found in those blocklists.

I'm realizing I don't know if a virtual table will ever return a count. Maybe it depends on the implementation. For these three, just checking now, it'll always return zero.

That said, I'm not sure there's any downside to having them return zero and caching that. (They're hidden, too.)

I wonder if there are any reasons that inspect SHOULD try to count virtual tables? Like are there any likely uses for a cirial table where the count is both interesting and likely to be accessed often enough that it's worth caching?

I have an issue open to add a setting to disable table counts entirely:

• 1818

Maybe that should be expanded to automatically disable row counts for virtual tables entirely? Which would mean no count would be shown for them in the UI.

If you desperately wanted a count you would then have to run a count(*) query against them explicitly.

It does work, though I think it's probably still worth excluding virtual tables that will always be zero. Here's the same inspection as before, now with --load-extension spatialite:

json { "alltheplaces": { "hash": "0843cfe414439ab903c22d1121b7ddbc643418c35c7f0edbcec82ef1452411df", "size": 963375104, "file": "alltheplaces.db", "tables": { "spatial_ref_sys": { "count": 6215 }, "spatialite_history": { "count": 18 }, "sqlite_sequence": { "count": 2 }, "geometry_columns": { "count": 3 }, "spatial_ref_sys_aux": { "count": 6164 }, "views_geometry_columns": { "count": 0 }, "virts_geometry_columns": { "count": 0 }, "geometry_columns_statistics": { "count": 3 }, "views_geometry_columns_statistics": { "count": 0 }, "virts_geometry_columns_statistics": { "count": 0 }, "geometry_columns_field_infos": { "count": 0 }, "views_geometry_columns_field_infos": { "count": 0 }, "virts_geometry_columns_field_infos": { "count": 0 }, "geometry_columns_time": { "count": 3 }, "geometry_columns_auth": { "count": 3 }, "views_geometry_columns_auth": { "count": 0 }, "virts_geometry_columns_auth": { "count": 0 }, "data_licenses": { "count": 10 }, "sql_statements_log": { "count": 0 }, "states": { "count": 56 }, "counties": { "count": 3234 }, "idx_states_geometry_rowid": { "count": 56 }, "idx_states_geometry_node": { "count": 3 }, "idx_states_geometry_parent": { "count": 2 }, "idx_counties_geometry_rowid": { "count": 3234 }, "idx_counties_geometry_node": { "count": 98 }, "idx_counties_geometry_parent": { "count": 97 }, "idx_places_geometry_rowid": { "count": 1236796 }, "idx_places_geometry_node": { "count": 38163 }, "idx_places_geometry_parent": { "count": 38162 }, "places": { "count": 1332609 }, "SpatialIndex": { "count": 0 }, "ElementaryGeometries": { "count": 0 }, "KNN": { "count": 0 }, "idx_states_geometry": { "count": 56 }, "idx_counties_geometry": { "count": 3234 }, "idx_places_geometry": { "count": 1236796 } } } }

Nothing spectacular yet but I think this falls under "cool/cute application of datasette": improving fakedata performance for fun. tl;dr I used datasette to visualize benchmarking data.

Hi @simonw and thanks for the great tools you're publishing, your dedication is inspiring!

I work for the French Ministry of Culture on a surveying tool for objects protected for their historical value. It is part of a program building modern public services called beta.gouv.fr.

In that context I'm using data published by the Ministry that I have ingested into datasette and published on a free Fly instance : https://collectif-objets-datasette.fly.dev . I have also ingested another data set with infos about french cities on this instance so that I can perform joined queries.

The surveying tool synchronizes its data regularly from this datasette instance, and I also use it to perform queries when asked generic questions about the distribution of objects. (The data is not very accessible as it's undocumented and for internal usage mostly)

The API explorer is now live here: https://latest-1-0-dev.datasette.io/-/api

I think the ability to create tokens should be protected by a create-tokens permission, not just a global setting.

Demo: https://latest-1-0-dev.datasette.io/-/api

I changed my mind about the "return_rows": true option - I'm going to rename it to "return": true.

Demo: https://latest-1-0-dev.datasette.io/-/api#path=%2Ffixtures%2Ffacetable%2F-%2Fdrop&json=&method=POST

Demo: https://latest-1-0-dev.datasette.io/-/api#path=%2Ffixtures%2Ffacetable%2F-%2Fdrop&json=%7B%22confirm%22%3A+true%7D&method=POST

I also changed the confirmation JSON returned by this endpoint to add the database and table like so: json { "ok": true, "database": "data", "table": "docs", "row_count": 1, "message": "Pass \"confirm\": true to confirm" } Updated docs: https://docs.datasette.io/en/1.0-dev/json_api.html#dropping-tables

Found a bug: you get a 500 error if you try this against an immutable database.

New documentation for these features currently lives here: https://docs.datasette.io/en/1.0-dev/json_api.html#the-json-write-api

I started a conversation about JSON error standards on Mastodon here: https://fedi.simonwillison.net/web/@simon/109338725610487457

Quite a few people pointed to this RFC independently.

Updated docs: https://docs.datasette.io/en/1.0-dev/json_api.html#dropping-tables

I'm going to add a "confirm": true option to the API. Without that, it returns a note about how many rows will be deleted.

In playing with the API explorer just now I realized it's way too easy to accidentally drop a table using it.

For the example links - I'm going to have these at the bottom of the page so you don't have to scroll past them.

Ideally these would take the user's permissions into account. This could make the page expensive to load, but I'm going to risk it for the moment.

Something like this then:

• data
• /data/-/create - create table
• /data/table1/-/insert - insert into table1
• /data/table1/-/drop - drop table1

I won't bother with per-row demo links (for update and delete) because there could be thousands of them for each table.

Two things left before I close this issue:

• [x] I want to preserve the state of the forms in the URL - probably after a #
• [ ] Instead of hard-coding the current examples, I want to provide a list of links which populate the forms

Tim Sherratt on Twitter: https://twitter.com/wragge/status/1591930345469153282

Where do I start? The #GLAMWorkbench now includes a number of examples where GLAM data is harvested, processed, and then made available for exploration via Datasette.

https://glam-workbench.net/

For example the GLAM Name Index Search brings together 10+ million entries from 240 indexes and provides an aggregated search using the Datasette search-all plugin:

https://glam-workbench.net/name-search/

Most recently I converted PDFs of the Tasmanian Postal Directories to a big Datasette instance: https://updates.timsherratt.org/2022/09/15/from-pdfs-to.html the process is documented and reusable.

I'm a cryptic crossword enthusiast and have spent a lot of time scraping and parsing cryptic crossword clues from various blogs, forums and publications. The result is over half a million clues from cryptic crosswords over the past twelve years, including the clue, answer, puzzle date, puzzle name and a link to the original source. This is all hosted using Datasette, which has been a delight to use: https://cryptics.georgeho.org/

This dataset is a significant work of crossword archivism and scholarship, as acquiring historical crosswords and structuring their contents require focused effort and tedious cleaning that few are willing to do for such trivial data - for example, according to this 2004 selection guide, the Library of Congress explicitly does not collect crossword puzzles. Anecdotally, I know that many constructors/setters of cryptic crosswords use this dataset as a resource, and some even simply call it "the database" - this is probably one of the most impactful data projects I've worked on!