This feels quite nice:

I want to package together all of the extras that are needed for the HTML format. A few options for doing that:

• Introduce ?_extra=_html where the leading underscore indicates that this is a "bundle" of extras, then define a bundle that's everything needed for the HTML renderer
• Have some other mechanism whereby different renderers can request a bundle of extras.

I'm leaning towards the first option. I'll try that and see what it looks like.

To get this unblocked, I'm going to allow myself to pass non-JSON-serializable objects to the HTML template version of things. If I can get that working (and get the existing tests to pass) I can consider a later change that makes those JSON serializable - or admit that it's OK for the templates to have non-JSON data passed to them and figure out how best to document those variables independently from the JSON documentation.

The reason for that Row thing is that it allows custom templates that do things like this:

https://docs.datasette.io/en/stable/changelog.html#easier-custom-templates-for-table-rows

html+jinja {% for row in display_rows %} <div> <h2>{{ row["title"] }}</h2> <p>{{ row["description"] }}<lp> <p>Category: {{ row.display("category_id") }}</p> </div> {% endfor %} Is that a good design? the .display() thing feels weird - I wonder if anyone has ever actually used that.

It's documented here: https://docs.datasette.io/en/0.64.2/custom_templates.html#custom-templates

If you want to output the rendered HTML version of a column, including any links to foreign keys, you can use {{ row.display("column_name") }}.

I can't see any examples of anyone using it in this code search: https://cs.github.com/?scopeName=All+repos&scope=&q=datasette+row.display

It is however useful to have some kind of abstraction layer here that insulates the SQLite Row object, since having an extra layer will help if Datasette ever grows support for alternative database backends such as DuckDB or PostgreSQL.

Aside idea: it might be interesting if there were "lazy" template variables available in the context: things that are not actually executed unless a template author requests them.

Imagine if metadata was a lazy template reference, such that custom templates that don't display any metadata don't trigger it to be resolved (which might involve additional database queries some day).

Figuring out what to do with display_columns_and_rows() is hard. That returns rows as this special kind of object, which is designed to be accessed from the HTML templates:

https://github.com/simonw/datasette/blob/96e94f9b7b2db53865e61390bcce6761727f26d8/datasette/views/table.py#L45-L71

For the HTML version, I need to decide where all of the stuff that happens in async def extra_template() is going to live.

I think it's another one of those extra functions, triggered for ?_extra=context.

https://github.com/simonw/datasette/blob/96e94f9b7b2db53865e61390bcce6761727f26d8/datasette/views/table.py#L813-L912

The ease with which I added that ?_extra=query feature in https://github.com/simonw/datasette/pull/1999/commits/96e94f9b7b2db53865e61390bcce6761727f26d8 made me feel really confident that this architecture is going in the right direction.

```diff diff --git a/datasette/views/table.py b/datasette/views/table.py index 8d3bb2c930..3e1db9c85f 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -1913,6 +1913,13 @@ async def extra_request(): "args": request.args._data, }

• async def extra_query():
• "Details of the underlying SQL query"
• return {
• "sql": sql,
• "params": params,
• } + async def extra_extras(): "Available ?_extra= blocks" return { @@ -1938,6 +1945,7 @@ async def extra_extras(): extra_primary_keys, extra_debug, extra_request,
• extra_query, extra_extras, ) ```

Just noticed that _json=colname is not working, and that's because it's handled by the renderer here:

https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/renderer.py#L29-L40

But that's not currently being called by my new code.

I'm going to hold off on that refactor until later, when I have tests to show me if the refactor works or not.

So maybe I can refactor it to look a bit more like this:

https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L1602-L1604

One thing that's useful here is that is_view is handled early, like this:

https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L466-L472

So if I omit the is_view bit from the extracted function I can simplify more.

I'd really like to extract this ugly logic out into a helper function:

https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L1617-L1684

I copied it in and asked ChatGPT to "List all of the variable that are either modified or assigned to by the above code":

Modified variables:

offset
where_clauses
params
order_by

Assigned variables:

sort_value
next_by_pk_clauses
order_by (when sort or sort_desc is not None)

Then I asked which variables were used as inputs, and argued with it a bit about whether it should be counting functions. Eventually got to this:

My apologies for the oversight. Here are the variables needed as input by the above code, excluding any functions:

_next
is_view
sort
sort_desc
use_rowid
pks
order_by

Note that use_rowid is actually defined earlier in terms of two of those other variables: https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L1540

This fix is released in 0.64.2 https://docs.datasette.io/en/stable/changelog.html#v0-64-2

Here's the https://latest.datasette.io/ deployment that just went out, further demonstrating that this change is working correctly:

Got the same failure again for a recent commit: https://github.com/simonw/datasette/actions/runs/4368239376/jobs/7640567282

It's this test here:

https://github.com/simonw/datasette/blob/1ad92a1d87d79084ebe524ed186c900ff042328c/tests/test_cli.py#L181-L189

Added in: - #2033

Instead of using isolated_filesystem() I could use a tmpdir fixture instead.

https://console.cloud.google.com/run/detail/us-central1/new-service/revisions?project=datasette-222320 confirms that the image deployed is:

Compared to https://console.cloud.google.com/run/detail/us-central1/datasette-io/revisions?project=datasette-222320 which shows that datasette.io is running:

Testing this manually:

``` % datasette publish cloudrun content.db --service new-service Creating temporary tarball archive of 2 file(s) totalling 13.8 MiB before compression. Uploading tarball of [.] to [gs://datasette-222320_cloudbuild/source/1678306859.271661-805303f364144b6094cc9c8532ab5133.tgz] Created [https://cloudbuild.googleapis.com/v1/projects/datasette-222320/locations/global/builds/290f41a4-e29a-443c-a1e5-c54513c6143d]. Logs are available at [ https://console.cloud.google.com/cloud-build/builds/290f41a4-e29a-443c-a1e5-c54513c6143d?project=99025868001 ]. ---- REMOTE BUILD OUTPUT ---- starting build "290f41a4-e29a-443c-a1e5-c54513c6143d"

FETCHSOURCE Fetching storage object: gs://datasette-222320_cloudbuild/source/1678306859.271661-805303f364144b6094cc9c8532ab5133.tgz#1678306862810483 Copying gs://datasette-222320_cloudbuild/source/1678306859.271661-805303f364144b6094cc9c8532ab5133.tgz#1678306862810483... / [1 files][ 3.9 MiB/ 3.9 MiB]
Operation completed over 1 objects/3.9 MiB. BUILD Already have image (with digest): gcr.io/cloud-builders/docker Sending build context to Docker daemon 14.52MB Step 1/9 : FROM python:3.11.0-slim-bullseye ... Installing collected packages: rfc3986, typing-extensions, sniffio, PyYAML, python-multipart, pluggy, pint, mergedeep, MarkupSafe, itsdangerous, idna, hupper, h11, click, certifi, asgiref, aiofiles, uvicorn, Jinja2, janus, click-default-group-wheel, asgi-csrf, anyio, httpcore, httpx, datasette Successfully installed Jinja2-3.1.2 MarkupSafe-2.1.2 PyYAML-6.0 aiofiles-23.1.0 anyio-3.6.2 asgi-csrf-0.9 asgiref-3.6.0 certifi-2022.12.7 click-8.1.3 click-default-group-wheel-1.2.2 datasette-0.64.1 h11-0.14.0 httpcore-0.16.3 httpx-0.23.3 hupper-1.11 idna-3.4 itsdangerous-2.1.2 janus-1.0.0 mergedeep-1.3.4 pint-0.20.1 pluggy-1.0.0 python-multipart-0.0.6 rfc3986-1.5.0 sniffio-1.3.0 typing-extensions-4.5.0 uvicorn-0.20.0 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip available: 22.3 -> 23.0.1 [notice] To update, run: pip install --upgrade pip Removing intermediate container 8ccebfebebc9 ---> b972c85b38bb ... Successfully built 606b7c286d7f Successfully tagged gcr.io/datasette-222320/datasette-new-service:latest PUSH Pushing gcr.io/datasette-222320/datasette-new-service The push refers to repository [gcr.io/datasette-222320/datasette-new-service] 667b1dc69e5e: Preparing ... d8ddfcff216f: Pushed latest: digest: sha256:452daffb2d3d7a8579c2ab39854be285155252c9428b4c1c50caac6a3a269e3f size: 2004 DONE

ID CREATE_TIME DURATION SOURCE IMAGES STATUS 290f41a4-e29a-443c-a1e5-c54513c6143d 2023-03-08T20:21:03+00:00 39S gs://datasette-222320_cloudbuild/source/1678306859.271661-805303f364144b6094cc9c8532ab5133.tgz gcr.io/datasette-222320/datasette-new-service (+1 more) SUCCESS Deploying container to Cloud Run service [new-service] in project [datasette-222320] region [us-central1] ✓ Deploying new service... Done.
✓ Creating Revision...
✓ Routing traffic...
✓ Setting IAM Policy...
Done.
Service [new-service] revision [new-service-00001-zon] has been deployed and is serving 100 percent of traffic. Service URL: https://new-service-j7hipcg4aq-uc.a.run.app ``` https://new-service-j7hipcg4aq-uc.a.run.app/ was deployed successfully.

I'm going to solve this by using the service name in that image_id instead:

python image_id = f"gcr.io/{project}/{service_name}" This is a nasty bug, so I'm going to backport it to a 0.64.2 release as well.

I think the code at fault is here:

https://github.com/simonw/datasette/blob/1ad92a1d87d79084ebe524ed186c900ff042328c/datasette/publish/cloudrun.py#L176-L182

That name ends up defaulting to datasette - so multiple different projects may end up deploying to the same image_id.

What I think happened in the datasette.io bug is that this workflow: https://github.com/simonw/simonwillisonblog-backup/blob/bfb573e96d8622ab52b22fdcd54724fe6e59fd24/.github/workflows/backup.yml and this workflow: https://github.com/simonw/datasette.io/blob/4676db5bf4a3fc9f792ee270ec0c59eb902cd2c3/.github/workflows/deploy.yml both happened to run at the exact same time.

And so the image that was pushed to gcr.io/datasette-222320/datasette:latest by the simonw/simonwillisonblog-backup action was then deployed by the simonw/datasette.io/ action, which broke the site.

+1, I have been wishing for this feature (also for use with template-sql). It was requested before here #1304.

I'm trying to get http://127.0.0.1:8001/fixtures/compound_three_primary_keys?_next=a,d,v to return the correct results.

Pushed that prototype to a branch: https://github.com/simonw/datasette/commit/0fe844e9adb006a0138e83102ced1329d9155c59 / https://github.com/simonw/datasette/compare/sql-list-parameters

I should consider the existing design of magic parameters here: https://docs.datasette.io/en/stable/sql_queries.html#magic-parameters

• _actor_*
• _header_*
• _cookie_
• _now_epoch
• _now_date_utc
• _now_datetime_utc
• _random_chars_*

Should this new id__list syntax look more like those magic parameters, or is it OK to use name__magic syntax here instead?

To implement this properly need to do the following: - Get the page to display multiple id: [ text input here ] fields such that re-submission works - Figure out how this should work for canned queries and for writable canned queries - Tests that cover queries, canned queries, writable canned queries

And a bonus feature: what if the Datasette UI layer spotted :id__list parameters and used them to add a bit of JavaScript that allowed users to click a + button next to an id form field to add another one?

Also, when a page is re-displayed for on of these queries it could potentially add an extra form field allowing people to add another value.

Though this has an annoying problem: how to tell the difference between an additional id input field that the user chose not to populate, v.s. one that is supposed to represent an empty string?

Maybe only support multiple id fields for users with JavaScript in order to avoid this problem.

Got a prototype working: ```diff diff --git a/datasette/views/database.py b/datasette/views/database.py index 8d289105..6f9d8a44 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -226,6 +226,12 @@ class QueryView(DataView): ): db = await self.ds.resolve_database(request) database = db.name + # Disallow x__list query string parameters + invalid_params = [k for k in request.args if k.endswith("__list")] + if invalid_params: + raise DatasetteError( + "Invalid query string parameters: {}".format(", ".join(invalid_params)) + ) params = {key: request.args.get(key) for key in request.args} if "sql" in params: params.pop("sql") @@ -258,6 +264,11 @@ class QueryView(DataView): for named_parameter in named_parameters if not named_parameter.startswith("_") } + # Handle any __list parameters + for named_parameter in named_parameters: + if named_parameter.endswith("__list"): + list_values = request.args.getlist(named_parameter[:-6]) + params[named_parameter] = json.dumps(list_values)

     # Set to blank string if missing from params
     for named_parameter in named_parameters:

`` This isn't yet doing the right thing on form re-submission: it breaks because it attempts to pass through the?id__list=` invalid parameter. But I did manage to get it to do this through careful editing of the URL:

That was this URL: http://127.0.0.1:8034/content?sql=select+%3Aid__list%2C*+from+releases+where+id+in+(select+value+from+json_each(%3Aid__list))&id=62642726&id=18402901&id=38714866

Also: datasette-explain may need to be updated to understand how to handle this:

ERROR: conn=<sqlite3.Connection object at 0x102834940>, sql = 'explain select * from releases where id in (select id from json_each(:id__list))', params = None: You did not supply a value for binding parameter :id__list.

Trickiest part of the implementation here is that it needs to know to output three id HTML form fields on the page, such that their values are persisted when the form is submitted a second time.

Since we are pre-1.0 it's still OK to implement a feature that disallows ?id__list= in the URL, but allows :id__list in SQL queries to reference the JSON list of parameters.

So I'm going to prototype this as the :id__list feature and see how it feels.

I'm pretty sold on this as a feature now. The main question I have is which of these options to implement:

1. ?id=1&?id=2 results in :id in the query being ["1", "2"] - no additional syntax required
2. :id in the query continues to reference just the first of those parameters - but :id__list (or some other custom syntax) instead gets ["1", "2"] - or, if the URL is ?id=1 - gets ["1"]

Actually on writing these out I realize that option 2 is the ONLY valid option. It's no good building a query that works against a JSON list if the user might pass just a single ID, ?id=1, resulting in their query breaking.

https://github.com/rclement/datasette-dashboards/issues/54 makes the excellent point that the <select multiple> default HTML widget produces this exact format of query string:

```html

`` Submitting that form with the middle two options selected navigates to:https://www.example.com/?id=32&id=15`

One variant on this idea: maybe you have to specify in your query that you want it to be the JSON list version, not the single item (first ?id= parameter version)? Maybe with syntax like this:

where id in (select value from json_each(:id__list))

Datasette would automatically pass {"id": "11", "id__list": '["11", "32", "62"]'} as arguments to the db.execute() method, if the page was called with ?id=11&id=32&id=62.

This is more explicit, though the syntax is a bit uglier (maybe there's a nicer design for this?). I also worry about ?id__list= conflicting with this, but I think that's a risk I can take - tell people not to do that, or even block ?id__list= style parameters entirely.

My current preferred solution is to lean into SQLite's JSON support.

What if the query page spotted ?id=11&id=32&id=62 and turned that into a JSON string called :id: with a value of ["11", "32", "62"]?

Note that this is still a string, not a list. This avoids a nasty problem that occurred in PHP world, where ?id[]=1&id[]=2 would result in an actual PHP array object, which often broke underlying code that had expected $_GET["id"] to be a string, not an array.

So in a query you'd be able to do this:

where id in (select value from json_each(:id))

And then call it with ?id=11&id=32&id=62.

One way to do this would be to dynamically generate the where id in (?, ?, ?) with the correct number of question marks, then feed in a list from request.args.getlist("id") - but that would require rewriting the SQL query text to add those question marks.

I managed to make it work by using nginx's 'exact match' (=) combined with 'prefix match'; that is, match explicitly on /, and redirect to /<db>/<table>, and then have the normal ProxyPath for the unadorned (prefix-matching) /.

location = / { return 302 /<db>/<table>; } location / { proxy_pass http://127.0.0.1:8001/; proxy_set_header Host $host; }

I've implemented the test (thanks for pointing me in the right direction!).

At tmcl-it/datasette:0.64.1+row-view-expand-labels I also have a variant of this patch that applies to the 0.64.x branch. Please let me know if you'd be interested in merging that as well and I'll open another PR.

This would be a nice feature to have with datasette publish too.

Documentation: https://docs.datasette.io/en/latest/plugins.html#installing-plugins

This is a very neat fix, for something I've been wanting for a while.

Add a unit test for the row HTML page - I suggest against this page: https://latest.datasette.io/fixtures/foreign_key_references/1 - and I'll land this PR.

You can model it on this test here: https://github.com/simonw/datasette/blob/a53b893c46453f35decc8c145c138671cee6140c/tests/test_table_html.py#L609-L632

I think adding it to test_table_html.py is OK, even though it's technically for the row page and not the table page.

Thanks!

Codecov Report

Patch and project coverage have no change.

Comparison is base (0b4a286) 92.11% compared to head (a8dde13) 92.11%.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Thanks!

I have this same issue, which is happening with both json links and facets. It is not happening with column sort links in the gear popup menus, but it is happening with the sort arrow that results after you use one of those links. I'm using Apache as a proxy to Datasette; the relevant configs are:

ProxyPass /datasette/ http://127.0.0.1:8000/datasette/ nocanon ProxyPreserveHost on

{ "base_url": "/datasette/" }

If it would be useful to get a look at the running installation via the Web, Simon, let me know.

I think I see what is happening here, although I haven't quite work out a fix yet. Usually:

• click.progressbar.render_progress() renders the cursor invisible on each invocation (update of the bar)
• When the progress bar goes out of scope, the __exit()__ method is invoked, which calls render_finish() to make the cursor re-appear.

(See terminal escape sequences BEFORE_BAR and AFTER_BAR in click).

However the sqlite-utils utils.file_progress context manager wraps click.progressbar and yields an instance of a helper class:

python @contextlib.contextmanager def file_progress(file, silent=False, **kwargs): ... with click.progressbar(length=file_length, **kwargs) as bar: yield UpdateWrapper(file, bar.update)

The yielded UpdateWrapper goes out of scope quickly and click.progressbar.__exit__() is called. The cursor is made un-invisible. Hoewever bar is still live and so when the caller iterates on the yielded wrapper this invokes the bar's update method, calling render_progress(), each time printing the "make cursor invisible" escape code. The progressbar.__exit__ function is not called again, so the cursor doesn't re-appear.

Thanks @dmick . I chose to create a firewall rule under my GCP to open the port of interest and datasette works.

@gk7279, you had asked in a separate bug about how to redirect web servers in general. The datasette docs actually have pretty good information on this for both nginx and apache2: https://docs.datasette.io/en/stable/deploying.html#running-datasette-behind-a-proxy

yes @dmick . How did you make your public IP redirect to your uvicorn server?

Instead of nginx, I have apache2 on my GCP VM. Any pointers here are helpful too.

Thanks.

Was that query to me, @gk7279?

Can you please help or share your expertise with #2030 ?

This is pretty annoying for FTS because sqlite throws an error instead of just doing something like returning all or no results. This makes users who are unfamiliar with SQL and Datasette think the canned query page is broken and is a frequent source of confusion.

To anyone dealing with this: My solution is to modify the canned query so that it returns no results which cues people to fill in the blank parameters.

So instead of emails_fts match escape_fts(:search))

My canned queries now look like this:

emails_fts match escape_fts(iif(:search=="", "*", :search))

There are no asterisks in my data so the result is always blank.

Ultimately it would be nice to be able to handle this in the metadata. Either making some named parameters required or setting some default values.

Meanwhile, a cheap workaround is to invalidate the registered function cache: python table.convert(...) db._registered_functions = set() table.convert(...)

Is this where we talk about i18n of results? Or is that a separate thread.

e.g. Having country_long show España in the Spanish version of the global power plants demo site instead of Spain.

Relevant: https://til.simonwillison.net/sqlite/enabling-wal-mode

Thanks for confirming my doubts! I removed it after opening this issue, yup, then had another issue with default_cache_ttl_hashed which I assume was removed at the same time. Sorry for not trying that before opening the issue.

I think this feature was removed in Datasette 0.61 and moved to a plugin. People who want hashed URLs can use the datasette-hashed-urls plugin to achieve the same affect.

It looks like you're trying to disable hashed urls, so I think you can just remove that config setting and things will work.

Nulls in primary keys, does it every time.

PR below

Just realized that it's useful to be able to tell what parameters were used to generate a page... but reflecting things like _next back in the JSON is confusing in the presence of next.

So I'm going to add an extra for that information too.

Not sure what to call it though:

• params - confusing because in the code that's usually used for params passed to SQL queries
• query_string - wouldn't that be a string, not params as a dictionary?

I'm going to experiment with a request extra that returns some bits of information about the request.

I live the patch here for others:

original code shell $ which sqlite-utils | xargs cat ```python

!/usr/bin/python3

-- coding: utf-8 --

import re import sys from sqlite_utils.cli import cli

if name == 'main': sys.argv[0] = re.sub(r'(-script.pyw|.exe)?$', '', sys.argv[0]) sys.exit(cli()) ```

patched/sqlite-utils.py ```python

!/usr/bin/python3

-- coding: utf-8 --

import re import sys from sqlite_utils.cli import cli

New imports

from unittest.mock import patch from sqlite_utils.cli import VALID_COLUMN_TYPES

if name == 'main': # Choices of the option --type cli.commands['transform'].params[2].type.types[1].choices.append('DATETIME')

# The dicts has to be extended with a new type
with patch.dict('sqlite_utils.db.COLUMN_TYPE_MAPPING', {'DATETIME': 'DATETIME'}),\
     patch('sqlite_utils.cli.VALID_COLUMN_TYPES', VALID_COLUMN_TYPES + ("DATETIME", )):

    # Command is unchanged
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(cli())

```

And now it's working ```bash $ sqlite-utils schema events.sqlite cards.chunk.get CREATE TABLE "cards.chunk.get" ( [id] INTEGER PRIMARY KEY NOT NULL, [timestamp] TEXT, )

$ python patched/sqlite-utils.py transform events.sqlite cards.chunk.get --type timestamp DATETIME

$ sqlite-utils schema events.sqlite cards.chunk.get CREATE TABLE "cards.chunk.get" ( [id] INTEGER PRIMARY KEY NOT NULL, [timestamp] DATETIME, ) ```

Next step: get ?_next=... working (it is ignored at the moment, even though the returned JSON includes the "next" key).

Then... figure out how to render HTML and other requested formats.

Then get the tests to pass!

Rather than duplicate this rather awful hack:

https://github.com/simonw/datasette/blob/0b4a28691468b5c758df74fa1d72a823813c96bf/datasette/views/table.py#L694-L714

I'm tempted to say that the code that calls the new pagination helper needs to ensure that the sort or sort_desc columns are selected. If it wants to ditch them later (e.g. because they were not included in ?_col=) it can do that later once the results have come back.

Most complicated example of a paginated query: https://latest.datasette.io/fixtures?sql=select%0D%0A++pk1%2C%0D%0A++pk2%2C%0D%0A++content%2C%0D%0A++sortable%2C%0D%0A++sortable_with_nulls%2C%0D%0A++sortable_with_nulls_2%2C%0D%0A++text%0D%0Afrom%0D%0A++sortable%0D%0Awhere%0D%0A++(%0D%0A++++sortable_with_nulls+is+null%0D%0A++++and+(%0D%0A++++++(pk1+%3E+%3Ap0)%0D%0A++++++or+(%0D%0A++++++++pk1+%3D+%3Ap0%0D%0A++++++++and+pk2+%3E+%3Ap1%0D%0A++++++)%0D%0A++++)%0D%0A++)%0D%0Aorder+by%0D%0A++sortable_with_nulls+desc%2C%0D%0A++pk1%2C%0D%0A++pk2%0D%0Alimit%0D%0A++101&p0=h&p1=r

sql select pk1, pk2, content, sortable, sortable_with_nulls, sortable_with_nulls_2, text from sortable where ( sortable_with_nulls is null and ( (pk1 > :p0) or ( pk1 = :p0 and pk2 > :p1 ) ) ) order by sortable_with_nulls desc, pk1, pk2 limit 101 Generated by this page: https://latest.datasette.io/fixtures/sortable?_next=%24null%2Ch%2Cr&_sort_desc=sortable_with_nulls

The _next= parameter there decodes as $null,h,r - and those components are tilde-encoded, so this can be distinguished from an actual $null value which would be represented as ~24null.

Hey, isn't this essentially the same issue as #448 ?

I'm going to build completely separate tests for this in test_pagination.py.

lets users make schema changes, so it's important to me that the tool work in a non-surprising way -- if you ask for a column of type X, you should get type X. If the column or table previously had CHECK constraints, they shouldn't be silently removed

I've got your concern. Let's see if we will be replied on it and i'll close the issue some later.

Ha, yes, I might end up making something very niche. That's OK.

I'm building a UI for Datasette that lets users make schema changes, so it's important to me that the tool work in a non-surprising way -- if you ask for a column of type X, you should get type X. If the column or table previously had CHECK constraints, they shouldn't be silently removed. And so on. I had hoped that I could just lean on sqlite-utils, but I think it's a little too surprising.

Ah, it looks like that is controlled by this dict: https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/db.py#L178

I suspect you could overwrite the datetime entry to achieve what you want

And thank you for pointing me to it. At least, i can make a monkey patch for my need...

Isn't your suggestion too fundamental for the utility?

The bigger flexibility, the bigger complexity. Your idea make sense defenitely, but how often do you make schema changes? And how many people could benefit from it, what do you think?

I think the bigger issue is that sqlite-utils mixes mechanism (it implements the 12-step way to alter SQLite tables) and policy (it has an opinionated stance on what column types should be used).

That might be a design choice to make it accessible to users by providing a reasonable set of defaults, but it doesn't quite fit my use case.

It might make sense to extract a separate library that provides just the mechanisms, and then sqlite-utils would sit on top of that library with its opinionated set of policies.

That would be a very big change, though.

I might take a stab at extracting the library, but just for the table schema migration piece, not all the other features that sqlite-utils supports. I wouldn't expect sqlite-utils to depend on it.

Part of my motivation is that I want to provide some other abilities, too, like support for CHECK constraints. I see that the issue in this repo (https://github.com/simonw/sqlite-utils/issues/358) proposes a bunch of short-hand constraints, which I wouldn't want to accidentally expose to people -- I want a layer that is a 1:1 mapping to SQLite.

Do you see a way to enable it without affecting existing users or bumping the major version number?

I don't see a clean solution, only extending code with a side variable that tells us we want to apply advanced types instead of basic.

it could be a similiar command like tranform-v2 --type column DATETIME or a cli option transform --adv-type column DATETIME along with a dict that contains the advanced types. Then with knowledge that we run an advanced command we take that dictionary somehow, we can wrap the current and new dictionaries by a superdict and work with it everywhere according to the knowledge. This way shouldn't affect users who are using the previous lib versions and it have to be merged in the next major one.

But this way looks a bad design, too messy.

I'd support that, but I'm not the author of this library.

One challenge is that would be a breaking change. Do you see a way to enable it without affecting existing users or bumping the major version number?

I could, of course.

Doest it worth bringing such the improvement to the library?

Is this feature covered by any more recent updates to datasette, or via any plugins that you're aware of?

Ah, it looks like that is controlled by this dict: https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/db.py#L178

I suspect you could overwrite the datetime entry to achieve what you want

That said, it looks like the check is only enforced at the CLI level. If you use the API directly, I think it'll work.

It works, but a column becomes TEXT

```python In [1]: import sqlite_utils In [2]: db = sqlite_utils.Database('events.sqlite') In [3]: table = db['cards.chunk.get'] In [4]: table.columns_dict Out[4]: {'id': int, 'timestamp': float, 'data_chunk_number': int, 'user_id': str, 'meta_duplication_source_id': int, 'context_sort_attribute': str, 'context_sort_order': str}

In [5]: from datetime import datetime In [7]: table.transform(types={'timestamp': datetime}) In [8]: table.columns_dict Out[8]: {'id': int, 'timestamp': str, 'data_chunk_number': int, 'user_id': str, 'meta_duplication_source_id': int, 'context_sort_attribute': str, 'context_sort_order': str} ```

bash ❯ sqlite-utils schema events.sqlite cards.chunk.get CREATE TABLE "cards.chunk.get" ( [id] INTEGER PRIMARY KEY NOT NULL, [timestamp] TEXT, ...

Doing this as a class makes sense to me. There are a few steps:

• Instantiate the class with the information it needs, which includes sort order, page size, tiebreaker columns and SQL query and parameters
• Generate the new SQL query that will actually be executed - maybe this takes the optional _next parameter? This returns the SQL and params that should be executed, where the SQL now includes pagination logic plus order by and limit
• The calling code then gets to execute the SQL query to fetch the rows
• Last step: those rows are passed to a paginator method which returns (rows, next) - where rows is the rows truncated to the correct length (really just with the last one cut off if it's too long for the length) and next is either None or a token, depending on if there should be a next page.

So I think I can write an abstraction that applies keyset pagination to ANY arbitrary SQL query provided it is given the query, the existing params (so it can pick names for the new params that won't overlap with them), the desired sort order, any existing _next token AND the columns that should be used to tie-break any duplicates.

Those tie breakers will be either the primary key(s) or rowid if none are provided.

What about the case of SQL views, where offset/limit should be used instead? I'm inclined to have that as a separate pagination abstraction entirely, with the calling code deciding which pagination helper to use based on if keyset pagination makes sense or not.

Might be easier to design a class structure for this starting with OffsetPaginator, then using that to inform the design of KeysetPaginator.

Might put these in datasette.utils.pagination to start off with, then maybe extract them out to sqlite-utils later once they've proven themselves.

Even more complicated: https://latest.datasette.io/fixtures/sortable?sortable_with_nulls__notnull=1&_next=0~2E692704598586882%2Ce%2Cr&_sort=sortable_with_nulls_2

The rewritten SQL for that is:

sql select * from (select pk1, pk2, content, sortable, sortable_with_nulls, sortable_with_nulls_2, text from sortable where "sortable_with_nulls" is not null) where (sortable_with_nulls_2 > :p2 or (sortable_with_nulls_2 = :p2 and ((pk1 > :p0) or (pk1 = :p0 and pk2 > :p1)))) order by sortable_with_nulls_2, pk1, pk2 limit 101 And it still has the same number of explain steps as the current SQL witohut the subselect.

A more complex example: https://latest.datasette.io/fixtures/sortable?_next=0~2E2650566289400591%2Ca%2Cu&_sort=sortable_with_nulls_2

SQL:

sql select pk1, pk2, content, sortable, sortable_with_nulls, sortable_with_nulls_2, text from sortable where (sortable_with_nulls_2 > :p2 or (sortable_with_nulls_2 = :p2 and ((pk1 > :p0) or (pk1 = :p0 and pk2 > :p1)))) order by sortable_with_nulls_2, pk1, pk2 limit 101

https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+content%2C+sortable%2C+sortable_with_nulls%2C+sortable_with_nulls_2%2C+text+from+sortable+where+%28sortable_with_nulls_2+%3E+%3Ap2+or+%28sortable_with_nulls_2+%3D+%3Ap2+and+%28%28pk1+%3E+%3Ap0%29%0A++or%0A%28pk1+%3D+%3Ap0+and+pk2+%3E+%3Ap1%29%29%29%29+order+by+sortable_with_nulls_2%2C+pk1%2C+pk2+limit+101&p0=a&p1=u&p2=0.2650566289400591

Here's the explain: 49 steps long https://latest.datasette.io/fixtures?sql=explain+select+pk1%2C+pk2%2C+content%2C+sortable%2C+sortable_with_nulls%2C+sortable_with_nulls_2%2C+text+from+sortable+where+%28sortable_with_nulls_2+%3E+%3Ap2+or+%28sortable_with_nulls_2+%3D+%3Ap2+and+%28%28pk1+%3E+%3Ap0%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3E+%3Ap1%29%29%29%29+order+by+sortable_with_nulls_2%2C+pk1%2C+pk2+limit+101&p2=0.2650566289400591&p0=a&p1=u

Rewritten with a subselect:

sql select * from ( select pk1, pk2, content, sortable, sortable_with_nulls, sortable_with_nulls_2, text from sortable ) where (sortable_with_nulls_2 > :p2 or (sortable_with_nulls_2 = :p2 and ((pk1 > :p0) or (pk1 = :p0 and pk2 > :p1)))) order by sortable_with_nulls_2, pk1, pk2 limit 101 https://latest.datasette.io/fixtures?sql=select+*+from+(%0D%0A++select+pk1%2C+pk2%2C+content%2C+sortable%2C+sortable_with_nulls%2C+sortable_with_nulls_2%2C+text+from+sortable%0D%0A)%0D%0Awhere+(sortable_with_nulls_2+%3E+%3Ap2+or+(sortable_with_nulls_2+%3D+%3Ap2+and+((pk1+%3E+%3Ap0)%0D%0A++or%0D%0A(pk1+%3D+%3Ap0+and+pk2+%3E+%3Ap1))))+order+by+sortable_with_nulls_2%2C+pk1%2C+pk2+limit+101&p2=0.2650566289400591&p0=a&p1=u

And here's the explain for that - also 49 steps: https://latest.datasette.io/fixtures?sql=explain+select+*+from+%28%0D%0A++select+pk1%2C+pk2%2C+content%2C+sortable%2C+sortable_with_nulls%2C+sortable_with_nulls_2%2C+text+from+sortable%0D%0A%29%0D%0Awhere+%28sortable_with_nulls_2+%3E+%3Ap2+or+%28sortable_with_nulls_2+%3D+%3Ap2+and+%28%28pk1+%3E+%3Ap0%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3E+%3Ap1%29%29%29%29+order+by+sortable_with_nulls_2%2C+pk1%2C+pk2+limit+101&p2=0.2650566289400591&p0=a&p1=u

For the SQL underlying this page (the second page in that compound primary key paginated sequence): https://latest.datasette.io/fixtures/compound_three_primary_keys?_next=a%2Cd%2Cv

The explain for the default query: https://latest.datasette.io/fixtures?sql=explain+select%0D%0A++pk1%2C%0D%0A++pk2%2C%0D%0A++pk3%2C%0D%0A++content%0D%0Afrom%0D%0A++compound_three_primary_keys%0D%0Awhere%0D%0A++%28%0D%0A++++%28pk1+%3E+%3Ap0%29%0D%0A++++or+%28%0D%0A++++++pk1+%3D+%3Ap0%0D%0A++++++and+pk2+%3E+%3Ap1%0D%0A++++%29%0D%0A++++or+%28%0D%0A++++++pk1+%3D+%3Ap0%0D%0A++++++and+pk2+%3D+%3Ap1%0D%0A++++++and+pk3+%3E+%3Ap2%0D%0A++++%29%0D%0A++%29%0D%0Aorder+by%0D%0A++pk1%2C%0D%0A++pk2%2C%0D%0A++pk3%0D%0Alimit%0D%0A++101&p0=a&p1=d&p2=v

The explain for that query rewritten as this:

sql explain select * from ( select pk1, pk2, pk3, content from compound_three_primary_keys ) where ( (pk1 > :p0) or ( pk1 = :p0 and pk2 > :p1 ) or ( pk1 = :p0 and pk2 = :p1 and pk3 > :p2 ) ) order by pk1, pk2, pk3 limit 101 https://latest.datasette.io/fixtures?sql=explain+select+*+from+%28select+%0D%0A++pk1%2C%0D%0A++pk2%2C%0D%0A++pk3%2C%0D%0A++content%0D%0Afrom%0D%0A++compound_three_primary_keys%0D%0A%29%0D%0A++where%0D%0A++%28%0D%0A++++%28pk1+%3E+%3Ap0%29%0D%0A++++or+%28%0D%0A++++++pk1+%3D+%3Ap0%0D%0A++++++and+pk2+%3E+%3Ap1%0D%0A++++%29%0D%0A++++or+%28%0D%0A++++++pk1+%3D+%3Ap0%0D%0A++++++and+pk2+%3D+%3Ap1%0D%0A++++++and+pk3+%3E+%3Ap2%0D%0A++++%29%0D%0A++%29%0D%0Aorder+by%0D%0A++pk1%2C%0D%0A++pk2%2C%0D%0A++pk3%0D%0Alimit%0D%0A++101&p0=a&p1=d&p2=v

Both explains have 31 steps and look pretty much identical.

Maybe the correct level of abstraction here is that pagination is something that happens to a SQL query that is defined as SQL and params, without an order by or limit. That's then wrapped in a sub-select and those things are added to it, plus the necessary where clauses depending on the page.

Need to check that the query plan for pagination of a subquery isn't slower than the plan for pagination as it works today.

Relevant issue: - https://github.com/simonw/datasette/issues/1773

Explains this comment: https://github.com/simonw/datasette/blob/0b4a28691468b5c758df74fa1d72a823813c96bf/datasette/views/table.py#L697

Found more logic relating to this:

https://github.com/simonw/datasette/blob/0b4a28691468b5c758df74fa1d72a823813c96bf/datasette/views/table.py#L684-L732

Crucial utility function: https://github.com/simonw/datasette/blob/0b4a28691468b5c758df74fa1d72a823813c96bf/datasette/utils/init.py#L137-L160

I should turn sort and sort_desc into an object representing the sort order earlier in the code.

I should also create something that bundles together pks and use_rowid and maybe is_view as well.

The inputs and outputs for this are pretty complex.

Inputs:

• ?_next= from the query string
• is_view - is this for a table or view? If it's a view it uses offset/limit pagination - which could actually work for arbitrary queries too. Also views could have keyset pagination if they are known to be sorted by a particular column.
• sort and sort_desc reflecting the current sort order
• use_rowid for if the table is a rowid table with no primary key of its own
• pks - the primary keys for the table
• params - the current set of parameters, I think used just to count their length so new params can be added as p5 etc without collisions. This could be handled with a s0, s1 etc naming convention instead.

Outputs:

• where_clauses - a list of where clauses to add to the query
• params - additional parameters to use with the query due to the new where clauses
• order_by - the order by clause

HTML mode needs a list of renderers so it can show links to .geojson etc - can do that as a hidden extra (maybe called renderers), repeating this code:

https://github.com/simonw/datasette/blob/e4ebef082de90db4e1b8527abc0d582b7ae0bc9d/datasette/views/base.py#L477-L497

I think that does make sense: ?_extra=table perhaps, which would add {"table": "..."}.

I think it's not currently possible: sqlite-utils requires that it be one of integer, text, float, blob (see code)

IMO, this is a bit of friction and it would be nice if it was more permissive. SQLite permits developers to use any data type when creating a table. For example, this is a perfectly cromulent sqlite session that creates a table with columns of type baz and bar:

``` sqlite> create table foo(column1 baz, column2 bar); sqlite> .schema foo CREATE TABLE foo(column1 baz, column2 bar); sqlite> select * from pragma_table_info('foo'); cid name type notnull dflt_value pk

0 column1 baz 0 0
1 column2 bar 0 0
```

The idea is that the application developer will know what meaning to ascribe to those types. For example, I'm working on a plugin to Datasette. Dates are tricky to handle. If you have some existing rows, you can look at the values in them to know how a user is serializing the dates -- as an ISO 8601 string? An RFC 3339 string? With millisecond precision? With timezone offset? But if you don't yet have any rows, you have to guess. If the column is of type TEXT, you don't even know that it's meant to hold a date! In this case, my plugin will look to see if the column is of type DATE or DATETIME, and assume a certain representation when writing.

Perhaps there is an argument that sqlite-utils is trying to conform to SQLite's strict mode, and that is why it limits the choices. In strict mode, SQLite requires that the data type be one of INT, INTEGER, REAL, TEXT, BLOB, ANY. But that can't be the case -- sqlite-utils supports FLOAT, which is not one of the valid types in strict mode, and it rejects INT, REAL and ANY, which are valid.

That said, it looks like the check is only enforced at the CLI level. If you use the API directly, I think it'll work.

SQLite doesn't have a native DATETIME type. It stores dates internally as strings and then has functions to work with date-like strings. Yes it's weird.

That's correct. But my issue is about the application level libraries that, i suppose, have better data understanding if see a specific type such as DATETIME.

I'm writing data with dataset i've mentioned. The lib changes its behavior depending on a type. I saw different behavior with types DATETIME, FLOAT, TEXT. Dataset, for their part, is built upon Sqlalchemy, you know what it is.

To be honest, i didn't dive into the details of why the behavior changes, but when i altered manually by other util a type of column to DATETIME things got back to normal.

On the matter, can i achieve it with Sqlite Utils at the moment?

SQLite doesn't have a native DATETIME type. It stores dates internally as strings and then has functions to work with date-like strings. Yes it's weird.

This absolutely makes sense. One of the biggest goals for Datasette 1.0 is "documented template contexts" - for any default template in Datasette that people might want to over-ride there should be documentation that describes the available context variables, plus tests that ensure they don't accidentally get broken by future changes.

Ensuring description/title/etc are available on the index page feels like it fits well into that bucket.

Came here to say that I also have this issue.

I think this is the relevant code:

https://github.com/simonw/datasette/blob/0b4a28691468b5c758df74fa1d72a823813c96bf/datasette/facets.py#L260-L268

Here's a quick prototype I knocked up for this: ```diff diff --git a/datasette/static/app.css b/datasette/static/app.css index 71437bd4..d763bcff 100644 --- a/datasette/static/app.css +++ b/datasette/static/app.css @@ -695,7 +695,48 @@ p.zero-results {

+/ Force table to not be like tables anymore / +body.row table.rows-and-columns, +body.row .rows-and-columns thead, +body.row .rows-and-columns tbody, +body.row .rows-and-columns th, +body.row .rows-and-columns td, +body.row .rows-and-columns tr { + display: block; +} + +/ Hide table headers (but not display: none;, for accessibility) / +body.row .rows-and-columns thead tr { + position: absolute; + top: -9999px; + left: -9999px; +} + +body.row .rows-and-columns tr { + border: 1px solid #ccc; + margin-bottom: 1em; + border-radius: 10px; + background-color: white; + padding: 0.2rem; +}

+body.row .rows-and-columns td { + / Behave like a "row" / + border: none; + border-bottom: 1px solid #eee; + padding: 0; + padding-left: 10%; + padding-bottom: 0.3em; +} + +body.row .rows-and-columns td:before { + display: block; + color: black; + padding-bottom: 0.2em; + font-size: 0.8em; + font-weight: bold; + background-color: #f5f5f5; +}

/ Overrides ===============================================================/ diff --git a/datasette/templates/row.html b/datasette/templates/row.html index 1d1b0bfd..339eb643 100644 --- a/datasette/templates/row.html +++ b/datasette/templates/row.html @@ -5,6 +5,9 @@ {% block extra_head %} {{- super() -}} <style> +{% for column in columns %} +body.row .rows-and-columns td:nth-of-type({{ loop.index }}):before { content: "{{ column|escape_css_string }}"; } +{% endfor %} @media only screen and (max-width: 576px) { {% for column in columns %} .rows-and-columns td:nth-of-type({{ loop.index }}):before { content: "{{ column|escape_css_string }}"; } ``` Now the row page looks like this at all page widths: I think that's better (could do with a bit of tightening up). One catch: you can't copy and paste the column labels, since they are added using generated content like this: https://github.com/simonw/datasette/blob/0b4a28691468b5c758df74fa1d72a823813c96bf/datasette/static/app.css#L752-L757 https://github.com/simonw/datasette/blob/0b4a28691468b5c758df74fa1d72a823813c96bf/datasette/templates/row.html#L9-L11 I think the row page should switch to different HTML entirely, rather than continuing to share the `<table>` that's used by the table page. This will be a breaking change for users who customize Datasette, so I should aim to ship it before 1.0.

I did some horrible things in https://github.com/cldellow/datasette-ui-extras/issues/2 to enable this in my plugin -- example here: https://dux-demo.fly.dev/cooking/posts?_facet=owner_user_id&owner_user_id=67

The implementation relies on two things:

• a filters_from_request hook that adds a good human description (unfortunately, without the benefit of the CSS styling you mention)
• doing something evil to hijack the exact and not operators in the Filters class. We can't leave them as is, or we'll get 2 human descriptions -- the built-in Datasette one and the one from my plugin. We can't remove them, or the filters UI will stop supporting the = and != operators

This got me thinking: it'd be neat if the list of operators that the filters UI supported wasn't a closed set.

A motivating example: adding a geospatial NEAR operator. Ideally it'd take two arguments - a target point and a radius, so you could express a filter like find me all rows whose lat/lng are within 10km of 43.4516° N, 80.4925° W. (Optionally, the UI could be enhanced if the geonames database was loaded and queried, so a user could say find me all rows whose lat/lng are within 10km of Kitchener, ON, and the city gets translated to a lat/lng for them)

We don't have any performance tests yet - would be a useful thing to add, I've not built anything like that before (at least not in CI, I've always done as-hoc performance testing using something like Locust) so I don't have a great feel for how it could work.

Had an interesting conversation about this just now: https://fedi.simonwillison.net/@simon/109773800944614366

There's a risk that different runs will return different results due to the shared resource nature of GitHub Actions runners, but a good fix for that is to run comparative tests where you run the benchmark against e.g. both main and the incoming PR branch and report back on any differences.

Performance tests - I think most places don't have them as a formal gate enforced by CI. TypeScript and scalac seem to have tests that run to capture timings. The timings are included by a bot as a comment or build check, and also stored in a database so you can graph changes over time to spot regressions. Probably overkill for Datasette!

Window functions - oh, good point. Looks like Ubuntu shipped JSON1 support as far back as sqlite 3.11. I'll let this PR linger until there's a way to run against different SQLite versions. For now, I'm shipping this with datasette-ui-extras, since I think it's OK for a plugin to enforce a higher minimum requirement.

Tests - there actually did end up being test changes to capture the undercount bug of the current implementation, so the current implementation would fail against the new tests.

Perhaps a non-window function version could be written that uses random() instead of row_number() over () in order to get a unique key. It's technically not unique, but in practice, I imagine it'll work well.

I don't know how/if you do automated tests for performance, so I haven't changed any of the tests.

We don't have any performance tests yet - would be a useful thing to add, I've not built anything like that before (at least not in CI, I've always done as-hoc performance testing using something like Locust) so I don't have a great feel for how it could work.

I see not having to change the tests at all for this change as a really positive sign. If you find any behaviour differences between this and the previous that's a sign we should add a mother test or two specifying the behaviour we want.