Fixed: https://sqlite-utils.datasette.io/en/latest/reference.html

It's working again now: https://docs.datasette.io/en/latest/sql_queries.html

Actually final Datasette fix was: ```html+jinja {% block scripts %} {{ super() }}

Note

This documentation covers the development version of Datasette.

See this page for the current stable release.

{% endblock %} ```

ChatGPT prompt:

`` jQuery(function ($) { // Show banner linking to /stable/ if this is a /latest/ page if (!/\/latest\//.test(location.pathname)) { return; } var stableUrl = location.pathname.replace("/latest/", "/stable/"); // Check it's not a 404 fetch(stableUrl, { method: "HEAD" }).then((response) => { if (response.status == 200) { var warning = $(

Note

This documentation covers the development version of Datasette.

See this page for the current stable release.

` ); warning.find("a").attr("href", stableUrl); $("article[role=main]").prepend(warning); } }); });

Rewrite this to use vanilla JavaScript, no jQuery ```

Got back:

javascript document.addEventListener("DOMContentLoaded", function() { // Show banner linking to /stable/ if this is a /latest/ page if (!/\/latest\//.test(location.pathname)) { return; } var stableUrl = location.pathname.replace("/latest/", "/stable/"); // Check it's not a 404 fetch(stableUrl, { method: "HEAD" }).then((response) => { if (response.status === 200) { var warning = document.createElement("div"); warning.className = "admonition warning"; warning.innerHTML = ` <p class="first admonition-title">Note</p> <p class="last"> This documentation covers the <strong>development version</strong> of Datasette. </p> <p> See <a href="${stableUrl}">this page</a> for the current stable release. </p> `; var mainArticle = document.querySelector("article[role=main]"); mainArticle.insertBefore(warning, mainArticle.firstChild); } }); });

Still getting this error:

Weird because jQuery is present in the devtools console.

This broke on the upgrade to Furo.

I fixed it in Datasette like this: https://github.com/simonw/datasette/commit/5c1cfa451d78e3935193f5e10eba59bf741241a1 and https://github.com/simonw/datasette/commit/db8cf899e286fbaa0a40f3a9ae8d5aaa1478822e

That didn't work, because jQuery isn't yet defined that high up the template.

I should use {% block scripts %} instead: https://github.com/pradyunsg/furo/blob/05350110cf02d033ebb57147eb54e9371a63545a/src/furo/theme/furo/base.html#L89-L102

This broke when I upgraded to the Furo theme.

I just landed this PR so this feature is now in main: - #1999

Still needs documentation and maybe some extra tests too.

This still needs documentation, but now the tests are passing I'm going to merge it into main!

I hacked at the CSV stuff until it worked.

I need to clean it up though, but I can do that in this separate task: - #1101

Oh this is a bit tricky.

I have a failing test because a plugin that uses the extra_css_urls hook can't see the columns for the page.

Turns out that bit comes from here:

https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/app.py#L1203-L1217

Which assumes the context has "columns" - but that's only now available if ?_extra=columns was passed.

Actually I think I can cheat here, since it's still getting the HTML context in order to render the template.

I rebased from main. Now:

FAILED tests/test_csv.py::test_table_csv - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_cors_headers - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_no_header - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_with_labels - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_with_nullable_labels - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_blob_columns - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_download - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_stream - assert 1 == 101 FAILED tests/test_csv.py::test_csv_trace - AttributeError: 'NoneType' object has no attribute 'text' FAILED tests/test_plugins.py::test_hook_extra_css_urls[/fixtures/sortable-expected_decoded_object2] - AssertionError: assert {'added': 15,...ortable', ...} == {'added': 15,...ortable', ...} FAILED tests/test_plugins.py::test_hook_render_cell_demo - AttributeError: 'NoneType' object has no attribute 'string' FAILED tests/test_plugins.py::test_hook_render_cell_async[/fixtures/simple_primary_key] - assert b'RENDER_CELL_ASYNC_RESULT' in b'<!DOCTYPE html>\n<html>\n<head>\n <title>Error 500</title>\n <link rel="stylesheet" href="/-/static/app.css?d5...ils != detailsClickedWithin\n ).forEach(details => details.open = false);\n});\n</script>\n\n\n\n\n</b...>', ...} == {'1+1': 2, 'c... 0xXXX>', ...} FAILED tests/test_permissions.py::test_permissions_checked[/fixtures/simple_primary_key-permissions3] - assert 500 in (200, 403) FAILED tests/test_table_html.py::test_table_csv_json_export_interface - assert 500 == 200 FAILED tests/test_table_html.py::test_table_metadata - assert 500 == 200 FAILED tests/test_html.py::test_css_classes_on_body[/fixtures/simple_primary_key-expected_classes3] - assert 500 == 200 FAILED tests/test_html.py::test_templates_considered[/fixtures/simple_primary_key-table-fixtures-simple_primary_key.html, *table.html] - assert 500 == 200 FAILED tests/test_plugins.py::test_hook_table_actions[simple_view] - AssertionError: assert [] == [{'href': '/'...simple_view'}] ERROR tests/test_internals_database.py::test_execute_write_fn_connection_exception - Failed: Timeout >1.0s ERROR tests/test_internals_database.py::test_mtime_ns - Failed: Timeout >1.0s ERROR tests/test_internals_database.py::test_mtime_ns_is_none_for_memory - Failed: Timeout >1.0s ERROR tests/test_internals_database.py::test_is_mutable - Failed: Timeout >1.0s ERROR tests/test_internals_database.py::test_database_memory_name - Failed: Timeout >1.0s ERROR tests/test_internals_database.py::test_in_memory_databases_forbid_writes - Failed: Timeout >1.0s =============== 21 failed, 1275 passed, 2 skipped, 1 xfailed, 6 errors in 59.18s ================

Getting close now! Only 13 failures left, mostly relating to CSV. FAILED tests/test_csv.py::test_table_csv - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_cors_headers - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_no_header - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_with_labels - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_with_nullable_labels - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_blob_columns - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_download - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_stream - assert 1 == 101 FAILED tests/test_plugins.py::test_hook_extra_css_urls[/fixtures/sortable-expected_decoded_object2] - AssertionError: assert {'added': 15,...ortable', ...} == {'added': 15,...ortable', ...} FAILED tests/test_plugins.py::test_hook_register_facet_classes - KeyError: 'suggested_facets' FAILED tests/test_csv.py::test_csv_trace - AttributeError: 'NoneType' object has no attribute 'text' FAILED tests/test_plugins.py::test_hook_extra_body_script[/fixtures/sortable-expected_extra_body_script2] - AssertionError: assert {'added': 15,...ixtures', ...} == {'added': 15,...ixtures', ...} FAILED tests/test_plugins.py::test_hook_register_output_renderer_all_parameters - assert {'1+1': 2, 'c... 0xXXX>', ...} == {'1+1': 2, 'c... 0xXXX>', ...} =============== 13 failed, 1287 passed, 2 skipped, 1 xfailed in 61.57s (0:01:01) ================

To replace this code: https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/views/base.py#L110-L122

Maybe datasette.render_template() should optionally accept a list of templates.

https://docs.datasette.io/en/stable/internals.html#await-render-template-template-context-none-request-none - turns out it does already:

If this is a list of template file names then the first one that exists will be loaded and rendered.

It doesn't have an easy way to populate that select_templates debug template variable though.

Removed code in https://github.com/simonw/datasette/commit/538ca9d2e2d41308f5b149c934e2bc43727fd77c - it will merge when I land: - #1999

This code here:

https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/views/table.py#L117-L169

Urgh getting CSV to work is going to be so hard, because the logic for that currently lives in a huge chunk of code in BaseView which depends on the old design of the data() method:

https://github.com/simonw/datasette/blob/4bb49848697e40b8b9a1557be42b8e59eac965b3/datasette/views/base.py#L177-L343

pytest -n 8 locally says:

FAILED tests/test_canned_queries.py::test_canned_query_form_csrf_hidden_field[add_name_specify_id-True] - assert '<input type="hidden" name="csrftoken" value="' in '<!DOCTYPE html>\n<html>\n<head>\n <title>Error 500</title>\n <link rel="stylesheet" href="/-/static/app.css?d59...ils != detailsClickedWithin\n ).forEach(details => details.open = false);\n});\n</... FAILED tests/test_canned_queries.py::test_canned_query_form_csrf_hidden_field[add_name-True] - assert '<input type="hidden" name="csrftoken" value="' in '<!DOCTYPE html>\n<html>\n<head>\n <title>Error 500</title>\n <link rel="stylesheet" href="/-/static/app.css?d59...ils != detailsClickedWithin\n ).forEach(details => details.open = false);\n});\n</... FAILED tests/test_canned_queries.py::test_canned_query_permissions - AssertionError: assert 200 == 500 FAILED tests/test_canned_queries.py::test_canned_query_with_named_parameter - KeyError: 'rows' FAILED tests/test_canned_queries.py::test_canned_write_custom_template - assert 500 == 200 FAILED tests/test_canned_queries.py::test_canned_write_query_disabled_for_immutable_database - assert 500 == 200 FAILED tests/test_canned_queries.py::test_custom_params - assert '<input type="text" id="qp3" name="extra" value="foo">' in '<!DOCTYPE html>\n<html>\n<head>\n <title>Error 500</title>\n <link rel="stylesheet" href="/-/static/app.css?d59...ils != detailsClickedWithin\n ).forEach(details => details.open = false);\... FAILED tests/test_canned_queries.py::test_insert - KeyError: 'ds_csrftoken' FAILED tests/test_canned_queries.py::test_insert_error - KeyError: 'ds_csrftoken' FAILED tests/test_canned_queries.py::test_magic_parameters_csrf_json[False-True] - KeyError: 'ds_csrftoken' FAILED tests/test_canned_queries.py::test_magic_parameters_csrf_json[True-True] - KeyError: 'ds_csrftoken' FAILED tests/test_canned_queries.py::test_vary_header - KeyError: 'vary' FAILED tests/test_csv.py::test_csv_trace - AssertionError: assert 'application/...charset=utf-8' == 'text/html; charset=utf-8' FAILED tests/test_csv.py::test_table_csv - AssertionError: assert 'application/...charset=utf-8' == 'text/plain; charset=utf-8' FAILED tests/test_csv.py::test_table_csv_blob_columns - AssertionError: assert 'application/...charset=utf-8' == 'text/plain; charset=utf-8' FAILED tests/test_csv.py::test_table_csv_download - AssertionError: assert 'application/...charset=utf-8' == 'text/csv; charset=utf-8' FAILED tests/test_csv.py::test_table_csv_no_header - AssertionError: assert 'application/...charset=utf-8' == 'text/plain; charset=utf-8' FAILED tests/test_csv.py::test_table_csv_stream - assert 1 == 101 FAILED tests/test_csv.py::test_table_csv_stream_does_not_calculate_counts - AttributeError: 'NoneType' object has no attribute 'text' FAILED tests/test_csv.py::test_table_csv_stream_does_not_calculate_facets - AttributeError: 'NoneType' object has no attribute 'text' FAILED tests/test_csv.py::test_table_csv_with_labels - AssertionError: assert 'application/...charset=utf-8' == 'text/plain; charset=utf-8' FAILED tests/test_csv.py::test_table_csv_with_nullable_labels - AssertionError: assert 'application/...charset=utf-8' == 'text/plain; charset=utf-8' FAILED tests/test_facets.py::test_array_facet_handle_duplicate_tags - KeyError: 'facet_results' FAILED tests/test_facets.py::test_conflicting_facet_names_json - AssertionError: assert {'results', 'timed_out'} == {'created', '...gs', 'tags_2'} FAILED tests/test_facets.py::test_facet_size - KeyError: 'suggested_facets' FAILED tests/test_facets.py::test_json_array_with_blanks_and_nulls - KeyError: 'suggested_facets' FAILED tests/test_facets.py::test_other_types_of_facet_in_metadata - assert '<strong>created (date)\n' in '<!DOCTYPE html>\n<html>\n<head>\n <title>fixtures: facetable: 15 rows</title>\n <link rel="stylesheet" href="/-...name": "table", "request_path": "/fixtures/facetable", "added": 15, "columns": null};</script>\n\n\n\n</bo... FAILED tests/test_html.py::test_alternate_url_json[/fixtures/facetable-http://localhost/fixtures/facetable.json] - KeyError: 'link' FAILED tests/test_html.py::test_alternate_url_json[/fixtures/table~2Fwith~2Fslashes~2Ecsv-http://localhost/fixtures/table~2Fwith~2Fslashes~2Ecsv.json] - KeyError: 'link' FAILED tests/test_html.py::test_templates_considered[/fixtures/simple_primary_key-table-fixtures-simple_primary_key.html, *table.html] - assert '<!-- Templates considered: table-fixtures-simple_primary_key.html, *table.html -->' in '<!DOCTYPE html>\n<html>\n<head>\n <title>fixtures: simple_primary_key: 5 rows</title>\n <link rel="stylesheet" ...able", "request_path": "/fixtures/simple_primary... FAILED tests/test_html.py::test_templates_considered[/fixtures/table~2Fwith~2Fslashes~2Ecsv-table-fixtures-tablewithslashescsv-fa7563.html, *table.html] - assert '<!-- Templates considered: table-fixtures-tablewithslashescsv-fa7563.html, *table.html -->' in '<!DOCTYPE html>\n<html>\n<head>\n <title>fixtures: table/with/slashes.csv: 1 row</title>\n <link rel="styleshee...quest_path": "/fixtures/table~2Fwith~2Fs... FAILED tests/test_plugins.py::test_hook_extra_body_script[/fixtures/sortable-expected_extra_body_script2] - AssertionError: assert {'added': 15,...ixtures', ...} == {'added': 15,...ixtures', ...} FAILED tests/test_plugins.py::test_hook_extra_css_urls[/fixtures/sortable-expected_decoded_object2] - AssertionError: assert {'added': 15,...ortable', ...} == {'added': 15,...ortable', ...} FAILED tests/test_plugins.py::test_hook_register_facet_classes - KeyError: 'suggested_facets' FAILED tests/test_plugins.py::test_hook_register_magic_parameters - KeyError: 'ds_csrftoken' FAILED tests/test_plugins.py::test_hook_register_output_renderer_all_parameters - assert {'next': None...', ...}, ...]} == {'1+1': 2, 'c... 0xXXX>', ...} FAILED tests/test_plugins.py::test_hook_register_output_renderer_no_parameters - assert b'Hello' == b'{"ok": true... "n": null}]}' FAILED tests/test_plugins.py::test_hook_register_output_renderer_returning_broken_value - assert 200 == 500 FAILED tests/test_plugins.py::test_hook_register_output_renderer_returning_response - assert {'next': None...', ...}, ...]} == {'this_is': 'json'} === 39 failed, 1259 passed, 2 skipped, 1 xfailed in 58.07s ===

Manual testing spotted a bug.

/content/repos.json?owner=9599&_facet_array=topics - does not return a facet_results key.

/content/repos.json?owner=9599&_facet_array=topics&_facet=owner does.

``` def test_routes(routes, path, expected_class, expected_matches): match, view = resolve_routes(routes, path) if expected_class is None: assert match is None else:

      assert view.view_class.__name__ == expected_class

E AttributeError: 'function' object has no attribute 'view_class' ```

Three failures in test_html.py: FAILED tests/test_html.py::test_templates_considered[/fixtures/simple_primary_key-table-fixtures-simple_primary_key.html, *table.html] - assert '<!-- Templates considered: table-fixtures-simple_primary_key.html, *table.html -->' in '<!DOCTYPE html>\n<html>\n<head>\n <title>fixtures: simple_primary_key: 5 rows</title>\n <link rel="stylesheet" ...able", "request_path": "/fixtures/simple_primary_key", "ad... FAILED tests/test_html.py::test_templates_considered[/fixtures/table~2Fwith~2Fslashes~2Ecsv-table-fixtures-tablewithslashescsv-fa7563.html, *table.html] - assert '<!-- Templates considered: table-fixtures-tablewithslashescsv-fa7563.html, *table.html -->' in '<!DOCTYPE html>\n<html>\n<head>\n <title>fixtures: table/with/slashes.csv: 1 row</title>\n <link rel="styleshee...quest_path": "/fixtures/table~2Fwith~2Fslashes~2Ec... FAILED tests/test_html.py::test_alternate_url_json[/fixtures/facetable-http://localhost/fixtures/facetable.json] - KeyError: 'link' FAILED tests/test_html.py::test_alternate_url_json[/fixtures/table~2Fwith~2Fslashes~2Ecsv-http://localhost/fixtures/table~2Fwith~2Fslashes~2Ecsv.json] - KeyError: 'link'

In CI I'm getting this: https://github.com/simonw/datasette/actions/runs/4472331872/jobs/7858360154

====== 44 failed, 1222 passed, 1 skipped, 1 xfailed in 171.24s (0:02:51) =======

Including failures in tests/test_plugins.py and tests/test_routes.py.

Down to just these failures: FAILED tests/test_table_html.py::test_sort_links - AssertionError: assert [{'attrs': {'class': ['col-Link'], 'scope': 'col', 'data-column': 'Link', 'data-column-type': '', 'data-column-not-null': '0', 'data-is-pk': '0'}, 'a_href': None}, {'attrs': {'class': ['col-pk1'], 'scope': 'col', 'data-column': 'pk1', 'data-column-typ... FAILED tests/test_table_html.py::test_table_html_simple_primary_key - AssertionError: assert 'id\xa0▼' == 'id' FAILED tests/test_table_html.py::test_table_csv_json_export_interface - AssertionError: assert ['/fixtures/simple_primary_key.json?id__gt=2', '/fixtures/simple_primary_key.testall?id__gt=2', '/fixtures/simple_primary_key.testnone?id__gt=2', '/fixtures/simple_primary_key.testresponse?id__gt=2', '/fixtures/simple_primary_key.csv?id__gt=2&_size=ma... FAILED tests/test_table_html.py::test_csv_json_export_links_include_labels_if_foreign_keys - AssertionError: assert ['/fixtures/facetable.json?_labels=on', '/fixtures/facetable.testall?_labels=on', '/fixtures/facetable.testnone?_labels=on', '/fixtures/facetable.testresponse?_labels=on', '/fixtures/facetable.csv?_labels=on&_size=max', '#export'] == ['/', '', '#export'] FAILED tests/test_table_html.py::test_rowid_sortable_no_primary_key - AssertionError: assert 'rowid\xa0▼' == 'rowid' FAILED tests/test_table_html.py::test_table_html_filter_form_column_options[/fixtures/infinity-expected_column_options0] - AssertionError: assert ['- column -', 'rowid', 'value'] == ['- column -'] FAILED tests/test_table_html.py::test_table_html_filter_form_column_options[/fixtures/primary_key_multiple_columns-expected_column_options1] - AssertionError: assert ['- column -', 'id', 'content', 'content2'] == ['- column -'] FAILED tests/test_table_html.py::test_table_html_filter_form_column_options[/fixtures/compound_primary_key-expected_column_options2] - AssertionError: assert ['- column -', 'pk1', 'pk2', 'content'] == ['- column -'] FAILED tests/test_table_html.py::test_table_html_filter_form_still_shows_nocol_columns - AssertionError: assert ['- column -'] == ['- column -', 'pk1', 'pk2', 'content', 'sortable_with_nulls', 'sortable_with_nulls_2', 'text', 'sortable'] FAILED tests/test_table_html.py::test_metadata_sort - AssertionError: assert ['id', 'name\xa0▼'] == ['id', 'name'] FAILED tests/test_table_html.py::test_metadata_sort_desc - AssertionError: assert ['pk\xa0▲', 'name'] == ['pk', 'name'] === 11 failed, 57 passed in 7.59s ===

Breaking this down into smaller steps:

• [x] Get ?_next= working
• [x] Implement extensions - so .json is needed again for the JSON version, and anything without an extension is passed through a new code path for HTML
• [ ] That HTML view should only access JSON data, which can be seen by using .context - this will require a lot of updates to templates (it may be necessary to still provide access to some helper functions though). This will form the basis of the ambition to fully document the template context.
• [ ] Get a bunch of the existing table HTML and JSON tests to pass
• [ ] Use those tests to refactor the nasty _next code, see https://github.com/simonw/datasette/pull/1999#issuecomment-1460905469
• [ ] Figure out how the register_output_renderer(datasette) plugin hook should work

Next big challenge: fix all the tests in test_table_html.py:

FAILED tests/test_table_html.py::test_add_filter_redirects - assert 500 == 302 FAILED tests/test_table_html.py::test_existing_filter_redirects - assert 500 == 302 FAILED tests/test_table_html.py::test_reflected_hidden_form_fields[_facet=_neighborhood-expected_hidden0] - AssertionError: assert {} == {'_facet': '_neighborhood'} FAILED tests/test_table_html.py::test_reflected_hidden_form_fields[_where=1+=+1&_col=_city_id-expected_hidden1] - AssertionError: assert {} == {'_col': '_ci...ere': '1 = 1'} FAILED tests/test_table_html.py::test_reflected_hidden_form_fields[_facet=_neighborhood&_neighborhood__exact=Downtown-expected_hidden2] - AssertionError: assert {} == {'_facet': '_neighborhood'} FAILED tests/test_table_html.py::test_reflected_hidden_form_fields[_facet=_neighborhood&_city_id__gt=1-expected_hidden3] - AssertionError: assert {} == {'_facet': '_neighborhood'} FAILED tests/test_table_html.py::test_empty_search_parameter_gets_removed - assert 500 == 302 FAILED tests/test_table_html.py::test_searchable_view_persists_fts_table - AssertionError: assert [('_fts_table...ts_pk', 'pk')] == [] FAILED tests/test_table_html.py::test_sort_by_desc_redirects - assert 500 == 302 FAILED tests/test_table_html.py::test_sort_links - AssertionError: assert [{'a_href': N...', ...}}, ...] == [{'a_href': N...', ...}}, ...] FAILED tests/test_table_html.py::test_facets_persist_through_filter_form - AssertionError: assert [] == [('_facet', '...ray', 'tags')] FAILED tests/test_table_html.py::test_next_does_not_persist_in_hidden_field - AssertionError: assert [] == [('_size', '1')] FAILED tests/test_table_html.py::test_table_html_simple_primary_key - AttributeError: 'NoneType' object has no attribute 'string' FAILED tests/test_table_html.py::test_table_csv_json_export_interface - AssertionError: assert ['/fixtures/s...x', '#export'] == ['/', '', '#export'] FAILED tests/test_table_html.py::test_csv_json_export_links_include_labels_if_foreign_keys - AssertionError: assert ['/fixtures/f...x', '#export'] == ['/', '', '#export'] FAILED tests/test_table_html.py::test_table_html_no_primary_key - AssertionError: assert ['content', 'a', 'b', 'c'] == ['a', 'b', 'c'] FAILED tests/test_table_html.py::test_rowid_sortable_no_primary_key - AttributeError: 'NoneType' object has no attribute 'string' FAILED tests/test_table_html.py::test_table_html_compound_primary_key - AssertionError: assert 'Link' == 'pk1' FAILED tests/test_table_html.py::test_table_html_foreign_key_links - assert [['<td class=...">\xa0</td>']] == [['<td class=...">\xa0</td>']] FAILED tests/test_table_html.py::test_table_html_disable_foreign_key_links_with_labels - assert [['<td class=...str">b</td>']] == [['<td class=...str">b</td>']] FAILED tests/test_table_html.py::test_table_html_foreign_key_custom_label_column - assert [['<td class=...1</em></td>']] == [['<td class=...1</em></td>']] FAILED tests/test_table_html.py::test_table_html_filter_form_column_options[/fixtures/infinity-expected_column_options0] - AssertionError: assert ['- column -'...wid', 'value'] == ['- column -'] FAILED tests/test_table_html.py::test_table_html_filter_form_column_options[/fixtures/primary_key_multiple_columns-expected_column_options1] - AssertionError: assert ['- column -'...', 'content2'] == ['- column -'] FAILED tests/test_table_html.py::test_table_html_filter_form_column_options[/fixtures/compound_primary_key-expected_column_options2] - AssertionError: assert ['- column -'...2', 'content'] == ['- column -'] FAILED tests/test_table_html.py::test_table_html_filter_form_still_shows_nocol_columns - AssertionError: assert ['- column -'] == ['- column -'...nulls_2', ...] FAILED tests/test_table_html.py::test_compound_primary_key_with_foreign_key_references - assert [['<td class=...ne</a></td>']] == [['<td class=...ne</a></td>']] FAILED tests/test_table_html.py::test_view_html - assert None is not None FAILED tests/test_table_html.py::test_extra_where_clauses - assert [('_where', "...'_city_id=1')] == [] FAILED tests/test_table_html.py::test_other_hidden_form_fields[/fixtures/facetable?_size=10-expected_hidden0] - AssertionError: assert [] == [('_size', '10')] FAILED tests/test_table_html.py::test_other_hidden_form_fields[/fixtures/facetable?_size=10&_ignore=1&_ignore=2-expected_hidden1] - AssertionError: assert [] == [('_size', '1...ignore', '2')] FAILED tests/test_table_html.py::test_search_and_sort_fields_not_duplicated[/fixtures/searchable?_sort=text2&_where=1-expected_hidden3] - AssertionError: assert [] == [('_where', '1')] FAILED tests/test_table_html.py::test_binary_data_display_in_table - assert [['<td class=...">\xa0</td>']] == [['<td class=...">\xa0</td>']] FAILED tests/test_table_html.py::test_metadata_sort - AttributeError: 'NoneType' object has no attribute 'string' FAILED tests/test_table_html.py::test_metadata_sort_desc - AttributeError: 'NoneType' object has no attribute 'string' FAILED tests/test_table_html.py::test_facet_more_links[5-/fixtures/facetable?_facet=_neighborhood-2-True-/fixtures/facetable?_facet=_neighborhood&_facet_size=max] - assert 0 == 2 FAILED tests/test_table_html.py::test_facet_more_links[5-/fixtures/facetable?_facet=_neighborhood&_facet_size=50-5-True-/fixtures/facetable?_facet=_neighborhood&_facet_size=max] - assert 0 == 5 FAILED tests/test_table_html.py::test_facet_total - assert 500 == 200

Deduped that's 30 tests:

FAILED tests/test_table_html.py::test_add_filter_redirects FAILED tests/test_table_html.py::test_binary_data_display_in_table FAILED tests/test_table_html.py::test_compound_primary_key_with_foreign_key_references FAILED tests/test_table_html.py::test_csv_json_export_links_include_labels_if_foreign_keys FAILED tests/test_table_html.py::test_empty_search_parameter_gets_removed FAILED tests/test_table_html.py::test_existing_filter_redirects FAILED tests/test_table_html.py::test_extra_where_clauses FAILED tests/test_table_html.py::test_facet_more_links FAILED tests/test_table_html.py::test_facet_total FAILED tests/test_table_html.py::test_facets_persist_through_filter_form FAILED tests/test_table_html.py::test_metadata_sort FAILED tests/test_table_html.py::test_metadata_sort_desc FAILED tests/test_table_html.py::test_next_does_not_persist_in_hidden_field FAILED tests/test_table_html.py::test_other_hidden_form_fields FAILED tests/test_table_html.py::test_reflected_hidden_form_fields FAILED tests/test_table_html.py::test_rowid_sortable_no_primary_key FAILED tests/test_table_html.py::test_search_and_sort_fields_not_duplicated FAILED tests/test_table_html.py::test_searchable_view_persists_fts_table FAILED tests/test_table_html.py::test_sort_by_desc_redirects FAILED tests/test_table_html.py::test_sort_links FAILED tests/test_table_html.py::test_table_csv_json_export_interface FAILED tests/test_table_html.py::test_table_html_compound_primary_key FAILED tests/test_table_html.py::test_table_html_disable_foreign_key_links_with_labels FAILED tests/test_table_html.py::test_table_html_filter_form_column_options FAILED tests/test_table_html.py::test_table_html_filter_form_still_shows_nocol_columns FAILED tests/test_table_html.py::test_table_html_foreign_key_custom_label_column FAILED tests/test_table_html.py::test_table_html_foreign_key_links FAILED tests/test_table_html.py::test_table_html_no_primary_key FAILED tests/test_table_html.py::test_table_html_simple_primary_key FAILED tests/test_table_html.py::test_view_html

I had to replicate quite a bit of this logic from base.py:

https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/views/base.py#L526-L544

I should refactor this when I move the canned / arbitrary query views away from that base class too.

test_paginate_using_link_header will be tricky.

The reason the tests are failing is that json_renderer() attempts to populate the link header using data["next_url"] - but that's not present unless ?_extra=next_url has been passed:

https://github.com/simonw/datasette/blob/2f38479dcc81f11a4362f4e28511fa88afc34e61/datasette/renderer.py#L101-L102

But I can only rely on next being present, not next_url.

I thought I could maybe assemble the link header using next, by turning that into a next_url link - but there's some custom logic which kicks in for pagination against SQL views (offset/limit pagination) to calculate the next_url which isn't easily replicable at the json_renderer() layer, in the _next_value_and_url() utility function:

https://github.com/simonw/datasette/blob/2f38479dcc81f11a4362f4e28511fa88afc34e61/datasette/views/table.py#L2275-L2282

Now 25 failures in test_table_api.py: FAILED tests/test_table_api.py::test_expand_labels - assert {'2': {'pk': 2, 'created': '2019-01-14 08:00:00', 'planet_int': 1, 'on_earth': 1, 'state': 'CA', '_city_id': 1, '_neighborhood': 'Dogpatch', 'tags': '["tag1", "tag3"]', 'complex_array'... FAILED tests/test_table_api.py::test_expand_label - AssertionError: assert {'1': {'pk': '1', 'foreign_key_with_label': '1', 'foreign_key_with_blank_label': '3', 'foreign_key_with_no_label': '1', 'foreign_key_compound_pk1': 'a', 'foreign_key_co... FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json-max-age=5] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=invalid-max-age=5] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=10-max-age=10] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=0-no-cache] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_infinity_returned_as_null - AssertionError: assert [{'rowid': 1, 'value': inf}, {'rowid': 2, 'value': -inf}, {'rowid': 3, 'value': 1.5}] == [{'rowid': 1, 'value': None}, {'rowid': 2, 'value': None}, {'rowid': 3, 'value'... FAILED tests/test_table_api.py::test_null_and_compound_foreign_keys_are_not_expanded - AssertionError: assert [{'pk': '1', 'foreign_key_with_label': '1', 'foreign_key_with_blank_label': '3', 'foreign_key_with_no_label': '1', 'foreign_key_compound_pk1': 'a', 'foreign_key_compoun... FAILED tests/test_table_api.py::test_binary_data_in_json[/fixtures/binary_data.json?_shape=array-expected_json0-None] - assert [{'rowid': 1, 'data': "b'\\x15\\x1c\\x02\\xc7\\xad\\x05\\xfe'"}, {'rowid': 2, 'data': "b'\\x15\\x1c\\x03\\xc7\\xad\\x05\\xfe'"}, {'rowid': 3, 'data': None}] == [{'rowid': 1, 'data': {'... FAILED tests/test_table_api.py::test_binary_data_in_json[/fixtures/binary_data.json?_shape=array&_nl=on-None-{"rowid": 1, "data": {"$base64": true, "encoded": "FRwCx60F/g=="}}\n{"rowid": 2, "data": {"$base64": true, "encoded": "FRwDx60F/g=="}}\n{"rowid": 3, "data": null}] - assert '{"ok": false, "error": "Object of type bytes is not JSON serializable", "status": 500, "title": null}' == '{"rowid": 1, "data": {"$base64": true, "encoded": "FRwCx60F/g=="}}\n{"rowid"... FAILED tests/test_table_api.py::test_paginate_using_link_header[] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=arrays] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=arrayfirst] - assert 400 == 200 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=object] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=objects] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=array] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=array&_nl=on] - assert 1 == 21 FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=created-expected_columns0] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_nocol=created-expected_columns1] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=created-expected_columns2] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=state-expected_columns3] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=created&_nocol=created-expected_columns4] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_nocol=state&_facet=state-expected_columns5] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/simple_view.json?_nocol=content-expected_columns6] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/simple_view.json?_col=content-expected_columns7] - KeyError: 'columns' ============================================================================= 25 failed, 86 passed, 1 xfailed in 7.18s =============================================================================

Here are the next set of tests to get passing: % pytest tests/test_table_api.py FAILED tests/test_table_api.py::test_facets[/fixtures/facetable.json?_facet=state&_facet=_city_id-expected_facet_results0] - KeyError: 'name' FAILED tests/test_table_api.py::test_facets[/fixtures/facetable.json?_facet=state&_facet=_city_id&state=MI-expected_facet_results1] - KeyError: 'name' FAILED tests/test_table_api.py::test_facets[/fixtures/facetable.json?_facet=planet_int-expected_facet_results2] - KeyError: 'name' FAILED tests/test_table_api.py::test_facets[/fixtures/facetable.json?_facet=planet_int&planet_int=1-expected_facet_results3] - KeyError: 'name' FAILED tests/test_table_api.py::test_suggested_facets - KeyError: 'suggested_facets' FAILED tests/test_table_api.py::test_allow_facet_off - KeyError: 'suggested_facets' FAILED tests/test_table_api.py::test_suggest_facets_off - KeyError: 'suggested_facets' FAILED tests/test_table_api.py::test_nofacet[True] - KeyError: 'suggested_facets' FAILED tests/test_table_api.py::test_nofacet[False] - KeyError: 'suggested_facets' FAILED tests/test_table_api.py::test_nosuggest[True] - KeyError: 'suggested_facets' FAILED tests/test_table_api.py::test_nosuggest[False] - KeyError: 'suggested_facets' FAILED tests/test_table_api.py::test_nocount[True-None] - KeyError: 'count' FAILED tests/test_table_api.py::test_nocount[False-15] - KeyError: 'count' FAILED tests/test_table_api.py::test_expand_labels - AssertionError: assert {'13': {'_cit...:00:00', ...}} == {'13': {'_cit...:00:00', ...}} FAILED tests/test_table_api.py::test_expand_label - AssertionError: assert {'1': {'forei...l': '1', ...}} == {'1': {'forei...': '1'}, ...}} FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json-max-age=5] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=invalid-max-age=5] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=10-max-age=10] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=0-no-cache] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_infinity_returned_as_null - AssertionError: assert [{'rowid': 1,...'value': 1.5}] == [{'rowid': 1,...'value': 1.5}] FAILED tests/test_table_api.py::test_null_and_compound_foreign_keys_are_not_expanded - AssertionError: assert [{'foreign_ke...': None, ...}] == [{'foreign_ke...': None, ...}] FAILED tests/test_table_api.py::test_binary_data_in_json[/fixtures/binary_data.json?_shape=array-expected_json0-None] - assert [{'data': "b'..., 'rowid': 3}] == [{'data': {'$..., 'rowid': 3}] FAILED tests/test_table_api.py::test_binary_data_in_json[/fixtures/binary_data.json?_shape=array&_nl=on-None-{"rowid": 1, "data": {"$base64": true, "encoded": "FRwCx60F/g=="}}\n{"rowid": 2, "data": {"$base64": true, "encoded": "FRwDx60F/g=="}}\n{"rowid": 3, "data": null}] - assert '{"ok": false...title": null}' == '{"rowid": 1,..."data": null}' FAILED tests/test_table_api.py::test_paginate_using_link_header[] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=arrays] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=arrayfirst] - assert 400 == 200 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=object] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=objects] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=array] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=array&_nl=on] - assert 1 == 21 FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=created-expected_columns0] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_nocol=created-expected_columns1] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=created-expected_columns2] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=state-expected_columns3] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=created&_nocol=created-expected_columns4] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_nocol=state&_facet=state-expected_columns5] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/simple_view.json?_nocol=content-expected_columns6] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/simple_view.json?_col=content-expected_columns7] - KeyError: 'columns' ============================================================================= 38 failed, 73 passed, 1 xfailed in 7.25s =============================================================================

Upgraded to Sphinx 6 locally and got the same error:

% just docs Cogging README.md Cogging docs/changelog.rst Cogging docs/cli-reference.rst Cogging docs/cli.rst Cogging docs/contributing.rst Cogging docs/index.rst Cogging docs/installation.rst Cogging docs/python-api.rst Cogging docs/reference.rst sphinx-autobuild -a -b html "." "_build" --watch ../sqlite_utils [sphinx-autobuild] > sphinx-build -b html -a /Users/simon/Dropbox/Development/sqlite-utils/docs /Users/simon/Dropbox/Development/sqlite-utils/docs/_build Running Sphinx v6.1.3 loading pickled environment... failed failed: Can't get attribute '_stable_repr_object' on <sphinx.deprecation._ModuleWrapper object at 0x103f155d0> building [mo]: all of 0 po files writing output... building [html]: all source files updating environment: [new config] 8 added, 0 changed, 0 removed reading sources... [ 12%] changelog Exception occurred: File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.11/site-packages/sphinx/ext/extlinks.py", line 103, in role title = caption % part ~~~~~~~~^~~~~~ TypeError: not all arguments converted during string formatting The full traceback has been saved in /var/folders/x6/31xf1vxj0nn9mxqq8z0mmcfw0000gn/T/sphinx-err-1ey36c1n.log, if you want to report the issue to the developers.

Could be the same problem as: - https://github.com/simonw/datasette/issues/1972

Which I fixed in https://github.com/simonw/datasette/commit/3af313e165215696af899e772f47bf7c27873ae3

Aah, I think I see why you wrote it like that.

The problem is that init_spatialite() does other stuff too:

https://github.com/simonw/sqlite-utils/blob/fc221f9b62ed8624b1d2098e564f525c84497969/sqlite_utils/db.py#L1161-L1171

So it needs to be able to load the SpatiaLite extension from the correct place, and THEN run select InitSpatialMetadata() to configure the database, if needed.

So the problem you're trying to solve here is to let people optionally pass in the path to SpatiaLite if it's not one of the ones that are searched by default.

Idea for if this change ends up making a bunch of breaking changes to the templates (which I think it should) - I can generate a GitHub diff link between the old and new templates and include that link in the 1.0 upgrade documentation to help people who wrote custom templates see what they might need to change - with minimal effort from myself to document those changes.

Here's an example of something that would break:

https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/templates/database.html#L12-L15

Because metadata is an empty dictionary sometimes, so {{ metadata.title or database }} would raise an error and need to be replaced by {{ metadata.get("title") or database }}.

I'm torn on this. It's useful for me right now for refactoring, but I feel like it should be a permanent thing, not a setting - or it should default to on and people should turn it off, but who would ever do that?

Renderers have an impact on three different pages: query results, table page and row page.

The row page feature is incomplete though:

https://congress-legislators.datasettes.com/legislators/social_media/A000055

Why is there no .csv link there?

I also need to figure out the renderers stuff, so I can link to the right URLs for CSV and JSON and other formats:

https://github.com/simonw/datasette/blob/6d07a7da1531cd749844fc6827d9a1e57009b2ea/datasette/views/base.py#L474-L518

Ironically the thing I most need right now is comprehensive documentation of what variables are being passed to the templates!

One big challenge is that I need to untangle the template context that happens in BaseView - I'm hacking that together at the moment, but I need a real answer for how that should work in a world in which view functions aren't using that base class.

https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/views/base.py#L110-L145

Now at 34 failed, 34 passed.

I need to figure out what to do about extra_context_from_filters - which was previously passed straight to the HTML context.

https://github.com/simonw/datasette/blob/11f7feb7a3f7166c71389786880863d60ed3d165/datasette/views/table.py#L406-L422

Found a neat trick: ```diff diff --git a/datasette/app.py b/datasette/app.py index 186f192d..40416713 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -23,7 +23,13 @@ from pathlib import Path

from markupsafe import Markup, escape from itsdangerous import URLSafeSerializer -from jinja2 import ChoiceLoader, Environment, FileSystemLoader, PrefixLoader +from jinja2 import ( + ChoiceLoader, + Environment, + FileSystemLoader, + PrefixLoader, + StrictUndefined, +) from jinja2.environment import Template from jinja2.exceptions import TemplateNotFound

@@ -394,7 +400,10 @@ class Datasette: ] ) self.jinja_env = Environment( - loader=template_loader, autoescape=True, enable_async=True + loader=template_loader, + autoescape=True, + enable_async=True, + undefined=StrictUndefined, ) self.jinja_env.filters["escape_css_string"] = escape_css_string self.jinja_env.filters["quote_plus"] = urllib.parse.quote_plus ``` This causes Jinja to raise a hard error if there are any variables referenced in the template that are not available in the context.

It's helping me spot things that are still missing, rather than just relying on failed unit tests.

I improved the readability by removing some unnecessary table aliases: sql with recursive nested_folders(folder_id, descendant_folder_id) as ( -- base case: select all immediate children of the root folder select id, id from folders where parent is null union all -- recursive case: select all children of the previous level of nested folders select nested_folders.folder_id, folders.id from nested_folders join folders on nested_folders.descendant_folder_id = folders.parent ) -- Find notes within all descendants of folder 1 select * from notes where folder in ( select descendant_folder_id from nested_folders where folder_id = 1 );

Here's a query that returns all notes in folder 1, including notes in descendant folders: sql with recursive nested_folders(folder_id, descendant_folder_id) as ( -- base case: select all immediate children of the root folder select id, id from folders where parent is null union all -- recursive case: select all children of the previous level of nested folders select nf.folder_id, f.id from nested_folders nf join folders f on nf.descendant_folder_id = f.parent ) -- Find notes within all descendants of folder 1 select * from notes where folder in ( select descendant_folder_id from nested_folders where folder_id = 1 ); With assistance from ChatGPT. Prompts were:

``` SQLite schema:

CREATE TABLE [folders] ( [id] INTEGER PRIMARY KEY, [long_id] TEXT, [name] TEXT, [parent] INTEGER, FOREIGN KEY([parent]) REFERENCES folders );

Write a recursive CTE that returns the following:

folder_id | descendant_folder_id

With a row for every nested child of every folder - so the top level folder has lots of rows Then I tweaked it a bit, then ran this: WITH RECURSIVE nested_folders(folder_id, descendant_folder_id) AS ( -- base case: select all immediate children of the root folder SELECT id, id FROM folders WHERE parent IS NULL UNION ALL -- recursive case: select all children of the previous level of nested folders SELECT nf.folder_id, f.id FROM nested_folders nf JOIN folders f ON nf.descendant_folder_id = f.parent ) -- select all rows from the recursive CTE SELECT * from notes where folder in (select descendant_folder_id FROM nested_folders where folder_id = 1)

Convert all SQL keywords to lower case, and re-indent ```

Here's what the CTE from that looks like:

I've implemented the test (thanks for pointing me in the right direction!).

At tmcl-it/datasette:0.64.1+row-view-expand-labels I also have a variant of this patch that applies to the 0.64.x branch. Please let me know if you'd be interested in merging that as well and I'll open another PR.

Sure, let's merge that one too - it can go out in the next 0.64.x series release (maybe even a 0.65).

Good catch, thanks.

My folders table will have:

• id - rowid
• long_id - that long unique string ID
• name - the name
• parent - foreign key to id

Improved script: zsh osascript -e 'tell application "Notes" set allFolders to folders repeat with aFolder in allFolders set folderId to id of aFolder set folderName to name of aFolder set folderContainer to container of aFolder if class of folderContainer is folder then set folderContainerId to id of folderContainer else set folderContainerId to "" end if log "ID: " & folderId log "Name: " & folderName log "Container: " & folderContainerId log " " end repeat end tell ' ``` ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6113 Name: Blog posts Container:

ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p698 Name: JSK Container:

ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7995 Name: Nested inside blog posts Container: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6113

ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p3526 Name: New Folder Container:

ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p3839 Name: New Folder 1 Container:

ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p2 Name: Notes Container:

ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6059 Name: Quick Notes Container:

ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7283 Name: UK Christmas 2022 Container: `` I filtered out things where the parent was an account and not a folder usingif class of folderContainer is folder then`.

Created through several rounds with ChatGPT (including hints like "rewrite that using setdefault()"): ```python def topological_sort(nodes): children = {} for node in nodes: parent_id = node["parent"] if parent_id is not None: children.setdefault(parent_id, []).append(node)

def traverse(node, result):
    result.append(node)
    if node["id"] in children:
        for child in children[node["id"]]:
            traverse(child, result)

sorted_data = []

for node in nodes:
    if node["parent"] is None:
        traverse(node, sorted_data)

return sorted_data

```

I used ChatGPT to write this: osascript -e 'tell application "Notes" set allFolders to folders repeat with aFolder in allFolders set folderId to id of aFolder set folderName to name of aFolder set folderContainer to container of aFolder set folderContainerName to name of folderContainer log "Folder ID: " & folderId log "Folder Name: " & folderName log "Folder Container: " & folderContainerName log " " --check for nested folders if count of folders of aFolder > 0 then set nestedFolders to folders of aFolder repeat with aNestedFolder in nestedFolders set nestedFolderId to id of aNestedFolder set nestedFolderName to name of aNestedFolder set nestedFolderContainer to container of aNestedFolder set nestedFolderContainerName to name of nestedFolderContainer log " Nested Folder ID: " & nestedFolderId log " Nested Folder Name: " & nestedFolderName log " Nested Folder Container: " & nestedFolderContainerName log " " end repeat end if end repeat end tell ' Which for my account output this: ``` Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6113 Folder Name: Blog posts Folder Container: iCloud

Nested Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7995
Nested Folder Name: Nested inside blog posts
Nested Folder Container: Blog posts

Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p698 Folder Name: JSK Folder Container: iCloud

Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7995 Folder Name: Nested inside blog posts Folder Container: Blog posts

Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p3526 Folder Name: New Folder Folder Container: iCloud

Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p3839 Folder Name: New Folder 1 Folder Container: iCloud

Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p2 Folder Name: Notes Folder Container: iCloud

Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6059 Folder Name: Quick Notes Folder Container: iCloud

Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7283 Folder Name: UK Christmas 2022 Folder Container: iCloud ``` So I think the correct approach here is to run code at the start to list all of the folders (no need to do fancy recursion though, just a flat list with the parent containers is enough) and create a model of that hierarchy in SQLite.

Then when I import notes I can foreign key reference them back to their containing folder.

I'm tempted to use rowid for the foreign keys because the official IDs are pretty long.

So it looks like folders can be hierarchical?

From the Script Editor library docs:

A note has a:

• container (folder), r/o) : the folder of the note

Here's what a folder looks like:

folder n : a folder containing notes elements:

• contains folders, notes; contained by application, accounts, folders.

properties:

• name (text) : the name of the folder
• id (text, r/o) : the unique identifier of the folder
• shared (boolean, r/o) : Is the folder shared?
• container (account or folder, r/o) : the container of the folder

In terms of the UI: I'm tempted to say that the default behaviour is for it to run until it sees a note that it already knows about AND that has matching update/created dates, and then stop.

You can do a full import again ignoring that logic with apple-notes-to-sqlite notes.db --full.

It looks like the iteration order is most-recently-modified-first - I tried editing a note a bit further back in my notes app and it was the first one output by apple-notes-to-sqlite --dump.

OK, this works!

It doesn't have tests yet. I guess I'll need to mock subprocess to test this.

Before that conversion:

Monday, March 6, 2023 at 11:55:15 AM

After:

2023-03-06T11:55:15

Converting AppleScript date strings to ISO format is hard!

https://forum.latenightsw.com/t/formatting-dates/841 has a recipe I'll try:

set todayISO to (todayDate as «class isot» as string)

Not clear to me how timezones work here. I'm going to ignore them for the moment.

My prototype showed that images embedded in notes come out in the HTML export as bas64 image URLs, which is neat.

I opened the "Script Editor" app on my computer, used Window -> Library to open the Library panel, then clicked on the Notes app there. I got this:

So the notes object has these properties:

• name (text) : the name of the note (normally the first line of the body)
• id (text, r/o) : the unique identifier of the note
• container (folder, r/o) : the folder of the note
• body (text) : the HTML content of the note
• plaintext (text, r/o) : the plaintext content of the note
• creation date (date, r/o) : the creation date of the note
• modification date (date, r/o) : the modification date of the note
• password protected (boolean, r/o) : Is the note password protected?
• shared (boolean, r/o) : Is the note shared?

I'm going to ignore the concept of attachments for the moment.

I think the AppleScript I want to pass to osascript looks like this: applescript tell application "Notes" repeat with eachNote in every note set noteId to the id of eachNote set noteTitle to the name of eachNote set noteBody to the body of eachNote log "------------------------" & "\n" log noteId & "\n" log noteTitle & "\n\n" log noteBody & "\n" end repeat end tell But there are a few more properties I'd like to get - created and updated date for example.

This did the job! Next step is to turn that into a Python script.

After a few more rounds I got to this script, which outputs them to a /tmp/notes.txt file:

```zsh

!/bin/zsh

osascript -e ' set notesFile to "/tmp/notes.txt" set fileRef to open for access notesFile with write permission tell application "Notes" repeat with eachNote in every note set noteId to the id of eachNote set noteTitle to the name of eachNote set noteBody to the body of eachNote write "------------------------" & "\n" to fileRef write noteId & "\n" to fileRef write noteTitle & "\n\n" to fileRef write noteBody & "\n" to fileRef end repeat end tell close access fileRef' Then I wrote this little Python script to load them into a database:python import sqlite_utils split = b"------------------------\n" s = open("/tmp/notes.txt", "rb").read() notes = [n.decode("mac_roman") for n in s.split(split) if n]

cleaned_notes = [{ "id": n.split("\n")[0], "title": n.split("\n")[1], "body": "\n".join(n.split("\n")[2:]).strip() } for n in notes]

db = sqlite_utils.Database("/tmp/notes.db") db["notes"].insert_all(cleaned_notes) ```

Me:

Write applescript to loop through all of my Apple Notes and output their contents

ChatGPT:

applescript tell application "Notes" repeat with eachNote in every note set noteTitle to the name of eachNote set noteBody to the body of eachNote display dialog noteTitle & return & return & noteBody end repeat end tell Me:

How can I turn that into a zsh script which will output the results to my terminal

ChatGPT: ```zsh

!/usr/bin/zsh

osascript -e 'tell application "Notes" repeat with eachNote in every note set noteTitle to the name of eachNote set noteBody to the body of eachNote set output to noteTitle & "\n" & noteBody & "\n" display dialog output log output end repeat end tell' `` I had to change that first line to#!/bin/zsh` - then when I ran it it spammed my screen with dialog boxes.

Me:

I want to log the notes contents to the terminal, not display it in a dialog

ChatGPT:

```zsh

!/bin/zsh

osascript -e 'tell application "Notes" repeat with eachNote in every note set noteTitle to the name of eachNote set noteBody to the body of eachNote set output to noteTitle & "\n" & noteBody & "\n" log output end repeat end tell' ``` This really helps!

Just ran into a no such table: columns error - which I think is because my new view code sometimes bypasses calling this method (currently done in BaseView.dispatch_request()):

https://github.com/simonw/datasette/blob/96e94f9b7b2db53865e61390bcce6761727f26d8/datasette/views/base.py#L101-L103

Or... I could temporarily build a quick additional CannedQueryView subclass that just does the necessary bits to get the existing code to work. I'm going to try that.

Solving this is proving difficult: https://github.com/simonw/datasette/blob/96e94f9b7b2db53865e61390bcce6761727f26d8/datasette/views/table.py#L1500-L1503

The problem is that calling .data() on QueryView only works here because we expect to ourselves be inside a .data() method, with all of the existing magic that knows how to render things that are returned by that.

So I may need to substantially re-engineer how QueryView works in order to get this to work.

127.0.0.1:8001/fixtures/neighborhood_search fails because the forwarding to a canned query does not yet work.

pytest tests/test_table_html.py

Currently 44 failed, 24 passed in 7.53s

Failures here: https://gist.github.com/simonw/df0a52cd7d820b776dc3dfc50e7cb778

I managed to get HTML view working! I did it by continuing to add more things to the extras and the _html bundle until the page loaded for me:

```diff async def extra_extras(): "Available ?_extra= blocks" return { @@ -1981,6 +2053,14 @@ async def extra_extras(): "query", "display_columns", "display_rows", + "database", + "table", + "database_color", + "table_actions", + "filters", + "renderers", + "custom_table_templates", + "sorted_facet_results", ] }

@@ -2006,6 +2086,14 @@ async def extra_extras(): extra_query, extra_metadata, extra_extras, + extra_database, + extra_table, + extra_database_color, + extra_table_actions, + extra_filters, + extra_renderers, + extra_custom_table_templates, + extra_sorted_facet_results, ) `` I'll probably refactor this into something cleaner, and maybe but a bunch of them in a"html"dictionary and update the templates to use{{ html.filters }}` or similar. Will look at that once the tests are passing.

This feels quite nice:

I want to package together all of the extras that are needed for the HTML format. A few options for doing that:

• Introduce ?_extra=_html where the leading underscore indicates that this is a "bundle" of extras, then define a bundle that's everything needed for the HTML renderer
• Have some other mechanism whereby different renderers can request a bundle of extras.

I'm leaning towards the first option. I'll try that and see what it looks like.

To get this unblocked, I'm going to allow myself to pass non-JSON-serializable objects to the HTML template version of things. If I can get that working (and get the existing tests to pass) I can consider a later change that makes those JSON serializable - or admit that it's OK for the templates to have non-JSON data passed to them and figure out how best to document those variables independently from the JSON documentation.

The reason for that Row thing is that it allows custom templates that do things like this:

https://docs.datasette.io/en/stable/changelog.html#easier-custom-templates-for-table-rows

html+jinja {% for row in display_rows %} <div> <h2>{{ row["title"] }}</h2> <p>{{ row["description"] }}<lp> <p>Category: {{ row.display("category_id") }}</p> </div> {% endfor %} Is that a good design? the .display() thing feels weird - I wonder if anyone has ever actually used that.

It's documented here: https://docs.datasette.io/en/0.64.2/custom_templates.html#custom-templates

If you want to output the rendered HTML version of a column, including any links to foreign keys, you can use {{ row.display("column_name") }}.

I can't see any examples of anyone using it in this code search: https://cs.github.com/?scopeName=All+repos&scope=&q=datasette+row.display

It is however useful to have some kind of abstraction layer here that insulates the SQLite Row object, since having an extra layer will help if Datasette ever grows support for alternative database backends such as DuckDB or PostgreSQL.

Aside idea: it might be interesting if there were "lazy" template variables available in the context: things that are not actually executed unless a template author requests them.

Imagine if metadata was a lazy template reference, such that custom templates that don't display any metadata don't trigger it to be resolved (which might involve additional database queries some day).

Figuring out what to do with display_columns_and_rows() is hard. That returns rows as this special kind of object, which is designed to be accessed from the HTML templates:

https://github.com/simonw/datasette/blob/96e94f9b7b2db53865e61390bcce6761727f26d8/datasette/views/table.py#L45-L71

For the HTML version, I need to decide where all of the stuff that happens in async def extra_template() is going to live.

I think it's another one of those extra functions, triggered for ?_extra=context.

https://github.com/simonw/datasette/blob/96e94f9b7b2db53865e61390bcce6761727f26d8/datasette/views/table.py#L813-L912

The ease with which I added that ?_extra=query feature in https://github.com/simonw/datasette/pull/1999/commits/96e94f9b7b2db53865e61390bcce6761727f26d8 made me feel really confident that this architecture is going in the right direction.

```diff diff --git a/datasette/views/table.py b/datasette/views/table.py index 8d3bb2c930..3e1db9c85f 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -1913,6 +1913,13 @@ async def extra_request(): "args": request.args._data, }

• async def extra_query():
• "Details of the underlying SQL query"
• return {
• "sql": sql,
• "params": params,
• } + async def extra_extras(): "Available ?_extra= blocks" return { @@ -1938,6 +1945,7 @@ async def extra_extras(): extra_primary_keys, extra_debug, extra_request,
• extra_query, extra_extras, ) ```

Just noticed that _json=colname is not working, and that's because it's handled by the renderer here:

https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/renderer.py#L29-L40

But that's not currently being called by my new code.

I'm going to hold off on that refactor until later, when I have tests to show me if the refactor works or not.

So maybe I can refactor it to look a bit more like this:

https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L1602-L1604

One thing that's useful here is that is_view is handled early, like this:

https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L466-L472

So if I omit the is_view bit from the extracted function I can simplify more.

I'd really like to extract this ugly logic out into a helper function:

https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L1617-L1684

I copied it in and asked ChatGPT to "List all of the variable that are either modified or assigned to by the above code":

Modified variables:

offset
where_clauses
params
order_by

Assigned variables:

sort_value
next_by_pk_clauses
order_by (when sort or sort_desc is not None)

Then I asked which variables were used as inputs, and argued with it a bit about whether it should be counting functions. Eventually got to this:

My apologies for the oversight. Here are the variables needed as input by the above code, excluding any functions:

_next
is_view
sort
sort_desc
use_rowid
pks
order_by

Note that use_rowid is actually defined earlier in terms of two of those other variables: https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L1540

This fix is released in 0.64.2 https://docs.datasette.io/en/stable/changelog.html#v0-64-2

Here's the https://latest.datasette.io/ deployment that just went out, further demonstrating that this change is working correctly:

Got the same failure again for a recent commit: https://github.com/simonw/datasette/actions/runs/4368239376/jobs/7640567282

It's this test here:

https://github.com/simonw/datasette/blob/1ad92a1d87d79084ebe524ed186c900ff042328c/tests/test_cli.py#L181-L189

Added in: - #2033

Instead of using isolated_filesystem() I could use a tmpdir fixture instead.

https://console.cloud.google.com/run/detail/us-central1/new-service/revisions?project=datasette-222320 confirms that the image deployed is:

Compared to https://console.cloud.google.com/run/detail/us-central1/datasette-io/revisions?project=datasette-222320 which shows that datasette.io is running:

Testing this manually:

``` % datasette publish cloudrun content.db --service new-service Creating temporary tarball archive of 2 file(s) totalling 13.8 MiB before compression. Uploading tarball of [.] to [gs://datasette-222320_cloudbuild/source/1678306859.271661-805303f364144b6094cc9c8532ab5133.tgz] Created [https://cloudbuild.googleapis.com/v1/projects/datasette-222320/locations/global/builds/290f41a4-e29a-443c-a1e5-c54513c6143d]. Logs are available at [ https://console.cloud.google.com/cloud-build/builds/290f41a4-e29a-443c-a1e5-c54513c6143d?project=99025868001 ]. ---- REMOTE BUILD OUTPUT ---- starting build "290f41a4-e29a-443c-a1e5-c54513c6143d"

FETCHSOURCE Fetching storage object: gs://datasette-222320_cloudbuild/source/1678306859.271661-805303f364144b6094cc9c8532ab5133.tgz#1678306862810483 Copying gs://datasette-222320_cloudbuild/source/1678306859.271661-805303f364144b6094cc9c8532ab5133.tgz#1678306862810483... / [1 files][ 3.9 MiB/ 3.9 MiB]
Operation completed over 1 objects/3.9 MiB. BUILD Already have image (with digest): gcr.io/cloud-builders/docker Sending build context to Docker daemon 14.52MB Step 1/9 : FROM python:3.11.0-slim-bullseye ... Installing collected packages: rfc3986, typing-extensions, sniffio, PyYAML, python-multipart, pluggy, pint, mergedeep, MarkupSafe, itsdangerous, idna, hupper, h11, click, certifi, asgiref, aiofiles, uvicorn, Jinja2, janus, click-default-group-wheel, asgi-csrf, anyio, httpcore, httpx, datasette Successfully installed Jinja2-3.1.2 MarkupSafe-2.1.2 PyYAML-6.0 aiofiles-23.1.0 anyio-3.6.2 asgi-csrf-0.9 asgiref-3.6.0 certifi-2022.12.7 click-8.1.3 click-default-group-wheel-1.2.2 datasette-0.64.1 h11-0.14.0 httpcore-0.16.3 httpx-0.23.3 hupper-1.11 idna-3.4 itsdangerous-2.1.2 janus-1.0.0 mergedeep-1.3.4 pint-0.20.1 pluggy-1.0.0 python-multipart-0.0.6 rfc3986-1.5.0 sniffio-1.3.0 typing-extensions-4.5.0 uvicorn-0.20.0 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip available: 22.3 -> 23.0.1 [notice] To update, run: pip install --upgrade pip Removing intermediate container 8ccebfebebc9 ---> b972c85b38bb ... Successfully built 606b7c286d7f Successfully tagged gcr.io/datasette-222320/datasette-new-service:latest PUSH Pushing gcr.io/datasette-222320/datasette-new-service The push refers to repository [gcr.io/datasette-222320/datasette-new-service] 667b1dc69e5e: Preparing ... d8ddfcff216f: Pushed latest: digest: sha256:452daffb2d3d7a8579c2ab39854be285155252c9428b4c1c50caac6a3a269e3f size: 2004 DONE

ID CREATE_TIME DURATION SOURCE IMAGES STATUS 290f41a4-e29a-443c-a1e5-c54513c6143d 2023-03-08T20:21:03+00:00 39S gs://datasette-222320_cloudbuild/source/1678306859.271661-805303f364144b6094cc9c8532ab5133.tgz gcr.io/datasette-222320/datasette-new-service (+1 more) SUCCESS Deploying container to Cloud Run service [new-service] in project [datasette-222320] region [us-central1] ✓ Deploying new service... Done.
✓ Creating Revision...
✓ Routing traffic...
✓ Setting IAM Policy...
Done.
Service [new-service] revision [new-service-00001-zon] has been deployed and is serving 100 percent of traffic. Service URL: https://new-service-j7hipcg4aq-uc.a.run.app ``` https://new-service-j7hipcg4aq-uc.a.run.app/ was deployed successfully.

I'm going to solve this by using the service name in that image_id instead:

python image_id = f"gcr.io/{project}/{service_name}" This is a nasty bug, so I'm going to backport it to a 0.64.2 release as well.

I think the code at fault is here:

https://github.com/simonw/datasette/blob/1ad92a1d87d79084ebe524ed186c900ff042328c/datasette/publish/cloudrun.py#L176-L182

That name ends up defaulting to datasette - so multiple different projects may end up deploying to the same image_id.

What I think happened in the datasette.io bug is that this workflow: https://github.com/simonw/simonwillisonblog-backup/blob/bfb573e96d8622ab52b22fdcd54724fe6e59fd24/.github/workflows/backup.yml and this workflow: https://github.com/simonw/datasette.io/blob/4676db5bf4a3fc9f792ee270ec0c59eb902cd2c3/.github/workflows/deploy.yml both happened to run at the exact same time.

And so the image that was pushed to gcr.io/datasette-222320/datasette:latest by the simonw/simonwillisonblog-backup action was then deployed by the simonw/datasette.io/ action, which broke the site.

I'm trying to get http://127.0.0.1:8001/fixtures/compound_three_primary_keys?_next=a,d,v to return the correct results.

Pushed that prototype to a branch: https://github.com/simonw/datasette/commit/0fe844e9adb006a0138e83102ced1329d9155c59 / https://github.com/simonw/datasette/compare/sql-list-parameters

I should consider the existing design of magic parameters here: https://docs.datasette.io/en/stable/sql_queries.html#magic-parameters

• _actor_*
• _header_*
• _cookie_
• _now_epoch
• _now_date_utc
• _now_datetime_utc
• _random_chars_*

Should this new id__list syntax look more like those magic parameters, or is it OK to use name__magic syntax here instead?

To implement this properly need to do the following: - Get the page to display multiple id: [ text input here ] fields such that re-submission works - Figure out how this should work for canned queries and for writable canned queries - Tests that cover queries, canned queries, writable canned queries

And a bonus feature: what if the Datasette UI layer spotted :id__list parameters and used them to add a bit of JavaScript that allowed users to click a + button next to an id form field to add another one?

Also, when a page is re-displayed for on of these queries it could potentially add an extra form field allowing people to add another value.

Though this has an annoying problem: how to tell the difference between an additional id input field that the user chose not to populate, v.s. one that is supposed to represent an empty string?

Maybe only support multiple id fields for users with JavaScript in order to avoid this problem.

Got a prototype working: ```diff diff --git a/datasette/views/database.py b/datasette/views/database.py index 8d289105..6f9d8a44 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -226,6 +226,12 @@ class QueryView(DataView): ): db = await self.ds.resolve_database(request) database = db.name + # Disallow x__list query string parameters + invalid_params = [k for k in request.args if k.endswith("__list")] + if invalid_params: + raise DatasetteError( + "Invalid query string parameters: {}".format(", ".join(invalid_params)) + ) params = {key: request.args.get(key) for key in request.args} if "sql" in params: params.pop("sql") @@ -258,6 +264,11 @@ class QueryView(DataView): for named_parameter in named_parameters if not named_parameter.startswith("_") } + # Handle any __list parameters + for named_parameter in named_parameters: + if named_parameter.endswith("__list"): + list_values = request.args.getlist(named_parameter[:-6]) + params[named_parameter] = json.dumps(list_values)

     # Set to blank string if missing from params
     for named_parameter in named_parameters:

`` This isn't yet doing the right thing on form re-submission: it breaks because it attempts to pass through the?id__list=` invalid parameter. But I did manage to get it to do this through careful editing of the URL:

That was this URL: http://127.0.0.1:8034/content?sql=select+%3Aid__list%2C*+from+releases+where+id+in+(select+value+from+json_each(%3Aid__list))&id=62642726&id=18402901&id=38714866

Also: datasette-explain may need to be updated to understand how to handle this:

ERROR: conn=<sqlite3.Connection object at 0x102834940>, sql = 'explain select * from releases where id in (select id from json_each(:id__list))', params = None: You did not supply a value for binding parameter :id__list.

Trickiest part of the implementation here is that it needs to know to output three id HTML form fields on the page, such that their values are persisted when the form is submitted a second time.

Since we are pre-1.0 it's still OK to implement a feature that disallows ?id__list= in the URL, but allows :id__list in SQL queries to reference the JSON list of parameters.

So I'm going to prototype this as the :id__list feature and see how it feels.

I'm pretty sold on this as a feature now. The main question I have is which of these options to implement:

1. ?id=1&?id=2 results in :id in the query being ["1", "2"] - no additional syntax required
2. :id in the query continues to reference just the first of those parameters - but :id__list (or some other custom syntax) instead gets ["1", "2"] - or, if the URL is ?id=1 - gets ["1"]

Actually on writing these out I realize that option 2 is the ONLY valid option. It's no good building a query that works against a JSON list if the user might pass just a single ID, ?id=1, resulting in their query breaking.