issue_comments

Table actions

7,703 rows where author_association = "OWNER" sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: reactions, created_at (date), updated_at (date)

issue >30

user 1

author_association 1

  • OWNER · 7,703 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
1477155602 https://github.com/simonw/datasette/issues/2041#issuecomment-1477155602 https://api.github.com/repos/simonw/datasette/issues/2041 IC_kwDOBm6k_c5YC5sS simonw 9599 2023-03-21T01:17:43Z 2023-03-21T01:17:43Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Remove obsolete table POST code 1633077183  
1477144853 https://github.com/simonw/datasette/issues/2041#issuecomment-1477144853 https://api.github.com/repos/simonw/datasette/issues/2041 IC_kwDOBm6k_c5YC3EV simonw 9599 2023-03-21T01:02:13Z 2023-03-21T01:02:13Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Remove obsolete table POST code 1633077183  
1477082852 https://github.com/simonw/datasette/pull/1999#issuecomment-1477082852 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5YCn7k simonw 9599 2023-03-20T23:27:25Z 2023-03-20T23:27:25Z OWNER

Urgh getting CSV to work is going to be so hard, because the logic for that currently lives in a huge chunk of code in BaseView which depends on the old design of the data() method:

https://github.com/simonw/datasette/blob/4bb49848697e40b8b9a1557be42b8e59eac965b3/datasette/views/base.py#L177-L343

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1476860334 https://github.com/simonw/datasette/pull/1999#issuecomment-1476860334 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5YBxmu simonw 9599 2023-03-20T20:06:56Z 2023-03-20T22:09:03Z OWNER

pytest -n 8 locally says:

FAILED tests/test_canned_queries.py::test_canned_query_form_csrf_hidden_field[add_name_specify_id-True] - assert '<input type="hidden" name="csrftoken" value="' in '<!DOCTYPE html>\n<html>\n<head>\n <title>Error 500</title>\n <link rel="stylesheet" href="/-/static/app.css?d59...ils != detailsClickedWithin\n ).forEach(details => details.open = false);\n});\n</... FAILED tests/test_canned_queries.py::test_canned_query_form_csrf_hidden_field[add_name-True] - assert '<input type="hidden" name="csrftoken" value="' in '<!DOCTYPE html>\n<html>\n<head>\n <title>Error 500</title>\n <link rel="stylesheet" href="/-/static/app.css?d59...ils != detailsClickedWithin\n ).forEach(details => details.open = false);\n});\n</... FAILED tests/test_canned_queries.py::test_canned_query_permissions - AssertionError: assert 200 == 500 FAILED tests/test_canned_queries.py::test_canned_query_with_named_parameter - KeyError: 'rows' FAILED tests/test_canned_queries.py::test_canned_write_custom_template - assert 500 == 200 FAILED tests/test_canned_queries.py::test_canned_write_query_disabled_for_immutable_database - assert 500 == 200 FAILED tests/test_canned_queries.py::test_custom_params - assert '<input type="text" id="qp3" name="extra" value="foo">' in '<!DOCTYPE html>\n<html>\n<head>\n <title>Error 500</title>\n <link rel="stylesheet" href="/-/static/app.css?d59...ils != detailsClickedWithin\n ).forEach(details => details.open = false);\... FAILED tests/test_canned_queries.py::test_insert - KeyError: 'ds_csrftoken' FAILED tests/test_canned_queries.py::test_insert_error - KeyError: 'ds_csrftoken' FAILED tests/test_canned_queries.py::test_magic_parameters_csrf_json[False-True] - KeyError: 'ds_csrftoken' FAILED tests/test_canned_queries.py::test_magic_parameters_csrf_json[True-True] - KeyError: 'ds_csrftoken' FAILED tests/test_canned_queries.py::test_vary_header - KeyError: 'vary' FAILED tests/test_csv.py::test_csv_trace - AssertionError: assert 'application/...charset=utf-8' == 'text/html; charset=utf-8' FAILED tests/test_csv.py::test_table_csv - AssertionError: assert 'application/...charset=utf-8' == 'text/plain; charset=utf-8' FAILED tests/test_csv.py::test_table_csv_blob_columns - AssertionError: assert 'application/...charset=utf-8' == 'text/plain; charset=utf-8' FAILED tests/test_csv.py::test_table_csv_download - AssertionError: assert 'application/...charset=utf-8' == 'text/csv; charset=utf-8' FAILED tests/test_csv.py::test_table_csv_no_header - AssertionError: assert 'application/...charset=utf-8' == 'text/plain; charset=utf-8' FAILED tests/test_csv.py::test_table_csv_stream - assert 1 == 101 FAILED tests/test_csv.py::test_table_csv_stream_does_not_calculate_counts - AttributeError: 'NoneType' object has no attribute 'text' FAILED tests/test_csv.py::test_table_csv_stream_does_not_calculate_facets - AttributeError: 'NoneType' object has no attribute 'text' FAILED tests/test_csv.py::test_table_csv_with_labels - AssertionError: assert 'application/...charset=utf-8' == 'text/plain; charset=utf-8' FAILED tests/test_csv.py::test_table_csv_with_nullable_labels - AssertionError: assert 'application/...charset=utf-8' == 'text/plain; charset=utf-8' FAILED tests/test_facets.py::test_array_facet_handle_duplicate_tags - KeyError: 'facet_results' FAILED tests/test_facets.py::test_conflicting_facet_names_json - AssertionError: assert {'results', 'timed_out'} == {'created', '...gs', 'tags_2'} FAILED tests/test_facets.py::test_facet_size - KeyError: 'suggested_facets' FAILED tests/test_facets.py::test_json_array_with_blanks_and_nulls - KeyError: 'suggested_facets' FAILED tests/test_facets.py::test_other_types_of_facet_in_metadata - assert '<strong>created (date)\n' in '<!DOCTYPE html>\n<html>\n<head>\n <title>fixtures: facetable: 15 rows</title>\n <link rel="stylesheet" href="/-...name": "table", "request_path": "/fixtures/facetable", "added": 15, "columns": null};</script>\n\n\n\n</bo... FAILED tests/test_html.py::test_alternate_url_json[/fixtures/facetable-http://localhost/fixtures/facetable.json] - KeyError: 'link' FAILED tests/test_html.py::test_alternate_url_json[/fixtures/table~2Fwith~2Fslashes~2Ecsv-http://localhost/fixtures/table~2Fwith~2Fslashes~2Ecsv.json] - KeyError: 'link' FAILED tests/test_html.py::test_templates_considered[/fixtures/simple_primary_key-table-fixtures-simple_primary_key.html, *table.html] - assert '<!-- Templates considered: table-fixtures-simple_primary_key.html, *table.html -->' in '<!DOCTYPE html>\n<html>\n<head>\n <title>fixtures: simple_primary_key: 5 rows</title>\n <link rel="stylesheet" ...able", "request_path": "/fixtures/simple_primary... FAILED tests/test_html.py::test_templates_considered[/fixtures/table~2Fwith~2Fslashes~2Ecsv-table-fixtures-tablewithslashescsv-fa7563.html, *table.html] - assert '<!-- Templates considered: table-fixtures-tablewithslashescsv-fa7563.html, *table.html -->' in '<!DOCTYPE html>\n<html>\n<head>\n <title>fixtures: table/with/slashes.csv: 1 row</title>\n <link rel="styleshee...quest_path": "/fixtures/table~2Fwith~2Fs... FAILED tests/test_plugins.py::test_hook_extra_body_script[/fixtures/sortable-expected_extra_body_script2] - AssertionError: assert {'added': 15,...ixtures', ...} == {'added': 15,...ixtures', ...} FAILED tests/test_plugins.py::test_hook_extra_css_urls[/fixtures/sortable-expected_decoded_object2] - AssertionError: assert {'added': 15,...ortable', ...} == {'added': 15,...ortable', ...} FAILED tests/test_plugins.py::test_hook_register_facet_classes - KeyError: 'suggested_facets' FAILED tests/test_plugins.py::test_hook_register_magic_parameters - KeyError: 'ds_csrftoken' FAILED tests/test_plugins.py::test_hook_register_output_renderer_all_parameters - assert {'next': None...', ...}, ...]} == {'1+1': 2, 'c... 0xXXX>', ...} FAILED tests/test_plugins.py::test_hook_register_output_renderer_no_parameters - assert b'Hello' == b'{"ok": true... "n": null}]}' FAILED tests/test_plugins.py::test_hook_register_output_renderer_returning_broken_value - assert 200 == 500 FAILED tests/test_plugins.py::test_hook_register_output_renderer_returning_response - assert {'next': None...', ...}, ...]} == {'this_is': 'json'} === 39 failed, 1259 passed, 2 skipped, 1 xfailed in 58.07s ===

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1476898261 https://github.com/simonw/datasette/pull/1999#issuecomment-1476898261 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5YB63V simonw 9599 2023-03-20T20:37:52Z 2023-03-20T20:37:52Z OWNER

Manual testing spotted a bug.

/content/repos.json?owner=9599&_facet_array=topics - does not return a facet_results key.

/content/repos.json?owner=9599&_facet_array=topics&_facet=owner does.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1476854645 https://github.com/simonw/datasette/pull/1999#issuecomment-1476854645 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5YBwN1 simonw 9599 2023-03-20T20:02:24Z 2023-03-20T20:02:24Z OWNER

``` def test_routes(routes, path, expected_class, expected_matches): match, view = resolve_routes(routes, path) if expected_class is None: assert match is None else:

      assert view.view_class.__name__ == expected_class

E AttributeError: 'function' object has no attribute 'view_class' ```

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1476851525 https://github.com/simonw/datasette/pull/1999#issuecomment-1476851525 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5YBvdF simonw 9599 2023-03-20T19:59:51Z 2023-03-20T20:01:40Z OWNER

Three failures in test_html.py: FAILED tests/test_html.py::test_templates_considered[/fixtures/simple_primary_key-table-fixtures-simple_primary_key.html, *table.html] - assert '<!-- Templates considered: table-fixtures-simple_primary_key.html, *table.html -->' in '<!DOCTYPE html>\n<html>\n<head>\n <title>fixtures: simple_primary_key: 5 rows</title>\n <link rel="stylesheet" ...able", "request_path": "/fixtures/simple_primary_key", "ad... FAILED tests/test_html.py::test_templates_considered[/fixtures/table~2Fwith~2Fslashes~2Ecsv-table-fixtures-tablewithslashescsv-fa7563.html, *table.html] - assert '<!-- Templates considered: table-fixtures-tablewithslashescsv-fa7563.html, *table.html -->' in '<!DOCTYPE html>\n<html>\n<head>\n <title>fixtures: table/with/slashes.csv: 1 row</title>\n <link rel="styleshee...quest_path": "/fixtures/table~2Fwith~2Fslashes~2Ec... FAILED tests/test_html.py::test_alternate_url_json[/fixtures/facetable-http://localhost/fixtures/facetable.json] - KeyError: 'link' FAILED tests/test_html.py::test_alternate_url_json[/fixtures/table~2Fwith~2Fslashes~2Ecsv-http://localhost/fixtures/table~2Fwith~2Fslashes~2Ecsv.json] - KeyError: 'link'

In CI I'm getting this: https://github.com/simonw/datasette/actions/runs/4472331872/jobs/7858360154

====== 44 failed, 1222 passed, 1 skipped, 1 xfailed in 171.24s (0:02:51) =======

Including failures in tests/test_plugins.py and tests/test_routes.py.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1476781401 https://github.com/simonw/datasette/pull/1999#issuecomment-1476781401 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5YBeVZ simonw 9599 2023-03-20T19:01:35Z 2023-03-20T19:01:35Z OWNER

Down to just these failures: FAILED tests/test_table_html.py::test_sort_links - AssertionError: assert [{'attrs': {'class': ['col-Link'], 'scope': 'col', 'data-column': 'Link', 'data-column-type': '', 'data-column-not-null': '0', 'data-is-pk': '0'}, 'a_href': None}, {'attrs': {'class': ['col-pk1'], 'scope': 'col', 'data-column': 'pk1', 'data-column-typ... FAILED tests/test_table_html.py::test_table_html_simple_primary_key - AssertionError: assert 'id\xa0▼' == 'id' FAILED tests/test_table_html.py::test_table_csv_json_export_interface - AssertionError: assert ['/fixtures/simple_primary_key.json?id__gt=2', '/fixtures/simple_primary_key.testall?id__gt=2', '/fixtures/simple_primary_key.testnone?id__gt=2', '/fixtures/simple_primary_key.testresponse?id__gt=2', '/fixtures/simple_primary_key.csv?id__gt=2&_size=ma... FAILED tests/test_table_html.py::test_csv_json_export_links_include_labels_if_foreign_keys - AssertionError: assert ['/fixtures/facetable.json?_labels=on', '/fixtures/facetable.testall?_labels=on', '/fixtures/facetable.testnone?_labels=on', '/fixtures/facetable.testresponse?_labels=on', '/fixtures/facetable.csv?_labels=on&_size=max', '#export'] == ['/', '', '#export'] FAILED tests/test_table_html.py::test_rowid_sortable_no_primary_key - AssertionError: assert 'rowid\xa0▼' == 'rowid' FAILED tests/test_table_html.py::test_table_html_filter_form_column_options[/fixtures/infinity-expected_column_options0] - AssertionError: assert ['- column -', 'rowid', 'value'] == ['- column -'] FAILED tests/test_table_html.py::test_table_html_filter_form_column_options[/fixtures/primary_key_multiple_columns-expected_column_options1] - AssertionError: assert ['- column -', 'id', 'content', 'content2'] == ['- column -'] FAILED tests/test_table_html.py::test_table_html_filter_form_column_options[/fixtures/compound_primary_key-expected_column_options2] - AssertionError: assert ['- column -', 'pk1', 'pk2', 'content'] == ['- column -'] FAILED tests/test_table_html.py::test_table_html_filter_form_still_shows_nocol_columns - AssertionError: assert ['- column -'] == ['- column -', 'pk1', 'pk2', 'content', 'sortable_with_nulls', 'sortable_with_nulls_2', 'text', 'sortable'] FAILED tests/test_table_html.py::test_metadata_sort - AssertionError: assert ['id', 'name\xa0▼'] == ['id', 'name'] FAILED tests/test_table_html.py::test_metadata_sort_desc - AssertionError: assert ['pk\xa0▲', 'name'] == ['pk', 'name'] === 11 failed, 57 passed in 7.59s ===

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1460759358 https://github.com/simonw/datasette/pull/1999#issuecomment-1460759358 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XEWs- simonw 9599 2023-03-08T19:48:13Z 2023-03-20T18:47:12Z OWNER

Breaking this down into smaller steps:

  • [x] Get ?_next= working
  • [x] Implement extensions - so .json is needed again for the JSON version, and anything without an extension is passed through a new code path for HTML
  • [ ] That HTML view should only access JSON data, which can be seen by using .context - this will require a lot of updates to templates (it may be necessary to still provide access to some helper functions though). This will form the basis of the ambition to fully document the template context.
  • [ ] Get a bunch of the existing table HTML and JSON tests to pass
  • [ ] Use those tests to refactor the nasty _next code, see https://github.com/simonw/datasette/pull/1999#issuecomment-1460905469
  • [ ] Figure out how the register_output_renderer(datasette) plugin hook should work
 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1475074311 https://github.com/simonw/datasette/pull/1999#issuecomment-1475074311 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5X69kH simonw 9599 2023-03-19T02:16:08Z 2023-03-19T02:17:33Z OWNER

Next big challenge: fix all the tests in test_table_html.py:

FAILED tests/test_table_html.py::test_add_filter_redirects - assert 500 == 302 FAILED tests/test_table_html.py::test_existing_filter_redirects - assert 500 == 302 FAILED tests/test_table_html.py::test_reflected_hidden_form_fields[_facet=_neighborhood-expected_hidden0] - AssertionError: assert {} == {'_facet': '_neighborhood'} FAILED tests/test_table_html.py::test_reflected_hidden_form_fields[_where=1+=+1&_col=_city_id-expected_hidden1] - AssertionError: assert {} == {'_col': '_ci...ere': '1 = 1'} FAILED tests/test_table_html.py::test_reflected_hidden_form_fields[_facet=_neighborhood&_neighborhood__exact=Downtown-expected_hidden2] - AssertionError: assert {} == {'_facet': '_neighborhood'} FAILED tests/test_table_html.py::test_reflected_hidden_form_fields[_facet=_neighborhood&_city_id__gt=1-expected_hidden3] - AssertionError: assert {} == {'_facet': '_neighborhood'} FAILED tests/test_table_html.py::test_empty_search_parameter_gets_removed - assert 500 == 302 FAILED tests/test_table_html.py::test_searchable_view_persists_fts_table - AssertionError: assert [('_fts_table...ts_pk', 'pk')] == [] FAILED tests/test_table_html.py::test_sort_by_desc_redirects - assert 500 == 302 FAILED tests/test_table_html.py::test_sort_links - AssertionError: assert [{'a_href': N...', ...}}, ...] == [{'a_href': N...', ...}}, ...] FAILED tests/test_table_html.py::test_facets_persist_through_filter_form - AssertionError: assert [] == [('_facet', '...ray', 'tags')] FAILED tests/test_table_html.py::test_next_does_not_persist_in_hidden_field - AssertionError: assert [] == [('_size', '1')] FAILED tests/test_table_html.py::test_table_html_simple_primary_key - AttributeError: 'NoneType' object has no attribute 'string' FAILED tests/test_table_html.py::test_table_csv_json_export_interface - AssertionError: assert ['/fixtures/s...x', '#export'] == ['/', '', '#export'] FAILED tests/test_table_html.py::test_csv_json_export_links_include_labels_if_foreign_keys - AssertionError: assert ['/fixtures/f...x', '#export'] == ['/', '', '#export'] FAILED tests/test_table_html.py::test_table_html_no_primary_key - AssertionError: assert ['content', 'a', 'b', 'c'] == ['a', 'b', 'c'] FAILED tests/test_table_html.py::test_rowid_sortable_no_primary_key - AttributeError: 'NoneType' object has no attribute 'string' FAILED tests/test_table_html.py::test_table_html_compound_primary_key - AssertionError: assert 'Link' == 'pk1' FAILED tests/test_table_html.py::test_table_html_foreign_key_links - assert [['<td class=...">\xa0</td>']] == [['<td class=...">\xa0</td>']] FAILED tests/test_table_html.py::test_table_html_disable_foreign_key_links_with_labels - assert [['<td class=...str">b</td>']] == [['<td class=...str">b</td>']] FAILED tests/test_table_html.py::test_table_html_foreign_key_custom_label_column - assert [['<td class=...1</em></td>']] == [['<td class=...1</em></td>']] FAILED tests/test_table_html.py::test_table_html_filter_form_column_options[/fixtures/infinity-expected_column_options0] - AssertionError: assert ['- column -'...wid', 'value'] == ['- column -'] FAILED tests/test_table_html.py::test_table_html_filter_form_column_options[/fixtures/primary_key_multiple_columns-expected_column_options1] - AssertionError: assert ['- column -'...', 'content2'] == ['- column -'] FAILED tests/test_table_html.py::test_table_html_filter_form_column_options[/fixtures/compound_primary_key-expected_column_options2] - AssertionError: assert ['- column -'...2', 'content'] == ['- column -'] FAILED tests/test_table_html.py::test_table_html_filter_form_still_shows_nocol_columns - AssertionError: assert ['- column -'] == ['- column -'...nulls_2', ...] FAILED tests/test_table_html.py::test_compound_primary_key_with_foreign_key_references - assert [['<td class=...ne</a></td>']] == [['<td class=...ne</a></td>']] FAILED tests/test_table_html.py::test_view_html - assert None is not None FAILED tests/test_table_html.py::test_extra_where_clauses - assert [('_where', "...'_city_id=1')] == [] FAILED tests/test_table_html.py::test_other_hidden_form_fields[/fixtures/facetable?_size=10-expected_hidden0] - AssertionError: assert [] == [('_size', '10')] FAILED tests/test_table_html.py::test_other_hidden_form_fields[/fixtures/facetable?_size=10&_ignore=1&_ignore=2-expected_hidden1] - AssertionError: assert [] == [('_size', '1...ignore', '2')] FAILED tests/test_table_html.py::test_search_and_sort_fields_not_duplicated[/fixtures/searchable?_sort=text2&_where=1-expected_hidden3] - AssertionError: assert [] == [('_where', '1')] FAILED tests/test_table_html.py::test_binary_data_display_in_table - assert [['<td class=...">\xa0</td>']] == [['<td class=...">\xa0</td>']] FAILED tests/test_table_html.py::test_metadata_sort - AttributeError: 'NoneType' object has no attribute 'string' FAILED tests/test_table_html.py::test_metadata_sort_desc - AttributeError: 'NoneType' object has no attribute 'string' FAILED tests/test_table_html.py::test_facet_more_links[5-/fixtures/facetable?_facet=_neighborhood-2-True-/fixtures/facetable?_facet=_neighborhood&_facet_size=max] - assert 0 == 2 FAILED tests/test_table_html.py::test_facet_more_links[5-/fixtures/facetable?_facet=_neighborhood&_facet_size=50-5-True-/fixtures/facetable?_facet=_neighborhood&_facet_size=max] - assert 0 == 5 FAILED tests/test_table_html.py::test_facet_total - assert 500 == 200

Deduped that's 30 tests:

FAILED tests/test_table_html.py::test_add_filter_redirects FAILED tests/test_table_html.py::test_binary_data_display_in_table FAILED tests/test_table_html.py::test_compound_primary_key_with_foreign_key_references FAILED tests/test_table_html.py::test_csv_json_export_links_include_labels_if_foreign_keys FAILED tests/test_table_html.py::test_empty_search_parameter_gets_removed FAILED tests/test_table_html.py::test_existing_filter_redirects FAILED tests/test_table_html.py::test_extra_where_clauses FAILED tests/test_table_html.py::test_facet_more_links FAILED tests/test_table_html.py::test_facet_total FAILED tests/test_table_html.py::test_facets_persist_through_filter_form FAILED tests/test_table_html.py::test_metadata_sort FAILED tests/test_table_html.py::test_metadata_sort_desc FAILED tests/test_table_html.py::test_next_does_not_persist_in_hidden_field FAILED tests/test_table_html.py::test_other_hidden_form_fields FAILED tests/test_table_html.py::test_reflected_hidden_form_fields FAILED tests/test_table_html.py::test_rowid_sortable_no_primary_key FAILED tests/test_table_html.py::test_search_and_sort_fields_not_duplicated FAILED tests/test_table_html.py::test_searchable_view_persists_fts_table FAILED tests/test_table_html.py::test_sort_by_desc_redirects FAILED tests/test_table_html.py::test_sort_links FAILED tests/test_table_html.py::test_table_csv_json_export_interface FAILED tests/test_table_html.py::test_table_html_compound_primary_key FAILED tests/test_table_html.py::test_table_html_disable_foreign_key_links_with_labels FAILED tests/test_table_html.py::test_table_html_filter_form_column_options FAILED tests/test_table_html.py::test_table_html_filter_form_still_shows_nocol_columns FAILED tests/test_table_html.py::test_table_html_foreign_key_custom_label_column FAILED tests/test_table_html.py::test_table_html_foreign_key_links FAILED tests/test_table_html.py::test_table_html_no_primary_key FAILED tests/test_table_html.py::test_table_html_simple_primary_key FAILED tests/test_table_html.py::test_view_html

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1475074025 https://github.com/simonw/datasette/pull/1999#issuecomment-1475074025 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5X69fp simonw 9599 2023-03-19T02:14:28Z 2023-03-19T02:14:51Z OWNER

I had to replicate quite a bit of this logic from base.py:

https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/views/base.py#L526-L544

I should refactor this when I move the canned / arbitrary query views away from that base class too.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1475016834 https://github.com/simonw/datasette/pull/1999#issuecomment-1475016834 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5X6viC simonw 9599 2023-03-18T22:30:31Z 2023-03-18T22:30:31Z OWNER

test_paginate_using_link_header will be tricky.

The reason the tests are failing is that json_renderer() attempts to populate the link header using data["next_url"] - but that's not present unless ?_extra=next_url has been passed:

https://github.com/simonw/datasette/blob/2f38479dcc81f11a4362f4e28511fa88afc34e61/datasette/renderer.py#L101-L102

But I can only rely on next being present, not next_url.

I thought I could maybe assemble the link header using next, by turning that into a next_url link - but there's some custom logic which kicks in for pagination against SQL views (offset/limit pagination) to calculate the next_url which isn't easily replicable at the json_renderer() layer, in the _next_value_and_url() utility function:

https://github.com/simonw/datasette/blob/2f38479dcc81f11a4362f4e28511fa88afc34e61/datasette/views/table.py#L2275-L2282

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1475003292 https://github.com/simonw/datasette/pull/1999#issuecomment-1475003292 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5X6sOc simonw 9599 2023-03-18T21:46:20Z 2023-03-18T21:46:20Z OWNER

Now 25 failures in test_table_api.py: FAILED tests/test_table_api.py::test_expand_labels - assert {'2': {'pk': 2, 'created': '2019-01-14 08:00:00', 'planet_int': 1, 'on_earth': 1, 'state': 'CA', '_city_id': 1, '_neighborhood': 'Dogpatch', 'tags': '["tag1", "tag3"]', 'complex_array'... FAILED tests/test_table_api.py::test_expand_label - AssertionError: assert {'1': {'pk': '1', 'foreign_key_with_label': '1', 'foreign_key_with_blank_label': '3', 'foreign_key_with_no_label': '1', 'foreign_key_compound_pk1': 'a', 'foreign_key_co... FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json-max-age=5] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=invalid-max-age=5] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=10-max-age=10] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=0-no-cache] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_infinity_returned_as_null - AssertionError: assert [{'rowid': 1, 'value': inf}, {'rowid': 2, 'value': -inf}, {'rowid': 3, 'value': 1.5}] == [{'rowid': 1, 'value': None}, {'rowid': 2, 'value': None}, {'rowid': 3, 'value'... FAILED tests/test_table_api.py::test_null_and_compound_foreign_keys_are_not_expanded - AssertionError: assert [{'pk': '1', 'foreign_key_with_label': '1', 'foreign_key_with_blank_label': '3', 'foreign_key_with_no_label': '1', 'foreign_key_compound_pk1': 'a', 'foreign_key_compoun... FAILED tests/test_table_api.py::test_binary_data_in_json[/fixtures/binary_data.json?_shape=array-expected_json0-None] - assert [{'rowid': 1, 'data': "b'\\x15\\x1c\\x02\\xc7\\xad\\x05\\xfe'"}, {'rowid': 2, 'data': "b'\\x15\\x1c\\x03\\xc7\\xad\\x05\\xfe'"}, {'rowid': 3, 'data': None}] == [{'rowid': 1, 'data': {'... FAILED tests/test_table_api.py::test_binary_data_in_json[/fixtures/binary_data.json?_shape=array&_nl=on-None-{"rowid": 1, "data": {"$base64": true, "encoded": "FRwCx60F/g=="}}\n{"rowid": 2, "data": {"$base64": true, "encoded": "FRwDx60F/g=="}}\n{"rowid": 3, "data": null}] - assert '{"ok": false, "error": "Object of type bytes is not JSON serializable", "status": 500, "title": null}' == '{"rowid": 1, "data": {"$base64": true, "encoded": "FRwCx60F/g=="}}\n{"rowid"... FAILED tests/test_table_api.py::test_paginate_using_link_header[] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=arrays] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=arrayfirst] - assert 400 == 200 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=object] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=objects] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=array] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=array&_nl=on] - assert 1 == 21 FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=created-expected_columns0] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_nocol=created-expected_columns1] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=created-expected_columns2] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=state-expected_columns3] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=created&_nocol=created-expected_columns4] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_nocol=state&_facet=state-expected_columns5] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/simple_view.json?_nocol=content-expected_columns6] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/simple_view.json?_col=content-expected_columns7] - KeyError: 'columns' ============================================================================= 25 failed, 86 passed, 1 xfailed in 7.18s =============================================================================

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1474704790 https://github.com/simonw/datasette/pull/1999#issuecomment-1474704790 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5X5jWW simonw 9599 2023-03-18T04:52:59Z 2023-03-18T04:52:59Z OWNER

Here are the next set of tests to get passing: % pytest tests/test_table_api.py FAILED tests/test_table_api.py::test_facets[/fixtures/facetable.json?_facet=state&_facet=_city_id-expected_facet_results0] - KeyError: 'name' FAILED tests/test_table_api.py::test_facets[/fixtures/facetable.json?_facet=state&_facet=_city_id&state=MI-expected_facet_results1] - KeyError: 'name' FAILED tests/test_table_api.py::test_facets[/fixtures/facetable.json?_facet=planet_int-expected_facet_results2] - KeyError: 'name' FAILED tests/test_table_api.py::test_facets[/fixtures/facetable.json?_facet=planet_int&planet_int=1-expected_facet_results3] - KeyError: 'name' FAILED tests/test_table_api.py::test_suggested_facets - KeyError: 'suggested_facets' FAILED tests/test_table_api.py::test_allow_facet_off - KeyError: 'suggested_facets' FAILED tests/test_table_api.py::test_suggest_facets_off - KeyError: 'suggested_facets' FAILED tests/test_table_api.py::test_nofacet[True] - KeyError: 'suggested_facets' FAILED tests/test_table_api.py::test_nofacet[False] - KeyError: 'suggested_facets' FAILED tests/test_table_api.py::test_nosuggest[True] - KeyError: 'suggested_facets' FAILED tests/test_table_api.py::test_nosuggest[False] - KeyError: 'suggested_facets' FAILED tests/test_table_api.py::test_nocount[True-None] - KeyError: 'count' FAILED tests/test_table_api.py::test_nocount[False-15] - KeyError: 'count' FAILED tests/test_table_api.py::test_expand_labels - AssertionError: assert {'13': {'_cit...:00:00', ...}} == {'13': {'_cit...:00:00', ...}} FAILED tests/test_table_api.py::test_expand_label - AssertionError: assert {'1': {'forei...l': '1', ...}} == {'1': {'forei...': '1'}, ...}} FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json-max-age=5] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=invalid-max-age=5] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=10-max-age=10] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=0-no-cache] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_infinity_returned_as_null - AssertionError: assert [{'rowid': 1,...'value': 1.5}] == [{'rowid': 1,...'value': 1.5}] FAILED tests/test_table_api.py::test_null_and_compound_foreign_keys_are_not_expanded - AssertionError: assert [{'foreign_ke...': None, ...}] == [{'foreign_ke...': None, ...}] FAILED tests/test_table_api.py::test_binary_data_in_json[/fixtures/binary_data.json?_shape=array-expected_json0-None] - assert [{'data': "b'..., 'rowid': 3}] == [{'data': {'$..., 'rowid': 3}] FAILED tests/test_table_api.py::test_binary_data_in_json[/fixtures/binary_data.json?_shape=array&_nl=on-None-{"rowid": 1, "data": {"$base64": true, "encoded": "FRwCx60F/g=="}}\n{"rowid": 2, "data": {"$base64": true, "encoded": "FRwDx60F/g=="}}\n{"rowid": 3, "data": null}] - assert '{"ok": false...title": null}' == '{"rowid": 1,..."data": null}' FAILED tests/test_table_api.py::test_paginate_using_link_header[] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=arrays] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=arrayfirst] - assert 400 == 200 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=object] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=objects] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=array] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=array&_nl=on] - assert 1 == 21 FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=created-expected_columns0] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_nocol=created-expected_columns1] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=created-expected_columns2] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=state-expected_columns3] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=created&_nocol=created-expected_columns4] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_nocol=state&_facet=state-expected_columns5] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/simple_view.json?_nocol=content-expected_columns6] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/simple_view.json?_col=content-expected_columns7] - KeyError: 'columns' ============================================================================= 38 failed, 73 passed, 1 xfailed in 7.25s =============================================================================

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1465303378 https://github.com/simonw/sqlite-utils/issues/533#issuecomment-1465303378 https://api.github.com/repos/simonw/sqlite-utils/issues/533 IC_kwDOCGYnMM5XVsFS simonw 9599 2023-03-12T21:24:04Z 2023-03-12T21:24:04Z OWNER

Upgraded to Sphinx 6 locally and got the same error:

% just docs Cogging README.md Cogging docs/changelog.rst Cogging docs/cli-reference.rst Cogging docs/cli.rst Cogging docs/contributing.rst Cogging docs/index.rst Cogging docs/installation.rst Cogging docs/python-api.rst Cogging docs/reference.rst sphinx-autobuild -a -b html "." "_build" --watch ../sqlite_utils [sphinx-autobuild] > sphinx-build -b html -a /Users/simon/Dropbox/Development/sqlite-utils/docs /Users/simon/Dropbox/Development/sqlite-utils/docs/_build Running Sphinx v6.1.3 loading pickled environment... failed failed: Can't get attribute '_stable_repr_object' on <sphinx.deprecation._ModuleWrapper object at 0x103f155d0> building [mo]: all of 0 po files writing output... building [html]: all source files updating environment: [new config] 8 added, 0 changed, 0 removed reading sources... [ 12%] changelog Exception occurred: File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.11/site-packages/sphinx/ext/extlinks.py", line 103, in role title = caption % part ~~~~~~~~^~~~~~ TypeError: not all arguments converted during string formatting The full traceback has been saved in /var/folders/x6/31xf1vxj0nn9mxqq8z0mmcfw0000gn/T/sphinx-err-1ey36c1n.log, if you want to report the issue to the developers.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ReadTheDocs error: not all arguments converted during string formatting 1620516340  
1465302936 https://github.com/simonw/sqlite-utils/issues/533#issuecomment-1465302936 https://api.github.com/repos/simonw/sqlite-utils/issues/533 IC_kwDOCGYnMM5XVr-Y simonw 9599 2023-03-12T21:22:09Z 2023-03-12T21:22:09Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ReadTheDocs error: not all arguments converted during string formatting 1620516340  
1465302343 https://github.com/simonw/sqlite-utils/pull/531#issuecomment-1465302343 https://api.github.com/repos/simonw/sqlite-utils/issues/531 IC_kwDOCGYnMM5XVr1H simonw 9599 2023-03-12T21:19:13Z 2023-03-12T21:19:13Z OWNER

Aah, I think I see why you wrote it like that.

The problem is that init_spatialite() does other stuff too:

https://github.com/simonw/sqlite-utils/blob/fc221f9b62ed8624b1d2098e564f525c84497969/sqlite_utils/db.py#L1161-L1171

So it needs to be able to load the SpatiaLite extension from the correct place, and THEN run select InitSpatialMetadata() to configure the database, if needed.

So the problem you're trying to solve here is to let people optionally pass in the path to SpatiaLite if it's not one of the ones that are searched by default.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Add paths for homebrew on Apple silicon 1620164673  
1463113856 https://github.com/simonw/datasette/pull/1999#issuecomment-1463113856 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XNViA simonw 9599 2023-03-10T02:13:15Z 2023-03-10T02:13:15Z OWNER

Idea for if this change ends up making a bunch of breaking changes to the templates (which I think it should) - I can generate a GitHub diff link between the old and new templates and include that link in the 1.0 upgrade documentation to help people who wrote custom templates see what they might need to change - with minimal effort from myself to document those changes.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1463112173 https://github.com/simonw/datasette/issues/2038#issuecomment-1463112173 https://api.github.com/repos/simonw/datasette/issues/2038 IC_kwDOBm6k_c5XNVHt simonw 9599 2023-03-10T02:11:06Z 2023-03-10T02:11:06Z OWNER

Here's an example of something that would break:

https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/templates/database.html#L12-L15

Because metadata is an empty dictionary sometimes, so {{ metadata.title or database }} would raise an error and need to be replaced by {{ metadata.get("title") or database }}.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Consider a `strict_templates` setting 1618249044  
1463110978 https://github.com/simonw/datasette/issues/2038#issuecomment-1463110978 https://api.github.com/repos/simonw/datasette/issues/2038 IC_kwDOBm6k_c5XNU1C simonw 9599 2023-03-10T02:09:41Z 2023-03-10T02:09:41Z OWNER

I'm torn on this. It's useful for me right now for refactoring, but I feel like it should be a permanent thing, not a setting - or it should default to on and people should turn it off, but who would ever do that?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Consider a `strict_templates` setting 1618249044  
1463024951 https://github.com/simonw/datasette/pull/1999#issuecomment-1463024951 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XM_03 simonw 9599 2023-03-10T00:17:58Z 2023-03-10T00:17:58Z OWNER

Renderers have an impact on three different pages: query results, table page and row page.

The row page feature is incomplete though:

https://congress-legislators.datasettes.com/legislators/social_media/A000055

Why is there no .csv link there?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1463023674 https://github.com/simonw/datasette/pull/1999#issuecomment-1463023674 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XM_g6 simonw 9599 2023-03-10T00:16:03Z 2023-03-10T00:16:03Z OWNER

I also need to figure out the renderers stuff, so I can link to the right URLs for CSV and JSON and other formats:

https://github.com/simonw/datasette/blob/6d07a7da1531cd749844fc6827d9a1e57009b2ea/datasette/views/base.py#L474-L518

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1463022397 https://github.com/simonw/datasette/pull/1999#issuecomment-1463022397 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XM_M9 simonw 9599 2023-03-10T00:14:21Z 2023-03-10T00:14:21Z OWNER

Ironically the thing I most need right now is comprehensive documentation of what variables are being passed to the templates!

One big challenge is that I need to untangle the template context that happens in BaseView - I'm hacking that together at the moment, but I need a real answer for how that should work in a world in which view functions aren't using that base class.

https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/views/base.py#L110-L145

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1463021383 https://github.com/simonw/datasette/pull/1999#issuecomment-1463021383 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XM-9H simonw 9599 2023-03-10T00:12:50Z 2023-03-10T00:12:50Z OWNER

Now at 34 failed, 34 passed.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1463005744 https://github.com/simonw/datasette/pull/1999#issuecomment-1463005744 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XM7Iw simonw 9599 2023-03-09T23:52:15Z 2023-03-09T23:52:23Z OWNER

I need to figure out what to do about extra_context_from_filters - which was previously passed straight to the HTML context.

https://github.com/simonw/datasette/blob/11f7feb7a3f7166c71389786880863d60ed3d165/datasette/views/table.py#L406-L422

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1462997800 https://github.com/simonw/datasette/pull/1999#issuecomment-1462997800 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XM5Mo simonw 9599 2023-03-09T23:39:47Z 2023-03-09T23:39:47Z OWNER

Found a neat trick: ```diff diff --git a/datasette/app.py b/datasette/app.py index 186f192d..40416713 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -23,7 +23,13 @@ from pathlib import Path

from markupsafe import Markup, escape from itsdangerous import URLSafeSerializer -from jinja2 import ChoiceLoader, Environment, FileSystemLoader, PrefixLoader +from jinja2 import ( + ChoiceLoader, + Environment, + FileSystemLoader, + PrefixLoader, + StrictUndefined, +) from jinja2.environment import Template from jinja2.exceptions import TemplateNotFound

@@ -394,7 +400,10 @@ class Datasette: ] ) self.jinja_env = Environment( - loader=template_loader, autoescape=True, enable_async=True + loader=template_loader, + autoescape=True, + enable_async=True, + undefined=StrictUndefined, ) self.jinja_env.filters["escape_css_string"] = escape_css_string self.jinja_env.filters["quote_plus"] = urllib.parse.quote_plus ``` This causes Jinja to raise a hard error if there are any variables referenced in the template that are not available in the context.

It's helping me spot things that are still missing, rather than just relying on failed unit tests.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1462921890 https://github.com/simonw/datasette/pull/2031#issuecomment-1462921890 https://api.github.com/repos/simonw/datasette/issues/2031 IC_kwDOBm6k_c5XMmqi simonw 9599 2023-03-09T22:35:30Z 2023-03-09T22:35:30Z OWNER

I've implemented the test (thanks for pointing me in the right direction!).

At tmcl-it/datasette:0.64.1+row-view-expand-labels I also have a variant of this patch that applies to the 0.64.x branch. Please let me know if you'd be interested in merging that as well and I'll open another PR.

Sure, let's merge that one too - it can go out in the next 0.64.x series release (maybe even a 0.65).

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Expand foreign key references in row view as well 1605481359  
1462921010 https://github.com/simonw/datasette/pull/2034#issuecomment-1462921010 https://api.github.com/repos/simonw/datasette/issues/2034 IC_kwDOBm6k_c5XMmcy simonw 9599 2023-03-09T22:34:29Z 2023-03-09T22:34:29Z OWNER

Good catch, thanks.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 remove an unused `app` var in cli.py 1613974869  
1461161256 https://github.com/simonw/datasette/pull/1999#issuecomment-1461161256 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XF40o simonw 9599 2023-03-09T02:10:07Z 2023-03-09T02:10:07Z OWNER

Just ran into a no such table: columns error - which I think is because my new view code sometimes bypasses calling this method (currently done in BaseView.dispatch_request()):

https://github.com/simonw/datasette/blob/96e94f9b7b2db53865e61390bcce6761727f26d8/datasette/views/base.py#L101-L103

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1461148579 https://github.com/simonw/datasette/pull/1999#issuecomment-1461148579 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XF1uj simonw 9599 2023-03-09T01:54:10Z 2023-03-09T01:55:33Z OWNER

Or... I could temporarily build a quick additional CannedQueryView subclass that just does the necessary bits to get the existing code to work. I'm going to try that.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1461148254 https://github.com/simonw/datasette/pull/1999#issuecomment-1461148254 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XF1pe simonw 9599 2023-03-09T01:53:41Z 2023-03-09T01:53:41Z OWNER

Solving this is proving difficult: https://github.com/simonw/datasette/blob/96e94f9b7b2db53865e61390bcce6761727f26d8/datasette/views/table.py#L1500-L1503

The problem is that calling .data() on QueryView only works here because we expect to ourselves be inside a .data() method, with all of the existing magic that knows how to render things that are returned by that.

So I may need to substantially re-engineer how QueryView works in order to get this to work.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1461075648 https://github.com/simonw/datasette/pull/1999#issuecomment-1461075648 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XFj7A simonw 9599 2023-03-09T00:24:22Z 2023-03-09T00:24:22Z OWNER

127.0.0.1:8001/fixtures/neighborhood_search fails because the forwarding to a canned query does not yet work.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1461074526 https://github.com/simonw/datasette/pull/1999#issuecomment-1461074526 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XFjpe simonw 9599 2023-03-09T00:23:06Z 2023-03-09T00:23:06Z OWNER 
pytest tests/test_table_html.py

Currently 44 failed, 24 passed in 7.53s

Failures here: https://gist.github.com/simonw/df0a52cd7d820b776dc3dfc50e7cb778

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1461070937 https://github.com/simonw/datasette/pull/1999#issuecomment-1461070937 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XFixZ simonw 9599 2023-03-09T00:18:52Z 2023-03-09T00:19:36Z OWNER

I managed to get HTML view working! I did it by continuing to add more things to the extras and the _html bundle until the page loaded for me:

```diff async def extra_extras(): "Available ?_extra= blocks" return { @@ -1981,6 +2053,14 @@ async def extra_extras(): "query", "display_columns", "display_rows", + "database", + "table", + "database_color", + "table_actions", + "filters", + "renderers", + "custom_table_templates", + "sorted_facet_results", ] }

@@ -2006,6 +2086,14 @@ async def extra_extras(): extra_query, extra_metadata, extra_extras, + extra_database, + extra_table, + extra_database_color, + extra_table_actions, + extra_filters, + extra_renderers, + extra_custom_table_templates, + extra_sorted_facet_results, ) `` I'll probably refactor this into something cleaner, and maybe but a bunch of them in a"html"dictionary and update the templates to use{{ html.filters }}` or similar. Will look at that once the tests are passing.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1461047607 https://github.com/simonw/datasette/pull/1999#issuecomment-1461047607 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XFdE3 simonw 9599 2023-03-08T23:51:46Z 2023-03-08T23:51:46Z OWNER

This feels quite nice:

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1461044477 https://github.com/simonw/datasette/pull/1999#issuecomment-1461044477 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XFcT9 simonw 9599 2023-03-08T23:47:26Z 2023-03-08T23:47:26Z OWNER

I want to package together all of the extras that are needed for the HTML format. A few options for doing that:

  • Introduce ?_extra=_html where the leading underscore indicates that this is a "bundle" of extras, then define a bundle that's everything needed for the HTML renderer
  • Have some other mechanism whereby different renderers can request a bundle of extras.

I'm leaning towards the first option. I'll try that and see what it looks like.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1461023559 https://github.com/simonw/datasette/pull/1999#issuecomment-1461023559 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XFXNH simonw 9599 2023-03-08T23:23:02Z 2023-03-08T23:23:02Z OWNER

To get this unblocked, I'm going to allow myself to pass non-JSON-serializable objects to the HTML template version of things. If I can get that working (and get the existing tests to pass) I can consider a later change that makes those JSON serializable - or admit that it's OK for the templates to have non-JSON data passed to them and figure out how best to document those variables independently from the JSON documentation.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1461002039 https://github.com/simonw/datasette/pull/1999#issuecomment-1461002039 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XFR83 simonw 9599 2023-03-08T22:58:16Z 2023-03-08T23:02:09Z OWNER

The reason for that Row thing is that it allows custom templates that do things like this:

https://docs.datasette.io/en/stable/changelog.html#easier-custom-templates-for-table-rows

html+jinja {% for row in display_rows %} <div> <h2>{{ row["title"] }}</h2> <p>{{ row["description"] }}<lp> <p>Category: {{ row.display("category_id") }}</p> </div> {% endfor %} Is that a good design? the .display() thing feels weird - I wonder if anyone has ever actually used that.

It's documented here: https://docs.datasette.io/en/0.64.2/custom_templates.html#custom-templates

If you want to output the rendered HTML version of a column, including any links to foreign keys, you can use {{ row.display("column_name") }}.

I can't see any examples of anyone using it in this code search: https://cs.github.com/?scopeName=All+repos&scope=&q=datasette+row.display

It is however useful to have some kind of abstraction layer here that insulates the SQLite Row object, since having an extra layer will help if Datasette ever grows support for alternative database backends such as DuckDB or PostgreSQL.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1460988975 https://github.com/simonw/datasette/pull/1999#issuecomment-1460988975 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XFOwv simonw 9599 2023-03-08T22:42:57Z 2023-03-08T22:42:57Z OWNER

Aside idea: it might be interesting if there were "lazy" template variables available in the context: things that are not actually executed unless a template author requests them.

Imagine if metadata was a lazy template reference, such that custom templates that don't display any metadata don't trigger it to be resolved (which might involve additional database queries some day).

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1460986533 https://github.com/simonw/datasette/pull/1999#issuecomment-1460986533 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XFOKl simonw 9599 2023-03-08T22:40:28Z 2023-03-08T22:40:28Z OWNER

Figuring out what to do with display_columns_and_rows() is hard. That returns rows as this special kind of object, which is designed to be accessed from the HTML templates:

https://github.com/simonw/datasette/blob/96e94f9b7b2db53865e61390bcce6761727f26d8/datasette/views/table.py#L45-L71

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1460970807 https://github.com/simonw/datasette/pull/1999#issuecomment-1460970807 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XFKU3 simonw 9599 2023-03-08T22:31:49Z 2023-03-08T22:33:03Z OWNER

For the HTML version, I need to decide where all of the stuff that happens in async def extra_template() is going to live.

I think it's another one of those extra functions, triggered for ?_extra=context.

https://github.com/simonw/datasette/blob/96e94f9b7b2db53865e61390bcce6761727f26d8/datasette/views/table.py#L813-L912

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1460943097 https://github.com/simonw/datasette/pull/1999#issuecomment-1460943097 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XFDj5 simonw 9599 2023-03-08T22:09:24Z 2023-03-08T22:09:47Z OWNER

The ease with which I added that ?_extra=query feature in https://github.com/simonw/datasette/pull/1999/commits/96e94f9b7b2db53865e61390bcce6761727f26d8 made me feel really confident that this architecture is going in the right direction.

```diff diff --git a/datasette/views/table.py b/datasette/views/table.py index 8d3bb2c930..3e1db9c85f 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -1913,6 +1913,13 @@ async def extra_request(): "args": request.args._data, }

  • async def extra_query():
  • "Details of the underlying SQL query"
  • return {
  • "sql": sql,
  • "params": params,
  • } + async def extra_extras(): "Available ?_extra= blocks" return { @@ -1938,6 +1945,7 @@ async def extra_extras(): extra_primary_keys, extra_debug, extra_request,
  • extra_query, extra_extras, ) ```
 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1460916405 https://github.com/simonw/datasette/pull/1999#issuecomment-1460916405 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XE9C1 simonw 9599 2023-03-08T21:43:27Z 2023-03-08T21:43:27Z OWNER

Just noticed that _json=colname is not working, and that's because it's handled by the renderer here:

https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/renderer.py#L29-L40

But that's not currently being called by my new code.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1460907148 https://github.com/simonw/datasette/pull/1999#issuecomment-1460907148 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XE6yM simonw 9599 2023-03-08T21:34:30Z 2023-03-08T21:34:30Z OWNER

I'm going to hold off on that refactor until later, when I have tests to show me if the refactor works or not.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1460906741 https://github.com/simonw/datasette/pull/1999#issuecomment-1460906741 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XE6r1 simonw 9599 2023-03-08T21:34:08Z 2023-03-08T21:34:08Z OWNER

So maybe I can refactor it to look a bit more like this:

https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L1602-L1604

One thing that's useful here is that is_view is handled early, like this:

https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L466-L472

So if I omit the is_view bit from the extracted function I can simplify more.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1460905469 https://github.com/simonw/datasette/pull/1999#issuecomment-1460905469 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XE6X9 simonw 9599 2023-03-08T21:32:56Z 2023-03-08T21:32:56Z OWNER

I'd really like to extract this ugly logic out into a helper function:

https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L1617-L1684

I copied it in and asked ChatGPT to "List all of the variable that are either modified or assigned to by the above code":

Modified variables:

offset
where_clauses
params
order_by

Assigned variables:

sort_value
next_by_pk_clauses
order_by (when sort or sort_desc is not None)

Then I asked which variables were used as inputs, and argued with it a bit about whether it should be counting functions. Eventually got to this:

My apologies for the oversight. Here are the variables needed as input by the above code, excluding any functions:

_next
is_view
sort
sort_desc
use_rowid
pks
order_by

Note that use_rowid is actually defined earlier in terms of two of those other variables: https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L1540

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1460866243 https://github.com/simonw/datasette/issues/2036#issuecomment-1460866243 https://api.github.com/repos/simonw/datasette/issues/2036 IC_kwDOBm6k_c5XEwzD simonw 9599 2023-03-08T20:57:34Z 2023-03-08T20:57:34Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `publish cloudrun` reuses image tags, which can lead to very surprising deploy problems 1615862295  
1460848869 https://github.com/simonw/datasette/issues/2036#issuecomment-1460848869 https://api.github.com/repos/simonw/datasette/issues/2036 IC_kwDOBm6k_c5XEsjl simonw 9599 2023-03-08T20:40:55Z 2023-03-08T20:40:55Z OWNER

Here's the https://latest.datasette.io/ deployment that just went out, further demonstrating that this change is working correctly:

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `publish cloudrun` reuses image tags, which can lead to very surprising deploy problems 1615862295  
1460840620 https://github.com/simonw/datasette/issues/2037#issuecomment-1460840620 https://api.github.com/repos/simonw/datasette/issues/2037 IC_kwDOBm6k_c5XEqis simonw 9599 2023-03-08T20:33:00Z 2023-03-08T20:33:00Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Test failure: FAILED tests/test_cli.py::test_install_requirements - FileNotFoundError 1615891776  
1460838797 https://github.com/simonw/datasette/issues/2037#issuecomment-1460838797 https://api.github.com/repos/simonw/datasette/issues/2037 IC_kwDOBm6k_c5XEqGN simonw 9599 2023-03-08T20:31:15Z 2023-03-08T20:31:15Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Test failure: FAILED tests/test_cli.py::test_install_requirements - FileNotFoundError 1615891776  
1460838109 https://github.com/simonw/datasette/issues/2037#issuecomment-1460838109 https://api.github.com/repos/simonw/datasette/issues/2037 IC_kwDOBm6k_c5XEp7d simonw 9599 2023-03-08T20:30:36Z 2023-03-08T20:30:36Z OWNER

Instead of using isolated_filesystem() I could use a tmpdir fixture instead.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Test failure: FAILED tests/test_cli.py::test_install_requirements - FileNotFoundError 1615891776  
1460827178 https://github.com/simonw/datasette/issues/2036#issuecomment-1460827178 https://api.github.com/repos/simonw/datasette/issues/2036 IC_kwDOBm6k_c5XEnQq simonw 9599 2023-03-08T20:25:10Z 2023-03-08T20:25:10Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `publish cloudrun` reuses image tags, which can lead to very surprising deploy problems 1615862295  
1460816528 https://github.com/simonw/datasette/issues/2036#issuecomment-1460816528 https://api.github.com/repos/simonw/datasette/issues/2036 IC_kwDOBm6k_c5XEkqQ simonw 9599 2023-03-08T20:22:50Z 2023-03-08T20:23:20Z OWNER

Testing this manually:

``` % datasette publish cloudrun content.db --service new-service Creating temporary tarball archive of 2 file(s) totalling 13.8 MiB before compression. Uploading tarball of [.] to [gs://datasette-222320_cloudbuild/source/1678306859.271661-805303f364144b6094cc9c8532ab5133.tgz] Created [https://cloudbuild.googleapis.com/v1/projects/datasette-222320/locations/global/builds/290f41a4-e29a-443c-a1e5-c54513c6143d]. Logs are available at [ https://console.cloud.google.com/cloud-build/builds/290f41a4-e29a-443c-a1e5-c54513c6143d?project=99025868001 ]. ---- REMOTE BUILD OUTPUT ---- starting build "290f41a4-e29a-443c-a1e5-c54513c6143d"

FETCHSOURCE Fetching storage object: gs://datasette-222320_cloudbuild/source/1678306859.271661-805303f364144b6094cc9c8532ab5133.tgz#1678306862810483 Copying gs://datasette-222320_cloudbuild/source/1678306859.271661-805303f364144b6094cc9c8532ab5133.tgz#1678306862810483... / [1 files][ 3.9 MiB/ 3.9 MiB]
Operation completed over 1 objects/3.9 MiB. BUILD Already have image (with digest): gcr.io/cloud-builders/docker Sending build context to Docker daemon 14.52MB Step 1/9 : FROM python:3.11.0-slim-bullseye ... Installing collected packages: rfc3986, typing-extensions, sniffio, PyYAML, python-multipart, pluggy, pint, mergedeep, MarkupSafe, itsdangerous, idna, hupper, h11, click, certifi, asgiref, aiofiles, uvicorn, Jinja2, janus, click-default-group-wheel, asgi-csrf, anyio, httpcore, httpx, datasette Successfully installed Jinja2-3.1.2 MarkupSafe-2.1.2 PyYAML-6.0 aiofiles-23.1.0 anyio-3.6.2 asgi-csrf-0.9 asgiref-3.6.0 certifi-2022.12.7 click-8.1.3 click-default-group-wheel-1.2.2 datasette-0.64.1 h11-0.14.0 httpcore-0.16.3 httpx-0.23.3 hupper-1.11 idna-3.4 itsdangerous-2.1.2 janus-1.0.0 mergedeep-1.3.4 pint-0.20.1 pluggy-1.0.0 python-multipart-0.0.6 rfc3986-1.5.0 sniffio-1.3.0 typing-extensions-4.5.0 uvicorn-0.20.0 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip available: 22.3 -> 23.0.1 [notice] To update, run: pip install --upgrade pip Removing intermediate container 8ccebfebebc9 ---> b972c85b38bb ... Successfully built 606b7c286d7f Successfully tagged gcr.io/datasette-222320/datasette-new-service:latest PUSH Pushing gcr.io/datasette-222320/datasette-new-service The push refers to repository [gcr.io/datasette-222320/datasette-new-service] 667b1dc69e5e: Preparing ... d8ddfcff216f: Pushed latest: digest: sha256:452daffb2d3d7a8579c2ab39854be285155252c9428b4c1c50caac6a3a269e3f size: 2004 DONE

ID CREATE_TIME DURATION SOURCE IMAGES STATUS 290f41a4-e29a-443c-a1e5-c54513c6143d 2023-03-08T20:21:03+00:00 39S gs://datasette-222320_cloudbuild/source/1678306859.271661-805303f364144b6094cc9c8532ab5133.tgz gcr.io/datasette-222320/datasette-new-service (+1 more) SUCCESS Deploying container to Cloud Run service [new-service] in project [datasette-222320] region [us-central1] ✓ Deploying new service... Done.
✓ Creating Revision...
✓ Routing traffic...
✓ Setting IAM Policy...
Done.
Service [new-service] revision [new-service-00001-zon] has been deployed and is serving 100 percent of traffic. Service URL: https://new-service-j7hipcg4aq-uc.a.run.app ``` https://new-service-j7hipcg4aq-uc.a.run.app/ was deployed successfully.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `publish cloudrun` reuses image tags, which can lead to very surprising deploy problems 1615862295  
1460810523 https://github.com/simonw/datasette/issues/2036#issuecomment-1460810523 https://api.github.com/repos/simonw/datasette/issues/2036 IC_kwDOBm6k_c5XEjMb simonw 9599 2023-03-08T20:17:01Z 2023-03-08T20:17:01Z OWNER

I'm going to solve this by using the service name in that image_id instead:

python image_id = f"gcr.io/{project}/{service_name}" This is a nasty bug, so I'm going to backport it to a 0.64.2 release as well.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `publish cloudrun` reuses image tags, which can lead to very surprising deploy problems 1615862295  
1460809643 https://github.com/simonw/datasette/issues/2036#issuecomment-1460809643 https://api.github.com/repos/simonw/datasette/issues/2036 IC_kwDOBm6k_c5XEi-r simonw 9599 2023-03-08T20:16:10Z 2023-03-08T20:16:10Z OWNER

I think the code at fault is here:

https://github.com/simonw/datasette/blob/1ad92a1d87d79084ebe524ed186c900ff042328c/datasette/publish/cloudrun.py#L176-L182

That name ends up defaulting to datasette - so multiple different projects may end up deploying to the same image_id.

What I think happened in the datasette.io bug is that this workflow: https://github.com/simonw/simonwillisonblog-backup/blob/bfb573e96d8622ab52b22fdcd54724fe6e59fd24/.github/workflows/backup.yml and this workflow: https://github.com/simonw/datasette.io/blob/4676db5bf4a3fc9f792ee270ec0c59eb902cd2c3/.github/workflows/deploy.yml both happened to run at the exact same time.

And so the image that was pushed to gcr.io/datasette-222320/datasette:latest by the simonw/simonwillisonblog-backup action was then deployed by the simonw/datasette.io/ action, which broke the site.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `publish cloudrun` reuses image tags, which can lead to very surprising deploy problems 1615862295  
1460760116 https://github.com/simonw/datasette/pull/1999#issuecomment-1460760116 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XEW40 simonw 9599 2023-03-08T19:48:52Z 2023-03-08T19:48:52Z OWNER

I'm trying to get http://127.0.0.1:8001/fixtures/compound_three_primary_keys?_next=a,d,v to return the correct results.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1460682625 https://github.com/simonw/datasette/issues/2035#issuecomment-1460682625 https://api.github.com/repos/simonw/datasette/issues/2035 IC_kwDOBm6k_c5XED-B simonw 9599 2023-03-08T18:40:57Z 2023-03-08T18:40:57Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Potential feature: special support for `?a=1&a=2` on the query page 1615692818  
1460679434 https://github.com/simonw/datasette/issues/2035#issuecomment-1460679434 https://api.github.com/repos/simonw/datasette/issues/2035 IC_kwDOBm6k_c5XEDMK simonw 9599 2023-03-08T18:39:35Z 2023-03-08T18:39:35Z OWNER

I should consider the existing design of magic parameters here: https://docs.datasette.io/en/stable/sql_queries.html#magic-parameters

  • _actor_*
  • _header_*
  • _cookie_
  • _now_epoch
  • _now_date_utc
  • _now_datetime_utc
  • _random_chars_*

Should this new id__list syntax look more like those magic parameters, or is it OK to use name__magic syntax here instead?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Potential feature: special support for `?a=1&a=2` on the query page 1615692818  
1460668431 https://github.com/simonw/datasette/issues/2035#issuecomment-1460668431 https://api.github.com/repos/simonw/datasette/issues/2035 IC_kwDOBm6k_c5XEAgP simonw 9599 2023-03-08T18:35:34Z 2023-03-08T18:35:34Z OWNER

To implement this properly need to do the following: - Get the page to display multiple id: [ text input here ] fields such that re-submission works - Figure out how this should work for canned queries and for writable canned queries - Tests that cover queries, canned queries, writable canned queries

And a bonus feature: what if the Datasette UI layer spotted :id__list parameters and used them to add a bit of JavaScript that allowed users to click a + button next to an id form field to add another one?

Also, when a page is re-displayed for on of these queries it could potentially add an extra form field allowing people to add another value.

Though this has an annoying problem: how to tell the difference between an additional id input field that the user chose not to populate, v.s. one that is supposed to represent an empty string?

Maybe only support multiple id fields for users with JavaScript in order to avoid this problem.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Potential feature: special support for `?a=1&a=2` on the query page 1615692818  
1460664619 https://github.com/simonw/datasette/issues/2035#issuecomment-1460664619 https://api.github.com/repos/simonw/datasette/issues/2035 IC_kwDOBm6k_c5XD_kr simonw 9599 2023-03-08T18:32:29Z 2023-03-08T18:32:29Z OWNER

Got a prototype working: ```diff diff --git a/datasette/views/database.py b/datasette/views/database.py index 8d289105..6f9d8a44 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -226,6 +226,12 @@ class QueryView(DataView): ): db = await self.ds.resolve_database(request) database = db.name + # Disallow x__list query string parameters + invalid_params = [k for k in request.args if k.endswith("__list")] + if invalid_params: + raise DatasetteError( + "Invalid query string parameters: {}".format(", ".join(invalid_params)) + ) params = {key: request.args.get(key) for key in request.args} if "sql" in params: params.pop("sql") @@ -258,6 +264,11 @@ class QueryView(DataView): for named_parameter in named_parameters if not named_parameter.startswith("_") } + # Handle any __list parameters + for named_parameter in named_parameters: + if named_parameter.endswith("__list"): + list_values = request.args.getlist(named_parameter[:-6]) + params[named_parameter] = json.dumps(list_values)

     # Set to blank string if missing from params
     for named_parameter in named_parameters:

`` This isn't yet doing the right thing on form re-submission: it breaks because it attempts to pass through the?id__list=` invalid parameter. But I did manage to get it to do this through careful editing of the URL:

That was this URL: http://127.0.0.1:8034/content?sql=select+%3Aid__list%2C*+from+releases+where+id+in+(select+value+from+json_each(%3Aid__list))&id=62642726&id=18402901&id=38714866

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Potential feature: special support for `?a=1&a=2` on the query page 1615692818  
1460659382 https://github.com/simonw/datasette/issues/2035#issuecomment-1460659382 https://api.github.com/repos/simonw/datasette/issues/2035 IC_kwDOBm6k_c5XD-S2 simonw 9599 2023-03-08T18:28:00Z 2023-03-08T18:28:00Z OWNER

Also: datasette-explain may need to be updated to understand how to handle this:

ERROR: conn=<sqlite3.Connection object at 0x102834940>, sql = 'explain select * from releases where id in (select id from json_each(:id__list))', params = None: You did not supply a value for binding parameter :id__list.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Potential feature: special support for `?a=1&a=2` on the query page 1615692818  
1460654136 https://github.com/simonw/datasette/issues/2035#issuecomment-1460654136 https://api.github.com/repos/simonw/datasette/issues/2035 IC_kwDOBm6k_c5XD9A4 simonw 9599 2023-03-08T18:25:46Z 2023-03-08T18:25:46Z OWNER

Trickiest part of the implementation here is that it needs to know to output three id HTML form fields on the page, such that their values are persisted when the form is submitted a second time.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Potential feature: special support for `?a=1&a=2` on the query page 1615692818  
1460639749 https://github.com/simonw/datasette/issues/2035#issuecomment-1460639749 https://api.github.com/repos/simonw/datasette/issues/2035 IC_kwDOBm6k_c5XD5gF simonw 9599 2023-03-08T18:17:31Z 2023-03-08T18:17:31Z OWNER

Since we are pre-1.0 it's still OK to implement a feature that disallows ?id__list= in the URL, but allows :id__list in SQL queries to reference the JSON list of parameters.

So I'm going to prototype this as the :id__list feature and see how it feels.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Potential feature: special support for `?a=1&a=2` on the query page 1615692818  
1460637906 https://github.com/simonw/datasette/issues/2035#issuecomment-1460637906 https://api.github.com/repos/simonw/datasette/issues/2035 IC_kwDOBm6k_c5XD5DS simonw 9599 2023-03-08T18:16:31Z 2023-03-08T18:16:31Z OWNER

I'm pretty sold on this as a feature now. The main question I have is which of these options to implement:

  1. ?id=1&?id=2 results in :id in the query being ["1", "2"] - no additional syntax required
  2. :id in the query continues to reference just the first of those parameters - but :id__list (or some other custom syntax) instead gets ["1", "2"] - or, if the URL is ?id=1 - gets ["1"]

Actually on writing these out I realize that option 2 is the ONLY valid option. It's no good building a query that works against a JSON list if the user might pass just a single ID, ?id=1, resulting in their query breaking.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Potential feature: special support for `?a=1&a=2` on the query page 1615692818  
1460632758 https://github.com/simonw/datasette/issues/2035#issuecomment-1460632758 https://api.github.com/repos/simonw/datasette/issues/2035 IC_kwDOBm6k_c5XD3y2 simonw 9599 2023-03-08T18:13:49Z 2023-03-08T18:13:49Z OWNER

https://github.com/rclement/datasette-dashboards/issues/54 makes the excellent point that the <select multiple> default HTML widget produces this exact format of query string:

```html

<form action="https://www.example.com/"> <select multiple name="id"> <option>21</option> <option>32</option> <option>15</option> <option>63</option> </select> </form>

`` Submitting that form with the middle two options selected navigates to:https://www.example.com/?id=32&id=15`

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Potential feature: special support for `?a=1&a=2` on the query page 1615692818  
1460628199 https://github.com/simonw/datasette/issues/2035#issuecomment-1460628199 https://api.github.com/repos/simonw/datasette/issues/2035 IC_kwDOBm6k_c5XD2rn simonw 9599 2023-03-08T18:11:31Z 2023-03-08T18:11:31Z OWNER

One variant on this idea: maybe you have to specify in your query that you want it to be the JSON list version, not the single item (first ?id= parameter version)? Maybe with syntax like this:

where id in (select value from json_each(:id__list))

Datasette would automatically pass {"id": "11", "id__list": '["11", "32", "62"]'} as arguments to the db.execute() method, if the page was called with ?id=11&id=32&id=62.

This is more explicit, though the syntax is a bit uglier (maybe there's a nicer design for this?). I also worry about ?id__list= conflicting with this, but I think that's a risk I can take - tell people not to do that, or even block ?id__list= style parameters entirely.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Potential feature: special support for `?a=1&a=2` on the query page 1615692818  
1460621871 https://github.com/simonw/datasette/issues/2035#issuecomment-1460621871 https://api.github.com/repos/simonw/datasette/issues/2035 IC_kwDOBm6k_c5XD1Iv simonw 9599 2023-03-08T18:08:25Z 2023-03-08T18:09:04Z OWNER

My current preferred solution is to lean into SQLite's JSON support.

What if the query page spotted ?id=11&id=32&id=62 and turned that into a JSON string called :id: with a value of ["11", "32", "62"]?

Note that this is still a string, not a list. This avoids a nasty problem that occurred in PHP world, where ?id[]=1&id[]=2 would result in an actual PHP array object, which often broke underlying code that had expected $_GET["id"] to be a string, not an array.

So in a query you'd be able to do this:

where id in (select value from json_each(:id))

And then call it with ?id=11&id=32&id=62.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Potential feature: special support for `?a=1&a=2` on the query page 1615692818  
1460618433 https://github.com/simonw/datasette/issues/2035#issuecomment-1460618433 https://api.github.com/repos/simonw/datasette/issues/2035 IC_kwDOBm6k_c5XD0TB simonw 9599 2023-03-08T18:06:34Z 2023-03-08T18:06:34Z OWNER

One way to do this would be to dynamically generate the where id in (?, ?, ?) with the correct number of question marks, then feed in a list from request.args.getlist("id") - but that would require rewriting the SQL query text to add those question marks.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Potential feature: special support for `?a=1&a=2` on the query page 1615692818  
1457117383 https://github.com/simonw/datasette/issues/2033#issuecomment-1457117383 https://api.github.com/repos/simonw/datasette/issues/2033 IC_kwDOBm6k_c5W2djH simonw 9599 2023-03-06T22:28:55Z 2023-03-06T22:28:55Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `datasette install -r requirements.txt` 1612296210  
1456997425 https://github.com/simonw/datasette/pull/2031#issuecomment-1456997425 https://api.github.com/repos/simonw/datasette/issues/2031 IC_kwDOBm6k_c5W2AQx simonw 9599 2023-03-06T21:04:27Z 2023-03-06T21:06:34Z OWNER

This is a very neat fix, for something I've been wanting for a while.

Add a unit test for the row HTML page - I suggest against this page: https://latest.datasette.io/fixtures/foreign_key_references/1 - and I'll land this PR.

You can model it on this test here: https://github.com/simonw/datasette/blob/a53b893c46453f35decc8c145c138671cee6140c/tests/test_table_html.py#L609-L632

I think adding it to test_table_html.py is OK, even though it's technically for the row page and not the table page.

Thanks!

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Expand foreign key references in row view as well 1605481359  
1456914694 https://github.com/simonw/datasette/pull/2028#issuecomment-1456914694 https://api.github.com/repos/simonw/datasette/issues/2028 IC_kwDOBm6k_c5W1sEG simonw 9599 2023-03-06T20:19:37Z 2023-03-06T20:19:37Z OWNER

Thanks!

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 add Python 3.11 classifier 1590839187  
1426031395 https://github.com/simonw/datasette/issues/2024#issuecomment-1426031395 https://api.github.com/repos/simonw/datasette/issues/2024 IC_kwDOBm6k_c5U_4Mj simonw 9599 2023-02-10T16:11:53Z 2023-02-10T16:11:53Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Mention WAL mode in documentation 1579973223  
1423067724 https://github.com/simonw/datasette/issues/262#issuecomment-1423067724 https://api.github.com/repos/simonw/datasette/issues/262 IC_kwDOBm6k_c5U0kpM simonw 9599 2023-02-08T18:33:32Z 2023-02-08T18:36:48Z OWNER

Just realized that it's useful to be able to tell what parameters were used to generate a page... but reflecting things like _next back in the JSON is confusing in the presence of next.

So I'm going to add an extra for that information too.

Not sure what to call it though:

  • params - confusing because in the code that's usually used for params passed to SQL queries
  • query_string - wouldn't that be a string, not params as a dictionary?

I'm going to experiment with a request extra that returns some bits of information about the request.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Add ?_extra= mechanism for requesting extra properties in JSON 323658641  
1421988953 https://github.com/simonw/datasette/pull/1999#issuecomment-1421988953 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5UwdRZ simonw 9599 2023-02-08T04:35:44Z 2023-02-08T05:27:48Z OWNER

Next step: get ?_next=... working (it is ignored at the moment, even though the returned JSON includes the "next" key).

Then... figure out how to render HTML and other requested formats.

Then get the tests to pass!

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1421784930 https://github.com/simonw/datasette/issues/2019#issuecomment-1421784930 https://api.github.com/repos/simonw/datasette/issues/2019 IC_kwDOBm6k_c5Uvrdi simonw 9599 2023-02-08T01:28:25Z 2023-02-08T01:40:46Z OWNER

Rather than duplicate this rather awful hack:

https://github.com/simonw/datasette/blob/0b4a28691468b5c758df74fa1d72a823813c96bf/datasette/views/table.py#L694-L714

I'm tempted to say that the code that calls the new pagination helper needs to ensure that the sort or sort_desc columns are selected. If it wants to ditch them later (e.g. because they were not included in ?_col=) it can do that later once the results have come back.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Refactor out the keyset pagination code 1573424830  
1421600789 https://github.com/simonw/datasette/issues/2019#issuecomment-1421600789 https://api.github.com/repos/simonw/datasette/issues/2019 IC_kwDOBm6k_c5Uu-gV simonw 9599 2023-02-07T23:12:40Z 2023-02-07T23:16:20Z OWNER

Most complicated example of a paginated query: https://latest.datasette.io/fixtures?sql=select%0D%0A++pk1%2C%0D%0A++pk2%2C%0D%0A++content%2C%0D%0A++sortable%2C%0D%0A++sortable_with_nulls%2C%0D%0A++sortable_with_nulls_2%2C%0D%0A++text%0D%0Afrom%0D%0A++sortable%0D%0Awhere%0D%0A++(%0D%0A++++sortable_with_nulls+is+null%0D%0A++++and+(%0D%0A++++++(pk1+%3E+%3Ap0)%0D%0A++++++or+(%0D%0A++++++++pk1+%3D+%3Ap0%0D%0A++++++++and+pk2+%3E+%3Ap1%0D%0A++++++)%0D%0A++++)%0D%0A++)%0D%0Aorder+by%0D%0A++sortable_with_nulls+desc%2C%0D%0A++pk1%2C%0D%0A++pk2%0D%0Alimit%0D%0A++101&p0=h&p1=r

sql select pk1, pk2, content, sortable, sortable_with_nulls, sortable_with_nulls_2, text from sortable where ( sortable_with_nulls is null and ( (pk1 > :p0) or ( pk1 = :p0 and pk2 > :p1 ) ) ) order by sortable_with_nulls desc, pk1, pk2 limit 101 Generated by this page: https://latest.datasette.io/fixtures/sortable?_next=%24null%2Ch%2Cr&_sort_desc=sortable_with_nulls

The _next= parameter there decodes as $null,h,r - and those components are tilde-encoded, so this can be distinguished from an actual $null value which would be represented as ~24null.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Refactor out the keyset pagination code 1573424830  
1421274434 https://github.com/simonw/datasette/issues/2019#issuecomment-1421274434 https://api.github.com/repos/simonw/datasette/issues/2019 IC_kwDOBm6k_c5Utu1C simonw 9599 2023-02-07T18:42:42Z 2023-02-07T18:42:42Z OWNER

I'm going to build completely separate tests for this in test_pagination.py.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Refactor out the keyset pagination code 1573424830  
1420109153 https://github.com/simonw/datasette/issues/2019#issuecomment-1420109153 https://api.github.com/repos/simonw/datasette/issues/2019 IC_kwDOBm6k_c5UpSVh simonw 9599 2023-02-07T02:32:36Z 2023-02-07T02:32:36Z OWNER

Doing this as a class makes sense to me. There are a few steps:

  • Instantiate the class with the information it needs, which includes sort order, page size, tiebreaker columns and SQL query and parameters
  • Generate the new SQL query that will actually be executed - maybe this takes the optional _next parameter? This returns the SQL and params that should be executed, where the SQL now includes pagination logic plus order by and limit
  • The calling code then gets to execute the SQL query to fetch the rows
  • Last step: those rows are passed to a paginator method which returns (rows, next) - where rows is the rows truncated to the correct length (really just with the last one cut off if it's too long for the length) and next is either None or a token, depending on if there should be a next page.
 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Refactor out the keyset pagination code 1573424830  
1420106315 https://github.com/simonw/datasette/issues/2019#issuecomment-1420106315 https://api.github.com/repos/simonw/datasette/issues/2019 IC_kwDOBm6k_c5UpRpL simonw 9599 2023-02-07T02:28:03Z 2023-02-07T02:28:36Z OWNER

So I think I can write an abstraction that applies keyset pagination to ANY arbitrary SQL query provided it is given the query, the existing params (so it can pick names for the new params that won't overlap with them), the desired sort order, any existing _next token AND the columns that should be used to tie-break any duplicates.

Those tie breakers will be either the primary key(s) or rowid if none are provided.

What about the case of SQL views, where offset/limit should be used instead? I'm inclined to have that as a separate pagination abstraction entirely, with the calling code deciding which pagination helper to use based on if keyset pagination makes sense or not.

Might be easier to design a class structure for this starting with OffsetPaginator, then using that to inform the design of KeysetPaginator.

Might put these in datasette.utils.pagination to start off with, then maybe extract them out to sqlite-utils later once they've proven themselves.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Refactor out the keyset pagination code 1573424830  
1420104254 https://github.com/simonw/datasette/issues/2019#issuecomment-1420104254 https://api.github.com/repos/simonw/datasette/issues/2019 IC_kwDOBm6k_c5UpRI- simonw 9599 2023-02-07T02:24:46Z 2023-02-07T02:24:46Z OWNER

Even more complicated: https://latest.datasette.io/fixtures/sortable?sortable_with_nulls__notnull=1&_next=0~2E692704598586882%2Ce%2Cr&_sort=sortable_with_nulls_2

The rewritten SQL for that is:

sql select * from (select pk1, pk2, content, sortable, sortable_with_nulls, sortable_with_nulls_2, text from sortable where "sortable_with_nulls" is not null) where (sortable_with_nulls_2 > :p2 or (sortable_with_nulls_2 = :p2 and ((pk1 > :p0) or (pk1 = :p0 and pk2 > :p1)))) order by sortable_with_nulls_2, pk1, pk2 limit 101 And it still has the same number of explain steps as the current SQL witohut the subselect.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Refactor out the keyset pagination code 1573424830  
1420101175 https://github.com/simonw/datasette/issues/2019#issuecomment-1420101175 https://api.github.com/repos/simonw/datasette/issues/2019 IC_kwDOBm6k_c5UpQY3 simonw 9599 2023-02-07T02:22:11Z 2023-02-07T02:22:11Z OWNER

A more complex example: https://latest.datasette.io/fixtures/sortable?_next=0~2E2650566289400591%2Ca%2Cu&_sort=sortable_with_nulls_2

SQL:

sql select pk1, pk2, content, sortable, sortable_with_nulls, sortable_with_nulls_2, text from sortable where (sortable_with_nulls_2 > :p2 or (sortable_with_nulls_2 = :p2 and ((pk1 > :p0) or (pk1 = :p0 and pk2 > :p1)))) order by sortable_with_nulls_2, pk1, pk2 limit 101

https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+content%2C+sortable%2C+sortable_with_nulls%2C+sortable_with_nulls_2%2C+text+from+sortable+where+%28sortable_with_nulls_2+%3E+%3Ap2+or+%28sortable_with_nulls_2+%3D+%3Ap2+and+%28%28pk1+%3E+%3Ap0%29%0A++or%0A%28pk1+%3D+%3Ap0+and+pk2+%3E+%3Ap1%29%29%29%29+order+by+sortable_with_nulls_2%2C+pk1%2C+pk2+limit+101&p0=a&p1=u&p2=0.2650566289400591

Here's the explain: 49 steps long https://latest.datasette.io/fixtures?sql=explain+select+pk1%2C+pk2%2C+content%2C+sortable%2C+sortable_with_nulls%2C+sortable_with_nulls_2%2C+text+from+sortable+where+%28sortable_with_nulls_2+%3E+%3Ap2+or+%28sortable_with_nulls_2+%3D+%3Ap2+and+%28%28pk1+%3E+%3Ap0%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3E+%3Ap1%29%29%29%29+order+by+sortable_with_nulls_2%2C+pk1%2C+pk2+limit+101&p2=0.2650566289400591&p0=a&p1=u

Rewritten with a subselect:

sql select * from ( select pk1, pk2, content, sortable, sortable_with_nulls, sortable_with_nulls_2, text from sortable ) where (sortable_with_nulls_2 > :p2 or (sortable_with_nulls_2 = :p2 and ((pk1 > :p0) or (pk1 = :p0 and pk2 > :p1)))) order by sortable_with_nulls_2, pk1, pk2 limit 101 https://latest.datasette.io/fixtures?sql=select+*+from+(%0D%0A++select+pk1%2C+pk2%2C+content%2C+sortable%2C+sortable_with_nulls%2C+sortable_with_nulls_2%2C+text+from+sortable%0D%0A)%0D%0Awhere+(sortable_with_nulls_2+%3E+%3Ap2+or+(sortable_with_nulls_2+%3D+%3Ap2+and+((pk1+%3E+%3Ap0)%0D%0A++or%0D%0A(pk1+%3D+%3Ap0+and+pk2+%3E+%3Ap1))))+order+by+sortable_with_nulls_2%2C+pk1%2C+pk2+limit+101&p2=0.2650566289400591&p0=a&p1=u

And here's the explain for that - also 49 steps: https://latest.datasette.io/fixtures?sql=explain+select+*+from+%28%0D%0A++select+pk1%2C+pk2%2C+content%2C+sortable%2C+sortable_with_nulls%2C+sortable_with_nulls_2%2C+text+from+sortable%0D%0A%29%0D%0Awhere+%28sortable_with_nulls_2+%3E+%3Ap2+or+%28sortable_with_nulls_2+%3D+%3Ap2+and+%28%28pk1+%3E+%3Ap0%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3E+%3Ap1%29%29%29%29+order+by+sortable_with_nulls_2%2C+pk1%2C+pk2+limit+101&p2=0.2650566289400591&p0=a&p1=u

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Refactor out the keyset pagination code 1573424830  
1420094396 https://github.com/simonw/datasette/issues/2019#issuecomment-1420094396 https://api.github.com/repos/simonw/datasette/issues/2019 IC_kwDOBm6k_c5UpOu8 simonw 9599 2023-02-07T02:18:11Z 2023-02-07T02:19:16Z OWNER

For the SQL underlying this page (the second page in that compound primary key paginated sequence): https://latest.datasette.io/fixtures/compound_three_primary_keys?_next=a%2Cd%2Cv

The explain for the default query: https://latest.datasette.io/fixtures?sql=explain+select%0D%0A++pk1%2C%0D%0A++pk2%2C%0D%0A++pk3%2C%0D%0A++content%0D%0Afrom%0D%0A++compound_three_primary_keys%0D%0Awhere%0D%0A++%28%0D%0A++++%28pk1+%3E+%3Ap0%29%0D%0A++++or+%28%0D%0A++++++pk1+%3D+%3Ap0%0D%0A++++++and+pk2+%3E+%3Ap1%0D%0A++++%29%0D%0A++++or+%28%0D%0A++++++pk1+%3D+%3Ap0%0D%0A++++++and+pk2+%3D+%3Ap1%0D%0A++++++and+pk3+%3E+%3Ap2%0D%0A++++%29%0D%0A++%29%0D%0Aorder+by%0D%0A++pk1%2C%0D%0A++pk2%2C%0D%0A++pk3%0D%0Alimit%0D%0A++101&p0=a&p1=d&p2=v

The explain for that query rewritten as this:

sql explain select * from ( select pk1, pk2, pk3, content from compound_three_primary_keys ) where ( (pk1 > :p0) or ( pk1 = :p0 and pk2 > :p1 ) or ( pk1 = :p0 and pk2 = :p1 and pk3 > :p2 ) ) order by pk1, pk2, pk3 limit 101 https://latest.datasette.io/fixtures?sql=explain+select+*+from+%28select+%0D%0A++pk1%2C%0D%0A++pk2%2C%0D%0A++pk3%2C%0D%0A++content%0D%0Afrom%0D%0A++compound_three_primary_keys%0D%0A%29%0D%0A++where%0D%0A++%28%0D%0A++++%28pk1+%3E+%3Ap0%29%0D%0A++++or+%28%0D%0A++++++pk1+%3D+%3Ap0%0D%0A++++++and+pk2+%3E+%3Ap1%0D%0A++++%29%0D%0A++++or+%28%0D%0A++++++pk1+%3D+%3Ap0%0D%0A++++++and+pk2+%3D+%3Ap1%0D%0A++++++and+pk3+%3E+%3Ap2%0D%0A++++%29%0D%0A++%29%0D%0Aorder+by%0D%0A++pk1%2C%0D%0A++pk2%2C%0D%0A++pk3%0D%0Alimit%0D%0A++101&p0=a&p1=d&p2=v

Both explains have 31 steps and look pretty much identical.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Refactor out the keyset pagination code 1573424830  
1420088670 https://github.com/simonw/datasette/issues/2019#issuecomment-1420088670 https://api.github.com/repos/simonw/datasette/issues/2019 IC_kwDOBm6k_c5UpNVe simonw 9599 2023-02-07T02:14:35Z 2023-02-07T02:14:35Z OWNER

Maybe the correct level of abstraction here is that pagination is something that happens to a SQL query that is defined as SQL and params, without an order by or limit. That's then wrapped in a sub-select and those things are added to it, plus the necessary where clauses depending on the page.

Need to check that the query plan for pagination of a subquery isn't slower than the plan for pagination as it works today.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Refactor out the keyset pagination code 1573424830  
1419953256 https://github.com/simonw/datasette/issues/2019#issuecomment-1419953256 https://api.github.com/repos/simonw/datasette/issues/2019 IC_kwDOBm6k_c5UosRo simonw 9599 2023-02-06T23:42:56Z 2023-02-06T23:43:10Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Refactor out the keyset pagination code 1573424830  
1419928455 https://github.com/simonw/datasette/issues/2019#issuecomment-1419928455 https://api.github.com/repos/simonw/datasette/issues/2019 IC_kwDOBm6k_c5UomOH simonw 9599 2023-02-06T23:21:50Z 2023-02-06T23:21:50Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Refactor out the keyset pagination code 1573424830  
1419921228 https://github.com/simonw/datasette/issues/2019#issuecomment-1419921228 https://api.github.com/repos/simonw/datasette/issues/2019 IC_kwDOBm6k_c5UokdM simonw 9599 2023-02-06T23:14:15Z 2023-02-06T23:14:15Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Refactor out the keyset pagination code 1573424830  
1419917661 https://github.com/simonw/datasette/issues/2019#issuecomment-1419917661 https://api.github.com/repos/simonw/datasette/issues/2019 IC_kwDOBm6k_c5Uojld simonw 9599 2023-02-06T23:10:51Z 2023-02-06T23:10:51Z OWNER

I should turn sort and sort_desc into an object representing the sort order earlier in the code.

I should also create something that bundles together pks and use_rowid and maybe is_view as well.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Refactor out the keyset pagination code 1573424830  
1419916684 https://github.com/simonw/datasette/issues/2019#issuecomment-1419916684 https://api.github.com/repos/simonw/datasette/issues/2019 IC_kwDOBm6k_c5UojWM simonw 9599 2023-02-06T23:09:51Z 2023-02-06T23:10:13Z OWNER

The inputs and outputs for this are pretty complex.

Inputs:

  • ?_next= from the query string
  • is_view - is this for a table or view? If it's a view it uses offset/limit pagination - which could actually work for arbitrary queries too. Also views could have keyset pagination if they are known to be sorted by a particular column.
  • sort and sort_desc reflecting the current sort order
  • use_rowid for if the table is a rowid table with no primary key of its own
  • pks - the primary keys for the table
  • params - the current set of parameters, I think used just to count their length so new params can be added as p5 etc without collisions. This could be handled with a s0, s1 etc naming convention instead.

Outputs:

  • where_clauses - a list of where clauses to add to the query
  • params - additional parameters to use with the query due to the new where clauses
  • order_by - the order by clause
 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Refactor out the keyset pagination code 1573424830  
1399343659 https://github.com/simonw/datasette/pull/1999#issuecomment-1399343659 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5TaEor simonw 9599 2023-01-21T22:19:20Z 2023-02-06T23:02:12Z OWNER

HTML mode needs a list of renderers so it can show links to .geojson etc - can do that as a hidden extra (maybe called renderers), repeating this code:

https://github.com/simonw/datasette/blob/e4ebef082de90db4e1b8527abc0d582b7ae0bc9d/datasette/views/base.py#L477-L497

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1418288327 https://github.com/simonw/datasette/issues/262#issuecomment-1418288327 https://api.github.com/repos/simonw/datasette/issues/262 IC_kwDOBm6k_c5UiVzH simonw 9599 2023-02-05T22:57:58Z 2023-02-06T23:01:15Z OWNER

I think that does make sense: ?_extra=table perhaps, which would add {"table": "..."}.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Add ?_extra= mechanism for requesting extra properties in JSON 323658641  
1418288077 https://github.com/simonw/datasette/issues/2016#issuecomment-1418288077 https://api.github.com/repos/simonw/datasette/issues/2016 IC_kwDOBm6k_c5UiVvN simonw 9599 2023-02-05T22:56:43Z 2023-02-05T22:56:43Z OWNER

This absolutely makes sense. One of the biggest goals for Datasette 1.0 is "documented template contexts" - for any default template in Datasette that people might want to over-ride there should be documentation that describes the available context variables, plus tests that ensure they don't accidentally get broken by future changes.

Ensuring description/title/etc are available on the index page feels like it fits well into that bucket.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Database metadata fields like description are not available in the index page template's context 1571207083  
1410827249 https://github.com/simonw/datasette/issues/2011#issuecomment-1410827249 https://api.github.com/repos/simonw/datasette/issues/2011 IC_kwDOBm6k_c5UF4Px simonw 9599 2023-01-31T17:58:54Z 2023-01-31T17:58:54Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Applied facet did not result in an "x" icon to dismiss it 1564769997  
1409406327 https://github.com/simonw/datasette/issues/2010#issuecomment-1409406327 https://api.github.com/repos/simonw/datasette/issues/2010 IC_kwDOBm6k_c5UAdV3 simonw 9599 2023-01-30T21:51:58Z 2023-01-30T21:51:58Z OWNER

Here's a quick prototype I knocked up for this: ```diff diff --git a/datasette/static/app.css b/datasette/static/app.css index 71437bd4..d763bcff 100644 --- a/datasette/static/app.css +++ b/datasette/static/app.css @@ -695,7 +695,48 @@ p.zero-results {

+/ Force table to not be like tables anymore / +body.row table.rows-and-columns, +body.row .rows-and-columns thead, +body.row .rows-and-columns tbody, +body.row .rows-and-columns th, +body.row .rows-and-columns td, +body.row .rows-and-columns tr { + display: block; +} + +/ Hide table headers (but not display: none;, for accessibility) / +body.row .rows-and-columns thead tr { + position: absolute; + top: -9999px; + left: -9999px; +} + +body.row .rows-and-columns tr { + border: 1px solid #ccc; + margin-bottom: 1em; + border-radius: 10px; + background-color: white; + padding: 0.2rem; +}

+body.row .rows-and-columns td { + / Behave like a "row" / + border: none; + border-bottom: 1px solid #eee; + padding: 0; + padding-left: 10%; + padding-bottom: 0.3em; +} + +body.row .rows-and-columns td:before { + display: block; + color: black; + padding-bottom: 0.2em; + font-size: 0.8em; + font-weight: bold; + background-color: #f5f5f5; +}

/ Overrides ===============================================================/ diff --git a/datasette/templates/row.html b/datasette/templates/row.html index 1d1b0bfd..339eb643 100644 --- a/datasette/templates/row.html +++ b/datasette/templates/row.html @@ -5,6 +5,9 @@ {% block extra_head %} {{- super() -}} <style> +{% for column in columns %} +body.row .rows-and-columns td:nth-of-type({{ loop.index }}):before { content: "{{ column|escape_css_string }}"; } +{% endfor %} @media only screen and (max-width: 576px) { {% for column in columns %} .rows-and-columns td:nth-of-type({{ loop.index }}):before { content: "{{ column|escape_css_string }}"; } ``` Now the row page looks like this at all page widths: I think that's better (could do with a bit of tightening up). One catch: you can't copy and paste the column labels, since they are added using generated content like this: https://github.com/simonw/datasette/blob/0b4a28691468b5c758df74fa1d72a823813c96bf/datasette/static/app.css#L752-L757 https://github.com/simonw/datasette/blob/0b4a28691468b5c758df74fa1d72a823813c96bf/datasette/templates/row.html#L9-L11 I think the row page should switch to different HTML entirely, rather than continuing to share the `<table>` that's used by the table page. This will be a breaking change for users who customize Datasette, so I should aim to ship it before 1.0. 

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Row page should default to card view 1563264257  
1407733793 https://github.com/simonw/datasette/pull/2008#issuecomment-1407733793 https://api.github.com/repos/simonw/datasette/issues/2008 IC_kwDOBm6k_c5T6FAh simonw 9599 2023-01-29T18:17:40Z 2023-01-29T18:17:40Z OWNER

We don't have any performance tests yet - would be a useful thing to add, I've not built anything like that before (at least not in CI, I've always done as-hoc performance testing using something like Locust) so I don't have a great feel for how it could work.

Had an interesting conversation about this just now: https://fedi.simonwillison.net/@simon/109773800944614366

There's a risk that different runs will return different results due to the shared resource nature of GitHub Actions runners, but a good fix for that is to run comparative tests where you run the benchmark against e.g. both main and the incoming PR branch and report back on any differences.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 array facet: don't materialize unnecessary columns 1560982210  
1407568923 https://github.com/simonw/datasette/pull/2008#issuecomment-1407568923 https://api.github.com/repos/simonw/datasette/issues/2008 IC_kwDOBm6k_c5T5cwb simonw 9599 2023-01-29T05:47:36Z 2023-01-29T05:47:36Z OWNER

I don't know how/if you do automated tests for performance, so I haven't changed any of the tests.

We don't have any performance tests yet - would be a useful thing to add, I've not built anything like that before (at least not in CI, I've always done as-hoc performance testing using something like Locust) so I don't have a great feel for how it could work.

I see not having to change the tests at all for this change as a really positive sign. If you find any behaviour differences between this and the previous that's a sign we should add a mother test or two specifying the behaviour we want.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 array facet: don't materialize unnecessary columns 1560982210  
1407567753 https://github.com/simonw/datasette/pull/2008#issuecomment-1407567753 https://api.github.com/repos/simonw/datasette/issues/2008 IC_kwDOBm6k_c5T5ceJ simonw 9599 2023-01-29T05:39:54Z 2023-01-29T05:40:34Z OWNER

I absolutely love this performance boost - really nice find.

One concern: this will be the first time Datasette ships a core feature that uses window functions.

Window functions were added to SQLite in version 3.25.0 on 2018-09-15 - which means it's still very common for Datasette to run on versions that don't yet support them.

So I see two options: - Detect window function support and switch between the old implementation and this better, new one - Detect window functions and disable the facet-by-JSON feature entirely if they are missing

I like the first option a bit better.

This also leads to a tricky CI challenge: Datasette needs to be able to run its test suite against more than one SQLite version to confidently test this feature going forward.

I don't yet have a good GitHub Actions recipe for this, but I really need one - for sqlite-utils too.

Might be able to use this trick for that: https://til.simonwillison.net/sqlite/ld-preload

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 array facet: don't materialize unnecessary columns 1560982210  
1405488884 https://github.com/simonw/datasette/issues/2006#issuecomment-1405488884 https://api.github.com/repos/simonw/datasette/issues/2006 IC_kwDOBm6k_c5Txg70 simonw 9599 2023-01-26T19:20:53Z 2023-01-26T19:20:53Z OWNER

I can run a GitHub code search a week or so before the release to give me time to get in touch with anyone who has actions that look like they might break.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Teach `datasette publish` to pin to `datasette<1.0` in a 0.x release 1558644003  
1405488523 https://github.com/simonw/datasette/issues/2006#issuecomment-1405488523 https://api.github.com/repos/simonw/datasette/issues/2006 IC_kwDOBm6k_c5Txg2L simonw 9599 2023-01-26T19:20:32Z 2023-01-26T19:20:32Z OWNER

This won't actually help that much if the user's GitHub Actions workflow does the equivalent of this: pip install datasette datasette publish cloudrun mydb.db ... Since they'll get the 1.0 release too.

Since they'll need to make changes anyway, maybe a better solution is to document this and tell people they should use this instead:

datasette publish cloudrun mydb.db ... --branch 0.64.1

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Teach `datasette publish` to pin to `datasette<1.0` in a 0.x release 1558644003  
1404458011 https://github.com/simonw/datasette/issues/2005#issuecomment-1404458011 https://api.github.com/repos/simonw/datasette/issues/2005 IC_kwDOBm6k_c5TtlQb simonw 9599 2023-01-26T01:41:30Z 2023-01-26T01:41:50Z OWNER

My code looked like this: ```python @hookimpl def extra_template_vars(datasette, view_name, database): async def inner(): if view_name == "database": return {"blah": 1}

return inner

`` It returnsNonein the case where theif` condition does not match.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `extra_template_vars` should be OK to return `None` 1557507274  
1404253358 https://github.com/simonw/datasette/issues/262#issuecomment-1404253358 https://api.github.com/repos/simonw/datasette/issues/262 IC_kwDOBm6k_c5TszSu simonw 9599 2023-01-25T21:35:32Z 2023-01-25T21:35:32Z OWNER

This issue here would benefit from some kid of mechanism for returning just the HTML of the table itself, without any of the surrounding material. I'm not sure if that would make sense as an extra or not: - https://github.com/simonw/datasette-search-all/issues/17

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Add ?_extra= mechanism for requesting extra properties in JSON 323658641  

Next page

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);