Here's what I have today - it's an undocumented datasette.metadata() method that returns a full JSON dictionary of values OR a single value if the optional key= argument is provided: https://github.com/simonw/datasette/blob/f2e2bfcdd9ad4891f3f66c9104c09943d943ffe4/datasette/app.py#L357-L388

In #942 I want to add support for per-column metadata - which means this new lookup mechanism will need to be able to answer the question "what description is available for this column".

So what should the .metadata() method look like? A couple of options:

• datasette.metadata("description", table=x, database=y) - can take optional column= too.
• datasette.table_metadata("description", table=x, database=y) and datasette.database_metadata("description", database=y) and so on - multiple methods for the different types of metadata.

I'll also allow any key to be looked up - so if users want to invent their own metadata keys other than the default license_url etc they can do so.

I'm going to go with a plugin hook (and Datasette method) that returns individual values - so you ask it for e.g. the license_url for a specific table and it returns a string or None.

The default plugin hook implementation that ships with Datasette will then implement cascading lookups against metadata.json - but other plugins will be able to provide their own implementations, which should make it easy to build a plugin that lets you keep metadata in a database file and edit it interactively.

Documentation: https://docs.datasette.io/en/latest/internals.html#setting-key

I'm going to make this a documented method in https://docs.datasette.io/en/latest/internals.html#datasette-class

This rebranding is complete - #1107 is a follow-up internal refactor.

I used an "exact redirect" instead and it worked:

https://user-images.githubusercontent.com/9599/100154418-60bf0d80-2e5a-11eb-92bf-527a14edb5c5.png">

https://user-images.githubusercontent.com/9599/100153884-a3341a80-2e59-11eb-8d8c-7027c714b9cf.png">

https://docs.datasette.io/en/latest/config.html isn't redirecting though, even after I tried running a rebuild of the latest version.

https://docs.datasette.io/en/latest/settings.html is now live - need to redirect https://docs.datasette.io/en/latest/config.html to it using the ReadTheDocs redirects interface.

Most of these use plugin_config which is unaffected. It looks like the only code I need to worry about is this trick in datasette-graphl: https://github.com/simonw/datasette-graphql/blob/483c9a9e203bb90365def3df8b8f01dda1e75865/datasette_graphql/utils.py#L456-L460

class DatasetteSpecialConfig(wrapt.ObjectProxy):
    def config(self, key):
        if key == "suggest_facets":
            return False
        return self.__wrapped__.config(key)

I ran rg '.config\(' datasette-*/ in my top-level directory:

datasette-sentry/test_datasette_sentry.py:    def plugin_config(self, name):
datasette-sentry/datasette_sentry.py:    config = datasette.plugin_config("datasette-sentry") or {}
datasette-render-markdown/datasette_render_markdown/__init__.py:        datasette.plugin_config(
datasette-render-images/datasette_render_images.py:        plugin_config = datasette.plugin_config("datasette-render-images") or {}
datasette-render-html/datasette_render_html.py:    config = datasette.plugin_config(
datasette-render-timestamps/datasette_render_timestamps/__init__.py:        datasette.plugin_config(
datasette-permissions-sql/datasette_permissions_sql/__init__.py:        for rule in datasette.plugin_config("datasette-permissions-sql") or []:
datasette-mask-columns/datasette_mask_columns/__init__.py:            datasette.plugin_config("datasette-mask-columns", database=database) or {}
datasette-mask-columns/datasette_mask_columns/__init__.py:    masks = datasette.plugin_config("datasette-mask-columns", database=database) or {}
datasette-media/datasette_media/__init__.py:    plugin_config = datasette.plugin_config("datasette-media") or {}
datasette-insert/datasette_insert/__init__.py:    plugin_config = datasette.plugin_config("datasette-insert") or {}
datasette-indieauth/datasette_indieauth/__init__.py:    plugin_config = datasette.plugin_config("datasette-indieauth") or {}
datasette-graphql/datasette_graphql/__init__.py:    config = datasette.plugin_config("datasette-graphql") or {}
datasette-graphql/datasette_graphql/__init__.py:    config = datasette.plugin_config("datasette-graphql") or {}
datasette-graphql/datasette_graphql/utils.py:            auto_camelcase=(datasette.plugin_config("datasette-graphql") or {}).get(
datasette-graphql/datasette_graphql/utils.py:    table_plugin_config = datasette.plugin_config(
datasette-graphql/datasette_graphql/utils.py:    def config(self, key):
datasette-graphql/datasette_graphql/utils.py:        return self.__wrapped__.config(key)
datasette-init/datasette_init/__init__.py:        config = datasette.plugin_config("datasette-init")
datasette-edit-templates/datasette_edit_templates/__init__.py:    plugin_config = datasette.plugin_config("datasette-edit-templates") or {}
datasette-cors/datasette_cors.py:    config = datasette.plugin_config("datasette-cors") or {}
datasette-cluster-map-old/build/lib/datasette_cluster_map/__init__.py:        datasette.plugin_config("datasette-cluster-map", database=database, table=table)
datasette-cluster-map/datasette_cluster_map/__init__.py:        datasette.plugin_config("datasette-cluster-map", database=database, table=table)
datasette-cluster-map/datasette_cluster_map/__init__.py:        datasette.plugin_config("datasette-cluster-map", database=database, table=table)
datasette-cluster-map/tests/test_cluster_map.py:async def test_plugin_config(db_path, config, table, expected_fragments):
datasette-configure-asgi/datasette_configure_asgi.py:        configs = datasette.plugin_config("datasette-configure-asgi") or []
datasette-configure-asgi/test_datasette_configure_asgi.py:    def plugin_config(self, name):
datasette-cluster-map-old/datasette_cluster_map/__init__.py:        datasette.plugin_config("datasette-cluster-map", database=database, table=table)
datasette-auth-passwords/datasette_auth_passwords/__init__.py:    config = datasette.plugin_config("datasette-auth-passwords") or {}
datasette-auth-github/datasette_auth_github/views.py:def verify_config(config):
datasette-auth-github/datasette_auth_github/views.py:    config = datasette.plugin_config("datasette-auth-github")
datasette-auth-github/datasette_auth_github/views.py:    verify_config(config)
datasette-auth-github/datasette_auth_github/views.py:    config = datasette.plugin_config("datasette-auth-github")
datasette-auth-github/datasette_auth_github/views.py:    verify_config(config)
datasette-atom/datasette_atom/__init__.py:        plugin_config = datasette.plugin_config("datasette-atom")
datasette-auth-google/datasette_auth_google/__init__.py:    config = datasette.plugin_config("datasette-auth-github") or {}
datasette-auth-existing-cookies/test_datasette_auth_existing_cookies.py:    def plugin_config(self, name):
datasette-auth-passwords/build/lib/datasette_auth_passwords/__init__.py:    config = datasette.plugin_config("datasette-auth-passwords") or {}
datasette-annotate/datasette_annotate/utils.py:    plugin_config = datasette.plugin_config("datasette-annotate") or {}
datasette-auth-existing-cookies/datasette_auth_existing_cookies/__init__.py:    config = datasette.plugin_config("datasette-auth-existing-cookies") or {}
datasette-auth-simple/datasette_auth_simple/__init__.py:datasette.plugin_config("datasette-auth-simple")
datasette-auth-tokens/datasette_auth_tokens/__init__.py:        config = datasette.plugin_config("datasette-auth-tokens") or {}
datasette-block-robots/datasette_block_robots/__init__.py:    config = datasette.plugin_config("datasette-block-robots") or {}
datasette-block-robots/datasette_block_robots/__init__.py:    config = datasette.plugin_config("datasette-block-robots") or {}

Are there any plugins that use this API even though it isn't documented?

https://readthedocs.org/dashboard/datasette/redirects/

I'll throw an error if a config.json file is detected.

This is well handled now: https://github.com/simonw/datasette/tree/0.51.1/tests/plugins

I'd like to redirect https://docs.datasette.io/en/stable/config.html to a new https://docs.datasette.io/en/stable/settings.html page too. I can use https://docs.readthedocs.io/en/stable/user-defined-redirects.html for that.

Part of #1105

Part of #1105

Part of #1105

I'll set up a redirect from /-/config to /-/settings.

Need to figure out the --setting foo bar alternative for this --config foo:bar logic: https://github.com/simonw/datasette/blob/4bac9f18f9d04e5ed10f072502bcc508e365438e/datasette/cli.py#L31-L63

I'm going to keep --config for the moment but show a deprecation warning that it will be gone in Datasette 1.0.

This is blocking progress on other metadata tickets like #860 because I want to split the concept of concrete metadata (source, license, etc) from configuration that currently lives in metadata (default sort order, default facets).

I'm going to solve this next to unblock that stuff.

I see two ways this plugin hook could work. It could be asked about a specific instance, database or table and return the full metadata for that object. OR it could ask for a specific metadata field - e.g. source_url for table X, and return that.

The more finely grained one would allow plugins to implement their own cascading rules pretty easily. Is there a reason it would be better for the hook to return an entire block of JSON for a specific table or database?

I also need to decide if this hook is just going to be about source/license/about displayed metadata, or if it will include the functionality that has been sneaking into metadata.json over time - stuff like page size, default sort order or default facets.

Perhaps I should split those out into a "configuration" concept first, after renaming --config to --setting in #992.

The trick I'm using here is to follow the next_url in order to paginate through all of the matching results. The loop calls the data() method multiple times, once for each page of results: https://github.com/simonw/datasette/blob/4bac9f18f9d04e5ed10f072502bcc508e365438e/datasette/views/base.py#L304-L307

Current design: https://docs.datasette.io/en/stable/plugin_hooks.html#register-output-renderer-datasette

@hookimpl
def register_output_renderer(datasette):
    return {
        "extension": "test",
        "render": render_demo,
        "can_render": can_render_demo,  # Optional
    }

Where render_demo looks something like this:

async def render_demo(datasette, columns, rows):
    db = datasette.get_database()
    result = await db.execute("select sqlite_version()")
    first_row = " | ".join(columns)
    lines = [first_row]
    lines.append("=" * len(first_row))
    for row in rows:
        lines.append(" | ".join(row))
    return Response(
        "\n".join(lines),
        content_type="text/plain; charset=utf-8",
        headers={"x-sqlite-version": result.first()[0]}
    )

Meanwhile here's where the CSV streaming mode is implemented: https://github.com/simonw/datasette/blob/4bac9f18f9d04e5ed10f072502bcc508e365438e/datasette/views/base.py#L297-L380

I'd like to implement this by first extending the register_output_renderer() hook to support streaming huge responses, then switching CSV to use the plugin hook in addition to TSV using it.

Documented behaviour right now, for metadata set at the instance level, is: https://docs.datasette.io/en/stable/metadata.html

The above metadata will be displayed on the index page of your Datasette-powered site. The source and license information will also be included in the footer of every page served by Datasette.

...

Metadata at the top level of the JSON will be shown on the index page and in the footer on every page of the site. The license and source is expected to apply to all of your data.

Open question: how should cascading work? If a table is missing a field but the database or instance has it, should that value cascade down to the table?

It feels like license should definitely cascade: if an instance lists a certain license that should absolutely filter through to all databases and tables.

But... should the other fields cascade? Cascading description doesn't feel right at all, and neither does title.

What about about and about_url and source and source_url? I'm a bit torn on whether they should cascade or not. I'm leaning towards cascading them.

There are three layers of metadata: table, database and instance.

Currently the metadata fields are (ignoring not-quite-metadata like sort and sort_desc):

• title
• description (or description_html)
• about / about_url
• source / source_url
• license / license_url

https://datasette-graphql-demo.datasette.io/github/users now demonstrates the fix.

I got to use CSS calc() for this: https://github.com/simonw/datasette/blob/4bac9f18f9d04e5ed10f072502bcc508e365438e/datasette/static/app.css#L367-L371

Fixed by positioning the menu relative to the <div class="page-header"> element rather than the cog icon:

https://user-images.githubusercontent.com/9599/99889950-02eab580-2c0f-11eb-85dc-dd4bc840a92e.png">

Another example of this bug: https://datasette-graphql-demo.datasette.io/github/users

https://user-images.githubusercontent.com/9599/99888639-3889a180-2c03-11eb-83b9-41f2eb12287e.png">

I can confirm this issue, running version 0.51.1 under Windows.

Fixed by commenting out the following line near the top of datasette\utils\asgi.py :

#from os import EX_CANTCREAT

Trying to run on Windows today, I get an error from the utils/asgi.py module.

It's trying from os import EX_CANTCREAT which is Unix-only. I commented this line out, and (so far) it's working.

I'm so sorry that you already have --since_id option and that's enough for the case I've mentioned. Thank you for this excellent tool!

We're using mod_proxy.

I don't think we are, but I'll check with Maruan.

I think this is the relevant part of our config?

  Alias "/base/" "/usr/share/corpora/"
  <Directory "/usr/share/corpora/">
    Options +Indexes -Multiviews
    AllowOverride None
  </Directory>

  ProxyPreserveHost On

  ProxyPass /datasette http://0.0.0.0:8001
  ProxyPassReverse /datasette http://0.0.0.0:8001

</VirtualHost>

I have a hunch that there may be some extra configuration in play here - could Apache itself be rewriting some of the links using mod_proxy_html?

Thanks, I'll look into this.

Schema for that broken example:

CREATE TABLE generators_eia860 (
    id INTEGER NOT NULL, 
    plant_id_eia INTEGER, 
    generator_id TEXT, 
    report_date DATE, 
    operational_status_code TEXT, 
    operational_status TEXT, 
    ownership_code TEXT, 
    utility_id_eia INTEGER, 
    capacity_mw FLOAT, 
    summer_capacity_mw FLOAT, 
    winter_capacity_mw FLOAT, 
    energy_source_code_1 TEXT, 
    energy_source_code_2 TEXT, 
    energy_source_code_3 TEXT, 
    energy_source_code_4 TEXT, 
    energy_source_code_5 TEXT, 
    energy_source_code_6 TEXT, 
    fuel_type_code_pudl TEXT, 
    multiple_fuels BOOLEAN, 
    deliver_power_transgrid BOOLEAN, 
    syncronized_transmission_grid BOOLEAN, 
    turbines_num INTEGER, 
    planned_modifications BOOLEAN, 
    planned_net_summer_capacity_uprate_mw FLOAT, 
    planned_net_winter_capacity_uprate_mw FLOAT, 
    planned_uprate_date DATE, 
    planned_net_summer_capacity_derate_mw FLOAT, 
    planned_net_winter_capacity_derate_mw FLOAT, 
    planned_derate_date DATE, 
    planned_new_prime_mover_code TEXT, 
    planned_energy_source_code_1 TEXT, 
    planned_repower_date DATE, 
    other_planned_modifications BOOLEAN, 
    other_modifications_date DATE, 
    planned_retirement_date DATE, 
    carbon_capture BOOLEAN, 
    startup_source_code_1 TEXT, 
    startup_source_code_2 TEXT, 
    startup_source_code_3 TEXT, 
    startup_source_code_4 TEXT, 
    technology_description TEXT, 
    turbines_inverters_hydrokinetics TEXT, 
    time_cold_shutdown_full_load_code TEXT, 
    planned_new_capacity_mw FLOAT, 
    cofire_fuels BOOLEAN, 
    switch_oil_gas BOOLEAN, 
    nameplate_power_factor FLOAT, 
    minimum_load_mw FLOAT, 
    uprate_derate_during_year BOOLEAN, 
    uprate_derate_completed_date DATE, 
    current_planned_operating_date DATE, 
    summer_estimated_capability_mw FLOAT, 
    winter_estimated_capability_mw FLOAT, 
    retirement_date DATE, 
    PRIMARY KEY (id), 
    FOREIGN KEY(plant_id_eia, generator_id) REFERENCES generators_entity_eia (plant_id_eia, generator_id), 
    FOREIGN KEY(utility_id_eia) REFERENCES utilities_entity_eia (utility_id_eia), 
    CHECK (multiple_fuels IN (0, 1)), 
    CHECK (deliver_power_transgrid IN (0, 1)), 
    CHECK (syncronized_transmission_grid IN (0, 1)), 
    CHECK (planned_modifications IN (0, 1)), 
    CHECK (other_planned_modifications IN (0, 1)), 
    CHECK (carbon_capture IN (0, 1)), 
    CHECK (cofire_fuels IN (0, 1)), 
    CHECK (switch_oil_gas IN (0, 1)), 
    CHECK (uprate_derate_during_year IN (0, 1))
);

https://pudl-datasette-xl7xwcpe2a-uc.a.run.app/pudl/generators_entity_eia is:

CREATE TABLE generators_entity_eia (
    plant_id_eia INTEGER NOT NULL, 
    generator_id TEXT NOT NULL, 
    prime_mover_code TEXT, 
    duct_burners BOOLEAN, 
    operating_date DATE, 
    topping_bottoming_code TEXT, 
    solid_fuel_gasification BOOLEAN, 
    pulverized_coal_tech BOOLEAN, 
    fluidized_bed_tech BOOLEAN, 
    subcritical_tech BOOLEAN, 
    supercritical_tech BOOLEAN, 
    ultrasupercritical_tech BOOLEAN, 
    stoker_tech BOOLEAN, 
    other_combustion_tech BOOLEAN, 
    bypass_heat_recovery BOOLEAN, 
    rto_iso_lmp_node_id TEXT, 
    rto_iso_location_wholesale_reporting_id TEXT, 
    associated_combined_heat_power BOOLEAN, 
    original_planned_operating_date DATE, 
    operating_switch TEXT, 
    previously_canceled BOOLEAN, 
    PRIMARY KEY (plant_id_eia, generator_id), 
    FOREIGN KEY(plant_id_eia) REFERENCES plants_entity_eia (plant_id_eia), 
    CHECK (duct_burners IN (0, 1)), 
    CHECK (solid_fuel_gasification IN (0, 1)), 
    CHECK (pulverized_coal_tech IN (0, 1)), 
    CHECK (fluidized_bed_tech IN (0, 1)), 
    CHECK (subcritical_tech IN (0, 1)), 
    CHECK (supercritical_tech IN (0, 1)), 
    CHECK (ultrasupercritical_tech IN (0, 1)), 
    CHECK (stoker_tech IN (0, 1)), 
    CHECK (other_combustion_tech IN (0, 1)), 
    CHECK (bypass_heat_recovery IN (0, 1)), 
    CHECK (associated_combined_heat_power IN (0, 1)), 
    CHECK (previously_canceled IN (0, 1))
);

Need to replicate this in the fixtures, then fix it.

Codecov Report

Merging #1097 (e89211d) into main (5eb8e9b) will not change coverage.
The diff coverage is 84.61%.

@@           Coverage Diff           @@
##             main    #1097   +/-   ##
=======================================
  Coverage   91.38%   91.38%           
=======================================
  Files          30       30           
  Lines        3785     3785           
=======================================
  Hits         3459     3459           
  Misses        326      326           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5eb8e9b...e89211d. Read the comment docs.

This will also benefit from the metadata plugin hook: #860

@tballison could I see the section of your Apache config that configures the proxying to /datasette/?

My headers aren't clickable/sortable with custom sql, but I think that's by design.

In the default view, https://corpora.tika.apache.org/datasette/file_profiles/file_profiles, ah, y, now I see that the headers should be sortable, but you're right the base_url is not applied.

base_url works with "View and Edit SQL" and with "(advanced)"

As you point out, does not work with the export csv, json, other or with the "Next page" navigational button at the bottom.

I'm starting this with docker like so:

docker run --name datasette -d -p 8001:8001 -vpwd:/mnt datasetteproject/datasette datasette -p 8001 -h 0.0.0.0 /mnt/file_profiles.db --config sql_time_limit_ms:120000 --config max_returned_rows:100000 --config base_url:/datasette/ --config cache_size_kb:50000

I'm not doing any templating or anything else custom.

Apropos of nothing, I swapped out a simpler db, so this query should now work:

https://corpora.tika.apache.org/datasette/file_profiles?sql=select%0D%0A++*%0D%0Afrom%0D%0A++file_profiles+fp%0D%0Alimit%0D%0A++10

Part of the challenge here is that this is the first time the TableView will have had a complete rewrite of the SQL it is going to execute. That SQL is currently constructed here: https://github.com/simonw/datasette/blob/5eb8e9bf250b26e30b017d39a392c33973997656/datasette/views/table.py#L628-L636

https://cloud.google.com/blog/products/serverless/cloud-run-now-supports-http-grpc-server-streaming indicates this limit should no longer apply:

With this addition, Cloud Run can now ... Send responses larger than the previous 32 MB limit

But I'm still getting errors from Cloud Run attempting to download .db files larger than 32 MB.

I filed a question in their issue tracker about that here: https://issuetracker.google.com/issues/173038375

Here's where url_csv comes from: https://github.com/simonw/datasette/blob/11eb1e026f3d84cb771f8d6e204939cbaee130cd/datasette/views/base.py#L542-L545

The sort headers are generated by this template code: https://github.com/simonw/datasette/blob/5eb8e9bf250b26e30b017d39a392c33973997656/datasette/templates/_table.html#L11-L15

The export links use this code:

https://github.com/simonw/datasette/blob/5eb8e9bf250b26e30b017d39a392c33973997656/datasette/templates/table.html#L134

https://github.com/simonw/datasette/blob/5eb8e9bf250b26e30b017d39a392c33973997656/datasette/templates/table.html#L180-L201

@tballison is there any chance you're running any custom templates in that installation? I'm really confused as to why I can't replicate the bug.

Hmm... it's not just the .csv and .json export links - it's the column headings (which can be clicked to change the sort order) as well. Here's an extract of the HTML from that page:

<p class="export-links">This data as 
  <a href="/corpora-metadata/REF_PARSE_EXCEPTION_TYPES.json">json</a>, 
  <a href="/corpora-metadata/REF_PARSE_EXCEPTION_TYPES.csv?_size=max">CSV</a> (
  <a href="#export">advanced</a>)
</p>
<div class="table-wrapper">
  <table class="rows-and-columns">
    <thead>
      <tr>
        <th class="col-Link" scope="col" data-column="Link" data-column-type="" data-column-not-null="0" data-is-pk="0">
          Link
        </th>
        <th class="col-rowid" scope="col" data-column="rowid" data-column-type="integer" data-column-not-null="0" data-is-pk="1">
          <a href="/corpora-metadata/REF_PARSE_EXCEPTION_TYPES?_sort_desc=rowid" rel="nofollow">rowid&nbsp;▼</a>
        </th>
        <th class="col-PARSE_EXCEPTION_ID" scope="col" data-column="PARSE_EXCEPTION_ID" data-column-type="INTEGER" data-column-not-null="0" data-is-pk="0">
          <a href="/corpora-metadata/REF_PARSE_EXCEPTION_TYPES?_sort=PARSE_EXCEPTION_ID" rel="nofollow">PARSE_EXCEPTION_ID</a>
        </th>
        <th class="col-PARSE_EXCEPTION_DESCRIPTION" scope="col" data-column="PARSE_EXCEPTION_DESCRIPTION" data-column-type="VARCHAR(128)" data-column-not-null="0" data-is-pk="0">
          <a href="/corpora-metadata/REF_PARSE_EXCEPTION_TYPES?_sort=PARSE_EXCEPTION_DESCRIPTION" rel="nofollow">PARSE_EXCEPTION_DESCRIPTION</a>
        </th>
      </tr>
    </thead>
    <tbody>
      <tr>
        <td class="col-Link type-pk">
          <a href="/datasette/corpora-metadata/REF_PARSE_EXCEPTION_TYPES/1">1</a>
        </td>
        <td class="col-rowid type-int">1</td>
        <td class="col-PARSE_EXCEPTION_ID type-int">0</td>
        <td class="col-PARSE_EXCEPTION_DESCRIPTION type-str">RUNTIME</td>
      </tr>
      <tr>
        <td class="col-Link type-pk">
          <a href="/datasette/corpora-metadata/REF_PARSE_EXCEPTION_TYPES/2">2</a>
        </td>
        <td class="col-rowid type-int">2</td>
        <td class="col-PARSE_EXCEPTION_ID type-int">1</td>
        <td class="col-PARSE_EXCEPTION_DESCRIPTION type-str">ENCRYPTION</td>
      </tr>
      <tr>
        <td class="col-Link type-pk">
          <a href="/datasette/corpora-metadata/REF_PARSE_EXCEPTION_TYPES/3">3</a>
        </td>
        <td class="col-rowid type-int">3</td>
        <td class="col-PARSE_EXCEPTION_ID type-int">2</td>
        <td class="col-PARSE_EXCEPTION_DESCRIPTION type-str">ACCESS_PERMISSION</td>
      </tr>
      <tr>
        <td class="col-Link type-pk">
          <a href="/datasette/corpora-metadata/REF_PARSE_EXCEPTION_TYPES/4">4</a>
        </td>
        <td class="col-rowid type-int">4</td>
        <td class="col-PARSE_EXCEPTION_ID type-int">3</td>
        <td class="col-PARSE_EXCEPTION_DESCRIPTION type-str">UNSUPPORTED_VERSION</td>
      </tr>
    </tbody>
  </table>
</div>
<div id="export" class="advanced-export">
  <h3>Advanced export</h3>
  <p>JSON shape:
    <a href="/corpora-metadata/REF_PARSE_EXCEPTION_TYPES.json">default</a>,
    <a href="/corpora-metadata/REF_PARSE_EXCEPTION_TYPES.json?_shape=array">array</a>,
    <a href="/corpora-metadata/REF_PARSE_EXCEPTION_TYPES.json?_shape=array&amp;_nl=on">newline-delimited</a>
  </p>
  <form action="/corpora-metadata/REF_PARSE_EXCEPTION_TYPES.csv" method="get">
    <p>
      CSV options:            
      <label>
        <input type="checkbox" name="_dl"> download file
        </label>
        <input type="submit" value="Export CSV">
          <input type="hidden" name="_size" value="max">
          </p>
        </form>
      </div>

But here's something really weird - the links to the individual rows DO include the /datasette/ prefix:

<td class="col-Link type-pk">
    <a href="/datasette/corpora-metadata/REF_PARSE_EXCEPTION_TYPES/2">2</a>
</td>

The navigation bar on that page is correct too:

<p class="crumbs">
        <a href="/datasette/">home</a> /
        <a href="/datasette/corpora-metadata">corpora-metadata</a>
</p>

I've also been unable to replicate this in my own local environment, running datasette fixtures.db --config base_url:/datasette/.

@tballison thanks, I've split that out into a new issue #1091

The same is true if I select advanced export and hit the 'export csv' at the bottom of the page.

Just tested with the latest Docker image, and it works pretty much everywhere! THANK YOU!

I did notice that if I try to export json or csv, the base is not applied. Not sure if I should reopen this issue or open a new one.

To see this, go here: https://corpora.tika.apache.org/datasette/corpora-metadata/REF_PARSE_EXCEPTION_TYPES

Click/hover over json or CSV and you'll see that the 'datasette' base is not included.

Again, many thanks!

Suggested list to look out for from that PR:

• simply/simple
• easy/easier/easiest
• obvious/obviously
• just
• merely
• straightforward
• ridiculous

I'm going to fix this without a test, because writing a test for this is a bit fiddly and it's a very minor bug. If it comes back again I'll do the work to test for it.

Yup, swapping QueryInterrupted fixed this against my giant database.

I'm suspicious of this code: https://github.com/simonw/datasette/blob/e8e0a6f284ca953b2980186c4356594c07bd1929/datasette/views/table.py#L1032-L1045

This code uses a different exception: https://github.com/simonw/datasette/blob/e8e0a6f284ca953b2980186c4356594c07bd1929/datasette/views/table.py#L658-L663

Codecov Report

Merging #1085 (51e7651) into main (2a981e2) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1085   +/-   ##
=======================================
  Coverage   91.38%   91.38%           
=======================================
  Files          30       30           
  Lines        3785     3785           
=======================================
  Hits         3459     3459           
  Misses        326      326           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2a981e2...51e7651. Read the comment docs.

Demo: https://latest.datasette.io/fixtures/foreign_key_references?_facet=foreign_key_with_blank_label

https://user-images.githubusercontent.com/9599/98877868-53575b80-2436-11eb-8669-f24eeb8d0f3c.png">

I'm just going to use a regular coloured hyphen.

Maybe use a grey hyphen here?

Yeah whatever process the have in place is working great without any extra intervention: they upgraded to 3.0 four hours ago.

I should depend on sqlite-fts4 - I'm doing that in sqlite-utils now and it works great: https://github.com/simonw/sqlite-utils/issues/198

This looks pretty good now!

% sqlite-utils search 24ways.db articles simon -c title -c author -t 
title                                                                          author
-----------------------------------------------------------------------------  ------------------
Don't be eval()                                                                Simon Willison
DOM Scripting Your Way to Better Blockquotes                                   Jeremy Keith
Swooshy Curly Quotes Without Images                                            Simon Collison
The Articulate Web Designer of Tomorrow                                        Simon Collison
Writing Responsible JavaScript                                                 Drew McLellan
Going Nuts with CSS Transitions                                                Natalie Downe
Managing a Mind                                                                Christopher Murphy
Design Systems                                                                 Laura Kalbag
Bringing Your Code to the Streets                                              Ruth John
Taming Complexity                                                              Simon Collison
Unobtrusively Mapping Microformats with jQuery                                 Simon Willison
Crafting the Front-end                                                         Ben Bodien
Develop Your Naturalist Superpowers with Observable Notebooks and iNaturalist  Natalie Downe
Fast Autocomplete Search for Your Website                                      Simon Willison

Need to fix this to close #192 and #197.

Here's the alpha release: https://github.com/simonw/sqlite-utils/releases/tag/3.0a0

Documentation: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#searching-with-table-search

The order= parameter should be called order_by for consistency with this:

for row in db["dogs"].rows_where("age > 1", order_by="age"):

(sqlite-utils) sqlite-utils % sqlite-utils rows 24ways-fts4.db articles -c title -c author | head -n 3
[{"title": "Why Bother with Accessibility?", "author": "Laura Kalbag"},
 {"title": "Levelling Up", "author": "Ashley Baxter"},
 {"title": "Project Hubs: A Home Base for Design Projects", "author": "Brad Frost"},

I'm going to have it only output the exact -c columns you requested (if you requested any). Add --rank to specify the rank column, since you may not know what its name is.

I'm going to release this as an alpha first and sit on it for a few days, since I don't want to ship any mistakes that would result in having to bump straight to 4.0!

This is now possible, for both FTS4 and FTS5 - see #197.

Also that order looks incorrect. It looks like most relevant came back last, not first.

This is a bit surprising:

(sqlite-utils) sqlite-utils % sqlite-utils search 24ways-fts4.db articles maps -c title      
[{"rowid": 41, "title": "What Is Vagrant and Why Should I Care?", "rank": -1.9252039178908076},
 {"rowid": 298, "title": "First Steps in VR", "rank": -1.9945466378736434},
 {"rowid": 43, "title": "Content Production Planning", "rank": -2.1928058363046143},
 {"rowid": 100, "title": "Moo'y Christmas", "rank": -2.2698482999851675},
 {"rowid": 91, "title": "Infinite Canvas: Moving Beyond the Page", "rank": -2.290928999035195},
 {"rowid": 175, "title": "Front-End Code Reusability with CSS and JavaScript", "rank": -2.498731782924352},
 {"rowid": 209, "title": "Feeding the Audio Graph", "rank": -2.619968930100356},
 {"rowid": 296, "title": "Animation in Design Systems", "rank": -2.62060151817201},
 {"rowid": 118, "title": "Ghosts On The Internet", "rank": -2.7224894534521087},
 {"rowid": 77, "title": "Colour Accessibility", "rank": -2.7389782859427343},
 {"rowid": 245, "title": "Web Content Accessibility Guidelines 2.1\u2014for People Who Haven\u2019t Read the Update", "rank": -2.9750992611162888},
 {"rowid": 56, "title": "Helping VIPs Care About Performance", "rank": -3.0819662908932535},
 {"rowid": 109, "title": "Geotag Everywhere with Fire Eagle", "rank": -3.1371975973877277},
 {"rowid": 203, "title": "Jobs-to-Be-Done in Your UX Toolbox", "rank": -3.2416719461682733},
 {"rowid": 276, "title": "Your jQuery: Now With 67% Less Suck", "rank": -3.4947916564653028},
 {"rowid": 58, "title": "Beyond the Style Guide", "rank": -3.7508321464447905},
 {"rowid": 225, "title": "Good Ideas Grow on Paper", "rank": -4.120077674716844},
 {"rowid": 168, "title": "Unobtrusively Mapping Microformats with jQuery", "rank": -4.662224207228984},
 {"rowid": 27, "title": "Putting Design on the Map", "rank": -5.667327088267961},
 {"rowid": 220, "title": "Finding Your Way with Static Maps", "rank": -9.952534352591737}]

I requested just -c title but also got back rowid and rank.

The breaking changes will be:

• table.search() now returns a generator that produces dictionaries, similar to table.rows_where()
• The -c shortcut no longer works, use --csv instead.
• The -f shortcut no longer works, use --fmt instead.

Still need to update docs.

Thanks to #198 (introducing a rank_bm25() custom function for FTS4) this feature will be able to offer relevance search for both FTS5 AND FTS4 tables.

Having .search() return tuples when .rows_where() returns dictionaries just feels like bad API design to me - it's inconsistent.

Something like this:

@db.register_function(replace=True)
def my_function(a):
    return a.upper()

If replace=True then this function will be registered even if a my_function of arity 1 has already been registered previously.

It defaults to False though which means the Database object tracks what functions and arities have been registered in the past and silently ignores any new attempts to register the same name/arity.

Should I register the custom rank_bm25 SQLite function for every connection, or should I register it against the connection just the first time the user attempts an FTS4 search? I think I'd rather register it only if it is needed.

Here's a demo of that rank query. I had to sort by rank instead of rank desc - need to double-check that:

https://datasette-sqlite-fts4.datasette.io/24ways-fts4?sql=with+original+as+(%0D%0A++++select%0D%0A++++++++rowid%2C%0D%0A++++++++*%0D%0A++++from+[articles]%0D%0A)%0D%0Aselect%0D%0A++++original.*%2C%0D%0A++++rank_bm25(matchinfo([articles_fts]%2C+%27pcnalx%27))+as+rank%0D%0Afrom%0D%0A++++[original]%0D%0A++++join+[articles_fts]+on+[original].rowid+%3D+[articles_fts].rowid%0D%0Awhere%0D%0A++++[articles_fts]+match+%3Aquery%0D%0Aorder+by%0D%0A++++rank%0D%0Alimit+20&query=jquery+maps

Here's the FTS5 query:

with original as (
    select
        rowid,
        *
    from [global-power-plants]
)
select
    original.*,
    [global-power-plants_fts].rank as rank
from
    [original]
    join [global-power-plants_fts] on [original].rowid = [global-power-plants_fts].rowid
where
    [global-power-plants_fts] match :query
order by
    rank desc
limit 20

The equivalent using rank_bm25() for FTS4 would be:

with original as (
    select
        rowid,
        *
    from [global-power-plants]
)
select
    original.*,
    rank_bm25(matchinfo([global-power-plants_fts], 'pcnalx')) as rank
from
    [original]
    join [global-power-plants_fts] on [original].rowid = [global-power-plants_fts].rowid
where
    [global-power-plants_fts] match :query
order by
    rank desc
limit 20

I released a 1.0 (and 1.0.1) version of https://github.com/simonw/sqlite-fts4

Example query (from the tests):

select c0, c1, rank_bm25(matchinfo(search, 'pcnalx')) as bm25
        from search where search match ?

I'd have to copy almost all of the code in https://github.com/simonw/sqlite-fts4/blob/master/sqlite_fts4/__init__.py so I think I will add it as a dependency instead.

This is likely to result in a 3.0 release due to a backwards-incompatible change to the current .search() method - #194

Latest test failure:

114  ->         assert [("racoons are biting trash pandas", "USA", "bar")] == table.search(
115                 "bite", order="rowid"
116             )
117     
118     
119     def test_optimize_fts(fresh_db):
(Pdb) table.search("bite")
[(2, 'racoons are biting trash pandas', 'USA', 'bar', -9.641434262948206e-07)]

The problem here is that the table.search() method now behaves differently for FTS4 v.s. FTS5 tables.

With FTS4 you get back just the table columns.

With FTS5 you also get back the rowid as the first column and the rank score as the last column.

This is weird. It also makes me question whether having .search() return a list of tuples is the right API design.

The finished monster:

_virtual_table_using_re = re.compile(r"""
^ # Start of string
\s*CREATE\s+VIRTUAL\s+TABLE\s+ # CREATE VIRTUAL TABLE
(
    '(?P<squoted_table>[^']*(?:''[^']*)*)' | # single quoted name
    "(?P<dquoted_table>[^"]*(?:""[^"]*)*)" | # double quoted name
    `(?P<backtick_table>[^`]+)`            | # `backtick` quoted name
    \[(?P<squarequoted_table>[^\]]+)\]     | # [...] quoted name
    (?P<identifier>                          # SQLite non-quoted identifier
        [A-Za-z_\u0080-\uffff]  # \u0080-\uffff = "any character larger than u007f"
        [A-Za-z_\u0080-\uffff0-9\$]* # zero-or-more alphanemuric or $
    )
)
\s+(IF\s+NOT\s+EXISTS\s+)?      # IF NOT EXISTS (optional)
USING\s+(?P<using>\w+)          # e.g. USING FTS5
""", re.VERBOSE | re.IGNORECASE)

And an identifier is "ALPHABETIC character and continue with zero or more ALPHANUMERIC characters and/or "$" (u0024) characters"

So...

[\u0041-\u005a\u0061-\u0071\u007f-\uffff\u005f][\u0041-\u005a\u0061-\u0071\u007f-\uffff\u005f\u0030-\u0039\u0024]+

So...

ALPHABETIC: [\u0041-\u005a\u0061-\u0071\u007f-\uffff\u005f]

NUMERIC: [\u0030-\u0039]

I think that's \u007F-\uFFFF in regex range speak.

"any other character larger than u007f." Need to figure that out!

More from that document, describing ALPHANUMERIC:

ALPHABETIC

Any of the characters in the range u0041 through u005a (letters "A" through "Z") or in the range u0061 through u007a (letters "a" through "z") or the character u005f ("_") or any other character larger than u007f.

NUMERIC

Any of the characters in the range u0030 through u0039 (digits "0" through "9")

ALPHANUMERIC

Any character which is either ALPHABETIC or NUMERIC

I'm missing the case where a table has no quotes around it at all - create virtual table foo using fts5

So I need to know how to create a regex for a SQLite identifier. https://www.sqlite.org/draft/tokenreq.html seems to be the only available documentation for that.

Identifier tokens

Identifiers follow the usual rules with the exception that SQLite allows the dollar-sign symbol in the interior of an identifier. The dollar-sign is for compatibility with Microsoft SQL-Server and is not part of the SQL standard.

H41130: SQLite shall recognize as an ID token any sequence of characters that begins with an ALPHABETIC character and continue with zero or more ALPHANUMERIC characters and/or "$" (u0024) characters and which is not a keyword token.

Identifiers can be arbitrary character strings within square brackets. This feature is also for compatibility with Microsoft SQL-Server and not a part of the SQL standard.

H41140: SQLite shall recognize as an ID token any sequence of non-zero characters that begins with "[" (u005b) and continuing through the first "]" (u005d) character.

The standard way of quoting SQL identifiers is to use double-quotes.

H41150: SQLite shall recognize as an ID token any sequence of characters that begins with a double-quote (u0022), is followed by zero or more non-zero characters and/or pairs of double-quotes (u0022) and terminates with a double-quote (u0022) that is not part of a pair.

MySQL allows identifiers to be quoted using the grave accent character. SQLite supports this for interoperability.

H41160: SQLite shall recognize as an ID token any sequence of characters that begins with a grave accent (u0060), is followed by zero or more non-zero characters and/or pairs ofgrave accents (u0060) and terminates with a grave accent (u0022) that is not part of a pair.

Next step: lots of unit tests.