@eyeseast I started work on that plugin: https://github.com/simonw/datasette-export

If I do that I should probably update insert to do those validation checks as well.

... which does imply that I'm going to do an extra layer of validation over what SQLite provides. SQLite will happily allow a text string to be added to a supposedly integer column. I'm not going to allow that - I'll return a validation error instead, unless the string can be safely coerced to the correct type.

Had a design conversation with myself in https://github.com/simonw/dclient/issues/6 where I decided that the API should allow string values to be sent to integer columns which would be automatically converted if possible to do so - as an API usability feature.

A want to call this datasette/exceptions.py inspired by Takahē: https://github.com/andrewgodwin/takahe/blob/f491fdb56e2de9200e14b855b5576009ca99dfa5/core/exceptions.py

Used it to deploy this: https://fivethirtyeight.datasettes.com/-/versions

The tests don't cover this bit at the moment.

Would be easier to write tests if there was a --generate-dir option as seen in https://datasette.io/plugins/datasette-publish-vercel

Tested a deploy:

``` % datasette publish heroku fixtures.db -n datasette-issue-1905 › Warning: heroku update available from 7.63.0 to 7.66.4. › Warning: heroku update available from 7.63.0 to 7.66.4. › Warning: heroku update available from 7.63.0 to 7.66.4. Creating datasette-issue-1905... done › Warning: heroku update available from 7.63.0 to 7.66.4. ▸ Couldn't detect GNU tar. Builds could fail due to decompression errors ▸ See https://devcenter.heroku.com/articles/platform-api-deploying-slugs#create-slug-archive ▸ Please install it, or specify the '--tar' option ▸ Falling back to node's built-in compressor

-----> Building on the Heroku-22 stack -----> Determining which buildpack to use for this app -----> Python app detected -----> Using Python version specified in runtime.txt -----> Installing python-3.11.0 -----> Installing pip 22.3.1, setuptools 63.4.3 and wheel 0.37.1 -----> Installing SQLite3 -----> Installing requirements with pip Collecting datasette Downloading datasette-0.63.1-py3-none-any.whl (231 kB) ... -----> Running post-compile hook -----> Discovering process types Procfile declares types -> web

-----> Compressing... Done: 28M -----> Launching... Released v3 https://datasette-issue-1905.herokuapp.com/ deployed to Heroku

Starting November 28th, 2022, free Heroku Dynos, free Heroku Postgres, and free Heroku Data for Redis® will no longer be available.

If you have apps using any of these resources, you must upgrade to paid plans by this date to ensure your apps continue to run and to retain your data. For students, we will announce a new program by the end of September. Learn more at https://blog.heroku.com/next-chapter ``` I had to then pay for the dino because I'd run out of free hours.

https://datasette-issue-1905.herokuapp.com/-/versions shows:

json { "python": { "version": "3.11.0", "full": "3.11.0 (main, Oct 24 2022, 21:34:02) [GCC 11.2.0]" }, "datasette": { "version": "0.63.1" }

This is a strong argument for extracting the Heroku support out to a plugin - it would allow this to be fixed with a plugin release without needing to push a full release of Datasette itself.

I think this actually needs to include a whole section of the documentation about the road to 1.0 - what to expect (planned breaking changes) etc. I can add that to the https://docs.datasette.io/en/stable/contributing.html page perhaps - or even create a Roadmap page.

New methods are documented here: https://docs.datasette.io/en/1.0-dev/internals.html#resolve-database-request

When I do this it's important to update the documentation for resolve_database() and the like:

https://github.com/simonw/datasette/blob/ee64130fa8a5ff4a24791916c696e10cf2375102/docs/internals.rst#L594

Found myself needing an await db.view_exists() method for this, similar to the existing await db.table_exists() one.

Open question: should resolve_table() know how to identify named canned queries too?

I think not, at least for the moment. Feels a bit too specialist to expose in a documented API.

Incomplete implementation of this view: ```python class RowUpdateView(BaseView): name = "row-update"

def __init__(self, datasette):
    self.ds = datasette

async def post(self, request):
    database_route = tilde_decode(request.url_vars["database"])
    table = tilde_decode(request.url_vars["table"])
    try:
        db = self.ds.get_database(route=database_route)
    except KeyError:
        return _error(["Database not found: {}".format(database_route)], 404)

    database_name = db.name
    if not await db.table_exists(table):
        return _error(["Table not found: {}".format(table)], 404)

    pk_values = urlsafe_components(request.url_vars["pks"])

    sql, params, pks = await row_sql_params_pks(db, table, pk_values)
    results = await db.execute(sql, params, truncate=True)
    rows = list(results.rows)
    if not rows:
        return _error([f"Record not found: {pk_values}"], 404)

    # Ensure user has permission to update this row
    if not await self.ds.permission_allowed(
        request.actor, "update-row", resource=(database_name, table)
    ):
        return _error(["Permission denied"], 403)

    body = await request.post_body()
    try:
        data = json.loads(body)
    except json.JSONDecodeError as e:
        return _error(["Invalid JSON: {}".format(e)])
    if not isinstance(data, dict):
        return _error(["JSON must be a dictionary"])

    def update_row(conn):
        sqlite_utils.Database(conn)[table].update(pk_values, updates)

    await db.execute_write_fn(update_row)
    result = {"ok": True}
    if data.get("return"):
        result["row"] = {"row-here": "TODO"}
    return Response.json(result, status=200)

``` This is before the refactor in: - #1896

Thanks!

Demo: https://latest.datasette.io/fixtures

This search helps too: https://ripgrep.datasette.io/-/ripgrep?pattern=%7B%25+block+nav&literal=on&ignore=on&glob=%21datasette%2F**

You get:

=> [internal] load metadata for docker.io/library/python:3.11.0-slim-bullseye 0.9s => [internal] load build context 2.3s => => transferring context: 72.38MB 2.3s => CACHED [1/6] FROM docker.io/library/python:3.11.0-slim-bullseye@sha256:1cd45c5dad845af18d71745c017325725dc979571c1bbe625b67e6051533716c 0.0s

I get:

=> [internal] load metadata for docker.io/library/python:3.11.0-slim-bullseye 1.0s => [internal] load build context 0.0s => => transferring context: 705B 0.0s => CACHED [1/6] FROM docker.io/library/python:3.11.0-slim-bullseye@sha256:1cd45c5dad845af18d71745c017325725dc979571c1bbe625b67e6051533716c 0.0s Both the image name and the hash are exactly the same. So why are you getting an error while mine works OK?

For my machine: ``` ~ % docker --version Docker version 20.10.12, build e91ed57

~ % uname -a Darwin Simons-MacBook-Pro-2.local 22.1.0 Darwin Kernel Version 22.1.0: Sun Oct 9 20:14:54 PDT 2022; root:xnu-8792.41.9~2/RELEASE_X86_64 x86_64 ```

This is so weird! What version of Datasette do you get from datasette --version there - and what's your Docker version / operating system version?

Here's the full list of 10 plugin releases for this issue:

• datasette-search-all 1.1.1
• datasette-ripgrep 0.7.1
• datasette-socrata 0.3.1
• datasette-configure-fts 1.1.1
• datasette-edit-templates 0.2
• datasette-copyable 0.3.2
• datasette-public 0.2.1
• datasette-import-table 0.3.1
• datasette-indieauth 1.2.2
• datasette-edit-schema 0.5.2

Other plugins this looks like it will affect:

• [x] datasette-ripgrep https://github.com/simonw/datasette-ripgrep/blob/03446464420130368582022eeb5944993f64ec8f/datasette_ripgrep/templates/ripgrep.html#L37-L42
• [x] datasette-socrata https://github.com/simonw/datasette-socrata/blob/32fb256a461bf0e790eca10bdc7dd9d96c20f7c4/datasette_socrata/templates/datasette_socrata_error.html#L5-L10
• [x] datasette-configure-fts https://github.com/simonw/datasette-configure-fts/blob/eca742e5d4b9190fc22d68bc0a406c575e6d09a0/datasette_configure_fts/templates/configure_fts_database.html#L9-L14
• [x] datasette-edit-templates https://github.com/simonw/datasette-edit-templates/blob/f772aff4a2a4080c949746668a8ec6302dbeb0d9/datasette_edit_templates/templates/edit_template.html#L17-L23
• [x] datasette-copyable https://github.com/simonw/datasette-copyable/blob/204d5c912a8d48c49155c67fba7339d4bb26ab9a/datasette_copyable/templates/copyable.html#L36-L43
• [x] datasette-public https://github.com/simonw/datasette-public/blob/32b6a0ba53bd5714b6b41eddd8705b213c105efc/datasette_public/templates/public_table_change_privacy.html#L5-L11
• [x] datasette-import-table https://github.com/simonw/datasettecloud-datasette/blob/37d0fe525c6649c1aec3d1ee8bc35a684570e87f/templates/import_data.html#L5-L10
• [x] datasette-edit-schema (three places)
• [x] datasette-indieauth https://github.com/simonw/datasette-indieauth/blob/a08ce67ddad6098b1240adbeff37d040e4df53b1/datasette_indieauth/templates/indieauth.html#L5-L10

That's all of them!

For datasette-copyable I want to show breadcrumbs that take database/instance permissions into account, so I'm removing {% block nav %} entirely and replacing it with this:

html+jinja {% block crumbs %} {{ crumbs.nav(request=request, database=database, table=table) }} {% endblock %}

I can't actually remember where that min-height: 70px came from. I just tried without it and it seems fine - especially since any time you add a newline in the editor it increases its height to fit.

I ran this in the DevTools console:

javascript document.querySelector('.cm-editor').style.minHeight = 'none';

Could you provide full steps to reproduce plus a SpatiaLite database file that triggered this for you? I'm not able to recreate the problem.

Did you use the --spatialite option?

I just tried this:

datasette package nps-spatialite.db

It built the image OK (I didn't see the error you reported), but running the container failed with an error:

``` /tmp % docker run -p 8001:8001 7298e8e6bbfb Usage: datasette serve [OPTIONS] [FILES]... Try 'datasette serve --help' for help.

Error: It looks like you're trying to load a SpatiaLite database without first loading the SpatiaLite module.

Read more: https://docs.datasette.io/en/stable/spatialite.html ```

Trying this out locally with this 69MB SpatiaLite file I happened to have lying around (from testing shapefile-to-sqlite a while ago): https://static.simonwillison.net/static/2022/nps-spatialite.db % datasette package nps-spatialite.db --spatialite ... => [2/6] COPY . /app 0.4s => [3/6] WORKDIR /app 0.0s => [4/6] RUN apt-get update && apt-get install -y python3-dev gcc libsqlite3-mod-spatialite && rm -rf /var/lib/apt/lists/* 29.6s => [5/6] RUN pip install -U datasette 12.0s => [6/6] RUN datasette inspect nps-spatialite.db --inspect-file inspect-data.json 2.6s => exporting to image 3.0s => => exporting layers 3.0s => => writing image sha256:4dfef1c373c5c057ef7ac22344f834d522acef24313a1b25d2eba9e500066b8f 0.0s And then:

docker run -p 8001:8001 4dfef1c373c5

This worked fine for me. I ran datasette package using Datasette 0.63.1.

Oh this is with datasette package? That should work. Will investigate.

Which Docker image are you using here? It looks like it's missing SpatiaLite from the image.

One of the big changes still left to do for Datasette 1.0 is to unify the JSON representation with the context psssed to the templates (via an ?_extra= mechanism to add extra context needed by the HTML templates), because a goal for 1.0 is for the template context to be a documented API contract such that custom templates won't break with future releases.

As such I expect to do quite a bit of refactoring and cleanup on how the template context works later on.

Also datasette-indieauth https://github.com/simonw/datasette-indieauth/blob/a08ce67ddad6098b1240adbeff37d040e4df53b1/datasette_indieauth/templates/indieauth.html#L5-L10

Looks like this issue could affect a bunch of other plugins too: https://cs.github.com/?scopeName=All+repos&scope=&q=%3Cp+class%3D%22crumbs%22%3E+user%3Asimonw

Good catch. Looks like that bug was introduced by this change: https://github.com/simonw/datasette/commit/1a5e5f2aa951e5bd731067a49819efba68fbe8ef

From:

• https://github.com/simonw/datasette/issues/1831

The search all plugin includes this code which interacts poorly with that refactor:

https://github.com/simonw/datasette-search-all/blob/847b55c368a285e4567627029624d7872ee75cac/datasette_search_all/templates/search_all.html#L31-L36

html+jinja {% block nav %} <p class="crumbs"> <a href="{{ urls.instance() }}">home</a> </p> {{ super() }} {% endblock %}

I decided to just go for the view names, not their columns.

Views aren't currently available in the _internal schema.

Just noticed that this isn't including views, which it should.

This function works even if the SQLite JSON functions are not available:

python async def _table_columns(datasette, database_name): internal = datasette.get_database("_internal") result = await internal.execute( "select table_name, name from columns where database_name = ?", [database_name], ) table_columns = {} for row in result.rows: table_columns.setdefault(row["table_name"], []).append(row["name"]) return table_columns

In terms of permissions: if you have execute-sql permission for a database then it's OK for you to see the table columns for that database.

I'm going to call this table_columns in the template context (because schema might mean CREATE TABLE ....

Here's where the schema is hard-coded at the moment: https://github.com/simonw/datasette/blob/00e233d7a7f6443cb95fb5227c23580c48551cad/datasette/templates/_codemirror_foot.html#L2-L7

I figured out how to extract that data from the _internal table in this comment: https://github.com/simonw/datasette/pull/1893#issuecomment-1317475720

Although that used JSON functions which may (in a real edge-case) not be available in the version of SQLite that Datasette is running on, so probably going to use a regular SQL query and then assemble the JSON separately.

I'm going to tackle #1897 in the next few minutes.

Tests failed due to Prettier check, just pushed a fix so it would ignore .bundle.js too.

I'll open a follow-up issue to fix the schema.

OK, let's do it! Thanks so much for this.

Deployed that to https://datasette-pr-1893.vercel.app/fixtures - looks good to me!

I can push up a commit that uses the static fixtures schema for testing, but given that the query used to generate it is authed we would still need some work to make that work on live data, right?

Yeah, push that up. I'm happy to wire in the query right after we land this.

Some ideas from walking the dog:

Challenge: standard URL routing of request to database/table/row

Standardize on the named components of the URL patterns - database, table, pks

Async function that takes the request and the Datasette instance and returns a Resolved instance with: .level - database or table or row (better name?) .database - the name of the database .db - the database object .table - the name of the table (or view) .is_view perhaps? .pk_values if it's a row Should this attempt to resolve names queries too? ``` .where_sql - the where fragment you use .where_params - accompanying dictionary

await datasette.resolve_request(request) Or even better three methods:python datasette.resolve_database(request) datasette.resolve_table(request) datasette.resolve_row(request) ``` These can be typed correctly

Methods raise NotFound if not found

In trying to write this I realize that there's a lot of duplicated code with delete row, specifically around resolving the incoming URL into a row (or a database or a table).

Since this is so common, I think it's worth extracting the logic out first.

Deployed 0a649e8f78c23e8db6869442eeb0dfe36a5443da: https://datasette-pr-1893.vercel.app/fixtures

Honestly I'm not too bothered if table names with weird characters don't work correctly here - I care about those in the Datasette fixtures.db database because Datasette aims to support ANY valid SQLite database, so I need stuff in the test suite that includes weird edge cases like this. But I would hope very few people actually create tables with spaces in their names, so it's not a huge concern to me if autocompletion doesn't work properly for those.

Here's a query that returns the exact JSON we need to pass to the schema:

https://latest.datasette.io/_internal?sql=with+inner+as+%28%0D%0A++select%0D%0A++++table_name%2C%0D%0A++++json_group_array%28name%29+as+table_columns%0D%0A++from%0D%0A++++columns%0D%0A++where%0D%0A++++database_name+%3D+%3Adatabase%0D%0A++group+by%0D%0A++++table_name%0D%0A%29%0D%0Aselect%0D%0A++json_group_object%28table_name%2C+table_columns%29%0D%0Afrom%0D%0A++inner&database=fixtures

sql with inner as ( select table_name, json_group_array(name) as table_columns from columns where database_name = :database group by table_name ) select json_group_object(table_name, table_columns) from inner Returns (after pretty-printing): json { "123_starts_with_digits": [ "content" ], "Table With Space In Name": [ "content", "pk" ], "attraction_characteristic": [ "name", "pk" ], "binary_data": [ "data" ], "complex_foreign_keys": [ "f1", "f2", "f3", "pk" ], "compound_primary_key": [ "content", "pk1", "pk2" ], "compound_three_primary_keys": [ "content", "pk1", "pk2", "pk3" ], "custom_foreign_key_label": [ "foreign_key_with_custom_label", "pk" ], "facet_cities": [ "id", "name" ], "facetable": [ "_city_id", "_neighborhood", "complex_array", "created", "distinct_some_null", "n", "on_earth", "pk", "planet_int", "state", "tags" ], "foreign_key_references": [ "foreign_key_compound_pk1", "foreign_key_compound_pk2", "foreign_key_with_blank_label", "foreign_key_with_label", "foreign_key_with_no_label", "pk" ], "infinity": [ "value" ], "no_primary_key": [ "a", "b", "c", "content" ], "primary_key_multiple_columns": [ "content", "content2", "id" ], "primary_key_multiple_columns_explicit_label": [ "content", "content2", "id" ], "roadside_attraction_characteristics": [ "attraction_id", "characteristic_id" ], "roadside_attractions": [ "address", "latitude", "longitude", "name", "pk", "url" ], "searchable": [ "name with . and spaces", "pk", "text1", "text2" ], "searchable_fts": [ "__langid", "docid", "name with . and spaces", "searchable_fts", "text1", "text2" ], "searchable_fts_docsize": [ "docid", "size" ], "searchable_fts_segdir": [ "end_block", "idx", "leaves_end_block", "level", "root", "start_block" ], "searchable_fts_segments": [ "block", "blockid" ], "searchable_fts_stat": [ "id", "value" ], "searchable_tags": [ "searchable_id", "tag" ], "select": [ "and", "group", "having", "json" ], "simple_primary_key": [ "content", "id" ], "sortable": [ "content", "pk1", "pk2", "sortable", "sortable_with_nulls", "sortable_with_nulls_2", "text" ], "table/with/slashes.csv": [ "content", "pk" ], "tags": [ "tag" ], "units": [ "distance", "frequency", "pk" ] }

I was worrying about the server-side overhead of collecting together all of the tables and column names for databases that might have hundreds of tables... but then I remember that I built the _internal table precisely for this kind of thing - so gathering all of that data should still only be a single SQL query against an in-memory database.

https://latest.datasette.io/login-as-root and then visit this page for an example query: https://latest.datasette.io/_internal?sql=select%0D%0A++database_name%2C%0D%0A++table_name%2C%0D%0A++json_group_array%28name%29%0D%0Afrom%0D%0A++columns%0D%0Awhere%0D%0A++database_name+%21%3D+%27_internal%27%0D%0Agroup+by%0D%0A++database_name%2C%0D%0A++table_name

sql select database_name, table_name, json_group_array(name) from columns where database_name != '_internal' group by database_name, table_name

database_name | table_name | json_group_array(name) -- | -- | -- extra_database | searchable | ["pk","text1","text2"] extra_database | searchable_fts | ["__langid","content","docid","searchable_fts","text1","text2"] extra_database | searchable_fts_content | ["c0text1","c1text2","c2content","docid"] extra_database | searchable_fts_segdir | ["end_block","idx","leaves_end_block","level","root","start_block"] extra_database | searchable_fts_segments | ["block","blockid"] fixtures | 123_starts_with_digits | ["content"] fixtures | Table With Space In Name | ["content","pk"] fixtures | attraction_characteristic | ["name","pk"] fixtures | binary_data | ["data"] fixtures | complex_foreign_keys | ["f1","f2","f3","pk"] fixtures | compound_primary_key | ["content","pk1","pk2"]

Tiny feature request (since you're in this code already) - I keep hitting Command+Enter on my macOS keyboard to submit the query, but the correct shortcut is Shift+Enter. Would be great if both worked!

Deployed latest copy with: datasette publish vercel fixtures.db \ --project datasette-pr-1893 \ --about 'PR 1893' \ --about_url https://github.com/simonw/datasette/pull/1893 \ --scope datasette \ --branch eccb1c6c781d69d8ec3c542ef65c78a4a0927a7c https://datasette-pr-1893.vercel.app/fixtures

I'm thinking of also adding count to the list since that's a common thing people would want to autocomplete. I notice BQ console highlights count in the same manner as other keywords like select as well.

Huh, yeah we should definitely have count - surprised it's not on the list on https://www.sqlite.org/lang_keywords.html which is why we didn't get it from the GPT-3 generated schema.

Tests passed.

Test failed again: https://github.com/simonw/sqlite-utils/actions/runs/3476950474/jobs/5812663096

E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/universe/s/spatialite/libsqlite3-mod-spatialite_4.3.0a-6build1_amd64.deb Unable to connect to azure.archive.ubuntu.com:http:

That looks like an intermittent error. I'll try running it again in the morning.

sqlite-utils % pipx run no_implicit_optional . Calculating full-repo metadata... Executing codemod... 11.43s 98% complete, 0.24s estimated for 5 files to go... Then: Finished codemodding 239 files! - Transformed 239 files successfully. - Skipped 0 files. - Failed to codemod 0 files. - 0 warnings were generated. Here's the diff: ```diff diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py index a06f4b7..e819d17 100644 --- a/sqlite_utils/db.py +++ b/sqlite_utils/db.py @@ -297,12 +297,12 @@ class Database:

 def __init__(
     self,

• filename_or_conn: Union[str, pathlib.Path, sqlite3.Connection] = None,
• filename_or_conn: Optional[Union[str, pathlib.Path, sqlite3.Connection]] = None, memory: bool = False,
• memory_name: str = None,
• memory_name: Optional[str] = None, recreate: bool = False, recursive_triggers: bool = True,
• tracer: Callable = None,

• tracer: Optional[Callable] = None, use_counts_table: bool = False, ): assert (filename_or_conn is not None and (not memory and not memory_name)) or ( @@ -341,7 +341,7 @@ class Database: self.conn.close()

  @contextlib.contextmanager - def tracer(self, tracer: Callable = None): + def tracer(self, tracer: Optional[Callable] = None): """ Context manager to temporarily set a tracer function - all executed SQL queries will be passed to this. @@ -378,7 +378,7 @@ class Database:

  def register_function( self, - fn: Callable = None, + fn: Optional[Callable] = None, deterministic: bool = False, replace: bool = False, name: Optional[str] = None, @@ -879,7 +879,7 @@ class Database: pk: Optional[Any] = None, foreign_keys: Optional[ForeignKeysType] = None, column_order: Optional[List[str]] = None, - not_null: Iterable[str] = None, + not_null: Optional[Iterable[str]] = None, defaults: Optional[Dict[str, Any]] = None, hash_id: Optional[str] = None, hash_id_columns: Optional[Iterable[str]] = None, @@ -1129,7 +1129,7 @@ class Database: sql += " [{}]".format(name) self.execute(sql)

• def init_spatialite(self, path: str = None) -> bool:

• def init_spatialite(self, path: Optional[str] = None) -> bool: """ The init_spatialite method will load and initialize the SpatiaLite extension. The path argument should be an absolute path to the compiled extension, which @@ -1182,7 +1182,7 @@ class Queryable:

  def count_where( self, - where: str = None, + where: Optional[str] = None, where_args: Optional[Union[Iterable, dict]] = None, ) -> int: """ @@ -1213,12 +1213,12 @@ class Queryable:

  def rows_where( self, - where: str = None, + where: Optional[str] = None, where_args: Optional[Union[Iterable, dict]] = None, - order_by: str = None, + order_by: Optional[str] = None, select: str = "*", - limit: int = None, - offset: int = None, + limit: Optional[int] = None, + offset: Optional[int] = None, ) -> Generator[dict, None, None]: """ Iterate over every row in this table or view that matches the specified where clause. @@ -1251,11 +1251,11 @@ class Queryable:

  def pks_and_rows_where( self, - where: str = None, + where: Optional[str] = None, where_args: Optional[Union[Iterable, dict]] = None, - order_by: str = None, - limit: int = None, - offset: int = None, + order_by: Optional[str] = None, + limit: Optional[int] = None, + offset: Optional[int] = None, ) -> Generator[Tuple[Any, Dict], None, None]: """ Like .rows_where() but returns (pk, row) pairs - pk can be a single value or tuple. @@ -1345,7 +1345,7 @@ class Table(Queryable): pk: Optional[Any] = None, foreign_keys: Optional[ForeignKeysType] = None, column_order: Optional[List[str]] = None, - not_null: Iterable[str] = None, + not_null: Optional[Iterable[str]] = None, defaults: Optional[Dict[str, Any]] = None, batch_size: int = 100, hash_id: Optional[str] = None, @@ -1545,7 +1545,7 @@ class Table(Queryable): pk: Optional[Any] = None, foreign_keys: Optional[ForeignKeysType] = None, column_order: Optional[List[str]] = None, - not_null: Iterable[str] = None, + not_null: Optional[Iterable[str]] = None, defaults: Optional[Dict[str, Any]] = None, hash_id: Optional[str] = None, hash_id_columns: Optional[Iterable[str]] = None, @@ -2464,7 +2464,7 @@ class Table(Queryable): columns: Optional[Iterable[str]] = None, limit: Optional[int] = None, offset: Optional[int] = None, - where: str = None, + where: Optional[str] = None, where_args: Optional[Union[Iterable, dict]] = None, quote: bool = False, ) -> Generator[dict, None, None]: @@ -2527,7 +2527,7 @@ class Table(Queryable):

  def delete_where( self, - where: str = None, + where: Optional[str] = None, where_args: Optional[Union[Iterable, dict]] = None, analyze: bool = False, ) -> "Table": ```

Should note though that this is a classic example of GPT-3 making stuff up in places.

current: Returns the current date, time, or timestamp

select current throws an error for me: https://latest.datasette.io/_memory?sql=select+current

select current_date, current_time, current_timestamp works though: https://latest.datasette.io/_memory?sql=select+current_date%2C+current_time%2C+current_timestamp

So let's drop current from the list. I'm OK with it though, I think it's likely good enough for the first attempt at this.

We should drop temp and temporary too.

So I think our dialect (at least to start with) should be:

keywords: "and as asc between by case cast current current_date current_time current_timestamp desc distinct each else escape except exists explain filter first for from full generated group having if in index inner intersect into isnull join last left like limit not null or order outer over pragma primary query raise range regexp right rollback row select set table temp temporary then to union unique using values view virtual when where", // https://www.sqlite.org/datatype3.html types: "null integer real text blob", builtin: "" I left builtin blank here because I don't think we need any of those things at all.

OK, I decided to play around with GPT-3 for this. https://beta.openai.com/playground

I prompted:

``` For each keyword say it it can be used in a SQLite select statement: and as asc between by case cast cross current current_date current_time current_timestamp desc detach distinct each else end escape except exclude exclusive exists explain fail filter first following for foreign from full generated glob group groups having if ignore immediate in index indexed initially inner insert instead intersect into is isnull join key last left like limit match materialized natural no not nothing notnull null nulls of offset on or order others outer over partition plan pragma preceding primary query raise range recursive references regexp reindex release rename replace restrict returning right rollback row rows savepoint select set table temp temporary then ties to transaction trigger unbounded union unique update using vacuum values view virtual when where window with without

• and: yes
• as: yes
• asc: yes
• between: yes
• by: yes
• case: yes
• cast: yes
• current: yes
• current_date: yes
• current_time: yes
• current_timestamp: yes
• desc: yes
• detach: no
• distinct: yes
• each: yes
• else: yes
• end: yes
• escape: yes
• except: no
• exclude: yes
• exclusive: no Having given it these examples I ran it a few times:
• exists: yes
• explain: no
• fail: no
• filter: yes
• first: yes
• following: no
• for: yes
• foreign: no
• from: yes
• full: no
• generated: no
• glob: yes
• group: yes
• groups: no
• having: yes
• if: yes
• ignore: no
• immediate: no
• in: yes
• index: yes
• indexed: no
• initially: no
• inner: no
• insert: no
• instead: no
• intersect: no
• into: yes
• is: yes
• isnull: yes
• join: yes
• key: no
• last: yes
• left: yes
• like: yes
• limit: yes
• match: no
• materialized: no -natural: yes
• no: no
• not: no
• nothing: no
• notnull: yes
• null: yes
• nulls: yes
• of: no
• offset: yes
• on: yes
• or: yes
• order: yes
• others: yes
• outer: no
• over: no
• partition: no
• pragma: no
• preceding: no
• primary: no
• query: no
• raise: no
• range: no
• recursive: no
• references: no
• regexp: yes
• reindex: no
• release: no
• rename: no
• replace: no
• restrict: no
• returning: yes
• right: yes
• rollback: no
• row: yes
• rows: no
• savepoint: no
• select: yes
• set: no
• table: yes
• temp: yes
• temporary: yes
• then: yes
• ties: no
• to: yes
• transaction: no
• trigger: no
• unbounded: no
• union: yes
• unique: yes
• update: no
• using: yes
• vacuum: no
• values: yes
• view: yes
• virtual: no
• when: yes
• where: yes
• window: no
• with: yes

• without: no I added this: A space separated list of all of the keywords for which the answer was yes: It replied: and as asc between by case cast current current_date current_time current_timestamp desc distinct each else escape except exists explain filter first for from full generated group having if in index inner intersect into isnull join last left like limit not null or order outer over pragma primary query raise range regexp right rollback row select set table temp temporary then to union unique using values view virtual when where Then I tried doing this: Produce a short summary of how each of those are used:

• and: ``` This kind of worked too! It ended up defining every keyword, not just the ones for which the "is it used in select" was "yes", but it gave me this list:

• and: Used to combine multiple conditions in a WHERE clause
• as: Rename a column or table in the output of a query
• asc: Sort results in ascending order
• between: Used in a WHERE clause to specify a range
• by: Specifies how to sort the results of a query
• case: Used to create different outputs based on conditions
• cast: Converts a value from one data type to another
• current: Returns the current date, time, or timestamp
• current_date: Returns the current date
• current_time: Returns the current time
• current_timestamp: Returns the current timestamp
• desc: Sort results in descending order
• distinct: Selects only unique values
• each: Used with aggregate functions
• else: Used with the CASE statement
• escape: Used to escape special characters in a string
• except: Returns all rows from the first query that are not in the second query
• exists: Used with subqueries
• explain: Analyzes a query and returns information about it
• fail: Used with the ON CONFLICT clause
• filter: Used to specify conditions for aggregate functions
• first: Used to specify conditions for aggregate functions
• following: Used with window functions
• for: Used to specify conditions for aggregate functions
• foreign: Creates a foreign key
• from: Specifies the table(s) to select data from
• full: Used with outer joins
• generated: Used with generated columns
• glob: Used to match a string with a pattern
• group: Groups together rows that have the same values
• groups: Used with aggregate functions
• having: Specifies conditions for groups
• if: Used with the ON CONFLICT clause
• ignore: Used with the ON CONFLICT clause
• immediate: Used with the ON CONFLICT clause
• in: Used to check if a value is in a list
• index: Creates an index on a column or columns
• indexed: Used with the ON CONFLICT clause
• initially: Used with the ON CONFLICT clause
• inner: Creates an inner join
• insert: Inserts new data into a table
• instead: Used with triggers
• intersect: Returns all rows that are in both queries
• into: Specifies the table to insert data into
• is: Used to compare values
• isnull: Checks if a value is null
• join: Joins two or more tables together
• key: Creates a primary key or unique key
• last: Used to specify conditions for aggregate functions
• left: Creates a left join
• like: Checks if a value matches a pattern
• limit: Specifies the maximum number of rows to return
• not: Negates a condition
• null: Specifies a null value
• nulls: Used with the ORDER BY clause
• of: Used with window functions
• offset: Specifies the number of rows to skip
• on: Specifies conditions for a join
• or: Checks if either condition is true
• order: Specifies how to sort the results of a query
• others: Used with the CASE statement
• outer: Creates an outer join
• over: Used with window functions
• partition: Used with window functions
• pragma: Used to query or change various internal settings
• preceding: Used with window functions
• primary: Creates a primary key
• query: Used with the ON CONFLICT clause
• raise: Used with the ON CONFLICT clause
• range: Used with window functions
• recursive: Used with common table expressions
• references: Creates a foreign key
• regexp: Checks if a value matches a regular expression
• reindex: Recreates an index
• release: Used with savepoints
• rename: Changes the name of a table or column
• replace: Inserts new data into a table or updates existing data
• restrict: Used with savepoints
• returning: Returns data from inserted or updated rows
• right: Creates a right join
• rollback: Rolls back a transaction
• row: Used with aggregate functions
• select: Selects data from a table
• set: Changes the value of a column
• table: Specifies the table(s) to select data from
• temp: Creates a temporary table
• temporary: Creates a temporary table
• then: Used with the CASE statement
• ties: Used with the ORDER BY clause
• to: Used with common table expressions
• transaction: Starts a transaction
• trigger: Creates a trigger
• unbounded: Used with window functions
• union: Returns all rows from both queries
• unique: Creates a unique constraint
• update: Updates existing data in a table
• using: Specifies the table(s) to join
• vacuum: Optimizes a database
• values: Specifies the values to insert into a table
• view: Creates a view
• virtual: Creates a virtual table
• when: Used with the CASE statement
• where: Specifies conditions for a query
• window: Used with window functions
• with: Specifies common table expressions used in a query
• without: Used with indexes

The main issue is that we don't pass the relevant table data down to QueryView.

If you can come up with a static example JSON data structure example that does the right thing, I'm happy to refactor QueryView to make that available to the template - or even have a separate fetch() that grabs just the data needed for the autocomplete as a separate hit when the page loads (whichever has better performance implications). I'm working a fair amount in the view classes at the moment so adding this to that work would make sense.

Have you ever seen CodeMirror correctly auto-completing columns? I'm not entirely sure I believe that the feature works anywhere else.

Deployed a fresh copy: datasette publish vercel fixtures.db \ --branch b7b2942b13f9ea09cfa9f8c73e2869b9bd2349ae \ --project datasette-pr-1893 \ --about 'PR 1893' \ --about_url https://github.com/simonw/datasette/pull/1893 \ --scope datasette https://datasette-pr-1893.vercel.app/fixtures

Actually this works as it should in desktop Safari:

I'm going to just put up with the weird behaviour in Mobile Safari.

Yeah I haven't written this down anywhere but Datasette definitely has an undocumented preference for lower-case SQL.

https://bugs.webkit.org/show_bug.cgi?id=201768 - " Datalist option's label not used" - marked as RESOLVED FIXED on March 31st 2020.

The commit: https://trac.webkit.org/changeset/259330/webkit

And here's the test mirrored on GitHub: https://cs.github.com/qtwebkit/webkit-mirror/blob/cc3fcd0b4bad1f7cf77c26e34aa01d16618d6d5e/LayoutTests/fast/forms/datalist/datalist-option-labels.html?q=datalist-option-labels.html

Here's a polyfill for <datalist>: https://github.com/mfranzke/datalist-polyfill

It shouldn't be necessary now that Safari has shipped support (apparently added in https://developer.apple.com/documentation/safari-release-notes/safari-12_1-release-notes#3130314 Safari 12.1 in March 2019).

But it does look like Safari doesn't support differing label and value attributes, though documentation about this is hard to come by.

If you rebase from main you should get the fix for that test failure.

Oops, introduced a test failure:

``` def test_table_html_foreign_key_facets(app_client): response = app_client.get( "/fixtures/foreign_key_references?_facet=foreign_key_with_blank_label" ) assert response.status == 200

  assert (
        '<li><a href="http://localhost/fixtures/foreign_key_references?_facet=foreign_key_with_blank_label&amp;foreign_key_with_blank_label=3">'
        "-</a> 1</li>"
    ) in response.text

E assert '

- 1

' in '<!DOCTYPE html>\n<html>\n<head>\n <title>fixtures: foreign_key_references: 2 rows</title>\n <link rel="styleshe.../script>\n\n\n<!-- Templates considered: table-fixtures-foreign_key_references.html, *table.html -->\n</body>\n</html>' E + where '<!DOCTYPE html>\n<html>\n<head>\n <title>fixtures: foreign_key_references: 2 rows</title>\n <link rel="styleshe.../script>\n\n\n\n</body>\n</html>' = <datasette.utils.testing.TestResponse object at 0x7fd1b0080640>.text ``` Need to fix this test:

https://github.com/simonw/datasette/blob/eac028d3f77aa5473a5fcf59240635a1bca80f7d/tests/test_table_html.py#L616-L624

Oops, the tests are failing because of a test failure I introduced here: - #1890

The resize handle doesn't appear on Mobile Safari on iPhone - I don't think that particularly matters though.

The textarea does get a weird border around it when focused on iPhone though.

Focused:

Not focused:

If you can get a version of this working with table and column autocompletion just using a static JavaScript object in the source code with the right tables and columns, I'm happy to take on the work of turning that static object into something that Datasette includes in the page itself with all of the correct values.

Resizing works great for me - and the page automatically sizes the editor to fit an existing query, e.g. on https://datasette-pr-1893.vercel.app/fixtures?sql=select+id%2C+content%2C+content2%0D%0A++from+primary_key_multiple_columns_explicit_label%0D%0A++order+by+id%0D%0A++limit+101

Autocomplete here looks promising (I've wanted that to work for years!), but it does currently show a whole bunch of suggestions which aren't part of the SQLite SQL dialect:

I just deployed a demo instance like this (using the commit hash from this PR):

bash datasette publish vercel fixtures.db \ --branch 544f7025900b78f63c34b9985522271ba5fd9c0f \ --project datasette-pr-1893 \ --scope datasette \ --about 'PR 1893' \ --about_url https://github.com/simonw/datasette/pull/1893 Here's the result: https://datasette-pr-1893.vercel.app/fixtures

If the update succeeds it will return {"ok": true}.

For consistency with /db/table/-/insert you can pass "return": true and it will return a "row" key with the now-updated full row.

I'm also not going to implement "alter": true yet (which would add any missing columns based on the update) - I'll hold that off for a later feature.

I'm going to use the error format I've been experimenting with here: - #1875

json { "type": "https://example.net/validation-error", "title": "Your request is not valid.", "errors": [ { "detail": "must be a positive integer", "pointer": "#/age" }, { "detail": "must be 'green', 'red' or 'blue'", "pointer": "#/profile/color" } ] } I'm not quite ready to commit to a type URL though, so I'll leave that to be solved later should I fully embrace that RFC.

The initial design I'm going to implement will look like this:

POST /db/table/1/-/update Authorization: Bearer xxx Content-Type: application/json json { "update": { "name": "New name" } } Any fields that are not yet columns will return an error.

Should it enforce types, in as much as an integer column should have a JSON integer passed to it, or should it allow strings containing valid integers?

I'm going to allow strings, mainly as a workaround for the fact that JavaScript integers have a maximum size.

One slight concern: https://latest.datasette.io/ will increasingly reflect a version that isn't the most recent production release.

I might setup https://stable.datasette.io/ as a demo instance of the most recent non-alpha release to compensate for that.

I'll do this after the 1.0a0 release: - #1708

Datasette 1.0 alpha 1

This alpha release is a preview of Datasette 1.0.

Datasette 1.0 marks a significant milestone in the project: it is the point from which various aspects of Datasette can be considered "stable", in that code developed against them should expect not to be broken by future releases in the 1.x series.

This will hold true until the next major version release, Datasette 2.0 - which we hope to hold off releasing for as long as possible.

The following Datasette components should be be considered stable after 1.0:

• The plugin API. Plugins developed against 1.0 should continue to work unmodified throughout the 1.x series.
• The JSON API. Code written that interacts with Datasette's default JSON web API should continue to work.
• The template context. If you build custom templates against Datasette your custom pages should continue to work.

Note that none of these components will cease to introduce new features. New plugin hooks, new JSON APIs and new template context variables can be introduced without breaking existing code.

Since this alpha release previews features that will be frozen for 1.0, please test this thoroughly against your existing Datasette projects.

You can install the alpha using:

pip install datasette==1.0a0

JSON API changes

The most significant changes introduced in this new alpha concern Datasette's JSON API.

The default JSON returned by the /database/table.json endpoint has changed. It now returns an object with two keys: rows - which contains a list of objects representing the rows in the table or query, and more containing a boolean that shows if there are more rows or if this object contains them all.

json { "rows": [{ "id": 1, "name": "Name 1" }, { "id": 2, "name": "Name 2" }], "more": false } [ Initially I thought about going with next_url, which would be null if you have reached the last page of records. Maybe that would be better? But since next_url cannot be provided on query pages, should this be part of the default format at all? ]

Use ?_extra= to retrieve extra fields

The default format can be expanded using one or more ?_extra= parameters. This takes names of extra keys you would like to include. These can be comma-separated or ?_extra= can be applied multiple times.

For example:

/database/table.json?_extra=total

This adds a "total": 124 field to the returned JSON.

[ Question: if you do ?_facet=foo then do you still need to do ?_extra=facets - I think not? ]

Annoying: Mobile Safari doesn't seem to support separate labels and values. I should probably disable this feature on that browser, at least for foreign key facets (for the moment).

Looks like I can fix that like so: html <datalist id="datalist-_city_id"> <option label="San Francisco" value="1"></option> <option label="Los Angeles" value="2"></option> <option label="Detroit" value="3"></option> <option label="Memnonia" value="4"></option> </datalist>

Spotted a bug with this on https://latest.datasette.io/fixtures/facetable?_facet=_city_id - the _city_id column is a foreign key, so you need to type 1 or 2 - but the autocomplete list shows the full text names for the cities.

Demo now live here: https://congress-legislators.datasettes.com/legislators/legislator_terms?_facet=party - select party and start typing.

Wrote a TIL about <datalist>: https://til.simonwillison.net/html/datalist

I implemented this as part of /db/-/create.

https://docs.datasette.io/en/1.0-dev/json_api.html#creating-a-table-from-example-data

That prototype actually works really well! I'm going to add that to table.js.

Here's a prototype: ```javascript function createDataLists() { var facetResults = document.querySelectorAll(".facet-results [data-column]"); Array.from(facetResults).forEach(function (facetResult) { // Use link text from all links in the facet result var linkTexts = Array.from( facetResult.querySelectorAll("li:not(.facet-truncated) a") ).map(function (link) { return link.textContent; }); // Create a datalist element var datalist = document.createElement("datalist"); datalist.id = "datalist-" + facetResult.dataset.column; // Create an option element for each link text linkTexts.forEach(function (linkText) { var option = document.createElement("option"); option.value = linkText; datalist.appendChild(option); }); // Add the datalist to the facet result facetResult.appendChild(datalist); }); } createDataLists();

// When any select with name=_filter_column changes, update the datalist document.body.addEventListener("change", function (event) { if (event.target.name === "_filter_column") { event.target .closest(".filter-row") .querySelector(".filter-value") .setAttribute("list", "datalist-" + event.target.value); } }); ```

This finds the right links on the page:

document.querySelectorAll('.facet-results [data-column] li:not(.facet-truncated) a')

This could start out as a purely JavaScript enhancement for pages that already figured out the available values through faceting, like you suggested.

I tried this out on https://congress-legislators.datasettes.com/legislators/legislator_terms for the party column - here's the demo:

I made this work by dropping the following HTML into the page in the browser DevTools: ```html <datalist id="party">

Oh interesting... this doesn't even need to be attached to the visible faceting feature, necessarily: Datasette could try to detect when a column has a limited number of options (which the faceting code handles already) and could turn those into an auto-complete interface.

There's actually a native HTML element for this these days: the <datalist> https://developer.mozilla.org/en-US/docs/Web/HTML/Element/datalist

Documentation:

• https://docs.datasette.io/en/1.0-dev/json_api.html#creating-a-table
• https://docs.datasette.io/en/1.0-dev/json_api.html#creating-a-table-from-example-data

Wrote a TIL about how I wrote some of those tests with Copilot: https://til.simonwillison.net/gpt3/writing-test-with-copilot

Rough initial prototype: ```diff diff --git a/datasette/views/table.py b/datasette/views/table.py index 8b987221..518ac578 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -1103,19 +1103,30 @@ class TableInsertView(BaseView): except json.JSONDecodeError as e: return _errors(["Invalid JSON: {}".format(e)]) if not isinstance(data, dict): - return _errors(["JSON must be a dictionary"]) + return _errors([{"detail": "JSON must be a dictionary", "pointer": "#/"}]) keys = data.keys()

     # keys must contain "row" or "rows"
     if "row" not in keys and "rows" not in keys:
         return _errors(['JSON must have one or other of "row" or "rows"'])
     rows = []

• was_single_row = False if "row" in keys: if "rows" in keys:
• return _errors(['Cannot use "row" and "rows" at the same time'])
• return _errors(
• [
• {
• "detail": 'Cannot use "row" and "rows" at the same time',
• "pointer": "#/row",
• }
• ]
• )
• was_single_row = True row = data["row"] if not isinstance(row, dict):
• return _errors(['"row" must be a dictionary'])
• return _errors(
• [{"detail": '"row" must be a dictionary', "pointer": "#/row"}]
• ) rows = [row] data["return"] = True else: @@ -1152,9 +1163,12 @@ class TableInsertView(BaseView): invalid_columns = set(row.keys()) - columns if invalid_columns: errors.append(
• "Row {} has invalid columns: {}".format(
• i, ", ".join(sorted(invalid_columns))
• )
• {
• "detail": "Invalid columns: {}".format(
• ", ".join(sorted(invalid_columns))
• ),
• "pointer": "#/blah/",
• } ) if errors: return _errors(errors) ```

Worth noting this bit in RFC 7807:

The fictional problem type here defines the "errors" extension, an array that describes the details of each validation error. Each member is an object containing "detail" to describe the issue, and "pointer" to locate the problem within the request's content using a JSON Pointer [JSON-POINTER].

So the list of "errors" with JSON Pointer isn't technically part of the spec, it's an imaginary extension.

It fits what I need to do though, so I'm inclined to stick with it anyway.

TIL: https://til.simonwillison.net/json/json-pointer

Spec looks pretty simple:

A JSON Pointer is a Unicode string (see [RFC4627], Section 3) containing a sequence of zero or more reference tokens, each prefixed by a / (%x2F) character.

Because the characters ~ (%x7E) and / (%x2F) have special meanings in JSON Pointer, ~ needs to be encoded as ~0 and / needs to be encoded as ~1 when these characters appear in a reference token.