# issue_comments

html_url: https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029338360
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/385
node_id: IC_kwDOCGYnMM49WnT4
created_at: 2022-02-03T19:43:56Z
updated_at: 2022-02-03T19:43:56Z
author_association: CONTRIBUTOR
body:
  Works for me. I was just looking at how the FTS extensions work and they're just methods, too. So this can be consistent with that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029370537
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/385
node_id: IC_kwDOCGYnMM49WvKp
created_at: 2022-02-03T20:25:58Z
updated_at: 2022-02-03T20:25:58Z
author_association: CONTRIBUTOR
body:
  OK, I moved all the GIS helpers into `db.py` as methods on `Database` and `Table`, and I put `find_spatialite` back in `utils.py`. I deleted `gis.py`, since there's nothing left it. Docs and tests are updated and passing.

  I think this is better.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/394#issuecomment-1029402029
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/394
node_id: IC_kwDOCGYnMM49W22t
created_at: 2022-02-03T21:06:35Z
updated_at: 2022-02-03T21:06:35Z
author_association: OWNER
body:
  This broke on Windows: https://github.com/simonw/sqlite-utils/runs/5056912641
  ```
              if recreate and os.path.exists(filename_or_conn):
  >               os.remove(filename_or_conn)
  E               PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_recreate_False_True_0\\data.db'
  ```
  I'm going to revert it from `main` for the moment.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1029402837
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/369
node_id: IC_kwDOCGYnMM49W3DV
created_at: 2022-02-03T21:07:35Z
updated_at: 2022-02-03T21:07:35Z
author_association: OWNER
body:
  Closing this - it was something I was curious about, but evidently not curious enough to actually do the work!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/363#issuecomment-1029469630
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/363
node_id: IC_kwDOCGYnMM49XHW-
created_at: 2022-02-03T22:42:36Z
updated_at: 2022-02-03T22:42:36Z
author_association: OWNER
body:
  > This check should run inside the `.insert_all()` method. It should raise a custom exception which the CLI code can then catch and turn into a click error.

  Actually no that doesn't work, because this line causes an error before we even get to `.insert_all()`:

  https://github.com/simonw/sqlite-utils/blob/7d928f83085fb285f294dbdaeb93fd94a44d5d44/sqlite_utils/cli.py#L1012-L1013

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/363#issuecomment-1029475387
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/363
node_id: IC_kwDOCGYnMM49XIw7
created_at: 2022-02-03T22:52:30Z
updated_at: 2022-02-03T22:52:30Z
author_association: OWNER
body:
  Demos:
  ```
  % sqlite-utils insert /tmp/all.db blah /tmp/log.log --convert '[1]' --text
  Error: Rows must all be dictionaries, got: 1
  % sqlite-utils insert /tmp/all.db blah /tmp/log.log --convert '1' --text             
  Error: --convert must return dict or iterator
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1080#issuecomment-1029691693
issue_url: https://api.github.com/repos/simonw/datasette/issues/1080
node_id: IC_kwDOBm6k_c49X9kt
created_at: 2022-02-04T06:16:45Z
updated_at: 2022-02-04T06:16:45Z
author_association: OWNER
body:
  Had a new, different idea for how this could work: support a `?_group_count=colname` parameter to the table view, which turns the page into a `select colname, count(*) ... group by colname` query - but keeps things like the filter interface, facet selection, search box and so on.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1080#issuecomment-1029695083
issue_url: https://api.github.com/repos/simonw/datasette/issues/1080
node_id: IC_kwDOBm6k_c49X-Zr
created_at: 2022-02-04T06:24:40Z
updated_at: 2022-02-04T06:25:18Z
author_association: OWNER
body:
  An initial prototype of that in my local `group-count` branch quickly started running into problems:
  ```diff
  diff --git a/datasette/views/table.py b/datasette/views/table.py
  index be9e9c3..d30efe1 100644
  --- a/datasette/views/table.py
  +++ b/datasette/views/table.py
  @@ -105,8 +105,12 @@ class RowTableShared(DataView):
                   type_ = "integer"
                   notnull = 0
               else:
  -                type_ = column_details[r[0]].type
  -                notnull = column_details[r[0]].notnull
  +                try:
  +                    type_ = column_details[r[0]].type
  +                    notnull = column_details[r[0]].notnull
  +                except KeyError: # Probably count(*)
  +                    type_ = "integer"
  +                    notnull = False
               columns.append(
                   {
                       "name": r[0],
  @@ -613,6 +617,15 @@ class TableView(RowTableShared):
               offset=offset,
           )
   
  +        # If ?_group_count we convert the SQL query here
  +        group_count = request.args.getlist("_group_count")
  +        if group_count:
  +            wrapped_sql = "select {cols}, count(*) from ({sql}) group by {cols}".format(
  +                cols=", ".join(group_count),
  +                sql=sql,
  +            )
  +            sql = wrapped_sql
  +
           if request.args.get("_timelimit"):
               extra_args["custom_time_limit"] = int(request.args.get("_timelimit"))
  ```
  Resulted in errors like this one:
  ```
      pk_path = path_from_row_pks(row, pks, not pks, False)
    File "/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py", line 82, in path_from_row_pks
      bits = [
    File "/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py", line 83, in <listcomp>
      row[pk]["value"] if isinstance(row[pk], dict) else row[pk] for pk in pks
  IndexError: No item with that key
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029703216
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/385
node_id: IC_kwDOCGYnMM49YAYw
created_at: 2022-02-04T06:45:43Z
updated_at: 2022-02-04T06:45:43Z
author_association: OWNER
body:
  Shipped this as `sqlite-utils` 3.23: https://sqlite-utils.datasette.io/en/stable/changelog.html#v3-23

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1629#issuecomment-1029980337
issue_url: https://api.github.com/repos/simonw/datasette/issues/1629
node_id: IC_kwDOBm6k_c49ZECx
created_at: 2022-02-04T13:21:09Z
updated_at: 2022-02-04T13:21:09Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1629?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1629](https://codecov.io/gh/simonw/datasette/pull/1629?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (1c0d848) into [main](https://codecov.io/gh/simonw/datasette/commit/1af1041f91a9b91b321078d354132d1df5204660?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (1af1041) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1629/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1629?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1629   +/-   ##
  =======================================
    Coverage   92.16%   92.16%           
  =======================================
    Files          34       34           
    Lines        4531     4531           
  =======================================
    Hits         4176     4176           
    Misses        355      355           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1629?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1629?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [1af1041...1c0d848](https://codecov.io/gh/simonw/datasette/pull/1629?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1030002502
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/385
node_id: IC_kwDOCGYnMM49ZJdG
created_at: 2022-02-04T13:50:19Z
updated_at: 2022-02-04T13:50:19Z
author_association: CONTRIBUTOR
body:
  Awesome. Thanks for your help getting it in. Will now look at adding CLI versions of this. It's going to be super helpful on a bunch of my projects.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1030454114
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/398
node_id: IC_kwDOCGYnMM49a3ti
created_at: 2022-02-05T00:14:47Z
updated_at: 2022-02-05T00:14:47Z
author_association: OWNER
body:
  I like these designs a lot. I would suggest `sqlite-utils create database.db --init-spatialite` there for consistency with the `sqlite-utils init-spatialite database.db` command.

  The other part of this story is how we support actually inserting spatial data from the command-line. I opened an issue about the challenges in doing that for the Python API here - #399 - but we need a good answer for the CLI too.

  I don't yet have any good ideas here. The `conversions=` option in the Python library was designed to cover these kinds of cases but it's pretty clunky and I don't think it's very widely used: https://sqlite-utils.datasette.io/en/stable/python-api.html#converting-column-values-using-sql-functions

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030455715
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/399
node_id: IC_kwDOCGYnMM49a4Gj
created_at: 2022-02-05T00:15:28Z
updated_at: 2022-02-05T00:15:28Z
author_association: OWNER
body:
  The `conversions=` argument to `.insert()` and friends is designed to handle this case, but I don't think it's very elegant: https://sqlite-utils.datasette.io/en/stable/python-api.html#converting-column-values-using-sql-functions

  ```python
  db["places"].insert(
      {"name": "Wales", "geometry": wkt},
      conversions={"geometry": "GeomFromText(?, 4326)"},
  )
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1030456717
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/398
node_id: IC_kwDOCGYnMM49a4WN
created_at: 2022-02-05T00:16:42Z
updated_at: 2022-02-05T00:16:42Z
author_association: OWNER
body:
  > The one thing worth highlighting in docs is that geometry columns can only be added to existing tables. Trying to add a geometry column to a table that doesn't exist yet might mean you have a schema like `{"rowid": int, "geometry": bytes}`. Might be worth nudging people to explicitly create a table first, then add geometry columns.

  That's a good call. I'm happy for `sqlite-utils add-geometry-column` to throw an error if the table doesn't exist yet.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030461163
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/399
node_id: IC_kwDOCGYnMM49a5br
created_at: 2022-02-05T00:30:18Z
updated_at: 2022-02-05T00:30:18Z
author_association: OWNER
body:
  I wonder what the most developer-friendly way to insert geometry data into SpatiaLite is?

  From https://www.gaia-gis.it/gaia-sins/spatialite-sql-latest.html it looks like these are the main options:

  - `GeomFromText( wkt String [ , SRID Integer] )`
  - `GeomFromWKB( wkbGeometry Binary [ , SRID Integer] )`
  - `GeomFromKml( KmlGeometry String )`
  - `GeomFromGML( gmlGeometry String )`
  - `GeomFromGeoJSON( geoJSONGeometry String )`
  - `GeomFromEWKB( ewkbGeometry String )`
  - `GeomFromEWKT( ewktGeometry String )`
  - `GeomFromFGF( fgfGeometry Binary [ , SRID Integer] )`
  - `GeomFromTWKB( twkbGeometry BLOB [ , SRID Integer] )`
  - `GeomFromGPB( geom GPKG Blob Geometry )` - GeoPackage format
  - `GeomFromExifGpsBlob( image BLOB )`

  Interesting that some accept an SRID and others do not - presumably `GeomFromGeoJSON()` always uses SRID=4326?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030465557
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/399
node_id: IC_kwDOCGYnMM49a6gV
created_at: 2022-02-05T00:39:09Z
updated_at: 2022-02-05T00:39:09Z
author_association: OWNER
body:
  I can't seem to get `GeomFromGeoJSON()` to work - example:

  https://calands.datasettes.com/calands?sql=select+IsValid%28SetSRID%28GeomFromGeoJSON%28%27%7B%0D%0A++++++++%22type%22%3A+%22Point%22%2C%0D%0A++++++++%22coordinates%22%3A+%5B%0D%0A++++++++++-94.921875%2C%0D%0A++++++++++45.460130637921004%0D%0A++++++++%5D%0D%0A++++++%7D%27%29%2C+4326%29%29

  ```sql
  select IsValid(SetSRID(GeomFromGeoJSON('{
          "type": "Point",
          "coordinates": [
            -94.921875,
            45.460130637921004
          ]
        }'), 4326))
  ```
  Returns `-1` suggesting the geometry is not valid. Just doing this (with or without that `SetSRID()` function) returns null:

  ```sql
  select SetSRID(GeomFromGeoJSON('{
          "type": "Point",
          "coordinates": [
            -94.921875,
            45.460130637921004
          ]
        }'), 4326)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030466255
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/399
node_id: IC_kwDOCGYnMM49a6rP
created_at: 2022-02-05T00:41:35Z
updated_at: 2022-02-05T00:42:23Z
author_association: OWNER
body:
  Wow, it was the newlines that broke it! This works fine: 
  ```sql
  select AsWKT(SetSRID(GeomFromGeoJSON('{"type": "Point","coordinates": [-94.921875,45.460130637921004]}'), 4326))
  ```
  https://calands.datasettes.com/calands?sql=select+AsWKT%28SetSRID%28GeomFromGeoJSON%28%27%7B%22type%22%3A+%22Point%22%2C%22coordinates%22%3A+%5B-94.921875%2C45.460130637921004%5D%7D%27%29%2C+4326%29%29

  And removing `SetSRID()` returns exactly the same result:

  https://calands.datasettes.com/calands?sql=select+AsWKT%28GeomFromGeoJSON%28%27%7B%22type%22%3A+%22Point%22%2C%22coordinates%22%3A+%5B-94.921875%2C45.460130637921004%5D%7D%27%29%29

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030468418
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/399
node_id: IC_kwDOCGYnMM49a7NC
created_at: 2022-02-05T00:49:08Z
updated_at: 2022-02-05T22:59:06Z
author_association: OWNER
body:
  I'm trying to think of ways to make this nicer from the perspective of someone calling the `.insert()` or `.insert_all()` methods against a table that has geometry columns.

  One option would be for the code to introspect the table (if it exists) before running the insert, looking for any geometry columns.

  This introspection isn't easy! The table schema just gives you `"name_of_column" point` or similar - to figure out the SRID and suchlike you need to consult the `geometry_columns` table, I think - which throws a 500 error on https://calands.datasettes.com/calands/geometry_columns for some reason. Also does the shape of that table change between SpatiaLite versions?

  Assuming we can introspect the table, what would we do with that information? We could add code that detects if the user attempted to pass GeoJSON objects and automatically inserts a `GeomFromGeoJSON()` function call - but detecting GeoJSON is a bit weird, and GeoJSON also isn't necessarily the nicest format for populating e.g. latitude/longitude points.

  Maybe we just support the simplest possible case: a tuple of floats, which we assume is `latitude, longitude` (or should we expect `longitude, latitude`, the eternal debate?) - if those are used against a geometry table (especially a point table) we assume they are coordinates that need to be converted using `GeomFromText('POINT(...`.

  Not crazy about either of these ideas. Is there something better?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1576#issuecomment-1030525218
issue_url: https://api.github.com/repos/simonw/datasette/issues/1576
node_id: IC_kwDOBm6k_c49bJEi
created_at: 2022-02-05T04:45:11Z
updated_at: 2022-02-05T04:45:11Z
author_association: OWNER
body:
  Got a prototype working with `contextvars` - it identified two parallel executing queries using the patch from above:

  ![CleanShot 2022-02-04 at 20 41 50@2x](https://user-images.githubusercontent.com/9599/152628949-cf766b13-13cf-4831-b48d-2f23cadb6a05.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1030629879
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/398
node_id: IC_kwDOCGYnMM49bin3
created_at: 2022-02-05T13:57:33Z
updated_at: 2022-02-05T19:49:38Z
author_association: CONTRIBUTOR
body:
  I'm mostly using [geojson-to-sqlite](https://github.com/simonw/geojson-to-sqlite) at the moment. Even with shapefiles, I'm usually converting to GeoJSON and projecting to EPSG:4326 (with [ogr2ogr](https://gdal.org/programs/ogr2ogr.html)) first. 

  I think an open question here is how much you want to leave to external libraries and how much you want here. My thinking has been that adding Spatialite helpers here would make external stuff easier, but it would be nice to have some standard way to insert geometries.

  I'm in the middle of adding GeoJSON and Spatialite support to [geocode-sqlite](https://github.com/eyeseast/geocode-sqlite), and that will probably use WKT. Since that's all points, I think I can just make the string inline. But for polygons, I'd generally use Shapely, which probably isn't a dependency you want to add to sqlite-utils.

  I've also been trying to get some of the approaches [here](https://www.gaia-gis.it/fossil/libspatialite/wiki?name=Supporting+GeoJSON) to work, but haven't had any success so far.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030712129
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/399
node_id: IC_kwDOCGYnMM49b2tB
created_at: 2022-02-05T23:08:45Z
updated_at: 2022-02-05T23:08:45Z
author_association: OWNER
body:
  Useful thoughts on Twitter regarding making coordinate pairs easy and more complex shapes possible:

  https://twitter.com/dbreunig/status/1490099303888547843
  > That is exactly where I was going: two modes.
  >
  > 1. Heuristics and assumptions to get coordinates as a pair (in tuple) or as columns (look for lat, lon, latitude, longitude, etc).
  > 2. GIS mode with projections, polys, etc
  >
  > Make it easy for people with csvs of coordinates. If you're using Geojson or shp files, you have to specify.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/397#issuecomment-1030726991
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/397
node_id: IC_kwDOCGYnMM49b6VP
created_at: 2022-02-06T01:13:58Z
updated_at: 2022-02-06T01:13:58Z
author_association: OWNER
body:
  This is a good idea. We already have that parameter for the `table.create_index()` method: https://sqlite-utils.datasette.io/en/stable/reference.html#sqlite_utils.db.Table.create_index

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/397#issuecomment-1030730108
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/397
node_id: IC_kwDOCGYnMM49b7F8
created_at: 2022-02-06T01:30:46Z
updated_at: 2022-02-06T01:30:46Z
author_association: OWNER
body:
  Updated documentation is here: https://sqlite-utils.datasette.io/en/latest/python-api.html#explicitly-creating-a-table

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/400#issuecomment-1030730748
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/400
node_id: IC_kwDOCGYnMM49b7P8
created_at: 2022-02-06T01:34:46Z
updated_at: 2022-02-06T01:34:46Z
author_association: OWNER
body:
  Actually this is not needed - there is already an option that does this, it's just called `--ignore` rather than `--if-not-exists`.

  The lack of consistency here is a little annoying, but not annoying enough to justify making a backwards incompatible change.

  ```
  % sqlite-utils create-table --help
  Usage: sqlite-utils create-table [OPTIONS] PATH TABLE COLUMNS...

    Add a table with the specified columns. Columns should be specified using
    name, type pairs, for example:

        sqlite-utils create-table my.db people \
            id integer \
            name text \
            height float \
            photo blob --pk id

  Options:
    --pk TEXT                 Column to use as primary key
    --not-null TEXT           Columns that should be created as NOT NULL
    --default <TEXT TEXT>...  Default value that should be set for a column
    --fk <TEXT TEXT TEXT>...  Column, other table, other column to set as a
                              foreign key
    --ignore                  If table already exists, do nothing
    --replace                 If table already exists, replace it
    --load-extension TEXT     SQLite extensions to load
    -h, --help                Show this message and exit.
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1030732093
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/398
node_id: IC_kwDOCGYnMM49b7k9
created_at: 2022-02-06T01:41:37Z
updated_at: 2022-02-06T01:41:37Z
author_association: OWNER
body:
  Yeah I'd like to avoid adding any geo-dependencies to `sqlite-utils` if I can avoid it. I'm fine using stuff that's going to be available in SpatiaLite itself (provided it's available as a SQLite module) since then I don't need to add any extra Python dependencies.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030732909
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/399
node_id: IC_kwDOCGYnMM49b7xt
created_at: 2022-02-06T01:47:06Z
updated_at: 2022-02-06T01:47:06Z
author_association: OWNER
body:
  Here's an idea for an API design:
  ```python
  geojson_geometry = {} # ... GeoJSON goes here
  db["places"].insert(
      {"name": "Wales", "geometry": geojson_geometry},
      geojson="geometry"
  )
  ```
  That `geojson=` parameter takes either a single column name or an iterable of column names. Any column in that list is expected to be a compatible `geometry` and the correct conversion functions will be applied.

  That solves for GeoJSON, but it's a bit ugly. Should I add `wkt=` and maybe even `kml=` and `gml=` and so-on too? Definitely not, that's way too many ugly and inscrutable new parameters.

  More importantly: if I want to support the following how would I do it?

  ```python
  db["places"].insert(
      {"name": "London", "point": (51.509865, -0.118092)}
  )
  ```
  Here I want to provide a `(latitude, longitude)` pair and have it inserted correctly into a `point` column.

  Could do this, but again it's messy:
  ```python
  db["places"].insert(
      {"name": "London", "point": (51.509865, -0.118092)},
      point="point"
  )
  ```
  And again, what about those `(longitude, latitude)` people?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/401#issuecomment-1030734937
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/401
node_id: IC_kwDOCGYnMM49b8RZ
created_at: 2022-02-06T02:02:24Z
updated_at: 2022-02-06T02:02:24Z
author_association: OWNER
body:
  The example also doesn't work right now - the code that fetches from Who's On First gets a 403 forbidden error.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/401#issuecomment-1030735372
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/401
node_id: IC_kwDOCGYnMM49b8YM
created_at: 2022-02-06T02:05:03Z
updated_at: 2022-02-06T02:05:03Z
author_association: OWNER
body:
  Improved version: https://sqlite-utils.datasette.io/en/latest/python-api.html#converting-column-values-using-sql-functions

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030735774
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/399
node_id: IC_kwDOCGYnMM49b8ee
created_at: 2022-02-06T02:08:19Z
updated_at: 2022-02-06T02:08:59Z
author_association: OWNER
body:
  Maybe I should leave this entirely up to documented patterns in the `conversions={}` dictionary?

  But even that's not ideal for the co-ordinate case. Consider the following:

  ```python
  db["places"].insert(
      {"name": "London", "point": (51.509865, -0.118092)},
      conversions={"point": "GeomFromText(?, 4326)"},
  )
  ```
  The challenge here is that the SpatiaLite function `GeomFromText()` expects a WKT string, which looks like this:

      POINT(-0.118092 51.509865)

  The existing `conversions=` mechanism doesn't support applying Python code to convert the `(lat, lon)` tuple to that value.

  It doesn't even support passing a Python tuple as a `?` parameter - so I don't think I could come up with a SQL string that would do the right thing here either.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030736047
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/399
node_id: IC_kwDOCGYnMM49b8iv
created_at: 2022-02-06T02:10:18Z
updated_at: 2022-02-06T02:10:18Z
author_association: OWNER
body:
  So maybe back to that earlier idea where the code introspects the table, figures out that `"point"` is a geometry table of type POINT, then applies the necessary conversions to the raw Python data?

  That feels overly-complicated to me, especially since nothing else in the `.insert()` method currently relies on table introspection.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030736589
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/399
node_id: IC_kwDOCGYnMM49b8rN
created_at: 2022-02-06T02:14:52Z
updated_at: 2022-02-06T02:14:52Z
author_association: OWNER
body:
  Another idea: introduce a helper function transform pattern, something a bit like this:

  ```python

  transformer = make_transformer({
    "point": lambda pair: "POINT({} {})".format(pair[1], pair[0])
  })

  db["places"].insert_all(
      transformer([{"name": "London", "point": (51.509865, -0.118092)}])
      conversions={"point": "GeomFromText(?, 4326)"},
  )
  ```
  The `make_transformer(...)` function builds an object that can work as a wrapping iterator, applying those transform functions to everything in the sequence that it wraps.

  So the above code would handle converting `(lat, lon)` to `POINT(lon lat)` - then the `conversions=` applies `GeomFromText`.

  Naming is a challenge here: `.transform()` and `.convert()` and `conversions=` all have existing meanings within the `sqlite-utils` Python library.

  It's also a bit of a messy way of solving this. It's not exactly a smooth API for inserting a bunch of lat/lon coordinate pairs!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030736848
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/399
node_id: IC_kwDOCGYnMM49b8vQ
created_at: 2022-02-06T02:17:35Z
updated_at: 2022-02-06T02:17:35Z
author_association: OWNER
body:
  Note that GeoJSON itself uses `(longitude, latitude)` so I should probably stick to that order here too. https://datatracker.ietf.org/doc/html/rfc7946#section-3.1.1

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030738023
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/399
node_id: IC_kwDOCGYnMM49b9Bn
created_at: 2022-02-06T02:28:05Z
updated_at: 2022-02-06T02:29:24Z
author_association: OWNER
body:
  Here's the definitive guide to `latitude, longitude` v.s. `longitude, latitude`: https://macwright.com/lonlat/

  > Which is right?
  >
  > Neither. This is an opinion with no right answer. Geographical tradition favors lat, lon. Math and software prefer lon, lat.

  I asked on Twitter here: https://twitter.com/simonw/status/1490148001569906688

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030739566
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/399
node_id: IC_kwDOCGYnMM49b9Zu
created_at: 2022-02-06T02:45:25Z
updated_at: 2022-02-06T02:50:27Z
author_association: OWNER
body:
  Another idea - my favourite option so far:
  ```python
  from sqlite_utils.utils import LongitudeLatitude

  db["places"].insert(
      {
          "name": "London",
          "point": (-0.118092, 51.509865)
      },
      conversions={"point": LongitudeLatitude},
  )
  ```
  Here `LongitudeLatitude` is a magical value which does TWO things: it sets up the `GeomFromText(?, 4326)` SQL function, and it handles converting the `(51.509865, -0.118092)` tuple into a `POINT({} {})` string.

  This would involve a change to the `conversions=` contract - where it usually expects a SQL string fragment, but it can also take an object which combines that SQL string fragment with a Python conversion function.

  Best of all... this resolves the `lat, lon` v.s. `lon, lat` dilemma because you can use `from sqlite_utils.utils import LongitudeLatitude` OR `from sqlite_utils.utils import LatitudeLongitude` depending on which you prefer!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030740570
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/399
node_id: IC_kwDOCGYnMM49b9pa
created_at: 2022-02-06T02:56:17Z
updated_at: 2022-02-06T02:57:00Z
author_association: OWNER
body:
  Thinking about types. The type of the `conversions` parameter right now is a bit lazy:

  ```python
  conversions: Optional[dict] = None,
  ```
  That becomes:
  ```python
  Optional[Dict[str, Union[str, Conversion]]]
  ```
  Where `Conversion` is an abstract base class which expects implementations to have a `.sql() -> str` and a `.convert(value) -> str` method.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030740653
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/399
node_id: IC_kwDOCGYnMM49b9qt
created_at: 2022-02-06T02:57:17Z
updated_at: 2022-02-06T02:57:17Z
author_association: CONTRIBUTOR
body:
  I like the idea of having stock conversions you could import. I'd actually move them to a dedicated module (call it `sqlite_utils.conversions` or something), because it's different from other utilities. Maybe they even take configuration, or they're composable.

  ```python
  from sqlite_utils.conversions import LongitudeLatitude

  db["places"].insert(
      {
          "name": "London",
          "lng": -0.118092,
          "lat": 51.509865,
      },
      conversions={"point": LongitudeLatitude("lng", "lat")},
  )
  ```

  I would definitely use that for every CSV I get with lat/lng columns where I actually need GeoJSON.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030740771
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/399
node_id: IC_kwDOCGYnMM49b9sj
created_at: 2022-02-06T02:58:29Z
updated_at: 2022-02-06T02:58:29Z
author_association: OWNER
body:
  That example you have there is really neat - I like the idea that they can also be used to populate completely new columns that are derived from the other column inputs.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030740826
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/399
node_id: IC_kwDOCGYnMM49b9ta
created_at: 2022-02-06T02:59:10Z
updated_at: 2022-02-06T02:59:10Z
author_association: CONTRIBUTOR
body:
  All this said, I don't think it's unreasonable to point people to dedicated tools like `geojson-to-sqlite`. If I'm dealing with a bunch of GeoJSON or Shapefiles, I need to something to read those anyway (or I need to figure out virtual tables). But something like this might make it easier to build those libraries, or standardize the underlying parts.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030740846
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/399
node_id: IC_kwDOCGYnMM49b9tu
created_at: 2022-02-06T02:59:21Z
updated_at: 2022-02-06T02:59:21Z
author_association: OWNER
body:
  I wonder if there are any interesting non-geospatial canned conversions that it would be worth including?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030740963
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/399
node_id: IC_kwDOCGYnMM49b9vj
created_at: 2022-02-06T03:00:33Z
updated_at: 2022-02-06T03:00:33Z
author_association: OWNER
body:
  Yeah, having this be a general purpose mechanism which has a few canned examples for handling geospatial stuff is a lot neater than having a mechanism for this that's exclusive to SpatiaLite.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030741289
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/399
node_id: IC_kwDOCGYnMM49b90p
created_at: 2022-02-06T03:03:43Z
updated_at: 2022-02-06T03:03:43Z
author_association: CONTRIBUTOR
body:
  > I wonder if there are any interesting non-geospatial canned conversions that it would be worth including?

  Off the top of my head:

  - Un-nesting JSON objects into columns
  - Splitting arrays
  - Normalizing dates and times
  - URL munging with `urlparse`
  - Converting strings to numbers

  Some of this is easy enough with SQL functions, some is easier in Python. Maybe that's where having pre-built classes gets really handy, because it saves you from thinking about which way it's implemented.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/957#issuecomment-1030761625
issue_url: https://api.github.com/repos/simonw/datasette/issues/957
node_id: IC_kwDOBm6k_c49cCyZ
created_at: 2022-02-06T06:30:32Z
updated_at: 2022-02-06T06:31:44Z
author_association: OWNER
body:
  I'm just going with:

  ```python
  from datasette import Response
  from datasette import Forbidden
  from datasette import NotFound
  from datasette import hookimpl
  from datasette import actor_matches_allow
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1176#issuecomment-1030762279
issue_url: https://api.github.com/repos/simonw/datasette/issues/1176
node_id: IC_kwDOBm6k_c49cC8n
created_at: 2022-02-06T06:38:08Z
updated_at: 2022-02-06T06:41:37Z
author_association: OWNER
body:
  Might do this using Sphinx auto-generated function and class documentation hooks, as seen here in `sqlite-utils`: https://sqlite-utils.datasette.io/en/stable/python-api.html#spatialite-helpers

  This would encourage me to add really good docstrings.

  ```
  .. _python_api_gis_find_spatialite:

  Finding SpatiaLite
  ------------------

  .. autofunction:: sqlite_utils.utils.find_spatialite
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030807433
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/399
node_id: IC_kwDOCGYnMM49cN-J
created_at: 2022-02-06T10:54:09Z
updated_at: 2022-02-06T10:54:09Z
author_association: NONE
body:
  > Interesting that some accept an SRID and others do not - presumably GeomFromGeoJSON() always uses SRID=4326?

  The ewtk/ewkb ones don't accept an SRID is because ewkt encodes the SRID in the string, so you would do this with a wkt string:

  `GeomFromText('POINT(529090 179645)', 27700)`

  but for ewkt it would be

  `GeomFromEWKT('SRID=27700;POINT(529090 179645)')`

  The specs for KML and GeoJSON specify a Coordinate Reference System for the format

  - https://datatracker.ietf.org/doc/html/rfc7946#section-4
  - https://docs.opengeospatial.org/is/12-007r2/12-007r2.html#1274

  GML can specify the SRID in the XML at feature level e.g:

  ```
  <gml:Point srsName="EPSG:27700">
    <gml:coordinates>529090, 179645</gml:coordinates>
  </gml:Point>
  ```

  There's a few more obscure formats in there, but broadly I think it is safe to assume an SRID param exists on the function for cases where the SRID is not implied by or specified in the input format.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030871591
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/399
node_id: IC_kwDOCGYnMM49cdon
created_at: 2022-02-06T16:57:22Z
updated_at: 2022-02-06T16:57:22Z
author_association: OWNER
body:
  I wonder if I could implement the above such that this *also* works:
  ```python
  db["places"].insert(
      {
          "name": "London",
          "point": LongitudeLatitude(-0.118092, 51.509865)
      }
  )
  ```
  This feels like a very natural way to work with single inserts.

  The challenge is writing the code inside `.insert_all()` such that it can handle these special objects in the input column values in addition to them being passed in `conversions=`.

  I'm feeling very good about this direction in general though, it feels like it takes the existing but not particularly elegant `conversions=` mechanism and upgrades it to be far more useful, while maintaining backwards compatibility.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1030901189
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/402
node_id: IC_kwDOCGYnMM49ck3F
created_at: 2022-02-06T19:48:36Z
updated_at: 2022-02-06T19:48:52Z
author_association: OWNER
body:
  From [that thread](https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030739566), two extra ideas which it may be possible to support in a single implementation:

  ```python
  from sqlite_utils.conversions import LongitudeLatitude

  db["places"].insert(
      {
          "name": "London",
          "lng": -0.118092,
          "lat": 51.509865,
      },
      conversions={"point": LongitudeLatitude("lng", "lat")},
  )
  ```
  And
  ```python
  db["places"].insert(
      {
          "name": "London",
          "point": LongitudeLatitude(-0.118092, 51.509865)
      }
  )
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1030901853
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/402
node_id: IC_kwDOCGYnMM49clBd
created_at: 2022-02-06T19:52:10Z
updated_at: 2022-02-06T19:52:10Z
author_association: OWNER
body:
  So the key idea here is to introduce a new abstract base class, `Conversion`, which has the following abilities:

  - Can wrap one or more Python values (if called using the constructor) such that the `.insert_all()` method knows how to transform those into a format that can be included in an insert - something like `GeomFromText(?, 4326)` with input `POINT(-0.118092 51.509865)`
  - Can be passed to `conversions={"point": LongitudeLatitude}` in a way that then knows to apply that conversion to every value in the `"point"` key of the data being inserted.
  - Maybe also extend `conversions=` to allow the definition of additional keys that use as input other rows? That's the `conversions={"point": LongitudeLatitude("lng", "lat")}` example above - it may not be possible to get this working with the rest of the design though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1030902102
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/402
node_id: IC_kwDOCGYnMM49clFW
created_at: 2022-02-06T19:53:34Z
updated_at: 2022-02-08T07:40:34Z
author_association: OWNER
body:
  I like the idea that the contract for `Conversion` (or rather for its subclasses) is that it can wrap a Python value and then return both the SQL fragment - e.g. `GeomFromText(?, 4326)` - and the values that should be used as the SQL parameters.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030902158
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/399
node_id: IC_kwDOCGYnMM49clGO
created_at: 2022-02-06T19:53:54Z
updated_at: 2022-02-06T19:53:54Z
author_association: OWNER
body:
  Moving the design of this new `Conversion` subclass mechanism to:
  - https://github.com/simonw/sqlite-utils/issues/402

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1030904948
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/402
node_id: IC_kwDOCGYnMM49clx0
created_at: 2022-02-06T20:09:42Z
updated_at: 2022-02-08T07:40:44Z
author_association: OWNER
body:
  I think this is the code that needs to become aware of this system: https://github.com/simonw/sqlite-utils/blob/fea8c9bcc509bcae75e99ae8870f520103b9aa58/sqlite_utils/db.py#L2453-L2469

  There's an earlier branch that runs for upserts which needs to be modified too: https://github.com/simonw/sqlite-utils/blob/fea8c9bcc509bcae75e99ae8870f520103b9aa58/sqlite_utils/db.py#L2417-L2440

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/932#issuecomment-1030940407
issue_url: https://api.github.com/repos/simonw/datasette/issues/932
node_id: IC_kwDOBm6k_c49cub3
created_at: 2022-02-06T23:31:22Z
updated_at: 2022-02-06T23:31:22Z
author_association: OWNER
body:
  Great argument for doing this from a conversation on Twitter about documentation-driven development:

  > Long ago, when the majority of commercial programs were desktop apps, I've read a very wise advice: The user manual should be written first, before even a single line if code.

  https://twitter.com/b11c/status/1490466703175823362

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1176#issuecomment-1031122800
issue_url: https://api.github.com/repos/simonw/datasette/issues/1176
node_id: IC_kwDOBm6k_c49da9w
created_at: 2022-02-07T06:34:21Z
updated_at: 2022-02-07T06:34:21Z
author_association: OWNER
body:
  New section is here: https://docs.datasette.io/en/latest/internals.html#the-datasette-utils-module

  But it's not correctly displaying the new autodoc stuff:

  <img width="688" alt="image" src="https://user-images.githubusercontent.com/9599/152736915-e6470e0a-05e2-4612-a2ff-ebd4f51b65cb.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1176#issuecomment-1031123719
issue_url: https://api.github.com/repos/simonw/datasette/issues/1176
node_id: IC_kwDOBm6k_c49dbMH
created_at: 2022-02-07T06:36:32Z
updated_at: 2022-02-07T06:36:32Z
author_association: OWNER
body:
  https://github.com/simonw/sqlite-utils/blob/main/.readthedocs.yaml looks like this (it works correctly):

  ```yaml
  version: 2

  sphinx:
    configuration: docs/conf.py

  python:
    version: "3.8"
    install:
    - method: pip
      path: .
      extra_requirements:
      - docs
  ```
  Compare to the current Datasette one here: https://github.com/simonw/datasette/blob/d9b508ffaa91f9f1840b366f5d282712d445f16b/.readthedocs.yaml#L1-L13

  Looks like I need this bit:
  ```python
  python:
    version: "3.8"
    install:
    - method: pip
      path: .
      extra_requirements:
      - docs
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1176#issuecomment-1031125347
issue_url: https://api.github.com/repos/simonw/datasette/issues/1176
node_id: IC_kwDOBm6k_c49dblj
created_at: 2022-02-07T06:40:16Z
updated_at: 2022-02-07T06:40:16Z
author_association: OWNER
body:
  Read The Docs error:

  > Problem in your project's configuration. Invalid "python.version": .readthedocs.yaml: Invalid configuration option: python.version. Make sure the key name is correct.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1176#issuecomment-1031126547
issue_url: https://api.github.com/repos/simonw/datasette/issues/1176
node_id: IC_kwDOBm6k_c49db4T
created_at: 2022-02-07T06:42:58Z
updated_at: 2022-02-07T06:42:58Z
author_association: OWNER
body:
  That fixed it: https://docs.datasette.io/en/latest/internals.html#parse-metadata-content

  <img width="695" alt="image" src="https://user-images.githubusercontent.com/9599/152737872-595e9a14-1a0d-47ff-9509-63ee765c0c5d.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1176#issuecomment-1031126801
issue_url: https://api.github.com/repos/simonw/datasette/issues/1176
node_id: IC_kwDOBm6k_c49db8R
created_at: 2022-02-07T06:43:31Z
updated_at: 2022-02-07T06:43:31Z
author_association: OWNER
body:
  Here's the new test: https://github.com/simonw/datasette/blob/03305ea183b1534bc4cef3a721fe5f3700273b84/tests/test_docs.py#L91-L104

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1631#issuecomment-1031463789
issue_url: https://api.github.com/repos/simonw/datasette/issues/1631
node_id: IC_kwDOBm6k_c49euNt
created_at: 2022-02-07T13:21:48Z
updated_at: 2022-02-07T13:21:48Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1631?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1631](https://codecov.io/gh/simonw/datasette/pull/1631?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (62eed84) into [main](https://codecov.io/gh/simonw/datasette/commit/03305ea183b1534bc4cef3a721fe5f3700273b84?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (03305ea) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1631/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1631?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1631   +/-   ##
  =======================================
    Coverage   92.19%   92.19%           
  =======================================
    Files          34       34           
    Lines        4546     4546           
  =======================================
    Hits         4191     4191           
    Misses        355      355           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1631?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1631?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [03305ea...62eed84](https://codecov.io/gh/simonw/datasette/pull/1631?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1031779460
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/402
node_id: IC_kwDOCGYnMM49f7SE
created_at: 2022-02-07T18:24:56Z
updated_at: 2022-02-07T18:24:56Z
author_association: CONTRIBUTOR
body:
  I wonder if there's any overlap with the goals here and the `sqlite3` module's concept of adapters and converters: https://docs.python.org/3/library/sqlite3.html#sqlite-and-python-types

  I'm not sure that's _exactly_ what we're talking about here, but it might be a parallel with some useful ideas to borrow.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1031787865
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/402
node_id: IC_kwDOCGYnMM49f9VZ
created_at: 2022-02-07T18:33:27Z
updated_at: 2022-02-07T18:33:27Z
author_association: OWNER
body:
  Hah, that's interesting - I've never used that mechanism before so it wasn't something that came to mind.

  They seem to be using a pretty surprising trick there that takes advantage of SQLite allowing you to define a column "type" using a made-up type name, which you can then introspect later.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1031791783
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/402
node_id: IC_kwDOCGYnMM49f-Sn
created_at: 2022-02-07T18:37:40Z
updated_at: 2022-02-07T18:37:40Z
author_association: CONTRIBUTOR
body:
  I've never used it either, but it's interesting, right? Feel like I should try it for something. 

  I'm trying to get my head around how this conversions feature might work, because I really like the idea of it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1632#issuecomment-1032032686
issue_url: https://api.github.com/repos/simonw/datasette/issues/1632
node_id: IC_kwDOBm6k_c49g5Gu
created_at: 2022-02-07T23:16:10Z
updated_at: 2022-02-07T23:16:10Z
author_association: OWNER
body:
  I found this bug while trying to get the following to work:

      datasette /data/one.db /data/two.db /data/*.db --create

  I want this to create any missing database files on startup out of that literal list of `one.db` and `two.db` and to also open any other `*.db` files in that folder - needed for `datasette-publish-fly` in https://github.com/simonw/datasette-publish-fly/pull/12#issuecomment-1032029874

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1632#issuecomment-1032036525
issue_url: https://api.github.com/repos/simonw/datasette/issues/1632
node_id: IC_kwDOBm6k_c49g6Ct
created_at: 2022-02-07T23:19:59Z
updated_at: 2022-02-07T23:19:59Z
author_association: OWNER
body:
  I'm going to fix this in the CLI code itself, rather than fixing it in the `Datasette` constructor. That way if someone has a truly weird reason to want this behaviour they can construct Datasette directly.

  https://github.com/simonw/datasette/blob/03305ea183b1534bc4cef3a721fe5f3700273b84/datasette/cli.py#L535-L550

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1632#issuecomment-1032037391
issue_url: https://api.github.com/repos/simonw/datasette/issues/1632
node_id: IC_kwDOBm6k_c49g6QP
created_at: 2022-02-07T23:21:07Z
updated_at: 2022-02-07T23:21:07Z
author_association: OWNER
body:
  For the record, here's the code that picks the `one_2` name if that stem is already used as a database name: https://github.com/simonw/datasette/blob/03305ea183b1534bc4cef3a721fe5f3700273b84/datasette/app.py#L401-L417

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1632#issuecomment-1032050489
issue_url: https://api.github.com/repos/simonw/datasette/issues/1632
node_id: IC_kwDOBm6k_c49g9c5
created_at: 2022-02-07T23:39:11Z
updated_at: 2022-02-07T23:42:08Z
author_association: OWNER
body:
  That implementation broke on Python 3.6 - which is still a supported Python version for the 0.60.x branch - `test_homepage` failed.

  ```
  >       assert (
              "2 rows in 1 table, 5 rows in 4 hidden tables, 1 view" == counts_p.text.strip()
          )
  E       AssertionError: assert '2 rows in 1 ...ables, 1 view' == '1 table, 4 h...ables, 1 view'
  E         - 1 table, 4 hidden tables, 1 view
  E         + 2 rows in 1 table, 5 rows in 4 hidden tables, 1 view
  E         ? ++++++++++         ++++++++++
  ```
  That's because this idiom isn't guaranteed to preserve order in versions earlier than Python 3.7:

  https://github.com/simonw/datasette/blob/fa5fc327adbbf70656ac533912f3fc0526a3873d/datasette/cli.py#L552-L553

  I could say that `0.60.2` is the first version to require Python 3.7 - but that feels a little surprising.

  I'm going to use a different idiom for order-preserving de-duplication from [this StackOverflow](https://stackoverflow.com/questions/480214/how-do-you-remove-duplicates-from-a-list-whilst-preserving-order) instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/403#issuecomment-1032126353
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/403
node_id: IC_kwDOCGYnMM49hP-R
created_at: 2022-02-08T01:45:15Z
updated_at: 2022-02-08T01:45:31Z
author_association: CONTRIBUTOR
body:
  you can hack something like this to achieve this result:

  `sqlite-utils convert my_database my_table rowid "{'id': value}" --multi`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1032294365
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/402
node_id: IC_kwDOCGYnMM49h4_d
created_at: 2022-02-08T07:32:09Z
updated_at: 2022-02-08T07:34:41Z
author_association: OWNER
body:
  I have an idea for how that third option could work - the one that creates a new column using values from the existing ones:
  ```python
  db["places"].insert(
      {
          "name": "London",
          "lng": -0.118092,
          "lat": 51.509865,
      },
      conversions={"point": LongitudeLatitude("lng", "lat")},
  )
  ```
  How about specifying that the values in that `conversion=` dictionary can be:

  - A SQL string fragment (as currently implemented)
  - A subclass of `Conversion` as described above
  - Or... a callable function that takes the row as an argument and returns either a `Conversion` subclass instance or a literal value to be jnserted into the database (a string, int or float)

  Then you could do this:

  ```python
  db["places"].insert(
      {
          "name": "London",
          "lng": -0.118092,
          "lat": 51.509865,
      },
      conversions={
          "point": lambda row: LongitudeLatitude(
              row["lng"], row["lat"]
          )
      }
  )
  ```
  Something I really like about this is that it expands the abilities of `conversions=` beyond the slightly obscure need to customize the SQL fragment into something that can solve other data insertion cleanup problems too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1032296717
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/402
node_id: IC_kwDOCGYnMM49h5kN
created_at: 2022-02-08T07:35:46Z
updated_at: 2022-02-08T07:35:46Z
author_association: OWNER
body:
  I'm going to write the documentation for this first, before the implementation, so I can see if it explains cleanly enough that the design appears to be sound.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1032732242
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/402
node_id: IC_kwDOCGYnMM49jj5S
created_at: 2022-02-08T15:26:59Z
updated_at: 2022-02-08T15:26:59Z
author_association: CONTRIBUTOR
body:
  What if you did something like this:

  ```python

  class Conversion:
      def __init__(self, *args, **kwargs):
          "Put whatever settings you need here"

      def python(self, row, column, value): # not sure on args here
          "Python step to transform value"
          return value

      def sql(self, row, column, value):
          "Return the actual sql that goes in the insert/update step, and maybe params"
          # value is the return of self.python()
          return value, []
  ```

  This way, you're always passing an instance, which has methods that do the conversion. (Or you're passing a SQL string, as you would now.) The `__init__` could take column names, or SRID, or whatever other setup state you need per row, but the row is getting processed with the `python` and `sql` methods (or whatever you want to call them). This is pretty rough, so do what you will with names and args and such.

  You'd then use it like this:

  ```python
  # subclass might be unneeded here, if methods are present
  class LngLatConversion(Conversion):
      def __init__(self, x="longitude", y="latitude"):
          self.x = x
          self.y = y

      def python(self, row, column, value):
          x = row[self.x]
          y = row[self.y]
          return x, y

      def sql(self, row, column, value):
          # value is now a tuple, returned above
          s = "GeomFromText(POINT(? ?))"
          return s, value

  table.insert_all(rows, conversions={"point": LngLatConversion("lng", "lat"))}
  ```

  I haven't thought through all the implementation details here, and it'll probably break in ways I haven't foreseen, but wanted to get this idea out of my head. Hope it helps.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/403#issuecomment-1032976720
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/403
node_id: IC_kwDOCGYnMM49kflQ
created_at: 2022-02-08T19:23:05Z
updated_at: 2022-02-08T19:23:27Z
author_association: OWNER
body:
  This is already possible using `sqlite-utils transform` like so:

  ```
  % echo '[{"name": "Barry"}, {"name": "Sandra"}]' | sqlite-utils insert rowid.db records -
  % sqlite-utils schema rowid.db                                                           
  CREATE TABLE [records] (
     [name] TEXT
  );
  % sqlite-utils rows rowid.db records
  [{"name": "Barry"},
   {"name": "Sandra"}]
  % sqlite-utils transform rowid.db records --pk id
  % sqlite-utils rows rowid.db records
  [{"id": 1, "name": "Barry"},
   {"id": 2, "name": "Sandra"}]
  % sqlite-utils schema rowid.db
  CREATE TABLE "records" (
     [id] INTEGER PRIMARY KEY,
     [name] TEXT
  );
  % echo '[{"name": "Barry 2"}, {"name": "Sandra 2"}]' | sqlite-utils insert rowid.db records -
  % sqlite-utils rows rowid.db records                                                         
  [{"id": 1, "name": "Barry"},
   {"id": 2, "name": "Sandra"},
   {"id": 3, "name": "Barry 2"},
   {"id": 4, "name": "Sandra 2"}]
  ```
  It's not covered in the documentation though: https://sqlite-utils.datasette.io/en/3.23/cli.html#transforming-tables

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/403#issuecomment-1032987901
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/403
node_id: IC_kwDOCGYnMM49kiT9
created_at: 2022-02-08T19:36:06Z
updated_at: 2022-02-08T19:36:06Z
author_association: OWNER
body:
  New documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#adding-a-primary-key-to-a-rowid-table

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1033366312
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/402
node_id: IC_kwDOCGYnMM49l-so
created_at: 2022-02-09T05:28:11Z
updated_at: 2022-02-09T07:28:48Z
author_association: OWNER
body:
  My hunch is that the case where you want to consider input from more than one column will actually be pretty rare - the only case I can think of where I would want to do that is for latitude/longitude columns - everything else that I'd want to use it for (which admittedly is still mostly SpatiaLite stuff) works against a single value.

  The reason I'm leaning towards using the constructor for the values is that I really like the look of this variant for common conversions:

  ```python
  db["places"].insert(
      {
          "name": "London",
          "boundary": GeometryFromGeoJSON({...})
      }
  )
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1607#issuecomment-1033403664
issue_url: https://api.github.com/repos/simonw/datasette/issues/1607
node_id: IC_kwDOBm6k_c49mH0Q
created_at: 2022-02-09T06:42:02Z
updated_at: 2022-02-09T06:42:02Z
author_association: OWNER
body:
  Deployed a new build of https://github.com/simonw/calands-datasette/actions/workflows/build-and-deploy.yml for a live demo:

  https://calands.datasettes.com/-/versions

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/404#issuecomment-1033407778
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/404
node_id: IC_kwDOCGYnMM49mI0i
created_at: 2022-02-09T06:50:26Z
updated_at: 2022-02-09T06:50:26Z
author_association: OWNER
body:
  I'll use this:

  ```
    sqlite-utils insert plants.db plants plants.csv --csv --convert '
    return {
        "name": row["name"].upper(),
        "latitude": float(row["latitude"]),
        "longitude": float(row["longitude"]),
    }'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/405#issuecomment-1033424454
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/405
node_id: IC_kwDOCGYnMM49mM5G
created_at: 2022-02-09T07:18:25Z
updated_at: 2022-02-09T07:18:25Z
author_association: OWNER
body:
  Writing tests against this is always a tiny bit fiddly since the created databases persist across the lifetime of the test run. Using randomly generated names helps.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/405#issuecomment-1033425512
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/405
node_id: IC_kwDOCGYnMM49mNJo
created_at: 2022-02-09T07:20:11Z
updated_at: 2022-02-09T07:20:11Z
author_association: OWNER
body:
  Datasette's implementation: https://github.com/simonw/datasette/blob/458f03ad3a454d271f47a643f4530bd8b60ddb76/datasette/database.py#L73-L79

  ```python
          if self.memory_name:
              uri = "file:{}?mode=memory&cache=shared".format(self.memory_name)
              conn = sqlite3.connect(
                  uri,
                  uri=True,
                  check_same_thread=False,
              )
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1033428967
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/402
node_id: IC_kwDOCGYnMM49mN_n
created_at: 2022-02-09T07:25:44Z
updated_at: 2022-02-09T07:28:11Z
author_association: OWNER
body:
  The CLI version of this could perhaps look like this:

      sqlite-utils insert a.db places places.json \
        --conversion boundary GeometryGeoJSON

  This will treat the boundary key as GeoJSON. It's equivalent to passing `conversions={"boundary": geometryGeoJSON}`

  The combined latitude/longitude case here can be handled by combining this with the existing `--convert` mechanism.
   
  Any `Conversion` subclass will be available to the  CLI in this way.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/203#issuecomment-1033641009
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/203
node_id: IC_kwDOCGYnMM49nBwx
created_at: 2022-02-09T11:06:18Z
updated_at: 2022-02-09T11:06:18Z
author_association: NONE
body:
  Is there any progress elsewhere on the handling of compound / composite foreign keys, or is this PR still effectively open?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/236#issuecomment-1033772902
issue_url: https://api.github.com/repos/simonw/datasette/issues/236
node_id: IC_kwDOBm6k_c49nh9m
created_at: 2022-02-09T13:40:52Z
updated_at: 2022-02-09T13:40:52Z
author_association: NONE
body:
  Hi @simonw, 

  I've received some inquiries over the last year or so about Datasette and how it might be supported by [Mangum](https://github.com/jordaneremieff/mangum). I maintain Mangum which is, as far as I know, the only project that provides support for ASGI applications in AWS Lambda.

  If there is anything that I can help with here, please let me know because I think what Datasette provides to the community (even beyond OSS) is noble and worthy of special consideration.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1035057014
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/402
node_id: IC_kwDOCGYnMM49sbd2
created_at: 2022-02-10T15:30:28Z
updated_at: 2022-02-10T15:30:40Z
author_association: CONTRIBUTOR
body:
  Yeah, the CLI experience is probably where any kind of multi-column, configured setup is going to fall apart. Sticking with GIS examples, one way I might think about this is using the [fiona CLI](https://fiona.readthedocs.io/en/latest/cli.html):

  ```sh
  # assuming a database is already created and has SpatiaLite
  fio cat boundary.shp | sqlite-utils insert boundaries --conversion geometry GeometryGeoJSON -
  ```

  Anyway, very interested to see where you land here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1634#issuecomment-1035664928
issue_url: https://api.github.com/repos/simonw/datasette/issues/1634
node_id: IC_kwDOBm6k_c49uv4g
created_at: 2022-02-11T00:10:07Z
updated_at: 2022-02-11T00:10:23Z
author_association: OWNER
body:
  Could also bump this up to Python 3.10: https://github.com/simonw/datasette/blob/5619069968ab39fd44c44a1888965e361c6f7fb9/Dockerfile#L1

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1634#issuecomment-1035667060
issue_url: https://api.github.com/repos/simonw/datasette/issues/1634
node_id: IC_kwDOBm6k_c49uwZ0
created_at: 2022-02-11T00:13:22Z
updated_at: 2022-02-11T00:13:22Z
author_association: OWNER
body:
  Looks like `3.10.2` is the latest: https://hub.docker.com/_/python?tab=tags&page=1&name=3.10.2-slim-bu

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1143#issuecomment-1038289584
issue_url: https://api.github.com/repos/simonw/datasette/issues/1143
node_id: IC_kwDOBm6k_c494wqw
created_at: 2022-02-13T17:40:50Z
updated_at: 2022-02-13T17:41:17Z
author_association: OWNER
body:
  The way Drupal does this is interesting; https://www.drupal.org/node/2715637 - it supports the following YAML:
  ```yaml
    # Configure Cross-Site HTTP requests (CORS).
    # Read https://developer.mozilla.org/en-US/docs/Web/HTTP/Access_control_CORS
    # for more information about the topic in general.
    # Note: By default the configuration is disabled.
   cors.config:
     enabled: false
     # Specify allowed headers, like 'x-allowed-header'.
     allowedHeaders: []
     # Specify allowed request methods, specify ['*'] to allow all possible ones.
     allowedMethods: []
     # Configure requests allowed from specific origins.
     allowedOrigins: ['*']
     # Sets the Access-Control-Expose-Headers header.
     exposedHeaders: false
     # Sets the Access-Control-Max-Age header.
     maxAge: false
     # Sets the Access-Control-Allow-Credentials header.
     supportsCredentials: false
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1038336591
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/398
node_id: IC_kwDOCGYnMM4948JP
created_at: 2022-02-13T18:48:21Z
updated_at: 2022-02-13T18:49:49Z
author_association: CONTRIBUTOR
body:
  Been chipping away at this between other things and realized `sqlite-utils init-spatialite` is probably unnecessary. Any of the other commands requires running `db.init_spatialite` to have the extension functions available, and that will do everything `init-spatialite` would do.

  I think it's probably worth keeping a SpatiaLite flag on `create-database` in case you wanted to create all the spatial metadata up front. Otherwise, it's going to get added the first time you run `add-geometry-column` or `create-spatial-index`, which is probably fine in most cases.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/407#issuecomment-1040519196
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/407
node_id: IC_kwDOCGYnMM4-BRAc
created_at: 2022-02-15T16:52:21Z
updated_at: 2022-02-15T18:12:03Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/407?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#407](https://codecov.io/gh/simonw/sqlite-utils/pull/407?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (a974da5) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/e7f040106b5f5a892ebd984f19b21c605e87c142?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (e7f0401) will **increase** coverage by `0.71%`.
  > The diff coverage is `85.00%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/407/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/407?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@            Coverage Diff             @@
  ##             main     #407      +/-   ##
  ==========================================
  + Coverage   95.91%   96.62%   +0.71%     
  ==========================================
    Files           6        6              
    Lines        2421     2460      +39     
  ==========================================
  + Hits         2322     2377      +55     
  + Misses         99       83      -16     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/407?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [sqlite\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/407/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.76% <85.00%> (+0.06%)` | :arrow_up: |
  | [sqlite\_utils/utils.py](https://codecov.io/gh/simonw/sqlite-utils/pull/407/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `94.59% <0.00%> (ø)` | |
  | [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/407/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.72% <0.00%> (+1.43%)` | :arrow_up: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/407?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/407?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [e7f0401...a974da5](https://codecov.io/gh/simonw/sqlite-utils/pull/407?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/407#issuecomment-1040596969
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/407
node_id: IC_kwDOCGYnMM4-Bj_p
created_at: 2022-02-15T17:56:22Z
updated_at: 2022-02-15T17:56:35Z
author_association: OWNER
body:
  We should add SpatiaLite to the action that calculates code coverage - that way we can calculate coverage across the new GIS tests as well: https://github.com/simonw/sqlite-utils/blob/main/.github/workflows/test-coverage.yml

  Should just be a case of adding this to that workflow - we can do this in the same PR.

  ```
      - name: Install SpatiaLite
        run: sudo apt-get install libsqlite3-mod-spatialite
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/407#issuecomment-1040598665
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/407
node_id: IC_kwDOCGYnMM4-BkaJ
created_at: 2022-02-15T17:58:11Z
updated_at: 2022-02-15T17:58:11Z
author_association: OWNER
body:
  Wow, just found out I can edit files in this PR branch by hitting `.` on my keyboard while looking at the PR, then making changes in the VS Code for web on `github.dev`!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/407#issuecomment-1040959312
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/407
node_id: IC_kwDOCGYnMM4-C8dQ
created_at: 2022-02-16T00:58:32Z
updated_at: 2022-02-16T00:58:32Z
author_association: OWNER
body:
  This is honestly one of the most complete PRs I've ever seen for a feature of this size. Thanks so much for this!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1040965672
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/398
node_id: IC_kwDOCGYnMM4-C-Ao
created_at: 2022-02-16T01:02:29Z
updated_at: 2022-02-16T01:02:29Z
author_association: OWNER
body:
  Documentation:

  - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#create-database
  - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#add-geometry-column
  - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#create-spatial-index
  - https://sqlite-utils.datasette.io/en/latest/cli.html#spatialite-helpers

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/406#issuecomment-1040974519
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/406
node_id: IC_kwDOCGYnMM4-DAK3
created_at: 2022-02-16T01:08:17Z
updated_at: 2022-02-16T01:08:17Z
author_association: OWNER
body:
  I had no idea this was possible! I guess SQLite will allow any text string as the column type, defaulting to `TEXT` as the underlying default representation if it doesn't recognize the type.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/406#issuecomment-1040978032
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/406
node_id: IC_kwDOCGYnMM4-DBBw
created_at: 2022-02-16T01:10:31Z
updated_at: 2022-02-16T01:10:31Z
author_association: OWNER
body:
  Allowing custom strings in the `create()` method, as you suggest in your example, feels like a reasonable way to support this.
  ```python
  db["dummy"].create({
      "title": str,
      "vector": "array",
  })
  ```
  I'm slightly nervous about that just because people might accidentally use this without realizig what they are doing - passing `"column-name": "string"` for example when they should have used `"column-name": str` in order to get a `TEXT` column.

  Alternatively, this could work:

  ```python
  db["dummy"].create({
      "title": str,
      "vector": CustomColumnType("array")
  })
  ```
  This would play better with `mypy` too I think.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/407#issuecomment-1040998433
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/407
node_id: IC_kwDOCGYnMM4-DGAh
created_at: 2022-02-16T01:29:39Z
updated_at: 2022-02-16T01:29:39Z
author_association: CONTRIBUTOR
body:
  Happy to do it and have it in the library. Going to use it a bunch. This whole SpatiaLite toolchain become a huge part of my work in the past year.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/406#issuecomment-1041313679
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/406
node_id: IC_kwDOCGYnMM4-ES-P
created_at: 2022-02-16T09:59:51Z
updated_at: 2022-02-16T10:00:10Z
author_association: NONE
body:
  The `CustomColumnType()` approach looks good. This pushes you into the mindspace that you are defining and working with a custom column type.

  When creating the table, you could then error, or at least warn, if someone wasn't setting a column on a `type` or a custom column type, which I guess is where `mypy` comes in?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1041325398
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/402
node_id: IC_kwDOCGYnMM4-EV1W
created_at: 2022-02-16T10:12:48Z
updated_at: 2022-02-16T10:18:55Z
author_association: NONE
body:
  > My hunch is that the case where you want to consider input from more than one column will actually be pretty rare - the only case I can think of where I would want to do that is for latitude/longitude columns

  Other possible pairs: unconventional date/datetime and timezone pairs eg `2022-02-16::17.00, London`; or more generally, numerical value and unit of measurement pairs (eg if you want to cast into and out of different measurement units using packages like `pint`) or currencies etc. Actually, in that case, I guess you may be presenting things that are unit typed already, and so a conversion would need to parse things into an appropriate, possibly two column `value, unit` format.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/406#issuecomment-1041363433
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/406
node_id: IC_kwDOCGYnMM4-EfHp
created_at: 2022-02-16T10:57:03Z
updated_at: 2022-02-16T10:57:19Z
author_association: NONE
body:
  Wondering if this actually relates to https://github.com/simonw/sqlite-utils/issues/402 ?

  I also wonder if this would be a sensible approach for eg registering `pint` based quantity conversions into and out of the db, perhaps storing the quantity as a serialised `magnitude measurement` single column string?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/327#issuecomment-1043609198
issue_url: https://api.github.com/repos/simonw/datasette/issues/327
node_id: IC_kwDOBm6k_c4-NDZu
created_at: 2022-02-17T23:21:36Z
updated_at: 2022-02-17T23:33:01Z
author_association: NONE
body:
  On fly.io. This particular database goes from 1.4GB to 200M. Slower, part of that might be having no `--inspect-file`?

  ```
  $ datasette publish fly ...   --generate-dir /tmp/deploy-this
  ...
  $ mksquashfs large.db large.squashfs
  $ rm large.db # don't accidentally put it in the image
  $ cat Dockerfile
  FROM python:3.8
  COPY . /app
  WORKDIR /app

  ENV DATASETTE_SECRET 'xyzzy'
  RUN pip install -U datasette
  # RUN datasette inspect large.db --inspect-file inspect-data.json
  ENV PORT 8080
  EXPOSE 8080
  CMD mount -o loop -t squashfs large.squashfs /mnt; datasette serve --host 0.0.0.0 -i /mnt/large.db --cors --port $PORT
  ```

  It would also be possible to copy the file onto the ~6GB available on the ephemeral container filesystem on startup. A little against the spirit of the thing? On this example the whole docker image is 2.42 GB and the squashfs version is 1.14 GB.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/327#issuecomment-1043626870
issue_url: https://api.github.com/repos/simonw/datasette/issues/327
node_id: IC_kwDOBm6k_c4-NHt2
created_at: 2022-02-17T23:37:24Z
updated_at: 2022-02-17T23:37:24Z
author_association: NONE
body:
  On second thought any kind of quick-to-decompress-on-startup could be helpful if we're paying for the container registry and deployment bandwidth but not ephemeral storage.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-1045024276
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c4-Sc4U
created_at: 2022-02-18T19:01:42Z
updated_at: 2022-02-18T19:55:24Z
author_association: OWNER
body:
  > Maybe I should use `-/` to encode forward slashes too, to defend against any ASGI servers that might not implement `raw_path` correctly.
  ```python
  def dash_encode(s):
      return s.replace("-", "--").replace(".", "-.").replace("/", "-/")

  def dash_decode(s):
      return s.replace("-/", "/").replace("-.", ".").replace("--", "-")
  ```

  ```pycon
  >>> dash_encode("foo/bar/baz.csv")
  'foo-/bar-/baz-.csv'
  >>> dash_decode('foo-/bar-/baz-.csv')
  'foo/bar/baz.csv'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-1045027067
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c4-Sdj7
created_at: 2022-02-18T19:03:26Z
updated_at: 2022-02-18T19:03:26Z
author_association: OWNER
body:
  (If I make this change it may break some existing Datasette installations when they upgrade - I could try and build a plugin for them which triggers on 404s and checks to see if the old format would return a 200 response, then returns that.)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-1045032377
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c4-Se25
created_at: 2022-02-18T19:06:50Z
updated_at: 2022-02-18T19:06:50Z
author_association: OWNER
body:
  How does URL routing for https://latest.datasette.io/fixtures/table%2Fwith%2Fslashes.csv work?

  Right now it's https://github.com/simonw/datasette/blob/7d24fd405f3c60e4c852c5d746c91aa2ba23cf5b/datasette/app.py#L1098-L1101

  That's not going to capture the dot-dash encoding version of that table name:
  ```pycon
  >>> dot_dash_encode("table/with/slashes.csv")
  'table-/with-/slashes-.csv'
  ```
  Probably needs a fancy regex trick like a negative lookbehind assertion or similar.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-1045055772
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c4-Skkc
created_at: 2022-02-18T19:23:33Z
updated_at: 2022-02-18T19:25:42Z
author_association: OWNER
body:
  I want a match for this URL:

      /db/table-/with-/slashes-.csv

  Maybe this:

      ^/(?P<db_name>[^/]+)/(?P<table_and_format>([^/]*|(\-/)*|(\-\.)*|(\.\.)*)*$)

  Here we are matching a sequence of:

      ([^/]*|(\-/)*|(\-\.)*|(\-\-)*)*

  So a combination of not-slashes OR -/ or -. Or -- sequences

  <img width="224" alt="image" src="https://user-images.githubusercontent.com/9599/154748362-84909d4e-dccf-454b-a9cd-a036f9f66f09.png">

      ^/(?P<db_name>[^/]+)/(?P<table_and_format>([^/]*|(\-/)*|(\-\.)*|(\-\-)*)*$)

  Try that with non-capturing bits:

      ^/(?P<db_name>[^/]+)/(?P<table_and_format>(?:[^/]*|(?:\-/)*|(?:\-\.)*|(?:\-\-)*)*$)

  `(?:[^/]*|(?:\-/)*|(?:\-\.)*|(?:\-\-)*)*` visualized is:

  <img width="193" alt="image" src="https://user-images.githubusercontent.com/9599/154748441-decea502-0d04-44f4-9ca9-fb6883767833.png">

  Here's the explanation on regex101.com https://regex101.com/r/CPnsIO/1

  <img width="1074" alt="image" src="https://user-images.githubusercontent.com/9599/154748720-cdda61db-5498-49a8-91c2-e726b394fa49.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-1045059427
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c4-Sldj
created_at: 2022-02-18T19:26:25Z
updated_at: 2022-02-18T19:26:25Z
author_association: OWNER
body:
  With this new pattern I could probably extract out the optional `.json` format string as part of the initial route capturing regex too, rather than the current `table_and_format` hack.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-1045069481
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c4-Sn6p
created_at: 2022-02-18T19:34:41Z
updated_at: 2022-03-05T21:32:22Z
author_association: OWNER
body:
  I think I got format extraction working! https://regex101.com/r/A0bW1D/1

      ^/(?P<database>[^/]+)/(?P<table>(?:[^\/\-\.]*|(?:\-/)*|(?:\-\.)*|(?:\-\-)*)*?)(?:(?<!\-)\.(?P<format>\w+))?$

  I had to make that crazy inner one even more complicated to stop it from capturing `.` that was not part of `-.`.

      (?:[^\/\-\.]*|(?:\-/)*|(?:\-\.)*|(?:\-\-)*)*

  Visualized:

  <img width="222" alt="image" src="https://user-images.githubusercontent.com/9599/154749714-44579899-5dc7-4e5f-ad4f-dc59dac48979.png">

  So now I have a regex which can extract out the dot-encoded table name AND spot if there is an optional `.format` at the end:

  <img width="1090" alt="image" src="https://user-images.githubusercontent.com/9599/156900484-7912073f-28aa-4301-86e2-e5cbe625e1d5.png">

  If I end up using this in Datasette it's going to need VERY comprehensive unit tests and inline documentation.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-1045075207
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c4-SpUH
created_at: 2022-02-18T19:39:35Z
updated_at: 2022-02-18T19:40:13Z
author_association: OWNER
body:
  > And if for some horific reason you had a table with the name `/db/table-.csv.csv` (so `/db/` was the first part of the actual table name in SQLite) the URLs would look like this:
  > 
  >     * `/db/%2Fdb%2Ftable---.csv-.csv` - the HTML version
  >     * `/db/%2Fdb%2Ftable---.csv-.csv.csv` - the CSV version
  >     * `/db/%2Fdb%2Ftable---.csv-.csv.json` - the JSON version

  Here's what those look like with the updated version of `dot_dash_encode()` that also encodes `/` as `-/`:

  - `/db/-/db-/table---.csv-.csv` - HTML
  - `/db/-/db-/table---.csv-.csv.csv` - CSV
  - `/db/-/db-/table---.csv-.csv.json` - JSON

  <img width="1050" alt="image" src="https://user-images.githubusercontent.com/9599/154750631-a8a23c62-3dfc-43e4-8026-4d117dc4bf8d.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-1045077590
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c4-Sp5W
created_at: 2022-02-18T19:41:37Z
updated_at: 2022-02-18T19:42:41Z
author_association: OWNER
body:
  Ugh, one disadvantage I just spotted with this: Datasette already has a `/-/versions.json` convention where "system" URLs are namespaced under `/-/` - but that could be confused under this new scheme with the `-/` escaping sequence.

  And I've thought about adding `/db/-/special` and `/db/table/-/special` URLs in the past too.

  Maybe change this system to use `.` as the escaping character instead of `-`?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-1045081042
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c4-SqvS
created_at: 2022-02-18T19:44:12Z
updated_at: 2022-02-18T19:51:34Z
author_association: OWNER
body:
  ```python
  def dot_encode(s):
      return s.replace(".", "..").replace("/", "./")

  def dot_decode(s):
      return s.replace("./", "/").replace("..", ".")
  ```
  No need for hyphen encoding in this variant at all, which simplifies things a bit.

  (Update: this is flawed, see https://github.com/simonw/datasette/issues/1439#issuecomment-1045086033)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-1045082891
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c4-SrML
created_at: 2022-02-18T19:45:32Z
updated_at: 2022-02-18T19:45:32Z
author_association: OWNER
body:
  ```pycon
  >>> dot_encode("/db/table-.csv.csv")
  './db./table-..csv..csv'
  >>> dot_decode('./db./table-..csv..csv')
  '/db/table-.csv.csv'
  ```
  I worry that web servers might treat `./` in a special way though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-1045086033
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c4-Sr9R
created_at: 2022-02-18T19:47:43Z
updated_at: 2022-02-18T19:51:11Z
author_association: OWNER
body:
  - https://datasette.io/-/asgi-scope/db/./db./table-..csv..csv
  - https://til.simonwillison.net/-/asgi-scope/db/./db./table-..csv..csv

  Do both of those survive the round-trip to populate `raw_path` correctly?

  No! In both cases the `/./` bit goes missing.

  It looks like this might even be a client issue - `curl` shows me this:

  ```
  ~ % curl -vv -i 'https://datasette.io/-/asgi-scope/db/./db./table-..csv..csv'
  *   Trying 216.239.32.21:443...
  * Connected to datasette.io (216.239.32.21) port 443 (#0)
  * ALPN, offering http/1.1
  * TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
  * Server certificate: datasette.io
  * Server certificate: R3
  * Server certificate: ISRG Root X1
  > GET /-/asgi-scope/db/db./table-..csv..csv HTTP/1.1
  ```
  So `curl` decided to turn `/-/asgi-scope/db/./db./table` into `/-/asgi-scope/db/db./table` before even sending the request.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-1045095348
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c4-SuO0
created_at: 2022-02-18T19:53:48Z
updated_at: 2022-02-18T19:53:48Z
author_association: OWNER
body:
  > Ugh, one disadvantage I just spotted with this: Datasette already has a `/-/versions.json` convention where "system" URLs are namespaced under `/-/` - but that could be confused under this new scheme with the `-/` escaping sequence.
  > 
  > And I've thought about adding `/db/-/special` and `/db/table/-/special` URLs in the past too.

  I don't think this matters. The new regex does indeed capture that kind of page:

  <img width="1052" alt="image" src="https://user-images.githubusercontent.com/9599/154752309-e1787755-3bdb-47c2-867c-7ac5fe65664d.png">

  But Datasette goes through configured route regular expressions in order - so I can have the regex that captures `/db/-/special` routes listed before the one that captures tables and formats.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-1045099290
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c4-SvMa
created_at: 2022-02-18T19:56:18Z
updated_at: 2022-02-18T19:56:30Z
author_association: OWNER
body:
  > ```python
  > def dash_encode(s):
  >     return s.replace("-", "--").replace(".", "-.").replace("/", "-/")
  > 
  > def dash_decode(s):
  >     return s.replace("-/", "/").replace("-.", ".").replace("--", "-")
  > ```

  I think **dash-encoding** (new name for this) is the right way forward here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-1045108611
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c4-SxeD
created_at: 2022-02-18T20:02:19Z
updated_at: 2022-02-18T20:08:34Z
author_association: OWNER
body:
  One other potential variant:
  ```python
  def dash_encode(s):
      return s.replace("-", "-dash-").replace(".", "-dot-").replace("/", "-slash-")

  def dash_decode(s):
      return s.replace("-slash-", "/").replace("-dot-", ".").replace("-dash-", "-")
  ```
  Except this has bugs - it doesn't round-trip safely, because it can get confused about things like `-dash-slash-` in terms of is that a `-dash-` or a `-slash-`?
  ```pycon
  >>> dash_encode("/db/table-.csv.csv")
  '-slash-db-slash-table-dash--dot-csv-dot-csv'
  >>> dash_decode('-slash-db-slash-table-dash--dot-csv-dot-csv')
  '/db/table-.csv.csv'
  >>> dash_encode('-slash-db-slash-table-dash--dot-csv-dot-csv')
  '-dash-slash-dash-db-dash-slash-dash-table-dash-dash-dash--dash-dot-dash-csv-dash-dot-dash-csv'
  >>> dash_decode('-dash-slash-dash-db-dash-slash-dash-table-dash-dash-dash--dash-dot-dash-csv-dash-dot-dash-csv')
  '-dash/dash-db-dash/dash-table-dash--dash.dash-csv-dash.dash-csv'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-1045111309
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c4-SyIN
created_at: 2022-02-18T20:04:24Z
updated_at: 2022-02-18T20:05:40Z
author_association: OWNER
body:
  This made me worry that my current `dash_decode()` implementation had unknown round-trip bugs, but thankfully this works OK:
  ```pycon
  >>> dash_encode("/db/table-.csv.csv")
  '-/db-/table---.csv-.csv'
  >>> dash_encode('-/db-/table---.csv-.csv')
  '---/db---/table-------.csv---.csv'
  >>> dash_decode('---/db---/table-------.csv---.csv')
  '-/db-/table---.csv-.csv'
  >>> dash_decode('-/db-/table---.csv-.csv')
  '/db/table-.csv.csv'
  ``` 
  The regex still works against that double-encoded example too:

  <img width="1032" alt="image" src="https://user-images.githubusercontent.com/9599/154753916-b7d2159e-4284-4c92-ae61-110671fa320e.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-1045117304
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c4-Szl4
created_at: 2022-02-18T20:09:22Z
updated_at: 2022-02-18T20:09:22Z
author_association: OWNER
body:
  Adopting this could result in supporting database files with surprising characters in their filename too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-1045131086
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c4-S29O
created_at: 2022-02-18T20:22:13Z
updated_at: 2022-02-18T20:22:47Z
author_association: OWNER
body:
  Should it encode `%` symbols too, since they have a special meaning in URLs and we can't guarantee that every single web server / proxy out there will round-trip them safely using percentage encoding? If so, would need to pick a different encoding character for them. Maybe `%` becomes `-p` - and in that case `/` could become `-s` too.

  Is it worth expanding dash-encoding outside of just `/` and `-` and `.` though? Not sure.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-1045134050
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c4-S3ri
created_at: 2022-02-18T20:25:04Z
updated_at: 2022-02-18T20:25:04Z
author_association: OWNER
body:
  Here's a useful modern spec for how existing URL percentage encoding is supposed to work: https://url.spec.whatwg.org/#percent-encoded-bytes

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-1045269544
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c4-TYwo
created_at: 2022-02-18T22:19:29Z
updated_at: 2022-02-18T22:19:29Z
author_association: OWNER
body:
  Note that I've ruled out using `Accept: application/json` to return JSON because it turns out Cloudflare and potentially other CDNs ignore the `Vary: Accept` header entirely:
  - https://github.com/simonw/datasette/issues/1534

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-1049124390
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c4-iF4m
created_at: 2022-02-23T19:15:00Z
updated_at: 2022-02-23T19:15:00Z
author_association: OWNER
body:
  I'll start by modifying this function: https://github.com/simonw/datasette/blob/458f03ad3a454d271f47a643f4530bd8b60ddb76/datasette/utils/__init__.py#L732-L749

  Later I want to move this to the routing layer to split out `format` automatically, as seen in the regexes here: https://github.com/simonw/datasette/issues/1439#issuecomment-1045069481

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-1049126151
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c4-iGUH
created_at: 2022-02-23T19:17:01Z
updated_at: 2022-02-23T19:17:01Z
author_association: OWNER
body:
  Actually the relevant code looks to be: https://github.com/simonw/datasette/blob/7d24fd405f3c60e4c852c5d746c91aa2ba23cf5b/datasette/views/base.py#L481-L498

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/62#issuecomment-1049775451
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/62
node_id: IC_kwDODEm0Qs4-kk1b
created_at: 2022-02-24T11:43:31Z
updated_at: 2022-02-24T11:43:31Z
author_association: NONE
body:
  i seem to have fixed this issue by applying for [elevated API access](https://developer.twitter.com/en/portal/products/elevated)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1641#issuecomment-1049879118
issue_url: https://api.github.com/repos/simonw/datasette/issues/1641
node_id: IC_kwDOBm6k_c4-k-JO
created_at: 2022-02-24T13:49:26Z
updated_at: 2022-02-24T13:49:26Z
author_association: CONTRIBUTOR
body:
  maybe worth considering adding buttons for paren, asterisk, etc. under the input text box on mobile?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/260#issuecomment-1051473892
issue_url: https://api.github.com/repos/simonw/datasette/issues/260
node_id: IC_kwDOBm6k_c4-rDfk
created_at: 2022-02-26T02:24:15Z
updated_at: 2022-02-26T02:24:15Z
author_association: NONE
body:
  Is there already functionality that can be used to validate the `metadata.json` file? Is there a JSON Schema that defines it? Or a validation that's available via datasette with Python? We're working on [automatically building the metadata](https://github.com/catalyst-cooperative/pudl/pull/1479) in CI and when we deploy to cloud run, and it would be nice to be able to check whether the the metadata we're outputting is valid in our tests.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-1053973425
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c4-0lux
created_at: 2022-02-28T07:40:12Z
updated_at: 2022-02-28T07:40:12Z
author_association: OWNER
body:
  If I make this change it will break existing links to one of the oldest Datasette demos: http://fivethirtyeight.datasettes.com/fivethirtyeight/avengers%2Favengers

  A plugin that fixes those by redirecting them on 404 would be neat.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/343#issuecomment-1055854884
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/343
node_id: IC_kwDOCGYnMM4-7xEk
created_at: 2022-03-01T21:03:45Z
updated_at: 2022-03-01T21:03:45Z
author_association: OWNER
body:
  Just found myself needing this capability myself!

  Relevant code:

  https://github.com/simonw/sqlite-utils/blob/8f386a0d300d1b1c76132bb75972b755049fb742/sqlite_utils/db.py#L2297-L2307

  https://github.com/simonw/sqlite-utils/blob/8f386a0d300d1b1c76132bb75972b755049fb742/sqlite_utils/db.py#L2996-L3001

  So various functions could grow a `hash_id_columns=("title", "date")` argument which causes just those columns to be included in the hash.

  Bonus: if you use `hash_id_columns=...` without setting `hash_id="id"` it could assume that you want the column to be called `id`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/343#issuecomment-1055855845
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/343
node_id: IC_kwDOCGYnMM4-7xTl
created_at: 2022-03-01T21:04:45Z
updated_at: 2022-03-01T22:43:38Z
author_association: OWNER
body:
  I'm going to make that `_hash()` utility function a documented, non-underscore-prefixed function too - called `hash_record()`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/409#issuecomment-1055930639
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/409
node_id: IC_kwDOCGYnMM4-8DkP
created_at: 2022-03-01T22:40:15Z
updated_at: 2022-03-01T22:40:15Z
author_association: OWNER
body:
  This test fails and I don't understand why:
  ```python
  from sqlite_utils import Database


  def test_transaction():
      db1 = Database(memory_name="transaction_test", tracer=print)
      db2 = Database(memory_name="transaction_test", tracer=print)
      with db1.conn:
          db1["t"].insert({"foo": 1})
          assert list(db2["t"].rows) == []
      assert list(db2["t"].rows) == [{"foo": 1}]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/343#issuecomment-1055991226
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/343
node_id: IC_kwDOCGYnMM4-8SW6
created_at: 2022-03-02T00:03:47Z
updated_at: 2022-03-02T00:03:47Z
author_association: OWNER
body:
  Oops, broke mypy:

  ```
  sqlite_utils/db.py:2600: error: Incompatible default for argument "hash_id_columns" (default has type "Default", argument has type "Optional[Iterable[str]]")
  Found 1 error in 1 file (checked 49 source files)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/343#issuecomment-1055992544
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/343
node_id: IC_kwDOCGYnMM4-8Srg
created_at: 2022-03-02T00:06:10Z
updated_at: 2022-03-02T00:06:10Z
author_association: OWNER
body:
  Updated documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#setting-an-id-based-on-the-hash-of-the-row-contents

  Documentation for the renamed `utils.hash_record()` function: https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite-utils-utils-hash-record

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/408#issuecomment-1055993700
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/408
node_id: IC_kwDOCGYnMM4-8S9k
created_at: 2022-03-02T00:08:10Z
updated_at: 2022-03-02T00:08:10Z
author_association: OWNER
body:
  I thought I'd made it so `deterministic=True` would be silently ignored in environments that don't support it, but clearly I missed a case here!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/408#issuecomment-1055995100
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/408
node_id: IC_kwDOCGYnMM4-8TTc
created_at: 2022-03-02T00:10:41Z
updated_at: 2022-03-02T00:10:41Z
author_association: OWNER
body:
  Here's the code in question:

  https://github.com/simonw/sqlite-utils/blob/521921b849003ed3742338f76f9d47ff3d95eaf3/sqlite_utils/db.py#L384-L394

  It's checking for Python 3.8, because that's the version of Python that added the `deterministic=True` option: https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.create_function

  But from your error message it looks like it should be checking the SQLite version too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/408#issuecomment-1055996626
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/408
node_id: IC_kwDOCGYnMM4-8TrS
created_at: 2022-03-02T00:12:21Z
updated_at: 2022-03-02T00:12:21Z
author_association: OWNER
body:
  Here's the SQLite changelog mentioning that it was added in 3.8.3: https://www.sqlite.org/changes.html#version_3_8_3

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1645#issuecomment-1059633902
issue_url: https://api.github.com/repos/simonw/datasette/issues/1645
node_id: IC_kwDOBm6k_c4_KLru
created_at: 2022-03-05T01:03:06Z
updated_at: 2022-03-05T01:03:06Z
author_association: OWNER
body:
  I agree: this is bad.

  Ideally, content served from `/static/` would apply best practices for static content serving - which to my mind means the following:

  - Where possible, serve with a far-future cache expiry header and use an asset URL that changes when the file itself changes
  - For assets without that, support conditional GET to avoid transferring the whole asset if it hasn't changed
  - Some kind of sensible mechanism for setting cache TTLs on assets that don't have a unique-file-per-version - in particular assets that might be served from plugins.

  Datasette half-implemented the first of these: if you view source on https://latest.datasette.io/ you'll see it links to `/-/static/app.css?cead5a` - which in the template looks like this:

  https://github.com/simonw/datasette/blob/dd94157f8958bdfe9f45575add934ccf1aba6d63/datasette/templates/base.html#L5

  I had forgotten I had implemented this! Here is how it is calculated:

  https://github.com/simonw/datasette/blob/458f03ad3a454d271f47a643f4530bd8b60ddb76/datasette/app.py#L510-L516

  So `app.css` right now could be safely served with a far-future cache header... only it isn't:
  ```
  ~ % curl -i 'https://latest.datasette.io/-/static/app.css?cead5a' 
  HTTP/2 200 
  content-type: text/css
  x-databases: _memory, _internal, fixtures, extra_database
  x-cloud-trace-context: 9ddc825620eb53d30fc127d1c750f342
  date: Sat, 05 Mar 2022 01:01:53 GMT
  server: Google Frontend
  content-length: 16178
  ```
  The larger question though is what to do about other assets. I'm particularly interested in plugin assets, since visualization plugins like `datasette-vega` and `datasette-cluster-map` ship with large amounts of JavaScript and I'd really like that to be sensibly cached by default.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1645#issuecomment-1059634412
issue_url: https://api.github.com/repos/simonw/datasette/issues/1645
node_id: IC_kwDOBm6k_c4_KLzs
created_at: 2022-03-05T01:04:53Z
updated_at: 2022-03-05T01:04:53Z
author_association: OWNER
body:
  The existing `app_css_hash` already isn't good enough, because I built that before `table.js` existed, and that file should obviously be smartly cached too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1645#issuecomment-1059634688
issue_url: https://api.github.com/repos/simonw/datasette/issues/1645
node_id: IC_kwDOBm6k_c4_KL4A
created_at: 2022-03-05T01:06:08Z
updated_at: 2022-03-05T01:06:08Z
author_association: OWNER
body:
  It sounds like you can workaround this with Varnish configuration for the moment, but I'm going to bump this up the list of things to fix - it's particularly relevant now as I'd like to get a solution in place before Datasette 1.0, since it's likely to be beneficial to plugins and hence should be part of the stable, documented plugin interface.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1642#issuecomment-1059635969
issue_url: https://api.github.com/repos/simonw/datasette/issues/1642
node_id: IC_kwDOBm6k_c4_KMMB
created_at: 2022-03-05T01:11:17Z
updated_at: 2022-03-05T01:11:17Z
author_association: OWNER
body:
  `pip install datasette` in a fresh virtual environment doesn't show any warnings.

  Neither does `pip install -e '.'` in a fresh checkout. Or `pip install -e '.[test]'`.

  Closing this as can't reproduce.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1640#issuecomment-1059636420
issue_url: https://api.github.com/repos/simonw/datasette/issues/1640
node_id: IC_kwDOBm6k_c4_KMTE
created_at: 2022-03-05T01:13:26Z
updated_at: 2022-03-05T01:13:26Z
author_association: OWNER
body:
  Hah, this is certainly unexpected.

  It looks like this is the code in question: https://github.com/simonw/datasette/blob/a6ff123de5464806441f6a6f95145c9a83b7f20b/datasette/utils/asgi.py#L259-L266

  You're right: it assumes that the file it is serving won't change length while it is serving it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1640#issuecomment-1059638778
issue_url: https://api.github.com/repos/simonw/datasette/issues/1640
node_id: IC_kwDOBm6k_c4_KM36
created_at: 2022-03-05T01:19:00Z
updated_at: 2022-03-05T01:19:00Z
author_association: OWNER
body:
  The reason I implemented it like this was to support things like the `curl` progress bar if users decide to serve up large files using the `--static` mechanism.

  Here's the code that hooks it up to the URL resolver:

  https://github.com/simonw/datasette/blob/458f03ad3a454d271f47a643f4530bd8b60ddb76/datasette/app.py#L1001-L1005

  Which uses this function:

  https://github.com/simonw/datasette/blob/a6ff123de5464806441f6a6f95145c9a83b7f20b/datasette/utils/asgi.py#L285-L310

  One option here would be to support a workaround that looks something like this:

      http://localhost:8001/my-static/log.txt?_unknown_size=1`

  The URL routing code could then look out for that `?_unknown_size=1` option and, if it's present, omit the `content-length` header entirely.

  It's a bit of a cludge, but it would be pretty straight-forward to implement.

  Would that work for you @broccolihighkicks?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059646247
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/412
node_id: IC_kwDOCGYnMM4_KOsn
created_at: 2022-03-05T01:51:03Z
updated_at: 2022-03-05T01:51:03Z
author_association: OWNER
body:
  I considered two ways of doing this.

  First, have methods such as `db.query_df()` and `table.rows_df` which do the same as `.query()` and `table.rows` but return a DataFrame instead of a generator of dictionaries.

  Second, have a compatibility class that is imported separately such as:
  ```python
  from sqlite_utils.pandas import Database
  ```
  Then have the `.query()` and `.rows` and other similar methods return dataframes.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059646543
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/412
node_id: IC_kwDOCGYnMM4_KOxP
created_at: 2022-03-05T01:52:47Z
updated_at: 2022-03-05T01:52:47Z
author_association: OWNER
body:
  I built a prototype of that second option and it looks pretty good:

  <img width="1108" alt="image" src="https://user-images.githubusercontent.com/9599/156862952-57cc937c-7f40-4137-b78f-ecc921a5d9a7.png">

  Here's the `pandas.py` prototype:

  ```python
  from .db import Database as _Database, Table as _Table, View as _View
  import pandas as pd
  from typing import (
      Iterable,
      Union,
      Optional,
  )


  class Database(_Database):
      def query(
          self, sql: str, params: Optional[Union[Iterable, dict]] = None
      ) -> pd.DataFrame:
          return pd.DataFrame(super().query(sql, params))

      def table(self, table_name: str, **kwargs) -> Union["Table", "View"]:
          "Return a table object, optionally configured with default options."
          klass = View if table_name in self.view_names() else Table
          return klass(self, table_name, **kwargs)


  class PandasQueryable:
      def rows_where(
          self,
          where: str = None,
          where_args: Optional[Union[Iterable, dict]] = None,
          order_by: str = None,
          select: str = "*",
          limit: int = None,
          offset: int = None,
      ) -> pd.DataFrame:
          return pd.DataFrame(
              super().rows_where(
                  where,
                  where_args,
                  order_by=order_by,
                  select=select,
                  limit=limit,
                  offset=offset,
              )
          )


  class Table(PandasQueryable, _Table):
      pass


  class View(PandasQueryable, _View):
      pass
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059646645
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/412
node_id: IC_kwDOCGYnMM4_KOy1
created_at: 2022-03-05T01:53:10Z
updated_at: 2022-03-05T01:53:10Z
author_association: OWNER
body:
  I'm not an experienced enough Pandas user to know if this design is right or not. I'm going to leave this open for a while and solicit some feedback.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059647114
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/412
node_id: IC_kwDOCGYnMM4_KO6K
created_at: 2022-03-05T01:54:24Z
updated_at: 2022-03-05T01:54:24Z
author_association: CONTRIBUTOR
body:
  I haven't tried this, but it looks like Pandas has a method for this: https://pandas.pydata.org/docs/reference/api/pandas.read_sql_query.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059649193
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/412
node_id: IC_kwDOCGYnMM4_KPap
created_at: 2022-03-05T02:00:02Z
updated_at: 2022-03-05T02:00:02Z
author_association: OWNER
body:
  Yeah, I imagine there are plenty of ways to do this with Pandas already - I'm opportunistically looking for a way to provide better integration with the rest of the Pandas situation from the work I've done in `sqlite-utils` already.

  Might be that this isn't worth doing at all.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059649803
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/412
node_id: IC_kwDOCGYnMM4_KPkL
created_at: 2022-03-05T02:02:41Z
updated_at: 2022-03-05T02:02:41Z
author_association: OWNER
body:
  It looks like the existing `pd.read_sql_query()` method has an optional dependency on SQLAlchemy:

  ```
  ...
  import pandas as pd
  pd.read_sql_query(db.conn, "select * from articles")
  # ImportError: Using URI string without sqlalchemy installed.
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059650190
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/412
node_id: IC_kwDOCGYnMM4_KPqO
created_at: 2022-03-05T02:04:43Z
updated_at: 2022-03-05T02:04:54Z
author_association: OWNER
body:
  To be honest, I'm having second thoughts about this now mainly because the idiom for turning a generator of dicts into a DataFrame is SO simple:

  ```python
  df = pd.DataFrame(db.query("select * from articles"))
  ```
  Given it's that simple, I'm questioning if there's any value to adding this to `sqlite-utils` at all. This likely becomes a documentation thing instead!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059651056
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/412
node_id: IC_kwDOCGYnMM4_KP3w
created_at: 2022-03-05T02:09:38Z
updated_at: 2022-03-05T02:09:38Z
author_association: OWNER
body:
  OK, so reading results from existing `sqlite-utils` into a Pandas DataFrame turns out to be trivial.

  How about writing a DataFrame to a database table?

  That feels like it could a lot more useful.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059651306
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/412
node_id: IC_kwDOCGYnMM4_KP7q
created_at: 2022-03-05T02:10:49Z
updated_at: 2022-03-05T02:10:49Z
author_association: OWNER
body:
  I could teach `.insert_all()` and `.upsert_all()` to optionally accept a DataFrame. A challenge there is `mypy` - if Pandas is an optional dependency, is it possibly to declare types that accept a Union that includes DataFrame?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059652538
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/412
node_id: IC_kwDOCGYnMM4_KQO6
created_at: 2022-03-05T02:13:17Z
updated_at: 2022-03-05T02:13:17Z
author_association: OWNER
body:
  > It looks like the existing `pd.read_sql_query()` method has an optional dependency on SQLAlchemy:
  > 
  > ```
  > ...
  > import pandas as pd
  > pd.read_sql_query(db.conn, "select * from articles")
  > # ImportError: Using URI string without sqlalchemy installed.
  > ```
  Hah, no I was wrong about this: SQLAlchemy is not needed for SQLite to work, I just had the arguments the wrong way round:
  ```python
  pd.read_sql_query("select * from articles", db.conn)
  # Shows a DateFrame
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059652834
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/412
node_id: IC_kwDOCGYnMM4_KQTi
created_at: 2022-03-05T02:14:40Z
updated_at: 2022-03-05T02:14:40Z
author_association: NONE
body:
  We do a lot of `df.to_sql()` to write into sqlite, mostly in [this moddule](https://github.com/catalyst-cooperative/pudl/blob/main/src/pudl/load.py#L25)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-1059802318
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c4_K0zO
created_at: 2022-03-05T17:34:33Z
updated_at: 2022-03-05T17:34:33Z
author_association: OWNER
body:
  Wrote documentation:

  <img width="741" alt="Dash encoding. Datasette uses a custom encoding scheme in some places, called dash encoding. This is primarily used for table names and row primary keys, to avoid any confusion between / characters in those values and the Datasette URL that references them. Dash encoding applies the following rules, in order: 1.    All single - characters are replaced by -- 2. . characters are replaced by -. 3. / characters are replaced by ./ These rules are applied in reverse order to decode a dash encoded string." src="https://user-images.githubusercontent.com/9599/156893903-5723f60e-e054-4365-84bc-f3084d11183d.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1647#issuecomment-1059804577
issue_url: https://api.github.com/repos/simonw/datasette/issues/1647
node_id: IC_kwDOBm6k_c4_K1Wh
created_at: 2022-03-05T17:49:46Z
updated_at: 2022-03-05T17:49:46Z
author_association: OWNER
body:
  My best guess is that this is an undocumented change in SQLite 3.38 - I get that test failure with that SQLite version.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1647#issuecomment-1059807598
issue_url: https://api.github.com/repos/simonw/datasette/issues/1647
node_id: IC_kwDOBm6k_c4_K2Fu
created_at: 2022-03-05T18:06:56Z
updated_at: 2022-03-05T18:08:00Z
author_association: OWNER
body:
  Had a look through the commits in https://github.com/sqlite/sqlite/compare/version-3.37.2...version-3.38.0 but couldn't see anything obvious that might have caused this.

  Really wish I had a good mechanism for running the test suite against different SQLite versions!

  May have to revisit this old trick: https://til.simonwillison.net/sqlite/ld-preload

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1647#issuecomment-1059819628
issue_url: https://api.github.com/repos/simonw/datasette/issues/1647
node_id: IC_kwDOBm6k_c4_K5Bs
created_at: 2022-03-05T19:28:54Z
updated_at: 2022-03-05T19:28:54Z
author_association: OWNER
body:
  OK, using that trick worked for testing this:

      docker run -it -p 8001:8001 ubuntu

  Then inside that container:

      apt-get install -y python3 build-essential tcl wget python3-pip git python3.8-venv

  For each version of SQLite I wanted to test I needed to figure out the tarball URL - for example, for `3.38.0` I navigated to https://www.sqlite.org/src/timeline?t=version-3.38.0 and clicked the "checkin" link and copied the tarball link:
  https://www.sqlite.org/src/tarball/40fa792d/SQLite-40fa792d.tar.gz

  Then to build it (the `CPPFLAGS` took some trial and error):
  ```
  cd /tmp
  wget https://www.sqlite.org/src/tarball/40fa792d/SQLite-40fa792d.tar.gz
  tar -xzvf SQLite-40fa792d.tar.gz
  cd SQLite-40fa792d
  CPPFLAGS="-DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_RTREE=1" ./configure
  make
  ```
  Then to test with Datasette:
  ```
  cd /tmp
  git clone https://github.com/simonw/datasette
  cd datasette
  python3 -m venv venv
  source venv/bin/activate
  pip install wheel # So bdist_wheel works in next step
  pip install -e '.[test]'
  LD_PRELOAD=/tmp/SQLite-40fa792d/.libs/libsqlite3.so pytest
  ```

  After some trial and error I proved that those tests passed with 3.36.0:
  ```
  cd /tmp
  wget https://www.sqlite.org/src/tarball/5c9a6c06/SQLite-5c9a6c06.tar.gz
  tar -xzvf SQLite-5c9a6c06.tar.gz
  cd SQLite-5c9a6c06
  CPPFLAGS="-DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_RTREE=1" ./configure
  make
  cd /tmp/datasette
  LD_PRELOAD=/tmp/SQLite-5c9a6c06/.libs/libsqlite3.so pytest tests/test_internals_database.py
  ```
  BUT failed with 3.37.0:
  ```
  # 3.37.0
  cd /tmp
  wget https://www.sqlite.org/src/tarball/bd41822c/SQLite-bd41822c.tar.gz
  tar -xzvf SQLite-bd41822c.tar.gz
  cd SQLite-bd41822c
  CPPFLAGS="-DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_RTREE=1" ./configure
  make
  cd /tmp/datasette
  LD_PRELOAD=/tmp/SQLite-bd41822c/.libs/libsqlite3.so pytest tests/test_internals_database.py
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1647#issuecomment-1059821674
issue_url: https://api.github.com/repos/simonw/datasette/issues/1647
node_id: IC_kwDOBm6k_c4_K5hq
created_at: 2022-03-05T19:44:32Z
updated_at: 2022-03-05T19:44:32Z
author_association: OWNER
body:
  I thought I'd need to introduce https://dirty-equals.helpmanual.io/types/string/ to help write tests for this, but I think I've found a good alternative that doesn't need a new dependency.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-1059822151
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c4_K5pH
created_at: 2022-03-05T19:48:35Z
updated_at: 2022-03-05T19:48:35Z
author_association: OWNER
body:
  Those new docs: https://github.com/simonw/datasette/blob/d1cb73180b4b5a07538380db76298618a5fc46b6/docs/internals.rst#dash-encoding

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1648#issuecomment-1059823151
issue_url: https://api.github.com/repos/simonw/datasette/issues/1648
node_id: IC_kwDOBm6k_c4_K54v
created_at: 2022-03-05T19:56:41Z
updated_at: 2022-03-07T15:38:08Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1648?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1648](https://codecov.io/gh/simonw/datasette/pull/1648?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (32548b8) into [main](https://codecov.io/gh/simonw/datasette/commit/7d24fd405f3c60e4c852c5d746c91aa2ba23cf5b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (7d24fd4) will **increase** coverage by `0.02%`.
  > The diff coverage is `100.00%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1648/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1648?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@            Coverage Diff             @@
  ##             main    #1648      +/-   ##
  ==========================================
  + Coverage   92.03%   92.05%   +0.02%     
  ==========================================
    Files          34       34              
    Lines        4557     4570      +13     
  ==========================================
  + Hits         4194     4207      +13     
    Misses        363      363              
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1648?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/url\_builder.py](https://codecov.io/gh/simonw/datasette/pull/1648/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3VybF9idWlsZGVyLnB5) | `100.00% <100.00%> (ø)` | |
  | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1648/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.97% <100.00%> (+0.10%)` | :arrow_up: |
  | [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1648/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `95.49% <100.00%> (ø)` | |
  | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1648/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `96.19% <100.00%> (ø)` | |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1648?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1648?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [7d24fd4...32548b8](https://codecov.io/gh/simonw/datasette/pull/1648?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-1059850369
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c4_LAiB
created_at: 2022-03-05T23:28:56Z
updated_at: 2022-03-05T23:28:56Z
author_association: OWNER
body:
  Lots of great conversations about the dash encoding implementation on Twitter: https://twitter.com/simonw/status/1500228316309061633

  @dracos helped me figure out a simpler regex: https://twitter.com/dracos/status/1500236433809973248

  `^/(?P<database>[^/]+)/(?P<table>[^\/\-\.]*|\-/|\-\.|\-\-)*(?P<format>\.\w+)?$`

  ![image](https://user-images.githubusercontent.com/9599/156903088-c01933ae-4713-4e91-8d71-affebf70b945.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-1059851259
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c4_LAv7
created_at: 2022-03-05T23:35:47Z
updated_at: 2022-03-05T23:35:59Z
author_association: OWNER
body:
  This [comment from glyph](https://twitter.com/glyph/status/1500244937312329730) got me thinking:

  > Have you considered replacing % with some other character and then using percent-encoding?

  What happens if a table name includes a `%` character and that ends up getting mangled by a misbehaving proxy?

  I should consider `%` in the escaping system too. And maybe go with that suggestion of using percent-encoding directly but with a different character.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-1059853526
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c4_LBTW
created_at: 2022-03-05T23:49:59Z
updated_at: 2022-03-05T23:49:59Z
author_association: OWNER
body:
  I want to try regular percentage encoding, except that it also encodes both the `-` and the `.` characters, AND it uses `-` instead of `%` as the encoding character.

  Should check what it does with emoji too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-1059854864
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c4_LBoQ
created_at: 2022-03-05T23:59:05Z
updated_at: 2022-03-05T23:59:05Z
author_association: OWNER
body:
  OK, for that percentage thing: the Python core implementation of URL percentage escaping deliberately ignores two of the characters we want to escape: `.` and `-`:

  https://github.com/python/cpython/blob/6927632492cbad86a250aa006c1847e03b03e70b/Lib/urllib/parse.py#L780-L783

  ```python
  _ALWAYS_SAFE = frozenset(b'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
                           b'abcdefghijklmnopqrstuvwxyz'
                           b'0123456789'
                           b'_.-~')
  ```
  It also defaults to skipping `/` (passed as a `safe=` parameter to various things).

  I'm going to try borrowing and modifying the core of the Python implementation: https://github.com/python/cpython/blob/6927632492cbad86a250aa006c1847e03b03e70b/Lib/urllib/parse.py#L795-L814
  ```python
  class _Quoter(dict):
      """A mapping from bytes numbers (in range(0,256)) to strings.
      String values are percent-encoded byte values, unless the key < 128, and
      in either of the specified safe set, or the always safe set.
      """
      # Keeps a cache internally, via __missing__, for efficiency (lookups
      # of cached keys don't call Python code at all).
      def __init__(self, safe):
          """safe: bytes object."""
          self.safe = _ALWAYS_SAFE.union(safe)

      def __repr__(self):
          return f"<Quoter {dict(self)!r}>"

      def __missing__(self, b):
          # Handle a cache miss. Store quoted string in cache and return.
          res = chr(b) if b in self.safe else '%{:02X}'.format(b)
          self[b] = res
          return res
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-1059855418
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c4_LBw6
created_at: 2022-03-06T00:00:53Z
updated_at: 2022-03-06T00:04:18Z
author_association: OWNER
body:
  ```python
  _ESCAPE_SAFE = frozenset(
      b'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
      b'abcdefghijklmnopqrstuvwxyz'
      b'0123456789_'
  )
  # I removed b'.-~')

  class Quoter(dict):
      # Keeps a cache internally, via __missing__
      def __missing__(self, b):
          # Handle a cache miss. Store quoted string in cache and return.
          res = chr(b) if b in _ESCAPE_SAFE else '-{:02X}'.format(b)
          self[b] = res
          return res

  quoter = Quoter().__getitem__

  ''.join([quoter(char) for char in b'foo/bar.csv'])
  # 'foo-2Fbar-2Ecsv'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-1059863997
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c4_LD29
created_at: 2022-03-06T00:57:57Z
updated_at: 2022-03-06T00:57:57Z
author_association: NONE
body:
  Probably too late… but I have just seen this because 
  http://simonwillison.net/2022/Mar/5/dash-encoding/#atom-everything

  And it reminded me of comma tools at W3C.
  http://www.w3.org/,tools

  Example, the text version of W3C homepage
  https://www.w3.org/,text


  > The challenge comes down to telling the difference between the following:
  > 
  >     * `/db/table` - an HTML table page

  `/db/table`

  >     * `/db/table.csv` - the CSV version of `/db/table`

  `/db/table,csv`

  >     * `/db/table.csv` - no this one is actually a database table called `table.csv`

  `/db/table.csv`

  >     * `/db/table.csv.csv` - the CSV version of `/db/table.csv`

  `/db/table.csv,csv`

  >     * `/db/table.csv.csv.csv` and so on...

  `/db/table.csv.csv,csv`


  I haven't checked all the cases in the thread.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-1059864154
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c4_LD5a
created_at: 2022-03-06T00:59:04Z
updated_at: 2022-03-06T00:59:04Z
author_association: OWNER
body:
  Needs more testing, but this seems to work for decoding the percent-escaped-with-dashes format: `urllib.parse.unquote(s.replace('-', '%'))`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-1059903309
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c4_LNdN
created_at: 2022-03-06T06:17:51Z
updated_at: 2022-03-06T06:17:51Z
author_association: OWNER
body:
  Suggestion from a conversation with Seth Michael Larson: it would be neat if plugins could easily integrate with whatever scheme this ends up using, maybe with the `/db/table/-/plugin-name` standardized pattern or similar.

  Making it easy for plugins to do the right, consistent thing is a good idea.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1648#issuecomment-1060016221
issue_url: https://api.github.com/repos/simonw/datasette/issues/1648
node_id: IC_kwDOBm6k_c4_LpBd
created_at: 2022-03-06T18:37:59Z
updated_at: 2022-03-06T18:37:59Z
author_association: OWNER
body:
  Change of plan: based on extensive conversations on Twitter - see https://github.com/simonw/datasette/issues/1439#issuecomment-1059851259 - I'm going to try a variant of this which is basically percent-encoding but with a hyphen instead of a percent symbol.

  Reason being that the current scheme doesn't handle the case of `%` being part of the table name, which could cause weird breakage due to some proxies decoding percent encoding before it gets to Datasette.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1649#issuecomment-1060021753
issue_url: https://api.github.com/repos/simonw/datasette/issues/1649
node_id: IC_kwDOBm6k_c4_LqX5
created_at: 2022-03-06T19:13:09Z
updated_at: 2022-03-06T19:13:09Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1649?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1649](https://codecov.io/gh/simonw/datasette/pull/1649?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (59b2c16) into [main](https://codecov.io/gh/simonw/datasette/commit/0499f174c063283aa9b589d475a32077aaf7adc5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0499f17) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1649/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1649?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1649   +/-   ##
  =======================================
    Coverage   92.03%   92.03%           
  =======================================
    Files          34       34           
    Lines        4557     4557           
  =======================================
    Hits         4194     4194           
    Misses        363      363           
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1649?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1649/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.86% <ø> (ø)` | |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1649?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1649?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [0499f17...59b2c16](https://codecov.io/gh/simonw/datasette/pull/1649?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1648#issuecomment-1060034562
issue_url: https://api.github.com/repos/simonw/datasette/issues/1648
node_id: IC_kwDOBm6k_c4_LtgC
created_at: 2022-03-06T20:36:12Z
updated_at: 2022-03-06T20:36:12Z
author_association: OWNER
body:
  Updated documentation: 
  ![image](https://user-images.githubusercontent.com/9599/156941171-89778c12-41bc-4951-97f2-ecc805025a53.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-1060044007
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c4_Lvzn
created_at: 2022-03-06T21:38:15Z
updated_at: 2022-03-06T21:38:15Z
author_association: OWNER
body:
  Test: https://github.com/simonw/datasette/blob/d2e3fe3facf0ed0abf8b00cd54463af90dd6904d/tests/test_utils.py#L651-L666

  One big advantage to this scheme is that redirecting old links to `%2F` pages (e.g. https://fivethirtyeight.datasettes.com/fivethirtyeight/twitter-ratio%2Fsenators) is easy - if you see a `%` in the `raw_path`, redirect to that page with the `%` replaced by `-`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1648#issuecomment-1060044592
issue_url: https://api.github.com/repos/simonw/datasette/issues/1648
node_id: IC_kwDOBm6k_c4_Lv8w
created_at: 2022-03-06T21:42:35Z
updated_at: 2022-03-06T21:42:35Z
author_association: OWNER
body:
  For consistency, I'm going to change how `?_next=` tokens work too. Right now they work like this:

  https://github.com/simonw/datasette/blob/de810f49cc57a4f88e4a1553d26c579253ce4531/datasette/views/table.py#L501-L507

  https://github.com/simonw/datasette/blob/de810f49cc57a4f88e4a1553d26c579253ce4531/datasette/utils/__init__.py#L114-L116

  I'm going to change those to use dash-encoding instead.

  I considered looking for `%` in those values and replacing that as `-` too, but since Datasette isn't 1.0 yet I'm going to risk breaking any pagination tokens that people might have saved away somewhere!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1648#issuecomment-1060056510
issue_url: https://api.github.com/repos/simonw/datasette/issues/1648
node_id: IC_kwDOBm6k_c4_Ly2-
created_at: 2022-03-06T23:02:05Z
updated_at: 2022-03-06T23:04:24Z
author_association: OWNER
body:
  Just spotted this: https://github.com/simonw/datasette/blob/de810f49cc57a4f88e4a1553d26c579253ce4531/datasette/views/base.py#L203-L216

  Maybe the db name should use dash encoding too?

  If so, relevant code includes this bit:

  https://github.com/simonw/datasette/blob/de810f49cc57a4f88e4a1553d26c579253ce4531/datasette/url_builder.py#L30-L38

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1648#issuecomment-1060065736
issue_url: https://api.github.com/repos/simonw/datasette/issues/1648
node_id: IC_kwDOBm6k_c4_L1HI
created_at: 2022-03-06T23:43:00Z
updated_at: 2022-03-06T23:43:11Z
author_association: OWNER
body:
  > * Maybe use dash encoding for database name too?

  Yes, I'm going to do this. At the moment if a DB file is called `fixx%tures.db` when you run it in Datasette the path is `/fix%2525tures` - which is liable to break.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1648#issuecomment-1060067031
issue_url: https://api.github.com/repos/simonw/datasette/issues/1648
node_id: IC_kwDOBm6k_c4_L1bX
created_at: 2022-03-06T23:50:40Z
updated_at: 2022-03-06T23:58:31Z
author_association: OWNER
body:
  I may have to do extra work here
  ```python
      def database(self, database, format=None):
          db = self.ds.databases[database]
          if self.ds.setting("hash_urls") and db.hash:
              path = self.path(
                  f"{dash_encode(database)}-{db.hash[:HASH_LENGTH]}", format=format
              )
          else:
              path = self.path(dash_encode(database), format=format)
          return path
  ```
  The URLs that incorporate a hash have a `dbname-hash` format - will that `-` in the middle there mess up the dash decoding mechanism? I think it will.

  Might be able to solve that like so:

  ```python
      async def resolve_db_name(self, request, db_name, **kwargs):
          hash = None
          name = None
          decoded_name = dash_decode(db_name)
          if decoded_name not in self.ds.databases and "-" in db_name:
              # No matching DB found, maybe it's a name-hash?
              name_bit, hash_bit = db_name.rsplit("-", 1)
              if dash_decode(name_bit) not in self.ds.databases:
                  raise NotFound(f"Database not found: {name}")
              else:
                  name = dash_decode(name_bit)
                  hash = hash_bit
          else:
              name = decoded_name
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1650#issuecomment-1060836262
issue_url: https://api.github.com/repos/simonw/datasette/issues/1650
node_id: IC_kwDOBm6k_c4_OxOm
created_at: 2022-03-07T15:52:09Z
updated_at: 2022-03-07T15:52:09Z
author_association: OWNER
body:
  This is a bit tricky.

  I tried this, sending a redirect only if a 404 happens:

  ```diff
  diff --git a/datasette/app.py b/datasette/app.py
  index 8c5480c..420664c 100644
  --- a/datasette/app.py
  +++ b/datasette/app.py
  @@ -1211,6 +1211,10 @@ class DatasetteRouter:
           return await self.handle_404(request, send)
   
       async def handle_404(self, request, send, exception=None):
  +        # If path contains % encoding, redirect to dash encoding
  +        if '%' in request.scope["path"]:
  +            await asgi_send_redirect(send, request.scope["path"].replace("%", "-"))
  +            return
           # If URL has a trailing slash, redirect to URL without it
           path = request.scope.get(
               "raw_path", request.scope["path"].encode("utf8")
  ```
  But this URL didn't work:

  - http://127.0.0.1:8001/fivethirtyeight/twitter-ratio%2Fsenators

  I was expecting that to redirect to this page:

  - http://127.0.0.1:8001/fivethirtyeight/twitter-2Dratio-2Fsenators

  But instead it took me to another 404:

  - http://127.0.0.1:8001/fivethirtyeight/twitter-ratio%2Fsenators

  This is because that URL contains both a %-escaped `/` AND a plain `-` - which was not escaped in the old system but is escaped in the new system.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1651#issuecomment-1060853226
issue_url: https://api.github.com/repos/simonw/datasette/issues/1651
node_id: IC_kwDOBm6k_c4_O1Xq
created_at: 2022-03-07T16:04:26Z
updated_at: 2022-03-07T16:04:26Z
author_association: OWNER
body:
  Here's the relevant code:

  https://github.com/simonw/datasette/blob/1baa030eca375f839f3471237547ab403523e643/datasette/utils/__init__.py#L753-L772

  https://github.com/simonw/datasette/blob/1baa030eca375f839f3471237547ab403523e643/datasette/views/base.py#L451-L479

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1650#issuecomment-1060863311
issue_url: https://api.github.com/repos/simonw/datasette/issues/1650
node_id: IC_kwDOBm6k_c4_O31P
created_at: 2022-03-07T16:13:17Z
updated_at: 2022-03-07T16:13:17Z
author_association: OWNER
body:
  This doesn't seem to work.

  https://latest.datasette.io/fixtures/table%2Fwith%2Fslashes.csv should be redirecting now that this is deployed - which it is, because https://latest.datasette.io/-/versions shows 644d25d1de78a36b105cca479e7b3e4375a6eadc - but I'm not getting that redirect.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1650#issuecomment-1060864823
issue_url: https://api.github.com/repos/simonw/datasette/issues/1650
node_id: IC_kwDOBm6k_c4_O4M3
created_at: 2022-03-07T16:14:33Z
updated_at: 2022-03-07T16:14:33Z
author_association: OWNER
body:
  Same problem here: https://fivethirtyeight.datasettes.com/fivethirtyeight/ahca-2Dpolls%2Fahca_polls should redirect but doesn't.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-1060870237
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c4_O5hd
created_at: 2022-03-07T16:19:22Z
updated_at: 2022-03-07T16:19:22Z
author_association: OWNER
body:
  I didn't need to do any of the fancy regular expression routing stuff after all, since the new dash encoding format avoids using `/` so a simple `[^/]+` can capture the correct segments from the URL.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1650#issuecomment-1061038414
issue_url: https://api.github.com/repos/simonw/datasette/issues/1650
node_id: IC_kwDOBm6k_c4_PilO
created_at: 2022-03-07T19:14:04Z
updated_at: 2022-03-07T19:14:04Z
author_association: OWNER
body:
  The problem seems to be that `http://127.0.0.1:8002/fixtures/table%2Fwith%2Fslashes.csv` doesn't result in a 404 at all. If it did, it would be redirected.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1650#issuecomment-1061041034
issue_url: https://api.github.com/repos/simonw/datasette/issues/1650
node_id: IC_kwDOBm6k_c4_PjOK
created_at: 2022-03-07T19:16:51Z
updated_at: 2022-03-07T19:16:51Z
author_association: OWNER
body:
  Here's the problem: https://github.com/simonw/datasette/blob/020effe47bf89f35182960a9645f2383a42ebd54/datasette/utils/__init__.py#L1173-L1175

  Which is called here:

  https://github.com/simonw/datasette/blob/1baa030eca375f839f3471237547ab403523e643/datasette/views/base.py#L469-L473

  So `table%2Fwith%2Fslashes` ends up decoded as if it was  using dash encoding.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1651#issuecomment-1061053094
issue_url: https://api.github.com/repos/simonw/datasette/issues/1651
node_id: IC_kwDOBm6k_c4_PmKm
created_at: 2022-03-07T19:29:01Z
updated_at: 2022-03-07T19:29:01Z
author_association: OWNER
body:
  I found an obscure bug in #1650 which I can fix with this too. The following test should pass:
  ```python
  @pytest.mark.parametrize(
      "path,expected",
      (
          (
              "/fivethirtyeight/twitter-ratio%2Fsenators",
              "/fivethirtyeight/twitter-2Dratio-2Fsenators",
          ),
          (
              "/fixtures/table%2Fwith%2Fslashes.csv",
              "/fixtures/table-2Fwith-2Fslashes-2Ecsv",
          ),
          # query string should be preserved
          ("/foo/bar%2Fbaz?id=5", "/foo/bar-2Fbaz?id=5"),
      ),
  )
  def test_redirect_percent_encoding_to_dash_encoding(app_client, path, expected):
      response = app_client.get(path)
      assert response.status == 302
      assert response.headers["location"] == expected
  ```
  It currently fails like this:
  ```
  >       assert response.headers["location"] == expected
  E       AssertionError: assert '/fixtures/table-2Fwith-2Fslashes.csv?_nofacet=1&_nocount=1' == '/fixtures/table-2Fwith-2Fslashes-2Ecsv'
  E         - /fixtures/table-2Fwith-2Fslashes-2Ecsv
  E         + /fixtures/table-2Fwith-2Fslashes.csv?_nofacet=1&_nocount=1
  ```
  Because the logic in that `get_format()` function notices that the table exists, and then weird things happen here: https://github.com/simonw/datasette/blob/1baa030eca375f839f3471237547ab403523e643/datasette/views/base.py#L288-L303

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1653#issuecomment-1061148807
issue_url: https://api.github.com/repos/simonw/datasette/issues/1653
node_id: IC_kwDOBm6k_c4_P9iH
created_at: 2022-03-07T21:21:23Z
updated_at: 2022-03-07T21:21:23Z
author_association: OWNER
body:
  This is currently blocked on the fact that Datasette doesn't have a mechanism for sorting by more than one column:
  - #197

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1653#issuecomment-1061150672
issue_url: https://api.github.com/repos/simonw/datasette/issues/1653
node_id: IC_kwDOBm6k_c4_P9_Q
created_at: 2022-03-07T21:23:39Z
updated_at: 2022-03-07T21:23:39Z
author_association: OWNER
body:
  There may be a short-term fix for this: table view could start accepting a `?_sort_sql=SQLfragment` parameter, similar to the `?_where=` parameter described here: https://docs.datasette.io/en/stable/json_api.html#special-table-arguments

  That fragment could then be pre-populated in `metadata`. Makes me think maybe that `?_where=` should be optionally settable in metadata too?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1651#issuecomment-1061169528
issue_url: https://api.github.com/repos/simonw/datasette/issues/1651
node_id: IC_kwDOBm6k_c4_QCl4
created_at: 2022-03-07T21:47:01Z
updated_at: 2022-03-07T21:47:01Z
author_association: OWNER
body:
  Wow, this code is difficult to follow! Look at this bit inside the `get_format()` method:

  https://github.com/simonw/datasette/blob/bb499942c15c4e2cfa4b6afab8f8debe5948c009/datasette/views/base.py#L469-L478

  That's modifying the arguments that were extracted from the path by the routing regular expressions to have `table` as ` dash-decoded value! So calling `.get_format()` has the side effect of decoding the table names for you. Nasty.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1651#issuecomment-1061170897
issue_url: https://api.github.com/repos/simonw/datasette/issues/1651
node_id: IC_kwDOBm6k_c4_QC7R
created_at: 2022-03-07T21:48:35Z
updated_at: 2022-03-07T21:48:35Z
author_association: OWNER
body:
  My attempts to simplify `get_format()` keep resulting in errors like this one:
  ```
    File "/Users/simon/Dropbox/Development/datasette/datasette/views/base.py", line 474, in view_get
      response_or_template_contexts = await self.data(
  TypeError: TableView.data() missing 1 required positional argument: 'table'
  ```
  I really need to clean this up.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1654#issuecomment-1061181089
issue_url: https://api.github.com/repos/simonw/datasette/issues/1654
node_id: IC_kwDOBm6k_c4_QFah
created_at: 2022-03-07T22:01:38Z
updated_at: 2022-03-07T22:01:38Z
author_association: OWNER
body:
  I'm going to use the [widely adopted](https://www.contributor-covenant.org/adopters/) Contributor Covenant:

  https://www.contributor-covenant.org/version/1/4/code-of-conduct/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1654#issuecomment-1061181530
issue_url: https://api.github.com/repos/simonw/datasette/issues/1654
node_id: IC_kwDOBm6k_c4_QFha
created_at: 2022-03-07T22:02:06Z
updated_at: 2022-03-07T22:02:06Z
author_association: OWNER
body:
  https://docs.github.com/en/communities/setting-up-your-project-for-healthy-contributions/adding-a-code-of-conduct-to-your-project says this should be called `CODE_OF_CONDUCT.md` in order for GitHub to pick it up.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1654#issuecomment-1061182132
issue_url: https://api.github.com/repos/simonw/datasette/issues/1654
node_id: IC_kwDOBm6k_c4_QFq0
created_at: 2022-03-07T22:02:43Z
updated_at: 2022-03-07T22:02:43Z
author_association: OWNER
body:
  Neat, GitHub have a template for this https://github.com/simonw/datasette/community/code-of-conduct/new?template=contributor-covenant

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1654#issuecomment-1061197133
issue_url: https://api.github.com/repos/simonw/datasette/issues/1654
node_id: IC_kwDOBm6k_c4_QJVN
created_at: 2022-03-07T22:19:35Z
updated_at: 2022-03-07T22:19:35Z
author_association: OWNER
body:
  Also now live on https://datasette.io

  ![CleanShot 2022-03-07 at 14 18 30@2x](https://user-images.githubusercontent.com/9599/157127424-805b3166-f0a8-4fac-be87-c055740af580.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1651#issuecomment-1061223822
issue_url: https://api.github.com/repos/simonw/datasette/issues/1651
node_id: IC_kwDOBm6k_c4_QP2O
created_at: 2022-03-07T22:54:54Z
updated_at: 2022-03-07T22:54:54Z
author_association: OWNER
body:
  I'm going to do a review of how URL routing works at the moment for the various views.

  I edited down [the full list](https://github.com/simonw/datasette/blob/c5791156d92615f25696ba93dae5bb2dcc192c98/datasette/app.py#L997-L1107) a bit - these are the most relevant:
  ```python
  add_route(IndexView.as_view(self), r"/(?P<as_format>(\.jsono?)?$)")
  add_route(
      DatabaseView.as_view(self),
      r"/(?P<db_name>[^/]+?)(?P<as_format>"
      + renderer_regex
      + r"|.jsono|\.csv)?$",
  )
  add_route(
      TableView.as_view(self),
      r"/(?P<db_name>[^/]+)/(?P<table_and_format>[^/]+?$)",
  )
  add_route(
      RowView.as_view(self),
      r"/(?P<db_name>[^/]+)/(?P<table>[^/]+?)/(?P<pk_path>[^/]+?)(?P<as_format>"
      + renderer_regex
      + r")?$",
  )
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/647#issuecomment-1061226942
issue_url: https://api.github.com/repos/simonw/datasette/issues/647
node_id: IC_kwDOBm6k_c4_QQm-
created_at: 2022-03-07T23:00:06Z
updated_at: 2022-03-07T23:00:06Z
author_association: OWNER
body:
  This needs to take into account the changes made here:
  - #1439

  In the new encoding scheme, `-` has a special meaning in a table name: https://docs.datasette.io/en/latest/internals.html#dash-encoding

  I think `~` is the right character to use to separate a database name from its hash. `~` should be a URL safe character according to Python's implementation of percent-encoding, see comment here:  https://github.com/simonw/datasette/blob/c5791156d92615f25696ba93dae5bb2dcc192c98/datasette/utils/__init__.py#L1146-L1152

  So the plugin could check for `dbname~hash` and react based on that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/647#issuecomment-1061267615
issue_url: https://api.github.com/repos/simonw/datasette/issues/647
node_id: IC_kwDOBm6k_c4_Qaif
created_at: 2022-03-08T00:05:43Z
updated_at: 2022-03-08T00:05:43Z
author_association: OWNER
body:
  Built a prototype of that plugin:
  ```python
  from datasette import hookimpl
  from functools import wraps


  @hookimpl
  def asgi_wrapper(datasette):
      def wrap_with_hashed_urls(app):
          @wraps(app)
          async def hashed_urls(scope, receive, send):
              # Only triggers on pages with a path not starting in /-/
              # and where the first page component matches a database name
              if scope.get("type") != "http":
                  await app(scope, receive, send)
                  return
              path = scope["path"].lstrip("/")
              if not path or path.startswith("-/"):
                  await app(scope, receive, send)
                  return
              potential_database = path.split("/")[0]
              # It may or may not be already dbname~hash
              if "~" in potential_database:
                  db_name, hash = potential_database.split("~", 1)
              else:
                  db_name = potential_database
                  hash = ""
              # Is db_name a database we have a hash for?
              try:
                  db = datasette.get_database(db_name)
              except KeyError:
                  await app(scope, receive, send)
                  return

              if db.hash is not None:
                  # TODO: make sure db.hash is documented
                  if db.hash[:7] != hash:
                      # Send a redirect
                      path_bits = path.split("/")
                      new_path = "/" + "/".join(["{}-{}".format(db_name, db.hash[:7])] + path_bits[1:])
                      if scope.get("query_string"):
                          new_path += "?" + scope["query_string"].decode("latin-1")

                      await send({
                          "type": "http.response.start",
                          "status": 302,
                          "headers": [
                              [b"location", new_path.encode("latin1")]
                          ],
                      })
                      await send({"type": "http.response.body", "body": b""})
                      return
                  else:
                      # Add a far-future cache header
                      async def wrapped_send(event):
                          if event["type"] == "http.response.start":
                              original_headers = event.get("headers") or []
                              event = {
                                  "type": event["type"],
                                  "status": event["status"],
                                  "headers": original_headers + [
                                      [b"Cache-Control", b"max-age=31536000"]
                                  ],
                              }
                          await send(event)

                      await app(scope, receive, wrapped_send)
                      return
                  
              await app(scope, receive, send)
          return hashed_urls
      return wrap_with_hashed_urls
  ```
  One catch: it doesn't affect the way URLs are generated - so every internal link within Datasette links to the non-hash version and then triggers a 302 redirect to the hashed version.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/647#issuecomment-1061272544
issue_url: https://api.github.com/repos/simonw/datasette/issues/647
node_id: IC_kwDOBm6k_c4_Qbvg
created_at: 2022-03-08T00:14:42Z
updated_at: 2022-03-08T00:14:42Z
author_association: OWNER
body:
  Maybe the plugin should interfere with `datasette.databases` on startup and change the registered name for each one?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/647#issuecomment-1061276399
issue_url: https://api.github.com/repos/simonw/datasette/issues/647
node_id: IC_kwDOBm6k_c4_Qcrv
created_at: 2022-03-08T00:21:47Z
updated_at: 2022-03-08T00:21:47Z
author_association: OWNER
body:
  This seems to do the job:
  ```python
  @hookimpl
  def startup(datasette):
      for name, database in datasette.databases.items():
          if database.hash:
              new_name = "{}_{}".format(name, database.hash[:7])
              del datasette.databases[name]
              datasette.databases[new_name] = database
  ```
  Would have to teach the rest of the plugin to split on `_` and to only redirect if the user seems to be hitting the URL for an old hash after which Datasette has been restarted with an updated database.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/647#issuecomment-1061282743
issue_url: https://api.github.com/repos/simonw/datasette/issues/647
node_id: IC_kwDOBm6k_c4_QeO3
created_at: 2022-03-08T00:32:34Z
updated_at: 2022-03-08T00:32:47Z
author_association: OWNER
body:
  It would be neat if the plugin could spot old-style hyphen hash URLs (maybe on 404) and redirect those too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1645#issuecomment-1061355871
issue_url: https://api.github.com/repos/simonw/datasette/issues/1645
node_id: IC_kwDOBm6k_c4_QwFf
created_at: 2022-03-08T02:59:28Z
updated_at: 2022-03-08T02:59:28Z
author_association: OWNER
body:
  Hah, found a TODO about this: https://github.com/simonw/datasette/blob/c5791156d92615f25696ba93dae5bb2dcc192c98/datasette/app.py#L997-L999

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1651#issuecomment-1061359915
issue_url: https://api.github.com/repos/simonw/datasette/issues/1651
node_id: IC_kwDOBm6k_c4_QxEr
created_at: 2022-03-08T03:08:14Z
updated_at: 2022-03-08T03:09:24Z
author_association: OWNER
body:
  A lot of the code complexity here is caused by `DataView` ([here](https://github.com/simonw/datasette/blob/c5791156d92615f25696ba93dae5bb2dcc192c98/datasette/views/base.py#L182-L669)), which has the logic for CSV streaming and plugin formats such that it can be shared between tables and custom queries.

  It would be good to get rid of that subclassed shared code, figure out how to do it via a utility function instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1384#issuecomment-1062124485
issue_url: https://api.github.com/repos/simonw/datasette/issues/1384
node_id: IC_kwDOBm6k_c4_TrvF
created_at: 2022-03-08T19:26:32Z
updated_at: 2022-03-08T19:26:32Z
author_association: NONE
body:
  Looks like I'm late to the party here, but wanted to join the convo if there's still time before this interface is solidified in v1.0. My plugin use case is for education / social science data, which is meta-data heavy in the documentation of measurement scales, instruments, collection procedures, etc. that I want to connect to columns, tables, and dbs (and render in static pages, but looks like I can do that with the jinja plugin hook). I'm still digging in and I think @brandonrobertz 's approach will work for me at least for now, but I want to bump this thread in the meantime -- are there still plans for an async metadata hook at some point in the future? (or are you considering other directions?)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1655#issuecomment-1062445113
issue_url: https://api.github.com/repos/simonw/datasette/issues/1655
node_id: IC_kwDOBm6k_c4_U6A5
created_at: 2022-03-09T01:01:24Z
updated_at: 2022-03-09T01:01:24Z
author_association: OWNER
body:
  https://labordata.bunkum.us/-/settings shows `max_returned_rows` had been increased to 5,000 for that instance - the default of 1,000 would help a bit here.

  Any thoughts on how Datasette could handle this kind of thing better?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1655#issuecomment-1062450649
issue_url: https://api.github.com/repos/simonw/datasette/issues/1655
node_id: IC_kwDOBm6k_c4_U7XZ
created_at: 2022-03-09T01:10:46Z
updated_at: 2022-03-09T01:10:46Z
author_association: CONTRIBUTOR
body:
  i increased the max_returned_row, because I have some scripts that get CSVs from this site, and this makes doing pagination of CSVs less annoying for many cases. i know that's streaming csvs is something you are hoping to address in 1.0. let me know if there's anything i can do to help with that.

  as for what if anything can be done about the size of the dom, I don't have any ideas right now, but i'll poke around.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/413#issuecomment-1065245831
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/413
node_id: IC_kwDOCGYnMM4_flyH
created_at: 2022-03-11T15:59:14Z
updated_at: 2022-03-11T15:59:14Z
author_association: OWNER
body:
  Hint from https://twitter.com/AdamChainz/status/1502311047612575745

  > Try:
  > 
  > `autodoc_typehints = 'description'`
  > 
  > For a list-of-arguments format
  > 
  > https://sphinx-doc.org/en/master/usage/extensions/autodoc.html#confval-autodoc_typehints

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/413#issuecomment-1065247619
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/413
node_id: IC_kwDOCGYnMM4_fmOD
created_at: 2022-03-11T16:01:20Z
updated_at: 2022-03-11T16:01:20Z
author_association: OWNER
body:
  Definitely an improvement!

  <img width="658" alt="image" src="https://user-images.githubusercontent.com/9599/157903133-999ee5f8-9f0c-44de-b0f5-817cad6b2651.png">

  It does highlight that I'm not currently using the `:param XXX: description` syntax though, which should move my descriptions of each parameter into that generated list.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/413#issuecomment-1065249883
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/413
node_id: IC_kwDOCGYnMM4_fmxb
created_at: 2022-03-11T16:03:35Z
updated_at: 2022-03-11T16:03:35Z
author_association: OWNER
body:
  Applying this change fixes that:
  ```diff
  diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py
  index 3bc528f..2a79711 100644
  --- a/sqlite_utils/db.py
  +++ b/sqlite_utils/db.py
  @@ -2293,18 +2293,18 @@ class Table(Queryable):
           """
           Apply conversion function ``fn`` to every value in the specified columns.
   
  -        - ``columns`` - a single column or list of string column names to convert.
  -        - ``fn`` - a callable that takes a single argument, ``value``, and returns it converted.
  -        - ``output`` - optional string column name to write the results to (defaults to the input column).
  -        - ``output_type`` - if the output column needs to be created, this is the type that will be used
  +        :param columns: a single column or list of string column names to convert.
  +        :param fn: a callable that takes a single argument, ``value``, and returns it converted.
  +        :param output: optional string column name to write the results to (defaults to the input column).
  +        :param output_type: if the output column needs to be created, this is the type that will be used
             for the new column.
  -        - ``drop`` - boolean, should the original column be dropped once the conversion is complete?
  -        - ``multi`` - boolean, if ``True`` the return value of ``fn(value)`` will be expected to be a
  +        :param drop: boolean, should the original column be dropped once the conversion is complete?
  +        :param multi: boolean, if ``True`` the return value of ``fn(value)`` will be expected to be a
             dictionary, and new columns will be created for each key of that dictionary.
  -        - ``where`` - a SQL fragment to use as a ``WHERE`` clause to limit the rows to which the conversion
  +        :param where: a SQL fragment to use as a ``WHERE`` clause to limit the rows to which the conversion
             is applied, for example ``age > ?`` or ``age > :age``.
  -        - ``where_args`` - a list of arguments (if using ``?``) or a dictionary (if using ``:age``).
  -        - ``show_progress`` - boolean, should a progress bar be displayed?
  +        :param where_args: a list of arguments (if using ``?``) or a dictionary (if using ``:age``).
  +        :param show_progress: boolean, should a progress bar be displayed?
   
           See :ref:`python_api_convert`.
           """
  ```
  <img width="671" alt="image" src="https://user-images.githubusercontent.com/9599/157903609-8b429b2b-9b9d-4be8-9eb7-bc4ab6315806.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/413#issuecomment-1065345515
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/413
node_id: IC_kwDOCGYnMM4_f-Hr
created_at: 2022-03-11T17:52:22Z
updated_at: 2022-03-11T17:52:22Z
author_association: OWNER
body:
  Well this is a huge improvement!

  https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Table.insert

  <img width="714" alt="image" src="https://user-images.githubusercontent.com/9599/157922673-3377cc75-d52a-4e86-acac-c5b5b6ddade6.png">

  I'm not crazy about the `extracts=<sqlite_utils.db.Default object>` thing though - I wonder if there's a neat way to customize that to be less verbose?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065377926
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/414
node_id: IC_kwDOCGYnMM4_gGCG
created_at: 2022-03-11T18:34:05Z
updated_at: 2022-03-11T18:34:05Z
author_association: OWNER
body:
  Two options:

  - Delete and recreate the release on GitHub, triggering it to be fixed on Read The Docs (as the `stable` version) - but cancel the push to PyPI, since that platform doesn't allow package versions to be over-written and in this case since the changelog file isn't included in the PyPI package there should be no change at all
  - Push a 3.25.2 release.

  I'm going to try and do that first option.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065379528
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/414
node_id: IC_kwDOCGYnMM4_gGbI
created_at: 2022-03-11T18:36:17Z
updated_at: 2022-03-11T18:36:17Z
author_association: OWNER
body:
  I created a new tag and release: https://github.com/simonw/sqlite-utils/releases/tag/3.25.1

  And I cancelled the publish workflow: https://github.com/simonw/sqlite-utils/actions/runs/1970415399

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065380286
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/414
node_id: IC_kwDOCGYnMM4_gGm-
created_at: 2022-03-11T18:37:23Z
updated_at: 2022-03-11T18:37:23Z
author_association: OWNER
body:
  On ReadTheDocs that triggered a new `stable` build but it didn't seem to trigger a new build of `3.25.1`: https://readthedocs.org/projects/sqlite-utils/builds/

  <img width="844" alt="image" src="https://user-images.githubusercontent.com/9599/157931210-80505416-509d-4c28-a894-b3f856ce0661.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065381047
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/414
node_id: IC_kwDOCGYnMM4_gGy3
created_at: 2022-03-11T18:38:27Z
updated_at: 2022-03-11T18:38:27Z
author_association: OWNER
body:
  OK that fixed it here: https://sqlite-utils.datasette.io/en/stable/changelog.html

  I'm going to trigger a rebuild of `3.25.1` too:

  <img width="881" alt="CleanShot 2022-03-11 at 10 37 33@2x" src="https://user-images.githubusercontent.com/9599/157931383-24682894-90be-4ede-b316-18efadbaf84b.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065382145
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/414
node_id: IC_kwDOCGYnMM4_gHEB
created_at: 2022-03-11T18:39:05Z
updated_at: 2022-03-11T18:39:05Z
author_association: OWNER
body:
  https://sqlite-utils.datasette.io/en/3.25.1/changelog.html is currently wrong:

  <img width="971" alt="image" src="https://user-images.githubusercontent.com/9599/157931484-479f04ea-700c-4daf-8b63-10f3d8d40a94.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065384183
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/414
node_id: IC_kwDOCGYnMM4_gHj3
created_at: 2022-03-11T18:40:39Z
updated_at: 2022-03-11T18:40:39Z
author_association: OWNER
body:
  That fixed it:

  <img width="971" alt="image" src="https://user-images.githubusercontent.com/9599/157931695-383a0454-127e-4978-8888-eeccae0b40ad.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065386352
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/411
node_id: IC_kwDOCGYnMM4_gIFw
created_at: 2022-03-11T18:41:37Z
updated_at: 2022-03-11T18:41:37Z
author_association: OWNER
body:
  I like `add-generated-column` - feels very clear to me, and is a nice place for adding logic that checks if the DB version supports it or not and shows a useful error.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065389386
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/411
node_id: IC_kwDOCGYnMM4_gI1K
created_at: 2022-03-11T18:42:53Z
updated_at: 2022-03-11T21:40:51Z
author_association: OWNER
body:
  The Python API could be:
  ```python
  db[table_name].add_generated_column("field", str, "json_extract(data, '$.field')", stored=True)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065402557
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/411
node_id: IC_kwDOCGYnMM4_gMC9
created_at: 2022-03-11T19:01:08Z
updated_at: 2022-03-11T21:42:25Z
author_association: OWNER
body:
  Just spotted this in https://www.sqlite.org/gencol.html

  > The only functional difference is that one cannot add new STORED columns using the [ALTER TABLE ADD COLUMN](https://www.sqlite.org/lang_altertable.html#altertabaddcol) command. Only VIRTUAL columns can be added using ALTER TABLE.

  So to add stored columns to an existing table we would need to use the `.transform()` trick. Which implies that this should actually be a capability of the various `.create()` methods, since transform works by creating a new table with those and then copying across the old data.

  Here's where `.transform()` calls `.create_table_sql()` under the hood:

  https://github.com/simonw/sqlite-utils/blob/9388edf57aa15719095e3cf0952c1653cd070c9b/sqlite_utils/db.py#L1627-L1637

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065440445
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/411
node_id: IC_kwDOCGYnMM4_gVS9
created_at: 2022-03-11T19:52:15Z
updated_at: 2022-03-11T19:52:15Z
author_association: OWNER
body:
  Two new parameters to `.create_table()` and friends:

   - `generated={...}` - generated column definitions
   - `generated_stored={...}` generated stored column definitions
   
  These columns will be added at the end of the table, but you can use the `column_order=` parameter to apply a different order.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065458729
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/411
node_id: IC_kwDOCGYnMM4_gZwp
created_at: 2022-03-11T19:58:50Z
updated_at: 2022-03-11T20:00:25Z
author_association: OWNER
body:
  I'm coming round to your suggestion to have this as extra arguments to `sqlite-utils add-column` now, especially since you also need to pass a column type.

  I'd like to come up with syntax for `sqlite-utils create-table` as well.

  https://sqlite-utils.datasette.io/en/stable/cli-reference.html#create-table

  Maybe extra `--generated-stored colname expression` (and `--generated`) options would work there.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065477258
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/411
node_id: IC_kwDOCGYnMM4_geSK
created_at: 2022-03-11T20:14:59Z
updated_at: 2022-03-11T20:14:59Z
author_association: CONTRIBUTOR
body:
  Good call on adding this to `create-table`, especially for stored columns. Having the stored/virtual split might make this tricky to implement, but I haven't gone any farther than thinking about what the CLI looks like. I'm going to try making the SQL side work first and figure that'll tell me more about what it needs.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065596417
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/411
node_id: IC_kwDOCGYnMM4_g7YB
created_at: 2022-03-11T22:30:15Z
updated_at: 2022-03-11T22:30:15Z
author_association: OWNER
body:
  I tried it out in Jupyter and it works as advertised:

  <img width="1076" alt="image" src="https://user-images.githubusercontent.com/9599/157980825-b362a162-b774-4fb5-8ec0-a7f539e62bd7.png">

  Introspection is a bit weird: there doesn't seem to be a way to introspect generated columns outside of parsing the stored SQL schema for the columns at the moment! And the `.columns` method doesn't return them at all:

  https://github.com/simonw/sqlite-utils/blob/433813612ff9b4b501739fd7543bef0040dd51fe/sqlite_utils/db.py#L1207-L1213

  Here's why:

  ```
  >>> db.execute("PRAGMA table_info('t')").fetchall()
  [(0, 'body', 'TEXT', 0, None, 0)]
  >>> db.execute("PRAGMA table_xinfo('t')").fetchall()
  [(0, 'body', 'TEXT', 0, None, 0, 0), (1, 'd', 'INT', 0, None, 0, 2)]
  ```
  So `table_xinfo()` is needed to get back columns including generated columns: https://www.sqlite.org/pragma.html#pragma_table_xinfo

  > **PRAGMA** *schema.***table_xinfo(***table-name***);**
  >
  > This pragma returns one row for each column in the named table, including [hidden columns](https://www.sqlite.org/vtab.html#hiddencol) in virtual tables. The output is the same as for [PRAGMA table_info](https://www.sqlite.org/pragma.html#pragma_table_info) except that hidden columns are shown rather than being omitted.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065597709
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/411
node_id: IC_kwDOCGYnMM4_g7sN
created_at: 2022-03-11T22:32:43Z
updated_at: 2022-03-11T22:32:43Z
author_association: OWNER
body:
  Trying to figure out what that extra field in `table_info` compared to `table_xinfo` is:
  ```
  >>> list(db.query("PRAGMA table_xinfo('t')"))
  [{'cid': 0,
    'name': 'body',
    'type': 'TEXT',
    'notnull': 0,
    'dflt_value': None,
    'pk': 0,
    'hidden': 0},
   {'cid': 1,
    'name': 'd',
    'type': 'INT',
    'notnull': 0,
    'dflt_value': None,
    'pk': 0,
    'hidden': 2}]
  ``
  Presumably `hidden` 0 v.s 2 v.s. other values has meaning.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1384#issuecomment-1065929510
issue_url: https://api.github.com/repos/simonw/datasette/issues/1384
node_id: IC_kwDOBm6k_c4_iMsm
created_at: 2022-03-12T17:49:59Z
updated_at: 2022-03-12T17:49:59Z
author_association: NONE
body:
  Ok, I'm taking a slightly different approach, which I think is sort of close to the in-memory _metadata table idea.

  I'm using a startup hook to load metadata / other info from the database, which I store in the datasette object for later:

  ```
  @hookimpl
  def startup(datasette):
      async def inner():
          datasette._mypluginmetadata = # await db query
      return inner
  ```

  Then, I can use this in other plugins:

  ```
  @hookimpl
  def render_cell(value, column, table, database, datasette):
      # use datasette._mypluginmetadata
  ```

  For my app I don't need anything to update dynamically so it's fine to pre-populate everything on startup. It's also good to have things precached especially for a hook like render_cell, which would otherwise require a ton of redundant db queries.

  Makes me wonder if we could take a sort of similar caching approach with the internal _metadata table. Like have a little watchdog that could query all of the attached dbs for their _metadata tables every 5min or so, which then could be merged into the in memory _metadata table which then could be accessed sync by the plugins, or something like that.

  For most the use cases I can think of, live updates don't need to take into effect immediately; refreshing a cache every 5min or on some other trigger (adjustable w a config setting) would be just fine. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1384#issuecomment-1065940779
issue_url: https://api.github.com/repos/simonw/datasette/issues/1384
node_id: IC_kwDOBm6k_c4_iPcr
created_at: 2022-03-12T18:49:29Z
updated_at: 2022-03-12T18:50:07Z
author_association: CONTRIBUTOR
body:
  Hello! Just wanted to chime in and note that there's a plugin to have Datasette [watch for updates to an external metadata.yaml/json and update the internal settings accordingly](https://datasette.io/plugins/datasette-remote-metadata), so I think the cache/poll use case is already covered. @khusmann If you don't need truly dynamic metadata then what you've come up with or the plugin ought to work fine.

  Making the get_metadata async won't improve the situation by itself as only some of the code paths accessing metadata use that hook. The other paths use the internal metadata dict. Trying to force all paths through a async hook would have performance ramifications and making everything use the internal meta will cause problems for users that need changes to take effect immediately. This is why I came to the non-async solution as it was the path of least change within Datasette. As always, open to new ideas, etc!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1384#issuecomment-1065951744
issue_url: https://api.github.com/repos/simonw/datasette/issues/1384
node_id: IC_kwDOBm6k_c4_iSIA
created_at: 2022-03-12T19:47:17Z
updated_at: 2022-03-12T19:47:17Z
author_association: NONE
body:
  Awesome, thanks @brandonrobertz !

  The plugin is close, but looks like it only grabs remote metadata, is that right? Instead what I'm wanting is to grab metadata embedded in the attached databases. Rather than extending that plugin, at this point I've realized I need a lot more flexibility in metadata for my data model (esp around formatting cell values and custom file exports) so rather than extending that I'll continue working on a plugin specific to my app.

  If I'm understanding your plugin code correctly, you query the db using the sync handle every time `get_metdata` is called, right? Won't this become a pretty big bottleneck if a hook into `render_cell` is trying to read metadata / plugin config?

  > Making the get_metadata async won't improve the situation by itself as only some of the code paths accessing metadata use that hook. The other paths use the internal metadata dict.

  I agree -- because things like `render_cell` will potentially want to read metadata/config, `get_metadata` should really remain sync and lightweight, which we can do with something like the remote-metadata plugin that could also poll metadata tables in attached databases.

  That leaves your app, where it sounds like you want changes made by the user in the browser in to be immediately reflected, rather than have to wait for the next metadata refresh. In this case I wonder if you could have your app make a sync write to the datasette object so the change would have the immediate effect, but then have a separate async polling mechanism to eventually write that change out to the database for long-term persistence. Then you'd have the best of both worlds, I think? But probably not worth the trouble if your use cases are small (and/or you're not reading metadata/config from tight loops like render_cell).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-1065987808
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c4_ia7g
created_at: 2022-03-13T00:02:32Z
updated_at: 2022-03-13T00:02:32Z
author_association: OWNER
body:
  OK, this has broken a lot more than I expected it would.

  Turns out `-` is a very common character in existing Datasette database names!

  https://datasette.io/-/databases for example has two:

  ```json
  [
      {
          "name": "docs-index",
          "path": "docs-index.db",
          "size": 1007616,
          "is_mutable": false,
          "is_memory": false,
          "hash": "0ac6c3de2762fcd174fd249fed8a8fa6046ea345173d22c2766186bf336462b2"
      },
      {
          "name": "dogsheep-index",
          "path": "dogsheep-index.db",
          "size": 5496832,
          "is_mutable": false,
          "is_memory": false,
          "hash": "d1ea238d204e5b9ae783c86e4af5bcdf21267c1f391de3e468d9665494ee012a"
      }
  ]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-1065988403
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c4_ibEz
created_at: 2022-03-13T00:06:38Z
updated_at: 2022-03-13T00:07:19Z
author_association: OWNER
body:
  If I want to reserve `-` as a character that CAN be used in URLs, the only remaining character that might make sense for escape sequences is `~` - based on this last line of characters that are escape from percentage encoding:

  ```python
  _ALWAYS_SAFE = frozenset(b'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
                           b'abcdefghijklmnopqrstuvwxyz'
                           b'0123456789'
                           b'_.-~')
  ```
  So I'd add both `-` and `_` back to the safe list, but use `~` to escape `.` and `/` and suchlike.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1384#issuecomment-1066006292
issue_url: https://api.github.com/repos/simonw/datasette/issues/1384
node_id: IC_kwDOBm6k_c4_ifcU
created_at: 2022-03-13T02:09:44Z
updated_at: 2022-03-13T02:09:44Z
author_association: CONTRIBUTOR
body:
  > If I'm understanding your plugin code correctly, you query the db using the sync handle every time `get_metdata` is called, right? Won't this become a pretty big bottleneck if a hook into `render_cell` is trying to read metadata / plugin config?

  Reading from sqlite DBs is pretty quick and I didn't notice significant performance issues when I was benchmarking. I tested on very large Datasette deployments (hundreds of DBs, millions of rows). See ["Many small queries are efficient in sqlite"](https://sqlite.org/np1queryprob.html) for more information on the rationale here. Also note that in the [datasette-live-config](https://github.com/next-LI/datasette-live-config) reference plugin, the DB connection is cached, so that eliminated most of the performance worries we had.

  If you need to ensure fresh metadata is being read inside of a `render_cell` hook specifically, you don't need to do anything further! `get_metadata` gets called before `render_cell` every request, so it already has access to the synced meta. There shouldn't be a need to call `get_metadata(...)` or `metadata(...)` inside `render_cell`, you can just use `datasette._metadata_local` if you're really worried about performance.

  > The plugin is close, but looks like it only grabs remote metadata, is that right? Instead what I'm wanting is to grab metadata embedded in the attached databases.

  Yes correct, the datadette-remote-metadata plugin doesn't do that. But the datasette-live-config plugin does. [It supports a `__metadata` table](https://github.com/next-LI/datasette-live-config/blob/main/datasette_live_config/__init__.py#L107-L138) that, when it exists on an attached DB, gets pulled into the Datasette internal `_metadata` and is also accessible via `get_metadata`. Updating is instantaneous so there's no gotchas for users or security issues for users relying on the metadata-based permissions. Simon talked about eventually making something like this a standard feature of Datasette, but I'm not sure what the status is on that!

  Good luck!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/408#issuecomment-1066139147
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/408
node_id: IC_kwDOCGYnMM4_i_4L
created_at: 2022-03-13T16:45:00Z
updated_at: 2022-03-13T16:54:09Z
author_association: NONE
body:
  @simonw 

  Now I get this:

  ```
  (app-root) sqlite-utils indexes global.db --table
  Error: near "(": syntax error
  (app-root) sqlite-utils --version
  sqlite-utils, version 3.25.1
  (app-root) sqlite3 --version
  3.36.0 2021-06-18 18:36:39
  (app-root) python --version
  Python 3.8.11
  ```


  Dockerfile
  ```
  FROM centos/python-38-centos7

  USER root

  RUN yum update -y
  RUN yum upgrade -y


  # epel
  RUN yum -y install epel-release && yum clean all

  # SQLite
  RUN yum -y install zlib-devel geos geos-devel proj proj-devel freexl freexl-devel libxml2-devel 

  WORKDIR /build/
  COPY sqlite-autoconf-3360000.tar.gz ./
  RUN tar -zxf sqlite-autoconf-3360000.tar.gz
  WORKDIR /build/sqlite-autoconf-3360000
  RUN ./configure
  RUN make
  RUN make install

  # 
  RUN /opt/app-root/bin/python3.8 -m pip install --upgrade pip
  RUN pip install sqlite-utils
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1384#issuecomment-1066143991
issue_url: https://api.github.com/repos/simonw/datasette/issues/1384
node_id: IC_kwDOBm6k_c4_jBD3
created_at: 2022-03-13T17:13:09Z
updated_at: 2022-03-13T17:13:09Z
author_association: NONE
body:
  Thanks for taking the time to reply @brandonrobertz , this is really helpful info.

  >  See "Many small queries are efficient in sqlite" for more information on the rationale here. Also note that in the datasette-live-config reference plugin, the DB connection is cached, so that eliminated most of the performance worries we had.

  Ah, that's nifty! Yeah, then caching on the python side is likely a waste :) I'm new to working with sqlite so this is super good to know the many-small-queries is a common pattern

  > I tested on very large Datasette deployments (hundreds of DBs, millions of rows).

  For my reference, did you include a `render_cell` plugin calling `get_metadata` in those tests? I'm less concerned now that I know a little more about sqlite's caching, but that special situation will jump you to a few orders of magnitude above what the sqlite article describes (e.g. 200 vs 20,000 queries+metadata merges for a page displaying 100 rows of a 200 column table). It wouldn't scale with db size as much as # of visible cells being rendered on the page, although they would be identical queries I suppose so will cache well.

  (If you didn't test this specific situation, no worries -- I'm just trying to calibrate my intuition on this and can do my own benchmarks at some point.)

  > Simon talked about eventually making something like this a standard feature of Datasette

  Yeah, getting metadata (and static pages as well for that matter) from internal tables definitely has my vote for including as a standard feature! Its really nice to be able to distribute a single *.db with all the metadata and static pages bundled. My metadata are sufficiently complex/domain specific that it makes sense to continue on my own plugin for now, but I'll be thinking about more general parts I can spin off as possible contributions to liveconfig (if you're open to them) or other plugins in this ecosystem.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1384#issuecomment-1066169718
issue_url: https://api.github.com/repos/simonw/datasette/issues/1384
node_id: IC_kwDOBm6k_c4_jHV2
created_at: 2022-03-13T19:48:49Z
updated_at: 2022-03-13T19:48:49Z
author_association: CONTRIBUTOR
body:
  > For my reference, did you include a `render_cell` plugin calling `get_metadata` in those tests?

  You shouldn't need to do this, as I mentioned previously. The code inside `render_cell` hook already has access to the most recently sync'd metadata via `datasette._metadata_local`. Refreshing the metadata for every cell seems ... excessive.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1384#issuecomment-1066194130
issue_url: https://api.github.com/repos/simonw/datasette/issues/1384
node_id: IC_kwDOBm6k_c4_jNTS
created_at: 2022-03-13T22:23:04Z
updated_at: 2022-03-13T22:23:04Z
author_association: NONE
body:
  Ah, sorry, I didn't get what you were saying you the first time. Using _metadata_local in that way makes total sense -- I agree, refreshing metadata each cell was seeming quite excessive. Now I'm on the same page! :)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1384#issuecomment-1066222323
issue_url: https://api.github.com/repos/simonw/datasette/issues/1384
node_id: IC_kwDOBm6k_c4_jULz
created_at: 2022-03-14T00:36:42Z
updated_at: 2022-03-14T00:36:42Z
author_association: CONTRIBUTOR
body:
  > Ah, sorry, I didn't get what you were saying you the first time. Using _metadata_local in that way makes total sense -- I agree, refreshing metadata each cell was seeming quite excessive. Now I'm on the same page! :)

  All good. Report back any issues you find with this stuff. Metadata/dynamic config hasn't been tested widely outside of what I've done AFAIK. If you find a strong use case for async meta, it's going to be better to know sooner rather than later!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1657#issuecomment-1067381556
issue_url: https://api.github.com/repos/simonw/datasette/issues/1657
node_id: IC_kwDOBm6k_c4_nvM0
created_at: 2022-03-14T22:57:27Z
updated_at: 2022-03-14T22:57:45Z
author_association: OWNER
body:
  The problem with the [dash encoding mechanism](https://simonwillison.net/2022/Mar/5/dash-encoding/) is that it turns out dashes are used in a LOT of existing Datasette instances - much of https://fivethirtyeight.datasettes.com/fivethirtyeight for example, and even https://datasette.io/ itself: https://datasette.io/dogsheep-index

  It's pretty ugly to force all of those to change to their dash-encoded equivalent - and in fact it broke https://datasette.io/ in a subtle way:

  - https://github.com/simonw/datasette.io/issues/94

  I'm going to try using `~` instead and see if that works as well and causes less breakage to existing sites.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1657#issuecomment-1067382232
issue_url: https://api.github.com/repos/simonw/datasette/issues/1657
node_id: IC_kwDOBm6k_c4_nvXY
created_at: 2022-03-14T22:58:47Z
updated_at: 2022-03-14T22:58:47Z
author_association: OWNER
body:
  Asked about this [on Twitter](https://twitter.com/simonw/status/1503499169775849473):

  > Anyone ever seen a proxy or other URL handling system do anything surprising with the tilde "~" character?
  >
  > I'm considering it as an escaping character, in place of "-" as described in

  Replies so far seem like it should be OK - Apache has supported this for home directories for a couple of decades now without any problems.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1657#issuecomment-1067413691
issue_url: https://api.github.com/repos/simonw/datasette/issues/1657
node_id: IC_kwDOBm6k_c4_n3C7
created_at: 2022-03-14T23:37:42Z
updated_at: 2022-03-14T23:37:42Z
author_association: OWNER
body:
  Relevant: https://datatracker.ietf.org/doc/html/rfc3986#section-2.1

  ```

        reserved    = gen-delims / sub-delims

        gen-delims  = ":" / "/" / "?" / "#" / "[" / "]" / "@"

        sub-delims  = "!" / "$" / "&" / "'" / "(" / ")"
                    / "*" / "+" / "," / ";" / "="
  ```
  Notably `~` is not in either of those lists.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1657#issuecomment-1067414156
issue_url: https://api.github.com/repos/simonw/datasette/issues/1657
node_id: IC_kwDOBm6k_c4_n3KM
created_at: 2022-03-14T23:38:41Z
updated_at: 2022-03-14T23:38:41Z
author_association: OWNER
body:
  And in https://datatracker.ietf.org/doc/html/rfc3986#section-2.3 "Unreserved Characters":

        unreserved  = ALPHA / DIGIT / "-" / "." / "_" / "~"

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1657#issuecomment-1067423720
issue_url: https://api.github.com/repos/simonw/datasette/issues/1657
node_id: IC_kwDOBm6k_c4_n5fo
created_at: 2022-03-14T23:59:56Z
updated_at: 2022-03-14T23:59:56Z
author_association: OWNER
body:
  Updated test:
  ```python
  @pytest.mark.parametrize(
      "original,expected",
      (
          ("abc", "abc"),
          ("/foo/bar", "~2Ffoo~2Fbar"),
          ("/-/bar", "~2F-~2Fbar"),
          ("-/db-/table.csv", "-~2Fdb-~2Ftable~2Ecsv"),
          (r"%~-/", "~25~7E-~2F"),
          ("~25~7E~2D~2F", "~7E25~7E7E~7E2D~7E2F"),
      ),
  )
  def test_tilde_encoding(original, expected):
      actual = utils.tilde_encode(original)
      assert actual == expected
      # And test round-trip
      assert original == utils.tilde_decode(actual)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/131#issuecomment-1067981656
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/131
node_id: IC_kwDOCGYnMM4_qBtY
created_at: 2022-03-15T13:21:42Z
updated_at: 2022-03-15T13:21:42Z
author_association: CONTRIBUTOR
body:
  Just ran into this issue last night. I have a big table that's _mostly_ numbers, but also a zip code column in a state where ZIP codes start with 0. Would be great to run something like this:

  ```sh
  sqlite-utils insert data.db places file.csv --csv --detect-types --type zipcode text
  ```

  Maybe I'll take a crack at this one.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1657#issuecomment-1068125636
issue_url: https://api.github.com/repos/simonw/datasette/issues/1657
node_id: IC_kwDOBm6k_c4_qk3E
created_at: 2022-03-15T15:30:54Z
updated_at: 2022-03-15T15:30:54Z
author_association: OWNER
body:
  I've made a real mess of this. I'm going to revert Datasette`main` back to the last commit that passed the tests and try this again in a branch.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1657#issuecomment-1068126821
issue_url: https://api.github.com/repos/simonw/datasette/issues/1657
node_id: IC_kwDOBm6k_c4_qlJl
created_at: 2022-03-15T15:31:54Z
updated_at: 2022-03-15T15:31:54Z
author_association: OWNER
body:
  The state I had got to prior to that revert is in https://github.com/simonw/datasette/tree/issue-1657-wip

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1657#issuecomment-1068148013
issue_url: https://api.github.com/repos/simonw/datasette/issues/1657
node_id: IC_kwDOBm6k_c4_qqUt
created_at: 2022-03-15T15:50:15Z
updated_at: 2022-03-15T15:50:15Z
author_association: OWNER
body:
  The thing that broke everything was this change:

  <img width="569" alt="image" src="https://user-images.githubusercontent.com/9599/158417416-285018c1-af45-49bc-9687-829c0c17ff05.png">

  I'm going to bring back the horrible `get_format()` method for the moment, with its weird mutations of the `args` object, then try and get rid of it again later.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1656#issuecomment-1068154183
issue_url: https://api.github.com/repos/simonw/datasette/issues/1656
node_id: IC_kwDOBm6k_c4_qr1H
created_at: 2022-03-15T15:55:34Z
updated_at: 2022-03-15T15:55:34Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1656?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1656](https://codecov.io/gh/simonw/datasette/pull/1656?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (5d9883f) into [main](https://codecov.io/gh/simonw/datasette/commit/c10cd48baf106659bf3f129ad7bfb2226be73821?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (c10cd48) will **not change** coverage.
  > The diff coverage is `n/a`.

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1656   +/-   ##
  =======================================
    Coverage   92.06%   92.06%           
  =======================================
    Files          34       34           
    Lines        4576     4576           
  =======================================
    Hits         4213     4213           
    Misses        363      363           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1656?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1656?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [c10cd48...5d9883f](https://codecov.io/gh/simonw/datasette/pull/1656?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1659#issuecomment-1068193035
issue_url: https://api.github.com/repos/simonw/datasette/issues/1659
node_id: IC_kwDOBm6k_c4_q1UL
created_at: 2022-03-15T16:28:25Z
updated_at: 2022-03-15T17:56:09Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1659?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1659](https://codecov.io/gh/simonw/datasette/pull/1659?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (85dde28) into [main](https://codecov.io/gh/simonw/datasette/commit/c10cd48baf106659bf3f129ad7bfb2226be73821?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (c10cd48) will **increase** coverage by `0.03%`.
  > The diff coverage is `100.00%`.

  > :exclamation: Current head 85dde28 differs from pull request most recent head 99b8263. Consider uploading reports for the commit 99b8263 to get more accurate results

  ```diff
  @@            Coverage Diff             @@
  ##             main    #1659      +/-   ##
  ==========================================
  + Coverage   92.06%   92.10%   +0.03%     
  ==========================================
    Files          34       34              
    Lines        4576     4584       +8     
  ==========================================
  + Hits         4213     4222       +9     
  + Misses        363      362       -1     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1659?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1659/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.36% <100.00%> (ø)` | |
  | [datasette/url\_builder.py](https://codecov.io/gh/simonw/datasette/pull/1659/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3VybF9idWlsZGVyLnB5) | `100.00% <100.00%> (ø)` | |
  | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1659/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.84% <100.00%> (-0.13%)` | :arrow_down: |
  | [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1659/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `96.07% <100.00%> (+0.58%)` | :arrow_up: |
  | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1659/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `96.21% <100.00%> (+0.01%)` | :arrow_up: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1659?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1659?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [c10cd48...99b8263](https://codecov.io/gh/simonw/datasette/pull/1659?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1657#issuecomment-1068306916
issue_url: https://api.github.com/repos/simonw/datasette/issues/1657
node_id: IC_kwDOBm6k_c4_rRHk
created_at: 2022-03-15T18:15:11Z
updated_at: 2022-03-15T18:15:11Z
author_association: OWNER
body:
  Now live here: https://fivethirtyeight.datasettes.com/fivethirtyeight/august-senate-polls~2Faugust_senate_polls

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1657#issuecomment-1068318454
issue_url: https://api.github.com/repos/simonw/datasette/issues/1657
node_id: IC_kwDOBm6k_c4_rT72
created_at: 2022-03-15T18:25:11Z
updated_at: 2022-03-15T18:25:11Z
author_association: OWNER
body:
  Demo:

  - https://latest.datasette.io/fixtures/table~2Fwith~2Fslashes~2Ecsv
  - https://latest.datasette.io/fixtures/table~2Fwith~2Fslashes~2Ecsv.csv
  - https://latest.datasette.io/fixtures/table~2Fwith~2Fslashes~2Ecsv.json

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1651#issuecomment-1068319530
issue_url: https://api.github.com/repos/simonw/datasette/issues/1651
node_id: IC_kwDOBm6k_c4_rUMq
created_at: 2022-03-15T18:25:42Z
updated_at: 2022-03-15T18:25:42Z
author_association: OWNER
body:
  Done:

  - https://latest.datasette.io/fixtures/table~2Fwith~2Fslashes~2Ecsv
  - https://latest.datasette.io/fixtures/table~2Fwith~2Fslashes~2Ecsv.csv
  - https://latest.datasette.io/fixtures/table~2Fwith~2Fslashes~2Ecsv.json

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1062#issuecomment-1068327874
issue_url: https://api.github.com/repos/simonw/datasette/issues/1062
node_id: IC_kwDOBm6k_c4_rWPC
created_at: 2022-03-15T18:33:49Z
updated_at: 2022-03-15T18:33:49Z
author_association: OWNER
body:
  I can get regular `.json` to stream too, using the pattern described in this TIL: https://til.simonwillison.net/python/output-json-array-streaming

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1660#issuecomment-1068415072
issue_url: https://api.github.com/repos/simonw/datasette/issues/1660
node_id: IC_kwDOBm6k_c4_rrhg
created_at: 2022-03-15T20:02:36Z
updated_at: 2022-03-15T20:02:36Z
author_association: OWNER
body:
  This is one of the worst bits - the `get_format()` method on the `DataView` base class actually modifies `args`, including removing keys! Really confusing: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L454-L482

  Then `BaseView` has some surprising responsibilities. It has a utility helper for checking multiple permissions at once:

  https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L81-L105

  And its own render method that adds extra stuff to the template context and handles the rel: alternate header:

  https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L131-L157

  Then `DataView` does all sorts of weird stuff - from handling database hashes (which I want to remove, see #647):

  https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L206-L219

  To streaming CSV responses: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L286-L308

  To handling SQLite exceptions: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L514-L526

  And a ton more. It' s a big mess.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1660#issuecomment-1068417357
issue_url: https://api.github.com/repos/simonw/datasette/issues/1660
node_id: IC_kwDOBm6k_c4_rsFN
created_at: 2022-03-15T20:05:08Z
updated_at: 2022-03-15T20:05:08Z
author_association: OWNER
body:
  `DataView` is used as the base class for:

  - `DatabaseView`
  - `DatabaseDownload` (just so the permissions checks can be called)
  - `QueryView` - which isn't routed to directly, it's called from `DatabaseView` if `?sql=` is available and `TableView` for canned queries
  - `RowTableShared` which is the base class for `TableView` and `RowView`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1510#issuecomment-1068443509
issue_url: https://api.github.com/repos/simonw/datasette/issues/1510
node_id: IC_kwDOBm6k_c4_ryd1
created_at: 2022-03-15T20:35:29Z
updated_at: 2022-03-15T20:35:29Z
author_association: OWNER
body:
  If I set a rule that everything available in the template context MUST also be available via the JSON API (maybe through an extras mechanism) I can combine this with API documentation and solve both at once.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1509#issuecomment-1068444767
issue_url: https://api.github.com/repos/simonw/datasette/issues/1509
node_id: IC_kwDOBm6k_c4_ryxf
created_at: 2022-03-15T20:37:03Z
updated_at: 2022-03-15T20:37:03Z
author_association: OWNER
body:
  Idea: I could add Pydantic https://pydantic-docs.helpmanual.io/usage/schema/ as an optional test dependency and use it to generate JSON schemas and run validation against examples in the API documentation.

  Maybe generate API documentation from it too?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1509#issuecomment-1068445412
issue_url: https://api.github.com/repos/simonw/datasette/issues/1509
node_id: IC_kwDOBm6k_c4_ry7k
created_at: 2022-03-15T20:37:50Z
updated_at: 2022-03-15T20:38:56Z
author_association: OWNER
body:
  ... maybe Datasette itself should include interactive API documentation, in addition to documenting it in the manual?

  `/dbname/table/-/apidocs` could return documentation about the specific table, taking into account columns and types.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/647#issuecomment-1068539404
issue_url: https://api.github.com/repos/simonw/datasette/issues/647
node_id: IC_kwDOBm6k_c4_sJ4M
created_at: 2022-03-15T22:49:01Z
updated_at: 2022-03-15T22:49:01Z
author_association: OWNER
body:
  I shipped the first version of this: https://github.com/simonw/datasette-hashed-urls

  Next step: test it with a live demo:
  - https://github.com/simonw/datasette-hashed-urls/issues/2

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/647#issuecomment-1068552696
issue_url: https://api.github.com/repos/simonw/datasette/issues/647
node_id: IC_kwDOBm6k_c4_sNH4
created_at: 2022-03-15T23:13:06Z
updated_at: 2022-03-15T23:13:06Z
author_association: OWNER
body:
  The plugin works. I'm going to implement one last feature for it:

  - https://github.com/simonw/datasette-hashed-urls/issues/3

  Then I can remove hashed URL mode in a separate issue.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1661#issuecomment-1068554827
issue_url: https://api.github.com/repos/simonw/datasette/issues/1661
node_id: IC_kwDOBm6k_c4_sNpL
created_at: 2022-03-15T23:16:58Z
updated_at: 2022-03-15T23:18:58Z
author_association: OWNER
body:
  If you attempt to use the [old setting](https://docs.datasette.io/en/stable/settings.html#hash-urls):

      datasette mydatabase.db --setting hash_urls 1

  It should error with a message saying that the feature has been moved to a plugin.

  I'll do this with a `deprecated_settings` mechanism so the error can be detected even though `datasette --help-settings` will no longer return the setting.

  https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/cli.py#L479-L489

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1661#issuecomment-1068628839
issue_url: https://api.github.com/repos/simonw/datasette/issues/1661
node_id: IC_kwDOBm6k_c4_sftn
created_at: 2022-03-16T01:21:36Z
updated_at: 2022-03-16T01:21:48Z
author_association: OWNER
body:
  From https://docs.datasette.io/en/0.60.2/performance.html#hashed-url-mode

  > You can enable these hashed URLs in two ways: using the [hash_urls](https://docs.datasette.io/en/0.60.2/settings.html#setting-hash-urls) configuration setting (which affects all requests to Datasette) or via the `?_hash=1` query string parameter (which only applies to the current request).

  I'm going to drop` ?_hash=1` entirely. I'd actually forgotten that feature existed!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1661#issuecomment-1068630353
issue_url: https://api.github.com/repos/simonw/datasette/issues/1661
node_id: IC_kwDOBm6k_c4_sgFR
created_at: 2022-03-16T01:24:56Z
updated_at: 2022-03-16T01:25:49Z
author_association: OWNER
body:
  Here's the only bit of code that references that `_hash` mechanism:

  https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L259-L265

  And here's the test:

  https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/tests/test_api.py#L828-L854

  Related issue:
  - #471

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1532#issuecomment-1069570893
issue_url: https://api.github.com/repos/simonw/datasette/issues/1532
node_id: IC_kwDOBm6k_c4_wFtN
created_at: 2022-03-16T20:11:41Z
updated_at: 2022-03-16T20:13:34Z
author_association: OWNER
body:
  Could also build a CLI Rich/Textual app to exercise the API - which could embed Datasette as a dependency and work using `datasette.client.get(...)` calls. 

  Could be a plugin that adds a `datasette tui` command.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1663#issuecomment-1071519407
issue_url: https://api.github.com/repos/simonw/datasette/issues/1663
node_id: IC_kwDOBm6k_c4_3hav
created_at: 2022-03-17T21:32:35Z
updated_at: 2022-03-17T21:32:35Z
author_association: OWNER
body:
  Updated docs:
  - https://docs.datasette.io/en/latest/internals.html#datasette-class
  - https://docs.datasette.io/en/latest/internals.html#db-hash

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1661#issuecomment-1071706993
issue_url: https://api.github.com/repos/simonw/datasette/issues/1661
node_id: IC_kwDOBm6k_c4_4PNx
created_at: 2022-03-17T22:42:21Z
updated_at: 2022-03-17T22:42:21Z
author_association: OWNER
body:
  As part of this I'm going to get rid of this mechanism: https://github.com/simonw/datasette/blob/30e5f0e67c38054a8087a2a4eae3fc4d1779af90/datasette/views/base.py#L170-L173

  Unwrapping `request.scope["url_route"]["kwargs"]` into keyword argument to view functions just made the code harder to follow.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1661#issuecomment-1071793307
issue_url: https://api.github.com/repos/simonw/datasette/issues/1661
node_id: IC_kwDOBm6k_c4_4kSb
created_at: 2022-03-17T23:17:32Z
updated_at: 2022-03-17T23:17:32Z
author_association: OWNER
body:
  Surprisingly I managed to break https://latest.datasette.io/fixtures/custom_foreign_key_label while working on this change:

  ![CleanShot 2022-03-17 at 16 16 54@2x](https://user-images.githubusercontent.com/9599/158909271-717b65e8-cfcc-44c4-b1cc-f34478b0f803.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1664#issuecomment-1071809988
issue_url: https://api.github.com/repos/simonw/datasette/issues/1664
node_id: IC_kwDOBm6k_c4_4oXE
created_at: 2022-03-17T23:24:57Z
updated_at: 2022-03-17T23:24:57Z
author_association: OWNER
body:
  My hunch is that this is broken because of this:

  https://github.com/simonw/datasette/blob/30e5f0e67c38054a8087a2a4eae3fc4d1779af90/datasette/app.py#L1098-L1107

  Note how the table uses `table_and_format` but the row uses just `table` - I think there's code that's getting confused by this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1664#issuecomment-1071813296
issue_url: https://api.github.com/repos/simonw/datasette/issues/1664
node_id: IC_kwDOBm6k_c4_4pKw
created_at: 2022-03-17T23:26:22Z
updated_at: 2022-03-17T23:26:22Z
author_association: OWNER
body:
  Probably caused by the convoluted code is `get_format()`: https://github.com/simonw/datasette/blob/30e5f0e67c38054a8087a2a4eae3fc4d1779af90/datasette/views/base.py#L466-L481

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1072833174
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/416
node_id: IC_kwDOCGYnMM4_8iKW
created_at: 2022-03-18T21:34:06Z
updated_at: 2022-03-18T21:34:06Z
author_association: OWNER
body:
  Good call-out: right now the `parsedate()` and `parsedatetime()` functions both terminate with an exception if they hit something invalid: https://sqlite-utils.datasette.io/en/stable/cli.html#sqlite-utils-convert-recipes

  It would be better if this was configurable by the user (and properly documented) - options could include "set null if date is invalid" and "leave the value as it is if invalid" in addition to throwing an error.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1072834273
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/416
node_id: IC_kwDOCGYnMM4_8ibh
created_at: 2022-03-18T21:36:05Z
updated_at: 2022-03-18T21:36:05Z
author_association: OWNER
body:
  Python's `str.encode()` method has a `errors=` parameter that does something along these lines: https://docs.python.org/3/library/stdtypes.html#str.encode

  > *errors* may be given to set a different error handling scheme. The default for *errors* is `'strict'`, meaning that encoding errors raise a [`UnicodeError`](https://docs.python.org/3/library/exceptions.html#UnicodeError "UnicodeError"). Other possible values are `'ignore'`, `'replace'`, `'xmlcharrefreplace'`, `'backslashreplace'` and any other name registered via [`codecs.register_error()`](https://docs.python.org/3/library/codecs.html#codecs.register_error "codecs.register_error"),

  Imitating this might be the way to go.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1664#issuecomment-1072890205
issue_url: https://api.github.com/repos/simonw/datasette/issues/1664
node_id: IC_kwDOBm6k_c4_8wFd
created_at: 2022-03-18T23:43:15Z
updated_at: 2022-03-18T23:43:15Z
author_association: OWNER
body:
  Now almost everything is working except for foreign key expansion:

  ![CleanShot 2022-03-18 at 16 41 39@2x](https://user-images.githubusercontent.com/9599/159097349-6f41dfdf-5bab-449b-a148-5cda3df6534c.png)

  Using the debugger I tracked it down to this code:

  https://github.com/simonw/datasette/blob/30e5f0e67c38054a8087a2a4eae3fc4d1779af90/datasette/views/table.py#L708-L715

  Turns out `default_labels` there is `None` - and it's a parameter to that `data()` method: https://github.com/simonw/datasette/blob/30e5f0e67c38054a8087a2a4eae3fc4d1779af90/datasette/views/table.py#L325-L334

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1664#issuecomment-1072890524
issue_url: https://api.github.com/repos/simonw/datasette/issues/1664
node_id: IC_kwDOBm6k_c4_8wKc
created_at: 2022-03-18T23:44:33Z
updated_at: 2022-03-19T00:06:51Z
author_association: OWNER
body:
  Looks like that was set here: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L490-L492

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1662#issuecomment-1072905467
issue_url: https://api.github.com/repos/simonw/datasette/issues/1662
node_id: IC_kwDOBm6k_c4_8zz7
created_at: 2022-03-19T00:42:23Z
updated_at: 2022-03-19T00:42:23Z
author_association: OWNER
body:
  Those client-side SQLite tricks are _really_ neat.

  `datasette publish` defaults to configuring it so the raw SQLite database can be downloaded from `/fixtures.db` - and this issue updated it to be served with a CORS header that would allow client-side scripts to load the file:

  - #1057

  If you're not going to run any server-side code at all you don't need Datasette for this - you can upload the SQLite database file to any static hosting with CORS headers and load it into the client that way.

  In terms of static publishing, I do think there's something interesting about using Datasette to generate static sites. There's an issue discussing options for that over here:

  - #1605

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1605#issuecomment-1072907200
issue_url: https://api.github.com/repos/simonw/datasette/issues/1605
node_id: IC_kwDOBm6k_c4_80PA
created_at: 2022-03-19T00:52:54Z
updated_at: 2022-03-19T00:53:45Z
author_association: OWNER
body:
  Had a thought about the implementation of this: it could make a really neat plugin.

  Something like `datasette-export` which adds a `export` command using https://docs.datasette.io/en/stable/plugin_hooks.html#register-commands-cli - then you could run:

      datasette export my-export-dir mydatabase.db -m metadata.json --template-dir templates/

  And the command would then:

  - Create a `Datasette()` instance with those databases/metadata/etc
  - Execute`await datasette.client.get("/")` to get the homepage HTML
  - Parse the HTML using BeautifulSoup to find all `a[href]`, `link[href]`, `script[src]`, `img[src]` elements that reference a relative path as opposed to one that starts with `http://`
  - Write out the homepage to `my-export-dir/index.html`
  - Recursively fetch and dump all of the other pages and assets that it found too

  All of that HTML parsing may be over-complicating things. It could alternatively accept options for which pages you want to export:

  ```
  datasette export my-export-dir \
    mydatabase.db -m metadata.json --template-dir templates/ \
    --path / \
    --path /mydatabase ...
  ```

  Or a really wild option: it could allow you to define the paths you want to export using a SQL query:

  ```
  datasette export my-export-dir \
    mydatabase.db -m metadata.json --template-dir templates/ \
    --sql "
  select '/' as path, 'index.html' as filename
    union all
  select '/mydatabase/articles/' || id as path, 'article-' || id || '.html' as filename
  from articles
    union all
  select '/mydatabase/tags/' || tag as path, 'tag-' || tag || '.html' as filename
  from tags
  "
  ```
  Which would save these files:
  - `index.html` as the content of `/`
  - `article-1.html` (and more) as the content of `/mydatabase/articles/1`
  - `tag-python.html` (and more) as the content of `/mydatabase/tags/python`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1228#issuecomment-1072907610
issue_url: https://api.github.com/repos/simonw/datasette/issues/1228
node_id: IC_kwDOBm6k_c4_80Va
created_at: 2022-03-19T00:55:29Z
updated_at: 2022-03-19T00:55:29Z
author_association: OWNER
body:
  It looks to me like something is causing the faceting query here to return a string when it was expected to return a number:

  https://github.com/simonw/datasette/blob/32963018e7edfab1233de7c7076c428d0e5c7813/datasette/facets.py#L153-L170

  I can't think of any way that  a `count(*) as n` would turn into a string though!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1228#issuecomment-1072908029
issue_url: https://api.github.com/repos/simonw/datasette/issues/1228
node_id: IC_kwDOBm6k_c4_80b9
created_at: 2022-03-19T00:57:54Z
updated_at: 2022-03-19T00:57:54Z
author_association: OWNER
body:
  Yes! That's the problem. I was able to replicate it like so:
  ```
  echo '[{                    
      "n": "one",
      "abc": 1
  }, {
      "n": "one",
      "abc": 2
  }, {
      "n": "two",
      "abc": 3
  }]' | sqlite-utils insert column-called-n.db t -
  ```
  <img width="564" alt="image" src="https://user-images.githubusercontent.com/9599/159100494-9a072e3c-7bad-4fc5-b90e-b5078c11fc44.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1666#issuecomment-1072933875
issue_url: https://api.github.com/repos/simonw/datasette/issues/1666
node_id: IC_kwDOBm6k_c4_86vz
created_at: 2022-03-19T04:03:42Z
updated_at: 2022-03-19T04:03:42Z
author_association: OWNER
body:
  Tests so far: https://github.com/simonw/datasette/blob/711767bcd3c1e76a0861fe7f24069ff1c8efc97a/tests/test_routes.py#L12-L34

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1561#issuecomment-1072939780
issue_url: https://api.github.com/repos/simonw/datasette/issues/1561
node_id: IC_kwDOBm6k_c4_88ME
created_at: 2022-03-19T04:45:40Z
updated_at: 2022-03-19T04:45:40Z
author_association: OWNER
body:
  I ended up moving hashed URL mode out to a plugin in:
  - #647

  If you're still interested in using it with `_memory` please open an issue in that repo here: https://github.com/simonw/datasette-hashed-urls

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1228#issuecomment-1072954795
issue_url: https://api.github.com/repos/simonw/datasette/issues/1228
node_id: IC_kwDOBm6k_c4_8_2r
created_at: 2022-03-19T06:44:40Z
updated_at: 2022-03-19T06:44:40Z
author_association: NONE
body:
  > ... unless your data had a column called `n`?

  Exactly, that's highly likely even though I can't double check from this computer just now. Thanks!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/878#issuecomment-1073037939
issue_url: https://api.github.com/repos/simonw/datasette/issues/878
node_id: IC_kwDOBm6k_c4_9UJz
created_at: 2022-03-19T16:19:30Z
updated_at: 2022-03-19T16:19:30Z
author_association: OWNER
body:
  On revisiting https://gist.github.com/simonw/281eac9c73b062c3469607ad86470eb2 a few months later I'm having second thoughts about using `@inject` on the `main()` method.

  But I still like the pattern as a way to resolve more complex cases like "to generate GeoJSON of the expanded view with labels, the label expansion code needs to run once at some before the GeoJSON formatting code does".

  So I'm going to stick with it a tiny bit longer, but maybe try to make it a lot more explicit when it's going to happen rather than having the main view methods themselves also use async DI.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1666#issuecomment-1073039670
issue_url: https://api.github.com/repos/simonw/datasette/issues/1666
node_id: IC_kwDOBm6k_c4_9Uk2
created_at: 2022-03-19T16:31:08Z
updated_at: 2022-03-19T16:31:57Z
author_association: OWNER
body:
  This does make it more interesting - it also highlights how inconsistent the way the capturing works is. Especially `as_format` which can be `None` or `""` or `.json` or `json` or not used at all in the case of `TableView`.

  https://github.com/simonw/datasette/blob/764738dfcb16cd98b0987d443f59d5baa9d3c332/tests/test_routes.py#L12-L36

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1667#issuecomment-1073040072
issue_url: https://api.github.com/repos/simonw/datasette/issues/1667
node_id: IC_kwDOBm6k_c4_9UrI
created_at: 2022-03-19T16:34:02Z
updated_at: 2022-03-19T16:34:02Z
author_association: OWNER
body:
  I called it `as_format` to avoid clashing with the Python built-in `format()` function when these things were turned into keyword arguments, but now that they're not I can use `format` instead.

  I think I'm going to go with `database`, `table`, `format` and `pks`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1667#issuecomment-1073042554
issue_url: https://api.github.com/repos/simonw/datasette/issues/1667
node_id: IC_kwDOBm6k_c4_9VR6
created_at: 2022-03-19T16:50:01Z
updated_at: 2022-03-19T16:52:35Z
author_association: OWNER
body:
  OK, I've made this more consistent - I still need to address the fact that `format` can be `.json` or `json` or not used at all before I close this issue.

  https://github.com/simonw/datasette/blob/61419388c134001118aaf7dfb913562d467d7913/tests/test_routes.py#L15-L35

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1668#issuecomment-1073043350
issue_url: https://api.github.com/repos/simonw/datasette/issues/1668
node_id: IC_kwDOBm6k_c4_9VeW
created_at: 2022-03-19T16:54:26Z
updated_at: 2022-03-19T16:54:26Z
author_association: OWNER
body:
  The `Database` class already has a `path` property but it means something else - it's the path to the `.db` file on disk:

  https://github.com/simonw/datasette/blob/61419388c134001118aaf7dfb913562d467d7913/datasette/database.py#L29-L50

  So need a different name for the path-that-is-used-in-the-URL.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1668#issuecomment-1073043713
issue_url: https://api.github.com/repos/simonw/datasette/issues/1668
node_id: IC_kwDOBm6k_c4_9VkB
created_at: 2022-03-19T16:56:19Z
updated_at: 2022-03-19T16:56:19Z
author_association: OWNER
body:
  Worth noting that the `name` right now is picked automatically to avoid conflicts:

  https://github.com/simonw/datasette/blob/61419388c134001118aaf7dfb913562d467d7913/datasette/app.py#L397-L413

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1668#issuecomment-1073073547
issue_url: https://api.github.com/repos/simonw/datasette/issues/1668
node_id: IC_kwDOBm6k_c4_9c2L
created_at: 2022-03-19T20:06:07Z
updated_at: 2022-03-19T20:06:07Z
author_association: OWNER
body:
  Implementing this is a little tricky because there's a whole lot of code that expects the `database` captured by the URL routing to be the name used to look up the database in `datasette.databases` - or via `.get_database()`.

  The `DataView.get()` method is a good example of the trickyness here. It even has code that dispatches out to plugin hooks that take `database` as a parameter.

  https://github.com/simonw/datasette/blob/61419388c134001118aaf7dfb913562d467d7913/datasette/views/base.py#L383-L555

  All the more reason to get rid of that `BaseView -> DataView -> TableView` hierarchy entirely:
  - #1660

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1668#issuecomment-1073075697
issue_url: https://api.github.com/repos/simonw/datasette/issues/1668
node_id: IC_kwDOBm6k_c4_9dXx
created_at: 2022-03-19T20:24:06Z
updated_at: 2022-03-19T20:24:06Z
author_association: OWNER
body:
  Right now if a database has a `.` in its name e.g. `fixtures.dot` the URL to that database is:

      /fixtures~2Edot

  But the output on `/-/databases` doesn't reflect that, it still shows the name with the dot.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1668#issuecomment-1073075913
issue_url: https://api.github.com/repos/simonw/datasette/issues/1668
node_id: IC_kwDOBm6k_c4_9dbJ
created_at: 2022-03-19T20:25:46Z
updated_at: 2022-03-19T20:26:08Z
author_association: OWNER
body:
  The output of `/.json` DOES use `path` to mean the URL path, not the path to the file on disk:

  ```
  {
    "fixtures.dot": {
      "name": "fixtures.dot",
      "hash": null,
      "color": "631f11",
      "path": "/fixtures~2Edot",
  ```
  So that's a problem already: having `db.path` refer to something different from that JSON is inconsistent.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1668#issuecomment-1073076110
issue_url: https://api.github.com/repos/simonw/datasette/issues/1668
node_id: IC_kwDOBm6k_c4_9deO
created_at: 2022-03-19T20:27:22Z
updated_at: 2022-03-19T20:27:22Z
author_association: OWNER
body:
  The docs do currently describe `path` as the filesystem path here: https://docs.datasette.io/en/stable/internals.html#database-class

  <img width="720" alt="image" src="https://user-images.githubusercontent.com/9599/159137373-5bbc7fea-7544-42b3-8423-be4e11eb4d52.png">

  Good thing I'm not at 1.0 yet so  I can change that!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1668#issuecomment-1073076136
issue_url: https://api.github.com/repos/simonw/datasette/issues/1668
node_id: IC_kwDOBm6k_c4_9deo
created_at: 2022-03-19T20:27:44Z
updated_at: 2022-03-19T20:27:44Z
author_association: OWNER
body:
  Pretty sure changing it will break some existing plugins though, including likely Datasette Desktop.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1668#issuecomment-1073076187
issue_url: https://api.github.com/repos/simonw/datasette/issues/1668
node_id: IC_kwDOBm6k_c4_9dfb
created_at: 2022-03-19T20:28:20Z
updated_at: 2022-03-19T20:28:20Z
author_association: OWNER
body:
  I'm going to keep `path` as the path to the file on disk. I'll pick a new name for what is currently `path` in that undocumented JSON API.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1667#issuecomment-1073076624
issue_url: https://api.github.com/repos/simonw/datasette/issues/1667
node_id: IC_kwDOBm6k_c4_9dmQ
created_at: 2022-03-19T20:31:44Z
updated_at: 2022-03-19T20:31:44Z
author_association: OWNER
body:
  I can now read `format` from `request.url_vars` and delete this code entirely: https://github.com/simonw/datasette/blob/b9c2b1cfc8692b9700416db98721fa3ec982f6be/datasette/views/base.py#L375-L381

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1668#issuecomment-1073097394
issue_url: https://api.github.com/repos/simonw/datasette/issues/1668
node_id: IC_kwDOBm6k_c4_9iqy
created_at: 2022-03-19T20:56:35Z
updated_at: 2022-03-19T20:56:35Z
author_association: OWNER
body:
  I'm trying to think if there's any reason not to use `route` for this. Would I possibly want to use that noun for something else in the future? I like it more than `route_path` because it has no underscore.

  Decision made: I'm going with `route`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1668#issuecomment-1073112104
issue_url: https://api.github.com/repos/simonw/datasette/issues/1668
node_id: IC_kwDOBm6k_c4_9mQo
created_at: 2022-03-19T21:08:21Z
updated_at: 2022-03-19T21:08:21Z
author_association: OWNER
body:
  I think I've got this working but I need to write a test for it that covers the rare case when the route is not the same thing as the database name.

  I'll do that with a new test.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/healthkit-to-sqlite/issues/14#issuecomment-1073123231
issue_url: https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/14
node_id: IC_kwDOC8tyDs4_9o-f
created_at: 2022-03-19T22:39:29Z
updated_at: 2022-03-19T22:39:29Z
author_association: NONE
body:
  I have this issue, too, with a fresh export. None of my `Workout` entries in `export.xml` have an `id` key, though [the sample `export.xml` in the tests folder doesn’t either](https://github.com/dogsheep/healthkit-to-sqlite/blob/main/tests/zip_contents/apple_health_export/export.xml#L14-L21), so I don’t think this is the culprit. Indeed, it seems @simonw is using the [`hash_id` function from `sqlite_utils`](https://sqlite-utils.datasette.io/en/stable/python-api.html#setting-an-id-based-on-the-hash-of-the-row-contents), which creates a column (`id`, in this case) based on a hash of the row’s contents.

  When I run the script, a `workouts` table is created, with one entry: my first workout. No `workout_points` table is created, as [I’d expect from `utils.py`](https://github.com/dogsheep/healthkit-to-sqlite/blob/main/healthkit_to_sqlite/utils.py#L89-L90). I then get essentially the same error as noted in this thread:

  ```Importing from HealthKit  [###################################-]   98%  00:00:01
  Traceback (most recent call last):
    File "/Users/lchski/.pyenv/versions/3.10.3/bin/healthkit-to-sqlite", line 8, in <module>
      sys.exit(cli())
    File "/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
      return self.main(*args, **kwargs)
    File "/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/click/core.py", line 1053, in main
      rv = self.invoke(ctx)
    File "/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/click/core.py", line 754, in invoke
      return __callback(*args, **kwargs)
    File "/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/healthkit_to_sqlite/cli.py", line 57, in cli
      convert_xml_to_sqlite(fp, db, progress_callback=bar.update, zipfile=zf)
    File "/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/healthkit_to_sqlite/utils.py", line 34, in convert_xml_to_sqlite
      workout_to_db(el, db, zipfile)
    File "/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/healthkit_to_sqlite/utils.py", line 57, in workout_to_db
      pk = db["workouts"].insert(record, alter=True, hash_id="id").last_pk
    File "/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/sqlite_utils/db.py", line 2822, in insert
      return self.insert_all(
    File "/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/sqlite_utils/db.py", line 2950, in insert_all
      self.insert_chunk(
    File "/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/sqlite_utils/db.py", line 2715, in insert_chunk
      result = self.db.execute(query, params)
    File "/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/sqlite_utils/db.py", line 458, in execute
      return self.conn.execute(sql, parameters)
  sqlite3.IntegrityError: UNIQUE constraint failed: workouts.id
  ```

  Are there maybe duplicate workouts in the data, which’d cause multiple rows to share the same `id`? It’s strange, though, that no `workout_points` is created at all. Export created from iOS 15.3.1.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1668#issuecomment-1073125334
issue_url: https://api.github.com/repos/simonw/datasette/issues/1668
node_id: IC_kwDOBm6k_c4_9pfW
created_at: 2022-03-19T22:53:55Z
updated_at: 2022-03-19T22:53:55Z
author_association: OWNER
body:
  Need to update documentation in a few places - e.g. https://docs.datasette.io/en/stable/internals.html#remove-database-name

  > This removes a database that has been previously added. `name=` is the unique name of that database, used in its URL path.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1668#issuecomment-1073126264
issue_url: https://api.github.com/repos/simonw/datasette/issues/1668
node_id: IC_kwDOBm6k_c4_9pt4
created_at: 2022-03-19T22:59:30Z
updated_at: 2022-03-19T22:59:30Z
author_association: OWNER
body:
  Also need to update the `datasette.urls` methods that construct the URL to a database/table/row - they take the database name but they need to know to look for the route.

  Need to add tests that check the links in the HTML and can confirm this is working correctly.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1668#issuecomment-1073134816
issue_url: https://api.github.com/repos/simonw/datasette/issues/1668
node_id: IC_kwDOBm6k_c4_9rzg
created_at: 2022-03-20T00:16:22Z
updated_at: 2022-03-20T00:16:22Z
author_association: OWNER
body:
  I'm going to add a `fixtures2.db` database which has that as the name but `alternative-route` as the route. I'll set that up using a custom plugin in the `plugins/` folder that gets deployed by https://github.com/simonw/datasette/blob/main/.github/workflows/deploy-latest.yml

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1668#issuecomment-1073135433
issue_url: https://api.github.com/repos/simonw/datasette/issues/1668
node_id: IC_kwDOBm6k_c4_9r9J
created_at: 2022-03-20T00:20:36Z
updated_at: 2022-03-20T00:20:36Z
author_association: OWNER
body:
  Building this plugin instantly revealed that all of the links - on the homepage and the database page and so on - are incorrect:
  ```python
  from datasette import hookimpl

  @hookimpl
  def startup(datasette):
      db = datasette.get_database("fixtures2")
      db.route = "alternative-route"
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1668#issuecomment-1073136686
issue_url: https://api.github.com/repos/simonw/datasette/issues/1668
node_id: IC_kwDOBm6k_c4_9sQu
created_at: 2022-03-20T00:31:13Z
updated_at: 2022-03-20T00:31:13Z
author_association: OWNER
body:
  That demo is now live:

  - https://latest.datasette.io/alternative-route
  - https://latest.datasette.io/alternative-route/attraction_characteristic

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1668#issuecomment-1073136896
issue_url: https://api.github.com/repos/simonw/datasette/issues/1668
node_id: IC_kwDOBm6k_c4_9sUA
created_at: 2022-03-20T00:33:23Z
updated_at: 2022-03-20T00:33:23Z
author_association: OWNER
body:
  I'm going to release this as a 0.61 alpha so I can more easily depend on it from `datasette-hashed-urls`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/healthkit-to-sqlite/issues/14#issuecomment-1073139067
issue_url: https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/14
node_id: IC_kwDOC8tyDs4_9s17
created_at: 2022-03-20T00:54:18Z
updated_at: 2022-03-20T00:54:18Z
author_association: NONE
body:
  Update: this appears to be because of running the command twice without clearing the DB in between. Tries to insert a Workout that already exists, causing a collision on the (auto-generated) `id` column. Had a different error with a clean DB, likely due to the workout points format; will make a new issue for that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/google-takeout-to-sqlite/issues/10#issuecomment-1073152522
issue_url: https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/10
node_id: IC_kwDODFE5qs4_9wIK
created_at: 2022-03-20T02:38:07Z
updated_at: 2022-03-20T02:38:07Z
author_association: NONE
body:
  [This line](https://github.com/dogsheep/google-takeout-to-sqlite/blob/e54e544427f1cc3ea8189f0e95f54046301a8645/google_takeout_to_sqlite/utils.py) needs to say `"MyActivity.json"` instead of `"My Activity.json"`. Google must have changed the file name.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/140#issuecomment-1073330388
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/140
node_id: IC_kwDOCGYnMM4_-bjU
created_at: 2022-03-20T19:44:39Z
updated_at: 2022-03-20T19:45:45Z
author_association: OWNER
body:
  Alternative idea for specifying types: accept a Python expression, then use Python type literal syntax. For example:

  ```
  sqlite-utils insert-files gifs.db images *.gif \
      -c path -c md5 -c last_modified:mtime \
      -a file_type '"gif"'
  ```

  Where `-a` indicates an additional column.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1660#issuecomment-1073355032
issue_url: https://api.github.com/repos/simonw/datasette/issues/1660
node_id: IC_kwDOBm6k_c4_-hkY
created_at: 2022-03-20T21:46:43Z
updated_at: 2022-03-20T21:46:43Z
author_association: OWNER
body:
  I think the way to get rid of most of the remaining complexity in `DataView` is to refactor how CSV stuff works - pulling it in line with other export factors and extracting the streaming mechanism. Opening a fresh issue for that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1672#issuecomment-1073355818
issue_url: https://api.github.com/repos/simonw/datasette/issues/1672
node_id: IC_kwDOBm6k_c4_-hwq
created_at: 2022-03-20T21:52:38Z
updated_at: 2022-03-20T21:52:38Z
author_association: OWNER
body:
  That means taking on these issues:

  - https://github.com/simonw/datasette/issues/1101
  - https://github.com/simonw/datasette/issues/1096
  - https://github.com/simonw/datasette/issues/1062

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1673#issuecomment-1073361986
issue_url: https://api.github.com/repos/simonw/datasette/issues/1673
node_id: IC_kwDOBm6k_c4_-jRC
created_at: 2022-03-20T22:31:41Z
updated_at: 2022-03-20T22:34:06Z
author_association: OWNER
body:
  Maybe it's because `supports_table_xinfo()` creates a brand new in-memory SQLite connection every time you call it?

  https://github.com/simonw/datasette/blob/798f075ef9b98819fdb564f9f79c78975a0f71e8/datasette/utils/sqlite.py#L22-L35

  Actually no, I'm caching that already:

  https://github.com/simonw/datasette/blob/798f075ef9b98819fdb564f9f79c78975a0f71e8/datasette/utils/sqlite.py#L12-L19

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1355#issuecomment-1073362979
issue_url: https://api.github.com/repos/simonw/datasette/issues/1355
node_id: IC_kwDOBm6k_c4_-jgj
created_at: 2022-03-20T22:38:53Z
updated_at: 2022-03-20T22:38:53Z
author_association: OWNER
body:
  Built a research prototype:
  ```diff
  diff --git a/datasette/app.py b/datasette/app.py
  index 5c8101a..5cd3e63 100644
  --- a/datasette/app.py
  +++ b/datasette/app.py
  @@ -1,6 +1,7 @@
   import asyncio
   import asgi_csrf
   import collections
  +import contextlib
   import datetime
   import functools
   import glob
  @@ -1490,3 +1491,11 @@ class DatasetteClient:
               return await client.request(
                   method, self._fix(path, avoid_path_rewrites), **kwargs
               )
  +
  +    @contextlib.asynccontextmanager
  +    async def stream(self, method, path, **kwargs):
  +        async with httpx.AsyncClient(app=self.app) as client:
  +            print("async with as client")
  +            async with client.stream(method, self._fix(path), **kwargs) as response:
  +                print("async with client.stream about to yield response")
  +                yield response
  diff --git a/datasette/cli.py b/datasette/cli.py
  index 3c6e1b2..3025ead 100644
  --- a/datasette/cli.py
  +++ b/datasette/cli.py
  @@ -585,11 +585,19 @@ def serve(
       asyncio.get_event_loop().run_until_complete(check_databases(ds))
   
       if get:
  -        client = TestClient(ds)
  -        response = client.get(get)
  -        click.echo(response.text)
  -        exit_code = 0 if response.status == 200 else 1
  -        sys.exit(exit_code)
  +
  +        async def _run_get():
  +            print("_run_get")
  +            async with ds.client.stream("GET", get) as response:
  +                print("Got response:", response)
  +                async for chunk in response.aiter_bytes(chunk_size=1024):
  +                    print("   chunk")
  +                    sys.stdout.buffer.write(chunk)
  +                    sys.stdout.buffer.flush()
  +                exit_code = 0 if response.status_code == 200 else 1
  +                sys.exit(exit_code)
  +
  +        asyncio.get_event_loop().run_until_complete(_run_get())
           return
   
       # Start the server
  ```
  But for some reason it didn't appear to stream out the response - it would print this out:
  ```
  % datasette covid.db --get '/covid/ny_times_us_counties.csv?_size=10&_stream=on'
  _run_get
  async with as client
  ```
  And then hang. I would expect it to start printing out chunks of CSV data here, but instead it looks like it waited for everything to be generated before returning anything to the console.

  No idea why. I dropped this for the moment.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1674#issuecomment-1073366436
issue_url: https://api.github.com/repos/simonw/datasette/issues/1674
node_id: IC_kwDOBm6k_c4_-kWk
created_at: 2022-03-20T22:58:40Z
updated_at: 2022-03-20T22:58:40Z
author_association: OWNER
body:
  This will probably happen as part of turning this into an officially documented API that serves the template context for the homepage:
  - #1510

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1510#issuecomment-1073366630
issue_url: https://api.github.com/repos/simonw/datasette/issues/1510
node_id: IC_kwDOBm6k_c4_-kZm
created_at: 2022-03-20T22:59:33Z
updated_at: 2022-03-20T22:59:33Z
author_association: OWNER
body:
  I really like the idea of making this effectively the same thing as the fully documented, stable JSON API that comes as part of 1.0. If you want to know what will be available to your templates, consult the API documentation.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073448904
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/416
node_id: IC_kwDOCGYnMM4_-4fI
created_at: 2022-03-21T03:28:12Z
updated_at: 2022-03-21T03:30:37Z
author_association: OWNER
body:
  Generating a test database using a pattern from https://www.geekytidbits.com/date-range-table-sqlite/
  ```
  sqlite-utils create-database test-dates.db
  sqlite-utils create-table test-dates.db dates id integer date text --pk id
  sqlite-utils test-dates.db "WITH RECURSIVE
    cnt(x) AS (
       SELECT 0
       UNION ALL
       SELECT x+1 FROM cnt
        LIMIT (SELECT ((julianday('2016-04-01') - julianday('2016-03-15'))) + 1)
    )
  insert into dates (date) select date(julianday('2016-03-15'), '+' || x || ' days') as date FROM cnt;"
  ```
  After running that:
  ```
  % sqlite-utils rows test-dates.db dates
  [{"id": 1, "date": "2016-03-15"},
   {"id": 2, "date": "2016-03-16"},
   {"id": 3, "date": "2016-03-17"},
   {"id": 4, "date": "2016-03-18"},
   {"id": 5, "date": "2016-03-19"},
   {"id": 6, "date": "2016-03-20"},
   {"id": 7, "date": "2016-03-21"},
   {"id": 8, "date": "2016-03-22"},
   {"id": 9, "date": "2016-03-23"},
   {"id": 10, "date": "2016-03-24"},
   {"id": 11, "date": "2016-03-25"},
   {"id": 12, "date": "2016-03-26"},
   {"id": 13, "date": "2016-03-27"},
   {"id": 14, "date": "2016-03-28"},
   {"id": 15, "date": "2016-03-29"},
   {"id": 16, "date": "2016-03-30"},
   {"id": 17, "date": "2016-03-31"},
   {"id": 18, "date": "2016-04-01"}]
  ```
  Then to make one of them invalid:

      sqlite-utils test-dates.db "update dates set date = '//' where id = 10"

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073450588
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/416
node_id: IC_kwDOCGYnMM4_-45c
created_at: 2022-03-21T03:32:58Z
updated_at: 2022-03-21T03:32:58Z
author_association: OWNER
body:
  Then I ran this to convert `2016-03-27` etc to `2016/03/27` so I could see which ones were later converted:

      sqlite-utils convert test-dates.db dates date 'value.replace("-", "/")'

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073451659
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/416
node_id: IC_kwDOCGYnMM4_-5KL
created_at: 2022-03-21T03:35:01Z
updated_at: 2022-03-21T03:35:01Z
author_association: OWNER
body:
  I confirmed that if it fails for any value ALL values are left alone, since it runs in a transaction.

  Here's the code that does that:

  https://github.com/simonw/sqlite-utils/blob/433813612ff9b4b501739fd7543bef0040dd51fe/sqlite_utils/db.py#L2523-L2526

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073453230
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/416
node_id: IC_kwDOCGYnMM4_-5iu
created_at: 2022-03-21T03:40:37Z
updated_at: 2022-03-21T03:40:37Z
author_association: OWNER
body:
  I think the options here should be:

  - On error, raise an exception and revert the transaction (the current default)
  - On error, leave the value as-is
  - On error, set the value to `None`

  These need to be indicated by parameters to the `r.parsedate()` function.

  Some design options:

  - `ignore=True` to ignore errors - but how does it know if it should leave the value or set it to `None`? This is similar to other `ignore=True` parameters elsewhere in the Python API.
  - `errors="ignore"`, `errors="set-null"` - I don't like magic string values very much, but this is similar to Python's `str.encode(errors=)` mechanism
  - `errors=r.IGNORE` - using constants, which at least avoids magic strings. The other one could be `errors=r.SET_NULL`
  - `error=lambda v: None` or `error=lambda v: v` - this is a bit confusing though, introducing another callback that gets to have a go at converting the error if the first callback failed? And what happens if that lambda itself raises an error?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073455905
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/416
node_id: IC_kwDOCGYnMM4_-6Mh
created_at: 2022-03-21T03:44:47Z
updated_at: 2022-03-21T03:45:00Z
author_association: OWNER
body:
  This is quite nice:
  ```
  % sqlite-utils convert test-dates.db dates date "r.parsedate(value, errors=r.IGNORE)"
    [####################################]  100%
  % sqlite-utils rows test-dates.db dates                                           
  [{"id": 1, "date": "2016-03-15"},
   {"id": 2, "date": "2016-03-16"},
   {"id": 3, "date": "2016-03-17"},
   {"id": 4, "date": "2016-03-18"},
   {"id": 5, "date": "2016-03-19"},
   {"id": 6, "date": "2016-03-20"},
   {"id": 7, "date": "2016-03-21"},
   {"id": 8, "date": "2016-03-22"},
   {"id": 9, "date": "2016-03-23"},
   {"id": 10, "date": "//"},
   {"id": 11, "date": "2016-03-25"},
   {"id": 12, "date": "2016-03-26"},
   {"id": 13, "date": "2016-03-27"},
   {"id": 14, "date": "2016-03-28"},
   {"id": 15, "date": "2016-03-29"},
   {"id": 16, "date": "2016-03-30"},
   {"id": 17, "date": "2016-03-31"},
   {"id": 18, "date": "2016-04-01"}]
  % sqlite-utils convert test-dates.db dates date "r.parsedate(value, errors=r.SET_NULL)"
    [####################################]  100%
  % sqlite-utils rows test-dates.db dates                                                
  [{"id": 1, "date": "2016-03-15"},
   {"id": 2, "date": "2016-03-16"},
   {"id": 3, "date": "2016-03-17"},
   {"id": 4, "date": "2016-03-18"},
   {"id": 5, "date": "2016-03-19"},
   {"id": 6, "date": "2016-03-20"},
   {"id": 7, "date": "2016-03-21"},
   {"id": 8, "date": "2016-03-22"},
   {"id": 9, "date": "2016-03-23"},
   {"id": 10, "date": null},
   {"id": 11, "date": "2016-03-25"},
   {"id": 12, "date": "2016-03-26"},
   {"id": 13, "date": "2016-03-27"},
   {"id": 14, "date": "2016-03-28"},
   {"id": 15, "date": "2016-03-29"},
   {"id": 16, "date": "2016-03-30"},
   {"id": 17, "date": "2016-03-31"},
   {"id": 18, "date": "2016-04-01"}]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073456155
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/416
node_id: IC_kwDOCGYnMM4_-6Qb
created_at: 2022-03-21T03:45:37Z
updated_at: 2022-03-21T03:45:37Z
author_association: OWNER
body:
  Prototype:
  ```diff
  diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py
  index 8255b56..0a3693e 100644
  --- a/sqlite_utils/cli.py
  +++ b/sqlite_utils/cli.py
  @@ -2583,7 +2583,11 @@ def _generate_convert_help():
       """
       ).strip()
       recipe_names = [
  -        n for n in dir(recipes) if not n.startswith("_") and n not in ("json", "parser")
  +        n
  +        for n in dir(recipes)
  +        if not n.startswith("_")
  +        and n not in ("json", "parser")
  +        and callable(getattr(recipes, n))
       ]
       for name in recipe_names:
           fn = getattr(recipes, name)
  diff --git a/sqlite_utils/recipes.py b/sqlite_utils/recipes.py
  index 6918661..569c30d 100644
  --- a/sqlite_utils/recipes.py
  +++ b/sqlite_utils/recipes.py
  @@ -1,17 +1,38 @@
   from dateutil import parser
   import json
   
  +IGNORE = object()
  +SET_NULL = object()
   
  -def parsedate(value, dayfirst=False, yearfirst=False):
  +
  +def parsedate(value, dayfirst=False, yearfirst=False, errors=None):
       "Parse a date and convert it to ISO date format: yyyy-mm-dd"
  -    return (
  -        parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).date().isoformat()
  -    )
  +    try:
  +        return (
  +            parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst)
  +            .date()
  +            .isoformat()
  +        )
  +    except parser.ParserError:
  +        if errors is IGNORE:
  +            return value
  +        elif errors is SET_NULL:
  +            return None
  +        else:
  +            raise
   
   
  -def parsedatetime(value, dayfirst=False, yearfirst=False):
  +def parsedatetime(value, dayfirst=False, yearfirst=False, errors=None):
       "Parse a datetime and convert it to ISO datetime format: yyyy-mm-ddTHH:MM:SS"
  -    return parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).isoformat()
  +    try:
  +        return parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).isoformat()
  +    except parser.ParserError:
  +        if errors is IGNORE:
  +            return value
  +        elif errors is SET_NULL:
  +            return None
  +        else:
  +            raise
   
   
   def jsonsplit(value, delimiter=",", type=str):
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/415#issuecomment-1073468996
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/415
node_id: IC_kwDOCGYnMM4_-9ZE
created_at: 2022-03-21T04:14:42Z
updated_at: 2022-03-21T04:14:42Z
author_association: OWNER
body:
  I can fix this like so:
  ```
  % sqlite-utils convert demo.db demo foo '{"foo": "bar"}' --multi --dry-run
  abc
   --- becomes:
  {"foo": "bar"}

  Would affect 1 row
  ```
  Diff is this:
  ```diff
  diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py
  index 0cf0468..b2a0440 100644
  --- a/sqlite_utils/cli.py
  +++ b/sqlite_utils/cli.py
  @@ -2676,7 +2676,10 @@ def convert(
           raise click.ClickException(str(e))
       if dry_run:
           # Pull first 20 values for first column and preview them
  -        db.conn.create_function("preview_transform", 1, lambda v: fn(v) if v else v)
  +        preview = lambda v: fn(v) if v else v
  +        if multi:
  +            preview = lambda v: json.dumps(fn(v), default=repr) if v else v
  +        db.conn.create_function("preview_transform", 1, preview)
           sql = """
               select
                   [{column}] as value,
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/526#issuecomment-1074019047
issue_url: https://api.github.com/repos/simonw/datasette/issues/526
node_id: IC_kwDOBm6k_c5ABDrn
created_at: 2022-03-21T15:09:56Z
updated_at: 2022-03-21T15:09:56Z
author_association: OWNER
body:
  I should research how much overhead creating a new connection costs - it may be that an easy way to solve this is to create A dedicated connection for the query and then close that connection at the end.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1660#issuecomment-1074136176
issue_url: https://api.github.com/repos/simonw/datasette/issues/1660
node_id: IC_kwDOBm6k_c5ABgRw
created_at: 2022-03-21T16:38:46Z
updated_at: 2022-03-21T16:38:46Z
author_association: OWNER
body:
  I'm going to refactor this stuff out and document it so it can be easily used by plugins:

  https://github.com/simonw/datasette/blob/4a4164b81191dec35e423486a208b05a9edc65e4/datasette/views/base.py#L69-L103

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1675#issuecomment-1074141457
issue_url: https://api.github.com/repos/simonw/datasette/issues/1675
node_id: IC_kwDOBm6k_c5ABhkR
created_at: 2022-03-21T16:44:09Z
updated_at: 2022-03-21T16:44:09Z
author_association: OWNER
body:
  A slightly odd thing about these methods is that they either fail silently or they raise a `Forbidden` exception.

  Maybe they should instead return `True` or `False` and the calling code could decide if it wants to raise the exception? That would make them more usable and a little less surprising.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1675#issuecomment-1074142617
issue_url: https://api.github.com/repos/simonw/datasette/issues/1675
node_id: IC_kwDOBm6k_c5ABh2Z
created_at: 2022-03-21T16:45:27Z
updated_at: 2022-03-21T16:45:27Z
author_association: OWNER
body:
  Though at that point `check_permission` is such a light wrapper around `self.ds.permission_allowed()` that there's little point in it existing at all.

  So maybe `check_permisions()` becomes `ds.permissions_allowed()`.

  `permission_allowed()` v.s. `permissions_allowed()` is a bit of a subtle naming difference, but I think it works.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1675#issuecomment-1074143209
issue_url: https://api.github.com/repos/simonw/datasette/issues/1675
node_id: IC_kwDOBm6k_c5ABh_p
created_at: 2022-03-21T16:46:05Z
updated_at: 2022-03-21T16:46:05Z
author_association: OWNER
body:
  The other difference though is that `ds.permission_allowed(...)` works against an actor, while `check_permission()` works against a request (though just to access `request.actor`).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1675#issuecomment-1074156779
issue_url: https://api.github.com/repos/simonw/datasette/issues/1675
node_id: IC_kwDOBm6k_c5ABlTr
created_at: 2022-03-21T16:55:08Z
updated_at: 2022-03-21T16:56:02Z
author_association: OWNER
body:
  One benefit of the current design of `check_permissions` that raises an exception is that the exception includes information on WHICH of the permission checks failed. Returning just `True` or `False` loses that information.

  I could return an object which evaluates to `False` but also carries extra information? Bit weird, I've never seen anything like that in other Python code.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1675#issuecomment-1074158890
issue_url: https://api.github.com/repos/simonw/datasette/issues/1675
node_id: IC_kwDOBm6k_c5ABl0q
created_at: 2022-03-21T16:57:15Z
updated_at: 2022-03-21T16:57:15Z
author_association: OWNER
body:
  Idea: `ds.permission_allowed()` continues to just return `True` or `False`.

  A new `ds.ensure_permissions(...)` method is added which raises a `Forbidden` exception if a check fails (hence the different name)`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1675#issuecomment-1074161523
issue_url: https://api.github.com/repos/simonw/datasette/issues/1675
node_id: IC_kwDOBm6k_c5ABmdz
created_at: 2022-03-21T16:59:55Z
updated_at: 2022-03-21T17:00:03Z
author_association: OWNER
body:
  Also calling that function `permissions_allowed()` is confusing because there is a plugin hook with a similar name already: https://docs.datasette.io/en/stable/plugin_hooks.html#permission-allowed-datasette-actor-action-resource

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1675#issuecomment-1074177827
issue_url: https://api.github.com/repos/simonw/datasette/issues/1675
node_id: IC_kwDOBm6k_c5ABqcj
created_at: 2022-03-21T17:14:31Z
updated_at: 2022-03-21T17:14:31Z
author_association: OWNER
body:
  Updated documentation: https://github.com/simonw/datasette/blob/e627510b760198ccedba9e5af47a771e847785c9/docs/internals.rst#await-ensure_permissionsactor-permissions

  > This method allows multiple permissions to be checked at onced. It raises a `datasette.Forbidden` exception if any of the checks are denied before one of them is explicitly granted.
  > 
  > This is useful when you need to check multiple permissions at once. For example, an actor should be able to view a table if either one of the following checks returns `True` or not a single one of them returns `False`:

  That's pretty hard to understand! I'm going to open a separate issue to reconsider if this is a useful enough abstraction given how confusing it is.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1676#issuecomment-1074178865
issue_url: https://api.github.com/repos/simonw/datasette/issues/1676
node_id: IC_kwDOBm6k_c5ABqsx
created_at: 2022-03-21T17:15:27Z
updated_at: 2022-03-21T17:15:27Z
author_association: OWNER
body:
  This method here: https://github.com/simonw/datasette/blob/e627510b760198ccedba9e5af47a771e847785c9/datasette/app.py#L632-L664

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1676#issuecomment-1074180312
issue_url: https://api.github.com/repos/simonw/datasette/issues/1676
node_id: IC_kwDOBm6k_c5ABrDY
created_at: 2022-03-21T17:16:45Z
updated_at: 2022-03-21T17:16:45Z
author_association: OWNER
body:
  When looking at this code earlier I assumed that the following would check each permission in turn and fail if any of them failed:
  ```python
  await self.ds.ensure_permissions(
      request.actor,
      [
          ("view-table", (database, table)),
          ("view-database", database),
          "view-instance",
      ]
  )
  ```
  But it's not quite that simple: if any of them fail, it fails... but if an earlier one returns `True` the whole stack passes even if there would have been a failure later on!

  If that is indeed the right abstraction, I need to work to make the documentation as clear as possible.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1677#issuecomment-1074184240
issue_url: https://api.github.com/repos/simonw/datasette/issues/1677
node_id: IC_kwDOBm6k_c5ABsAw
created_at: 2022-03-21T17:20:17Z
updated_at: 2022-03-21T17:20:17Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/blob/e627510b760198ccedba9e5af47a771e847785c9/datasette/views/base.py#L69-L77

  This is weirdly different from how `check_permissions()` used to work, in that it doesn't differentiate between `None` and `False`.

  https://github.com/simonw/datasette/blob/4a4164b81191dec35e423486a208b05a9edc65e4/datasette/views/base.py#L79-L103

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/417#issuecomment-1074243540
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/417
node_id: IC_kwDOCGYnMM5AB6fU
created_at: 2022-03-21T18:08:03Z
updated_at: 2022-03-21T18:08:03Z
author_association: OWNER
body:
  I've not really thought about standards as much here as I should. It looks like there are two competing specs for newline-delimited JSON!

  http://ndjson.org/ is the one I've been using in `sqlite-utils` - and https://github.com/ndjson/ndjson-spec#31-serialization says:

  > The JSON texts MUST NOT contain newlines or carriage returns.

  https://jsonlines.org/ is the other one. It is slightly less clear, but it does say this:

  > 2. Each Line is a Valid JSON Value
  >
  > The most common values will be objects or arrays, but any JSON value is permitted.

  My interpretation of both of these is that newlines in the middle of a JSON object shouldn't be allowed.

  So what's `jq` doing here? It looks to me like that `jq` format is its own thing - it's not actually compatible with either of those two loose specs described above.

  The `jq` docs seem to call this "whitespace-separated JSON": https://stedolan.github.io/jq/manual/v1.6/#Invokingjq

  The thing I like about newline-delimited JSON is that it's really trivial to parse - loop through each line, run it through `json.loads()` and that's it. No need to try and unwrap JSON objects that might span multiple lines.

  Unless someone has written a robust Python implementation of a `jq`-compatible whitespace-separated JSON parser, I'm inclined to leave this as is. I'd be fine adding some documentation that helps point people towards `jq -c` though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/417#issuecomment-1074256603
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/417
node_id: IC_kwDOCGYnMM5AB9rb
created_at: 2022-03-21T18:19:41Z
updated_at: 2022-03-21T18:19:41Z
author_association: NONE
body:
  That makes sense; just a little hint that points folks towards doing the right thing might be helpful!

  fwiw, the reason I was using jq in the first place was just a quick way to extract one attribute from an actual JSON array. When I initially imported it, I got a table with a bunch of embedded JSON values, rather than a native table, because each array entry had two attributes, one with the data I _actually_ wanted. Not sure how common a use-case this is, though (and easily fixed, aside from the jq weirdness!)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1678#issuecomment-1074302559
issue_url: https://api.github.com/repos/simonw/datasette/issues/1678
node_id: IC_kwDOBm6k_c5ACI5f
created_at: 2022-03-21T19:04:03Z
updated_at: 2022-03-21T19:04:03Z
author_association: OWNER
body:
  Documentation: https://docs.datasette.io/en/latest/internals.html#await-check-visibility-actor-action-resource-none

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1660#issuecomment-1074321862
issue_url: https://api.github.com/repos/simonw/datasette/issues/1660
node_id: IC_kwDOBm6k_c5ACNnG
created_at: 2022-03-21T19:19:01Z
updated_at: 2022-03-21T19:19:01Z
author_association: OWNER
body:
  I've simplified this a ton now. I'm going to keep working on this in the long-term but I think this issue can be closed.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1679#issuecomment-1074331743
issue_url: https://api.github.com/repos/simonw/datasette/issues/1679
node_id: IC_kwDOBm6k_c5ACQBf
created_at: 2022-03-21T19:30:05Z
updated_at: 2022-03-21T19:30:05Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/blob/1a7750eb29fd15dd2eea3b9f6e33028ce441b143/datasette/app.py#L118-L122 sets it to 50ms for facet suggestion but that's not going to pass `ms < 50`:

  ```python
      Setting(
          "facet_suggest_time_limit_ms",
          50,
          "Time limit for calculating a suggested facet",
      ),
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1679#issuecomment-1074337997
issue_url: https://api.github.com/repos/simonw/datasette/issues/1679
node_id: IC_kwDOBm6k_c5ACRjN
created_at: 2022-03-21T19:37:08Z
updated_at: 2022-03-21T19:37:08Z
author_association: OWNER
body:
  This is weird:
  ```python
  import sqlite3

  db = sqlite3.connect(":memory:")

  i = 0

  def count():
      global i
      i += 1


  db.set_progress_handler(count, 1)

  db.execute("""
  with recursive counter(x) as (
    select 0
      union
    select x + 1 from counter
  )
  select * from counter limit 10000;
  """)

  print(i)
  ```
  Outputs `24`. But if you try the same thing in the SQLite console:
  ```
  sqlite> .stats vmstep
  sqlite> with recursive counter(x) as (
     ...>   select 0
     ...>     union
     ...>   select x + 1 from counter
     ...> )
     ...> select * from counter limit 10000;
  ...
  VM-steps: 200007
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1679#issuecomment-1074341924
issue_url: https://api.github.com/repos/simonw/datasette/issues/1679
node_id: IC_kwDOBm6k_c5ACSgk
created_at: 2022-03-21T19:42:08Z
updated_at: 2022-03-21T19:42:08Z
author_association: OWNER
body:
  Here's the Python-C implementation of `set_progress_handler`: https://github.com/python/cpython/blob/4674fd4e938eb4a29ccd5b12c15455bd2a41c335/Modules/_sqlite/connection.c#L1177-L1201

  It calls `sqlite3_progress_handler(self->db, n, progress_callback, ctx);`

  https://www.sqlite.org/c3ref/progress_handler.html says:

  > The parameter N is the approximate number of [virtual machine instructions](https://www.sqlite.org/opcode.html) that are evaluated between successive invocations of the callback X

  So maybe VM-steps and virtual machine instructions are different things?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1679#issuecomment-1074439309
issue_url: https://api.github.com/repos/simonw/datasette/issues/1679
node_id: IC_kwDOBm6k_c5ACqSN
created_at: 2022-03-21T21:28:58Z
updated_at: 2022-03-21T21:28:58Z
author_association: OWNER
body:
  David Raymond solved it there: https://sqlite.org/forum/forumpost/330c8532d8a88bcd

  > Don't forget to step through the results. All .execute() has done is prepared it.
  >
  >     db.execute(query).fetchall()

  Sure enough, adding that gets the VM steps number up to 190,007 which is close enough that I'm happy.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1679#issuecomment-1074446576
issue_url: https://api.github.com/repos/simonw/datasette/issues/1679
node_id: IC_kwDOBm6k_c5ACsDw
created_at: 2022-03-21T21:38:27Z
updated_at: 2022-03-21T21:38:27Z
author_association: OWNER
body:
  OK here's a microbenchmark script:
  ```python
  import sqlite3
  import timeit

  db = sqlite3.connect(":memory:")
  db_with_progress_handler_1 = sqlite3.connect(":memory:")
  db_with_progress_handler_1000 = sqlite3.connect(":memory:")

  db_with_progress_handler_1.set_progress_handler(lambda: None, 1)
  db_with_progress_handler_1000.set_progress_handler(lambda: None, 1000)

  def execute_query(db):
      cursor = db.execute("""
      with recursive counter(x) as (
      select 0
          union
      select x + 1 from counter
      )
      select * from counter limit 10000;
      """)
      list(cursor.fetchall())


  print("Without progress_handler")
  print(timeit.timeit(lambda: execute_query(db), number=100))

  print("progress_handler every 1000 ops")
  print(timeit.timeit(lambda: execute_query(db_with_progress_handler_1000), number=100))

  print("progress_handler every 1 op")
  print(timeit.timeit(lambda: execute_query(db_with_progress_handler_1), number=100))
  ```
  Results:
  ```
  % python3 bench.py
  Without progress_handler
  0.8789225700311363
  progress_handler every 1000 ops
  0.8829826560104266
  progress_handler every 1 op
  2.8892734259716235
  ```

  So running every 1000 ops makes almost no difference at all, but running every single op is a 3.2x performance degradation.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1679#issuecomment-1074454687
issue_url: https://api.github.com/repos/simonw/datasette/issues/1679
node_id: IC_kwDOBm6k_c5ACuCf
created_at: 2022-03-21T21:48:02Z
updated_at: 2022-03-21T21:48:02Z
author_association: OWNER
body:
  Here's another microbenchmark that measures how many nanoseconds it takes to run 1,000 vmops:

  ```python
  import sqlite3
  import time

  db = sqlite3.connect(":memory:")

  i = 0
  out = []

  def count():
      global i
      i += 1000
      out.append(((i, time.perf_counter_ns())))

  db.set_progress_handler(count, 1000)

  print("Start:", time.perf_counter_ns())
  all = db.execute("""
  with recursive counter(x) as (
    select 0
      union
    select x + 1 from counter
  )
  select * from counter limit 10000;
  """).fetchall()
  print("End:", time.perf_counter_ns())

  print()
  print("So how long does it take to execute 1000 ops?")

  prev_time_ns = None
  for i, time_ns in out:
      if prev_time_ns is not None:
          print(time_ns - prev_time_ns, "ns")
      prev_time_ns = time_ns
  ```
  Running it:
  ```
  % python nanobench.py
  Start: 330877620374821
  End: 330877632515822

  So how long does it take to execute 1000 ops?
  47290 ns
  49573 ns
  48226 ns
  45674 ns
  53238 ns
  47313 ns
  52346 ns
  48689 ns
  47092 ns
  87596 ns
  69999 ns
  52522 ns
  52809 ns
  53259 ns
  52478 ns
  53478 ns
  65812 ns
  ```
  87596ns is 0.087596ms - so even a measure rate of every 1000 ops is easily finely grained enough to capture differences of less than 0.1ms.

  If anything I could bump that default 1000 up - and I can definitely eliminate the `if ms < 50` branch entirely.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1679#issuecomment-1074458506
issue_url: https://api.github.com/repos/simonw/datasette/issues/1679
node_id: IC_kwDOBm6k_c5ACu-K
created_at: 2022-03-21T21:53:47Z
updated_at: 2022-03-21T21:53:47Z
author_association: OWNER
body:
  Oh interesting, it turns out there is ONE place in the code that sets the `ms` to less than 20 - this test fixture: https://github.com/simonw/datasette/blob/4e47a2d894b96854348343374c8e97c9d7055cf6/tests/fixtures.py#L224-L226

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1679#issuecomment-1074459746
issue_url: https://api.github.com/repos/simonw/datasette/issues/1679
node_id: IC_kwDOBm6k_c5ACvRi
created_at: 2022-03-21T21:55:45Z
updated_at: 2022-03-21T21:55:45Z
author_association: OWNER
body:
  I'm going to change the original logic to set n=1 for times that are `<= 20ms` - and update the comments to make it more obvious what is happening.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1671#issuecomment-1074465536
issue_url: https://api.github.com/repos/simonw/datasette/issues/1671
node_id: IC_kwDOBm6k_c5ACwsA
created_at: 2022-03-21T22:04:31Z
updated_at: 2022-03-21T22:04:31Z
author_association: OWNER
body:
  Oh this is fascinating! I replicated the bug (thanks for the steps to reproduce) and it looks like this is down to the following:

  <img width="1276" alt="image" src="https://user-images.githubusercontent.com/9599/159370986-1e2fc513-6d6c-4a2f-96dd-dccf5a680fe4.png">

  Against views, `where has_expired = 1` returns different results from `where has_expired = '1'`

  This doesn't happen against tables because of SQLite's [type affinity](https://www.sqlite.org/datatype3.html#type_affinity) mechanism, which handles the type conversion automatically.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1671#issuecomment-1074468450
issue_url: https://api.github.com/repos/simonw/datasette/issues/1671
node_id: IC_kwDOBm6k_c5ACxZi
created_at: 2022-03-21T22:08:35Z
updated_at: 2022-03-21T22:10:00Z
author_association: OWNER
body:
  Relevant section of the SQLite documentation: [3.2. Affinity Of Expressions](https://www.sqlite.org/datatype3.html#affinity_of_expressions):

  > When an expression is a simple reference to a column of a real table (not a [VIEW](https://www.sqlite.org/lang_createview.html) or subquery) then the expression has the same affinity as the table column.

  In your example, `has_expired` is no longer a simple reference to a column of a real table, hence the bug.

  Then [4.2. Type Conversions Prior To Comparison](https://www.sqlite.org/datatype3.html#type_conversions_prior_to_comparison) fills in the rest:

  > SQLite may attempt to convert values between the storage classes INTEGER, REAL, and/or TEXT before performing a comparison. Whether or not any conversions are attempted before the comparison takes place depends on the type affinity of the operands. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1671#issuecomment-1074470568
issue_url: https://api.github.com/repos/simonw/datasette/issues/1671
node_id: IC_kwDOBm6k_c5ACx6o
created_at: 2022-03-21T22:11:14Z
updated_at: 2022-03-21T22:12:49Z
author_association: OWNER
body:
  I wonder if this will be a problem with generated columns, or with SQLite strict tables?

  My hunch is that strict tables will continue to work without any changes, because https://www.sqlite.org/stricttables.html says nothing about their impact on comparison operations. I should test this to make absolutely sure though.

  Generated columns have a type, so my hunch is they will continue to work fine too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1671#issuecomment-1074478299
issue_url: https://api.github.com/repos/simonw/datasette/issues/1671
node_id: IC_kwDOBm6k_c5ACzzb
created_at: 2022-03-21T22:20:26Z
updated_at: 2022-03-21T22:20:26Z
author_association: OWNER
body:
  Thinking about options for fixing this...

  The following query works fine:
  ```sql
  select * from test_view where cast(has_expired as text) = '1'
  ```
  I don't want to start using this for every query, because one of the goals of Datasette is to help people who are learning SQL:
  - #1613

  If someone clicks on "View and edit SQL" from a filtered table page I don't want them to have to wonder why that `cast` is there.

  But... for querying views, the `cast` turns out to be necessary.

  So one fix would be to get the SQL generating logic to use casts like this any time it is operating against a view.

  An even better fix would be to detect which columns in a view come from a table and which ones might not, and only use casts for the columns that aren't definitely from a table.

  The trick I was exploring here might be able to help with that:
  - #1293 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/276#issuecomment-1074479768
issue_url: https://api.github.com/repos/simonw/datasette/issues/276
node_id: IC_kwDOBm6k_c5AC0KY
created_at: 2022-03-21T22:22:20Z
updated_at: 2022-03-21T22:22:20Z
author_association: OWNER
body:
  I'm closing this issue because this is now solved by a number of neat plugins:

  - https://datasette.io/plugins/datasette-geojson-map shows the geometry from SpatiaLite columns on a map
  - https://datasette.io/plugins/datasette-leaflet-geojson can be used to display inline maps next to each column

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1671#issuecomment-1075425513
issue_url: https://api.github.com/repos/simonw/datasette/issues/1671
node_id: IC_kwDOBm6k_c5AGbDp
created_at: 2022-03-22T17:31:53Z
updated_at: 2022-03-22T17:31:53Z
author_association: OWNER
body:
  The alternative to using `cast` here would be for Datasette to convert the `"1"` to a `1` in Python code before passing it as a param.

  This feels a bit neater to me, but I still then need to solve the problem of how to identify the "type" of a column that I want to use in a query.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1671#issuecomment-1075428030
issue_url: https://api.github.com/repos/simonw/datasette/issues/1671
node_id: IC_kwDOBm6k_c5AGbq-
created_at: 2022-03-22T17:34:30Z
updated_at: 2022-03-22T17:34:30Z
author_association: OWNER
body:
  No, I think I need to use `cast` - I can't think of any way to ask SQLite "for this query, what types are the columns that will come back from it?"

  Even the details from the `explain` trick explored in #1293 don't seem to come back with column type information: https://latest.datasette.io/fixtures?sql=explain+select+pk%2C+text1%2C+text2%2C+[name+with+.+and+spaces]+from+searchable_view+where+%22pk%22+%3D+%3Ap0&p0=1

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1671#issuecomment-1075432283
issue_url: https://api.github.com/repos/simonw/datasette/issues/1671
node_id: IC_kwDOBm6k_c5AGctb
created_at: 2022-03-22T17:39:04Z
updated_at: 2022-03-22T17:43:12Z
author_association: OWNER
body:
  Note that Datasette does already have special logic to convert parameters to integers for numeric comparisons like `>`:

  https://github.com/simonw/datasette/blob/c4c9dbd0386e46d2bf199f0ed34e4895c98cb78c/datasette/filters.py#L203-L212

  Though... it looks like there's a bug in that? It doesn't account for `float` values - `"3.5".isdigit()` return `False` - probably for the best, because `int(3.5)` would break that value anyway.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1671#issuecomment-1075435185
issue_url: https://api.github.com/repos/simonw/datasette/issues/1671
node_id: IC_kwDOBm6k_c5AGdax
created_at: 2022-03-22T17:42:09Z
updated_at: 2022-03-22T17:42:09Z
author_association: OWNER
body:
  Also made me realize that this query:
  ```sql
  select * from sortable where sortable > :p0
  ```
  Only works here thanks to the column affinity thing kicking in too: https://latest.datasette.io/fixtures?sql=select+*+from+sortable+where+sortable+%3E+%3Ap0&p0=70

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1681#issuecomment-1075437598
issue_url: https://api.github.com/repos/simonw/datasette/issues/1681
node_id: IC_kwDOBm6k_c5AGeAe
created_at: 2022-03-22T17:44:42Z
updated_at: 2022-03-22T17:45:04Z
author_association: OWNER
body:
  My hunch is that this mechanism doesn't actually do anything useful at all, because of the type conversion that automatically happens for data from tables based on the column type affinities, see:
  - #1671

  So either remove the `self.numeric` type conversion bit entirely, or prove that it is necessary and upgrade it to be able to handle floating point values too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1681#issuecomment-1075438684
issue_url: https://api.github.com/repos/simonw/datasette/issues/1681
node_id: IC_kwDOBm6k_c5AGeRc
created_at: 2022-03-22T17:45:50Z
updated_at: 2022-03-22T17:49:09Z
author_association: OWNER
body:
  I would expect this to break against SQL views that include calculated columns though - something like this:

  ```sql
  create view this_will_break as select pk + 1 as pk_plus_one, 0.5 as score from searchable;
  ```
  Confirmed: the filter interface for that view plain doesn't work for any comparison against that table - except for `score > 0` since `0` is converted to an integer. `0.1` breaks though because it doesn't get converted as it doesn't match `.isdigit()`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1670#issuecomment-1076638278
issue_url: https://api.github.com/repos/simonw/datasette/issues/1670
node_id: IC_kwDOBm6k_c5ALDJG
created_at: 2022-03-23T17:50:55Z
updated_at: 2022-03-23T17:50:55Z
author_association: OWNER
body:
  Release notes are mostly written for the alpha, just need to clean them up a bit https://github.com/simonw/datasette/blob/c4c9dbd0386e46d2bf199f0ed34e4895c98cb78c/docs/changelog.rst#061a0-2022-03-19

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1665#issuecomment-1076644362
issue_url: https://api.github.com/repos/simonw/datasette/issues/1665
node_id: IC_kwDOBm6k_c5ALEoK
created_at: 2022-03-23T17:55:39Z
updated_at: 2022-03-23T17:55:39Z
author_association: OWNER
body:
  Thanks for the PR - I spotted an error about this and went through and fixed this in all of my repos the other day: https://github.com/search?o=desc&q=user%3Asimonw+google-github-actions%2Fsetup-gcloud%40v0&s=committer-date&type=Commits

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1574#issuecomment-1076645636
issue_url: https://api.github.com/repos/simonw/datasette/issues/1574
node_id: IC_kwDOBm6k_c5ALE8E
created_at: 2022-03-23T17:56:35Z
updated_at: 2022-03-23T17:56:35Z
author_association: OWNER
body:
  I'd actually like to switch to slim as the default - I think Datasette should ship the smallest possible container that can still support extra packages being installed using `apt-get install`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1670#issuecomment-1076647495
issue_url: https://api.github.com/repos/simonw/datasette/issues/1670
node_id: IC_kwDOBm6k_c5ALFZH
created_at: 2022-03-23T17:58:16Z
updated_at: 2022-03-23T17:58:16Z
author_association: OWNER
body:
  I think the release notes are fine, but they need an opening paragraph highlighting the changes that are most likely to break backwards compatibility.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1670#issuecomment-1076652046
issue_url: https://api.github.com/repos/simonw/datasette/issues/1670
node_id: IC_kwDOBm6k_c5ALGgO
created_at: 2022-03-23T18:02:30Z
updated_at: 2022-03-23T18:02:30Z
author_association: OWNER
body:
  Two new things to add to the release notes from https://github.com/simonw/datasette/compare/0.61a0...main
  - https://github.com/simonw/datasette/issues/1678
  - https://github.com/simonw/datasette/issues/1675 (now also a documented API)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/419#issuecomment-1076662556
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/419
node_id: IC_kwDOCGYnMM5ALJEc
created_at: 2022-03-23T18:12:47Z
updated_at: 2022-03-23T18:12:47Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/419?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#419](https://codecov.io/gh/simonw/sqlite-utils/pull/419?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (228f736) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/93fa79d30b1531bea281d0eb6b925c4e61bc1aa6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (93fa79d) will **not change** coverage.
  > The diff coverage is `n/a`.

  ```diff
  @@           Coverage Diff           @@
  ##             main     #419   +/-   ##
  =======================================
    Coverage   96.55%   96.55%           
  =======================================
    Files           6        6           
    Lines        2498     2498           
  =======================================
    Hits         2412     2412           
    Misses         86       86           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/419?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/419?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [93fa79d...228f736](https://codecov.io/gh/simonw/sqlite-utils/pull/419?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1670#issuecomment-1076683297
issue_url: https://api.github.com/repos/simonw/datasette/issues/1670
node_id: IC_kwDOBm6k_c5ALOIh
created_at: 2022-03-23T18:32:32Z
updated_at: 2022-03-23T18:32:32Z
author_association: OWNER
body:
  Added this to news on https://datasette.io/ https://github.com/simonw/datasette.io/commit/fd3ec57cdd5b935f75cbf52a86b3aabf2c97d217

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1682#issuecomment-1076696791
issue_url: https://api.github.com/repos/simonw/datasette/issues/1682
node_id: IC_kwDOBm6k_c5ALRbX
created_at: 2022-03-23T18:45:49Z
updated_at: 2022-03-23T18:45:49Z
author_association: OWNER
body:
  The problem is here in `QueryView`: https://github.com/simonw/datasette/blob/d7c793d7998388d915f8d270079c68a77a785051/datasette/views/database.py#L206-L238

  It should be resolving `database` based on the route path, as seen in other methods like this one: https://github.com/simonw/datasette/blob/d7c793d7998388d915f8d270079c68a77a785051/datasette/views/table.py#L270-L279

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1077671779
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/399
node_id: IC_kwDOCGYnMM5AO_dj
created_at: 2022-03-24T14:11:33Z
updated_at: 2022-03-24T14:11:43Z
author_association: CONTRIBUTOR
body:
  Coming back to this. I was about to add a utility function to [datasette-geojson]() to convert lat/lng columns to geometries. Thankfully I googled first. There's a SpatiaLite function for this: [MakePoint](https://www.gaia-gis.it/gaia-sins/spatialite-sql-latest.html#p0).

  ```sql
  select MakePoint(longitude, latitude) as geometry from places;
  ```

  I'm not sure if that would work with `conversions`, since it needs two columns, but it's an option for tables that already have latitude, longitude columns.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1684#issuecomment-1078126065
issue_url: https://api.github.com/repos/simonw/datasette/issues/1684
node_id: IC_kwDOBm6k_c5AQuXx
created_at: 2022-03-24T20:08:56Z
updated_at: 2022-03-24T20:13:19Z
author_association: CONTRIBUTOR
body:
  would be nice if the behavior was

  1. try to facet all the columns
  2. for bigger tables try to facet the indexed columns
  3. for the biggest tables, turn off autofacetting completely

  This is based on my assumption that what determines autofaceting is the rarity of unique values. Which may not be true!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1078315922
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/420
node_id: IC_kwDOCGYnMM5ARcuS
created_at: 2022-03-24T21:09:27Z
updated_at: 2022-03-24T21:09:27Z
author_association: OWNER
body:
  Yeah, this is WAY harder than it should be.

  There's a clumsy workaround you could use which looks something like this: create a file `my_enchant.py` containing:

  ```python
  import enchant
  d = enchant.Dict("en_US")

  def check(word):
      return d.check(word)
  ```
  Then run `sqlite-utils` like this:

  ```
  PYTHONPATH=. cat items.json | jq '.data' | sqlite-utils insert listings.db listings - --convert 'my_enchant.check(value)' --import my_enchant
  ```
  Except I tried that and it doesn't work! I don't know the right pattern for getting `--import` to work with modules in the same directory.

  So yeah, this is definitely a big feature gap.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1078322301
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/420
node_id: IC_kwDOCGYnMM5AReR9
created_at: 2022-03-24T21:10:52Z
updated_at: 2022-03-24T21:10:52Z
author_association: OWNER
body:
  I can think of three ways forward:

  - Figure out a pattern that gets that local file import workaround to work
  - Add another option such as `--convert-init` that lets you pass code that will be executed once at the start
  - Come up with a pattern where the `--convert` code can run some initialization code and then return a function which will be called against each value

  I quite like the idea of that third option - I'm going to prototype it and see if I can work something out.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1078328774
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/420
node_id: IC_kwDOCGYnMM5ARf3G
created_at: 2022-03-24T21:12:33Z
updated_at: 2022-03-24T21:12:33Z
author_association: OWNER
body:
  Here's how the `_compile_code()` mechanism works at the moment: https://github.com/simonw/sqlite-utils/blob/396f80fcc60da8dd844577114f7920830a2e5403/sqlite_utils/utils.py#L308-L342

  At the end it does this:
  ```python
      return locals["fn"]
  ```
  So it's already building and then returning a function.

  The question is if there's a sensible way to allow people to further customize that function by executing some code first, in a way that's easy to explain.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1078343231
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/420
node_id: IC_kwDOCGYnMM5ARjY_
created_at: 2022-03-24T21:16:10Z
updated_at: 2022-03-24T21:17:20Z
author_association: OWNER
body:
  Aha! This may be possible already: https://github.com/simonw/sqlite-utils/blob/396f80fcc60da8dd844577114f7920830a2e5403/sqlite_utils/utils.py#L311-L316

  And yes, this does indeed work - you can do something like this:

  ```
  echo '{"name": "harry"}' | sqlite-utils insert db.db people - --convert '
  import time
  # Simulate something expensive
  time.sleep(1)

  def convert(row):
      row["upper"] = row["name"].upper()
  '
  ```
  And after running that:
  ```
  sqlite-utils dump db.db 
  BEGIN TRANSACTION;
  CREATE TABLE [people] (
     [name] TEXT,
     [upper] TEXT
  );
  INSERT INTO "people" VALUES('harry','HARRY');
  COMMIT;
  ```
  So this is a documentation issue - there's a trick for it but I didn't know what the trick was!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1685#issuecomment-1079018557
issue_url: https://api.github.com/repos/simonw/datasette/issues/1685
node_id: IC_kwDOBm6k_c5AUIQ9
created_at: 2022-03-25T13:16:48Z
updated_at: 2022-03-25T13:16:48Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1685?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1685](https://codecov.io/gh/simonw/datasette/pull/1685?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (933ce47) into [main](https://codecov.io/gh/simonw/datasette/commit/c496f2b663ff0cef908ffaaa68b8cb63111fb5f2?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (c496f2b) will **not change** coverage.
  > The diff coverage is `n/a`.

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1685   +/-   ##
  =======================================
    Coverage   91.74%   91.74%           
  =======================================
    Files          34       34           
    Lines        4565     4565           
  =======================================
    Hits         4188     4188           
    Misses        377      377           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1685?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1685?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [c496f2b...933ce47](https://codecov.io/gh/simonw/datasette/pull/1685?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1079243535
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/420
node_id: IC_kwDOCGYnMM5AU_MP
created_at: 2022-03-25T17:25:12Z
updated_at: 2022-03-25T17:25:12Z
author_association: OWNER
body:
  That documentation is split across a few places. This is the only bit that talks about `def convert()` pattern right now:

  - https://sqlite-utils.datasette.io/en/stable/cli.html#converting-data-in-columns

  But that's for `sqlite-utils convert` - the documentation for `sqlite-utils insert --convert` at https://sqlite-utils.datasette.io/en/stable/cli.html#applying-conversions-while-inserting-data doesn't mention it.

  Since both `sqlite-utils convert` and `sqlite-utils insert --convert` apply the same rules to the code, they should link to a shared explanation in the documentation.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1079376283
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/420
node_id: IC_kwDOCGYnMM5AVfmb
created_at: 2022-03-25T19:39:30Z
updated_at: 2022-03-25T19:43:35Z
author_association: OWNER
body:
  Actually this doesn't work as I thought. This demo shows that the initialization code is run once per item, not a single time at the start of the run:
  ```
  % sqlite-utils insert dogs.db dogs dogs.json --convert '
  import random
  print("seeding")
  random.seed(10)
  print(random.random())

  def convert(row):
      print(row)
      row["random_score"] = random.random()
  '
  seeding
  0.5714025946899135
  seeding
  0.5714025946899135
  seeding
  0.5714025946899135
  seeding
  0.5714025946899135
  ```
  Also that `print(row)` line is not being printed anywhere that gets to the console for some reason.

  ... my mistake, that happened because I changed this line in order to try to get local imports to work:
  ```python
      try:
          exec(code, globals, locals)
          return globals["convert"]
      except (AttributeError, SyntaxError, NameError, KeyError, TypeError):
  ```
  It should be `locals["convert"]`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1079384771
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/420
node_id: IC_kwDOCGYnMM5AVhrD
created_at: 2022-03-25T19:51:34Z
updated_at: 2022-03-25T19:53:01Z
author_association: OWNER
body:
  This works:
  ```
  % sqlite-utils insert dogs.db dogs dogs.json --convert '
  import random
  print("seeding")
  random.seed(10)
  print(random.random())

  def convert(row):
      global random
      print(row)
      row["random_score"] = random.random()
  '
  seeding
  0.5714025946899135
  {'id': 1, 'name': 'Cleo'}
  {'id': 2, 'name': 'Pancakes'}
  {'id': 3, 'name': 'New dog'}
  (sqlite-utils) sqlite-utils % sqlite-utils rows dogs.db dogs
  [{"id": 1, "name": "Cleo", "random_score": 0.4288890546751146},
   {"id": 2, "name": "Pancakes", "random_score": 0.5780913011344704},
   {"id": 3, "name": "New dog", "random_score": 0.20609823213950174}]
  ```
  Having to use `global random` inside the function is frustrating but apparently necessary. https://stackoverflow.com/a/56552138/6083

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1079404281
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/420
node_id: IC_kwDOCGYnMM5AVmb5
created_at: 2022-03-25T20:19:50Z
updated_at: 2022-03-25T20:19:50Z
author_association: OWNER
body:
  Now documented here: https://sqlite-utils.datasette.io/en/latest/cli.html#using-a-convert-function-to-execute-initialization

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/422#issuecomment-1079406708
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/422
node_id: IC_kwDOCGYnMM5AVnB0
created_at: 2022-03-25T20:23:21Z
updated_at: 2022-03-25T20:23:21Z
author_association: OWNER
body:
  Fixing this would require a bump to 4.0 because it would break existing code.

  The alternative would be to introduce a new `ignore_nulls=True` parameter which users can change to `ignore_nulls=False`. Or come up with better wording for that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1079407962
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/421
node_id: IC_kwDOCGYnMM5AVnVa
created_at: 2022-03-25T20:25:10Z
updated_at: 2022-03-25T20:25:18Z
author_association: OWNER
body:
  Can you share either your whole `global.db` table or a shrunk down example that illustrates the bug?

  My hunch is that you may have a table or column with a name that triggers the error.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/417#issuecomment-1079441621
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/417
node_id: IC_kwDOCGYnMM5AVvjV
created_at: 2022-03-25T21:18:37Z
updated_at: 2022-03-25T21:18:37Z
author_association: OWNER
body:
  Updated documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#inserting-newline-delimited-json

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1688#issuecomment-1079550754
issue_url: https://api.github.com/repos/simonw/datasette/issues/1688
node_id: IC_kwDOBm6k_c5AWKMi
created_at: 2022-03-26T01:27:27Z
updated_at: 2022-03-26T03:16:29Z
author_association: CONTRIBUTOR
body:
  > Is there a way to serve a static assets when using the plugins/ directory method instead of installing plugins as a new python package?

  As a workaround, I found I can serve my statics from a non-plugin specific folder using the [--static](https://docs.datasette.io/en/stable/custom_templates.html#serving-static-files) CLI flag.

  ```bash
  datasette ~/Library/Safari/History.db \
    --plugins-dir=plugins/ \
    --static assets:dist/
  ```

  It's not ideal because it means I'll change the cache pattern path depending on how the plugin is running (via pip install or as a one off script), but it's usable as a workaround.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1688#issuecomment-1079582485
issue_url: https://api.github.com/repos/simonw/datasette/issues/1688
node_id: IC_kwDOBm6k_c5AWR8V
created_at: 2022-03-26T03:15:34Z
updated_at: 2022-03-26T03:15:34Z
author_association: OWNER
body:
  Yup, you're right in what you figured out here: stand-alone plugins can't currently package static assets other then using the static folder.

  The `datasette-plugin` cookiecutter template should make creating a Python package pretty easy though: https://github.com/simonw/datasette-plugin

  You can run that yourself, or you can run it using this GitHub template repository: https://github.com/simonw/datasette-plugin-template-repository 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1689#issuecomment-1079779040
issue_url: https://api.github.com/repos/simonw/datasette/issues/1689
node_id: IC_kwDOBm6k_c5AXB7g
created_at: 2022-03-26T21:35:57Z
updated_at: 2022-03-26T21:35:57Z
author_association: OWNER
body:
  Fixed: https://docs.datasette.io/en/latest/internals.html#add-message-request-message-type-datasette-info

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1690#issuecomment-1079788346
issue_url: https://api.github.com/repos/simonw/datasette/issues/1690
node_id: IC_kwDOBm6k_c5AXEM6
created_at: 2022-03-26T22:42:40Z
updated_at: 2022-03-26T22:42:40Z
author_association: OWNER
body:
  I don't want to do a `response.set_actor_cookie()` method because I like `Response` not to carry too many Datasette-specific features.

  So `datasette.set_actor_cookie(response, actor, expire_after=None)` would be a better place for this I think.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1690#issuecomment-1079788375
issue_url: https://api.github.com/repos/simonw/datasette/issues/1690
node_id: IC_kwDOBm6k_c5AXENX
created_at: 2022-03-26T22:43:00Z
updated_at: 2022-03-26T22:43:00Z
author_association: OWNER
body:
  Then I can update this section of the documentation which currently recommends the above pattern: https://docs.datasette.io/en/stable/authentication.html#the-ds-actor-cookie

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1688#issuecomment-1079806857
issue_url: https://api.github.com/repos/simonw/datasette/issues/1688
node_id: IC_kwDOBm6k_c5AXIuJ
created_at: 2022-03-27T01:01:14Z
updated_at: 2022-03-27T01:01:14Z
author_association: CONTRIBUTOR
body:
  Thank you! I went through the cookiecutter template, and published my first package here: https://github.com/hydrosquall/datasette-nteract-data-explorer

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1080141111
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/420
node_id: IC_kwDOCGYnMM5AYaU3
created_at: 2022-03-28T03:25:57Z
updated_at: 2022-03-28T03:54:37Z
author_association: OWNER
body:
  So now this should solve your problem:
  ```
  echo '[{"name": "notaword"}, {"name": "word"}]
  ' | python3 -m sqlite_utils insert listings.db listings - --convert '
  import enchant
  d = enchant.Dict("en_US")

  def convert(row):
      global d
      row["is_dictionary_word"] = d.check(row["name"])
  '
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1081047053
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/420
node_id: IC_kwDOCGYnMM5Ab3gN
created_at: 2022-03-28T19:22:37Z
updated_at: 2022-03-28T19:22:37Z
author_association: OWNER
body:
  Wrote about this in my weeknotes: https://simonwillison.net/2022/Mar/28/datasette-auth0/#new-features-as-documentation

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1081079506
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/421
node_id: IC_kwDOCGYnMM5Ab_bS
created_at: 2022-03-28T19:58:55Z
updated_at: 2022-03-28T20:05:57Z
author_association: NONE
body:
  Sure, it is from the documentation example:
  [Extracting columns into a separate table](https://sqlite-utils.datasette.io/en/stable/cli.html#extracting-columns-into-a-separate-table)


  ```
  wget "https://github.com/wri/global-power-plant-database/blob/232a6666/output_database/global_power_plant_database.csv?raw=true"

  sqlite-utils insert global.db power_plants \
      'global_power_plant_database.csv?raw=true' --csv
  # Extract those columns:
  sqlite-utils extract global.db power_plants country country_long \
      --table countries \
      --fk-column country_id \
      --rename country_long name
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1694#issuecomment-1081860312
issue_url: https://api.github.com/repos/simonw/datasette/issues/1694
node_id: IC_kwDOBm6k_c5Ae-DY
created_at: 2022-03-29T13:17:30Z
updated_at: 2022-03-29T13:17:30Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1694?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1694](https://codecov.io/gh/simonw/datasette/pull/1694?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (83ff967) into [main](https://codecov.io/gh/simonw/datasette/commit/e73fa72917ca28c152208d62d07a490c81cadf52?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (e73fa72) will **not change** coverage.
  > The diff coverage is `n/a`.

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1694   +/-   ##
  =======================================
    Coverage   91.74%   91.74%           
  =======================================
    Files          34       34           
    Lines        4565     4565           
  =======================================
    Hits         4188     4188           
    Misses        377      377           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1694?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1694?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [e73fa72...83ff967](https://codecov.io/gh/simonw/datasette/pull/1694?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1693#issuecomment-1081861670
issue_url: https://api.github.com/repos/simonw/datasette/issues/1693
node_id: IC_kwDOBm6k_c5Ae-Ym
created_at: 2022-03-29T13:18:47Z
updated_at: 2022-05-20T20:36:30Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1693?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1693](https://codecov.io/gh/simonw/datasette/pull/1693?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (65a5d5e) into [main](https://codecov.io/gh/simonw/datasette/commit/1465fea4798599eccfe7e8f012bd8d9adfac3039?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (1465fea) will **not change** coverage.
  > The diff coverage is `n/a`.

  > :exclamation: Current head 65a5d5e differs from pull request most recent head ec2d1e4. Consider uploading reports for the commit ec2d1e4 to get more accurate results

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1693   +/-   ##
  =======================================
    Coverage   91.67%   91.67%           
  =======================================
    Files          36       36           
    Lines        4658     4658           
  =======================================
    Hits         4270     4270           
    Misses        388      388           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1693?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1693?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [1d33fd0...ec2d1e4](https://codecov.io/gh/simonw/datasette/pull/1693?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1695#issuecomment-1082617241
issue_url: https://api.github.com/repos/simonw/datasette/issues/1695
node_id: IC_kwDOBm6k_c5Ah22Z
created_at: 2022-03-30T04:45:55Z
updated_at: 2022-03-30T04:45:55Z
author_association: OWNER
body:
  Relevant template: https://github.com/simonw/datasette/blob/e73fa72917ca28c152208d62d07a490c81cadf52/datasette/templates/table.html#L168-L172

  Populated from here: https://github.com/simonw/datasette/blob/c496f2b663ff0cef908ffaaa68b8cb63111fb5f2/datasette/facets.py#L246-L253

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1692#issuecomment-1082663746
issue_url: https://api.github.com/repos/simonw/datasette/issues/1692
node_id: IC_kwDOBm6k_c5AiCNC
created_at: 2022-03-30T06:14:39Z
updated_at: 2022-03-30T06:14:51Z
author_association: OWNER
body:
  I like your design, though I think it should be `"nomodule": True` for consistency with the other options.

  I think `"async": True` is worth supporting too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1696#issuecomment-1083351437
issue_url: https://api.github.com/repos/simonw/datasette/issues/1696
node_id: IC_kwDOBm6k_c5AkqGN
created_at: 2022-03-30T16:20:49Z
updated_at: 2022-03-30T16:21:02Z
author_association: OWNER
body:
  Maybe like this:

  <img width="499" alt="image" src="https://user-images.githubusercontent.com/9599/160883280-c19d5a22-e923-491f-8bf4-1a4f5215d684.png">

  ```html
  <h3>283 rows
    where dcode = 3 <span style="color: #aaa; font-size: 0.9em">(Human Related: Other)</span>
      </h3>
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1574#issuecomment-1084216224
issue_url: https://api.github.com/repos/simonw/datasette/issues/1574
node_id: IC_kwDOBm6k_c5An9Og
created_at: 2022-03-31T07:45:25Z
updated_at: 2022-03-31T07:45:25Z
author_association: NONE
body:
  @simonw I like that you want to go "slim by default". Do you want another PR for that or should I just wait?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1549#issuecomment-1087428593
issue_url: https://api.github.com/repos/simonw/datasette/issues/1549
node_id: IC_kwDOBm6k_c5A0Nfx
created_at: 2022-04-04T11:17:13Z
updated_at: 2022-04-04T11:17:13Z
author_association: CONTRIBUTOR
body:
  another way to get the behavior of downloading the file is to use the download attribute of the anchor tag

  https://developer.mozilla.org/en-US/docs/Web/HTML/Element/a#attr-download

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1699#issuecomment-1092321966
issue_url: https://api.github.com/repos/simonw/datasette/issues/1699
node_id: IC_kwDOBm6k_c5BG4Ku
created_at: 2022-04-08T00:20:32Z
updated_at: 2022-04-08T00:20:56Z
author_association: OWNER
body:
  If we do this I'm keen to have it be more than just an alternative to the existing `sqlite-utils` command - especially since if I add `sqlite-utils` as a dependency of Datasette in the future that command will be installed as part of `pip install datasette` anyway.

  My best thought on how to differentiate them so far is plugins: if Datasette plugins that provide alternative outputs - like `.geojson` and `.yml` and suchlike - also work for the `datasette query` command that would make a lot of sense to me.

  One way that could work: a `--fmt geojson` option to this command which uses the plugin that was registered for the specified extension.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1699#issuecomment-1092357672
issue_url: https://api.github.com/repos/simonw/datasette/issues/1699
node_id: IC_kwDOBm6k_c5BHA4o
created_at: 2022-04-08T01:39:40Z
updated_at: 2022-04-08T01:39:40Z
author_association: CONTRIBUTOR
body:
  > My best thought on how to differentiate them so far is plugins: if Datasette plugins that provide alternative outputs - like .geojson and .yml and suchlike - also work for the datasette query command that would make a lot of sense to me.

  That's my thinking, too. It's really the thing I've been wanting since writing `datasette-geojson`, since I'm always exporting with `datasette --get`. The workflow I'm always looking for is something like this:

  ```sh
  cd alltheplaces-datasette
  datasette query dunkin_in_suffolk -f geojson -o dunkin_in_suffolk.geojson
  ```

  I think this probably needs either a new plugin hook separate from `register_output_renderer` or a way to use that without going through the HTTP stack. Or maybe a render mode that writes to a stream instead of a response. Maybe there's a new key in the dictionary that `register_output_renderer` returns that handles CLI exports.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1699#issuecomment-1092361727
issue_url: https://api.github.com/repos/simonw/datasette/issues/1699
node_id: IC_kwDOBm6k_c5BHB3_
created_at: 2022-04-08T01:47:43Z
updated_at: 2022-04-08T01:47:43Z
author_association: OWNER
body:
  A render mode for that plugin hook that writes to a stream is exactly what I have in mind:
  - #1062 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1699#issuecomment-1092370880
issue_url: https://api.github.com/repos/simonw/datasette/issues/1699
node_id: IC_kwDOBm6k_c5BHEHA
created_at: 2022-04-08T02:07:40Z
updated_at: 2022-04-08T02:07:40Z
author_association: CONTRIBUTOR
body:
  So maybe `render_output_render` returns something like this:

  ```python
  @hookimpl
  def register_output_renderer(datasette):
      return {
          "extension": "geojson",
          "render": render_geojson,
          "stream": stream_geojson,
          "can_render": can_render_geojson,
      }
  ```

  And stream gets an iterator, instead of a list of rows, so it can efficiently handle large queries. Maybe it also gets passed a destination stream, or it returns an iterator. I'm not sure what makes more sense. Either way, that might cover both CLI exports and streaming responses.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1699#issuecomment-1092386254
issue_url: https://api.github.com/repos/simonw/datasette/issues/1699
node_id: IC_kwDOBm6k_c5BHH3O
created_at: 2022-04-08T02:39:25Z
updated_at: 2022-04-08T02:39:25Z
author_association: CONTRIBUTOR
body:
  And just to think this through a little more, here's what `stream_geojson` might look like:

  ```python
  async def stream_geojson(datasette, columns, rows, database, stream):
      db = datasette.get_database(database)
      for row in rows:
          feature = await row_to_geojson(row, db)
          stream.write(feature + "\n") # just assuming newline mode for now
  ```

  Alternately, that could be an async generator, like this:

  ```python
  async def stream_geojson(datasette, columns, rows, database):
      db = datasette.get_database(database)
      for row in rows:
          feature = await row_to_geojson(row, db)
          yield feature
  ```

  Not sure which makes more sense, but I think this pattern would open up a lot of possibility. If you had your [stream_indented_json](https://til.simonwillison.net/python/output-json-array-streaming) function, you could do `yield from stream_indented_json(rows, 2)` and be one your way.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1703#issuecomment-1092850719
issue_url: https://api.github.com/repos/simonw/datasette/issues/1703
node_id: IC_kwDOBm6k_c5BI5Qf
created_at: 2022-04-08T13:18:04Z
updated_at: 2022-04-08T13:18:04Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1703?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1703](https://codecov.io/gh/simonw/datasette/pull/1703?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (73aabe6) into [main](https://codecov.io/gh/simonw/datasette/commit/90d1be9952db9aaddc21a536e4d00a8de44765d7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (90d1be9) will **not change** coverage.
  > The diff coverage is `n/a`.

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1703   +/-   ##
  =======================================
    Coverage   91.75%   91.75%           
  =======================================
    Files          34       34           
    Lines        4573     4573           
  =======================================
    Hits         4196     4196           
    Misses        377      377           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1703?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1703?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [90d1be9...73aabe6](https://codecov.io/gh/simonw/datasette/pull/1703?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1693#issuecomment-1093454899
issue_url: https://api.github.com/repos/simonw/datasette/issues/1693
node_id: IC_kwDOBm6k_c5BLMwz
created_at: 2022-04-08T23:07:04Z
updated_at: 2022-04-08T23:07:04Z
author_association: OWNER
body:
  Tests failed here due to this issue:
  - https://github.com/psf/black/pull/2987

  A future Black release should fix that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1699#issuecomment-1094453751
issue_url: https://api.github.com/repos/simonw/datasette/issues/1699
node_id: IC_kwDOBm6k_c5BPAn3
created_at: 2022-04-11T01:32:12Z
updated_at: 2022-04-11T01:32:12Z
author_association: CONTRIBUTOR
body:
  Was looking through old issues and realized a bunch of this got discussed in #1101 (including by me!), so sorry to rehash all this. Happy to help with whatever piece of it I can. Would be very excited to be able to use format plugins with exports.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1707#issuecomment-1095277937
issue_url: https://api.github.com/repos/simonw/datasette/issues/1707
node_id: IC_kwDOBm6k_c5BSJ1x
created_at: 2022-04-11T16:32:31Z
updated_at: 2022-04-11T16:33:00Z
author_association: OWNER
body:
  That's a really interesting idea!

  That page is one of the least developed at the moment. There's plenty of room for it to grow new useful features.

  I like this suggestion because it feels like a good opportunity to introduce some unobtrusive JavaScript. Could use a details/summary element that uses `fetch()` to load in the extra data for example.

  Could even do something with the `<datasette-table>` Web Component here... https://github.com/simonw/datasette-table

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1708#issuecomment-1095675839
issue_url: https://api.github.com/repos/simonw/datasette/issues/1708
node_id: IC_kwDOBm6k_c5BTq-_
created_at: 2022-04-11T23:06:30Z
updated_at: 2022-11-15T19:57:53Z
author_association: OWNER
body:
  # Datasette 1.0 alpha 1

  This alpha release is a preview of Datasette 1.0.

  Datasette 1.0 marks a significant milestone in the project: it is the point from which various aspects of Datasette can be considered "stable", in that code developed against them should expect not to be broken by future releases in the 1.x series.

  This will hold true until the next major version release, Datasette 2.0 - which we hope to hold off releasing for as long as possible.

  The following Datasette components should be be considered stable after 1.0:

  - The plugin API. Plugins developed against 1.0 should continue to work unmodified throughout the 1.x series.
  - The JSON API. Code written that interacts with Datasette's default JSON web API should continue to work.
  - The template context. If you build custom templates against Datasette your custom pages should continue to work.

  Note that none of these components will cease to introduce new features. New plugin hooks, new JSON APIs and new template context variables can be introduced without breaking existing code.

  Since this alpha release previews features that will be frozen for 1.0, please test this thoroughly against your existing Datasette projects.

  You can install the alpha using:

      pip install datasette==1.0a0

  ## JSON API changes

  The most significant changes introduced in this new alpha concern Datasette's JSON API.

  The default JSON returned by the `/database/table.json` endpoint has changed. It now returns an object with two keys: `rows` - which contains a list of objects representing the rows in the table or query, and `more` containing a `boolean` that shows if there are more rows or if this object contains them all.

  ```json
  {
    "rows": [{
      "id": 1,
      "name": "Name 1"
    }, {
      "id": 2,
      "name": "Name 2"
    }],
    "more": false
  }
  ```
  [ Initially I thought about going with `next_url`, which would be `null` if you have reached the last page of records. Maybe that would be better? But since `next_url` cannot be provided on query pages, should this be part of the default format at all? ]

  ## Use ?_extra= to retrieve extra fields

  The default format can be expanded using one or more `?_extra=` parameters. This takes names of extra keys you would like to include. These can be comma-separated or `?_extra=` can be applied multiple times.

  For example:

      /database/table.json?_extra=total

  This adds a `"total": 124` field to the returned JSON.

  [ Question: if you do `?_facet=foo` then do you still need to do `?_extra=facets` - I think not? ]

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1708#issuecomment-1095687566
issue_url: https://api.github.com/repos/simonw/datasette/issues/1708
node_id: IC_kwDOBm6k_c5BTt2O
created_at: 2022-04-11T23:24:30Z
updated_at: 2022-04-11T23:24:30Z
author_association: OWNER
body:
  ## Redesigned template context

  **Warning:** if you use any custom templates with your Datasette instance they are likely to break when you upgrade to 1.0.

  The template context has been redesigned to be based on the documented JSON API. This means that the template context can be considered stable going forward, so any custom templates you implement should continue to work when you upgrade Datasette in the future.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1712#issuecomment-1097068474
issue_url: https://api.github.com/repos/simonw/datasette/issues/1712
node_id: IC_kwDOBm6k_c5BY--6
created_at: 2022-04-12T18:38:18Z
updated_at: 2022-04-12T18:38:18Z
author_association: OWNER
body:
  <img width="633" alt="image" src="https://user-images.githubusercontent.com/9599/163030785-9dcc5a21-6a1b-42a7-97de-10e7d2874412.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098288158
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/421
node_id: IC_kwDOCGYnMM5Bdowe
created_at: 2022-04-13T17:07:53Z
updated_at: 2022-04-13T17:07:53Z
author_association: OWNER
body:
  I can't replicate the bug I'm afraid:
  ```
  % wget "https://github.com/wri/global-power-plant-database/blob/232a6666/output_database/global_power_plant_database.csv?raw=true"               
  ...
  2022-04-13 10:06:29 (8.97 MB/s) - ‘global_power_plant_database.csv?raw=true’ saved [8856038/8856038]
  % sqlite-utils insert global.db power_plants \                      
      'global_power_plant_database.csv?raw=true' --csv
    [------------------------------------]    0%
    [###################################-]   99%  00:00:00%
  % sqlite-utils indexes global.db --table                            
  table    index_name    seqno    cid    name    desc    coll    key
  -------  ------------  -------  -----  ------  ------  ------  -----
  % sqlite-utils extract global.db power_plants country country_long \
      --table countries \
      --fk-column country_id \
      --rename country_long name
  % sqlite-utils indexes global.db --table                            
  table      index_name                    seqno    cid  name       desc  coll      key
  ---------  --------------------------  -------  -----  -------  ------  ------  -----
  countries  idx_countries_country_name        0      1  country       0  BINARY      1
  countries  idx_countries_country_name        1      2  name          0  BINARY      1
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098295517
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/421
node_id: IC_kwDOCGYnMM5Bdqjd
created_at: 2022-04-13T17:16:20Z
updated_at: 2022-04-13T17:16:20Z
author_association: OWNER
body:
  Aha! I was able to replicate the bug using your `Dockerfile` - thanks very much for providing that.
  ```
  (app-root) sqlite-utils indexes global.db --table
  Error: near "(": syntax error
  ```
  (That wa sbefore I even ran the `extract` command.)

  To build your `Dockerfile` I copied it into an empty folder and ran the following:
  ```
  wget https://www.sqlite.org/2021/sqlite-autoconf-3360000.tar.gz
  docker build . -t centos-sqlite-utils
  docker run -it centos-sqlite-utils /bin/bash
  ```
  This gave me a shell in which I could replicate the bug.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098531354
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/421
node_id: IC_kwDOCGYnMM5BekIa
created_at: 2022-04-13T22:08:20Z
updated_at: 2022-04-13T22:08:20Z
author_association: OWNER
body:
  OK I figured out what's going on here. First I added an extra `print(sql)` statement to the `indexes` command to see what SQL it was running:
  ```
  (app-root) sqlite-utils indexes global.db --table

      select
        sqlite_master.name as "table",
        indexes.name as index_name,
        xinfo.*
      from sqlite_master
        join pragma_index_list(sqlite_master.name) indexes
        join pragma_index_xinfo(index_name) xinfo
      where
        sqlite_master.type = 'table'
       and xinfo.key = 1
  Error: near "(": syntax error
  ```
  This made me suspicious that the SQLite version being used here didn't support joining against the `pragma_index_list(...)` table-valued functions in that way. So I checked the version:
  ```
  (app-root) sqlite3
  SQLite version 3.36.0 2021-06-18 18:36:39
  ```
  That version should be fine - it's the one you compiled in the Dockerfile.

  Then I checked the version that `sqlite-utils` itself was using:
  ```
  (app-root) sqlite-utils memory 'select sqlite_version()'
  [{"sqlite_version()": "3.7.17"}]
  ```
  It's running SQLite 3.7.17!

  So the problem here is that the Python in that Docker image is running a very old version of SQLite.

  I tried using the trick in https://til.simonwillison.net/sqlite/ld-preload as a workaround, and it almost worked:

  ```
  (app-root) python3 -c 'import sqlite3; print(sqlite3.connect(":memory").execute("select sqlite_version()").fetchone())'
  ('3.7.17',)
  (app-root) LD_PRELOAD=./build/sqlite-autoconf-3360000/.libs/libsqlite3.so python3 -c 'import sqlite3; print(sqlite3.connect(":memory").execute("select sqlite_version()").fetchone())'
  ('3.36.0',)
  ```
  But when I try to run `sqlite-utils` like that I get an error:

  ```
  (app-root) LD_PRELOAD=./build/sqlite-autoconf-3360000/.libs/libsqlite3.so sqlite-utils indexes /tmp/global.db 
  ...
    File "/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/cli.py", line 1624, in query
      db.register_fts4_bm25()
    File "/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py", line 412, in register_fts4_bm25
      self.register_function(rank_bm25, deterministic=True)
    File "/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py", line 408, in register_function
      register(fn)
    File "/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py", line 401, in register
      self.conn.create_function(name, arity, fn, **kwargs)
  sqlite3.NotSupportedError: deterministic=True requires SQLite 3.8.3 or higher
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098532220
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/421
node_id: IC_kwDOCGYnMM5BekV8
created_at: 2022-04-13T22:09:52Z
updated_at: 2022-04-13T22:09:52Z
author_association: OWNER
body:
  That error is weird - it's not supposed to happen according to this code here: https://github.com/simonw/sqlite-utils/blob/95522ad919f96eb6cc8cd3cd30389b534680c717/sqlite_utils/db.py#L389-L400

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098535531
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/421
node_id: IC_kwDOCGYnMM5BelJr
created_at: 2022-04-13T22:15:48Z
updated_at: 2022-04-13T22:15:48Z
author_association: OWNER
body:
  Trying this alternative implementation of the `register()` method:

  ```python
          def register(fn):
              name = fn.__name__
              arity = len(inspect.signature(fn).parameters)
              if not replace and (name, arity) in self._registered_functions:
                  return fn
              kwargs = {}
              done = False
              if deterministic:
                  # Try this, but fall back if sqlite3.NotSupportedError
                  try:
                      self.conn.create_function(name, arity, fn, **dict(kwargs, deterministic=True))
                      done = True
                  except sqlite3.NotSupportedError:
                      pass
              if not done:
                  self.conn.create_function(name, arity, fn, **kwargs)
              self._registered_functions.add((name, arity))
              return fn
  ```
  With that fix, the following worked!
  ```
  LD_PRELOAD=./build/sqlite-autoconf-3360000/.libs/libsqlite3.so sqlite-utils indexes /tmp/global.db --table
  table      index_name                    seqno    cid  name       desc  coll      key
  ---------  --------------------------  -------  -----  -------  ------  ------  -----
  countries  idx_countries_country_name        0      1  country       0  BINARY      1
  countries  idx_countries_country_name        1      2  name          0  BINARY      1
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/425#issuecomment-1098537000
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/425
node_id: IC_kwDOCGYnMM5Belgo
created_at: 2022-04-13T22:18:22Z
updated_at: 2022-04-13T22:18:22Z
author_association: OWNER
body:
  I figured out a workaround in https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098535531

  The current `register(fn)` method looks like this: https://github.com/simonw/sqlite-utils/blob/95522ad919f96eb6cc8cd3cd30389b534680c717/sqlite_utils/db.py#L389-L403

  This alternative implementation worked in the environment where that failed:

  ```python
          def register(fn):
              name = fn.__name__
              arity = len(inspect.signature(fn).parameters)
              if not replace and (name, arity) in self._registered_functions:
                  return fn
              kwargs = {}
              done = False
              if deterministic:
                  # Try this, but fall back if sqlite3.NotSupportedError
                  try:
                      self.conn.create_function(name, arity, fn, **dict(kwargs, deterministic=True))
                      done = True
                  except sqlite3.NotSupportedError:
                      pass
              if not done:
                  self.conn.create_function(name, arity, fn, **kwargs)
              self._registered_functions.add((name, arity))
              return fn
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/425#issuecomment-1098545390
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/425
node_id: IC_kwDOCGYnMM5Benju
created_at: 2022-04-13T22:34:52Z
updated_at: 2022-04-13T22:34:52Z
author_association: OWNER
body:
  That broke Python 3.7 because it doesn't support `deterministic=True` even being passed:

  > function takes at most 3 arguments (4 given)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/424#issuecomment-1098548090
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/424
node_id: IC_kwDOCGYnMM5BeoN6
created_at: 2022-04-13T22:40:15Z
updated_at: 2022-04-13T22:40:15Z
author_association: OWNER
body:
  New error:
  ```pycon
  >>> from sqlite_utils import Database
  >>> db = Database(memory=True)
  >>> db["foo"].create({})
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py", line 1465, in create
      self.db.create_table(
    File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py", line 885, in create_table
      sql = self.create_table_sql(
    File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py", line 771, in create_table_sql
      assert columns, "Tables must have at least one column"
  AssertionError: Tables must have at least one column
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098548931
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/421
node_id: IC_kwDOCGYnMM5BeobD
created_at: 2022-04-13T22:41:59Z
updated_at: 2022-04-13T22:41:59Z
author_association: OWNER
body:
  I'm going to close this ticket since it looks like this is a bug in the way the Dockerfile builds Python, but I'm going to ship a fix for that issue I found so the `LD_PRELOAD` workaround above should work OK with the next release of `sqlite-utils`. Thanks for the detailed bug report!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1713#issuecomment-1098628334
issue_url: https://api.github.com/repos/simonw/datasette/issues/1713
node_id: IC_kwDOBm6k_c5Be7zu
created_at: 2022-04-14T01:43:00Z
updated_at: 2022-04-14T01:43:13Z
author_association: OWNER
body:
  Current workaround for fast publishing to S3:

      datasette fixtures.db --get /fixtures/facetable.json | \
        s3-credentials put-object my-bucket facetable.json -

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1713#issuecomment-1099443468
issue_url: https://api.github.com/repos/simonw/datasette/issues/1713
node_id: IC_kwDOBm6k_c5BiC0M
created_at: 2022-04-14T17:26:27Z
updated_at: 2022-04-14T17:26:27Z
author_association: NONE
body:
  What would be an awesome feature as a plugin would be to be able to save a query (and possibly even results) to a github gist. Being able to share results that way would be super fantastic. Possibly even in Jupyter Notebook format (since github and github gists nicely render those)! 

  I know there's the handy datasette-saved-queries plugin, but a button that could export stuff out and then even possibly import stuff back in (I'm sort of thinking the way that Google Colab allows you to save to github, and then pull the notebook back in is a really great workflow 
  ![image](https://user-images.githubusercontent.com/9308268/163441612-9ad2649f-c73e-4557-aaf2-e3d0fdc48fbf.png)
  https://github.com/cincinnatilibrary/collection-analysis/blob/master/reports/colab_datasette_example.ipynb )

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1713#issuecomment-1099540225
issue_url: https://api.github.com/repos/simonw/datasette/issues/1713
node_id: IC_kwDOBm6k_c5BiacB
created_at: 2022-04-14T19:09:57Z
updated_at: 2022-04-14T19:09:57Z
author_association: CONTRIBUTOR
body:
  I wonder if this overlaps with what I outlined in #1605. You could run something like this:

  ```sh
  datasette freeze -d exports/
  aws s3 cp exports/ s3://my-export-bucket/$(date)
  ```

  And maybe that does what you need. Of course, that plugin isn't built yet. But that's the idea.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1574#issuecomment-1105464661
issue_url: https://api.github.com/repos/simonw/datasette/issues/1574
node_id: IC_kwDOBm6k_c5B5A1V
created_at: 2022-04-21T16:51:24Z
updated_at: 2022-04-21T16:51:24Z
author_association: NONE
body:
  tfw you have more ephemeral storage than upstream bandwidth

  ```
  FROM python:3.10-slim AS base

  RUN apt update && apt -y install zstd

  ENV DATASETTE_SECRET 'sosecret'
  RUN --mount=type=cache,target=/root/.cache/pip
      pip install -U datasette datasette-pretty-json datasette-graphql

  ENV PORT 8080
  EXPOSE 8080

  FROM base AS pack

  COPY . /app
  WORKDIR /app

  RUN datasette inspect --inspect-file inspect-data.json
  RUN zstd --rm *.db

  FROM base AS unpack

  COPY --from=pack /app /app
  WORKDIR /app

  CMD ["/bin/bash", "-c", "shopt -s nullglob && zstd --rm -d *.db.zst && datasette serve --host 0.0.0.0 --cors --inspect-file inspect-data.json --metadata metadata.json --create --port $PORT *.db"]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/72#issuecomment-1105474232
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/72
node_id: IC_kwDODFdgUs5B5DK4
created_at: 2022-04-21T17:02:15Z
updated_at: 2022-04-21T17:02:15Z
author_association: MEMBER
body:
  That's interesting - yeah it looks like the number of pages can be derived from the `Link` header, which is enough information to show a progress bar, probably using Click just to avoid adding another dependency.

  https://docs.github.com/en/rest/guides/traversing-with-pagination

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1101#issuecomment-1105571003
issue_url: https://api.github.com/repos/simonw/datasette/issues/1101
node_id: IC_kwDOBm6k_c5B5ay7
created_at: 2022-04-21T18:10:38Z
updated_at: 2022-04-21T18:10:46Z
author_association: OWNER
body:
  Maybe the simplest design for this is to add an optional `can_stream` to the contract:

  ```python
      @hookimpl
      def register_output_renderer(datasette):
          return {
              "extension": "tsv",
              "render": render_tsv,
              "can_render": lambda: True,
              "can_stream": lambda: True
          }
  ```
  When streaming, a new parameter could be passed to the render function - maybe `chunks` - which is an iterator/generator over a sequence of chunks of rows.

  Or it could use the existing `rows` parameter but treat that as an iterator?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1101#issuecomment-1105588651
issue_url: https://api.github.com/repos/simonw/datasette/issues/1101
node_id: IC_kwDOBm6k_c5B5fGr
created_at: 2022-04-21T18:15:39Z
updated_at: 2022-04-21T18:15:39Z
author_association: CONTRIBUTOR
body:
  What if you split rendering and streaming into two things:

  - `render` is a function that returns a response
  - `stream` is a function that sends chunks, or yields chunks passed to an ASGI `send` callback

  That way current plugins still work, and streaming is purely additive. A `stream` function could get a cursor or iterator of rows, instead of a list, so it could more efficiently handle large queries.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1101#issuecomment-1105608964
issue_url: https://api.github.com/repos/simonw/datasette/issues/1101
node_id: IC_kwDOBm6k_c5B5kEE
created_at: 2022-04-21T18:26:29Z
updated_at: 2022-04-21T18:26:29Z
author_association: OWNER
body:
  I'm questioning if the mechanisms should be separate at all now - a single response rendering is really just a case of a streaming response that only pulls the first N records from the iterator.

  It probably needs to be an `async for` iterator, which I've not worked with much before. Good opportunity to learn.

  This actually gets a fair bit more complicated due to the work I'm doing right now to improve the default JSON API:

  - #1709

  I want to do things like make faceting results optionally available to custom renderers - which is a separate concern from streaming rows.

  I'm going to poke around with a bunch of prototypes and see what sticks.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1101#issuecomment-1105615625
issue_url: https://api.github.com/repos/simonw/datasette/issues/1101
node_id: IC_kwDOBm6k_c5B5lsJ
created_at: 2022-04-21T18:31:41Z
updated_at: 2022-04-21T18:32:22Z
author_association: OWNER
body:
  The `datasette-geojson` plugin is actually an interesting case here, because of the way it converts SpatiaLite geometries into GeoJSON: https://github.com/eyeseast/datasette-geojson/blob/602c4477dc7ddadb1c0a156cbcd2ef6688a5921d/datasette_geojson/__init__.py#L61-L66

  ```python

      if isinstance(geometry, bytes):
          results = await db.execute(
              "SELECT AsGeoJSON(:geometry)", {"geometry": geometry}
          )
          return geojson.loads(results.single_value())
  ```
  That actually seems to work really well as-is, but it does worry me a bit that it ends up having to execute an extra `SELECT` query for every single returned row - especially in streaming mode where it might be asked to return 1m rows at once.

  My PostgreSQL/MySQL engineering brain says that this would be better handled by doing a chunk of these (maybe 100) at once, to avoid the per-query-overhead - but with SQLite that might not be necessary.

  At any rate, this is one of the reasons I'm interested in "iterate over this sequence of chunks of 100 rows at a time" as a potential option here.

  Of course, a better solution would be for `datasette-geojson` to have a way to influence the SQL query before it is executed, adding a `AsGeoJSON(geometry)` clause to it - so that's something I'm open to as well.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1101#issuecomment-1105642187
issue_url: https://api.github.com/repos/simonw/datasette/issues/1101
node_id: IC_kwDOBm6k_c5B5sLL
created_at: 2022-04-21T18:59:08Z
updated_at: 2022-04-21T18:59:08Z
author_association: CONTRIBUTOR
body:
  Ha! That was your idea (and a good one).

  But it's probably worth measuring to see what overhead it adds. It did require both passing in the database and making the whole thing `async`. 

  Just timing the queries themselves:

  1. [Using `AsGeoJSON(geometry) as geometry`](https://alltheplaces-datasette.fly.dev/alltheplaces?sql=select%0D%0A++id%2C%0D%0A++properties%2C%0D%0A++AsGeoJSON%28geometry%29+as+geometry%2C%0D%0A++spider%0D%0Afrom%0D%0A++places%0D%0Aorder+by%0D%0A++id%0D%0Alimit%0D%0A++1000) takes 10.235 ms
  2. [Leaving as binary](https://alltheplaces-datasette.fly.dev/alltheplaces?sql=select%0D%0A++id%2C%0D%0A++properties%2C%0D%0A++geometry%2C%0D%0A++spider%0D%0Afrom%0D%0A++places%0D%0Aorder+by%0D%0A++id%0D%0Alimit%0D%0A++1000) takes 8.63 ms

  Looking at the network panel:

  1. Takes about 200 ms for the `fetch` request
  2. Takes about 300 ms

  I'm not sure how best to time the GeoJSON generation, but it would be interesting to check. Maybe I'll write a plugin to add query times to response headers.

  The other thing to consider with async streaming is that it might be well-suited for a slower response. When I have to get the whole result and send a response in a fixed amount of time, I need the most efficient query possible. If I can hang onto a connection and get things one chunk at a time, maybe it's ok if there's some overhead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1715#issuecomment-1106908642
issue_url: https://api.github.com/repos/simonw/datasette/issues/1715
node_id: IC_kwDOBm6k_c5B-hXi
created_at: 2022-04-22T21:47:55Z
updated_at: 2022-04-22T21:47:55Z
author_association: OWNER
body:
  I need a `asyncio.Registry` with functions registered to perform the role of the table view.

  Something like this perhaps:
  ```python
  def table_html_context(facet_results, query, datasette, rows):
      return {...}
  ```
  That then gets called like this:
  ```python
  async def view(request):
      registry = Registry(facet_results, query, datasette, rows)
      context = await registry.resolve(table_html, request=request, datasette=datasette)
      return Reponse.html(await datasette.render("table.html", context)
  ```
  It's also interesting to start thinking about this from a Python client library point of view. If I'm writing code outside of the HTTP request cycle, what would it look like?

  One thing I could do: break out is the code that turns a request into a list of pairs extracted from the request - this code here: https://github.com/simonw/datasette/blob/8338c66a57502ef27c3d7afb2527fbc0663b2570/datasette/views/table.py#L442-L449

  I could turn that into a typed dependency injection function like this:

  ```python
  def filter_args(request: Request) -> List[Tuple[str, str]]:
      # Arguments that start with _ and don't contain a __ are
      # special - things like ?_search= - and should not be
      # treated as filters.
      filter_args = []
      for key in request.args:
          if not (key.startswith("_") and "__" not in key):
              for v in request.args.getlist(key):
                  filter_args.append((key, v))
      return filter_args
  ```
  Then I can either pass a `request` into a `.resolve()` call, or I can instead skip that function by passing:

  ```python
  output = registry.resolve(table_context, filter_args=[("foo", "bar")])
  ```
  I do need to think about where plugins get executed in all of this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1716#issuecomment-1106923258
issue_url: https://api.github.com/repos/simonw/datasette/issues/1716
node_id: IC_kwDOBm6k_c5B-k76
created_at: 2022-04-22T22:02:07Z
updated_at: 2022-04-22T22:02:07Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/blame/main/datasette/views/base.py

  <img width="1373" alt="image" src="https://user-images.githubusercontent.com/9599/164801564-d8a11ce9-7d9b-4e85-8947-a547d2986ef3.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1715#issuecomment-1106945876
issue_url: https://api.github.com/repos/simonw/datasette/issues/1715
node_id: IC_kwDOBm6k_c5B-qdU
created_at: 2022-04-22T22:24:29Z
updated_at: 2022-04-22T22:24:29Z
author_association: OWNER
body:
  Looking at the start of `TableView.data()`:

  https://github.com/simonw/datasette/blob/d57c347f35bcd8cff15f913da851b4b8eb030867/datasette/views/table.py#L333-L346

  I'm going to resolve `table_name` and `database` from the URL - `table_name` will be a string, `database` will be the DB object returned by `datasette.get_database()`. Then those can be passed in separately too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1715#issuecomment-1106947168
issue_url: https://api.github.com/repos/simonw/datasette/issues/1715
node_id: IC_kwDOBm6k_c5B-qxg
created_at: 2022-04-22T22:25:57Z
updated_at: 2022-04-22T22:26:06Z
author_association: OWNER
body:
  ```python
  async def database(request: Request, datasette: Datasette) -> Database:
      database_route = tilde_decode(request.url_vars["database"])
      try:
          return datasette.get_database(route=database_route)
      except KeyError:
          raise NotFound("Database not found: {}".format(database_route))

  async def table_name(request: Request) -> str:
      return tilde_decode(request.url_vars["table"])
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1715#issuecomment-1106989581
issue_url: https://api.github.com/repos/simonw/datasette/issues/1715
node_id: IC_kwDOBm6k_c5B-1IN
created_at: 2022-04-22T23:03:29Z
updated_at: 2022-04-22T23:03:29Z
author_association: OWNER
body:
  I'm having second thoughts about injecting `request` - might be better to have the view function pull the relevant pieces out of the request before triggering the rest of the resolution.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1717#issuecomment-1107459446
issue_url: https://api.github.com/repos/simonw/datasette/issues/1717
node_id: IC_kwDOBm6k_c5CAn12
created_at: 2022-04-23T11:56:36Z
updated_at: 2022-04-23T11:56:36Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1717?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1717](https://codecov.io/gh/simonw/datasette/pull/1717?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (9b9a314) into [main](https://codecov.io/gh/simonw/datasette/commit/d57c347f35bcd8cff15f913da851b4b8eb030867?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (d57c347) will **increase** coverage by `0.00%`.
  > The diff coverage is `100.00%`.

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1717   +/-   ##
  =======================================
    Coverage   91.75%   91.75%           
  =======================================
    Files          34       34           
    Lines        4574     4575    +1     
  =======================================
  + Hits         4197     4198    +1     
    Misses        377      377           
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1717?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/publish/cloudrun.py](https://codecov.io/gh/simonw/datasette/pull/1717/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3B1Ymxpc2gvY2xvdWRydW4ucHk=) | `97.05% <100.00%> (+0.04%)` | :arrow_up: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1717?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1717?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [d57c347...9b9a314](https://codecov.io/gh/simonw/datasette/pull/1717?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1718#issuecomment-1107863365
issue_url: https://api.github.com/repos/simonw/datasette/issues/1718
node_id: IC_kwDOBm6k_c5CCKdF
created_at: 2022-04-24T15:26:41Z
updated_at: 2022-04-24T15:26:41Z
author_association: OWNER
body:
  Tried this:
  ```
  pip install blacken-docs
  blacken-docs docs/*.rst
  git diff | pbcopy
  ```
  Got this:
  ```diff
   diff --git a/docs/authentication.rst b/docs/authentication.rst
  index 0d98cf8..8008023 100644
  --- a/docs/authentication.rst
  +++ b/docs/authentication.rst
  @@ -381,11 +381,7 @@ Authentication plugins can set signed ``ds_actor`` cookies themselves like so:
   .. code-block:: python
   
       response = Response.redirect("/")
  -    response.set_cookie("ds_actor", datasette.sign({
  -        "a": {
  -            "id": "cleopaws"
  -        }
  -    }, "actor"))
  +    response.set_cookie("ds_actor", datasette.sign({"a": {"id": "cleopaws"}}, "actor"))
   
   Note that you need to pass ``"actor"`` as the namespace to :ref:`datasette_sign`.
   
  @@ -412,12 +408,16 @@ To include an expiry, add a ``"e"`` key to the cookie value containing a `base62
       expires_at = int(time.time()) + (24 * 60 * 60)
   
       response = Response.redirect("/")
  -    response.set_cookie("ds_actor", datasette.sign({
  -        "a": {
  -            "id": "cleopaws"
  -        },
  -        "e": baseconv.base62.encode(expires_at),
  -    }, "actor"))
  +    response.set_cookie(
  +        "ds_actor",
  +        datasette.sign(
  +            {
  +                "a": {"id": "cleopaws"},
  +                "e": baseconv.base62.encode(expires_at),
  +            },
  +            "actor",
  +        ),
  +    )
   
   The resulting cookie will encode data that looks something like this:
   
  diff --git a/docs/spatialite.rst b/docs/spatialite.rst
  index d1b300b..556bad8 100644
  --- a/docs/spatialite.rst
  +++ b/docs/spatialite.rst
  @@ -58,19 +58,22 @@ Here's a recipe for taking a table with existing latitude and longitude columns,
   .. code-block:: python
   
       import sqlite3
  -    conn = sqlite3.connect('museums.db')
  +
  +    conn = sqlite3.connect("museums.db")
       # Lead the spatialite extension:
       conn.enable_load_extension(True)
  -    conn.load_extension('/usr/local/lib/mod_spatialite.dylib')
  +    conn.load_extension("/usr/local/lib/mod_spatialite.dylib")
       # Initialize spatial metadata for this database:
  -    conn.execute('select InitSpatialMetadata(1)')
  +    conn.execute("select InitSpatialMetadata(1)")
       # Add a geometry column called point_geom to our museums table:
       conn.execute("SELECT AddGeometryColumn('museums', 'point_geom', 4326, 'POINT', 2);")
       # Now update that geometry column with the lat/lon points
  -    conn.execute('''
  +    conn.execute(
  +        """
           UPDATE museums SET
           point_geom = GeomFromText('POINT('||"longitude"||' '||"latitude"||')',4326);
  -    ''')
  +    """
  +    )
       # Now add a spatial index to that column
       conn.execute('select CreateSpatialIndex("museums", "point_geom");')
       # If you don't commit your changes will not be persisted:
  @@ -186,13 +189,14 @@ Here's Python code to create a SQLite database, enable SpatiaLite, create a plac
   .. code-block:: python
   
       import sqlite3
  -    conn = sqlite3.connect('places.db')
  +
  +    conn = sqlite3.connect("places.db")
       # Enable SpatialLite extension
       conn.enable_load_extension(True)
  -    conn.load_extension('/usr/local/lib/mod_spatialite.dylib')
  +    conn.load_extension("/usr/local/lib/mod_spatialite.dylib")
       # Create the masic countries table
  -    conn.execute('select InitSpatialMetadata(1)')
  -    conn.execute('create table places (id integer primary key, name text);')
  +    conn.execute("select InitSpatialMetadata(1)")
  +    conn.execute("create table places (id integer primary key, name text);")
       # Add a MULTIPOLYGON Geometry column
       conn.execute("SELECT AddGeometryColumn('places', 'geom', 4326, 'MULTIPOLYGON', 2);")
       # Add a spatial index against the new column
  @@ -201,13 +205,17 @@ Here's Python code to create a SQLite database, enable SpatiaLite, create a plac
       from shapely.geometry.multipolygon import MultiPolygon
       from shapely.geometry import shape
       import requests
  -    geojson = requests.get('https://data.whosonfirst.org/404/227/475/404227475.geojson').json()
  +
  +    geojson = requests.get(
  +        "https://data.whosonfirst.org/404/227/475/404227475.geojson"
  +    ).json()
       # Convert to "Well Known Text" format
  -    wkt = shape(geojson['geometry']).wkt
  +    wkt = shape(geojson["geometry"]).wkt
       # Insert and commit the record
  -    conn.execute("INSERT INTO places (id, name, geom) VALUES(null, ?, GeomFromText(?, 4326))", (
  -       "Wales", wkt
  -    ))
  +    conn.execute(
  +        "INSERT INTO places (id, name, geom) VALUES(null, ?, GeomFromText(?, 4326))",
  +        ("Wales", wkt),
  +    )
       conn.commit()
   
   Querying polygons using within()
  diff --git a/docs/writing_plugins.rst b/docs/writing_plugins.rst
  index bd60a4b..5af01f6 100644
  --- a/docs/writing_plugins.rst
  +++ b/docs/writing_plugins.rst
  @@ -18,9 +18,10 @@ The quickest way to start writing a plugin is to create a ``my_plugin.py`` file
   
       from datasette import hookimpl
   
  +
       @hookimpl
       def prepare_connection(conn):
  -        conn.create_function('hello_world', 0, lambda: 'Hello world!')
  +        conn.create_function("hello_world", 0, lambda: "Hello world!")
   
   If you save this in ``plugins/my_plugin.py`` you can then start Datasette like this::
   
  @@ -60,22 +61,18 @@ The example consists of two files: a ``setup.py`` file that defines the plugin:
   
       from setuptools import setup
   
  -    VERSION = '0.1'
  +    VERSION = "0.1"
   
       setup(
  -        name='datasette-plugin-demos',
  -        description='Examples of plugins for Datasette',
  -        author='Simon Willison',
  -        url='https://github.com/simonw/datasette-plugin-demos',
  -        license='Apache License, Version 2.0',
  +        name="datasette-plugin-demos",
  +        description="Examples of plugins for Datasette",
  +        author="Simon Willison",
  +        url="https://github.com/simonw/datasette-plugin-demos",
  +        license="Apache License, Version 2.0",
           version=VERSION,
  -        py_modules=['datasette_plugin_demos'],
  -        entry_points={
  -            'datasette': [
  -                'plugin_demos = datasette_plugin_demos'
  -            ]
  -        },
  -        install_requires=['datasette']
  +        py_modules=["datasette_plugin_demos"],
  +        entry_points={"datasette": ["plugin_demos = datasette_plugin_demos"]},
  +        install_requires=["datasette"],
       )
   
   And a Python module file, ``datasette_plugin_demos.py``, that implements the plugin:
  @@ -88,12 +85,12 @@ And a Python module file, ``datasette_plugin_demos.py``, that implements the plu
   
       @hookimpl
       def prepare_jinja2_environment(env):
  -        env.filters['uppercase'] = lambda u: u.upper()
  +        env.filters["uppercase"] = lambda u: u.upper()
   
   
       @hookimpl
       def prepare_connection(conn):
  -        conn.create_function('random_integer', 2, random.randint)
  +        conn.create_function("random_integer", 2, random.randint)
   
   
   Having built a plugin in this way you can turn it into an installable package using the following command::
  @@ -123,11 +120,13 @@ To bundle the static assets for a plugin in the package that you publish to PyPI
   
   .. code-block:: python
   
  -        package_data={
  -            'datasette_plugin_name': [
  -                'static/plugin.js',
  -            ],
  -        },
  +        package_data = (
  +            {
  +                "datasette_plugin_name": [
  +                    "static/plugin.js",
  +                ],
  +            },
  +        )
   
   Where ``datasette_plugin_name`` is the name of the plugin package (note that it uses underscores, not hyphens) and ``static/plugin.js`` is the path within that package to the static file.
   
  @@ -152,11 +151,13 @@ Templates should be bundled for distribution using the same ``package_data`` mec
   
   .. code-block:: python
   
  -        package_data={
  -            'datasette_plugin_name': [
  -                'templates/my_template.html',
  -            ],
  -        },
  +        package_data = (
  +            {
  +                "datasette_plugin_name": [
  +                    "templates/my_template.html",
  +                ],
  +            },
  +        )
   
   You can also use wildcards here such as ``templates/*.html``. See `datasette-edit-schema <https://github.com/simonw/datasette-edit-schema>`__ for an example of this pattern.
   ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1718#issuecomment-1107863924
issue_url: https://api.github.com/repos/simonw/datasette/issues/1718
node_id: IC_kwDOBm6k_c5CCKl0
created_at: 2022-04-24T15:30:03Z
updated_at: 2022-04-24T15:30:03Z
author_association: OWNER
body:
  On the one hand, I'm not crazy about some of the indentation decisions Black made here - in particular this one, which I had indented deliberately for readability:
  ```diff
   diff --git a/docs/authentication.rst b/docs/authentication.rst
  index 0d98cf8..8008023 100644
  --- a/docs/authentication.rst
  +++ b/docs/authentication.rst
  @@ -381,11 +381,7 @@ Authentication plugins can set signed ``ds_actor`` cookies themselves like so:
   .. code-block:: python
   
       response = Response.redirect("/")
  -    response.set_cookie("ds_actor", datasette.sign({
  -        "a": {
  -            "id": "cleopaws"
  -        }
  -    }, "actor"))
  +    response.set_cookie("ds_actor", datasette.sign({"a": {"id": "cleopaws"}}, "actor"))
  ```
  But... consistency is a virtue. Maybe I'm OK with just this one disagreement?

  Also: I've been mentally trying to keep the line lengths a bit shorter to help them  be more readable on mobile devices.

  I'll try a different line length using `blacken-docs -l 60 docs/*.rst` instead.

  I like this more - here's the result for that example:
  ```diff
  diff --git a/docs/authentication.rst b/docs/authentication.rst
  index 0d98cf8..2496073 100644
  --- a/docs/authentication.rst
  +++ b/docs/authentication.rst
  @@ -381,11 +381,10 @@ Authentication plugins can set signed ``ds_actor`` cookies themselves like so:
   .. code-block:: python
   
       response = Response.redirect("/")
  -    response.set_cookie("ds_actor", datasette.sign({
  -        "a": {
  -            "id": "cleopaws"
  -        }
  -    }, "actor"))
  +    response.set_cookie(
  +        "ds_actor",
  +        datasette.sign({"a": {"id": "cleopaws"}}, "actor"),
  +    )
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1718#issuecomment-1107865493
issue_url: https://api.github.com/repos/simonw/datasette/issues/1718
node_id: IC_kwDOBm6k_c5CCK-V
created_at: 2022-04-24T15:39:02Z
updated_at: 2022-04-24T15:39:02Z
author_association: OWNER
body:
  There's no `blacken-docs --check` option so I filed a feature request:
  - https://github.com/asottile/blacken-docs/issues/161

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1718#issuecomment-1107866013
issue_url: https://api.github.com/repos/simonw/datasette/issues/1718
node_id: IC_kwDOBm6k_c5CCLGd
created_at: 2022-04-24T15:42:07Z
updated_at: 2022-04-24T15:42:07Z
author_association: OWNER
body:
  In the absence of `--check` I can use this to detect if changes are applied:
  ```zsh
  % git diff-index --quiet HEAD --
  % echo $?                       
  0
  % blacken-docs -l 60 docs/*.rst
  docs/authentication.rst: Rewriting...
  ...
  % git diff-index --quiet HEAD --
  % echo $?                       
  1
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1718#issuecomment-1107867281
issue_url: https://api.github.com/repos/simonw/datasette/issues/1718
node_id: IC_kwDOBm6k_c5CCLaR
created_at: 2022-04-24T15:49:23Z
updated_at: 2022-04-24T15:49:23Z
author_association: OWNER
body:
  I'm going to push the first commit with a deliberate missing formatting to check that the tests fail.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1718#issuecomment-1107868585
issue_url: https://api.github.com/repos/simonw/datasette/issues/1718
node_id: IC_kwDOBm6k_c5CCLup
created_at: 2022-04-24T15:57:10Z
updated_at: 2022-04-24T15:57:19Z
author_association: OWNER
body:
  The tests failed there because of what I thought were warnings but turn out to be treated as errors:
  ```
  % blacken-docs -l 60 docs/*.rst                                        
  docs/internals.rst:196: code block parse error Cannot parse: 14:0: <line number missing in source>
  docs/json_api.rst:449: code block parse error Cannot parse: 1:0: <link rel="alternate"
  docs/plugin_hooks.rst:250: code block parse error Cannot parse: 6:4:     ]
  docs/plugin_hooks.rst:311: code block parse error Cannot parse: 38:0: <line number missing in source>
  docs/testing_plugins.rst:135: code block parse error Cannot parse: 5:0: <line number missing in source>
  % echo $?
  1
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1718#issuecomment-1107869556
issue_url: https://api.github.com/repos/simonw/datasette/issues/1718
node_id: IC_kwDOBm6k_c5CCL90
created_at: 2022-04-24T16:02:27Z
updated_at: 2022-04-24T16:02:27Z
author_association: OWNER
body:
  Looking at that first error it appears to be a place where I had deliberately omitted the body of the function:

  https://github.com/simonw/datasette/blob/36573638b0948174ae237d62e6369b7d55220d7f/docs/internals.rst#L196-L211

  I can use `...` as the function body here to get it to pass.

  Fixing those warnings actually helped me spot a couple of bugs, so I'm glad this happened.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1718#issuecomment-1107873271
issue_url: https://api.github.com/repos/simonw/datasette/issues/1718
node_id: IC_kwDOBm6k_c5CCM33
created_at: 2022-04-24T16:23:57Z
updated_at: 2022-04-24T16:23:57Z
author_association: OWNER
body:
  Turns out I didn't need that `git diff-index` trick after all - the `blacken-docs` command returns a non-zero exit code if it changes any files.

  Submitted a documentation PR to that project instead:

  - https://github.com/asottile/blacken-docs/pull/162

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1715#issuecomment-1108875068
issue_url: https://api.github.com/repos/simonw/datasette/issues/1715
node_id: IC_kwDOBm6k_c5CGBc8
created_at: 2022-04-25T18:03:13Z
updated_at: 2022-04-25T18:06:33Z
author_association: OWNER
body:
  The `RowTableShared` class is making this a whole lot more complicated.

  I'm going to split the `RowView` view out into an entirely separate `views/row.py` module.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1719#issuecomment-1108884171
issue_url: https://api.github.com/repos/simonw/datasette/issues/1719
node_id: IC_kwDOBm6k_c5CGDrL
created_at: 2022-04-25T18:10:46Z
updated_at: 2022-04-25T18:12:45Z
author_association: OWNER
body:
  It looks like the only class method from that shared class needed  by `RowView` is `self.display_columns_and_rows()`.

  Which I've been wanting to refactor to provide to `QueryView` too:

  - #715

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1719#issuecomment-1108888494
issue_url: https://api.github.com/repos/simonw/datasette/issues/1719
node_id: IC_kwDOBm6k_c5CGEuu
created_at: 2022-04-25T18:15:42Z
updated_at: 2022-04-25T18:15:42Z
author_association: OWNER
body:
  Here's an undocumented feature I forgot existed: https://latest.datasette.io/fixtures/simple_primary_key/1.json?_extras=foreign_key_tables

  `?_extras=foreign_key_tables`

  https://github.com/simonw/datasette/blob/0bc5186b7bb4fc82392df08f99a9132f84dcb331/datasette/views/table.py#L1021-L1024

  It's even covered by the tests:

  https://github.com/simonw/datasette/blob/b9c2b1cfc8692b9700416db98721fa3ec982f6be/tests/test_api.py#L691-L703

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/262#issuecomment-1108890170
issue_url: https://api.github.com/repos/simonw/datasette/issues/262
node_id: IC_kwDOBm6k_c5CGFI6
created_at: 2022-04-25T18:17:09Z
updated_at: 2022-04-25T18:18:39Z
author_association: OWNER
body:
  I spotted in https://github.com/simonw/datasette/issues/1719#issuecomment-1108888494 that there's actually already an undocumented implementation of `?_extras=foreign_key_tables` - https://latest.datasette.io/fixtures/simple_primary_key/1.json?_extras=foreign_key_tables

  I added that feature all the way back in November 2017! https://github.com/simonw/datasette/commit/a30c5b220c15360d575e94b0e67f3255e120b916

reactions:
[cell repeats an earlier one in this file: 112 characters]


