# issue_comments

html_url: https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864207841
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/279
node_id: MDEyOklzc3VlQ29tbWVudDg2NDIwNzg0MQ==
created_at: 2021-06-18T18:28:40Z
updated_at: 2021-06-18T18:28:46Z
author_association: OWNER
body:
  ```python
  def detect_format(fp):
      # ...
      return "csv", fp, dialect
      # or
      return "json", fp, parsed_data
      # or
      return "json-nl", fp, docs
  ```
  The mixed return types here are ugly. In all of these cases what we really want is to return a generator of `{...}` objects. So maybe it returns that instead.
  ```python
  def filepointer_to_documents(fp):
      # ...
      yield from documents
  ```
  I can refactor `sqlite-utils insert` to use this new code too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864208476
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/279
node_id: MDEyOklzc3VlQ29tbWVudDg2NDIwODQ3Ng==
created_at: 2021-06-18T18:30:08Z
updated_at: 2021-06-18T23:30:19Z
author_association: OWNER
body:
  So maybe this is a function which can either be told the format or, if none is provided, it detects one for itself.
  ```python
  def rows_from_file(fp, format=None):
      # ...
      yield from rows
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/281#issuecomment-864323438
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/281
node_id: MDEyOklzc3VlQ29tbWVudDg2NDMyMzQzOA==
created_at: 2021-06-18T23:55:06Z
updated_at: 2021-06-18T23:55:06Z
author_association: OWNER
body:
  The `-:json` idea is flawed: Click thinks that's the syntax for an option called `:json`.

  I'm going to do `stdin:json` - which means you can't open a file called `stdin` - but you could use `cat stdin | sqlite-utils memory stdin:json ...`  instead which is an OK workaround.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864328927
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/279
node_id: MDEyOklzc3VlQ29tbWVudDg2NDMyODkyNw==
created_at: 2021-06-19T00:25:08Z
updated_at: 2021-06-19T00:25:17Z
author_association: OWNER
body:
  I tried writing this function with type hints, but eventually gave up:
  ```python
  def rows_from_file(
      fp: BinaryIO,
      format: Optional[Format] = None,
      dialect: Optional[Type[csv.Dialect]] = None,
      encoding: Optional[str] = None,
  ) -> Generator[dict, None, None]:
      if format == Format.JSON:
          decoded = json.load(fp)
          if isinstance(decoded, dict):
              decoded = [decoded]
          if not isinstance(decoded, list):
              raise RowsFromFileBadJSON("JSON must be a list or a dictionary")
          yield from decoded
      elif format == Format.CSV:
          decoded_fp = io.TextIOWrapper(fp, encoding=encoding or "utf-8-sig")
          yield from csv.DictReader(decoded_fp)
      elif format == Format.TSV:
          yield from rows_from_file(
              fp, format=Format.CSV, dialect=csv.excel_tab, encoding=encoding
          )
      elif format is None:
          # Detect the format, then call this recursively
          buffered = io.BufferedReader(fp, buffer_size=4096)
          first_bytes = buffered.peek(2048).strip()
          if first_bytes[0] in (b"[", b"{"):
              # TODO: Detect newline-JSON
              yield from rows_from_file(fp, format=Format.JSON)
          else:
              dialect = csv.Sniffer().sniff(first_bytes.decode(encoding, "ignore"))
              yield from rows_from_file(
                  fp, format=Format.CSV, dialect=dialect, encoding=encoding
              )
      else:
          raise RowsFromFileError("Bad format")
  ```
  mypy said:
  ```
  sqlite_utils/utils.py:157: error: Argument 1 to "BufferedReader" has incompatible type "BinaryIO"; expected "RawIOBase"
  sqlite_utils/utils.py:163: error: Argument 1 to "decode" of "bytes" has incompatible type "Optional[str]"; expected "str"
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864330508
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/279
node_id: MDEyOklzc3VlQ29tbWVudDg2NDMzMDUwOA==
created_at: 2021-06-19T00:34:24Z
updated_at: 2021-06-19T00:34:24Z
author_association: OWNER
body:
  Got this working:

      % curl 'https://api.github.com/repos/simonw/datasette/issues' | sqlite-utils memory - 'select id from stdin' 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/282#issuecomment-864348954
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/282
node_id: MDEyOklzc3VlQ29tbWVudDg2NDM0ODk1NA==
created_at: 2021-06-19T03:34:42Z
updated_at: 2021-06-19T03:35:46Z
author_association: OWNER
body:
  I built some prototype code here for something which looks at every row in a CSV import and records the likely types: https://gist.github.com/simonw/465f9356f175d1cf86957947dff501d4

  This could be used by the command-line tools to figure out what `table.transform(types=...)` method to use at the end.

  This is a different approach to the pure SQL version I tried building in https://github.com/simonw/sqlite-utils/issues/179 - I think this is a better approach though, it's less prone to weird idiosyncrasies of SQLite types, and it's also easy for us to add on to the existing CSV import code in a way that won't require scanning the data twice.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/282#issuecomment-864349123
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/282
node_id: MDEyOklzc3VlQ29tbWVudDg2NDM0OTEyMw==
created_at: 2021-06-19T03:36:54Z
updated_at: 2021-06-19T03:36:54Z
author_association: OWNER
body:
  I may change the default for `sqlite-utils insert` to detect types if I release `sqlite-utils` 4.0, as a backwards-incompatible change.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/282#issuecomment-864350407
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/282
node_id: MDEyOklzc3VlQ29tbWVudDg2NDM1MDQwNw==
created_at: 2021-06-19T03:52:20Z
updated_at: 2021-06-19T03:52:20Z
author_association: OWNER
body:
  I'll have an environment variable for `--detect-types` so users who really want that as the default option can turn it on.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/282#issuecomment-864354627
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/282
node_id: MDEyOklzc3VlQ29tbWVudDg2NDM1NDYyNw==
created_at: 2021-06-19T04:42:03Z
updated_at: 2021-06-19T04:42:03Z
author_association: OWNER
body:
  Demo:

      curl -s 'https://api.github.com/users/simonw/repos?per_page=100' | \
        sqlite-utils memory - 'select sum(size), sum(stargazers_count) from stdin limit 1'
      [{"sum(size)": 2042547, "sum(stargazers_count)": 6769}]

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/284#issuecomment-864358680
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/284
node_id: MDEyOklzc3VlQ29tbWVudDg2NDM1ODY4MA==
created_at: 2021-06-19T05:27:13Z
updated_at: 2021-06-19T05:27:13Z
author_association: OWNER
body:
  How easy is it to detect a `rowid` table? Is it as simple as `.pks` returning `None`? If so the documentation should mention that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/283#issuecomment-864416086
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/283
node_id: MDEyOklzc3VlQ29tbWVudDg2NDQxNjA4Ng==
created_at: 2021-06-19T14:49:06Z
updated_at: 2021-06-19T14:49:13Z
author_association: OWNER
body:
  Once again, this is difficult because of the use of a generator here - `rows_from_file()` only yields rows, so there is no obvious mechanism for it to communicate back to the wrapping code that the detected format was CSV or TSV as opposed to JSON.

  I'm going to change `rows_from_file()` to return a `(generator, detected_format)` tuple.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/284#issuecomment-864416785
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/284
node_id: MDEyOklzc3VlQ29tbWVudDg2NDQxNjc4NQ==
created_at: 2021-06-19T14:54:41Z
updated_at: 2021-06-19T14:54:41Z
author_association: OWNER
body:
  ```pycon
  >>> db = sqlite_utils.Database(memory=True)
  >>> db["rowid_table"].insert({"name": "Cleo"})
  <Table rowid_table (name)>
  >>> db["regular_table"].insert({"id": 1, "name": "Cleo"}, pk="id")
  <Table regular_table (id, name)>
  >>> db["rowid_table"].pks
  ['rowid']
  >>> db["regular_table"].pks
  ['id']
  ```
  I think I need an introspection property for working out if a table is a `rowid` table or not.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/284#issuecomment-864416911
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/284
node_id: MDEyOklzc3VlQ29tbWVudDg2NDQxNjkxMQ==
created_at: 2021-06-19T14:55:45Z
updated_at: 2021-06-19T14:55:45Z
author_association: OWNER
body:
  https://github.com/simonw/sqlite-utils/blob/dc94f4bb8cfe922bb2f9c89f8f0f29092ea63133/sqlite_utils/db.py#L805-L810

  So I can indeed detect a `rowid` table by looking for no `is_pk` columns.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864417031
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/285
node_id: MDEyOklzc3VlQ29tbWVudDg2NDQxNzAzMQ==
created_at: 2021-06-19T14:56:45Z
updated_at: 2021-06-19T14:56:45Z
author_association: OWNER
body:
  ```pycon
  >>> db = sqlite_utils.Database(memory=True)
  >>> db["rowid_table"].insert({"name": "Cleo"})
  <Table rowid_table (name)>
  >>> db["regular_table"].insert({"id": 1, "name": "Cleo"}, pk="id")
  <Table regular_table (id, name)>
  >>> db["rowid_table"].pks
  ['rowid']
  >>> db["regular_table"].pks
  ['id']
  ```
  But that's because the `.pks` property hides the difference: https://github.com/simonw/sqlite-utils/blob/dc94f4bb8cfe922bb2f9c89f8f0f29092ea63133/sqlite_utils/db.py#L805-L810

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864417133
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/285
node_id: MDEyOklzc3VlQ29tbWVudDg2NDQxNzEzMw==
created_at: 2021-06-19T14:57:36Z
updated_at: 2021-06-19T14:57:36Z
author_association: OWNER
body:
  So the logic is:
  ```python
  [column.name for column in self.columns if column.is_pk]
  ```
  I need to decide on a property name. Existing names are documented here: https://sqlite-utils.datasette.io/en/stable/python-api.html#introspection

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864417493
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/285
node_id: MDEyOklzc3VlQ29tbWVudDg2NDQxNzQ5Mw==
created_at: 2021-06-19T15:00:43Z
updated_at: 2021-06-19T15:00:43Z
author_association: OWNER
body:
  I have to be careful about the language I use here. Here's the official definition: https://www.sqlite.org/rowidtable.html

  > A "rowid table" is any table in an SQLite schema that
  > 
  > -   is *not* a [virtual table](https://www.sqlite.org/vtab.html), and
  > -   is *not* a [WITHOUT ROWID](https://www.sqlite.org/withoutrowid.html) table.
  > 
  > Most tables in a typical SQLite database schema are rowid tables.
  > 
  > Rowid tables are distinguished by the fact that they all have a unique, non-NULL, signed 64-bit integer [rowid](https://www.sqlite.org/lang_createtable.html#rowid) that is used as the access key for the data in the underlying [B-tree](https://www.sqlite.org/fileformat2.html#btree) storage engine.

  So it's not correct to call a table a "rowid table" only if it is missing its own primary keys.

  Maybe `table.has_rowid` is the right language to use here? No, that's no good - because tables with their own primary keys usually also have a rowid.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864417765
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/285
node_id: MDEyOklzc3VlQ29tbWVudDg2NDQxNzc2NQ==
created_at: 2021-06-19T15:02:42Z
updated_at: 2021-06-19T15:02:42Z
author_association: OWNER
body:
  Some options:

  - `table.rowid_only`
  - `table.rowid_as_pk`
  - `table.no_pks`
  - `table.no_pk`
  - `table.uses_rowid`
  - `table.use_rowid`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864418188
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/285
node_id: MDEyOklzc3VlQ29tbWVudDg2NDQxODE4OA==
created_at: 2021-06-19T15:05:53Z
updated_at: 2021-06-19T15:05:53Z
author_association: OWNER
body:
  ```python
      @property
      def uses_rowid(self):
          return not any(column for column in self.columns if column.is_pk)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864418795
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/285
node_id: MDEyOklzc3VlQ29tbWVudDg2NDQxODc5NQ==
created_at: 2021-06-19T15:11:05Z
updated_at: 2021-06-19T15:11:14Z
author_association: OWNER
body:
  Actually I'm going to go with `use_rowid` instead - because the table doesn't inherently use a rowid itself, but you should use one if you want to query it in a way that gives you back a primary key.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/284#issuecomment-864419283
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/284
node_id: MDEyOklzc3VlQ29tbWVudDg2NDQxOTI4Mw==
created_at: 2021-06-19T15:15:34Z
updated_at: 2021-06-19T15:15:34Z
author_association: OWNER
body:
  I think this code is at fault: https://github.com/simonw/sqlite-utils/blob/5b257949d996fe43dc5d218d4308b88796a90740/sqlite_utils/db.py#L1017-L1023

  It's using `.pks` which adds `rowid` if it's missing.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/272#issuecomment-864476167
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDg2NDQ3NjE2Nw==
created_at: 2021-06-19T23:36:48Z
updated_at: 2021-06-19T23:36:48Z
author_association: OWNER
body:
  Wrote this up on my blog here: https://simonwillison.net/2021/Jun/19/sqlite-utils-memory/ - with a video demo here: https://www.youtube.com/watch?v=OUjd0rkc678

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1382#issuecomment-864480051
issue_url: https://api.github.com/repos/simonw/datasette/issues/1382
node_id: MDEyOklzc3VlQ29tbWVudDg2NDQ4MDA1MQ==
created_at: 2021-06-20T00:20:06Z
updated_at: 2021-06-20T00:21:02Z
author_association: OWNER
body:
  Yes you can - thanks for pointing this out, I've added a comment to the `install.sh` script in the `datasette-csvs` Glitch project:

  ```bash
  pip3 install -U --no-cache-dir -r requirements.txt --user && \
    mkdir -p .data && \
    rm .data/data.db || true && \
    for f in *.csv
      do
        # Add --encoding=latin-1 to the following if your CSVs use a different encoding:
        sqlite-utils insert .data/data.db ${f%.*} $f --csv
      done
  ```

  So if you edit that file in your own project and change the line to this:

        sqlite-utils insert .data/data.db ${f%.*} $f --csv --encoding=iso-8859-1

  It should fix this for you.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/286#issuecomment-864594956
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/286
node_id: MDEyOklzc3VlQ29tbWVudDg2NDU5NDk1Ng==
created_at: 2021-06-20T18:38:05Z
updated_at: 2021-06-20T18:38:05Z
author_association: OWNER
body:
  3.10 is out in Homebrew now (they turn that around so fast): https://formulae.brew.sh/formula/sqlite-utils

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/278#issuecomment-864621099
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/278
node_id: MDEyOklzc3VlQ29tbWVudDg2NDYyMTA5OQ==
created_at: 2021-06-20T22:39:57Z
updated_at: 2021-06-20T22:39:57Z
author_association: CONTRIBUTOR
body:
  Fair. I looked into it, it looks like it could be done, but it would be _a bit ugly_. I can upload and link a gist of my exploration. **Click** can parse a first argument while still recognizing it as a sub-command keyword. From there, the program could:
  1. ignore it preemptively if it matches a sub-command
  2. and/or check if a (db) file exists at the path.

  It would then also need to set a shared db argument variable.

  Click also makes it easy to parse arguments from environment variables.  If you're amenable, I may submit a patch for only that, which would update each sub-command to check for a DB/SQLITE_UTILS_DB environment variable. The goal would be usage that looks like: `DB=./convenient.db sqlite-utils [operation] [args]`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1368#issuecomment-865160132
issue_url: https://api.github.com/repos/simonw/datasette/issues/1368
node_id: MDEyOklzc3VlQ29tbWVudDg2NTE2MDEzMg==
created_at: 2021-06-21T16:07:06Z
updated_at: 2021-06-21T16:08:48Z
author_association: OWNER
body:
  A few tests failed - Black, the test that checks the docs mention the new hook - the most interesting failing test looks like this one:
  ```
              updated_metadata["databases"]["fixtures"]["queries"]["magic_parameters"][
                  "allow"
              ] = (allow if "query" in permissions else deny)
  >           cascade_app_client.ds._metadata = updated_metadata
  E           AttributeError: can't set attribute
  ```
  From https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/tests/test_permissions.py#L439-L467

  This test is directly manipulating `_metadata` purely for the purposes of simulating different permissions - I think updating it to manipulate `_local_metadata` instead would fix that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1368#issuecomment-865204472
issue_url: https://api.github.com/repos/simonw/datasette/issues/1368
node_id: MDEyOklzc3VlQ29tbWVudDg2NTIwNDQ3Mg==
created_at: 2021-06-21T17:11:37Z
updated_at: 2021-06-21T17:11:37Z
author_association: CONTRIBUTOR
body:
  If this is a concept ACK then I will move onto fixing the tests (adding new ones) and updating the documentation for the new plugin hook.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/290#issuecomment-865491922
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/290
node_id: MDEyOklzc3VlQ29tbWVudDg2NTQ5MTkyMg==
created_at: 2021-06-22T03:05:35Z
updated_at: 2021-06-22T03:05:35Z
author_association: OWNER
body:
  Potential names:

  - `db.query(sql)` - it's weird to have both this and `db.execute()` but it is at least short and memorable
  - `db.sql(sql)`
  - `db.execute_d(sql)` - ugly
  - `db.execute_dicts(sql)` - confusing
  - `db.execute_sql(sql)` - easily confused with `db.execute(sql)`

  I think `db.query(sql)` may be the best option here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/290#issuecomment-865495370
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/290
node_id: MDEyOklzc3VlQ29tbWVudDg2NTQ5NTM3MA==
created_at: 2021-06-22T03:14:30Z
updated_at: 2021-06-22T03:14:30Z
author_association: OWNER
body:
  One small problem with the existing method:
  https://github.com/simonw/sqlite-utils/blob/8cedc6a8b29180e68326f6b76f249d5e39e4b591/sqlite_utils/db.py#L362-L365

  It returns a full list, but what if the user would rather have a generator they can iterate over without loading the results into memory in one go?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/290#issuecomment-865497846
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/290
node_id: MDEyOklzc3VlQ29tbWVudDg2NTQ5Nzg0Ng==
created_at: 2021-06-22T03:21:38Z
updated_at: 2021-06-22T03:21:38Z
author_association: OWNER
body:
  The Python docs say: https://docs.python.org/3/library/sqlite3.html

  > To retrieve data after executing a SELECT statement, you can either treat the cursor as an iterator, call the cursor’s `fetchone()` method to retrieve a single matching row, or call `fetchall()` to get a list of the matching rows.

  Looking at the C source code, both `fetchmany()` and `fetchall()` work under the hood by assembling a Python list: https://github.com/python/cpython/blob/be1cb3214d09d4bf0288bc45f3c1f167f67e4514/Modules/_sqlite/cursor.c#L907-L972 - see calls to `PyList_Append()`

  So it looks like the most efficient way to iterate over a cursor may well be `for row in cursor:` - which I think calls this C function: https://github.com/python/cpython/blob/be1cb3214d09d4bf0288bc45f3c1f167f67e4514/Modules/_sqlite/cursor.c#L813-L876

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/290#issuecomment-865510796
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/290
node_id: MDEyOklzc3VlQ29tbWVudDg2NTUxMDc5Ng==
created_at: 2021-06-22T04:04:40Z
updated_at: 2021-06-22T04:04:48Z
author_association: OWNER
body:
  Still needs documentation, which will involve rewriting the whole [Executing queries](https://sqlite-utils.datasette.io/en/3.11/python-api.html#executing-queries) section.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/290#issuecomment-865511810
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/290
node_id: MDEyOklzc3VlQ29tbWVudDg2NTUxMTgxMA==
created_at: 2021-06-22T04:07:34Z
updated_at: 2021-06-22T18:26:21Z
author_association: OWNER
body:
  That documentation section is pretty weak at the moment - here's the whole thing:

  > ### Executing queries
  >
  > The `db.execute()` and `db.executescript()` methods provide wrappers around `.execute()` and `.executescript()` on the underlying SQLite connection. These wrappers log to the tracer function if one has been registered.
  > ```python
  > db = Database(memory=True)
  > db["dogs"].insert({"name": "Cleo"})
  > db.execute("update dogs set name = 'Cleopaws'")
  > ```
  > You can pass parameters as an optional second argument, using either a list or a dictionary. These will be correctly quoted and escaped.
  > ```python
  > # Using ? and a list:
  > db.execute("update dogs set name = ?", ["Cleopaws"])
  > # Or using :name and a dictionary:
  > db.execute("update dogs set name = :name", {"name": "Cleopaws"})
  > ```

  - Talks about `.execute()` - I want to talk about `.query()` instead
  - Doesn't clarify that `.execute()` returns a `Cursor` - and assumes you know what to do with one
  - Doesn't show an example of a `select` query at all
  - The "tracer function" bit is confusing (should at least link to docs further down)
  - For `UPDATE` should show how to access the number of rows modified (probably using `.execute()` there)

  It does at least cover the two types of parameters, though that could be bulked out.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/267#issuecomment-866182655
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/267
node_id: MDEyOklzc3VlQ29tbWVudDg2NjE4MjY1NQ==
created_at: 2021-06-22T17:24:03Z
updated_at: 2021-06-22T17:24:03Z
author_association: OWNER
body:
  I'm re-opening this as a research task because it may be possible to cleanly implement this using a `dict` subclass - some notes on that here: https://treyhunner.com/2019/04/why-you-shouldnt-inherit-from-list-and-dict-in-python/

  Since this would just be for adding methods (and maybe a property for returning the primary keys for a row) the usual disadvantages of subclassing `dict` described in that article shouldn't apply.

  One catch: dictionaries already have a `.update()` method! So would have to pick another name.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/267#issuecomment-866184260
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/267
node_id: MDEyOklzc3VlQ29tbWVudDg2NjE4NDI2MA==
created_at: 2021-06-22T17:26:18Z
updated_at: 2021-06-22T17:27:27Z
author_association: OWNER
body:
  If an`.update()` method doesn't work because it collides with an existing dictionary method a `.pk` property could still be nice:
  ```python
  for row in db["sometable"].rows:
      db["sometable"].update(row.pk, {"modified": 1})
  ```
      

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/291#issuecomment-866461926
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/291
node_id: MDEyOklzc3VlQ29tbWVudDg2NjQ2MTkyNg==
created_at: 2021-06-23T01:59:57Z
updated_at: 2021-06-23T01:59:57Z
author_association: OWNER
body:
  That shouldn't be failing: it's complaining about these: https://github.com/simonw/sqlite-utils/blob/02898bf7af4a4e484ecc8ec852d5fee98463277b/tests/test_register_function.py#L56-L67

  But I added `# noqa: F811` to them.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/291#issuecomment-866466388
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/291
node_id: MDEyOklzc3VlQ29tbWVudDg2NjQ2NjM4OA==
created_at: 2021-06-23T02:10:24Z
updated_at: 2021-06-23T02:10:24Z
author_association: OWNER
body:
  This already helped me spot a bug in a test: https://github.com/simonw/sqlite-utils/commit/90e211e3e2f36d2ff911ecf1afe4470ff45c7c0d#diff-4e8715c7a425ee52e74b7df4d34efd32e8c92f3e60bd51bc2e1ad5943b82032eL1160-L1161

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/290#issuecomment-868021624
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/290
node_id: MDEyOklzc3VlQ29tbWVudDg2ODAyMTYyNA==
created_at: 2021-06-24T23:17:38Z
updated_at: 2021-06-24T23:17:38Z
author_association: OWNER
body:
  The new documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#executing-queries

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/293#issuecomment-868125750
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/293
node_id: MDEyOklzc3VlQ29tbWVudDg2ODEyNTc1MA==
created_at: 2021-06-25T01:42:43Z
updated_at: 2021-06-25T01:42:43Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/293?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#293](https://codecov.io/gh/simonw/sqlite-utils/pull/293?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (ae0f46a) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/747be6057d09a4e5d9d726e29d5cf99b10c59dea?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (747be60) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/293/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/293?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main     #293   +/-   ##
  =======================================
    Coverage   96.03%   96.03%           
  =======================================
    Files           4        4           
    Lines        1994     1994           
  =======================================
    Hits         1915     1915           
    Misses         79       79           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/293?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/293?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [747be60...ae0f46a](https://codecov.io/gh/simonw/sqlite-utils/pull/293?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/293#issuecomment-868134040
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/293
node_id: MDEyOklzc3VlQ29tbWVudDg2ODEzNDA0MA==
created_at: 2021-06-25T01:49:44Z
updated_at: 2021-06-25T01:50:33Z
author_association: OWNER
body:
  Test failed on 3.10 with `numpy` on macOS:
  ```
  sqlite_utils/__init__.py:1: in <module>
  11
      from .db import Database
  12
  sqlite_utils/db.py:48: in <module>
  13
      import numpy as np  # type: ignore
  14
  ../../../hostedtoolcache/Python/3.10.0-beta.3/x64/lib/python3.10/site-packages/numpy/__init__.py:391: in <module>
  15
      raise RuntimeError(msg)
  16
  E   RuntimeError: Polyfit sanity test emitted a warning, most likely due to using a buggy Accelerate backend. If you compiled yourself, more information is available at https://numpy.org/doc/stable/user/building.html#accelerated-blas-lapack-libraries Otherwise report this to the vendor that provided NumPy.
  17
  E   RankWarning: Polyfit may be poorly conditioned
  18
  Error: Process completed with exit code 4.
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/37#issuecomment-868881033
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/37
node_id: MDEyOklzc3VlQ29tbWVudDg2ODg4MTAzMw==
created_at: 2021-06-25T23:23:49Z
updated_at: 2021-06-25T23:23:49Z
author_association: OWNER
body:
  Twitter conversation about how to add types to the `.create_table(columns=)` parameter: https://twitter.com/simonw/status/1408532867592818693

  > Anyone know how to write a mypy type definition for this?
  >
  > {"id": int, "name": str, "image": bytes, "weight": float}
  >
  > It's a dict where keys are strings and values are one of int/str/bytes/float (weird API design I know, but I designed this long before I was thinking about mypy)

  Looks like this could work:

  ```python
      def create_table(
          self,
          name,
          columns: Dict[str, Union[Type[int], Type[bytes], Type[str], Type[float]]],
          pk=None,
          foreign_keys=None,
          column_order=None,
          not_null=None,
          defaults=None,
          hash_id=None,
          extracts=None,
      ):
  ```
  Except... that method can optionally also accept `numpy` types if `numpy` is installed. I don't know if it's possible to dynamically change a signature based on an import, since `mypy` is a static type analyzer and doesn't ever execute the code.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/37#issuecomment-868881190
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/37
node_id: MDEyOklzc3VlQ29tbWVudDg2ODg4MTE5MA==
created_at: 2021-06-25T23:24:28Z
updated_at: 2021-06-25T23:24:28Z
author_association: OWNER
body:
  Maybe I could release a separate Python package `types-sqlite-utils-numpy` which adds an over-ridden type definition that includes the `numpy` types?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1384#issuecomment-869069655
issue_url: https://api.github.com/repos/simonw/datasette/issues/1384
node_id: MDEyOklzc3VlQ29tbWVudDg2OTA2OTY1NQ==
created_at: 2021-06-26T22:36:14Z
updated_at: 2021-06-26T22:37:37Z
author_association: OWNER
body:
  Documentation for the new hook is now live at https://docs.datasette.io/en/latest/plugin_hooks.html#get-metadata-datasette-key-database-table-fallback

  Link to the current snapshot of that documentation: https://github.com/simonw/datasette/blob/05a312caf3debb51aa1069939923a49e21cd2bd1/docs/plugin_hooks.rst#get-metadata-datasette-key-database-table-fallback

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1384#issuecomment-869069926
issue_url: https://api.github.com/repos/simonw/datasette/issues/1384
node_id: MDEyOklzc3VlQ29tbWVudDg2OTA2OTkyNg==
created_at: 2021-06-26T22:40:15Z
updated_at: 2021-06-26T22:40:53Z
author_association: OWNER
body:
  The documentation says:

  > **datasette**:     You can use this to access plugin configuration options via `datasette.plugin_config(your_plugin_name)`, or to execute SQL queries.

  That's not accurate: since the plugin hook is a regular function, not an awaitable, you can't use it to run `await db.execute(...)` so you can't execute SQL queries.

  I can fix this with the await-me-maybe pattern, used for other plugin hooks: https://simonwillison.net/2020/Sep/2/await-me-maybe/

  BUT... that requires changing the `ds.metadata()` function to be awaitable, which will affect every existing plugn that uses that documented internal method!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1384#issuecomment-869070076
issue_url: https://api.github.com/repos/simonw/datasette/issues/1384
node_id: MDEyOklzc3VlQ29tbWVudDg2OTA3MDA3Ng==
created_at: 2021-06-26T22:42:21Z
updated_at: 2021-06-26T22:42:21Z
author_association: OWNER
body:
  Hmmm... that's tricky, since one of the most obvious ways to use this hook is to load metadata from database tables using SQL queries.

  @brandonrobertz do you have a working example of using this hook to populate metadata from database tables I can try?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1384#issuecomment-869070348
issue_url: https://api.github.com/repos/simonw/datasette/issues/1384
node_id: MDEyOklzc3VlQ29tbWVudDg2OTA3MDM0OA==
created_at: 2021-06-26T22:46:18Z
updated_at: 2021-06-26T22:46:18Z
author_association: OWNER
body:
  Here's where the plugin hook is called, demonstrating the `fallback=` argument: https://github.com/simonw/datasette/blob/05a312caf3debb51aa1069939923a49e21cd2bd1/datasette/app.py#L426-L472

  I'm not convinced of the use-case for passing `fallback=` to the hook here - is there a reason a plugin might care whether fallback is `True` or `False`, seeing as the `metadata()` method already respects that fallback logic on line 459?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1384#issuecomment-869070941
issue_url: https://api.github.com/repos/simonw/datasette/issues/1384
node_id: MDEyOklzc3VlQ29tbWVudDg2OTA3MDk0MQ==
created_at: 2021-06-26T22:53:34Z
updated_at: 2021-06-26T22:53:34Z
author_association: OWNER
body:
  The `await` thing is worrying me a lot - it feels like this plugin hook is massively less useful if it can't make it's own DB queries and generally do asynchronous stuff - but I'd also like not to break every existing plugin that calls `datasette.metadata(...)`.

  One solution that could work: introduce a new method, maybe `await datasette.get_metadata(...)`, which uses this plugin hook - and keep the existing `datasette.metadata()` method (which doesn't call the hook) around. This would ensure existing plugins keep on working.

  Then, upgrade those plugins separately - with the goal of deprecating and removing `.metadata()` entirely in Datasette 1.0 - having upgraded the plugins in the meantime.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1384#issuecomment-869071167
issue_url: https://api.github.com/repos/simonw/datasette/issues/1384
node_id: MDEyOklzc3VlQ29tbWVudDg2OTA3MTE2Nw==
created_at: 2021-06-26T22:55:36Z
updated_at: 2021-06-26T22:55:36Z
author_association: OWNER
body:
  Just realized I already have an issue open for this, at #860. I'm going to close that and continue work on this in this issue.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1384#issuecomment-869071435
issue_url: https://api.github.com/repos/simonw/datasette/issues/1384
node_id: MDEyOklzc3VlQ29tbWVudDg2OTA3MTQzNQ==
created_at: 2021-06-26T22:59:26Z
updated_at: 2021-06-26T22:59:26Z
author_association: OWNER
body:
  The other alternative is to finish the work to build a `_metadata` internal table, see #1168. The idea there was that if we want to support efficient pagination and search across the metadata for thousands of attached tables powering it with a plugin hook doesn't work well - we don't want to call the hook once for every one of 1,000+ tables just to implement the homepage.

  So instead, all metadata for all attached databases would be loaded into an in-memory database called `_metadata`. Plugins that want to modify stored metadata could then do so by directly writing to that table.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1384#issuecomment-869071790
issue_url: https://api.github.com/repos/simonw/datasette/issues/1384
node_id: MDEyOklzc3VlQ29tbWVudDg2OTA3MTc5MA==
created_at: 2021-06-26T23:04:12Z
updated_at: 2021-06-26T23:04:12Z
author_association: OWNER
body:
  > Hmmm... that's tricky, since one of the most obvious ways to use this hook is to load metadata from database tables using SQL queries.
  > 
  > @brandonrobertz do you have a working example of using this hook to populate metadata from database tables I can try?

  Answering my own question: here's how Brandon implements it in his `datasette-live-config` plugin: https://github.com/next-LI/datasette-live-config/blob/72e335e887f1c69c54c6c2441e07148955b0fc9f/datasette_live_config/__init__.py#L50-L160

  That's using a completely separate SQLite connection (actually wrapped in `sqlite-utils`) and making blocking synchronous calls to it.

  This is a pragmatic solution, which works - and likely performs just fine, because SQL queries like this against a small database are so fast that not running them asynchronously isn't actually a problem.

  But... it's weird. Everywhere else in Datasette land uses `await db.execute(...)` - but here's an example where users are encouraged to use blocking calls instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1384#issuecomment-869074182
issue_url: https://api.github.com/repos/simonw/datasette/issues/1384
node_id: MDEyOklzc3VlQ29tbWVudDg2OTA3NDE4Mg==
created_at: 2021-06-26T23:37:42Z
updated_at: 2021-06-26T23:37:42Z
author_association: CONTRIBUTOR
body:
  > > Hmmm... that's tricky, since one of the most obvious ways to use this hook is to load metadata from database tables using SQL queries.
  > > @brandonrobertz do you have a working example of using this hook to populate metadata from database tables I can try?
  > 
  > Answering my own question: here's how Brandon implements it in his `datasette-live-config` plugin: https://github.com/next-LI/datasette-live-config/blob/72e335e887f1c69c54c6c2441e07148955b0fc9f/datasette_live_config/__init__.py#L50-L160
  > 
  > That's using a completely separate SQLite connection (actually wrapped in `sqlite-utils`) and making blocking synchronous calls to it.
  > 
  > This is a pragmatic solution, which works - and likely performs just fine, because SQL queries like this against a small database are so fast that not running them asynchronously isn't actually a problem.
  > 
  > But... it's weird. Everywhere else in Datasette land uses `await db.execute(...)` - but here's an example where users are encouraged to use blocking calls instead.

  _Ideally_ this hook would be asynchronous, but when I started down that path I quickly realized how large of a change this would be, since metadata gets used synchronously across the entire Datasette codebase. (And calling async code from sync is non-trivial.)

  In my live-configuration implementation I use synchronous reads using a persistent sqlite connection. This works pretty well in practice, but I agree it's limiting. My thinking around this was to go with the path of least change as `Datasette.metadata()` is a critical core function.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1384#issuecomment-869074701
issue_url: https://api.github.com/repos/simonw/datasette/issues/1384
node_id: MDEyOklzc3VlQ29tbWVudDg2OTA3NDcwMQ==
created_at: 2021-06-26T23:45:18Z
updated_at: 2021-06-26T23:45:37Z
author_association: CONTRIBUTOR
body:
  > Here's where the plugin hook is called, demonstrating the `fallback=` argument:
  > 
  > https://github.com/simonw/datasette/blob/05a312caf3debb51aa1069939923a49e21cd2bd1/datasette/app.py#L426-L472
  > 
  > I'm not convinced of the use-case for passing `fallback=` to the hook here - is there a reason a plugin might care whether fallback is `True` or `False`, seeing as the `metadata()` method already respects that fallback logic on line 459?

  I think you're right. I can't think of a reason why the plugin would care about the `fallback` parameter since plugins are currently mandated to return a full, global metadata dict.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1384#issuecomment-869075368
issue_url: https://api.github.com/repos/simonw/datasette/issues/1384
node_id: MDEyOklzc3VlQ29tbWVudDg2OTA3NTM2OA==
created_at: 2021-06-26T23:53:55Z
updated_at: 2021-06-26T23:53:55Z
author_association: OWNER
body:
  Great, let's drop fallback then.

  My instinct at the moment is to ship this plugin hook as-is but with a warning that it may change before Datasette 1.0 - then before 1.0 either figure out an async variant or finish the database-backed metadata concept from #1168 and recommend that as an alternative.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1168#issuecomment-869076254
issue_url: https://api.github.com/repos/simonw/datasette/issues/1168
node_id: MDEyOklzc3VlQ29tbWVudDg2OTA3NjI1NA==
created_at: 2021-06-27T00:03:16Z
updated_at: 2021-06-27T00:05:51Z
author_association: CONTRIBUTOR
body:
  > Related: Here's an implementation of a `get_metadata()` plugin hook by @brandonrobertz [next-LI@3fd8ce9](https://github.com/next-LI/datasette/commit/3fd8ce91f3108c82227bf65ff033929426c60437)

  Here's a plugin that implements metadata-within-DBs: [next-LI/datasette-live-config](https://github.com/next-LI/datasette-live-config)

  How it works: If a database has a `__metadata` table, then it gets parsed and included in the global metadata. It also implements a database-action hook with a UI for managing config.

  More context: https://github.com/next-LI/datasette-live-config/blob/72e335e887f1c69c54c6c2441e07148955b0fc9f/datasette_live_config/__init__.py#L109-L140

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1385#issuecomment-869105782
issue_url: https://api.github.com/repos/simonw/datasette/issues/1385
node_id: MDEyOklzc3VlQ29tbWVudDg2OTEwNTc4Mg==
created_at: 2021-06-27T05:48:55Z
updated_at: 2021-09-13T17:29:30Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1385?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1385](https://codecov.io/gh/simonw/datasette/pull/1385?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (db78094) into [main](https://codecov.io/gh/simonw/datasette/commit/ea627baccf980d7d8ebc9e1ffff1fe34d556e56f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (ea627ba) will **not change** coverage.
  > The diff coverage is `n/a`.

  > :exclamation: Current head db78094 differs from pull request most recent head 8d78c8c. Consider uploading reports for the commit 8d78c8c to get more accurate results
  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1385/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1385?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1385   +/-   ##
  =======================================
    Coverage   91.70%   91.70%           
  =======================================
    Files          34       34           
    Lines        4364     4364           
  =======================================
    Hits         4002     4002           
    Misses        362      362           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1385?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1385?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [ea627ba...8d78c8c](https://codecov.io/gh/simonw/datasette/pull/1385?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1386#issuecomment-869677851
issue_url: https://api.github.com/repos/simonw/datasette/issues/1386
node_id: MDEyOklzc3VlQ29tbWVudDg2OTY3Nzg1MQ==
created_at: 2021-06-28T13:18:50Z
updated_at: 2021-06-28T13:18:50Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1386?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1386](https://codecov.io/gh/simonw/datasette/pull/1386?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (e974ed1) into [main](https://codecov.io/gh/simonw/datasette/commit/ea627baccf980d7d8ebc9e1ffff1fe34d556e56f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (ea627ba) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1386/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1386?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1386   +/-   ##
  =======================================
    Coverage   91.70%   91.70%           
  =======================================
    Files          34       34           
    Lines        4364     4364           
  =======================================
    Hits         4002     4002           
    Misses        362      362           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1386?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1386?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [ea627ba...e974ed1](https://codecov.io/gh/simonw/datasette/pull/1386?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1101#issuecomment-869812567
issue_url: https://api.github.com/repos/simonw/datasette/issues/1101
node_id: MDEyOklzc3VlQ29tbWVudDg2OTgxMjU2Nw==
created_at: 2021-06-28T16:06:57Z
updated_at: 2021-06-28T16:07:24Z
author_association: OWNER
body:
  Relevant blog post: https://simonwillison.net/2021/Jun/25/streaming-large-api-responses/ - including notes on efficiently streaming formats with some kind of separator in between the records (regular JSON).

  > Some export formats are friendlier for streaming than others. CSV and TSV are pretty easy to stream, as is newline-delimited JSON.
  > 
  > Regular JSON requires a bit more thought: you can output a `[` character, then output each row in a stream with a comma suffix, then skip the comma for the last row and output a `]`. Doing that requires peeking ahead (looping two at a time) to verify that you haven't yet reached the end.
  > 
  > Or... Martin De Wulf [pointed out](https://twitter.com/madewulf/status/1405559088994467844) that you can output the first row, then output every other row with a preceeding comma---which avoids the whole "iterate two at a time" problem entirely.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1387#issuecomment-873134866
issue_url: https://api.github.com/repos/simonw/datasette/issues/1387
node_id: MDEyOklzc3VlQ29tbWVudDg3MzEzNDg2Ng==
created_at: 2021-07-02T16:58:52Z
updated_at: 2021-07-02T16:58:52Z
author_association: OWNER
body:
  What's weird here is that the URL itself is correct - it starts with `/collection-analysis/` as expected - but the hostname is wrong.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1387#issuecomment-873136440
issue_url: https://api.github.com/repos/simonw/datasette/issues/1387
node_id: MDEyOklzc3VlQ29tbWVudDg3MzEzNjQ0MA==
created_at: 2021-07-02T17:01:48Z
updated_at: 2021-07-02T17:01:48Z
author_association: OWNER
body:
  Here's what's happening: https://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/views/table.py#L827-L829

  This is being run through `absolute_url()` - defined here: https://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/app.py#L633-L637

  That's because the `next_url` in the JSON needs to be a full URL that a client can retrieve - as  opposed to the other links on that page which are all relative links that start with `/`: https://github.com/simonw/datasette/blob/ea627baccf980d7d8ebc9e1ffff1fe34d556e56f/datasette/templates/_table.html#L11-L15

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1387#issuecomment-873137935
issue_url: https://api.github.com/repos/simonw/datasette/issues/1387
node_id: MDEyOklzc3VlQ29tbWVudDg3MzEzNzkzNQ==
created_at: 2021-07-02T17:03:36Z
updated_at: 2021-07-02T17:03:36Z
author_association: OWNER
body:
  And the links to apply a facet value are broken too! https://ilsweb.cincinnatilibrary.org/collection-analysis/current_collection-3d4a4b7/bib?_facet=bib_level_callnumber

  ```json
          {
            "value": "g l fiction",
            "label": "g l fiction",
            "count": 212,
            "toggle_url": "https://127.0.0.1:8010/collection-analysis/current_collection-3d4a4b7/bib.json?_facet=bib_level_callnumber&bib_level_callnumber=g+l+fiction",
            "selected": false
          }
  ```
  Same problem: https://github.com/simonw/datasette/blob/ea627baccf980d7d8ebc9e1ffff1fe34d556e56f/datasette/facets.py#L251-L261

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1387#issuecomment-873139138
issue_url: https://api.github.com/repos/simonw/datasette/issues/1387
node_id: MDEyOklzc3VlQ29tbWVudDg3MzEzOTEzOA==
created_at: 2021-07-02T17:05:47Z
updated_at: 2021-07-02T17:05:47Z
author_association: OWNER
body:
  In this case the proxy is Apache. So there are a couple of potential fixes:

  - Configure Apache to pass the original HTTP request `Host:` header through to the proxied application. This should then be documented.
  - Add a new optional feature to Datasette called something like `base_host` which, if set, is always used in place of the host in `request.url` when constructing new URLs.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1387#issuecomment-873140742
issue_url: https://api.github.com/repos/simonw/datasette/issues/1387
node_id: MDEyOklzc3VlQ29tbWVudDg3MzE0MDc0Mg==
created_at: 2021-07-02T17:08:40Z
updated_at: 2021-07-02T17:08:40Z
author_association: OWNER
body:
  `ProxyPreserveHost On` is the Apache setting - it defaults to `Off`: https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypreservehost 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1387#issuecomment-873141222
issue_url: https://api.github.com/repos/simonw/datasette/issues/1387
node_id: MDEyOklzc3VlQ29tbWVudDg3MzE0MTIyMg==
created_at: 2021-07-02T17:09:32Z
updated_at: 2021-07-02T17:09:32Z
author_association: OWNER
body:
  I'm going to add this to the suggested Apache configuration at https://docs.datasette.io/en/stable/deploying.html#apache-proxy-configuration

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1387#issuecomment-873156408
issue_url: https://api.github.com/repos/simonw/datasette/issues/1387
node_id: MDEyOklzc3VlQ29tbWVudDg3MzE1NjQwOA==
created_at: 2021-07-02T17:37:30Z
updated_at: 2021-07-02T17:37:30Z
author_association: OWNER
body:
  Updated documentation is here: https://docs.datasette.io/en/latest/deploying.html#apache-proxy-configuration

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1387#issuecomment-873166836
issue_url: https://api.github.com/repos/simonw/datasette/issues/1387
node_id: MDEyOklzc3VlQ29tbWVudDg3MzE2NjgzNg==
created_at: 2021-07-02T17:58:23Z
updated_at: 2021-07-02T17:58:23Z
author_association: NONE
body:
  Thanks Simon for nailing that one down! It does seem a little confusing that the ProxyPreservehost option is set to Off By default, but this config totally did the trick and fixed the issue

  ```
  <Location /collection-analysis/>
     ProxyPass http://127.0.0.1:8010/collection-analysis/
     ProxyPreservehost On
  </Location>
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1388#issuecomment-875740085
issue_url: https://api.github.com/repos/simonw/datasette/issues/1388
node_id: MDEyOklzc3VlQ29tbWVudDg3NTc0MDA4NQ==
created_at: 2021-07-07T16:17:08Z
updated_at: 2021-07-07T16:17:08Z
author_association: OWNER
body:
  Looks pretty easy to implement - here's a hint from Uvicorn source code: https://github.com/encode/uvicorn/blob/b5af1049e63c059dc750a450c807b9768f91e906/uvicorn/main.py#L368

  Need to work out a simple pattern for testing this too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1388#issuecomment-875741410
issue_url: https://api.github.com/repos/simonw/datasette/issues/1388
node_id: MDEyOklzc3VlQ29tbWVudDg3NTc0MTQxMA==
created_at: 2021-07-07T16:18:50Z
updated_at: 2021-07-07T16:18:50Z
author_association: OWNER
body:
  You could actually run Datasette like this today without modifications by running a thin Python script that imports from `datasette.app`, instantiates the ASGI app and passes that to `uvicorn.run` - but I like this as a supported feature too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1388#issuecomment-875742910
issue_url: https://api.github.com/repos/simonw/datasette/issues/1388
node_id: MDEyOklzc3VlQ29tbWVudDg3NTc0MjkxMA==
created_at: 2021-07-07T16:20:50Z
updated_at: 2021-07-07T16:23:02Z
author_association: OWNER
body:
  I wonder if `--fd` is worth supporting too? Uvicorn documentation says that's useful for running under process managers, I'd want to understand exactly how to use that (and test it) before adding the feature though.

  https://www.uvicorn.org/settings/

  Docs on how to use a process manager: https://www.uvicorn.org/deployment/#supervisor

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1388#issuecomment-876213177
issue_url: https://api.github.com/repos/simonw/datasette/issues/1388
node_id: MDEyOklzc3VlQ29tbWVudDg3NjIxMzE3Nw==
created_at: 2021-07-08T07:47:17Z
updated_at: 2021-07-08T07:47:17Z
author_association: CONTRIBUTOR
body:
  > This sounds like a valuable feature for people running Datasette behind a proxy.

  Yes, in some cases it is easer to use e.g. Apache's [ProxyPass Directive](https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypass) with Unix Domain Socket like  `unix:/home/www.socket|http://localhost/whatever/`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/268#issuecomment-876428348
issue_url: https://api.github.com/repos/simonw/datasette/issues/268
node_id: MDEyOklzc3VlQ29tbWVudDg3NjQyODM0OA==
created_at: 2021-07-08T13:13:12Z
updated_at: 2021-07-08T13:13:12Z
author_association: NONE
body:
  I had setup a full text search on my instance of Datasette for title data for our public library, and was noticing that some of the features of the SQLite FTS weren't working as expected ... and maybe the issue is in the `escape_fts()` function

  ![image](https://user-images.githubusercontent.com/9308268/124925900-f1ea8b00-dfca-11eb-895e-59cc083d6524.png)
  vs removing the function...
  ![image](https://user-images.githubusercontent.com/9308268/124925971-0464c480-dfcb-11eb-8fbf-8e9b5d6e0861.png)

  Also, on the issue of sorting by rank by default .. perhaps something like this could work for the baked-in default SQL query for Datasette?
  ![image](https://user-images.githubusercontent.com/9308268/124927191-5a863780-dfcc-11eb-9908-3f63577d5ff5.png)

  [link to the above search in my instance of Datasette](https://ilsweb.cincinnatilibrary.org/collection-analysis/current_collection-87a9011?sql=with+fts_search+as+%28%0D%0A++select%0D%0A++rowid%2C%0D%0A++rank%0D%0A++++from%0D%0A++++++bib_fts%0D%0A++++where%0D%0A++++++bib_fts+match+%3Asearch%0D%0A%29%0D%0A%0D%0Aselect%0D%0A++%0D%0A++bib_record_num%2C%0D%0A++creation_date%2C%0D%0A++record_last_updated%2C%0D%0A++isbn%2C%0D%0A++best_author%2C%0D%0A++best_title%2C%0D%0A++publisher%2C%0D%0A++publish_year%2C%0D%0A++bib_level_callnumber%2C%0D%0A++indexed_subjects%0D%0Afrom%0D%0A++fts_search%0D%0A++join+bib+on+bib.rowid+%3D+fts_search.rowid%0D%0A++%0D%0Aorder+by%0D%0Arank%0D%0A&search=black+death+NOT+fiction)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/268#issuecomment-876616414
issue_url: https://api.github.com/repos/simonw/datasette/issues/268
node_id: MDEyOklzc3VlQ29tbWVudDg3NjYxNjQxNA==
created_at: 2021-07-08T17:29:04Z
updated_at: 2021-07-08T17:29:04Z
author_association: OWNER
body:
  > I had setup a full text search on my instance of Datasette for title data for our public library, and was noticing that some of the features of the SQLite FTS weren't working as expected ... and maybe the issue is in the `escape_fts()` function

  That's a deliberate feature (albeit controversial, see #759) - part of the main problem here is that it's easy to construct a SQLite full-text search string which results in a database error. This is a bad user-experience!

  You can opt-in to raw SQL queries by appending `?_searchmode=raw` to the page, see https://docs.datasette.io/en/stable/full_text_search.html#advanced-sqlite-search-queries

  But maybe there should be an option for turning that on by default without needing the query string?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1389#issuecomment-876618582
issue_url: https://api.github.com/repos/simonw/datasette/issues/1389
node_id: MDEyOklzc3VlQ29tbWVudDg3NjYxODU4Mg==
created_at: 2021-07-08T17:32:40Z
updated_at: 2021-07-08T17:32:40Z
author_association: OWNER
body:
  This makes sense to me since other useful querystring arguments like this can be set as defaults in the `metadata.json` configuration.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1389#issuecomment-876618847
issue_url: https://api.github.com/repos/simonw/datasette/issues/1389
node_id: MDEyOklzc3VlQ29tbWVudDg3NjYxODg0Nw==
created_at: 2021-07-08T17:33:08Z
updated_at: 2021-07-08T17:33:08Z
author_association: OWNER
body:
  Relevant documentation: https://docs.datasette.io/en/0.57.1/full_text_search.html#advanced-sqlite-search-queries

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1389#issuecomment-876619271
issue_url: https://api.github.com/repos/simonw/datasette/issues/1389
node_id: MDEyOklzc3VlQ29tbWVudDg3NjYxOTI3MQ==
created_at: 2021-07-08T17:33:49Z
updated_at: 2021-07-08T17:33:49Z
author_association: OWNER
body:
  Relevant code: https://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/views/table.py#L491-L498

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1389#issuecomment-876619531
issue_url: https://api.github.com/repos/simonw/datasette/issues/1389
node_id: MDEyOklzc3VlQ29tbWVudDg3NjYxOTUzMQ==
created_at: 2021-07-08T17:34:16Z
updated_at: 2021-07-08T17:34:16Z
author_association: OWNER
body:
  If I implement this I'll also set it up so `?_searchmode=default` can be used to over-ride that and go back to the default behaviour.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/268#issuecomment-876721585
issue_url: https://api.github.com/repos/simonw/datasette/issues/268
node_id: MDEyOklzc3VlQ29tbWVudDg3NjcyMTU4NQ==
created_at: 2021-07-08T20:22:17Z
updated_at: 2021-07-08T20:22:17Z
author_association: NONE
body:
  I do like the idea of there being a option for turning that on by default so that you could use those terms in the default "Search" bar presented when you browse to a table where FTS has been enabled. Maybe even a small inline pop up with a short bit explaining the FTS feature and the keywords (e.g. case matters). What are the side-effects of turning that on in the query string, or even by default as you suggested? I see that you stated in the docs... "to ensure they do not cause any confusion for users who are not aware of them", but I'm not sure what those could be.

  Isn't it the case that those keywords are only picked up by sqlite in where you're using the MATCH clause?

  Seems like a really powerful feature (even though there are a lot of hurdles around setting it up in the sqlite db ... sqlite-utils makes that so simple by the way!)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1389#issuecomment-877681031
issue_url: https://api.github.com/repos/simonw/datasette/issues/1389
node_id: MDEyOklzc3VlQ29tbWVudDg3NzY4MTAzMQ==
created_at: 2021-07-10T18:17:29Z
updated_at: 2021-07-10T18:17:29Z
author_association: OWNER
body:
  I don't like `?_searchmode=default` because it suggests "use the default" - but it actually over-rides the default that was specified by `"searchmode": "raw"` in `metadata.json`.

  I'm going with `?_searchmode=escaped` instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1391#issuecomment-877682533
issue_url: https://api.github.com/repos/simonw/datasette/issues/1391
node_id: MDEyOklzc3VlQ29tbWVudDg3NzY4MjUzMw==
created_at: 2021-07-10T18:28:05Z
updated_at: 2021-07-10T18:28:05Z
author_association: OWNER
body:
  Here's the test in question: https://github.com/simonw/datasette/blob/a6c55afe8c82ead8deb32f90c9324022fd422324/tests/test_api.py#L2033-L2046

  Various other places in the test code also need changing - anything that calls `supports_generated_columns()`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1391#issuecomment-877686784
issue_url: https://api.github.com/repos/simonw/datasette/issues/1391
node_id: MDEyOklzc3VlQ29tbWVudDg3NzY4Njc4NA==
created_at: 2021-07-10T18:56:03Z
updated_at: 2021-07-10T18:56:03Z
author_association: OWNER
body:
  Here's the SQL used to generate the table for the test: https://github.com/simonw/datasette/blob/02b19c7a9afd328f22040ab33b5c1911cd904c7c/tests/fixtures.py#L723-L733

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1391#issuecomment-877687196
issue_url: https://api.github.com/repos/simonw/datasette/issues/1391
node_id: MDEyOklzc3VlQ29tbWVudDg3NzY4NzE5Ng==
created_at: 2021-07-10T18:58:40Z
updated_at: 2021-07-10T18:58:40Z
author_association: OWNER
body:
  I can use the `extra_databases` mechanism as demonstrated here: https://github.com/simonw/datasette/blob/9552414e1f968c6fc704031cec349c05e6bc2371/tests/test_canned_queries.py#L8-L12

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1391#issuecomment-877691427
issue_url: https://api.github.com/repos/simonw/datasette/issues/1391
node_id: MDEyOklzc3VlQ29tbWVudDg3NzY5MTQyNw==
created_at: 2021-07-10T19:26:00Z
updated_at: 2021-07-10T19:26:00Z
author_association: OWNER
body:
  I had to run the tests locally on my macOS laptop using `pysqlite3` to get a version that supported generated columns - wrote up a TIL about that here: https://til.simonwillison.net/sqlite/pysqlite3-on-macos

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1391#issuecomment-877691558
issue_url: https://api.github.com/repos/simonw/datasette/issues/1391
node_id: MDEyOklzc3VlQ29tbWVudDg3NzY5MTU1OA==
created_at: 2021-07-10T19:26:57Z
updated_at: 2021-07-10T19:26:57Z
author_association: OWNER
body:
  The `https://latest.datasette.io/fixtures.db` file no longer includes generated columns, which will help avoid confusion such as seen in #1376.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1388#issuecomment-877715654
issue_url: https://api.github.com/repos/simonw/datasette/issues/1388
node_id: MDEyOklzc3VlQ29tbWVudDg3NzcxNTY1NA==
created_at: 2021-07-10T23:15:06Z
updated_at: 2021-07-10T23:15:06Z
author_association: OWNER
body:
  I can run tests against it using `httpx`: https://www.python-httpx.org/advanced/#usage_1

  > ```pycon
  > >>> import httpx
  > >>> # Connect to the Docker API via a Unix Socket.
  > >>> transport = httpx.HTTPTransport(uds="/var/run/docker.sock")
  > >>> client = httpx.Client(transport=transport)
  > >>> response = client.get("http://docker/info")
  > ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1388#issuecomment-877716156
issue_url: https://api.github.com/repos/simonw/datasette/issues/1388
node_id: MDEyOklzc3VlQ29tbWVudDg3NzcxNjE1Ng==
created_at: 2021-07-10T23:22:21Z
updated_at: 2021-07-10T23:22:21Z
author_association: OWNER
body:
  I don't have the Datasette test suite running on Windows yet, but I'd like it to run there some day - so ideally this test would be skipped if Unix domain sockets are not supported by the underlying operating system.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1388#issuecomment-877716359
issue_url: https://api.github.com/repos/simonw/datasette/issues/1388
node_id: MDEyOklzc3VlQ29tbWVudDg3NzcxNjM1OQ==
created_at: 2021-07-10T23:24:58Z
updated_at: 2021-07-10T23:24:58Z
author_association: OWNER
body:
  Apparently Windows 10 has Unix domain socket support: https://bugs.python.org/issue33408

  > Unix socket (AF_UNIX) is now avalible in Windows 10 (April 2018 Update). Please add Python support for it.
  > More details about it on https://blogs.msdn.microsoft.com/commandline/2017/12/19/af_unix-comes-to-windows/

  But it's not clear if this is going to work. That same issue thread (the issue is still open) suggests using `hasattr(socket, 'AF_UNIX'))` to detect support in tests.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1388#issuecomment-877716993
issue_url: https://api.github.com/repos/simonw/datasette/issues/1388
node_id: MDEyOklzc3VlQ29tbWVudDg3NzcxNjk5Mw==
created_at: 2021-07-10T23:34:02Z
updated_at: 2021-07-10T23:34:02Z
author_association: OWNER
body:
  Figured out an example nginx configuration. This in `nginx.conf`:

      daemon off;
      events {
        worker_connections  1024;
      }
      http {
        server {
          listen 8092;
          location / {
            proxy_pass              http://datasette;
            proxy_set_header        X-Real-IP $remote_addr;
            proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
          }
        }
        upstream datasette {
          server unix:/tmp/datasette.sock;
        }
      }

  Then run `datasette --uds /tmp/datasette.sock`

  Then run nginx like this:

      nginx -c ./nginx.conf

  Then hits to `http://localhost:8092/` will be proxied to Datasette.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1388#issuecomment-877717262
issue_url: https://api.github.com/repos/simonw/datasette/issues/1388
node_id: MDEyOklzc3VlQ29tbWVudDg3NzcxNzI2Mg==
created_at: 2021-07-10T23:37:54Z
updated_at: 2021-07-10T23:37:54Z
author_association: OWNER
body:
  > I wonder if `--fd` is worth supporting too?

  I'm going to hold off on implementing this until someone asks for it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/511#issuecomment-877717791
issue_url: https://api.github.com/repos/simonw/datasette/issues/511
node_id: MDEyOklzc3VlQ29tbWVudDg3NzcxNzc5MQ==
created_at: 2021-07-10T23:45:35Z
updated_at: 2021-07-10T23:45:35Z
author_association: OWNER
body:
  > Trying to run on Windows today, I get an error from the utils/asgi.py module.
  > 
  > It's trying `from os import EX_CANTCREAT` which is Unix-only. I commented this line out, and (so far) it's working.

  Good news: that line was removed in #1094.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/511#issuecomment-877718286
issue_url: https://api.github.com/repos/simonw/datasette/issues/511
node_id: MDEyOklzc3VlQ29tbWVudDg3NzcxODI4Ng==
created_at: 2021-07-10T23:53:29Z
updated_at: 2021-07-10T23:53:29Z
author_association: OWNER
body:
  Test suite on Windows seems to run a lot slower:

  <img width="391" alt="Ran_Black__refs__1388_·_simonw_datasette_518cb06" src="https://user-images.githubusercontent.com/9599/125179033-4e9e9f00-e19f-11eb-8c2e-8df09fafa795.png">

  From https://github.com/simonw/datasette/actions/runs/1018938850 which is still going.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/511#issuecomment-877718364
issue_url: https://api.github.com/repos/simonw/datasette/issues/511
node_id: MDEyOklzc3VlQ29tbWVudDg3NzcxODM2NA==
created_at: 2021-07-10T23:54:37Z
updated_at: 2021-07-10T23:54:37Z
author_association: OWNER
body:
  Looks like it's not even 10% of the way through, and already a bunch of errors:

  <img width="777" alt="Ran_Black__refs__1388_·_simonw_datasette_518cb06" src="https://user-images.githubusercontent.com/9599/125179059-81e12e00-e19f-11eb-94d9-0f2d9ce8afad.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/511#issuecomment-877725742
issue_url: https://api.github.com/repos/simonw/datasette/issues/511
node_id: MDEyOklzc3VlQ29tbWVudDg3NzcyNTc0Mg==
created_at: 2021-07-11T01:25:01Z
updated_at: 2021-07-11T01:26:38Z
author_association: OWNER
body:
  That's weird. https://github.com/simonw/datasette/runs/3037862798 finished running and came up green - but actually a TON of the tests failed on Windows. Not sure why that didn't fail the whole test suite:

  <img width="1203" alt="Ran_Black__refs__1388_·_simonw_datasette_518cb06" src="https://user-images.githubusercontent.com/9599/125180192-12257000-e1ac-11eb-8657-d46b7bcdc1b2.png">

  Also the test suite took 50 minutes on Windows!

  Here's a copy of the full log file for the tests on Python 3.8 on Windows: https://gist.github.com/simonw/2900ef33693c1bbda09188eb31c8212d

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/511#issuecomment-877726288
issue_url: https://api.github.com/repos/simonw/datasette/issues/511
node_id: MDEyOklzc3VlQ29tbWVudDg3NzcyNjI4OA==
created_at: 2021-07-11T01:29:41Z
updated_at: 2021-07-11T01:29:41Z
author_association: OWNER
body:
  Lots of errors that look like this:
  ```
  2021-07-11T00:40:32.1189321Z E           NotADirectoryError: [WinError 267] The directory name is invalid: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpdr41pgwg\\data.db'
  2021-07-11T00:40:32.1190083Z 
  2021-07-11T00:40:32.1191128Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\shutil.py:596: NotADirectoryError
  2021-07-11T00:40:32.1191999Z ___________________ ERROR at teardown of test_insert_error ____________________
  2021-07-11T00:40:32.1192842Z [gw1] win32 -- Python 3.8.10 c:\hostedtoolcache\windows\python\3.8.10\x64\python.exe
  2021-07-11T00:40:32.1193387Z 
  2021-07-11T00:40:32.1193930Z path = 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpry729pq_'
  2021-07-11T00:40:32.1194876Z onerror = <function TemporaryDirectory._rmtree.<locals>.onerror at 0x00000291FCEA93A0>
  2021-07-11T00:40:32.1195480Z 
  2021-07-11T00:40:32.1195927Z     def _rmtree_unsafe(path, onerror):
  2021-07-11T00:40:32.1196435Z         try:
  2021-07-11T00:40:32.1196910Z             with os.scandir(path) as scandir_it:
  2021-07-11T00:40:32.1197504Z                 entries = list(scandir_it)
  2021-07-11T00:40:32.1198002Z         except OSError:
  2021-07-11T00:40:32.1198607Z             onerror(os.scandir, path, sys.exc_info())
  2021-07-11T00:40:32.1199137Z             entries = []
  2021-07-11T00:40:32.1199637Z         for entry in entries:
  2021-07-11T00:40:32.1200184Z             fullname = entry.path
  2021-07-11T00:40:32.1200692Z             if _rmtree_isdir(entry):
  2021-07-11T00:40:32.1201198Z                 try:
  2021-07-11T00:40:32.1201643Z                     if entry.is_symlink():
  2021-07-11T00:40:32.1202280Z                         # This can only happen if someone replaces
  2021-07-11T00:40:32.1202944Z                         # a directory with a symlink after the call to
  2021-07-11T00:40:32.1203623Z                         # os.scandir or entry.is_dir above.
  2021-07-11T00:40:32.1204303Z                         raise OSError("Cannot call rmtree on a symbolic link")
  2021-07-11T00:40:32.1204942Z                 except OSError:
  2021-07-11T00:40:32.1206416Z                     onerror(os.path.islink, fullname, sys.exc_info())
  2021-07-11T00:40:32.1207022Z                     continue
  2021-07-11T00:40:32.1207584Z                 _rmtree_unsafe(fullname, onerror)
  2021-07-11T00:40:32.1208074Z             else:
  2021-07-11T00:40:32.1208496Z                 try:
  2021-07-11T00:40:32.1208926Z >                   os.unlink(fullname)
  2021-07-11T00:40:32.1210053Z E                   PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpry729pq_\\data.db'
  2021-07-11T00:40:32.1210974Z 
  2021-07-11T00:40:32.1211638Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\shutil.py:616: PermissionError
  2021-07-11T00:40:32.1212211Z 
  2021-07-11T00:40:32.1212846Z During handling of the above exception, another exception occurred:
  2021-07-11T00:40:32.1213320Z 
  2021-07-11T00:40:32.1213797Z func = <built-in function unlink>
  2021-07-11T00:40:32.1214529Z path = 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpry729pq_\\data.db'
  2021-07-11T00:40:32.1215763Z exc_info = (<class 'PermissionError'>, PermissionError(13, 'The process cannot access the file because it is being used by another process'), <traceback object at 0x00000291FB4D7040>)
  2021-07-11T00:40:32.1217263Z 
  2021-07-11T00:40:32.1217777Z     def onerror(func, path, exc_info):
  2021-07-11T00:40:32.1218421Z         if issubclass(exc_info[0], PermissionError):
  2021-07-11T00:40:32.1219079Z             def resetperms(path):
  2021-07-11T00:40:32.1219518Z                 try:
  2021-07-11T00:40:32.1219992Z                     _os.chflags(path, 0)
  2021-07-11T00:40:32.1220535Z                 except AttributeError:
  2021-07-11T00:40:32.1221110Z                     pass
  2021-07-11T00:40:32.1221545Z                 _os.chmod(path, 0o700)
  2021-07-11T00:40:32.1221984Z     
  2021-07-11T00:40:32.1222330Z             try:
  2021-07-11T00:40:32.1222768Z                 if path != name:
  2021-07-11T00:40:32.1223332Z                     resetperms(_os.path.dirname(path))
  2021-07-11T00:40:32.1223963Z                 resetperms(path)
  2021-07-11T00:40:32.1224408Z     
  2021-07-11T00:40:32.1224749Z                 try:
  2021-07-11T00:40:32.1225954Z >                   _os.unlink(path)
  2021-07-11T00:40:32.1227032Z E                   PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpry729pq_\\data.db'
  2021-07-11T00:40:32.1227927Z 
  2021-07-11T00:40:32.1228646Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\tempfile.py:802: PermissionError
  2021-07-11T00:40:32.1229200Z 
  2021-07-11T00:40:32.1229842Z During handling of the above exception, another exception occurred:
  2021-07-11T00:40:32.1230355Z 
  2021-07-11T00:40:32.1230783Z     @pytest.fixture
  2021-07-11T00:40:32.1231322Z     def canned_write_client():
  2021-07-11T00:40:32.1231805Z         with make_app_client(
  2021-07-11T00:40:32.1232467Z             extra_databases={"data.db": "create table names (name text)"},
  2021-07-11T00:40:32.1233104Z             metadata={
  2021-07-11T00:40:32.1233535Z                 "databases": {
  2021-07-11T00:40:32.1233989Z                     "data": {
  2021-07-11T00:40:32.1234416Z                         "queries": {
  2021-07-11T00:40:32.1235001Z                             "canned_read": {"sql": "select * from names"},
  2021-07-11T00:40:32.1235527Z                             "add_name": {
  2021-07-11T00:40:32.1236117Z                                 "sql": "insert into names (name) values (:name)",
  2021-07-11T00:40:32.1236686Z                                 "write": True,
  2021-07-11T00:40:32.1237317Z                                 "on_success_redirect": "/data/add_name?success",
  2021-07-11T00:40:32.1237882Z                             },
  2021-07-11T00:40:32.1238331Z                             "add_name_specify_id": {
  2021-07-11T00:40:32.1239009Z                                 "sql": "insert into names (rowid, name) values (:rowid, :name)",
  2021-07-11T00:40:32.1239610Z                                 "write": True,
  2021-07-11T00:40:32.1240259Z                                 "on_error_redirect": "/data/add_name_specify_id?error",
  2021-07-11T00:40:32.1240839Z                             },
  2021-07-11T00:40:32.1241320Z                             "delete_name": {
  2021-07-11T00:40:32.1242504Z                                 "sql": "delete from names where rowid = :rowid",
  2021-07-11T00:40:32.1243127Z                                 "write": True,
  2021-07-11T00:40:32.1243721Z                                 "on_success_message": "Name deleted",
  2021-07-11T00:40:32.1244282Z                                 "allow": {"id": "root"},
  2021-07-11T00:40:32.1244749Z                             },
  2021-07-11T00:40:32.1245959Z                             "update_name": {
  2021-07-11T00:40:32.1246614Z                                 "sql": "update names set name = :name where rowid = :rowid",
  2021-07-11T00:40:32.1247267Z                                 "params": ["rowid", "name", "extra"],
  2021-07-11T00:40:32.1247828Z                                 "write": True,
  2021-07-11T00:40:32.1248247Z                             },
  2021-07-11T00:40:32.1248653Z                         }
  2021-07-11T00:40:32.1249166Z                     }
  2021-07-11T00:40:32.1249577Z                 }
  2021-07-11T00:40:32.1249962Z             },
  2021-07-11T00:40:32.1250333Z         ) as client:
  2021-07-11T00:40:32.1250822Z >           yield client
  2021-07-11T00:40:32.1251078Z 
  2021-07-11T00:40:32.1251678Z D:\a\datasette\datasette\tests\test_canned_queries.py:43: 
  2021-07-11T00:40:32.1252347Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  2021-07-11T00:40:32.1253040Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\contextlib.py:120: in __exit__
  2021-07-11T00:40:32.1253759Z     next(self.gen)
  2021-07-11T00:40:32.1254398Z D:\a\datasette\datasette\tests\fixtures.py:156: in make_app_client
  2021-07-11T00:40:32.1255098Z     yield TestClient(ds)
  2021-07-11T00:40:32.1255796Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\tempfile.py:827: in __exit__
  2021-07-11T00:40:32.1256510Z     self.cleanup()
  2021-07-11T00:40:32.1257200Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\tempfile.py:831: in cleanup
  2021-07-11T00:40:32.1257961Z     self._rmtree(self.name)
  2021-07-11T00:40:32.1258712Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\tempfile.py:813: in _rmtree
  2021-07-11T00:40:32.1259487Z     _shutil.rmtree(name, onerror=onerror)
  2021-07-11T00:40:32.1260280Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\shutil.py:740: in rmtree
  2021-07-11T00:40:32.1261039Z     return _rmtree_unsafe(path, onerror)
  2021-07-11T00:40:32.1261843Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\shutil.py:618: in _rmtree_unsafe
  2021-07-11T00:40:32.1262633Z     onerror(os.unlink, fullname, sys.exc_info())
  2021-07-11T00:40:32.1263456Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\tempfile.py:805: in onerror
  2021-07-11T00:40:32.1264175Z     cls._rmtree(path)
  2021-07-11T00:40:32.1264848Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\tempfile.py:813: in _rmtree
  2021-07-11T00:40:32.1266329Z     _shutil.rmtree(name, onerror=onerror)
  2021-07-11T00:40:32.1267082Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\shutil.py:740: in rmtree
  2021-07-11T00:40:32.1267858Z     return _rmtree_unsafe(path, onerror)
  2021-07-11T00:40:32.1268615Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\shutil.py:599: in _rmtree_unsafe
  2021-07-11T00:40:32.1269440Z     onerror(os.scandir, path, sys.exc_info())
  2021-07-11T00:40:32.1269979Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  2021-07-11T00:40:32.1270287Z 
  2021-07-11T00:40:32.1270947Z path = 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpry729pq_\\data.db'
  2021-07-11T00:40:32.1273356Z onerror = <function TemporaryDirectory._rmtree.<locals>.onerror at 0x00000291FCF40E50>
  2021-07-11T00:40:32.1273999Z 
  2021-07-11T00:40:32.1274493Z     def _rmtree_unsafe(path, onerror):
  2021-07-11T00:40:32.1274953Z         try:
  2021-07-11T00:40:32.1275461Z >           with os.scandir(path) as scandir_it:
  2021-07-11T00:40:32.1276459Z E           NotADirectoryError: [WinError 267] The directory name is invalid: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpry729pq_\\data.db'
  2021-07-11T00:40:32.1277220Z 
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/511#issuecomment-877726495
issue_url: https://api.github.com/repos/simonw/datasette/issues/511
node_id: MDEyOklzc3VlQ29tbWVudDg3NzcyNjQ5NQ==
created_at: 2021-07-11T01:32:27Z
updated_at: 2021-07-11T01:32:27Z
author_association: OWNER
body:
  I'm using `pytest-xdist` and this:

      pytest -n auto -m "not serial"

  I'll try not using the `-n auto` bit on Windows and see if that helps.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/healthkit-to-sqlite/issues/12#issuecomment-877805513
issue_url: https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/12
node_id: MDEyOklzc3VlQ29tbWVudDg3NzgwNTUxMw==
created_at: 2021-07-11T14:03:01Z
updated_at: 2021-07-11T14:03:01Z
author_association: NONE
body:
  Hi Simon -- just experimenting with your excellent software!

  Up to this point in time I have been using the (paid) [HealthFit App](https://apps.apple.com/au/app/healthfit/id1202650514) to export my workouts from my Apple Watch, one walk at the time into either .GPX or .FIT format and then using another library to suck it into Python and eventually here to my "Emmaus Walking" app:

  https://share.streamlit.io/mjboothaus/emmaus_walking/emmaus_walking/app.py

  I just used `healthkit-to-sqlite` to convert my export.zip file and it all "just worked".

  I did notice the issue with various numeric fields being stored in the SQLite db as TEXT for now and just thought I'd flag it - but you're already self-reported this issue.

  Keep up the great work!

  I was curious if you have any thoughts about periodically exporting "export.zip" and how to just update the SQLite file instead of re-creating it each time. Hopefully Apple will give some thought to managing this data in a more sensible fashion as it grows over time.  Ideally one could pull it from iCloud (where it is allegedly being backed up).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/511#issuecomment-877835171
issue_url: https://api.github.com/repos/simonw/datasette/issues/511
node_id: MDEyOklzc3VlQ29tbWVudDg3NzgzNTE3MQ==
created_at: 2021-07-11T17:23:05Z
updated_at: 2021-07-11T17:23:05Z
author_association: OWNER
body:
      == 87 failed, 819 passed, 7 skipped, 29 errors in 2584.85s (0:43:04) ==

  https://github.com/simonw/datasette/runs/3038188870?check_suite_focus=true

  Full copy of log here: https://gist.github.com/simonw/4b1fdd24496b989fca56bc757be345ad

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/healthkit-to-sqlite/issues/12#issuecomment-877874117
issue_url: https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/12
node_id: MDEyOklzc3VlQ29tbWVudDg3Nzg3NDExNw==
created_at: 2021-07-11T23:03:37Z
updated_at: 2021-07-11T23:03:37Z
author_association: NONE
body:
  P.s. wondering if you have explored using the spatialite functionality with the location data in workouts?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/healthkit-to-sqlite/issues/12#issuecomment-879477586
issue_url: https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/12
node_id: MDEyOklzc3VlQ29tbWVudDg3OTQ3NzU4Ng==
created_at: 2021-07-13T23:50:06Z
updated_at: 2021-07-13T23:50:06Z
author_association: MEMBER
body:
  Unfortunately I don't think updating the database is practical, because the export doesn't include unique identifiers which can be used to update existing records and create new ones. Recreating from scratch works around that limitation.

  I've not explored workouts with SpatiaLite but that's a really good idea.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/268#issuecomment-880150755
issue_url: https://api.github.com/repos/simonw/datasette/issues/268
node_id: MDEyOklzc3VlQ29tbWVudDg4MDE1MDc1NQ==
created_at: 2021-07-14T19:26:47Z
updated_at: 2021-07-14T19:29:08Z
author_association: OWNER
body:
  > What are the side-effects of turning that on in the query string, or even by default as you suggested? I see that you stated in the docs... "to ensure they do not cause any confusion for users who are not aware of them", but I'm not sure what those could be.

  Mainly that it's possible to generate SQL queries that crash with an error. This was the example that convinced me to default to escaping:

  - https://www.niche-museums.com/browse/museums?_search=park.&_searchmode=raw (returns `fts5: syntax error near "."`)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/268#issuecomment-880153069
issue_url: https://api.github.com/repos/simonw/datasette/issues/268
node_id: MDEyOklzc3VlQ29tbWVudDg4MDE1MzA2OQ==
created_at: 2021-07-14T19:31:00Z
updated_at: 2021-07-14T19:31:00Z
author_association: OWNER
body:
  ... though interestingly I can't replicate that error on `latest.datasette.io` - https://latest.datasette.io/fixtures/searchable?_search=park.&_searchmode=raw

  That's running https://latest.datasette.io/-/versions SQLite 3.35.4 whereas https://www.niche-museums.com/-/versions is running 3.27.2 (the most recent version available with Vercel) - but there's nothing in the SQLite changelog between those two versions that suggests changes to how the FTS5 parser works. https://www.sqlite.org/changes.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/297#issuecomment-880256058
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/297
node_id: MDEyOklzc3VlQ29tbWVudDg4MDI1NjA1OA==
created_at: 2021-07-14T22:40:01Z
updated_at: 2021-07-14T22:40:47Z
author_association: OWNER
body:
  Full docs here: https://www.sqlite.org/draft/cli.html#csv

  One catch: how this works has changed in recent SQLite versions: https://www.sqlite.org/changes.html

  - 2020-12-01 (3.34.0) - "Table name quoting works correctly for the .import dot-command"
  - 2020-05-22 (3.32.0) - "Add options to the .import command: --csv, --ascii, --skip"
  - 2017-08-01 (3.20.0) - " The ".import" command ignores an initial UTF-8 BOM."

  The "skip" feature is particularly important to understand. https://www.sqlite.org/draft/cli.html#csv says:

  > There are two cases to consider: (1) Table "tab1" does not previously exist and (2) table "tab1" does already exist.
  >
  > In the first case, when the table does not previously exist, the table is automatically created and the content of the first row of the input CSV file is used to determine the name of all the columns in the table. In other words, if the table does not previously exist, the first row of the CSV file is interpreted to be column names and the actual data starts on the second row of the CSV file.
  >
  > For the second case, when the table already exists, every row of the CSV file, including the first row, is assumed to be actual content. If the CSV file contains an initial row of column labels, you can cause the .import command to skip that initial row using the "--skip 1" option.

  But the `--skip 1` option is only available in 3.32.0 and higher.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/297#issuecomment-880256865
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/297
node_id: MDEyOklzc3VlQ29tbWVudDg4MDI1Njg2NQ==
created_at: 2021-07-14T22:42:11Z
updated_at: 2021-07-14T22:42:11Z
author_association: OWNER
body:
  Potential workaround for missing `--skip` implementation is that the filename can be a command instead, so maybe it could shell out to `tail -n +1 filename`:

  > The source argument is the name of a file to be read or, if it begins with a "|" character, specifies a command which will be run to produce the input CSV data.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/297#issuecomment-880257587
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/297
node_id: MDEyOklzc3VlQ29tbWVudDg4MDI1NzU4Nw==
created_at: 2021-07-14T22:44:05Z
updated_at: 2021-07-14T22:44:05Z
author_association: OWNER
body:
  https://unix.stackexchange.com/a/642364 suggests you can also use this to import from stdin, like so:

      sqlite3 -csv $database_file_name ".import '|cat -' $table_name"

  Here the `sqlite3 -csv` is an alternative to using `.mode csv`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/297#issuecomment-880259255
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/297
node_id: MDEyOklzc3VlQ29tbWVudDg4MDI1OTI1NQ==
created_at: 2021-07-14T22:48:41Z
updated_at: 2021-07-14T22:48:41Z
author_association: OWNER
body:
  Should also take advantage of `.mode tabs` to support `sqlite-utils insert blah.db blah blah.csv --tsv --fast`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1394#issuecomment-880278256
issue_url: https://api.github.com/repos/simonw/datasette/issues/1394
node_id: MDEyOklzc3VlQ29tbWVudDg4MDI3ODI1Ng==
created_at: 2021-07-14T23:35:18Z
updated_at: 2021-07-14T23:35:18Z
author_association: OWNER
body:
  The challenge here is that faceting doesn't currently modify the inner SQL at all - it wraps it so that it can work against any SQL statement (though Datasette itself does not yet take advantage of that ability, only offering faceting on table pages).

  So just removing the order by wouldn't be appropriate if the inner query looked something like this:

  ```sql
  select * from items order by created desc limit 100
  ```
  Since the intent there would be to return facet counts against only the most recent 100 items.

  In SQLite the `limit` has to come after the `order by` though, so the fix here could be as easy as using a regular expression to identify queries that end with `order by COLUMN (desc)?` and stripping off that clause.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1394#issuecomment-880287483
issue_url: https://api.github.com/repos/simonw/datasette/issues/1394
node_id: MDEyOklzc3VlQ29tbWVudDg4MDI4NzQ4Mw==
created_at: 2021-07-15T00:01:47Z
updated_at: 2021-07-15T00:01:47Z
author_association: OWNER
body:
  I wrote this code:
  ```python
  _order_by_re = re.compile(r"(^.*) order by [a-zA-Z_][a-zA-Z0-9_]+( desc)?$", re.DOTALL)
  _order_by_braces_re = re.compile(r"(^.*) order by \[[^\]]+\]( desc)?$", re.DOTALL)


  def strip_order_by(sql):
      for regex in (_order_by_re, _order_by_braces_re):
          match = regex.match(sql)
          if match is not None:
              return match.group(1)
      return sql

  @pytest.mark.parametrize(
      "sql,expected",
      [
          ("blah", "blah"),
          ("select * from foo", "select * from foo"),
          ("select * from foo order by bah", "select * from foo"),
          ("select * from foo order by bah desc", "select * from foo"),
          ("select * from foo order by [select]", "select * from foo"),
          ("select * from foo order by [select] desc", "select * from foo"),
      ],
  )
  def test_strip_order_by(sql, expected):
      assert strip_order_by(sql) == expected
  ```
  But it turns out I don't need it! The SQL that is passed to the facet class is created by this code: https://github.com/simonw/datasette/blob/ba11ef27edd6981eeb26d7ecf5aa236707f5f8ce/datasette/views/table.py#L677-L684

  And the only place that uses that `sql_no_limit` variable is here: https://github.com/simonw/datasette/blob/ba11ef27edd6981eeb26d7ecf5aa236707f5f8ce/datasette/views/table.py#L733-L745

  So I can change that to `sql_no_limit_no_order` and fix the bug that way instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1396#issuecomment-880324637
issue_url: https://api.github.com/repos/simonw/datasette/issues/1396
node_id: MDEyOklzc3VlQ29tbWVudDg4MDMyNDYzNw==
created_at: 2021-07-15T01:46:26Z
updated_at: 2021-07-15T01:46:26Z
author_association: OWNER
body:
  I manually published the Docker image using https://github.com/simonw/datasette/actions/workflows/push_docker_tag.yml https://github.com/simonw/datasette/runs/3072505126

  The 0.58 release shows up on https://hub.docker.com/r/datasetteproject/datasette/tags?page=1&ordering=last_updated now - BUT the `latest` tag still points to a version from a month ago.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1396#issuecomment-880325004
issue_url: https://api.github.com/repos/simonw/datasette/issues/1396
node_id: MDEyOklzc3VlQ29tbWVudDg4MDMyNTAwNA==
created_at: 2021-07-15T01:47:17Z
updated_at: 2021-07-15T01:47:17Z
author_association: OWNER
body:
  This is the part of the publish workflow that failed and threw the "invalid reference format" error: https://github.com/simonw/datasette/blob/084cfe1e00e1a4c0515390a513aca286eeea20c2/.github/workflows/publish.yml#L100-L119

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1396#issuecomment-880325362
issue_url: https://api.github.com/repos/simonw/datasette/issues/1396
node_id: MDEyOklzc3VlQ29tbWVudDg4MDMyNTM2Mg==
created_at: 2021-07-15T01:48:11Z
updated_at: 2021-07-15T01:48:11Z
author_association: OWNER
body:
  In particular these three lines: https://github.com/simonw/datasette/blob/084cfe1e00e1a4c0515390a513aca286eeea20c2/.github/workflows/publish.yml#L117-L119

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1396#issuecomment-880326049
issue_url: https://api.github.com/repos/simonw/datasette/issues/1396
node_id: MDEyOklzc3VlQ29tbWVudDg4MDMyNjA0OQ==
created_at: 2021-07-15T01:50:05Z
updated_at: 2021-07-15T01:50:05Z
author_association: OWNER
body:
  I think I made a mistake in this commit: https://github.com/simonw/datasette/commit/0486303b60ce2784fd2e2ecdbecf304b7d6e6659

  <img width="770" alt="Explicitly_push_version_tag__refs__1281_·_simonw_datasette_0486303" src="https://user-images.githubusercontent.com/9599/125715241-dc96d147-191a-4a69-bdf0-1cb092d97330.png">

  It looks like I copied `$VERSION_TAG` from here - but it's not available in the `publish.yml` flow: https://github.com/simonw/datasette/blob/0486303b60ce2784fd2e2ecdbecf304b7d6e6659/.github/workflows/push_docker_tag.yml#L18-L25

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1396#issuecomment-880374156
issue_url: https://api.github.com/repos/simonw/datasette/issues/1396
node_id: MDEyOklzc3VlQ29tbWVudDg4MDM3NDE1Ng==
created_at: 2021-07-15T04:03:18Z
updated_at: 2021-07-15T04:03:18Z
author_association: OWNER
body:
  I fixed  `datasette:latest` by running the following on my laptop:
  ```
  docker pull datasetteproject/datasette:0.58
  docker tag datasetteproject/datasette:0.58 datasetteproject/datasette:latest
  docker login -u datasetteproject -p ...
  docker push datasetteproject/datasette:latest
  ```
  Confirmed on https://hub.docker.com/r/datasetteproject/datasette/tags?page=1&ordering=last_updated that `datasette:latest` and `datasette:0.58` both now have the same digest of `3b5ba478040e`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1394#issuecomment-880900534
issue_url: https://api.github.com/repos/simonw/datasette/issues/1394
node_id: MDEyOklzc3VlQ29tbWVudDg4MDkwMDUzNA==
created_at: 2021-07-15T17:58:03Z
updated_at: 2021-07-15T17:58:03Z
author_association: OWNER
body:
  Started a conversation about this on the SQLite forum: https://sqlite.org/forum/forumpost/2d76f2bcf65d256a?t=h

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/759#issuecomment-881125124
issue_url: https://api.github.com/repos/simonw/datasette/issues/759
node_id: IC_kwDOBm6k_c40hOcE
created_at: 2021-07-16T02:11:48Z
updated_at: 2021-07-16T02:11:54Z
author_association: OWNER
body:
  I added `"searchmode": "raw"` as a supported option for table metadata in #1389 and released that in Datasette 0.58.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1394#issuecomment-881129149
issue_url: https://api.github.com/repos/simonw/datasette/issues/1394
node_id: IC_kwDOBm6k_c40hPa9
created_at: 2021-07-16T02:23:32Z
updated_at: 2021-07-16T02:23:32Z
author_association: OWNER
body:
  Wrote about this in the annotated release notes for 0.58: https://simonwillison.net/2021/Jul/16/datasette-058/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1231#issuecomment-881204343
issue_url: https://api.github.com/repos/simonw/datasette/issues/1231
node_id: IC_kwDOBm6k_c40hhx3
created_at: 2021-07-16T06:13:11Z
updated_at: 2021-07-16T06:13:11Z
author_association: OWNER
body:
  This just broke the `datasette-graphql` test suite: https://github.com/simonw/datasette-graphql/issues/77 - I need to figure out a solution here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1231#issuecomment-881204782
issue_url: https://api.github.com/repos/simonw/datasette/issues/1231
node_id: IC_kwDOBm6k_c40hh4u
created_at: 2021-07-16T06:14:12Z
updated_at: 2021-07-16T06:14:12Z
author_association: OWNER
body:
  Here's the traceback I got from `datasette-graphql` (annoyingly only running the tests in GitHub Actions CI - I've not been able to replicate on my laptop yet):

  ```
  tests/test_utils.py .                                                    [100%]

  =================================== FAILURES ===================================
  _________________________ test_graphql_examples[path0] _________________________

  ds = <datasette.app.Datasette object at 0x7f6b8b6f8fd0>
  path = PosixPath('/home/runner/work/datasette-graphql/datasette-graphql/examples/filters.md')

      @pytest.mark.asyncio
      @pytest.mark.parametrize(
          "path", (pathlib.Path(__file__).parent.parent / "examples").glob("*.md")
      )
      async def test_graphql_examples(ds, path):
          content = path.read_text()
          query = graphql_re.search(content)[1]
          try:
              variables = variables_re.search(content)[1]
          except TypeError:
              variables = "{}"
          expected = json.loads(json_re.search(content)[1])
          response = await ds.client.post(
              "/graphql",
              json={
                  "query": query,
                  "variables": json.loads(variables),
              },
          )
  >       assert response.status_code == 200, response.json()
  E       AssertionError: {'data': {'repos_arraycontains': None, 'users_contains': None, 'users_date': None, 'users_endswith': None, ...}, 'erro...", 'path': ['users_gt']}, {'locations': [{'column': 5, 'line': 34}], 'message': "'rows'", 'path': ['users_gte']}, ...]}
  E       assert 500 == 200
  E        +  where 500 = <Response [500 Internal Server Error]>.status_code

  tests/test_graphql.py:142: AssertionError
  ----------------------------- Captured stderr call -----------------------------
  table databases already exists
  table databases already exists
  table databases already exists
  table databases already exists
  table databases already exists
  table databases already exists
  table databases already exists
  table databases already exists
  table databases already exists
  table databases already exists
  table databases already exists
  table databases already exists
  table databases already exists
  table databases already exists
  table databases already exists
  table databases already exists
  table databases already exists
  table databases already exists
  table databases already exists
  table databases already exists
  table databases already exists
  Traceback (most recent call last):
    File "/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/app.py", line 1171, in route_path
      response = await view(request, send)
    File "/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/views/base.py", line 151, in view
      request, **request.scope["url_route"]["kwargs"]
    File "/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/views/base.py", line 123, in dispatch_request
      await self.ds.refresh_schemas()
    File "/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/app.py", line 338, in refresh_schemas
      await init_internal_db(internal_db)
    File "/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/utils/internal_db.py", line 16, in init_internal_db
      block=True,
    File "/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/database.py", line 102, in execute_write
      return await self.execute_write_fn(_inner, block=block)
    File "/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/database.py", line 118, in execute_write_fn
      raise result
    File "/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/database.py", line 139, in _execute_writes
      result = task.fn(conn)
    File "/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/database.py", line 100, in _inner
      return conn.execute(sql, params or [])
  sqlite3.OperationalError: table databases already exists
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1231#issuecomment-881663968
issue_url: https://api.github.com/repos/simonw/datasette/issues/1231
node_id: IC_kwDOBm6k_c40jR_g
created_at: 2021-07-16T19:18:42Z
updated_at: 2021-07-16T19:18:42Z
author_association: OWNER
body:
  The race condition happens inside this method - initially with the call to `await init_internal_db()`: https://github.com/simonw/datasette/blob/dd5ee8e66882c94343cd3f71920878c6cfd0da41/datasette/app.py#L334-L359

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1231#issuecomment-881664408
issue_url: https://api.github.com/repos/simonw/datasette/issues/1231
node_id: IC_kwDOBm6k_c40jSGY
created_at: 2021-07-16T19:19:35Z
updated_at: 2021-07-16T19:19:35Z
author_association: OWNER
body:
  The only place that calls `refresh_schemas()` is here: https://github.com/simonw/datasette/blob/dd5ee8e66882c94343cd3f71920878c6cfd0da41/datasette/views/base.py#L120-L124

  Ideally only one call to `refresh_schemas()` would be running at any one time.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1231#issuecomment-881665383
issue_url: https://api.github.com/repos/simonw/datasette/issues/1231
node_id: IC_kwDOBm6k_c40jSVn
created_at: 2021-07-16T19:21:35Z
updated_at: 2021-07-16T19:21:35Z
author_association: OWNER
body:
  https://stackoverflow.com/a/25799871/6083 has a good example of using `asyncio.Lock()`:

  ```python
  stuff_lock = asyncio.Lock()

  async def get_stuff(url):
      async with stuff_lock:
          if url in cache:
              return cache[url]
          stuff = await aiohttp.request('GET', url)
          cache[url] = stuff
          return stuff
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1231#issuecomment-881668759
issue_url: https://api.github.com/repos/simonw/datasette/issues/1231
node_id: IC_kwDOBm6k_c40jTKX
created_at: 2021-07-16T19:27:46Z
updated_at: 2021-07-16T19:27:46Z
author_association: OWNER
body:
  Second attempt at this:
  ```diff
  diff --git a/datasette/app.py b/datasette/app.py
  index 5976d8b..5f348cb 100644
  --- a/datasette/app.py
  +++ b/datasette/app.py
  @@ -224,6 +224,7 @@ class Datasette:
           self.inspect_data = inspect_data
           self.immutables = set(immutables or [])
           self.databases = collections.OrderedDict()
  +        self._refresh_schemas_lock = asyncio.Lock()
           self.crossdb = crossdb
           if memory or crossdb or not self.files:
               self.add_database(Database(self, is_memory=True), name="_memory")
  @@ -332,6 +333,12 @@ class Datasette:
           self.client = DatasetteClient(self)
   
       async def refresh_schemas(self):
  +        if self._refresh_schemas_lock.locked():
  +            return
  +        async with self._refresh_schemas_lock:
  +            await self._refresh_schemas()
  +
  +    async def _refresh_schemas(self):
           internal_db = self.databases["_internal"]
           if not self.internal_db_created:
               await init_internal_db(internal_db)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1231#issuecomment-881674857
issue_url: https://api.github.com/repos/simonw/datasette/issues/1231
node_id: IC_kwDOBm6k_c40jUpp
created_at: 2021-07-16T19:38:39Z
updated_at: 2021-07-16T19:38:39Z
author_association: OWNER
body:
  I can't replicate the race condition locally with or without this patch. I'm going to push the commit and then test the CI run from `datasette-graphql` that was failing against it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1231#issuecomment-881677620
issue_url: https://api.github.com/repos/simonw/datasette/issues/1231
node_id: IC_kwDOBm6k_c40jVU0
created_at: 2021-07-16T19:44:12Z
updated_at: 2021-07-16T19:44:12Z
author_association: OWNER
body:
  That fixed the race condition in the `datasette-graphql` tests, which is the only place that I've been able to successfully replicate this. I'm going to land this change.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1396#issuecomment-881686662
issue_url: https://api.github.com/repos/simonw/datasette/issues/1396
node_id: IC_kwDOBm6k_c40jXiG
created_at: 2021-07-16T20:02:44Z
updated_at: 2021-07-16T20:02:44Z
author_association: OWNER
body:
  Confirmed fixed: 0.58.1 was successfully published to Docker Hub in https://github.com/simonw/datasette/runs/3089447346 and the `latest` tag on https://hub.docker.com/r/datasetteproject/datasette/tags was updated.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1199#issuecomment-881932880
issue_url: https://api.github.com/repos/simonw/datasette/issues/1199
node_id: IC_kwDOBm6k_c40kTpQ
created_at: 2021-07-17T17:39:17Z
updated_at: 2021-07-17T17:39:17Z
author_association: OWNER
body:
  I asked about optimizing performance on the SQLite forum and this came up as a suggestion: https://sqlite.org/forum/forumpost/9a6b9ae8e2048c8b?t=c

  I can start by trying this:

      PRAGMA mmap_size=268435456;

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/297#issuecomment-882052693
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/297
node_id: IC_kwDOCGYnMM40kw5V
created_at: 2021-07-18T12:57:54Z
updated_at: 2022-06-21T13:17:15Z
author_association: OWNER
body:
  Another implementation option would be to use the CSV virtual table mechanism. This could avoid shelling out to the `sqlite3` binary, but requires solving the harder problem of compiling and distributing a loadable SQLite module: https://www.sqlite.org/csv.html

  (Would be neat to produce a Python wheel of this, see https://simonwillison.net/2022/May/23/bundling-binary-tools-in-python-wheels/)

  This would also help solve the challenge of making this optimization available to the `sqlite-utils memory` command. That command operates against an in-memory database so it's not obvious how it could shell out to a binary.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/297#issuecomment-882052852
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/297
node_id: IC_kwDOCGYnMM40kw70
created_at: 2021-07-18T12:59:20Z
updated_at: 2021-07-18T12:59:20Z
author_association: OWNER
body:
  I'm not too worried about `sqlite-utils memory` because if your data is large enough that you can benefit from this optimization you probably should use a real file as opposed to a disposable memory database when analyzing it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/123#issuecomment-882096402
issue_url: https://api.github.com/repos/simonw/datasette/issues/123
node_id: IC_kwDOBm6k_c40k7kS
created_at: 2021-07-18T18:07:29Z
updated_at: 2021-07-18T18:07:29Z
author_association: NONE
body:
  I also love the idea for this feature and wonder if it could work without having to download the whole database into memory at once if it's a rather large db. Obviously this could be slower but could have many use cases.

  My comment is partially inspired by this post about streaming sqlite dbs from github pages or such
  https://news.ycombinator.com/item?id=27016630

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/123#issuecomment-882138084
issue_url: https://api.github.com/repos/simonw/datasette/issues/123
node_id: IC_kwDOBm6k_c40lFvk
created_at: 2021-07-19T00:04:31Z
updated_at: 2021-07-19T00:04:31Z
author_association: OWNER
body:
  I've been thinking more about this one today too. An extension of this (touched on in #417, Datasette Library) would be to support pointing Datasette at a directory and having it automatically load any CSV files it finds anywhere in that folder or its descendants - either loading them fully, or providing a UI that allows users to select a file to open it in Datasette.

  For larger files I think the right thing to do is import them into an on-disk SQLite database, which is limited only by available disk space. For smaller files loading them into an in-memory database should work fine.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1400#issuecomment-882542519
issue_url: https://api.github.com/repos/simonw/datasette/issues/1400
node_id: IC_kwDOBm6k_c40moe3
created_at: 2021-07-19T13:20:52Z
updated_at: 2021-07-19T13:20:52Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1400?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1400](https://codecov.io/gh/simonw/datasette/pull/1400?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (e95c685) into [main](https://codecov.io/gh/simonw/datasette/commit/c73af5dd72305f6a01ea94a2c76d52e5e26de38b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (c73af5d) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1400/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1400?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1400   +/-   ##
  =======================================
    Coverage   91.62%   91.62%           
  =======================================
    Files          34       34           
    Lines        4371     4371           
  =======================================
    Hits         4005     4005           
    Misses        366      366           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1400?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1400?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [c73af5d...e95c685](https://codecov.io/gh/simonw/datasette/pull/1400?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-884672647
issue_url: https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5
node_id: IC_kwDODFE5qs40uwiH
created_at: 2021-07-22T05:56:31Z
updated_at: 2021-07-22T14:03:08Z
author_association: NONE
body:
  How does this commit look? https://github.com/maxhawkins/google-takeout-to-sqlite/commit/72802a83fee282eb5d02d388567731ba4301050d

  It seems that Takeout's mbox format is pretty simple, so we can get away with just splitting the file on lines begining with `From `. My commit just splits the file every time a line starts with `From ` and uses `email.message_from_bytes` to parse each chunk.

  I was able to load a 12GB takeout mbox without the program using more than a couple hundred MB of memory during the import process. It does make us lose the progress bar, but maybe I can add that back in a later commit.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/32#issuecomment-884688833
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/32
node_id: IC_kwDOD079W840u0fB
created_at: 2021-07-22T06:40:25Z
updated_at: 2021-07-22T06:40:25Z
author_association: NONE
body:
  The solution here is to upload an image to the bucket first. It is caused because it does not properly handle the case when there are no images in the bucket.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-885022230
issue_url: https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5
node_id: IC_kwDODFE5qs40wF4W
created_at: 2021-07-22T15:51:46Z
updated_at: 2021-07-22T15:51:46Z
author_association: NONE
body:
  One thing I noticed is this importer doesn't save attachments along with the body of the emails. It would be nice if those got stored as blobs in a separate attachments table so attachments can be included while fetching search results.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-885094284
issue_url: https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5
node_id: IC_kwDODFE5qs40wXeM
created_at: 2021-07-22T17:41:32Z
updated_at: 2021-07-22T17:41:32Z
author_association: NONE
body:
  I added a follow-up commit that deals with emails that don't have a `Date` header: https://github.com/maxhawkins/google-takeout-to-sqlite/commit/4bc70103582c10802c85a523ef1e99a8a2154aa9

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-885098025
issue_url: https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5
node_id: IC_kwDODFE5qs40wYYp
created_at: 2021-07-22T17:47:50Z
updated_at: 2021-07-22T17:47:50Z
author_association: NONE
body:
  Hi @maxhawkins , I'm sorry, I haven't had any time to work on this. I'll have some time tomorrow to test your commits. I think they look great.  I'm great with your commits superseding my initial attempt here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/251#issuecomment-886122696
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/251
node_id: IC_kwDOCGYnMM400SjI
created_at: 2021-07-24T23:21:32Z
updated_at: 2021-07-24T23:21:32Z
author_association: OWNER
body:
  > ```
  > sqlite-utils convert jsonsplit mydb.db mytable mycolumn
  > sqlite-utils convert parsedatetime mydb.db mytable mycolumn
  > sqlite-utils convert parsedate mydb.db mytable mycolumn
  > sqlite-utils convert lambda mydb.db mytable mycolumn --code='str(value).upper()'
  > ```

  This is a bit verbose - and having added `--multi` and `--output` the `lambda` command keeps getting more and more flexible compared to the others.

  New idea: ditch the sub-sub-commands and move the `jsonsplit` and `parsedate` recipes to be options of `convert` - maybe like this:

      sqlite-utils convert my.db mytable col1 --jsonsplit

  or:

      sqlite-utils convert my.db mytable col1 --recipe jsonsplit

  or `-r jsonsplit` for short.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/hacker-news-to-sqlite/issues/2#issuecomment-886135562
issue_url: https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/2
node_id: IC_kwDODtX3eM400VsK
created_at: 2021-07-25T02:01:11Z
updated_at: 2021-07-25T02:01:11Z
author_association: MEMBER
body:
  That page doesn't have an API but does look easy to scrape.

  The other option here is the HN Search API powered by Algolia, documented at https://hn.algolia.com/api

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/hacker-news-to-sqlite/issues/2#issuecomment-886135922
issue_url: https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/2
node_id: IC_kwDODtX3eM400Vxy
created_at: 2021-07-25T02:06:20Z
updated_at: 2021-07-25T02:06:20Z
author_association: MEMBER
body:
  https://hn.algolia.com/api/v1/search_by_date?query=simonwillison.net&restrictSearchableAttributes=url looks like it does what I want.

  https://hn.algolia.com/api/v1/search_by_date?query=simonwillison.net&restrictSearchableAttributes=url&hitsPerPage=1000 - returns 1000 at once.

  Otherwise you have to paginate using `&page=2` etc - up to `nbPages` pages.

  https://www.algolia.com/doc/api-reference/api-parameters/hitsPerPage/ says 1000 is the maximum.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/hacker-news-to-sqlite/issues/2#issuecomment-886136224
issue_url: https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/2
node_id: IC_kwDODtX3eM400V2g
created_at: 2021-07-25T02:08:29Z
updated_at: 2021-07-25T02:08:29Z
author_association: MEMBER
body:
  Prototype:

      curl "https://hn.algolia.com/api/v1/search_by_date?query=simonwillison.net&restrictSearchableAttributes=url&hitsPerPage=1000" | \
        jq .hits | sqlite-utils insert hn.db items - --pk objectID --alter

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/hacker-news-to-sqlite/issues/2#issuecomment-886140431
issue_url: https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/2
node_id: IC_kwDODtX3eM400W4P
created_at: 2021-07-25T03:12:57Z
updated_at: 2021-07-25T03:12:57Z
author_association: MEMBER
body:
  I'm going to build a general-purpose `hacker-new-to-sqlite search ...` command, where one of the options is to search within the URL.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/hacker-news-to-sqlite/issues/3#issuecomment-886142671
issue_url: https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/3
node_id: IC_kwDODtX3eM400XbP
created_at: 2021-07-25T03:51:05Z
updated_at: 2021-07-25T03:51:05Z
author_association: MEMBER
body:
  Prototype:

      curl 'https://hn.algolia.com/api/v1/items/27941108' \
        | jq '[recurse(.children[]) | del(.children)]' \
        | sqlite-utils insert hn.db items - --pk id

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/hacker-news-to-sqlite/issues/3#issuecomment-886237834
issue_url: https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/3
node_id: IC_kwDODtX3eM400uqK
created_at: 2021-07-25T18:05:32Z
updated_at: 2021-07-25T18:05:32Z
author_association: MEMBER
body:
  If you hit the endpoint for a comment that's part of a thread you get that comment and its recursive children: https://hn.algolia.com/api/v1/items/27941552

  You can tell that it's not the top-level because the `parent_id` isn't `null`. You can use `story_id` to figure out what the top-level item is.

  ```json
  {
    "id": 27941552,
    "created_at": "2021-07-24T15:08:39.000Z",
    "created_at_i": 1627139319,
    "type": "comment",
    "author": "nine_k",
    "title": null,
    "url": null,
    "text": "<p>I wish ...",
    "points": null,
    "parent_id": 27941108,
    "story_id": 27941108
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1402#issuecomment-886968648
issue_url: https://api.github.com/repos/simonw/datasette/issues/1402
node_id: IC_kwDOBm6k_c403hFI
created_at: 2021-07-26T19:30:14Z
updated_at: 2021-07-26T19:30:14Z
author_association: OWNER
body:
  I really like this idea. I was thinking it might make a good plugin, but there's not a great mechanism for plugins to inject extra `<head>` content at the moment - plus this actually feels like a reasonable feature for Datasette core itself.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1402#issuecomment-886969541
issue_url: https://api.github.com/repos/simonw/datasette/issues/1402
node_id: IC_kwDOBm6k_c403hTF
created_at: 2021-07-26T19:31:40Z
updated_at: 2021-07-26T19:31:40Z
author_association: OWNER
body:
  Datasette could do a pretty good job of this by default, using `twitter:card` and `og:url` tags - like on https://til.simonwillison.net/jq/extracting-objects-recursively

  I could also provide a mechanism to customize these - in particular to add images of some sort.

  It feels like something that should tie in to the metadata mechanism.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1404#issuecomment-887095569
issue_url: https://api.github.com/repos/simonw/datasette/issues/1404
node_id: IC_kwDOBm6k_c404AER
created_at: 2021-07-26T23:27:07Z
updated_at: 2021-07-26T23:27:07Z
author_association: OWNER
body:
  Updated documentation: https://github.com/simonw/datasette/blob/eccfeb0871dd4bc27870faf64f80ac68e5b6bc0d/docs/plugin_hooks.rst#register-routes-datasette

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-888075098
issue_url: https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5
node_id: IC_kwDODFE5qs407vNa
created_at: 2021-07-28T07:18:56Z
updated_at: 2021-07-28T07:18:56Z
author_association: NONE
body:
  > I'm not sure why but my most recent import, when displayed in Datasette, looks like this:
  > 
  > <img alt="mbox__mbox_emails__753_446_rows" width="574" src="https://user-images.githubusercontent.com/9599/109985836-0ab00080-7cba-11eb-97d5-0631a0835b61.png">

  I did some investigation into this issue and made a fix [here](https://github.com/dogsheep/google-takeout-to-sqlite/pull/8/commits/8ee555c2889a38ff42b95664ee074b4a01a82f06). The problem was that some messages (like gchat logs) don't have a `Message-Id` and we need to use `X-GM-THRID` as the pkey instead.

  @simonw While looking into this I found something unexpected about how sqlite_utils handles upserts if the pkey column is `None`. When the pkey is NULL I'd expect the function to either use rowid or throw an exception. Instead, it seems upsert_all creates a row where all columns are NULL instead of using the values provided as parameters.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1405#issuecomment-888694144
issue_url: https://api.github.com/repos/simonw/datasette/issues/1405
node_id: IC_kwDOBm6k_c40-GWA
created_at: 2021-07-28T23:51:59Z
updated_at: 2021-07-28T23:51:59Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/blob/eccfeb0871dd4bc27870faf64f80ac68e5b6bc0d/datasette/utils/__init__.py#L918-L926

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1405#issuecomment-889525741
issue_url: https://api.github.com/repos/simonw/datasette/issues/1405
node_id: IC_kwDOBm6k_c41BRXt
created_at: 2021-07-29T23:33:30Z
updated_at: 2021-07-29T23:33:30Z
author_association: OWNER
body:
  New documentation section for `datasette.utils` is here: https://docs.datasette.io/en/latest/internals.html#the-datasette-utils-module

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1241#issuecomment-889539227
issue_url: https://api.github.com/repos/simonw/datasette/issues/1241
node_id: IC_kwDOBm6k_c41BUqb
created_at: 2021-07-30T00:15:26Z
updated_at: 2021-07-30T00:15:26Z
author_association: OWNER
body:
  One possible treatment:

  <img width="442" alt="fixtures__sortable__201_rows" src="https://user-images.githubusercontent.com/9599/127581105-dca1193c-1bdd-491e-89d1-ac3136c05c3b.png">

  ```html
  <style>
  a.action-button {
      display: inline-block;
      border: 1.5px solid #666;
      border-radius: 0.5em;
      background-color: #ffffff;
      color: #666;
      padding: 0.1em 0.8em;
      text-decoration: none;
      margin-right: 0.3em;
      font-size: 0.85em;
  }
  </style>
  <p>
      {% if query.sql and allow_execute_sql %}
          <a class="action-button"
              title="{{ query.sql }}"
              href="{{ urls.database(database) }}?{{ {'sql': query.sql}|urlencode|safe }}{% if query.params %}&amp;{{ query.params|urlencode|safe }}{% endif %}">
              View and edit SQL
          </a>
      {% endif %}
      <a href="#" class="action-button">Copy and share link</a>
  </p>
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1406#issuecomment-889547142
issue_url: https://api.github.com/repos/simonw/datasette/issues/1406
node_id: IC_kwDOBm6k_c41BWmG
created_at: 2021-07-30T00:39:49Z
updated_at: 2021-07-30T00:39:49Z
author_association: OWNER
body:
  It happens in CI but not on my laptop. I think I need to run the tests on my laptop like this: https://github.com/simonw/datasette/blob/121e10c29c5b412fddf0326939f1fe46c3ad9d4a/.github/workflows/test.yml#L27-L30

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1406#issuecomment-889548536
issue_url: https://api.github.com/repos/simonw/datasette/issues/1406
node_id: IC_kwDOBm6k_c41BW74
created_at: 2021-07-30T00:43:47Z
updated_at: 2021-07-30T00:43:47Z
author_association: OWNER
body:
  Still couldn't replicate on my laptop. On a hunch, I'm going to add `@pytest.mark.serial` to every test that uses `runner.isolated_filesystem()`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1406#issuecomment-889550391
issue_url: https://api.github.com/repos/simonw/datasette/issues/1406
node_id: IC_kwDOBm6k_c41BXY3
created_at: 2021-07-30T00:49:31Z
updated_at: 2021-07-30T00:49:31Z
author_association: OWNER
body:
  That fixed it. My hunch is that Click's `runner.isolated_filesystem()` mechanism doesn't play well with `pytest-xdist`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1406#issuecomment-889555977
issue_url: https://api.github.com/repos/simonw/datasette/issues/1406
node_id: IC_kwDOBm6k_c41BYwJ
created_at: 2021-07-30T01:06:57Z
updated_at: 2021-07-30T01:06:57Z
author_association: OWNER
body:
  Looking at the source code in Click for `isolated_filesystem()`: https://github.com/pallets/click/blob/9da166957f5848b641231d485467f6140bca2bc0/src/click/testing.py#L450-L468

  ```python
      @contextlib.contextmanager
      def isolated_filesystem(
          self, temp_dir: t.Optional[t.Union[str, os.PathLike]] = None
      ) -> t.Iterator[str]:
          """A context manager that creates a temporary directory and
          changes the current working directory to it. This isolates tests
          that affect the contents of the CWD to prevent them from
          interfering with each other.
          :param temp_dir: Create the temporary directory under this
              directory. If given, the created directory is not removed
              when exiting.
          .. versionchanged:: 8.0
              Added the ``temp_dir`` parameter.
          """
          cwd = os.getcwd()
          t = tempfile.mkdtemp(dir=temp_dir)
          os.chdir(t)
  ```
  How about if I pass in that optional `temp_dir` as a temp directory created using the `pytest-xdist` aware pytest mechanisms: https://docs.pytest.org/en/6.2.x/tmpdir.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1406#issuecomment-890259755
issue_url: https://api.github.com/repos/simonw/datasette/issues/1406
node_id: IC_kwDOBm6k_c41EEkr
created_at: 2021-07-31T00:04:54Z
updated_at: 2021-07-31T00:04:54Z
author_association: OWNER
body:
  STILL failing. I'm going to try removing all instances of `isolated_filesystem()` in favour of a different pattern using pytest temporary files, then see if I can get that to work without the serial hack. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1407#issuecomment-890388200
issue_url: https://api.github.com/repos/simonw/datasette/issues/1407
node_id: IC_kwDOBm6k_c41Ej7o
created_at: 2021-07-31T18:38:41Z
updated_at: 2021-07-31T18:38:41Z
author_association: OWNER
body:
  The `path` variable there looked like this:

  `/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-696/popen-gw0/uds0/datasette.sock`

  I think what's happening here is that `pytest-xdist` causes `tmp_path_factory.mktemp("uds")` to create significantly longer paths, which in this case is breaking some limit.

  So for this code to work with `pytest-xdist` I need to make sure the random path to `datasette.sock` is shorter.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1407#issuecomment-890388656
issue_url: https://api.github.com/repos/simonw/datasette/issues/1407
node_id: IC_kwDOBm6k_c41EkCw
created_at: 2021-07-31T18:42:41Z
updated_at: 2021-07-31T18:42:41Z
author_association: OWNER
body:
  I'll try `tempfile.gettempdir()` - on macOS it returns something like `'/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T'` which is still long but hopefully not too long.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1406#issuecomment-890390198
issue_url: https://api.github.com/repos/simonw/datasette/issues/1406
node_id: IC_kwDOBm6k_c41Eka2
created_at: 2021-07-31T18:55:33Z
updated_at: 2021-07-31T18:55:33Z
author_association: OWNER
body:
  To clarify: the core problem here is that an error is thrown any time you call `os.getcwd()` but the directory you are currently in has been deleted.

  `runner.isolated_filesystem()` assumes that the current directory in has not been deleted. But the various temporary directory utilities in `pytest` work by creating directories and then deleting them.

  Maybe there's a larger problem here that I play a bit fast and loose with `os.chdir()` in both the test suite and in various lines of code in Datasette itself (in particular in the publish commands)?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1406#issuecomment-890390342
issue_url: https://api.github.com/repos/simonw/datasette/issues/1406
node_id: IC_kwDOBm6k_c41EkdG
created_at: 2021-07-31T18:56:35Z
updated_at: 2021-07-31T18:56:35Z
author_association: OWNER
body:
  But... I've lost enough time to this already, and removing `runner.isolated_filesystem()` has the tests passing again. So I'm not going to work on this any more.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1408#issuecomment-890390845
issue_url: https://api.github.com/repos/simonw/datasette/issues/1408
node_id: IC_kwDOBm6k_c41Ekk9
created_at: 2021-07-31T19:00:32Z
updated_at: 2021-07-31T19:00:32Z
author_association: OWNER
body:
  When I revisit this I can also look at dropping the `@pytest.mark.serial` hack, and maybe the `restore_working_directory()` fixture hack too:

  https://github.com/simonw/datasette/blob/ff253f5242e4b0b5d85d29d38b8461feb5ea997a/pytest.ini#L9-L10

  https://github.com/simonw/datasette/blob/ff253f5242e4b0b5d85d29d38b8461feb5ea997a/tests/conftest.py#L62-L75

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1409#issuecomment-890397124
issue_url: https://api.github.com/repos/simonw/datasette/issues/1409
node_id: IC_kwDOBm6k_c41EmHE
created_at: 2021-07-31T19:51:10Z
updated_at: 2021-07-31T19:51:10Z
author_association: OWNER
body:
  I think I may like `disable_sql` better. Some options:

  - `--setting allow_sql off` (consistent with `allow_facet` and `allow_download` and `allow_csv_stream` - all which default to `on` already)
  - `--setting disable_sql on`
  - `--setting disable_custom_sql on`

  The existence of three `allow_*` settings does make a strong argument for staying consistent with that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1409#issuecomment-890397261
issue_url: https://api.github.com/repos/simonw/datasette/issues/1409
node_id: IC_kwDOBm6k_c41EmJN
created_at: 2021-07-31T19:52:25Z
updated_at: 2021-07-31T19:52:25Z
author_association: OWNER
body:
  I think I can make this modification by teaching the default permissions code here to take the `allow_sql` setting into account: https://github.com/simonw/datasette/blob/ff253f5242e4b0b5d85d29d38b8461feb5ea997a/datasette/default_permissions.py#L38-L45

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1409#issuecomment-890397652
issue_url: https://api.github.com/repos/simonw/datasette/issues/1409
node_id: IC_kwDOBm6k_c41EmPU
created_at: 2021-07-31T19:56:48Z
updated_at: 2021-07-31T19:56:48Z
author_association: OWNER
body:
  The other option would be to use the setting to pick the `default=` argument when calling `self.ds.permission_allowed( request.actor, "execute-sql", resource=database, default=True)`.

  The problem with that is that there are actually a few different places which perform that check, so changing all of them raises the risk of missing one in the future:

  https://github.com/simonw/datasette/blob/a6c8e7fa4cffdeff84e9e755dcff4788fd6154b8/datasette/views/table.py#L436-L444

  https://github.com/simonw/datasette/blob/a6c8e7fa4cffdeff84e9e755dcff4788fd6154b8/datasette/views/table.py#L964-L966

  https://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/views/database.py#L220-L221

  https://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/views/database.py#L343-L345

  https://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/views/database.py#L134-L136

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1409#issuecomment-890397753
issue_url: https://api.github.com/repos/simonw/datasette/issues/1409
node_id: IC_kwDOBm6k_c41EmQ5
created_at: 2021-07-31T19:57:56Z
updated_at: 2021-07-31T19:57:56Z
author_association: OWNER
body:
  I think the correct solution is for the default permissions logic to take the `allow_sql` setting into account, and to return `False` if that setting is set to `off` AND the current actor fails the `actor_matches_allow` checks.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1409#issuecomment-890399806
issue_url: https://api.github.com/repos/simonw/datasette/issues/1409
node_id: IC_kwDOBm6k_c41Emw-
created_at: 2021-07-31T20:18:46Z
updated_at: 2021-07-31T20:18:46Z
author_association: OWNER
body:
  My rationale for removing it: https://github.com/simonw/datasette/issues/813#issuecomment-640916290

  > Naming problem: Datasette already has a config option with this name:
  >
  >     $ datasette serve data.db --config allow_sql:1
  >
  > https://datasette.readthedocs.io/en/stable/config.html#allow-sql
  >
  > It's confusing to have two things called `allow_sql` that do slightly different things.
  >
  > I could retire the `--config allow_sql:0` option entirely, since the new `metadata.json` mechanism can be used to achieve the exact same thing.
  > 
  > I'm going to do that.

  This is true. The `"allow_sql"` permissions block in `metadata.json` does indeed have a name that is easily confused with `--setting allow_sql off`.

  So I definitely need to pick a different name from the setting. `--setting default_allow_sql off` is a good option here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1409#issuecomment-890400059
issue_url: https://api.github.com/repos/simonw/datasette/issues/1409
node_id: IC_kwDOBm6k_c41Em07
created_at: 2021-07-31T20:21:51Z
updated_at: 2021-07-31T20:21:51Z
author_association: OWNER
body:
  One of these two options:

  - `--setting default_allow_sql off`
  - `--setting allow_sql_default off`

  Existing settings from https://docs.datasette.io/en/0.58.1/settings.html with similar names that I need to be consistent with:

  - `default_page_size`
  - `allow_facet`
  - `default_facet_size`
  - `allow_download`
  - `default_cache_ttl`
  - `default_cache_ttl_hashed`
  - `allow_csv_stream`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1409#issuecomment-890400121
issue_url: https://api.github.com/repos/simonw/datasette/issues/1409
node_id: IC_kwDOBm6k_c41Em15
created_at: 2021-07-31T20:22:21Z
updated_at: 2021-07-31T20:23:34Z
author_association: OWNER
body:
  I think `default_allow_sql` is more consistent with the current naming conventions, because both `allow` and `default` are used as prefixes at the moment but neither of them are ever used as a suffix.

  Plus `default_allow_sql off` makes sense to me but `allow_default_sql off` does not - what is "default SQL"?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1409#issuecomment-890400425
issue_url: https://api.github.com/repos/simonw/datasette/issues/1409
node_id: IC_kwDOBm6k_c41Em6p
created_at: 2021-07-31T20:25:16Z
updated_at: 2021-07-31T20:26:25Z
author_association: OWNER
body:
  If I was prone to over-thinking (which I am) I'd note that `allow_facet` and `allow_download` and `allow_csv_stream` are all settings that do NOT have an equivalent in the newer permissions system, which is itself a little weird and inconsistent.

  So maybe there's a future task where I introduce those as both permissions and metadata `"allow_x"` blocks, then rename the settings themselves to be called `default_allow_facet` and `default_allow_download` and `default_allow_csv_stream`.

  If I was going to do that I should get it in before Datasette 1.0.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1411#issuecomment-890441844
issue_url: https://api.github.com/repos/simonw/datasette/issues/1411
node_id: IC_kwDOBm6k_c41ExB0
created_at: 2021-08-01T03:27:30Z
updated_at: 2021-08-01T03:27:30Z
author_association: OWNER
body:
  Confirmed: https://latest.datasette.io/fixtures/neighborhood_search?text=cork&_hide_sql=1 no longer exhibits the bug.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/251#issuecomment-890443079
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/251
node_id: IC_kwDOCGYnMM41ExVH
created_at: 2021-08-01T03:46:43Z
updated_at: 2021-08-01T03:46:43Z
author_association: OWNER
body:
  Note that there's already a concept of `conversions` which might be confused with `convert`? https://sqlite-utils.datasette.io/en/stable/python-api.html#converting-column-values-using-sql-functions

  ```python
  db["example"].insert({
      "name": "The Bigfoot Discovery Museum"
  }, conversions={"name": "upper(?)"})
  ```
  I think that's OK though - that's a Python library feature, `sqlite-utils convert` is a CLI thing.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/251#issuecomment-890446166
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/251
node_id: IC_kwDOCGYnMM41EyFW
created_at: 2021-08-01T04:14:26Z
updated_at: 2021-08-01T04:14:26Z
author_association: OWNER
body:
  Problem with the `-r/--recipe` idea: the `parsedate` and `parsedatetime` and `jsonsplit` recipes in the current `sqlite-transform` tool all take additional options.

  For `sqlite-transform parsedate` and `parsedatetime`:

  ```python
  @click.option(
      "--dayfirst",
      is_flag=True,
      help="Assume day comes first in ambiguous dates, e.g. 03/04/05",
  )
  @click.option(
      "--yearfirst",
      is_flag=True,
      help="Assume year comes first in ambiguous dates, e.g. 03/04/05",
  )
  ```
  For `jsonsplit`:
  ```python
  @click.option("--delimiter", default=",", help="Delimiter to split on")
  @click.option(
      "--type",
      type=click.Choice(("int", "float")),
      help="Type to use for values - int or float (defaults to string)",
  )
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/251#issuecomment-890446506
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/251
node_id: IC_kwDOCGYnMM41EyKq
created_at: 2021-08-01T04:16:36Z
updated_at: 2021-08-01T04:16:36Z
author_association: OWNER
body:
  Back to the design board then. One way to handle this would be the long-form:

  ```
  sqlite-utils convert jsonsplit mydb.db mytable mycolumn
  sqlite-utils convert parsedatetime mydb.db mytable mycolumn
  sqlite-utils convert parsedate mydb.db mytable mycolumn
  sqlite-utils convert lambda mydb.db mytable mycolumn --code='str(value).upper()'
  ```
  I like the idea that `lambda` is the default action, but in this form it's required that the second argument (the word after `convert`) be the name of the recipe that is being applied to avoid any potential confusion with the database filename.

  An ugly solution would be to make all four of those options available on `sqlite-utils convert` - and return an error if you try and use one of those without specifying the accompanying recipe. That's a bit gross though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/251#issuecomment-890446808
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/251
node_id: IC_kwDOCGYnMM41EyPY
created_at: 2021-08-01T04:18:18Z
updated_at: 2021-08-01T04:28:18Z
author_association: OWNER
body:
  Or.... how about making the `parsedate()` and `parsedatetime()` and `jsonsplit()` functions available within the namespace that is configured for the `--code` block?

  Then you could do something like this:

  ```
  sqlite-utils convert mydb.db mytable col --code 'parsedatetime(value, dayfirst=True)'
  sqlite-utils convert mydb.db mytable col --code 'jsonsplit(value, delimiter=":")'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/251#issuecomment-890446943
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/251
node_id: IC_kwDOCGYnMM41EyRf
created_at: 2021-08-01T04:19:09Z
updated_at: 2021-08-01T04:19:09Z
author_association: OWNER
body:
  That's a pretty neat fix, though it's a bit more challenging on the documentation front - maybe the help text for `sqlite-utils convert --help` gets a fair bit longer?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/251#issuecomment-890447102
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/251
node_id: IC_kwDOCGYnMM41EyT-
created_at: 2021-08-01T04:20:18Z
updated_at: 2021-08-01T04:29:26Z
author_association: OWNER
body:
  I could stick them in a `recipe` namespace so you do this:
  ```
  sqlite-utils convert mydb.db mytable col --code 'recipe.parsedatetime(value, dayfirst=True)'
  sqlite-utils convert mydb.db mytable col --code 'recipe.jsonsplit(value, delimiter=":")'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/251#issuecomment-890448119
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/251
node_id: IC_kwDOCGYnMM41Eyj3
created_at: 2021-08-01T04:28:05Z
updated_at: 2021-08-01T04:30:28Z
author_association: OWNER
body:
  In which case I think `--code` should be a positional argument instead:
  ```
  sqlite-utils convert mydb.db mytable col 'recipe.parsedatetime(value, dayfirst=True)'
  sqlite-utils convert mydb.db mytable col 'recipe.jsonsplit(value, delimiter=":")'
  sqlite-utils convert mydb.db mytable col 'recipe.jsonsplit(value, delimiter=":")'
  sqlite-utils convert mydb.db mytable col '{"lower": value.lower(), "upper": value.upper()}' --multi
  ```
  One problem with this: we already accept one or more columns. I think that's OK though since the code is now a required argument, so it means we have to treat everything between the table and the final code argument as a column.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/251#issuecomment-890448190
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/251
node_id: IC_kwDOCGYnMM41Eyk-
created_at: 2021-08-01T04:28:49Z
updated_at: 2021-08-01T04:28:49Z
author_association: OWNER
body:
  Would make sense to accept code from standard input too:

      echo 'value.upper()' | sqlite-utils convert my.db mytable col -

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/251#issuecomment-890448623
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/251
node_id: IC_kwDOCGYnMM41Eyrv
created_at: 2021-08-01T04:33:30Z
updated_at: 2021-08-01T04:33:30Z
author_association: OWNER
body:
  I've started an implementation in the `convert` branch - no documentation yet, and I've not implemented the recipes.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/302#issuecomment-890548009
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/302
node_id: IC_kwDOCGYnMM41FK8p
created_at: 2021-08-01T16:18:13Z
updated_at: 2021-08-01T16:18:13Z
author_association: OWNER
body:
  Basic API design:

      db[table].convert("headline", lambda v: v.upper())

  You can pass a list of columns instead of a single game column string to apply it to multiple columns.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/303#issuecomment-890553014
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/303
node_id: IC_kwDOCGYnMM41FMK2
created_at: 2021-08-01T16:53:35Z
updated_at: 2021-08-02T04:45:19Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/303?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#303](https://codecov.io/gh/simonw/sqlite-utils/pull/303?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (4c3bf97) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/c7e8d72be9fe8fe0811f685a18eebc637662d41b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (c7e8d72) will **increase** coverage by `0.24%`.
  > The diff coverage is `100.00%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/303/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/303?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@            Coverage Diff             @@
  ##             main     #303      +/-   ##
  ==========================================
  + Coverage   96.04%   96.28%   +0.24%     
  ==========================================
    Files           4        5       +1     
    Lines        1998     2129     +131     
  ==========================================
  + Hits         1919     2050     +131     
    Misses         79       79              
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/303?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [sqlite\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/303/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `94.91% <100.00%> (+0.36%)` | :arrow_up: |
  | [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/303/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.97% <100.00%> (+0.10%)` | :arrow_up: |
  | [sqlite\_utils/recipes.py](https://codecov.io/gh/simonw/sqlite-utils/pull/303/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3JlY2lwZXMucHk=) | `100.00% <100.00%> (ø)` | |
  | [sqlite\_utils/utils.py](https://codecov.io/gh/simonw/sqlite-utils/pull/303/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `92.39% <100.00%> (+0.48%)` | :arrow_up: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/303?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/303?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [c7e8d72...4c3bf97](https://codecov.io/gh/simonw/sqlite-utils/pull/303?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/251#issuecomment-890553783
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/251
node_id: IC_kwDOCGYnMM41FMW3
created_at: 2021-08-01T16:59:09Z
updated_at: 2021-08-01T16:59:09Z
author_association: OWNER
body:
  I'm going with `recipes.jsonsplit()` rather than `recipe.jsonsplit()` because the Python module containing the recipes will be called `recipes`. I'll set up a `r.jsonsplit()` shortcut too as a convenience.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/304#issuecomment-890704624
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/304
node_id: IC_kwDOCGYnMM41FxLw
created_at: 2021-08-02T04:28:42Z
updated_at: 2021-08-02T04:28:42Z
author_association: OWNER
body:
  For the command-line version this can duplicate the `--param` option to allow named parameters in the where clause: https://github.com/simonw/sqlite-utils/blob/c7e8d72be9fe8fe0811f685a18eebc637662d41b/sqlite_utils/cli.py#L1096-L1102

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/305#issuecomment-890713185
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/305
node_id: IC_kwDOCGYnMM41FzRh
created_at: 2021-08-02T04:53:53Z
updated_at: 2021-08-02T04:53:53Z
author_association: OWNER
body:
  I'm going to rename `.execute_count()` to `.count_where()` - but I'll keep `.execute_count()` in the codebase since it's a documented API at https://sqlite-utils.datasette.io/en/3.13/python-api.html#count and I don't want to do a major version bump.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/304#issuecomment-891257730
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/304
node_id: IC_kwDOCGYnMM41H4OC
created_at: 2021-08-02T19:00:00Z
updated_at: 2021-08-02T19:00:00Z
author_association: OWNER
body:
  Documented here:

  - https://sqlite-utils.datasette.io/en/latest/python-api.html#converting-data-in-columns
  - https://sqlite-utils.datasette.io/en/latest/cli.html#converting-data-in-columns

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/306#issuecomment-891352587
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/306
node_id: IC_kwDOCGYnMM41IPYL
created_at: 2021-08-02T21:39:34Z
updated_at: 2021-08-02T21:39:34Z
author_association: OWNER
body:
  This older TIL would have saved me some time! https://til.simonwillison.net/sphinx/sphinx-ext-extlinks

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/300#issuecomment-891356906
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/300
node_id: IC_kwDOCGYnMM41IQbq
created_at: 2021-08-02T21:49:10Z
updated_at: 2021-08-02T21:49:10Z
author_association: OWNER
body:
  I'm nervous to set this by default in the Python library because it's a global setting, so I can't isolate it to just code that is being called by a `sqlite_utils` method.

  The new `sqlite-utils convert` command from #251 DOES turn it on - I think having it on by default for a CLI tool that uses user-defined functions makes sense: https://github.com/simonw/sqlite-utils/blob/e83aef951bd3e8c179511faddb607239a5fa8682/sqlite_utils/cli.py#L2006

  I like the suggestion to include this in the documentation. I'll do that!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/298#issuecomment-891359751
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/298
node_id: IC_kwDOCGYnMM41IRIH
created_at: 2021-08-02T21:55:16Z
updated_at: 2021-08-02T21:55:16Z
author_association: OWNER
body:
  This is a feature already! You can do this:

      sqlite-utils insert nl-demo.db mytable data.ndjson --nl

  See https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-newline-delimited-json

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1417#issuecomment-891979858
issue_url: https://api.github.com/repos/simonw/datasette/issues/1417
node_id: IC_kwDOBm6k_c41KohS
created_at: 2021-08-03T16:15:30Z
updated_at: 2021-08-03T16:15:30Z
author_association: OWNER
body:
  Docs: https://pypi.org/project/codespell/

  There's a `codespell --ignore-words=FILE` option for ignoring words. I don't have any that need ignoring yet but I'm going to add that file anyway, that way I can have codespell be a failing test but still provide a way to work around it if it incorrectly flags a correct word.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1418#issuecomment-891984359
issue_url: https://api.github.com/repos/simonw/datasette/issues/1418
node_id: IC_kwDOBm6k_c41Kpnn
created_at: 2021-08-03T16:21:56Z
updated_at: 2021-08-03T16:21:56Z
author_association: OWNER
body:
  Failed with:
  ```
  docs/authentication.rst:63: perfom ==> perform
  docs/authentication.rst:76: perfom ==> perform
  docs/changelog.rst:429: repsonse ==> response
  docs/changelog.rst:503: permissons ==> permissions
  docs/changelog.rst:717: compatibilty ==> compatibility
  docs/changelog.rst:1172: browseable ==> browsable
  docs/deploying.rst:191: similiar ==> similar
  docs/internals.rst:434: Respons ==> Response, respond
  docs/internals.rst:440: Respons ==> Response, respond
  docs/internals.rst:717: tha ==> than, that, the
  docs/performance.rst:42: databse ==> database
  docs/plugin_hooks.rst:667: utilites ==> utilities
  docs/publish.rst:168: countainer ==> container
  docs/settings.rst:352: inalid ==> invalid
  docs/sql_queries.rst:406: preceeded ==> preceded, proceeded
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1418#issuecomment-891987129
issue_url: https://api.github.com/repos/simonw/datasette/issues/1418
node_id: IC_kwDOBm6k_c41KqS5
created_at: 2021-08-03T16:26:01Z
updated_at: 2021-08-03T16:32:20Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1418?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1418](https://codecov.io/gh/simonw/datasette/pull/1418?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (532170f) into [main](https://codecov.io/gh/simonw/datasette/commit/54b6e96ee8aa553b6671e341a1944f93f3fb89c3?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (54b6e96) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1418/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1418?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1418   +/-   ##
  =======================================
    Coverage   91.64%   91.64%           
  =======================================
    Files          34       34           
    Lines        4382     4382           
  =======================================
    Hits         4016     4016           
    Misses        366      366           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1418?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1418?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [54b6e96...532170f](https://codecov.io/gh/simonw/datasette/pull/1418?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1419#issuecomment-892276385
issue_url: https://api.github.com/repos/simonw/datasette/issues/1419
node_id: IC_kwDOBm6k_c41Lw6h
created_at: 2021-08-04T00:58:49Z
updated_at: 2021-08-04T00:58:49Z
author_association: CONTRIBUTOR
body:
  yes, [filter clause on aggregate queries were added to sqlite3 in 3.30](https://www.sqlite.org/releaselog/3_30_1.html)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1420#issuecomment-892365639
issue_url: https://api.github.com/repos/simonw/datasette/issues/1420
node_id: IC_kwDOBm6k_c41MGtH
created_at: 2021-08-04T05:05:07Z
updated_at: 2021-08-04T05:05:07Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/blob/cd8b7bee8fb5c1cdce7c8dbfeb0166011abc72c6/datasette/publish/cloudrun.py#L153-L158

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1420#issuecomment-892372509
issue_url: https://api.github.com/repos/simonw/datasette/issues/1420
node_id: IC_kwDOBm6k_c41MIYd
created_at: 2021-08-04T05:22:29Z
updated_at: 2021-08-04T05:22:29Z
author_association: OWNER
body:
  Testing this manually with:

      datasette publish cloudrun fixtures.db --memory 8G --cpu 4 \
        --service fixtures-over-provisioned-issue-1420 --install datasette-psutil

  And for comparison:

      datasette publish cloudrun fixtures.db --service fixtures-default-issue-1420 \
        --install datasette-psutil

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1420#issuecomment-892374253
issue_url: https://api.github.com/repos/simonw/datasette/issues/1420
node_id: IC_kwDOBm6k_c41MIzt
created_at: 2021-08-04T05:27:21Z
updated_at: 2021-08-04T05:29:59Z
author_association: OWNER
body:
  I'll delete these services shortly afterwards. Right now:

  https://fixtures-over-provisioned-issue-1420-j7hipcg4aq-uc.a.run.app/-/psutil (deployed with `--memory 8G --cpu 4`) returns:
  ```
  process.memory_info()
    pmem(rss=60456960, vms=518930432, shared=0, text=0, lib=0, data=0, dirty=0)

  ...

  psutil.cpu_times(True)
    scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0)
    scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0)
    scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0)
    scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0)

  psutil.virtual_memory()
    svmem(total=2147483648, available=2092531712, percent=2.6, used=33103872, free=2092531712, active=44130304, inactive=10792960, buffers=0, cached=21848064, shared=262144, slab=0)
  ```
  https://fixtures-default-issue-1420-j7hipcg4aq-uc.a.run.app/-/psutil returns:
  ```
  process.memory_info()
    pmem(rss=49324032, vms=140595200, shared=0, text=0, lib=0, data=0, dirty=0)

  ...

  psutil.cpu_times(True)
    scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0)
    scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0)

  psutil.virtual_memory()
    svmem(total=2147483648, available=2091188224, percent=2.6, used=40071168, free=2091188224, active=41586688, inactive=7983104, buffers=0, cached=16224256, shared=262144, slab=0)
  ```
  These numbers are different enough that I assume this works as advertised. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1420#issuecomment-892376353
issue_url: https://api.github.com/repos/simonw/datasette/issues/1420
node_id: IC_kwDOBm6k_c41MJUh
created_at: 2021-08-04T05:33:12Z
updated_at: 2021-08-04T05:33:12Z
author_association: OWNER
body:
  In the Cloud Run console (before I deleted these services) when I click "Edit and deploy new revision" I see this for the default one:

  <img width="551" alt="Deploy_new_revision_–_Cloud_Run_–_datasette_–_Google_Cloud_Platform" src="https://user-images.githubusercontent.com/9599/128127309-7ec2064f-b107-4d99-b630-ee58734c4a72.png">

  And this for the big one:

  <img width="546" alt="Deploy_new_revision_–_Cloud_Run_–_datasette_–_Google_Cloud_Platform" src="https://user-images.githubusercontent.com/9599/128127366-7cdcb245-2e83-4b88-9d4f-0c595a486139.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/308#issuecomment-892988609
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/308
node_id: IC_kwDOCGYnMM41OezB
created_at: 2021-08-04T21:30:59Z
updated_at: 2021-08-04T21:30:59Z
author_association: OWNER
body:
  OK, this works great as a proof-of-concept (though maybe I should remove the output from the cells so people have to run it themselves in the Binder environment?)

  Next challenge: how extensive should it be? What features should I cover? Should I do a basic tutorial and then several advanced tutorials?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1420#issuecomment-893079520
issue_url: https://api.github.com/repos/simonw/datasette/issues/1420
node_id: IC_kwDOBm6k_c41O0_g
created_at: 2021-08-05T00:54:59Z
updated_at: 2021-08-05T00:54:59Z
author_association: OWNER
body:
  Just saw this error:

  `ERROR: (gcloud.run.deploy) The `--cpu` flag is not supported on the fully managed version of Cloud Run. Specify `--platform gke` or run `gcloud config set run/platform gke` to work with Cloud Run for Anthos deployed on Google Cloud.`

  Which is weird because I managed to run this successfully the other day. Maybe a region difference thing?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1419#issuecomment-893114612
issue_url: https://api.github.com/repos/simonw/datasette/issues/1419
node_id: IC_kwDOBm6k_c41O9j0
created_at: 2021-08-05T02:29:06Z
updated_at: 2021-08-05T02:29:06Z
author_association: CONTRIBUTOR
body:
  there's a lot of complexity here, that's probably not worth addressing. i got what i needed by patching the dockerfile that cloudrun uses to install a newer version of sqlite.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1422#issuecomment-893122356
issue_url: https://api.github.com/repos/simonw/datasette/issues/1422
node_id: IC_kwDOBm6k_c41O_c0
created_at: 2021-08-05T02:52:31Z
updated_at: 2021-08-05T02:52:44Z
author_association: OWNER
body:
  If you do this it should still be possible to view the SQL - which means we need a new parameter. I propose `?_show_sql=1` to over-ride the hidden default.

  I think the configuration should use `hide_sql: true` - looking like this:
  ```yaml
  databases:
    fixtures:
      queries:
        neighborhood_search:
          hide_sql: true
          sql: |-
            select neighborhood, facet_cities.name, state
            from facetable
              join facet_cities on facetable.city_id = facet_cities.id
            where neighborhood like '%' || :text || '%'
            order by neighborhood
          title: Search neighborhoods
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1422#issuecomment-893131703
issue_url: https://api.github.com/repos/simonw/datasette/issues/1422
node_id: IC_kwDOBm6k_c41PBu3
created_at: 2021-08-05T03:16:46Z
updated_at: 2021-08-05T03:16:46Z
author_association: OWNER
body:
  The logic for this is a little bit fiddly, due to the need to switch to using `?_show_sql=1` on the link depending on the context.

  - If metadata says hide and there's no query string, hide and link to `?_show_sql=1`
  - If metadata says hide but query string says `?_show_sql=1`, show and have hide link linking to URL without `?_show_sql=1`
  - Otherwise, show and link to `?_hide_sql=1`
  - ... or if that query string is there then hide and link to URL without `?_hide_sql=1`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1419#issuecomment-893133496
issue_url: https://api.github.com/repos/simonw/datasette/issues/1419
node_id: IC_kwDOBm6k_c41PCK4
created_at: 2021-08-05T03:22:44Z
updated_at: 2021-08-05T03:22:44Z
author_association: OWNER
body:
  I ran into this exact same problem today! I only just learned how to use filter on aggregates: https://til.simonwillison.net/sqlite/sqlite-aggregate-filter-clauses

  A workaround I used is to add this to the deploy command:

      datasette publish cloudrun ... --install=pysqlite3-binary

  This will install the https://pypi.org/project/pysqlite3-binary for package which bundles a more recent SQLite version.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1423#issuecomment-893996604
issue_url: https://api.github.com/repos/simonw/datasette/issues/1423
node_id: IC_kwDOBm6k_c41SU48
created_at: 2021-08-06T04:43:07Z
updated_at: 2021-08-06T04:43:37Z
author_association: OWNER
body:
  Problem: on a page which doesn't have quite enough facet values to trigger the display of the "..." link that links to `?_facet_size=max` the user would still have to manually count the values - up to 30 by default.

  So maybe the count should always be shown, perhaps as a non-bold light colored number?

  I could even hide it in a non-discoverable tooltip.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1423#issuecomment-894452990
issue_url: https://api.github.com/repos/simonw/datasette/issues/1423
node_id: IC_kwDOBm6k_c41UET-
created_at: 2021-08-06T18:49:37Z
updated_at: 2021-08-06T18:49:37Z
author_association: OWNER
body:
  Could display them always, like this:

  <img width="1299" alt="fixtures__compound_three_primary_keys__1_001_rows_and_School_enrollment_analysis_Colorado_-_Google_Docs" src="https://user-images.githubusercontent.com/9599/128557997-6bedb52b-45f1-4ab1-9a79-ddf6c01f6036.png">

  That's with `<span style="font-size: 0.8em;color: #666;">23</span>`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1423#issuecomment-894453520
issue_url: https://api.github.com/repos/simonw/datasette/issues/1423
node_id: IC_kwDOBm6k_c41UEcQ
created_at: 2021-08-06T18:50:40Z
updated_at: 2021-08-06T18:50:40Z
author_association: OWNER
body:
  Point of confusion: if only 30 options are shown, but there's a `...` at the end, what would the number be? It can't be the total number of facets because we haven't counted them all - but if it's just the number of displayed facets that's like to be confusing.

  So the original idea of showing the counts only if `_facet_size=max` is a good one.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1423#issuecomment-894454087
issue_url: https://api.github.com/repos/simonw/datasette/issues/1423
node_id: IC_kwDOBm6k_c41UElH
created_at: 2021-08-06T18:51:42Z
updated_at: 2021-08-06T18:51:42Z
author_association: OWNER
body:
  The invisible tooltip could say "Showing 30 items, more available" (helping save you from counting up to 20 if you know about the secret feature). The numbers could then be fully displayed on the "..." page.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1423#issuecomment-894454644
issue_url: https://api.github.com/repos/simonw/datasette/issues/1423
node_id: IC_kwDOBm6k_c41UEt0
created_at: 2021-08-06T18:52:49Z
updated_at: 2021-08-06T18:52:49Z
author_association: OWNER
body:
  This means that the counts would be unavailable to users who cannot see tooltips (e.g. mobile users) on pages that did not have any facets that broke the 30 limit and hence displayed that "..." link.

  I think I'm OK with that, for the moment. May revisit in the future.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/google-takeout-to-sqlite/pull/8#issuecomment-894581223
issue_url: https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/8
node_id: IC_kwDODFE5qs41Ujnn
created_at: 2021-08-07T00:57:48Z
updated_at: 2021-08-07T00:57:48Z
author_association: NONE
body:
  Just added two more fixes:

  * Added parsing for rfc 2047 encoded unicode headers
  * Body is now stored as TEXT rather than a BLOB regardless of what order the messages are parsed in.

  I was able to run this on my Takeout export and everything seems to work fine. @simonw let me know if this looks good to merge.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1422#issuecomment-894589140
issue_url: https://api.github.com/repos/simonw/datasette/issues/1422
node_id: IC_kwDOBm6k_c41UljU
created_at: 2021-08-07T01:58:16Z
updated_at: 2021-08-07T01:58:24Z
author_association: OWNER
body:
  Also need to consider this hidden field - it should pass the `_hide_sql` or `_show_sql` parameters depending on the same logic: https://github.com/simonw/datasette/blob/acc22436622ff8476c30acf45ed60f54b4aaa5d9/datasette/templates/query.html#L47-L49

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1421#issuecomment-894606796
issue_url: https://api.github.com/repos/simonw/datasette/issues/1421
node_id: IC_kwDOBm6k_c41Up3M
created_at: 2021-08-07T05:16:39Z
updated_at: 2021-08-07T05:16:39Z
author_association: OWNER
body:
  Urgh, yeah I've seen this one before. Fixing it pretty much requires writing a full SQLite SQL syntax parser in Python, which is frustratingly complicated for solving this issue!

  You can work around this for a canned query by using the optional `params:` argument documented here: https://docs.datasette.io/en/stable/sql_queries.html#canned-query-parameters

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1421#issuecomment-894606843
issue_url: https://api.github.com/repos/simonw/datasette/issues/1421
node_id: IC_kwDOBm6k_c41Up37
created_at: 2021-08-07T05:17:12Z
updated_at: 2021-08-07T05:17:12Z
author_association: OWNER
body:
  Marking this blocked because I don't have a way around the needing-a-SQLite-SQL-parser problem at the moment.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1422#issuecomment-894607989
issue_url: https://api.github.com/repos/simonw/datasette/issues/1422
node_id: IC_kwDOBm6k_c41UqJ1
created_at: 2021-08-07T05:31:57Z
updated_at: 2021-08-07T05:31:57Z
author_association: OWNER
body:
  Demo: https://latest.datasette.io/fixtures/neighborhood_search

  Documentation: https://docs.datasette.io/en/latest/sql_queries.html#additional-canned-query-options

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1424#issuecomment-894864404
issue_url: https://api.github.com/repos/simonw/datasette/issues/1424
node_id: IC_kwDOBm6k_c41VowU
created_at: 2021-08-08T22:24:06Z
updated_at: 2021-08-08T22:24:06Z
author_association: OWNER
body:
  Relevant code: https://github.com/simonw/datasette/blob/de5ce2e56339ad8966f417a4758f7c210c017dec/datasette/database.py#L176-L200

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1424#issuecomment-894864616
issue_url: https://api.github.com/repos/simonw/datasette/issues/1424
node_id: IC_kwDOBm6k_c41Vozo
created_at: 2021-08-08T22:26:08Z
updated_at: 2021-08-08T22:26:08Z
author_association: OWNER
body:
  - `datasette.database.QueryInterrupted` for queries that were interrupted
  - `sqlite3.OperationalError`
  - `sqlite3.DatabaseError` and more

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1424#issuecomment-894864682
issue_url: https://api.github.com/repos/simonw/datasette/issues/1424
node_id: IC_kwDOBm6k_c41Vo0q
created_at: 2021-08-08T22:26:46Z
updated_at: 2021-08-08T22:26:46Z
author_association: OWNER
body:
  Note that the `sqlite3` exceptions are in `sqlite3` if using the Python standard library but are in `pysqlite3` if that module is being used instead.

  So maybe encourage people to use them from `datasette.sqlite.sqlite3` instead, which will point to the correct package.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1424#issuecomment-894864744
issue_url: https://api.github.com/repos/simonw/datasette/issues/1424
node_id: IC_kwDOBm6k_c41Vo1o
created_at: 2021-08-08T22:27:31Z
updated_at: 2021-08-08T22:27:31Z
author_association: OWNER
body:
  https://docs.python.org/3/library/sqlite3.html#exceptions is useful - it looks like `sqlite3.DatabaseError` is the super-class of all of the other exceptions that we might see.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1425#issuecomment-894865323
issue_url: https://api.github.com/repos/simonw/datasette/issues/1425
node_id: IC_kwDOBm6k_c41Vo-r
created_at: 2021-08-08T22:33:19Z
updated_at: 2021-08-08T22:33:19Z
author_association: OWNER
body:
  I can do this with the `await_me_maybe()` function, as seen here: https://github.com/simonw/datasette/blob/a21853c9dade240734abc6b4f750fae09a3e840a/datasette/app.py#L864-L873

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1425#issuecomment-894869692
issue_url: https://api.github.com/repos/simonw/datasette/issues/1425
node_id: IC_kwDOBm6k_c41VqC8
created_at: 2021-08-08T23:08:29Z
updated_at: 2021-08-08T23:08:29Z
author_association: OWNER
body:
  Updated documentation: https://docs.datasette.io/en/latest/plugin_hooks.html#render-cell-value-column-table-database-datasette

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1425#issuecomment-894881016
issue_url: https://api.github.com/repos/simonw/datasette/issues/1425
node_id: IC_kwDOBm6k_c41Vsz4
created_at: 2021-08-09T00:21:53Z
updated_at: 2021-08-09T00:21:53Z
author_association: OWNER
body:
  Still one test failure:
  ```
      def test_hook_render_cell_link_from_json(app_client):
          sql = """
              select '{"href": "http://example.com/", "label":"Example"}'
          """.strip()
          path = "/fixtures?" + urllib.parse.urlencode({"sql": sql})
          response = app_client.get(path)
          td = Soup(response.body, "html.parser").find("table").find("tbody").find("td")
          a = td.find("a")
  >       assert a is not None, str(a)
  E       AssertionError: None
  E       assert None is not None
  ```
  The weird thing about this one is that I can't replicate it on my laptop - but it happens in CI every time, including when I shell in and try to run that single test.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1425#issuecomment-894881448
issue_url: https://api.github.com/repos/simonw/datasette/issues/1425
node_id: IC_kwDOBm6k_c41Vs6o
created_at: 2021-08-09T00:24:25Z
updated_at: 2021-08-09T00:24:39Z
author_association: OWNER
body:
  My hunch is that the "skip this `render_cell()` result if it returns `None`" logic isn't working correctly, ever since I added the `await_me_maybe` line.

  Could that be because Pluggy handles the "do the next if `None` is returned" logic itself, but I'm no-longer returning `None`, I'm returning an awaitable which when awaited returns `None`.

  This would suggest that all of the `await_me_maybe()` plugin hooks have the same bug. That's definitely possible - it may well be that no-one has yet stumbled across a bug caused by a plugin returning an awaitable and hence not being skipped, because plugin hooks that return awaitable are rare enough that no-one has tried two plugins which both use that trick.

  Still don't see why it would pass on my laptop but fail in CI though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1425#issuecomment-894882123
issue_url: https://api.github.com/repos/simonw/datasette/issues/1425
node_id: IC_kwDOBm6k_c41VtFL
created_at: 2021-08-09T00:27:43Z
updated_at: 2021-08-09T00:27:43Z
author_association: OWNER
body:
  Good news: `render_cell()` is the only hook to use `firstresult=True`:

  https://github.com/simonw/datasette/blob/f3c9edb376a13c09b5ecf97c7390f4e49efaadf2/datasette/hookspecs.py#L62-L64

  https://pluggy.readthedocs.io/en/latest/#first-result-only

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1425#issuecomment-894882642
issue_url: https://api.github.com/repos/simonw/datasette/issues/1425
node_id: IC_kwDOBm6k_c41VtNS
created_at: 2021-08-09T00:29:57Z
updated_at: 2021-08-09T00:29:57Z
author_association: OWNER
body:
  Here's the code in `pluggy` that implements this: https://github.com/pytest-dev/pluggy/blob/0a064fe275060dbdb1fe6e10c888e72bc400fb33/src/pluggy/callers.py#L31-L43

  ```python
                  if hook_impl.hookwrapper:
                      try:
                          gen = hook_impl.function(*args)
                          next(gen)  # first yield
                          teardowns.append(gen)
                      except StopIteration:
                          _raise_wrapfail(gen, "did not yield")
                  else:
                      res = hook_impl.function(*args)
                      if res is not None:
                          results.append(res)
                          if firstresult:  # halt further impl calls
                              break
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1425#issuecomment-894883664
issue_url: https://api.github.com/repos/simonw/datasette/issues/1425
node_id: IC_kwDOBm6k_c41VtdQ
created_at: 2021-08-09T00:33:56Z
updated_at: 2021-08-09T00:33:56Z
author_association: OWNER
body:
  I could extract that code out and write my own function which implements the equivalent of calling `pm.hook.render_cell(...)` but runs `await_me_maybe()` before checking if `res is not None`.

  That's pretty nasty.

  Could I instead call the plugin hook normally, but then have additional logic which says "if I await it and it returns `None` then try calling the hook again but skip this one" - not sure if there's a way to do that either.

  I could remove the `firstresult=True` from the hookspec - which would cause it to call and return ALL hooks - but then in my own code use only the first one. This is slightly less efficient (since it calls all the hooks and then discards all-but-one value) but it's the least unpleasant in terms of the code I would have to write - plus I don't think it's going to be THAT common for someone to have multiple expensive `render_cell()` hooks installed at once (they are usually pretty cheap).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1425#issuecomment-894893319
issue_url: https://api.github.com/repos/simonw/datasette/issues/1425
node_id: IC_kwDOBm6k_c41Vv0H
created_at: 2021-08-09T01:08:56Z
updated_at: 2021-08-09T01:09:12Z
author_association: OWNER
body:
  Demo: https://latest.datasette.io/fixtures/simple_primary_key shows `RENDER_CELL_ASYNC_RESULT` where the CSV version shows `RENDER_CELL_ASYNC`: https://latest.datasette.io/fixtures/simple_primary_key.csv - because of this test plugin code: https://github.com/simonw/datasette/blob/a390bdf9cef01d8723d025fc3348e81345ff4856/tests/plugins/my_plugin.py#L98-L122

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1425#issuecomment-894900267
issue_url: https://api.github.com/repos/simonw/datasette/issues/1425
node_id: IC_kwDOBm6k_c41Vxgr
created_at: 2021-08-09T01:31:22Z
updated_at: 2021-08-09T01:31:22Z
author_association: OWNER
body:
  I used this to build a new plugin: https://github.com/simonw/datasette-query-links

  Demo here: https://latest-with-plugins.datasette.io/fixtures?sql=select%0D%0A++%27select+*+from+[facetable]%27+as+query%0D%0Aunion%0D%0Aselect%0D%0A++%27select+sqlite_version()%27%0D%0Aunion%0D%0Aselect%0D%0A++%27select+this+is+invalid+SQL+so+will+not+be+linked%27

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1421#issuecomment-894921512
issue_url: https://api.github.com/repos/simonw/datasette/issues/1421
node_id: IC_kwDOBm6k_c41V2so
created_at: 2021-08-09T03:05:26Z
updated_at: 2021-08-09T03:05:26Z
author_association: OWNER
body:
  I may have a way to work around this, using `explain`. Consider this query:

  ```sql
  select * from facetable
  where state = :state
  and on_earth = :on_earth
  and neighborhood not like '00:04'
  ```
  Datasette currently gets confused and shows three form fields: https://latest.datasette.io/fixtures?sql=select+*+from+facetable%0D%0Awhere+state+%3D+%3Astate%0D%0Aand+on_earth+%3D+%3Aon_earth%0D%0Aand+neighborhood+not+like+%2700%3A04%27&state=&on_earth=&04=

  <img width="698" alt="fixtures__select___from_facetable_where_state____state_and_on_earth____on_earth_and_neighborhood_not_like__00_04__and_pyinfra_pip_py_at_current_·_Fizzadar_pyinfra" src="https://user-images.githubusercontent.com/9599/128656369-8af860bf-f7c1-4c9c-beba-1eb6887a8336.png">

  But... if I run `explain` [against that](https://latest.datasette.io/fixtures?sql=explain+select+*+from+facetable%0D%0Awhere+state+%3D+%3Astate%0D%0Aand+on_earth+%3D+%3Aon_earth%0D%0Aand+neighborhood+not+like+%2700%3A04%27&state=&on_earth=&04=) I get this (truncated):

  addr | opcode | p1 | p2 | p3 | p4 | p5 | comment
  -- | -- | -- | -- | -- | -- | -- | --
  20 | ResultRow | 6 | 10 | 0 |   | 0 |  
  21 | Next | 0 | 3 | 0 |   | 1 |  
  22 | Halt | 0 | 0 | 0 |   | 0 |  
  23 | Transaction | 0 | 0 | 35 | 0 | 1 |  
  24 | Variable | 1 | 2 | 0 | :state | 0 |  
  25 | Variable | 2 | 3 | 0 | :on_earth | 0 |  
  26 | String8 | 0 | 4 | 0 | 00:04 | 0 |  
  27 | Goto | 0 | 1 | 0 |   | 0 |  

  Could it be as simple as pulling out those `Variable` rows to figure out the names of the variables in the query?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1421#issuecomment-894922145
issue_url: https://api.github.com/repos/simonw/datasette/issues/1421
node_id: IC_kwDOBm6k_c41V22h
created_at: 2021-08-09T03:07:38Z
updated_at: 2021-08-09T03:07:38Z
author_association: OWNER
body:
  I hoped this would work:
  ```sql
  with foo as (
    explain select * from facetable
    where state = :state
    and on_earth = :on_earth
    and neighborhood not like '00:04'
  )
  select p4 from foo where opcode = 'Variable'
  ```
   But sadly [it returns an error](https://latest.datasette.io/fixtures?sql=with+foo+as+%28%0D%0A++explain+select+*+from+facetable%0D%0A++where+state+%3D+%3Astate%0D%0A++and+on_earth+%3D+%3Aon_earth%0D%0A++and+neighborhood+not+like+%2700%3A04%27%0D%0A%29%0D%0Aselect+p4+from+foo+where+opcode+%3D+%27Variable%27&state=&on_earth=&04=):

  > near "explain": syntax error

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1421#issuecomment-894922703
issue_url: https://api.github.com/repos/simonw/datasette/issues/1421
node_id: IC_kwDOBm6k_c41V2_P
created_at: 2021-08-09T03:09:29Z
updated_at: 2021-08-09T03:09:29Z
author_association: OWNER
body:
  Relevant code: https://github.com/simonw/datasette/blob/ad90a72afa21b737b162e2bbdddc301a97d575cd/datasette/views/database.py#L225-L231

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1421#issuecomment-894925437
issue_url: https://api.github.com/repos/simonw/datasette/issues/1421
node_id: IC_kwDOBm6k_c41V3p9
created_at: 2021-08-09T03:19:00Z
updated_at: 2021-08-09T03:19:00Z
author_association: OWNER
body:
  This may not work:

  > `ERROR: sql = 'explain select 1 + :one + :two', params = None: You did not supply a value for binding 1.`

  The `explain` queries themselves want me to pass them parameters.

  I could try using the regex to pull out candidates and passing `None` for each of those, including incorrect ones like `:31`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1421#issuecomment-894925914
issue_url: https://api.github.com/repos/simonw/datasette/issues/1421
node_id: IC_kwDOBm6k_c41V3xa
created_at: 2021-08-09T03:20:42Z
updated_at: 2021-08-09T03:20:42Z
author_association: OWNER
body:
  I think this works!

  ```python
  _re_named_parameter = re.compile(":([a-zA-Z0-9_]+)")

  async def derive_named_parameters(db, sql):
      explain = 'explain {}'.format(sql.strip().rstrip(";"))
      possible_params = _re_named_parameter.findall(sql)
      try:
          results = await db.execute(explain, {p: None for p in possible_params})
          return [row["p4"].lstrip(":") for row in results if row["opcode"] == "Variable"]
      except sqlite3.DatabaseError:
          return []
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1421#issuecomment-894927185
issue_url: https://api.github.com/repos/simonw/datasette/issues/1421
node_id: IC_kwDOBm6k_c41V4FR
created_at: 2021-08-09T03:25:01Z
updated_at: 2021-08-09T03:25:01Z
author_association: OWNER
body:
  One catch with this approach: if the SQL query is invalid, the parameters will not be extracted and shown as form fields.

  Maybe that's completely fine? Why display a form if it's going to break when the user actually runs the query?

  But it does bother me. I worry that someone who is still iterating on and editing their query before actually starting to use it might find the behaviour confusing.

  So maybe if the query raises an exception it could fall back on the regular expression results?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1421#issuecomment-894929080
issue_url: https://api.github.com/repos/simonw/datasette/issues/1421
node_id: IC_kwDOBm6k_c41V4i4
created_at: 2021-08-09T03:33:02Z
updated_at: 2021-08-09T03:33:02Z
author_association: OWNER
body:
  Fixed! Fantastic, this one has been bothering me for *years*.

  https://latest.datasette.io/fixtures?sql=select+*+from+facetable%0D%0Awhere+state+%3D+%3Astate%0D%0Aand+on_earth+%3D+%3Aon_earth%0D%0Aand+neighborhood+not+like+%2700%3A04%27

  <img width="707" alt="fixtures__select___from_facetable_where_state____state_and_on_earth____on_earth_and_neighborhood_not_like__00_04__and_pyinfra_pip_py_at_current_·_Fizzadar_pyinfra" src="https://user-images.githubusercontent.com/9599/128657807-80ac818d-6fd9-4f70-ad26-900cec6a7482.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1421#issuecomment-894929769
issue_url: https://api.github.com/repos/simonw/datasette/issues/1421
node_id: IC_kwDOBm6k_c41V4tp
created_at: 2021-08-09T03:36:49Z
updated_at: 2021-08-09T03:36:49Z
author_association: OWNER
body:
  SQLite carries a warning about using `EXPLAIN` like this: https://www.sqlite.org/lang_explain.html

  > The output from EXPLAIN and EXPLAIN QUERY PLAN is intended for interactive analysis and troubleshooting only. The details of the output format are subject to change from one release of SQLite to the next. Applications should not use EXPLAIN or EXPLAIN QUERY PLAN since their exact behavior is variable and only partially documented.

  I think that's OK here, because of the regular expression fallback. If the format changes in the future in a way that breaks the query the error should be caught and the regex-captured parameters should be returned instead.

  Hmmm... actually that's not entirely true:

  https://github.com/simonw/datasette/blob/b1fed48a95516ae84c0f020582303ab50ab817e2/datasette/utils/__init__.py#L1084-L1091

  If the format changes such that the same columns are returned but the `[row["p4"].lstrip(":") for row in results if row["opcode"] == "Variable"]` list comprehension returns an empty array it will break Datasette!

  I'm going to take that risk for the moment, but I'll actively watch out for problems in the future. If this does turn out to be bad I can always go back to the pure regular expression mechanism.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1421#issuecomment-894930013
issue_url: https://api.github.com/repos/simonw/datasette/issues/1421
node_id: IC_kwDOBm6k_c41V4xd
created_at: 2021-08-09T03:38:06Z
updated_at: 2021-08-09T03:38:06Z
author_association: OWNER
body:
  Amusing edge-case: if you run this against a `explain ...` query it falls back to using regular expressions, because `explain explain select ...` is invalid SQL. https://latest.datasette.io/fixtures?sql=explain+select+*+from+facetable%0D%0Awhere+state+%3D+%3Astate%0D%0Aand+on_earth+%3D+%3Aon_earth%0D%0Aand+neighborhood+not+like+%2700%3A04%27&state=&on_earth=

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1425#issuecomment-895003796
issue_url: https://api.github.com/repos/simonw/datasette/issues/1425
node_id: IC_kwDOBm6k_c41WKyU
created_at: 2021-08-09T07:14:35Z
updated_at: 2021-08-09T07:14:35Z
author_association: CONTRIBUTOR
body:
  I believe this also provides a workaround for the problem I face in https://github.com/simonw/datasette/issues/1300. 

  Now I should be able to get table PKs and generate a row URL. I'll test this out and report my findings.


  ```py
  from datasette.utils import path_from_row_pks

  pks = await db.primary_keys(table)
  url = self.ds.urls.row_blob(
      database,
      table,
      path_from_row_pks(row, pks, not pks),
      column,
  )
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1426#issuecomment-895500565
issue_url: https://api.github.com/repos/simonw/datasette/issues/1426
node_id: IC_kwDOBm6k_c41YEEV
created_at: 2021-08-09T20:00:04Z
updated_at: 2021-08-09T20:00:04Z
author_association: OWNER
body:
  A few options for how this would work:

  - `datasette ... --robots allow`
  - `datasette ... --setting robots allow`

  Options could be:

  - `allow` - allow all crawling
  - `deny` - deny all crawling
  - `limited` - allow access to the homepage and the index pages for each database and each table, but disallow crawling any further than that

  The "limited" mode is particularly interesting. Could even make it the default, but I think that may be a bit too confusing. Idea would be to get the key pages indexed but use `nofollow` to discourage crawlers from indexing individual row pages or deep pages like `https://datasette.io/content/repos?_facet=owner&_facet=language&_facet_array=topics&topics__arraycontains=sqlite#facet-owner`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1426#issuecomment-895509536
issue_url: https://api.github.com/repos/simonw/datasette/issues/1426
node_id: IC_kwDOBm6k_c41YGQg
created_at: 2021-08-09T20:12:57Z
updated_at: 2021-08-09T20:12:57Z
author_association: OWNER
body:
  I could try out the `X-Robots` HTTP header too: https://developers.google.com/search/docs/advanced/robots/robots_meta_tag#xrobotstag

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1426#issuecomment-895510773
issue_url: https://api.github.com/repos/simonw/datasette/issues/1426
node_id: IC_kwDOBm6k_c41YGj1
created_at: 2021-08-09T20:14:50Z
updated_at: 2021-08-09T20:19:22Z
author_association: OWNER
body:
  https://twitter.com/mal/status/1424825895139876870

  > True pinging google should be part of the build process on a static site :)

  That's another aspect of this: if you DO want your site crawled, teaching the `datasette publish` command how to ping Google when a deploy has gone out could be a nice improvement.

  Annoyingly it looks like you need to configure an auth token of some sort in order to use their API though, which is likely too much hassle to be worth building into Datasette itself: https://developers.google.com/search/apis/indexing-api/v3/using-api

  ```
  curl -X POST https://indexing.googleapis.com/v3/urlNotifications:publish -d '{
    "url": "https://careers.google.com/jobs/google/technical-writer",
    "type": "URL_UPDATED"
  }' -H "Content-Type: application/json"

  {
    "error": {
      "code": 401,
      "message": "Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.",
      "status": "UNAUTHENTICATED"
    }
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1426#issuecomment-895522818
issue_url: https://api.github.com/repos/simonw/datasette/issues/1426
node_id: IC_kwDOBm6k_c41YJgC
created_at: 2021-08-09T20:34:10Z
updated_at: 2021-08-09T20:34:10Z
author_association: OWNER
body:
  At the very least Datasette should serve a blank `/robots.txt` by default - I'm seeing a ton of 404s for it in the logs.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/310#issuecomment-895571420
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/310
node_id: IC_kwDOCGYnMM41YVXc
created_at: 2021-08-09T21:44:38Z
updated_at: 2021-08-09T21:44:38Z
author_association: OWNER
body:
  When I ship this I should update the TILs at https://til.simonwillison.net/cloudrun/tailing-cloud-run-request-logs and https://til.simonwillison.net/jq/flatten-nested-json-objects-jq to reference it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/309#issuecomment-895577012
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/309
node_id: IC_kwDOCGYnMM41YWu0
created_at: 2021-08-09T21:55:52Z
updated_at: 2021-08-09T21:59:03Z
author_association: OWNER
body:
  Yeah this error message could certainly be more helpful.

  I thought `OverflowError` might be one of the SQLite exceptions: https://docs.python.org/3/library/sqlite3.html#exceptions - but it turns out it's actually reusing the Python built-in `OverflowError` class:
  ```python
  import sqlite3
  db = sqlite3.connect(":memory:")
  caught = []
  try:
      db.execute("create table foo (number integer)")
      db.execute("insert into foo (number) values (?)", [34223049823094832094802398430298048240])
  except Exception as e:
      print(e)
      caught.append(e)
  isinstance(caught[0], OverflowError)
  ```
  Here's where that happens in the Python `sqlite3` module code: https://github.com/python/cpython/blob/058fb35b57ca8c5063d16ec818e668b3babfea65/Modules/_sqlite/util.c#L123-L124

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/309#issuecomment-895581038
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/309
node_id: IC_kwDOCGYnMM41YXtu
created_at: 2021-08-09T22:03:54Z
updated_at: 2021-08-09T23:39:53Z
author_association: OWNER
body:
  Steps to reproduce:

      echo '{"v": 34223049823094832094802398430298048240}' | sqlite-utils insert /tmp/blah.db row -

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/309#issuecomment-895587282
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/309
node_id: IC_kwDOCGYnMM41YZPS
created_at: 2021-08-09T22:15:25Z
updated_at: 2021-08-09T22:15:25Z
author_association: OWNER
body:
  I'm going to use a bit of a dirty trick for this one: I'm going to recursively inspect the stack on an error and try to find the `sql` and `params` variables.

  That way I can handle this all at the CLI layer without changing the exceptions that are being raised by the Python library.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/309#issuecomment-895587441
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/309
node_id: IC_kwDOCGYnMM41YZRx
created_at: 2021-08-09T22:15:45Z
updated_at: 2021-08-09T22:15:45Z
author_association: OWNER
body:
  ```
  OverflowError: Python int too large to convert to SQLite INTEGER
  >>> import sys
  >>> def find_variables(tb, vars):
          to_find = list(vars)
          found = {}
          for var in to_find:
              if var in tb.tb_frame.f_locals:
                  vars.remove(var)
                  found[var] = tb.tb_frame.f_locals[var]
          if vars and tb.tb_next:
              found.update(find_variables(tb.tb_next, vars))
          return found
  ... 
  >>> find_variables(sys.last_traceback, ["sql", "params"])
  {'params': [34223049823094832094802398430298048240], 'sql': 'INSERT INTO [row] ([v]) VALUES (?);'}
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/309#issuecomment-895592507
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/309
node_id: IC_kwDOCGYnMM41Yag7
created_at: 2021-08-09T22:26:28Z
updated_at: 2021-08-09T22:33:48Z
author_association: OWNER
body:
  Demo:
  ```
  $ echo '{"v": 34223049823094832094802398430298048240}' | sqlite-utils insert /tmp/blah.db row -                   
  Error: Python int too large to convert to SQLite INTEGER

  sql = INSERT INTO [row] ([v]) VALUES (?);
  parameters = [34223049823094832094802398430298048240]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/309#issuecomment-895622908
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/309
node_id: IC_kwDOCGYnMM41Yh78
created_at: 2021-08-09T23:40:29Z
updated_at: 2021-08-09T23:40:29Z
author_association: OWNER
body:
  TIL about how the stack inspection works: https://til.simonwillison.net/python/find-local-variables-in-exception-traceback

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/311#issuecomment-896131902
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/311
node_id: IC_kwDOCGYnMM41aeM-
created_at: 2021-08-10T16:31:51Z
updated_at: 2021-08-10T16:31:51Z
author_association: OWNER
body:
  `make livehtml` wasn't picking up changes I made to the docstrings `.py` files.

  Fix was to change it to this:
  ```
  sphinx-autobuild -a -b html "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(0) --watch ../sqlite_utils
  ```
  See https://github.com/executablebooks/sphinx-autobuild#relevant-sphinx-bugs - though that suggested `-a` but didn't suggest `--watch`, which is a tip I got from https://github.com/executablebooks/sphinx-autobuild#working-on-a-sphinx-html-theme

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896154028
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/312
node_id: IC_kwDOCGYnMM41ajms
created_at: 2021-08-10T17:01:06Z
updated_at: 2021-08-10T17:01:06Z
author_association: OWNER
body:
  On Python 3.6:

  ```
  sqlite_utils/db.py:366: in Database
      def tables(self) -> List[Table]:
  E   NameError: name 'Table' is not defined
  ```
  Python 3.7 can fix this with `from __future__ import annotations` but since we still support 3.6 I'll have to use a string instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896156971
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/312
node_id: IC_kwDOCGYnMM41akUr
created_at: 2021-08-10T17:04:22Z
updated_at: 2021-08-10T17:05:59Z
author_association: OWNER
body:
  I'm going to get Read The Docs to build the docs for this branch too - on https://readthedocs.org/projects/sqlite-utils/versions/ I am clicking this button:

  <img width="881" alt="Versions___Read_the_Docs" src="https://user-images.githubusercontent.com/9599/128903015-0f2649e7-34a8-4dfd-984c-b1b5bd8a98be.png">

  I then set it to "active" (so pushes to the branch will build it) and "hidden" (so it wouldn't show up in search or in the navigation menu). https://docs.readthedocs.io/en/stable/versions.html#version-states

  <img width="846" alt="autodoc_-_sqlite-utils___Read_the_Docs" src="https://user-images.githubusercontent.com/9599/128903217-efe827c6-4f54-42dd-943e-ddf9ab3a05d8.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896162082
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/312
node_id: IC_kwDOCGYnMM41alki
created_at: 2021-08-10T17:10:39Z
updated_at: 2021-08-10T23:07:35Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/312?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#312](https://codecov.io/gh/simonw/sqlite-utils/pull/312?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (43bc064) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/ee469e3122d6f5973ec2584c1580d930daca2e7c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (ee469e3) will **decrease** coverage by `0.02%`.
  > The diff coverage is `96.84%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/312/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/312?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@            Coverage Diff             @@
  ##             main     #312      +/-   ##
  ==========================================
  - Coverage   96.30%   96.28%   -0.03%     
  ==========================================
    Files           5        5              
    Lines        2168     2179      +11     
  ==========================================
  + Hits         2088     2098      +10     
  - Misses         80       81       +1     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/312?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/312/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.91% <96.84%> (-0.08%)` | :arrow_down: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/312?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/312?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [ee469e3...43bc064](https://codecov.io/gh/simonw/sqlite-utils/pull/312?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896174456
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/312
node_id: IC_kwDOCGYnMM41aol4
created_at: 2021-08-10T17:27:01Z
updated_at: 2021-08-10T17:27:01Z
author_association: OWNER
body:
  Docs are now building at https://sqlite-utils.datasette.io/en/autodoc/reference.html

  But there's a problem! The page is semi-blank:

  <img width="663" alt="Reference_—_sqlite-utils_3_15-6-gc11ff89_documentation" src="https://user-images.githubusercontent.com/9599/128906373-7463e0c9-3b08-4b15-a73a-84b8ba6ab342.png">

  I need to teach Read The Docs how to ensure `sqlite_utils` is available for introspection.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896175438
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/312
node_id: IC_kwDOCGYnMM41ao1O
created_at: 2021-08-10T17:28:19Z
updated_at: 2021-08-10T17:28:19Z
author_association: OWNER
body:
  https://sphinx-rtd-tutorial.readthedocs.io/en/latest/sphinx-config.html#autodoc-configuration says do something like this at the top of `conf.py`:

  ```python
  import os
  import sys
  sys.path.insert(0, os.path.abspath('../../simpleble/'))
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896180956
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/312
node_id: IC_kwDOCGYnMM41aqLc
created_at: 2021-08-10T17:35:51Z
updated_at: 2021-08-10T17:35:51Z
author_association: OWNER
body:
  Reading the rest of https://sphinx-rtd-tutorial.readthedocs.io/en/latest/sphinx-config.html#autodoc-configuration it suggests using a `requirements.txt` file to install dependencies - but I use `setup.py` for that so I need to figure out a different pattern here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896182934
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/312
node_id: IC_kwDOCGYnMM41aqqW
created_at: 2021-08-10T17:38:44Z
updated_at: 2021-08-10T17:38:44Z
author_association: OWNER
body:
  From https://docs.readthedocs.io/en/stable/config-file/v2.html#packages it looks like I can tell Read The Docs to run `pip install -e .` using a `.readthedocs.yaml` configuration:

  ```yaml
  version: 2

  sphinx:
     configuration: docs/conf.py

  python:
    version: "3.9"
    install:
      - method: pip
        path: .
        extra_requirements:
          - docs
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896200682
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/312
node_id: IC_kwDOCGYnMM41au_q
created_at: 2021-08-10T18:03:40Z
updated_at: 2021-08-10T18:03:40Z
author_association: OWNER
body:
  Adding type signatures to `create_table()` and `.create_table_sql()` is a bit too involved, I'll do that in a separate issue.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896284722
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/312
node_id: IC_kwDOCGYnMM41bDgy
created_at: 2021-08-10T20:08:03Z
updated_at: 2021-08-10T20:08:21Z
author_association: OWNER
body:
  Spotted a rogue backtick:

  ![A0147E27-7506-49B0-BEFB-20D99BBFEBAD](https://user-images.githubusercontent.com/9599/128927930-b3333dee-a385-409b-a945-f108e6ea40df.jpeg)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/314#issuecomment-896344833
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/314
node_id: IC_kwDOCGYnMM41bSMB
created_at: 2021-08-10T22:07:34Z
updated_at: 2021-08-10T22:07:34Z
author_association: OWNER
body:
  Also the `.insert()` family of methods - they look pretty ugly in Sphinx right now:

  <img width="688" alt="API_Reference_—_sqlite-utils_3_15-16-gf51b712_documentation" src="https://user-images.githubusercontent.com/9599/128941267-51b290cb-6c3d-44b7-a584-e3ae3a819e8b.png">

  I should probably define reusable types for things like `pk=`, which have complex type signatures (a string or a list/tuple of strings) and show up in multiple places.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/314#issuecomment-896369551
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/314
node_id: IC_kwDOCGYnMM41bYOP
created_at: 2021-08-10T23:06:41Z
updated_at: 2021-08-10T23:06:41Z
author_association: OWNER
body:
  I took a big bite out of this when I annotated the ``.insert()`` method - but there are a bunch of other places that still need doing: https://github.com/simonw/sqlite-utils/blob/43bc06481783c3cfcee70c0cb541a686e8894adb/sqlite_utils/db.py#L2382-L2397

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/google-takeout-to-sqlite/pull/8#issuecomment-896378525
issue_url: https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/8
node_id: IC_kwDODFE5qs41baad
created_at: 2021-08-10T23:28:45Z
updated_at: 2021-08-10T23:28:45Z
author_association: NONE
body:
  I added parsing of text/html emails using BeautifulSoup.

  Around half of the emails in my archive don't include a text/plain payload so adding html parsing makes a good chunk of them searchable.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/311#issuecomment-896381184
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/311
node_id: IC_kwDOCGYnMM41bbEA
created_at: 2021-08-10T23:33:33Z
updated_at: 2021-08-10T23:33:33Z
author_association: OWNER
body:
  Now live at https://sqlite-utils.datasette.io/en/latest/reference.html

  TIL from what I learned today here: https://til.simonwillison.net/sphinx/sphinx-autodoc

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/186#issuecomment-897588624
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/186
node_id: IC_kwDOCGYnMM41gB2Q
created_at: 2021-08-12T12:13:25Z
updated_at: 2021-08-12T12:13:25Z
author_association: NONE
body:
  I think I ran into an issue that's perhaps related with `extract()`

  I have a case where I want to create a lookup table for all the related title data where there are possibly multiple null values in the related columns ....
  ```python3
  sql = """\
  INSERT INTO "circulation_info" ("item_id", "bib_title", "bib_creator", "bib_format", "bib_pub_year", "checkout_date")
  VALUES
  (1, "title one", "creator one", "Book", 2018, "2021-08-12 00:01"),
  (2, "title two", "creator one", "Book", 2019, "2021-08-12 00:02"),
  (3, "title three", NULL, "DVD", 2020, "2021-08-12 00:03"),
  (4, "title four", NULL, "DVD", NULL, "2021-08-12 00:04"),
  (5, "title five", NULL, "DVD", NULL, "2021-08-12 00:05")
  """

  with sqlite3.connect('test_bib.db') as con:
      con.execute(sql)
  ```

  when I run the `extract()` method ... 

  ```python3
  db["circulation_info"].extract(
      [
          "bib_title",
          "bib_creator",
          "bib_format"        
      ],
      table="bib_info", 
      fk_column="bib_info_id"
  )

  db = sqlite_utils.Database("test_bib.db")

  for row in db["circulation_info"].rows:
      print(row)

  print("\n---\n")

  for row in db["bib_info"].rows:
      print(row)
  ```

  results in this .. 
  ```
  {'id': 1, 'item_id': 1, 'bib_info_id': 1, 'bib_pub_year': 2018, 'checkout_date': '2021-08-12 00:01'}
  {'id': 2, 'item_id': 2, 'bib_info_id': 2, 'bib_pub_year': 2019, 'checkout_date': '2021-08-12 00:02'}
  {'id': 3, 'item_id': 3, 'bib_info_id': None, 'bib_pub_year': 2020, 'checkout_date': '2021-08-12 00:03'}
  {'id': 4, 'item_id': 4, 'bib_info_id': None, 'bib_pub_year': None, 'checkout_date': '2021-08-12 00:04'}
  {'id': 5, 'item_id': 5, 'bib_info_id': None, 'bib_pub_year': None, 'checkout_date': '2021-08-12 00:05'}

  ---

  {'id': 1, 'bib_title': 'title one', 'bib_creator': 'creator one', 'bib_format': 'Book'}
  {'id': 2, 'bib_title': 'title two', 'bib_creator': 'creator one', 'bib_format': 'Book'}
  {'id': 3, 'bib_title': 'title three', 'bib_creator': None, 'bib_format': 'DVD'}
  {'id': 4, 'bib_title': 'title four', 'bib_creator': None, 'bib_format': 'DVD'}
  {'id': 5, 'bib_title': 'title five', 'bib_creator': None, 'bib_format': 'DVD'}
  ```

  Seems like it's correctly generating the row data for those lookups, but it's not correctly updating the foreign key back to the primary table? Looks like it just results in a `NULL` value in that original table.

  Any ideas on why? Thanks again!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/186#issuecomment-897600677
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/186
node_id: IC_kwDOCGYnMM41gEyl
created_at: 2021-08-12T12:32:14Z
updated_at: 2021-08-12T12:32:14Z
author_association: NONE
body:
  Actually, I forgot to include the `bib_pub_year` in the extract ... 

  But also, I tried again with empty string values instead of `NULL` values and it seems to place the foreign key properly / correctly... 

  ```python3
  sql = """\
  INSERT INTO "circulation_info" ("item_id", "bib_title", "bib_creator", "bib_format", "bib_pub_year", "checkout_date")
  VALUES
  (1, "title one", "creator one", "Book", 2018, "2021-08-12 00:01"),
  (2, "title two", "creator one", "Book", 2019, "2021-08-12 00:02"),
  (3, "title three", "", "DVD", 2020, "2021-08-12 00:03"),
  (4, "title four", "", "DVD", "", "2021-08-12 00:04"),
  (5, "title five", "", "DVD", "", "2021-08-12 00:05")
  """

  with sqlite3.connect('test_bib_2.db') as con:
      con.execute(sql)
  ```

  ```python3
  db["circulation_info"].extract(
      [
          "bib_title",
          "bib_creator",
          "bib_format",
          "bib_pub_year"
      ],
      table="bib_info", 
      fk_column="bib_info_id"
  )
  ```

  ```
  {'id': 1, 'item_id': 1, 'bib_info_id': 1, 'bib_pub_year': 2018, 'checkout_date': '2021-08-12 00:01'}
  {'id': 2, 'item_id': 2, 'bib_info_id': 2, 'bib_pub_year': 2019, 'checkout_date': '2021-08-12 00:02'}
  {'id': 3, 'item_id': 3, 'bib_info_id': 3, 'bib_pub_year': 2020, 'checkout_date': '2021-08-12 00:03'}
  {'id': 4, 'item_id': 4, 'bib_info_id': 4, 'bib_pub_year': '', 'checkout_date': '2021-08-12 00:04'}
  {'id': 5, 'item_id': 5, 'bib_info_id': 5, 'bib_pub_year': '', 'checkout_date': '2021-08-12 00:05'}

  ---

  {'id': 1, 'bib_title': 'title one', 'bib_creator': 'creator one', 'bib_format': 'Book'}
  {'id': 2, 'bib_title': 'title two', 'bib_creator': 'creator one', 'bib_format': 'Book'}
  {'id': 3, 'bib_title': 'title three', 'bib_creator': '', 'bib_format': 'DVD'}
  {'id': 4, 'bib_title': 'title four', 'bib_creator': '', 'bib_format': 'DVD'}
  {'id': 5, 'bib_title': 'title five', 'bib_creator': '', 'bib_format': 'DVD'}
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/942#issuecomment-897996296
issue_url: https://api.github.com/repos/simonw/datasette/issues/942
node_id: IC_kwDOBm6k_c41hlYI
created_at: 2021-08-12T22:01:36Z
updated_at: 2021-08-12T22:01:36Z
author_association: OWNER
body:
  I'm going with `"columns": {"name-of-column": "description-of-column"}`.

  If I decide to make `"col"` and `"nocol"` available in metadata I'll use those as the keys in the metadata, for consistency with the existing query string parameters.

  I'm OK with having both `"columns": ...` and `"col": ...` keys in the metadata, even though they could be a tiny bit confusing without the documentation.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/942#issuecomment-898021895
issue_url: https://api.github.com/repos/simonw/datasette/issues/942
node_id: IC_kwDOBm6k_c41hroH
created_at: 2021-08-12T22:51:36Z
updated_at: 2021-08-12T22:51:36Z
author_association: OWNER
body:
  Prototype:

  <img width="650" alt="fixtures__sortable__201_rows" src="https://user-images.githubusercontent.com/9599/129279808-08d56b2f-a4d6-4147-b3a6-542c62b566c1.png">

  ```diff
  diff --git a/datasette/static/app.css b/datasette/static/app.css
  index c6be1e9..5ca64cb 100644
  --- a/datasette/static/app.css
  +++ b/datasette/static/app.css
  @@ -784,9 +784,14 @@ svg.dropdown-menu-icon {
       font-size: 0.7em;
       color: #666;
       margin: 0;
  -    padding: 0;
       padding: 4px 8px 4px 8px;
   }
  +.dropdown-menu .dropdown-column-description {
  +    margin: 0;
  +    color: #666;
  +    padding: 4px 8px 4px 8px;
  +    max-width: 20em;
  +}
   .dropdown-menu li {
       border-bottom: 1px solid #ccc;
   }
  diff --git a/datasette/static/table.js b/datasette/static/table.js
  index 991346d..a903112 100644
  --- a/datasette/static/table.js
  +++ b/datasette/static/table.js
  @@ -9,6 +9,7 @@ var DROPDOWN_HTML = `<div class="dropdown-menu">
     <li><a class="dropdown-not-blank" href="#">Show not-blank rows</a></li>
   </ul>
   <p class="dropdown-column-type"></p>
  +<p class="dropdown-column-description"></p>
   </div>`;
   
   var DROPDOWN_ICON_SVG = `<svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
  @@ -166,6 +167,14 @@ var DROPDOWN_ICON_SVG = `<svg xmlns="http://www.w3.org/2000/svg" width="14" heig
       } else {
         columnTypeP.style.display = "none";
       }
  +
  +    var columnDescriptionP = menu.querySelector(".dropdown-column-description");
  +    if (th.dataset.columnDescription) {
  +      columnDescriptionP.innerText = th.dataset.columnDescription;
  +      columnDescriptionP.style.display = 'block';
  +    } else {
  +      columnDescriptionP.style.display = 'none';
  +    }
       menu.style.position = "absolute";
       menu.style.top = menuTop + 6 + "px";
       menu.style.left = menuLeft + "px";
  diff --git a/datasette/templates/_table.html b/datasette/templates/_table.html
  index d765937..649f517 100644
  --- a/datasette/templates/_table.html
  +++ b/datasette/templates/_table.html
  @@ -4,7 +4,7 @@
           <thead>
               <tr>
                   {% for column in display_columns %}
  -                    <th class="col-{{ column.name|to_css_class }}" scope="col" data-column="{{ column.name }}" data-column-type="{{ column.type }}" data-column-not-null="{{ column.notnull }}" data-is-pk="{% if column.is_pk %}1{% else %}0{% endif %}">
  +                    <th {% if column.description %}data-column-description="{{ column.description }}" {% endif %}class="col-{{ column.name|to_css_class }}" scope="col" data-column="{{ column.name }}" data-column-type="{{ column.type }}" data-column-not-null="{{ column.notnull }}" data-is-pk="{% if column.is_pk %}1{% else %}0{% endif %}">
                           {% if not column.sortable %}
                               {{ column.name }}
                           {% else %}
  diff --git a/datasette/views/table.py b/datasette/views/table.py
  index 456d806..486a613 100644
  --- a/datasette/views/table.py
  +++ b/datasette/views/table.py
  @@ -125,6 +125,7 @@ class RowTableShared(DataView):
           """Returns columns, rows for specified table - including fancy foreign key treatment"""
           db = self.ds.databases[database]
           table_metadata = self.ds.table_metadata(database, table)
  +        column_descriptions = table_metadata.get("columns") or {}
           column_details = {col.name: col for col in await db.table_column_details(table)}
           sortable_columns = await self.sortable_columns_for_table(database, table, True)
           pks = await db.primary_keys(table)
  @@ -147,6 +148,7 @@ class RowTableShared(DataView):
                       "is_pk": r[0] in pks_for_display,
                       "type": type_,
                       "notnull": notnull,
  +                    "description": column_descriptions.get(r[0]),
                   }
               )

  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/942#issuecomment-898022235
issue_url: https://api.github.com/repos/simonw/datasette/issues/942
node_id: IC_kwDOBm6k_c41hrtb
created_at: 2021-08-12T22:52:23Z
updated_at: 2021-08-12T22:52:23Z
author_association: OWNER
body:
  I like this. Need to solve for mobile though where the cog menu isn't visible - I think I'll do that with a definition list at the top of the page.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/942#issuecomment-898032118
issue_url: https://api.github.com/repos/simonw/datasette/issues/942
node_id: IC_kwDOBm6k_c41huH2
created_at: 2021-08-12T23:12:00Z
updated_at: 2021-08-12T23:12:00Z
author_association: NONE
body:
  This looks awesome. We'll definitely make extensive use of this feature!

  On Thu, Aug 12, 2021 at 5:52 PM Simon Willison ***@***.***>
  wrote:

  > I like this. Need to solve for mobile though where the cog menu isn't
  > visible - I think I'll do that with a definition list at the top of the
  > page.
  >
  > —
  > You are receiving this because you are subscribed to this thread.
  > Reply to this email directly, view it on GitHub
  > <https://github.com/simonw/datasette/issues/942#issuecomment-898022235>,
  > or unsubscribe
  > <https://github.com/notifications/unsubscribe-auth/AAERSNYEF6QRZO2HRJGRIWDT4RGDFANCNFSM4QEC6ATA>
  > .
  > Triage notifications on the go with GitHub Mobile for iOS
  > <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
  > or Android
  > <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email>
  > .
  >


  -- 
  Zane A. Selvans, PhD
  Chief Data Wrangler
  Catalyst Cooperative
  https://catalyst.coop
  ***@***.***
  Signal/WhatsApp/SMS: +1 720 443 1363
  Twitter: @ZaneSelvans <https://twitter.com/ZaneSelvans>
  PGP <https://www.gnupg.org/>: 0x64F7B56F3A127B04

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/942#issuecomment-898037456
issue_url: https://api.github.com/repos/simonw/datasette/issues/942
node_id: IC_kwDOBm6k_c41hvbQ
created_at: 2021-08-12T23:23:34Z
updated_at: 2021-08-12T23:23:34Z
author_association: OWNER
body:
  Prototype with a `<dl>`:

  <img width="721" alt="fixtures__sortable__201_rows" src="https://user-images.githubusercontent.com/9599/129282175-c94dcf74-beb3-4f0a-9106-f75d19369fb0.png">

  ```diff
  diff --git a/datasette/static/app.css b/datasette/static/app.css
  index c6be1e9..bf068fd 100644
  --- a/datasette/static/app.css
  +++ b/datasette/static/app.css
  @@ -836,6 +841,16 @@ svg.dropdown-menu-icon {
       background-repeat: no-repeat;
   }
   
  +dl.column-descriptions dt {
  +    font-weight: bold;
  +}
  +dl.column-descriptions dd {
  +    padding-left: 1.5em;
  +    white-space: pre-wrap;
  +    line-height: 1.1em;
  +    color: #666;
  +}
  +
   .anim-scale-in {
       animation-name: scale-in;
       animation-duration: 0.15s;
  diff --git a/datasette/templates/table.html b/datasette/templates/table.html
  index 211352b..466e8a4 100644
  --- a/datasette/templates/table.html
  +++ b/datasette/templates/table.html
  @@ -51,6 +51,14 @@
   
   {% block description_source_license %}{% include "_description_source_license.html" %}{% endblock %}
   
  +{% if metadata.columns %}
  +<dl class="column-descriptions">
  +    {% for column_name, column_description in metadata.columns.items() %}
  +        <dt>{{ column_name }}</dt><dd>{{ column_description }}</dd>
  +    {% endfor %}
  +</dl>
  +{% endif %}
  +
   {% if filtered_table_rows_count or human_description_en %}
       <h3>{% if filtered_table_rows_count or filtered_table_rows_count == 0 %}{{ "{:,}".format(filtered_table_rows_count) }} row{% if filtered_table_rows_count == 1 %}{% else %}s{% endif %}{% endif %}
           {% if human_description_en %}{{ human_description_en }}{% endif %}
   ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/942#issuecomment-898037650
issue_url: https://api.github.com/repos/simonw/datasette/issues/942
node_id: IC_kwDOBm6k_c41hveS
created_at: 2021-08-12T23:23:54Z
updated_at: 2021-08-12T23:23:54Z
author_association: OWNER
body:
  I like this enough that I'm going to ship it as an alpha and try it out on a couple of live projects.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1430#issuecomment-898043575
issue_url: https://api.github.com/repos/simonw/datasette/issues/1430
node_id: IC_kwDOBm6k_c41hw63
created_at: 2021-08-12T23:39:36Z
updated_at: 2021-08-12T23:49:51Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1430?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1430](https://codecov.io/gh/simonw/datasette/pull/1430?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (9419947) into [main](https://codecov.io/gh/simonw/datasette/commit/b1fed48a95516ae84c0f020582303ab50ab817e2?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (b1fed48) will **increase** coverage by `0.00%`.
  > The diff coverage is `100.00%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1430/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1430?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1430   +/-   ##
  =======================================
    Coverage   91.71%   91.71%           
  =======================================
    Files          34       34           
    Lines        4417     4418    +1     
  =======================================
  + Hits         4051     4052    +1     
    Misses        366      366           
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1430?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1430/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `96.00% <100.00%> (+<0.01%)` | :arrow_up: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1430?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1430?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [b1fed48...9419947](https://codecov.io/gh/simonw/datasette/pull/1430?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/942#issuecomment-898050457
issue_url: https://api.github.com/repos/simonw/datasette/issues/942
node_id: IC_kwDOBm6k_c41hymZ
created_at: 2021-08-12T23:59:53Z
updated_at: 2021-08-12T23:59:53Z
author_association: OWNER
body:
  Documentation: https://docs.datasette.io/en/latest/metadata.html#column-descriptions

  Live demo: https://latest.datasette.io/fixtures/roadside_attractions

  <img width="1045" alt="fixtures__roadside_attractions__4_rows" src="https://user-images.githubusercontent.com/9599/129284675-ef2fd6ed-54ba-4f6a-add5-7a9253921279.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/942#issuecomment-898051645
issue_url: https://api.github.com/repos/simonw/datasette/issues/942
node_id: IC_kwDOBm6k_c41hy49
created_at: 2021-08-13T00:02:25Z
updated_at: 2021-08-13T00:02:25Z
author_association: OWNER
body:
  And on mobile:

  ![5FAF8D73-7199-4BB7-A5B8-9E46DCB4A985](https://user-images.githubusercontent.com/9599/129284817-dc13cbf4-144e-4f4c-8fb7-470602e2eea0.jpeg)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-898056013
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: IC_kwDOBm6k_c41hz9N
created_at: 2021-08-13T00:12:09Z
updated_at: 2021-08-13T00:12:09Z
author_association: OWNER
body:
  Having added column metadata in #1430 (ref #942) I could also include a definition list at the top of the query results page exposing the column descriptions for any columns, using the same EXPLAIN mechanism.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-898063815
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: IC_kwDOBm6k_c41h13H
created_at: 2021-08-13T00:33:17Z
updated_at: 2021-08-13T00:33:17Z
author_association: OWNER
body:
  Improved version of that function:
  ```python
  def columns_for_query(conn, sql):
      """
      Given a SQLite connection ``conn`` and a SQL query ``sql``,
      returns a list of ``(table_name, column_name)`` pairs, one
      per returned column. ``(None, None)`` if no table and column
      could be derived.
      """
      rows = conn.execute('explain ' + sql).fetchall()
      table_rootpage_by_register = {r['p1']: r['p2'] for r in rows if r['opcode'] == 'OpenRead'}
      names_by_rootpage = dict(
          conn.execute(
              'select rootpage, name from sqlite_master where rootpage in ({})'.format(
                  ', '.join(map(str, table_rootpage_by_register.values()))
              )
          )
      )
      columns_by_column_register = {}
      for row in rows:
          if row['opcode'] in ('Rowid', 'Column'):
              addr, opcode, table_id, cid, column_register, p4, p5, comment = row
              table = names_by_rootpage[table_rootpage_by_register[table_id]]
              columns_by_column_register[column_register] = (table, cid)
      result_row = [dict(r) for r in rows if r['opcode'] == 'ResultRow'][0]
      registers = list(range(result_row["p1"], result_row["p1"] + result_row["p2"]))
      all_column_names = {}
      for table in names_by_rootpage.values():
          table_xinfo = conn.execute('pragma table_xinfo({})'.format(table)).fetchall()
          for row in table_xinfo:
              all_column_names[(table, row["cid"])] = row["name"]
      final_output = []
      for r in registers:
          try:
              table, cid = columns_by_column_register[r]
              final_output.append((table, all_column_names[table, cid]))
          except KeyError:
              final_output.append((None, None))
      return final_output
  ```
  It works!

  <img width="1440" alt="Banners_and_Alerts_and_fixtures__select_attraction_id__roadside_attractions_name__characteristic_id__attraction_characteristic_name_as_characteristic_from_roadside_attraction_characteristics_join_roadside_attractions_on_roadside_attractions" src="https://user-images.githubusercontent.com/9599/129286745-be5bb268-c31c-413b-9530-c327ae7fd481.png">

  ```diff
  diff --git a/datasette/templates/query.html b/datasette/templates/query.html
  index 75f7f1b..9fe1d4f 100644
  --- a/datasette/templates/query.html
  +++ b/datasette/templates/query.html
  @@ -67,6 +67,8 @@
       </p>
   </form>
   
  +extra_column_info: {{ extra_column_info }}
  +
   {% if display_rows %}
   <p class="export-links">This data as {% for name, url in renderers.items() %}<a href="{{ url }}">{{ name }}</a>{{ ", " if not loop.last }}{% endfor %}, <a href="{{ url_csv }}">CSV</a></p>
   <div class="table-wrapper"><table class="rows-and-columns">
  diff --git a/datasette/views/database.py b/datasette/views/database.py
  index 7c36034..02f8039 100644
  --- a/datasette/views/database.py
  +++ b/datasette/views/database.py
  @@ -10,6 +10,7 @@ import markupsafe
   from datasette.utils import (
       await_me_maybe,
       check_visibility,
  +    columns_for_query,
       derive_named_parameters,
       to_css_class,
       validate_sql_select,
  @@ -248,6 +249,8 @@ class QueryView(DataView):
   
           query_error = None
   
  +        extra_column_info = None
  +
           # Execute query - as write or as read
           if write:
               if request.method == "POST":
  @@ -334,6 +337,10 @@ class QueryView(DataView):
                       database, sql, params_for_query, truncate=True, **extra_args
                   )
                   columns = [r[0] for r in results.description]
  +
  +                # Try to figure out extra column information
  +                db = self.ds.get_database(database)
  +                extra_column_info = await db.execute_fn(lambda conn: columns_for_query(conn, sql))
               except sqlite3.DatabaseError as e:
                   query_error = e
                   results = None
  @@ -462,6 +469,7 @@ class QueryView(DataView):
                   "show_hide_text": show_hide_text,
                   "show_hide_hidden": markupsafe.Markup(show_hide_hidden),
                   "hide_sql": hide_sql,
  +                "extra_column_info": extra_column_info,
               }
   
           return (
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-898065011
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: IC_kwDOBm6k_c41h2Jz
created_at: 2021-08-13T00:36:30Z
updated_at: 2021-08-13T00:36:30Z
author_association: OWNER
body:
  > https://latest.datasette.io/fixtures?sql=explain+select+*+from+paginated_view will be an interesting test query - because `paginated_view` is defined like this:
  > 
  > ```sql
  > CREATE VIEW paginated_view AS
  >     SELECT
  >         content,
  >         '- ' || content || ' -' AS content_extra
  >     FROM no_primary_key;
  > ```
  > 
  > So this will help test that the mechanism isn't confused by output columns that are created through a concatenation expression.

  Here's what it does for that:

  <img width="748" alt="fixtures__select___from_paginated_view" src="https://user-images.githubusercontent.com/9599/129286962-426bfa56-3946-447a-996d-668b4d80f5c1.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-898065948
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: IC_kwDOBm6k_c41h2Yc
created_at: 2021-08-13T00:38:58Z
updated_at: 2021-08-13T00:38:58Z
author_association: OWNER
body:
  Trying to run `explain select * from facetable` fails with an error in my prototype, because it tries to execute `explain explain select * from facetable` - so I need to spot that error and ignore it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-898066466
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: IC_kwDOBm6k_c41h2gi
created_at: 2021-08-13T00:40:24Z
updated_at: 2021-08-13T00:40:24Z
author_association: OWNER
body:
  It figures out renamed columns too:

  <img width="694" alt="fixtures__select_created__state_as_the_state_from_facetable" src="https://user-images.githubusercontent.com/9599/129287208-1347fe80-f62e-4ed2-80c6-06a223cbe749.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1431#issuecomment-898072940
issue_url: https://api.github.com/repos/simonw/datasette/issues/1431
node_id: IC_kwDOBm6k_c41h4Fs
created_at: 2021-08-13T00:58:40Z
updated_at: 2021-08-13T00:58:40Z
author_association: OWNER
body:
  While I'm doing this I should rename this internal variable to avoid confusion in the future:

  https://github.com/simonw/datasette/blob/e837095ef35ae155b4c78cc9a8b7133a48c94f03/datasette/app.py#L203

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1432#issuecomment-898074849
issue_url: https://api.github.com/repos/simonw/datasette/issues/1432
node_id: IC_kwDOBm6k_c41h4jh
created_at: 2021-08-13T01:03:40Z
updated_at: 2021-08-13T01:03:40Z
author_association: OWNER
body:
  Also this method: https://github.com/simonw/datasette/blob/77f46297a88ac7e49dad2139410b01ee56d5f99c/datasette/app.py#L422-L424

  And the places that use it:

  https://github.com/simonw/datasette/blob/fc4846850fffd54561bc125332dfe97bb41ff42e/datasette/views/base.py#L617

  https://github.com/simonw/datasette/blob/fc4846850fffd54561bc125332dfe97bb41ff42e/datasette/views/database.py#L459

  Which is used in this template: https://github.com/simonw/datasette/blob/77f46297a88ac7e49dad2139410b01ee56d5f99c/datasette/templates/table.html#L204

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1432#issuecomment-898079507
issue_url: https://api.github.com/repos/simonw/datasette/issues/1432
node_id: IC_kwDOBm6k_c41h5sT
created_at: 2021-08-13T01:08:42Z
updated_at: 2021-08-13T01:09:41Z
author_association: OWNER
body:
  This is going to break some plugins: https://ripgrep.datasette.io/-/ripgrep?pattern=config%3D&literal=on&glob=%21datasette%2F**

  > ### datasette-cluster-map/tests/test_cluster_map.py
  > 
  > @pytest.mark.asyncio
  > 
  > async def test_respects_base_url():
  > ds = Datasette([], memory=True, config={"base_url": "/foo/"})
  > response = await ds.client.get("/:memory:?sql=select+1+as+latitude,+2+as+longitude")
  > assert (
  > 
  > ### datasette-export-notebook/tests/test_export_notebook.py
  > 
  > @pytest.mark.asyncio
  > 
  > async def test_notebook_no_csv(db_path):
  > datasette = Datasette([db_path], config={"allow_csv_stream": False})
  > response = await datasette.client.get("/db/big.Notebook")
  > assert ".csv" not in response.text
  > 
  > ### datasette-publish-vercel/tests/test_publish_vercel.py
  > metadata=metadata,
  > cors=True,
  > config={"default_page_size": 10, "sql_time_limit_ms": 2000}
  > ).app()
  > """
  > 
  > ### datasette-publish-vercel/datasette_publish_vercel/__init__.py
  > metadata=metadata{extras},
  > cors=True,
  > config={settings}
  > 
  > ).app()
  > 
  > """.strip()
  > 
  > ### datasette-search-all/tests/test_search_all.py
  > 
  > async def test_base_url(db_path, path):
  > sqlite_utils.Database(db_path)["creatures"].enable_fts(["name", "description"])
  > datasette = Datasette([db_path], config={"base_url": "/foo/"})
  > response = await datasette.client.get(path)
  > assert response.status_code == 200

  I should fix those as soon as this goes out in a release. I won't close this issue until then.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1432#issuecomment-898084675
issue_url: https://api.github.com/repos/simonw/datasette/issues/1432
node_id: IC_kwDOBm6k_c41h69D
created_at: 2021-08-13T01:11:30Z
updated_at: 2021-08-13T01:11:30Z
author_association: OWNER
body:
  It's only `datasette-publish-vercel` that will break the actual functionality - the others will have broken tests.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1433#issuecomment-898450402
issue_url: https://api.github.com/repos/simonw/datasette/issues/1433
node_id: IC_kwDOBm6k_c41jUPi
created_at: 2021-08-13T13:15:55Z
updated_at: 2021-08-13T13:15:55Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1433?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1433](https://codecov.io/gh/simonw/datasette/pull/1433?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (ddba6cc) into [main](https://codecov.io/gh/simonw/datasette/commit/2883098770fc66e50183b2b231edbde20848d4d6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (2883098) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1433/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1433?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1433   +/-   ##
  =======================================
    Coverage   91.82%   91.82%           
  =======================================
    Files          34       34           
    Lines        4418     4418           
  =======================================
    Hits         4057     4057           
    Misses        361      361           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1433?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1433?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [2883098...ddba6cc](https://codecov.io/gh/simonw/datasette/pull/1433?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-898517872
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: IC_kwDOBm6k_c41jktw
created_at: 2021-08-13T15:00:50Z
updated_at: 2021-08-13T15:00:50Z
author_association: OWNER
body:
  The primary key column (or `rowid`) often resolves to an `index` record in the `sqlite_master` table, e.g. the second row in this: 

  type | name | tbl_name | rootpage | sql
  -- | -- | -- | -- | --
  table | simple_primary_key | simple_primary_key | 2 | CREATE TABLE simple_primary_key (   id varchar(30) primary key,   content text )
  index | sqlite_autoindex_simple_primary_key_1 | simple_primary_key | 3 |  

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-898519924
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: IC_kwDOBm6k_c41jlN0
created_at: 2021-08-13T15:03:36Z
updated_at: 2021-08-13T15:03:36Z
author_association: OWNER
body:
  Weird edge-case: adding an `order by` changes the order of the columns with respect to the information I am deriving about them.

  Without order by this gets it right:

  <img width="702" alt="fixtures__select_neighborhood__facet_cities_name__state_from_facetable_join_facet_cities_on_facetable_city_id___facet_cities_id_where_neighborhood_like_________text________" src="https://user-images.githubusercontent.com/9599/129377247-ec1f67fd-5fc5-46a2-92ef-629276446621.png">

  With order by:

  <img width="708" alt="fixtures__select_neighborhood__facet_cities_name__state_from_facetable_join_facet_cities_on_facetable_city_id___facet_cities_id_where_neighborhood_like_________text________order_by_neighborhood" src="https://user-images.githubusercontent.com/9599/129377339-5b338432-6db8-43ac-9408-48a87c03e5e9.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-898524057
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: IC_kwDOBm6k_c41jmOZ
created_at: 2021-08-13T15:06:37Z
updated_at: 2021-08-13T15:06:37Z
author_association: OWNER
body:
  Comparing the `explain` for the two versions of that query - one with the order by and one without:

  <img width="1031" alt="fixtures__explain_select_neighborhood__facet_cities_name__state_from_facetable_join_facet_cities_on_facetable_city_id___facet_cities_id_where_neighborhood_like_________text________order_by_neighborhood_and_fixtures__explain_select_neighborh" src="https://user-images.githubusercontent.com/9599/129377790-52af28ab-5110-470f-bb1b-a400455e6717.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-898527525
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: IC_kwDOBm6k_c41jnEl
created_at: 2021-08-13T15:08:03Z
updated_at: 2021-08-13T15:08:03Z
author_association: OWNER
body:
  Am I going to need to look at the `ResultRow` and its columns but then wind back to that earlier `MakeRecord` and its columns?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-898536181
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: IC_kwDOBm6k_c41jpL1
created_at: 2021-08-13T15:17:20Z
updated_at: 2021-08-13T15:20:33Z
author_association: OWNER
body:
  Documentation for `MakeRecord`: https://www.sqlite.org/opcode.html#MakeRecord

  Running `explain` inside `sqlite3` provides extra comments and indentation which make it easier to understand:
  ```
  sqlite> explain select neighborhood, facet_cities.name, state
     ...> from facetable
     ...>     join facet_cities
     ...>         on facetable.city_id = facet_cities.id
     ...> where neighborhood like '%bob%';
  addr  opcode         p1    p2    p3    p4             p5  comment      
  ----  -------------  ----  ----  ----  -------------  --  -------------
  0     Init           0     15    0                    00  Start at 15  
  1     OpenRead       0     43    0     7              00  root=43 iDb=0; facetable
  2     OpenRead       1     42    0     2              00  root=42 iDb=0; facet_cities
  3     Rewind         0     14    0                    00               
  4       Column         0     6     3                    00  r[3]=facetable.neighborhood
  5       Function0      1     2     1     like(2)        02  r[1]=func(r[2..3])
  6       IfNot          1     13    1                    00               
  7       Column         0     5     4                    00  r[4]=facetable.city_id
  8       SeekRowid      1     13    4                    00  intkey=r[4]  
  9       Column         0     6     5                    00  r[5]=facetable.neighborhood
  10      Column         1     1     6                    00  r[6]=facet_cities.name
  11      Column         0     4     7                    00  r[7]=facetable.state
  12      ResultRow      5     3     0                    00  output=r[5..7]
  13    Next           0     4     0                    01               
  14    Halt           0     0     0                    00               
  15    Transaction    0     0     35    0              01  usesStmtJournal=0
  16    String8        0     2     0     %bob%          00  r[2]='%bob%' 
  17    Goto           0     1     0                    00               
  ```
  Compared with:
  ```
  sqlite> explain select neighborhood, facet_cities.name, state
     ...> from facetable
     ...>     join facet_cities
     ...>         on facetable.city_id = facet_cities.id
     ...> where neighborhood like '%bob%' order by neighborhood
     ...> ;
  addr  opcode         p1    p2    p3    p4             p5  comment      
  ----  -------------  ----  ----  ----  -------------  --  -------------
  0     Init           0     25    0                    00  Start at 25  
  1     SorterOpen     2     5     0     k(1,B)         00               
  2     OpenRead       0     43    0     7              00  root=43 iDb=0; facetable
  3     OpenRead       1     42    0     2              00  root=42 iDb=0; facet_cities
  4     Rewind         0     16    0                    00               
  5       Column         0     6     3                    00  r[3]=facetable.neighborhood
  6       Function0      1     2     1     like(2)        02  r[1]=func(r[2..3])
  7       IfNot          1     15    1                    00               
  8       Column         0     5     4                    00  r[4]=facetable.city_id
  9       SeekRowid      1     15    4                    00  intkey=r[4]  
  10      Column         1     1     6                    00  r[6]=facet_cities.name
  11      Column         0     4     7                    00  r[7]=facetable.state
  12      Column         0     6     5                    00  r[5]=facetable.neighborhood
  13      MakeRecord     5     3     9                    00  r[9]=mkrec(r[5..7])
  14      SorterInsert   2     9     5     3              00  key=r[9]     
  15    Next           0     5     0                    01               
  16    OpenPseudo     3     10    5                    00  5 columns in r[10]
  17    SorterSort     2     24    0                    00               
  18      SorterData     2     10    3                    00  r[10]=data   
  19      Column         3     2     8                    00  r[8]=state   
  20      Column         3     1     7                    00  r[7]=facet_cities.name
  21      Column         3     0     6                    00  r[6]=neighborhood
  22      ResultRow      6     3     0                    00  output=r[6..8]
  23    SorterNext     2     18    0                    00               
  24    Halt           0     0     0                    00               
  25    Transaction    0     0     35    0              01  usesStmtJournal=0
  26    String8        0     2     0     %bob%          00  r[2]='%bob%' 
  27    Goto           0     1     0                    00               
  ```
  So actually it looks like the `SorterSort` may be key to understanding this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-898540260
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: IC_kwDOBm6k_c41jqLk
created_at: 2021-08-13T15:23:28Z
updated_at: 2021-08-13T15:23:28Z
author_association: OWNER
body:
  SorterInsert:
  > Register P2 holds an SQL index key made using the MakeRecord instructions. This opcode writes that key into the sorter P1. Data for the entry is nil.

  SorterData:
  > Write into register P2 the current sorter data for sorter cursor P1. Then clear the column header cache on cursor P3.
  >
  > This opcode is normally use to move a record out of the sorter and into a register that is the source for a pseudo-table cursor created using OpenPseudo. That pseudo-table cursor is the one that is identified by parameter P3. Clearing the P3 column cache as part of this opcode saves us from having to issue a separate NullRow instruction to clear that cache.

  OpenPseudo:
  > Open a new cursor that points to a fake table that contains a single row of data. The content of that one row is the content of memory register P2. In other words, cursor P1 becomes an alias for the MEM_Blob content contained in register P2.
  >
  > A pseudo-table created by this opcode is used to hold a single row output from the sorter so that the row can be decomposed into individual columns using the Column opcode. The Column opcode is the only cursor opcode that works with a pseudo-table.
  >
  > P3 is the number of fields in the records that will be stored by the pseudo-table.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-898541543
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: IC_kwDOBm6k_c41jqfn
created_at: 2021-08-13T15:25:26Z
updated_at: 2021-08-13T15:25:26Z
author_association: OWNER
body:
  But the debug output here seems to be saying what we want it to say:
  ```
  17    SorterSort     2     24    0                    00               
  18      SorterData     2     10    3                    00  r[10]=data   
  19      Column         3     2     8                    00  r[8]=state   
  20      Column         3     1     7                    00  r[7]=facet_cities.name
  21      Column         3     0     6                    00  r[6]=neighborhood
  22      ResultRow      6     3     0                    00  output=r[6..8]
  ```
  We want to get back `neighborhood`, `facet_cities.name`, `state`.

  Why then are we seeing `[('facet_cities', 'name'), ('facetable', 'state'), (None, None)]`?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-898541972
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: IC_kwDOBm6k_c41jqmU
created_at: 2021-08-13T15:26:06Z
updated_at: 2021-08-13T15:29:06Z
author_association: OWNER
body:
  ResultRow:
  > The registers P1 through P1+P2-1 contain a single row of results. This opcode causes the sqlite3_step() call to terminate with an SQLITE_ROW return code and it sets up the sqlite3_stmt structure to provide access to the r(P1)..r(P1+P2-1) values as the result row.

  Column:
  > Interpret the data that cursor P1 points to as a structure built using the MakeRecord instruction. (See the MakeRecord opcode for additional information about the format of the data.) Extract the P2-th column from this record. If there are less that (P2+1) values in the record, extract a NULL.
  >
  > The value extracted is stored in register P3.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-898545815
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: IC_kwDOBm6k_c41jriX
created_at: 2021-08-13T15:31:53Z
updated_at: 2021-08-13T15:31:53Z
author_association: OWNER
body:
  My hunch here is that registers or columns are being reused in a way that makes my code break - my code is pretty dumb, there are places in it where maybe the first mention of a register wins instead of the last one?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-898554427
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: IC_kwDOBm6k_c41jto7
created_at: 2021-08-13T15:45:32Z
updated_at: 2021-08-13T15:45:32Z
author_association: OWNER
body:
  Some useful debug output:
  ```
  table_rootpage_by_register={0: 43, 1: 42}
  names_and_types_by_rootpage={42: ('facet_cities', 'table'), 43: ('facetable', 'table')}
  result_registers=[6, 7, 8]
  columns_by_column_register={3: ('facetable', 6), 4: ('facetable', 5), 6: ('facet_cities', 1), 7: ('facetable', 4), 5: ('facetable', 6)}
  all_column_names={('facet_cities', 0): 'id', ('facet_cities', 1): 'name', ('facetable', 0): 'pk', ('facetable', 1): 'created', ('facetable', 2): 'planet_int', ('facetable', 3): 'on_earth', ('facetable', 4): 'state', ('facetable', 5): 'city_id', ('facetable', 6): 'neighborhood', ('facetable', 7): 'tags', ('facetable', 8): 'complex_array', ('facetable', 9): 'distinct_some_null'}
  ```
  The `result_registers` should each correspond to the correct entry in `columns_by_column_register` but they do not.

  Python code:
  ```python
  def columns_for_query(conn, sql, params=None):
      """
      Given a SQLite connection ``conn`` and a SQL query ``sql``, returns a list of
      ``(table_name, column_name)`` pairs corresponding to the columns that would be
      returned by that SQL query.

      Each pair indicates the source table and column for the returned column, or
      ``(None, None)`` if no table and column could be derived (e.g. for "select 1")
      """
      if sql.lower().strip().startswith("explain"):
          return []
      opcodes = conn.execute("explain " + sql, params).fetchall()
      table_rootpage_by_register = {
          r["p1"]: r["p2"] for r in opcodes if r["opcode"] == "OpenRead"
      }
      print(f"{table_rootpage_by_register=}")
      names_and_types_by_rootpage = dict(
          [(r[0], (r[1], r[2])) for r in conn.execute(
              "select rootpage, name, type from sqlite_master where rootpage in ({})".format(
                  ", ".join(map(str, table_rootpage_by_register.values()))
              )
          )]
      )
      print(f"{names_and_types_by_rootpage=}")
      columns_by_column_register = {}
      for opcode in opcodes:
          if opcode["opcode"] in ("Rowid", "Column"):
              addr, opcode, table_id, cid, column_register, p4, p5, comment = opcode
              try:
                  table = names_and_types_by_rootpage[table_rootpage_by_register[table_id]][0]
                  columns_by_column_register[column_register] = (table, cid)
              except KeyError:
                  pass
      result_row = [dict(r) for r in opcodes if r["opcode"] == "ResultRow"][0]
      result_registers = list(range(result_row["p1"], result_row["p1"] + result_row["p2"]))
      print(f"{result_registers=}")
      print(f"{columns_by_column_register=}")
      all_column_names = {}
      for (table, _) in names_and_types_by_rootpage.values():
          table_xinfo = conn.execute("pragma table_xinfo({})".format(table)).fetchall()
          for column_info in table_xinfo:
              all_column_names[(table, column_info["cid"])] = column_info["name"]
      print(f"{all_column_names=}")
      final_output = []
      for register in result_registers:
          try:
              table, cid = columns_by_column_register[register]
              final_output.append((table, all_column_names[table, cid]))
          except KeyError:
              final_output.append((None, None))
      return final_output
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-898564705
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: IC_kwDOBm6k_c41jwJh
created_at: 2021-08-13T16:02:12Z
updated_at: 2021-08-13T16:04:06Z
author_association: OWNER
body:
  More debug output:
  ```
  table_rootpage_by_register={0: 43, 1: 42}
  names_and_types_by_rootpage={42: ('facet_cities', 'table'), 43: ('facetable', 'table')}
  table_id=0 cid=6 column_register=3
  table_id=0 cid=5 column_register=4
  table_id=1 cid=1 column_register=6
  table_id=0 cid=4 column_register=7
  table_id=0 cid=6 column_register=5
  table_id=3 cid=2 column_register=8
  table_id=3 cid=2 column_register=8
    KeyError
      3
      table = names_and_types_by_rootpage[table_rootpage_by_register[table_id]][0]
      names_and_types_by_rootpage={42: ('facet_cities', 'table'), 43: ('facetable', 'table')} table_rootpage_by_register={0: 43, 1: 42} table_id=3
      columns_by_column_register[column_register] = (table, cid)
      column_register=8 = (table='facetable', cid=2)
  table_id=3 cid=1 column_register=7
    KeyError
      3
      table = names_and_types_by_rootpage[table_rootpage_by_register[table_id]][0]
      names_and_types_by_rootpage={42: ('facet_cities', 'table'), 43: ('facetable', 'table')} table_rootpage_by_register={0: 43, 1: 42} table_id=3
      columns_by_column_register[column_register] = (table, cid)
      column_register=7 = (table='facetable', cid=1)
  table_id=3 cid=0 column_register=6
    KeyError
      3
      table = names_and_types_by_rootpage[table_rootpage_by_register[table_id]][0]
      names_and_types_by_rootpage={42: ('facet_cities', 'table'), 43: ('facetable', 'table')} table_rootpage_by_register={0: 43, 1: 42} table_id=3
      columns_by_column_register[column_register] = (table, cid)
      column_register=6 = (table='facetable', cid=0)
  result_registers=[6, 7, 8]
  columns_by_column_register={3: ('facetable', 6), 4: ('facetable', 5), 6: ('facet_cities', 1), 7: ('facetable', 4), 5: ('facetable', 6)}
  all_column_names={('facet_cities', 0): 'id', ('facet_cities', 1): 'name', ('facetable', 0): 'pk', ('facetable', 1): 'created', ('facetable', 2): 'planet_int', ('facetable', 3): 'on_earth', ('facetable', 4): 'state', ('facetable', 5): 'city_id', ('facetable', 6): 'neighborhood', ('facetable', 7): 'tags', ('facetable', 8): 'complex_array', ('facetable', 9): 'distinct_some_null'}
  ```
  Those `KeyError` are happening here because of a lookup in `table_rootpage_by_register` for `table_id=3` - but `table_rootpage_by_register` only has keys 0 and 1.

  It looks like that `3` actually corresponds to the `OpenPseudo` table from here:

  ```
  16    OpenPseudo     3     10    5                    00  5 columns in r[10]
  17    SorterSort     2     24    0                    00               
  18      SorterData     2     10    3                    00  r[10]=data   
  19      Column         3     2     8                    00  r[8]=state   
  20      Column         3     1     7                    00  r[7]=facet_cities.name
  21      Column         3     0     6                    00  r[6]=neighborhood
  22      ResultRow      6     3     0                    00  output=r[6..8]
  ```

  Python code:

  ```python
  def columns_for_query(conn, sql, params=None):
      """
      Given a SQLite connection ``conn`` and a SQL query ``sql``, returns a list of
      ``(table_name, column_name)`` pairs corresponding to the columns that would be
      returned by that SQL query.

      Each pair indicates the source table and column for the returned column, or
      ``(None, None)`` if no table and column could be derived (e.g. for "select 1")
      """
      if sql.lower().strip().startswith("explain"):
          return []
      opcodes = conn.execute("explain " + sql, params).fetchall()
      table_rootpage_by_register = {
          r["p1"]: r["p2"] for r in opcodes if r["opcode"] == "OpenRead"
      }
      print(f"{table_rootpage_by_register=}")
      names_and_types_by_rootpage = dict(
          [(r[0], (r[1], r[2])) for r in conn.execute(
              "select rootpage, name, type from sqlite_master where rootpage in ({})".format(
                  ", ".join(map(str, table_rootpage_by_register.values()))
              )
          )]
      )
      print(f"{names_and_types_by_rootpage=}")
      columns_by_column_register = {}
      for opcode_row in opcodes:
          if opcode_row["opcode"] in ("Rowid", "Column"):
              addr, opcode, table_id, cid, column_register, p4, p5, comment = opcode_row
              print(f"{table_id=} {cid=} {column_register=}")
              try:
                  table = names_and_types_by_rootpage[table_rootpage_by_register[table_id]][0]
                  columns_by_column_register[column_register] = (table, cid)
              except KeyError as e:
                  print("  KeyError")
                  print("   ", e)
                  print("    table = names_and_types_by_rootpage[table_rootpage_by_register[table_id]][0]")
                  print(f"    {names_and_types_by_rootpage=} {table_rootpage_by_register=} {table_id=}")
                  print("    columns_by_column_register[column_register] = (table, cid)")
                  print(f"    {column_register=} = ({table=}, {cid=})")
                  pass
      result_row = [dict(r) for r in opcodes if r["opcode"] == "ResultRow"][0]
      result_registers = list(range(result_row["p1"], result_row["p1"] + result_row["p2"]))
      print(f"{result_registers=}")
      print(f"{columns_by_column_register=}")
      all_column_names = {}
      for (table, _) in names_and_types_by_rootpage.values():
          table_xinfo = conn.execute("pragma table_xinfo({})".format(table)).fetchall()
          for column_info in table_xinfo:
              all_column_names[(table, column_info["cid"])] = column_info["name"]
      print(f"{all_column_names=}")
      final_output = []
      for register in result_registers:
          try:
              table, cid = columns_by_column_register[register]
              final_output.append((table, all_column_names[table, cid]))
          except KeyError:
              final_output.append((None, None))
      return final_output

  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-898567974
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: IC_kwDOBm6k_c41jw8m
created_at: 2021-08-13T16:07:00Z
updated_at: 2021-08-13T16:07:00Z
author_association: OWNER
body:
  So this line:
  ```
  19      Column         3     2     8                    00  r[8]=state
  ```
  Means "Take column 2 of table 3 (the pseudo-table) and store it in register 8"

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-898569319
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: IC_kwDOBm6k_c41jxRn
created_at: 2021-08-13T16:09:01Z
updated_at: 2021-08-13T16:10:48Z
author_association: OWNER
body:
  Need to figure out what column 2 of that pseudo-table is.

  I think the answer is here:

  ```
  4     Rewind         0     16    0                    00               
  5       Column         0     6     3                    00  r[3]=facetable.neighborhood
  6       Function0      1     2     1     like(2)        02  r[1]=func(r[2..3])
  7       IfNot          1     15    1                    00               
  8       Column         0     5     4                    00  r[4]=facetable.city_id
  9       SeekRowid      1     15    4                    00  intkey=r[4]  
  10      Column         1     1     6                    00  r[6]=facet_cities.name
  11      Column         0     4     7                    00  r[7]=facetable.state
  12      Column         0     6     5                    00  r[5]=facetable.neighborhood
  13      MakeRecord     5     3     9                    00  r[9]=mkrec(r[5..7])
  14      SorterInsert   2     9     5     3              00  key=r[9]     
  15    Next           0     5     0                    01               
  16    OpenPseudo     3     10    5                    00  5 columns in r[10]
  ```
  I think the `OpenPseduo` line puts five columns in `r[10]` - and those five columns are the five from the previous block - maybe the five leading up to the `MakeRecord` call on line 13.

  In which case column 2 would be `facet_cities.name` - assuming we start counting from 0.

  But the debug code said "r[8]=state".

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-898572065
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: IC_kwDOBm6k_c41jx8h
created_at: 2021-08-13T16:13:16Z
updated_at: 2021-08-13T16:13:16Z
author_association: OWNER
body:
  Aha! That `MakeRecord` line says `r[5..7]` - and r5 = neighborhood, r6 = facet_cities.name, r7 = facetable.state

  So if the `MakeRecord` defines what goes into that pseudo-table column 2 of that pseudo-table would be `state` - which is what we want.

  This is really convoluted. I'm no longer confident I can get this to work in a sensible way, especially since I've not started exploring what complex nested tables with CTEs and sub-selects do yet.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-898576097
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: IC_kwDOBm6k_c41jy7h
created_at: 2021-08-13T16:19:57Z
updated_at: 2021-08-13T16:19:57Z
author_association: OWNER
body:
  I think I need to look out for `OpenPseudo` and, when that occurs, take a look at the most recent `SorterInsert` and use that to find the `MakeRecord` and then use the `MakeRecord` to figure out the columns that went into it.

  After all of that I'll be able to resolve that "table 3" reference.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-898760020
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: IC_kwDOBm6k_c41kf1U
created_at: 2021-08-13T23:00:28Z
updated_at: 2021-08-13T23:01:27Z
author_association: OWNER
body:
  New theory: this is all about `SorterOpen` and `SorterInsert`. Consider the following with extra annotations at the end of the lines after the `--`:
  ```
  addr  opcode         p1    p2    p3    p4             p5  comment      
  ----  -------------  ----  ----  ----  -------------  --  -------------
  0     Init           0     25    0                    00  Start at 25  
  1     SorterOpen     2     5     0     k(1,B)         00               -- New SORTER in r2 with 5 slots
  2     OpenRead       0     43    0     7              00  root=43 iDb=0; facetable
  3     OpenRead       1     42    0     2              00  root=42 iDb=0; facet_cities
  4     Rewind         0     16    0                    00               
  5       Column         0     6     3                    00  r[3]=facetable.neighborhood
  6       Function0      1     2     1     like(2)        02  r[1]=func(r[2..3])
  7       IfNot          1     15    1                    00               
  8       Column         0     5     4                    00  r[4]=facetable.city_id
  9       SeekRowid      1     15    4                    00  intkey=r[4]  
  10      Column         1     1     6                    00  r[6]=facet_cities.name
  11      Column         0     4     7                    00  r[7]=facetable.state
  12      Column         0     6     5                    00  r[5]=facetable.neighborhood
  13      MakeRecord     5     3     9                    00  r[9]=mkrec(r[5..7])
  14      SorterInsert   2     9     5     3              00  key=r[9]-- WRITES record from r9 (line above) into sorter in r2
  15    Next           0     5     0                    01               
  16    OpenPseudo     3     10    5                    00  5 columns in r[10]
  17    SorterSort     2     24    0                    00 -- runs the sort, not relevant to my goal
  18      SorterData     2     10    3                    00  r[10]=data -- "Write into register P2 (r10) the current sorter data for sorter cursor P1 (sorter 2)"
  19      Column         3     2     8                    00  r[8]=state   
  20      Column         3     1     7                    00  r[7]=facet_cities.name
  21      Column         3     0     6                    00  r[6]=neighborhood
  22      ResultRow      6     3     0                    00  output=r[6..8]
  23    SorterNext     2     18    0                    00               
  24    Halt           0     0     0                    00               
  25    Transaction    0     0     35    0              01  usesStmtJournal=0
  26    String8        0     2     0     %bob%          00  r[2]='%bob%' 
  27    Goto           0     1     0                    00               
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-898760808
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: IC_kwDOBm6k_c41kgBo
created_at: 2021-08-13T23:03:01Z
updated_at: 2021-08-13T23:03:01Z
author_association: OWNER
body:
  Another idea: strip out any `order by` clause to try and keep this simpler. I doubt that's going to cope with complex nested queries though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-898788262
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: IC_kwDOBm6k_c41kmum
created_at: 2021-08-14T01:22:26Z
updated_at: 2021-08-14T01:51:08Z
author_association: OWNER
body:
  Tried a more complicated query:
  ```sql
  explain select pk, text1, text2, [name with . and spaces] from searchable where rowid in (select rowid from searchable_fts where searchable_fts match escape_fts(:search)) order by text1 desc limit 101
  ```
  Here's the explain:
  ```
  sqlite> explain select pk, text1, text2, [name with . and spaces] from searchable where rowid in (select rowid from searchable_fts where searchable_fts match escape_fts(:search)) order by text1 desc limit 101
     ...> ;
  addr  opcode         p1    p2    p3    p4             p5  comment      
  ----  -------------  ----  ----  ----  -------------  --  -------------
  0     Init           0     41    0                    00  Start at 41  
  1     OpenEphemeral  2     6     0     k(1,-B)        00  nColumn=6    
  2     Integer        101   1     0                    00  r[1]=101; LIMIT counter
  3     OpenRead       0     32    0     4              00  root=32 iDb=0; searchable
  4     Integer        16    3     0                    00  r[3]=16; return address
  5     Once           0     16    0                    00               
  6     OpenEphemeral  3     1     0     k(1,)          00  nColumn=1; Result of SELECT 1
  7     VOpen          1     0     0     vtab:7FCBCA72BE80  00               
  8     Function0      1     7     6     unknown(-1)    01  r[6]=func(r[7])
  9     Integer        5     4     0                    00  r[4]=5       
  10    Integer        1     5     0                    00  r[5]=1       
  11    VFilter        1     16    4                    00  iplan=r[4] zplan=''
  12      Rowid          1     8     0                    00  r[8]=rowid   
  13      MakeRecord     8     1     9     C              00  r[9]=mkrec(r[8])
  14      IdxInsert      3     9     8     1              00  key=r[9]     
  15    VNext          1     12    0                    00               
  16    Return         3     0     0                    00               
  17    Rewind         3     33    0                    00               
  18      Column         3     0     2                    00  r[2]=        
  19      IsNull         2     32    0                    00  if r[2]==NULL goto 32
  20      SeekRowid      0     32    2                    00  intkey=r[2]  
  21      Column         0     1     10                   00  r[10]=searchable.text1
  22      Sequence       2     11    0                    00  r[11]=cursor[2].ctr++
  23      IfNotZero      1     27    0                    00  if r[1]!=0 then r[1]--, goto 27
  24      Last           2     0     0                    00               
  25      IdxLE          2     32    10    1              00  key=r[10]    
  26      Delete         2     0     0                    00               
  27      Rowid          0     12    0                    00  r[12]=rowid  
  28      Column         0     2     13                   00  r[13]=searchable.text2
  29      Column         0     3     14                   00  r[14]=searchable.name with . and spaces
  30      MakeRecord     10    5     16                   00  r[16]=mkrec(r[10..14])
  31      IdxInsert      2     16    10    5              00  key=r[16]    
  32    Next           3     18    0                    00               
  33    Sort           2     40    0                    00               
  34      Column         2     4     15                   00  r[15]=[name with . and spaces]
  35      Column         2     3     14                   00  r[14]=text2  
  36      Column         2     0     13                   00  r[13]=text1  
  37      Column         2     2     12                   00  r[12]=pk     
  38      ResultRow      12    4     0                    00  output=r[12..15]
  39    Next           2     34    0                    00               
  40    Halt           0     0     0                    00               
  41    Transaction    0     0     35    0              01  usesStmtJournal=0
  42    Variable       1     7     0     :search        00  r[7]=parameter(1,:search)
  43    Goto           0     1     0                    00               
  ```
  Here the `ResultRow` is for registers `12..15` - but those all refer to `Column` records in `2` - where `2` is the first `OpenEphemeral` declared right at the start. I'm having enormous trouble figuring out how that ephemeral table gets populated by the other operations in a way that would let me derive which columns end up in the `ResultRow`.

  Frustratingly SQLite seems to be able to figure that out just fine, see the column of comments on the right hand side - but I only get those in the `sqlite3` CLI shell, they're not available to me with SQLite when called as a library from Python.

  Maybe the key to that is this section:
  ```
  27      Rowid          0     12    0                    00  r[12]=rowid  
  28      Column         0     2     13                   00  r[13]=searchable.text2
  29      Column         0     3     14                   00  r[14]=searchable.name with . and spaces
  30      MakeRecord     10    5     16                   00  r[16]=mkrec(r[10..14])
  31      IdxInsert      2     16    10    5              00  key=r[16]    
  ```
  MakeRecord:

  > Convert P2 registers beginning with P1 into the record format use as a data record in a database table or as a key in an index. The Column opcode can decode the record later.
  > 
  > P4 may be a string that is P2 characters long. The N-th character of the string indicates the column affinity that should be used for the N-th field of the index key.
  > 
  > The mapping from character to affinity is given by the SQLITE_AFF_ macros defined in sqliteInt.h.
  > 
  > If P4 is NULL then all index fields have the affinity BLOB.
  > 
  > The meaning of P5 depends on whether or not the SQLITE_ENABLE_NULL_TRIM compile-time option is enabled:
  > 
  > * If SQLITE_ENABLE_NULL_TRIM is enabled, then the P5 is the index of the right-most table that can be null-trimmed.
  > 
  > * If SQLITE_ENABLE_NULL_TRIM is omitted, then P5 has the value OPFLAG_NOCHNG_MAGIC if the MakeRecord opcode is allowed to accept no-change records with serial_type 10. This value is only used inside an assert() and does not affect the end result.

  IdxInsert:
  > Register P2 holds an SQL index key made using the MakeRecord instructions. This opcode writes that key into the index P1. Data for the entry is nil.
  > 
  > If P4 is not zero, then it is the number of values in the unpacked key of reg(P2). In that case, P3 is the index of the first register for the unpacked key. The availability of the unpacked key can sometimes be an optimization.
  > 
  > If P5 has the OPFLAG_APPEND bit set, that is a hint to the b-tree layer that this insert is likely to be an append.
  > 
  > If P5 has the OPFLAG_NCHANGE bit set, then the change counter is incremented by this instruction. If the OPFLAG_NCHANGE bit is clear, then the change counter is unchanged.
  > 
  > If the OPFLAG_USESEEKRESULT flag of P5 is set, the implementation might run faster by avoiding an unnecessary seek on cursor P1. However, the OPFLAG_USESEEKRESULT flag must only be set if there have been no prior seeks on the cursor or if the most recent seek used a key equivalent to P2.
  >
  > This instruction only works for indices. The equivalent instruction for tables is Insert.

  IdxLE:
  > The P4 register values beginning with P3 form an unpacked index key that omits the PRIMARY KEY or ROWID. Compare this key value against the index that P1 is currently pointing to, ignoring the PRIMARY KEY or ROWID on the P1 index.
  >
  > If the P1 index entry is less than or equal to the key value then jump to P2. Otherwise fall through to the next instruction.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-898913554
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: IC_kwDOBm6k_c41lFUS
created_at: 2021-08-14T16:13:40Z
updated_at: 2021-08-14T16:13:40Z
author_association: OWNER
body:
  I think I need to care about the following:

  - `ResultRow` and `Column` for the final result
  - `OpenRead` for opening tables
  - `OpenEphemeral` then `MakeRecord` and `IdxInsert` for writing records into ephemeral tables

  `Column` may reference either a table (from `OpenRead`) or an ephemeral table (from `OpenEphemeral`).

  That *might* be enough.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-898913629
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: IC_kwDOBm6k_c41lFVd
created_at: 2021-08-14T16:14:12Z
updated_at: 2021-08-14T16:14:12Z
author_association: OWNER
body:
  I would feel a lot more comfortable about all of this if I had a robust mechanism for running the Datasette test suite against multiple versions of SQLite itself.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-898933865
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: IC_kwDOBm6k_c41lKRp
created_at: 2021-08-14T17:27:16Z
updated_at: 2021-08-14T17:28:29Z
author_association: OWNER
body:
  Maybe I split this out into a separate Python library that gets tested against *every* SQLite release I can possibly try it against, and then bakes out the supported release versions into the library code itself?

  Datasette could depend on that library. The library could be released independently of Datasette any time a new SQLite version comes out.

  I could even run a separate git scraper repo that checks for new SQLite releases and submits PRs against the library when a new release comes out.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-898936068
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: IC_kwDOBm6k_c41lK0E
created_at: 2021-08-14T17:44:54Z
updated_at: 2021-08-14T17:44:54Z
author_association: OWNER
body:
  Another interesting query to consider: https://latest.datasette.io/fixtures?sql=explain+select+*+from++pragma_table_info%28+%27123_starts_with_digits%27%29

  That one shows `VColumn` instead of `Column`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-898961535
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: IC_kwDOBm6k_c41lRB_
created_at: 2021-08-14T21:37:24Z
updated_at: 2021-08-14T21:37:24Z
author_association: OWNER
body:
  Did some more research into building SQLite custom versions via `pysqlite3` - here's what I figured out for macOS (which should hopefully work for Linux too): https://til.simonwillison.net/sqlite/build-specific-sqlite-pysqlite-macos

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1423#issuecomment-899749881
issue_url: https://api.github.com/repos/simonw/datasette/issues/1423
node_id: IC_kwDOBm6k_c41oRf5
created_at: 2021-08-16T19:07:02Z
updated_at: 2021-08-16T19:07:02Z
author_association: OWNER
body:
  Demo: https://latest.datasette.io/fixtures/compound_three_primary_keys?_facet=content&_facet_size=max&_facet=pk1&_facet=pk2

  <img width="686" alt="fixtures__compound_three_primary_keys__1_001_rows" src="https://user-images.githubusercontent.com/9599/129616596-cc51b668-7cb8-482f-9e20-e0d8ca4b71be.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-899915829
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: IC_kwDOBm6k_c41o6A1
created_at: 2021-08-17T01:02:35Z
updated_at: 2021-08-17T01:02:35Z
author_association: OWNER
body:
  New approach: this time I'm building a simplified executor for the bytecode operations themselves.
  ```python
  def execute_operations(operations, max_iterations = 100, trace=None):
      trace = trace or (lambda *args: None)
      registers: Dict[int, Any] = {}
      cursors: Dict[int, Tuple[str, Dict]] = {}
      instruction_pointer = 0
      iterations = 0
      result_row = None
      while True:
          iterations += 1
          if iterations > max_iterations:
              break
          operation = operations[instruction_pointer]
          trace(instruction_pointer, dict(operation))
          opcode = operation["opcode"]
          if opcode == "Init":
              if operation["p2"] != 0:
                  instruction_pointer = operation["p2"]
                  continue
              else:
                  instruction_pointer += 1
                  continue
          elif opcode == "Goto":
              instruction_pointer = operation["p2"]
              continue
          elif opcode == "Halt":
              break
          elif opcode == "OpenRead":
              cursors[operation["p1"]] = ("database_table", {
                  "rootpage": operation["p2"],
                  "connection": operation["p3"],
              })
          elif opcode == "OpenEphemeral":
              cursors[operation["p1"]] = ("ephemeral", {
                  "num_columns": operation["p2"],
                  "index_keys": [],
              })
          elif opcode == "MakeRecord":
              registers[operation["p3"]] = ("MakeRecord", {
                  "registers": list(range(operation["p1"] + operation["p2"]))
              })
          elif opcode == "IdxInsert":
              record = registers[operation["p2"]]
              cursors[operation["p1"]][1]["index_keys"].append(record)
          elif opcode == "Rowid":
              registers[operation["p2"]] = ("rowid", {
                  "table": operation["p1"]
              })
          elif opcode == "Sequence":
              registers[operation["p2"]] = ("sequence", {
                  "next_from_cursor": operation["p1"]
              })
          elif opcode == "Column":
              registers[operation["p3"]] = ("column", {
                  "cursor": operation["p1"],
                  "column_offset": operation["p2"]
              })
          elif opcode == "ResultRow":
              p1 = operation["p1"]
              p2 = operation["p2"]
              trace("ResultRow: ", list(range(p1, p1 + p2)), registers)
              result_row = [registers.get(i) for i in range(p1, p1 + p2)]
          elif opcode == "Integer":
              registers[operation["p2"]] = ("Integer", operation["p1"])
          elif opcode == "String8":
              registers[operation["p2"]] = ("String", operation["p4"])
          instruction_pointer += 1
      return {"registers": registers, "cursors": cursors, "result_row": result_row}
  ```
  Results are promising!
  ```
  execute_operations(db.execute("explain select 'hello', 55, rowid, * from searchable").fetchall())

  {'registers': {1: ('String', 'hello'),
    2: ('Integer', 55),
    3: ('rowid', {'table': 0}),
    4: ('rowid', {'table': 0}),
    5: ('column', {'cursor': 0, 'column_offset': 1}),
    6: ('column', {'cursor': 0, 'column_offset': 2}),
    7: ('column', {'cursor': 0, 'column_offset': 3})},
   'cursors': {0: ('database_table', {'rootpage': 32, 'connection': 0})},
   'result_row': [('String', 'hello'),
    ('Integer', 55),
    ('rowid', {'table': 0}),
    ('rowid', {'table': 0}),
    ('column', {'cursor': 0, 'column_offset': 1}),
    ('column', {'cursor': 0, 'column_offset': 2}),
    ('column', {'cursor': 0, 'column_offset': 3})]}
  ```
  Here's what happens with a union across three tables:
  ```
  execute_operations(db.execute(f"""
  explain select data as content from binary_data
  union
  select pk as content from complex_foreign_keys
  union
  select name as content from facet_cities
  """}).fetchall())

  {'registers': {1: ('column', {'cursor': 4, 'column_offset': 0}),
    2: ('MakeRecord', {'registers': [0, 1, 2, 3]}),
    3: ('column', {'cursor': 0, 'column_offset': 1}),
    4: ('column', {'cursor': 3, 'column_offset': 0})},
   'cursors': {3: ('ephemeral',
     {'num_columns': 1,
      'index_keys': [('MakeRecord', {'registers': [0, 1]}),
       ('MakeRecord', {'registers': [0, 1]}),
       ('MakeRecord', {'registers': [0, 1, 2, 3]})]}),
    2: ('database_table', {'rootpage': 44, 'connection': 0}),
    4: ('database_table', {'rootpage': 24, 'connection': 0}),
    0: ('database_table', {'rootpage': 42, 'connection': 0})},
   'result_row': [('column', {'cursor': 3, 'column_offset': 0})]}
  ```
  Note how the result_row refers to cursor 3, which is an ephemeral table which had three different sets of `MakeRecord` index keys assigned to it - indicating that the output column is NOT from the same underlying table source.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1438#issuecomment-900500824
issue_url: https://api.github.com/repos/simonw/datasette/issues/1438
node_id: IC_kwDOBm6k_c41rI1Y
created_at: 2021-08-17T17:38:16Z
updated_at: 2021-08-17T17:38:16Z
author_association: OWNER
body:
  Relevant template code: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/datasette/templates/query.html#L71

  `renderers` comes from here: https://github.com/simonw/datasette/blob/2883098770fc66e50183b2b231edbde20848d4d6/datasette/views/base.py#L593-L608

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1438#issuecomment-900502364
issue_url: https://api.github.com/repos/simonw/datasette/issues/1438
node_id: IC_kwDOBm6k_c41rJNc
created_at: 2021-08-17T17:40:41Z
updated_at: 2021-08-17T17:40:41Z
author_association: OWNER
body:
  Bug is likely in `path_with_format` itself: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/datasette/utils/__init__.py#L710-L729

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1438#issuecomment-900513267
issue_url: https://api.github.com/repos/simonw/datasette/issues/1438
node_id: IC_kwDOBm6k_c41rL3z
created_at: 2021-08-17T17:57:05Z
updated_at: 2021-08-17T17:57:05Z
author_association: OWNER
body:
  I'm having trouble replicating this bug outside of Vercel. Against Cloud Run: view-source:https://latest.datasette.io/fixtures?sql=select+*+from+searchable+where+text1+like+%22%25cat%25%22

  The HTML here is:

  ```html
  <p class="export-links">This data as
    <a href="/fixtures.json?sql=select+*+from+searchable+where+text1+like+%22%25cat%25%22">json</a>, 
   ...
    <a href="/fixtures.csv?sql=select+*+from+searchable+where+text1+like+%22%25cat%25%22&amp;_size=max">CSV</a>
  </p>
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1438#issuecomment-900516826
issue_url: https://api.github.com/repos/simonw/datasette/issues/1438
node_id: IC_kwDOBm6k_c41rMva
created_at: 2021-08-17T18:02:27Z
updated_at: 2021-08-17T18:02:27Z
author_association: OWNER
body:
  The key difference I can spot between Vercel and Cloud Run is that `+` in a query string gets converted to `%20` by Vercel before it gets to my app, but does not for Cloud Run:
  ```
  # Vercel
  ~ % curl -s 'https://til.simonwillison.net/-/asgi-scope?sql=select+*+from+tunes+where+name+like+%22%25wise+maid%25%22%0D%0A' | rg 'query_string' -C 2
   'method': 'GET',
   'path': '/-/asgi-scope',
   'query_string': b'sql=select%20*%20from%20tunes%20where%20name%20like%20%22%25'
                   b'wise%20maid%25%22%0D%0A',
   'raw_path': b'/-/asgi-scope',

  # Cloud Run
  ~ % curl -s 'https://latest-with-plugins.datasette.io/-/asgi-scope?sql=select+*+from+tunes+where+name+like+%22%25wise+maid%25%22%0D%0A' | rg 'query_string' -C 2
   'method': 'GET',
   'path': '/-/asgi-scope',
   'query_string': b'sql=select+*+from+tunes+where+name+like+%22%25wise+maid%25%2'
                   b'2%0D%0A',
   'raw_path': b'/-/asgi-scope',
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1438#issuecomment-900518343
issue_url: https://api.github.com/repos/simonw/datasette/issues/1438
node_id: IC_kwDOBm6k_c41rNHH
created_at: 2021-08-17T18:04:42Z
updated_at: 2021-08-17T18:04:42Z
author_association: OWNER
body:
  Here's how `request.query_string` works: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/datasette/utils/asgi.py#L86-L88

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1438#issuecomment-900681413
issue_url: https://api.github.com/repos/simonw/datasette/issues/1438
node_id: IC_kwDOBm6k_c41r07F
created_at: 2021-08-17T22:47:44Z
updated_at: 2021-08-17T22:47:44Z
author_association: OWNER
body:
  I deployed another copy of `fixtures.db` on Vercel at https://til.simonwillison.net/fixtures so I can compare it with `fixtures.db` on Cloud Run at https://latest.datasette.io/fixtures

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1438#issuecomment-900690998
issue_url: https://api.github.com/repos/simonw/datasette/issues/1438
node_id: IC_kwDOBm6k_c41r3Q2
created_at: 2021-08-17T23:11:16Z
updated_at: 2021-08-17T23:12:25Z
author_association: OWNER
body:
  I have completely failed to replicate this initial bug - but it's still there on the `thesession.vercel.app` deployment (even though my own deployments to Vercel do not exhibit it). Here's a one-liner to replicate it against that deployment:

  `curl -s 'https://thesession.vercel.app/thesession?sql=select+*+from+tunes+where+name+like+%22%25wise+maid%25%22' | rg '.csv'`

  Whit outputs this:

  `<p class="export-links">This data as <a href="/thesession.json?sql=select * from tunes where name like &#34;%wise maid%&#34;">json</a>, <a href="/thesession.csv?sql=select * from tunes where name like &#34;%wise maid%&#34;&amp;_size=max">CSV</a></p>`

  It looks like, rather than being URL-encoded, the original query string is somehow making it through to Jinja and then being auto-escaped there.

  The weird thing is that the equivalent query executed against my `til.simonwillison.net` Vercel instance does this:

  `curl -s 'https://til.simonwillison.net/fixtures?sql=select+*+from+searchable+where+text1+like+%22%25a%25%22' | rg '.csv'`

  `<p class="export-links">This data as <a href="/fixtures.json?sql=select%20*%20from%20searchable%20where%20text1%20like%20%22%25a%25%22">json</a>, <a href="/fixtures.csv?sql=select%20*%20from%20searchable%20where%20text1%20like%20%22%25a%25%22&amp;_size=max">CSV</a></p>`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-900699670
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c41r5YW
created_at: 2021-08-17T23:34:23Z
updated_at: 2021-08-17T23:34:23Z
author_association: OWNER
body:
  The challenge comes down to telling the difference between the following:

  - `/db/table` - an HTML table page
  - `/db/table.csv` - the CSV version of `/db/table`
  - `/db/table.csv` - no this one is actually a database table called `table.csv`
  - `/db/table.csv.csv` - the CSV version of `/db/table.csv`
  - `/db/table.csv.csv.csv` and so on...

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-900705226
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c41r6vK
created_at: 2021-08-17T23:50:32Z
updated_at: 2021-08-17T23:50:47Z
author_association: OWNER
body:
  An alternative solution would be to use some form of escaping for the characters that form the name of the table.

  The obvious way to do this would be URL-encoding - but it doesn't hold for `.` characters. The hex for that is `%2E` but watch what happens with that in a URL:

  ```
  # Against Cloud Run:
  curl -s 'https://datasette.io/-/asgi-scope/foo/bar%2Fbaz%2E' | rg path
   'path': '/-/asgi-scope/foo/bar/baz.',
   'raw_path': b'/-/asgi-scope/foo/bar%2Fbaz.',
   'root_path': '',
  # Against Vercel:
  curl -s 'https://til.simonwillison.net/-/asgi-scope/foo/bar%2Fbaz%2E' | rg path
   'path': '/-/asgi-scope/foo/bar%2Fbaz%2E',
   'raw_path': b'/-/asgi-scope/foo/bar%2Fbaz%2E',
   'root_path': '',
  ```
  Surprisingly in this case Vercel DOES keep it intact, but Cloud Run does not.

  It's still no good though: I need a solution that works on Vercel, Cloud Run and every other potential hosting provider too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-900709703
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c41r71H
created_at: 2021-08-18T00:03:09Z
updated_at: 2021-08-18T00:03:09Z
author_association: OWNER
body:
  But... what if I invent my own escaping scheme?

  I actually did this once before, in https://github.com/simonw/datasette/commit/9fdb47ca952b93b7b60adddb965ea6642b1ff523 - while I was working on porting Datasette to ASGI in https://github.com/simonw/datasette/issues/272#issuecomment-494192779 because ASGI didn't yet have the `raw_path` mechanism.

  I could bring that back - it looked like this:

  ```
      "table/and/slashes" => "tableU+002FandU+002Fslashes"
      "~table" => "U+007Etable"
      "+bobcats!" => "U+002Bbobcats!"
      "U+007Etable" => "UU+002B007Etable"
  ```
  But I didn't particularly like it - it was quite verbose.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-900711967
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c41r8Yf
created_at: 2021-08-18T00:08:09Z
updated_at: 2021-08-18T00:08:09Z
author_association: OWNER
body:
  Here's an alternative I just made up which I'm calling "dot dash" encoding:

  ```python
  def dot_dash_encode(s):
      return s.replace("-", "--").replace(".", "-.")

  def dot_dash_decode(s):
      return s.replace("-.", ".").replace("--", "-")
  ```
  And some examples:
  ```python
  for example in (
    "hello",
    "hello.csv",
    "hello-and-so-on.csv",
    "hello-.csv",
    "hello--and--so--on-.csv",
    "hello.csv.",
    "hello.csv.-",
    "hello.csv.--",
  ):
      print(example)
      print(dot_dash_encode(example))
      print(example == dot_dash_decode(dot_dash_encode(example)))
      print()
  ```
  Outputs:
  ```
  hello
  hello
  True

  hello.csv
  hello-.csv
  True

  hello-and-so-on.csv
  hello--and--so--on-.csv
  True

  hello-.csv
  hello---.csv
  True

  hello--and--so--on-.csv
  hello----and----so----on---.csv
  True

  hello.csv.
  hello-.csv-.
  True

  hello.csv.-
  hello-.csv-.--
  True

  hello.csv.--
  hello-.csv-.----
  True
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-900712981
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c41r8oV
created_at: 2021-08-18T00:09:59Z
updated_at: 2021-08-18T00:12:32Z
author_association: OWNER
body:
  So given the original examples, a table called `table.csv` would have the following URLs:

  - `/db/table-.csv` - the HTML version
  - `/db/table-.csv.csv` - the CSV version
  - `/db/table-.csv.json` - the JSON version

  And if for some horific reason you had a table with the name `/db/table-.csv.csv` (so `/db/` was the first part of the actual table name in SQLite) the URLs would look like this:

  - `/db/%2Fdb%2Ftable---.csv-.csv` - the HTML version
  - `/db/%2Fdb%2Ftable---.csv-.csv.csv` - the CSV version
  - `/db/%2Fdb%2Ftable---.csv-.csv.json` - the JSON version

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-900714630
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c41r9CG
created_at: 2021-08-18T00:13:33Z
updated_at: 2021-08-18T00:13:33Z
author_association: OWNER
body:
  The documentation should definitely cover how table names become URLs, in case any third party code needs to be able to calculate this themselves.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1439#issuecomment-900715375
issue_url: https://api.github.com/repos/simonw/datasette/issues/1439
node_id: IC_kwDOBm6k_c41r9Nv
created_at: 2021-08-18T00:15:28Z
updated_at: 2021-08-18T00:15:28Z
author_association: OWNER
body:
  Maybe I should use `-/` to encode forward slashes too, to defend against any ASGI servers that might not implement `raw_path` correctly.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/317#issuecomment-901337305
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/317
node_id: IC_kwDOCGYnMM41uVDZ
created_at: 2021-08-18T18:30:59Z
updated_at: 2021-08-18T18:30:59Z
author_association: OWNER
body:
  I'm just going to remove this - I added it when the library was mostly undocumented, but it has comprehensive documentation now.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/296#issuecomment-901338356
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/296
node_id: IC_kwDOCGYnMM41uVT0
created_at: 2021-08-18T18:32:39Z
updated_at: 2021-08-18T18:32:39Z
author_association: OWNER
body:
  This is a good call. I have a fix for this in Datasette but it's not in `sqlite-utils` yet: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/datasette/utils/__init__.py#L824-L835

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/296#issuecomment-901338841
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/296
node_id: IC_kwDOCGYnMM41uVbZ
created_at: 2021-08-18T18:33:26Z
updated_at: 2021-08-18T18:45:12Z
author_association: OWNER
body:
  I think I'll do this as an optional `table.search(..., escape=True)` parameter.

  Actually I'll do `quote=True` for consistency with the new `db.quote_fts()` method.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/247#issuecomment-901344634
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/247
node_id: IC_kwDOCGYnMM41uW16
created_at: 2021-08-18T18:42:54Z
updated_at: 2021-08-18T18:42:54Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/247?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#247](https://codecov.io/gh/simonw/sqlite-utils/pull/247?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (af989af) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/1fe73c898b44695052f1a9ca832818d50cecf662?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (1fe73c8) will **decrease** coverage by `0.03%`.
  > The diff coverage is `85.71%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/247/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/247?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@            Coverage Diff             @@
  ##             main     #247      +/-   ##
  ==========================================
  - Coverage   96.28%   96.24%   -0.04%     
  ==========================================
    Files           5        5              
    Lines        2179     2186       +7     
  ==========================================
  + Hits         2098     2104       +6     
  - Misses         81       82       +1     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/247?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.84% <85.71%> (-0.08%)` | :arrow_down: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/247?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/247?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [1fe73c8...af989af](https://codecov.io/gh/simonw/sqlite-utils/pull/247?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/246#issuecomment-901345800
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/246
node_id: IC_kwDOCGYnMM41uXII
created_at: 2021-08-18T18:44:48Z
updated_at: 2021-08-18T18:44:48Z
author_association: OWNER
body:
  The `db.quote_fts(value)` method from #247 can now be used for this - documentation here: https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Database.quote_fts

  I'll be adding further improvements relating to this (a `table.search(q, quote=True)` parameter) in #296.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/246#issuecomment-901353345
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/246
node_id: IC_kwDOCGYnMM41uY-B
created_at: 2021-08-18T18:57:13Z
updated_at: 2021-08-18T18:57:13Z
author_association: OWNER
body:
  More documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#quoting-characters-for-use-in-search

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/296#issuecomment-901390635
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/296
node_id: IC_kwDOCGYnMM41uiEr
created_at: 2021-08-18T19:58:53Z
updated_at: 2021-08-18T19:58:53Z
author_association: OWNER
body:
  ```
  sqlite-utils search fixtures.db searchable 'dog"'
  Error: malformed MATCH expression: [dog"]
  ```
  This error message could suggest retrying with `--quote`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/296#issuecomment-901398216
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/296
node_id: IC_kwDOCGYnMM41uj7I
created_at: 2021-08-18T20:11:01Z
updated_at: 2021-08-18T20:11:01Z
author_association: OWNER
body:
  ```
  % sqlite-utils search fixtures.db searchable 'dog"'
  Error: malformed MATCH expression: [dog"]

  Try running this again with the --quote option
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/296#issuecomment-901399139
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/296
node_id: IC_kwDOCGYnMM41ukJj
created_at: 2021-08-18T20:12:34Z
updated_at: 2021-08-18T20:13:12Z
author_association: OWNER
body:
  Documentation for `table.search(..., quote=True)`: https://sqlite-utils.datasette.io/en/latest/python-api.html#searching-with-table-search

  In the API reference: https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Table.search

  And for the CLI `--quote` option: https://sqlite-utils.datasette.io/en/latest/cli.html#executing-searches

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/318#issuecomment-901440207
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/318
node_id: IC_kwDOCGYnMM41uuLP
created_at: 2021-08-18T21:24:28Z
updated_at: 2021-08-18T21:24:49Z
author_association: OWNER
body:
  Something like this then:

      sqlite-utils file.db "select * from t" --csv --gz > t.csv.gz

  Maybe add a `-o t.csv.gz` option too so you don't have to use a `>`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/318#issuecomment-901440752
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/318
node_id: IC_kwDOCGYnMM41uuTw
created_at: 2021-08-18T21:25:30Z
updated_at: 2021-08-18T21:25:30Z
author_association: OWNER
body:
  Some questions:

  - Should this support compression formats other than gzip?
  - Should `memory` learn to auto-detect gzipped data?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/37#issuecomment-901452199
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/37
node_id: IC_kwDOCGYnMM41uxGn
created_at: 2021-08-18T21:48:57Z
updated_at: 2021-08-18T21:48:57Z
author_association: OWNER
body:
  I did a bunch of work on this in #266. The library is now pretty thoroughly typed, and I even found a couple of bugs using `mypy` along the way: #313 and #315.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-901475812
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: IC_kwDOBm6k_c41u23k
created_at: 2021-08-18T22:41:19Z
updated_at: 2021-08-18T22:41:19Z
author_association: OWNER
body:
  > Maybe I split this out into a separate Python library that gets tested against _every_ SQLite release I can possibly try it against, and then bakes out the supported release versions into the library code itself?

  I'm going to do this, and call the Python library `sqlite-explain`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1442#issuecomment-902191150
issue_url: https://api.github.com/repos/simonw/datasette/issues/1442
node_id: IC_kwDOBm6k_c41xlgu
created_at: 2021-08-19T19:43:05Z
updated_at: 2021-08-19T19:43:59Z
author_association: OWNER
body:
  Maybe as simple as teaching https://github.com/simonw/datasette/blob/main/.github/workflows/deploy-latest.yml to run on pushes to ALL branches:

  https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/.github/workflows/deploy-latest.yml#L3-L6

  And then quit early if the branch is not in some allow-list.

  If it IS in the allow-list, use the name of the branch to dynamically construct the name of the Cloud Run service here: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/.github/workflows/deploy-latest.yml#L60

  Need to skip the documentation build and deployment stuff for other branches though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1442#issuecomment-902217726
issue_url: https://api.github.com/repos/simonw/datasette/issues/1442
node_id: IC_kwDOBm6k_c41xr_-
created_at: 2021-08-19T20:21:47Z
updated_at: 2021-08-19T20:21:47Z
author_association: OWNER
body:
  I think the neatest way to implement this would be for the `on -> push -> branches` list to be the list of branches that should be deployed in this way. The rest of the code can react to that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1442#issuecomment-902235714
issue_url: https://api.github.com/repos/simonw/datasette/issues/1442
node_id: IC_kwDOBm6k_c41xwZC
created_at: 2021-08-19T20:50:38Z
updated_at: 2021-08-19T20:50:38Z
author_association: OWNER
body:
  Would this allow anyone to push a PR to this repo that would result in their code being deployed against my Cloud Run account? I'm reasonably confident that it would not, since the secrets would not be visible to their PR branch.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1442#issuecomment-902239215
issue_url: https://api.github.com/repos/simonw/datasette/issues/1442
node_id: IC_kwDOBm6k_c41xxPv
created_at: 2021-08-19T20:56:46Z
updated_at: 2021-08-19T20:56:46Z
author_association: OWNER
body:
  I'm going to only run the tests if it's a push to `main` - that way I can ship demo branches really quickly, even if they don't yet have passing tests.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1442#issuecomment-902243498
issue_url: https://api.github.com/repos/simonw/datasette/issues/1442
node_id: IC_kwDOBm6k_c41xySq
created_at: 2021-08-19T21:04:01Z
updated_at: 2021-08-19T21:04:01Z
author_association: OWNER
body:
  That successfully deployed to https://datasette-latest-deploy-this-branch-j7hipcg4aq-uc.a.run.app/ even though the tests failed.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1415#issuecomment-902250361
issue_url: https://api.github.com/repos/simonw/datasette/issues/1415
node_id: IC_kwDOBm6k_c41xz95
created_at: 2021-08-19T21:12:28Z
updated_at: 2021-08-19T21:12:28Z
author_association: OWNER
body:
  I would love to know this too! I always find figuring out minimal permissions to be really difficult.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1415#issuecomment-902251316
issue_url: https://api.github.com/repos/simonw/datasette/issues/1415
node_id: IC_kwDOBm6k_c41x0M0
created_at: 2021-08-19T21:14:15Z
updated_at: 2021-08-19T21:14:15Z
author_association: OWNER
body:
  https://github.com/ahmetb/cloud-run-faq#how-do-i-continuously-deploy-to-cloud-run suggests the following:

  > - `roles/run.admin` to deploy applications
  > - `roles/iam.serviceAccountUser` on the service account that your app will use

  It also links to https://cloud.google.com/run/docs/reference/iam/roles

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1434#issuecomment-902254712
issue_url: https://api.github.com/repos/simonw/datasette/issues/1434
node_id: IC_kwDOBm6k_c41x1B4
created_at: 2021-08-19T21:18:31Z
updated_at: 2021-08-19T21:18:57Z
author_association: OWNER
body:
  I deployed a demo to https://datasette-latest-query-info-j7hipcg4aq-uc.a.run.app using the mechanism from #1442.

  e.g. demo here: https://datasette-latest-query-info-j7hipcg4aq-uc.a.run.app/fixtures?sql=select+*+from+searchable

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1426#issuecomment-902260338
issue_url: https://api.github.com/repos/simonw/datasette/issues/1426
node_id: IC_kwDOBm6k_c41x2Zy
created_at: 2021-08-19T21:28:25Z
updated_at: 2021-08-19T21:29:40Z
author_association: OWNER
body:
  Actually it looks like you can send a `sitemap.xml` to Google using an unauthenticated GET request to:

      https://www.google.com/ping?sitemap=FULL_URL_OF_SITEMAP

  According to https://developers.google.com/search/docs/advanced/sitemaps/build-sitemap

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1426#issuecomment-902260799
issue_url: https://api.github.com/repos/simonw/datasette/issues/1426
node_id: IC_kwDOBm6k_c41x2g_
created_at: 2021-08-19T21:29:13Z
updated_at: 2021-08-19T21:29:13Z
author_association: OWNER
body:
  Bing's equivalent is: https://www.bing.com/webmasters/help/Sitemaps-3b5cf6ed

      http://www.bing.com/ping?sitemap=FULL_URL_OF_SITEMAP

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1426#issuecomment-902263367
issue_url: https://api.github.com/repos/simonw/datasette/issues/1426
node_id: IC_kwDOBm6k_c41x3JH
created_at: 2021-08-19T21:33:51Z
updated_at: 2021-08-19T21:36:28Z
author_association: OWNER
body:
  I was worried about if it's possible to allow access to `/fixtures` but deny access to `/fixtures?sql=...`

  From various answers on Stack Overflow it looks like this should handle that:

  ```
  User-agent: *
  Disallow: /fixtures?
  ```
  I could use this for tables too - it may well be OK to access table index pages while still avoiding pagination, facets etc. I think this should block both query strings and row pages while allowing the table page itself:
  ```
  User-agent: *
  Disallow: /fixtures/searchable?
  Disallow: /fixtures/searchable/*
  ```
  Could even accompany that with a `sitemap.xml` that explicitly lists all of the tables - which would mean adding sitemaps to Datasette core too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902328369
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57
node_id: IC_kwDODEm0Qs41yHAx
created_at: 2021-08-19T23:56:26Z
updated_at: 2021-08-19T23:56:26Z
author_association: MEMBER
body:
  https://developer.twitter.com/en/docs/twitter-api/v1/tweets/timelines/api-reference/get-statuses-user_timeline says the API has been replaced by the new v2 one, but it should still work - and the `since_id` parameter is still documented on that page.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902328760
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57
node_id: IC_kwDODEm0Qs41yHG4
created_at: 2021-08-19T23:57:41Z
updated_at: 2021-08-19T23:57:41Z
author_association: MEMBER
body:
  Weird, added debug code and got this: `{'screen_name': 'simonw', 'count': 200, 'since_id': 'False', 'tweet_mode': 'extended'}` -  so maybe it's a `twitter-to-sqlite` bug where somehow the string `False` is being passed somewhere.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902329455
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57
node_id: IC_kwDODEm0Qs41yHRv
created_at: 2021-08-19T23:59:56Z
updated_at: 2021-08-19T23:59:56Z
author_association: MEMBER
body:
  This looks like the bug to me:

  https://github.com/dogsheep/twitter-to-sqlite/blob/197e69cec40052c423a5ed071feb5f7cccea41b9/twitter_to_sqlite/cli.py#L239-L241

  `type=str, default=False`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/healthkit-to-sqlite/issues/20#issuecomment-902355471
issue_url: https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/20
node_id: IC_kwDOC8tyDs41yNoP
created_at: 2021-08-20T01:09:07Z
updated_at: 2021-08-20T01:09:07Z
author_association: MEMBER
body:
  Workaround:

      sqlite-utils create-index healthkit.db workout_points -- -date

  See https://sqlite-utils.datasette.io/en/stable/cli.html#creating-indexes

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/320#issuecomment-903288430
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/320
node_id: IC_kwDOCGYnMM411xZu
created_at: 2021-08-22T15:44:55Z
updated_at: 2021-08-22T15:45:52Z
author_association: OWNER
body:
  ```
  curl 'https://api.github.com/users/dogsheep/repos' | sqlite-utils memory - --analyze
  ```
  ```
  stdin.id: (1/73)

    Total rows: 13
    Null rows: 0
    Blank rows: 0

    Distinct values: 13

  stdin.node_id: (2/73)

    Total rows: 13
    Null rows: 0
    Blank rows: 0

    Distinct values: 13

  stdin.name: (3/73)

    Total rows: 13
    Null rows: 0
    Blank rows: 0

    Distinct values: 13

  stdin.full_name: (4/73)

    Total rows: 13
    Null rows: 0
    Blank rows: 0

    Distinct values: 13

  stdin.private: (5/73)

    Total rows: 13
    Null rows: 0
    Blank rows: 0

    Distinct values: 1

    Most common:
      13: 0

  stdin.owner: (6/73)

    Total rows: 13
    Null rows: 0
    Blank rows: 0

    Distinct values: 1

    Most common:
      13: {"login": "dogsheep", "id": 53015001, "node_id": "MDEyOk9yZ2FuaXphdGlvbjUzMDE1MD...

  stdin.html_url: (7/73)

    Total rows: 13
    Null rows: 0
    Blank rows: 0

    Distinct values: 13

  stdin.description: (8/73)

    Total rows: 13
    Null rows: 0
    Blank rows: 0

    Distinct values: 13

  stdin.fork: (9/73)

    Total rows: 13
    Null rows: 0
    Blank rows: 0

    Distinct values: 1

    Most common:
      13: 0

  stdin.url: (10/73)

    Total rows: 13
    Null rows: 0
    Blank rows: 0

    Distinct values: 13

  stdin.forks_url: (11/73)

    Total rows: 13
    Null rows: 0
    Blank rows: 0

    Distinct values: 13

  stdin.keys_url: (12/73)

    Total rows: 13
  ...
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/healthkit-to-sqlite/issues/21#issuecomment-903950096
issue_url: https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/21
node_id: IC_kwDOC8tyDs414S8Q
created_at: 2021-08-23T17:00:59Z
updated_at: 2021-08-23T17:00:59Z
author_association: NONE
body:
  I think the issue is that I have records like these:

  ```xml
   <Record type="HKQuantityTypeIdentifierDietaryCholesterol" sourceName="MyFitnessPal" sourceVersion="35120" unit="mg" creationDate="2021-07-04 20:55:27 +0200" startDate="2021-07-04 20:55:00 +0200" endDate="2021-07-04 20:55:00 +0200" value="124">
    <MetadataEntry key="meal" value="Dinner"/>
    <MetadataEntry key="Meal" value="Dinner"/>
   </Record>
  ```

  And if sqlite is case insensitive, then `metadata_meal` and `metadata_Meal` result in the same column.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1445#issuecomment-904024939
issue_url: https://api.github.com/repos/simonw/datasette/issues/1445
node_id: IC_kwDOBm6k_c414lNr
created_at: 2021-08-23T18:52:35Z
updated_at: 2021-08-23T18:52:35Z
author_association: OWNER
body:
  The downside of the current implementation of this trick is that it only works for exact LIKE partial matches in a specific table - if you search for `dog cat` and `dog` appears in `title` but `cat` appears in `description` you won't get back that result.

  I think that's fine though. If you want more advanced search there are other mechanisms you can use. This is meant to be a very quick and dirty starting point for exploring a brand new table.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1445#issuecomment-904026253
issue_url: https://api.github.com/repos/simonw/datasette/issues/1445
node_id: IC_kwDOBm6k_c414liN
created_at: 2021-08-23T18:54:49Z
updated_at: 2021-08-23T18:54:49Z
author_association: OWNER
body:
  The bigger problem here is UI design. This feels like a pretty niche requirement to me, so adding a prominent search box to the table page (which already has the filters interface, plus the full-text search box for tables that have FTS configured) feels untidy.

  I could tuck it away in the table cog menu, but that's a weird place for something like this to live.

  Maybe add it as a new type of filter? Filters apply to specific columns though, so this would be the first filter that applied to _all_ columns - which doesn't really fit the existing filter interface very well.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1445#issuecomment-904027166
issue_url: https://api.github.com/repos/simonw/datasette/issues/1445
node_id: IC_kwDOBm6k_c414lwe
created_at: 2021-08-23T18:56:20Z
updated_at: 2021-08-23T18:56:20Z
author_association: OWNER
body:
  A related but potentially even more useful ability would be running a search across every column of every table in a whole database. For anything less than a few 100MB this could be incredibly useful.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1445#issuecomment-904036200
issue_url: https://api.github.com/repos/simonw/datasette/issues/1445
node_id: IC_kwDOBm6k_c414n9o
created_at: 2021-08-23T19:08:54Z
updated_at: 2021-08-23T19:08:54Z
author_association: OWNER
body:
  Figured out a query for searching across every column in every table! https://til.simonwillison.net/datasette/search-all-columns-trick#user-content-same-trick-for-the-entire-database

  ```sql
  with tables as (
    select
      name as table_name
    from
      sqlite_master
    where
      type = 'table'
  ),
  queries as (
    select
      'select ''' || tables.table_name || ''' as _table, rowid from "' || tables.table_name || '" where ' || group_concat(
        '"' || name || '" like ''%'' || :search || ''%''',
        ' or '
      ) as query
    from
      pragma_table_info(tables.table_name),
      tables
    group by
      tables.table_name
  )
  select
    group_concat(query, ' union all ')
  from
    queries
  ```
  The SQL query this generates for larger databases is _extremely_ long - but it does seem to work for smaller databases.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1445#issuecomment-904037087
issue_url: https://api.github.com/repos/simonw/datasette/issues/1445
node_id: IC_kwDOBm6k_c414oLf
created_at: 2021-08-23T19:10:17Z
updated_at: 2021-08-23T19:10:17Z
author_association: OWNER
body:
  Rather than trying to run that monstrosity in a single `union all` query, a better approach may be to use `fetch()` requests as seen in https://datasette.io/plugins/datasette-search-all

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1446#issuecomment-904866495
issue_url: https://api.github.com/repos/simonw/datasette/issues/1446
node_id: IC_kwDOBm6k_c417yq_
created_at: 2021-08-24T18:13:49Z
updated_at: 2021-08-24T18:13:49Z
author_association: OWNER
body:
  OK, now the following optional CSS gives us a sticky footer:
  ```css
  html, body {
    height: 100%;
  }
  body {
    display: flex;
    flex-direction: column;
  }
  .not-footer {
    flex: 1 0 auto;
  }
  footer {
    flex-shrink: 0;
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1446#issuecomment-904954530
issue_url: https://api.github.com/repos/simonw/datasette/issues/1446
node_id: IC_kwDOBm6k_c418IKi
created_at: 2021-08-24T20:32:47Z
updated_at: 2021-08-24T20:32:47Z
author_association: OWNER
body:
  Pasting that CSS into the styles editor in the developer tools on https://latest.datasette.io/ has the desired effect: footer at the bottom of the window unless the page is too long, in which case the footer is at the bottom of the scroll.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/859#issuecomment-904982056
issue_url: https://api.github.com/repos/simonw/datasette/issues/859
node_id: IC_kwDOBm6k_c418O4o
created_at: 2021-08-24T21:15:04Z
updated_at: 2021-08-24T21:15:30Z
author_association: CONTRIBUTOR
body:
  I'm running into issues with this as well. All other pages seem to work with lots of DBs except the home page, which absolutely tanks. Would be willing to put some work into this, if there's been any kind of progress on concepts on how this ought to work.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905001586
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/319
node_id: IC_kwDOCGYnMM418Tpy
created_at: 2021-08-24T21:52:50Z
updated_at: 2021-08-24T21:52:50Z
author_association: OWNER
body:
  Will need to re-title this section of the documentation: https://sqlite-utils.datasette.io/en/3.16/cli.html#inserting-binary-data-from-files - "Inserting binary data from files" will become "Inserting data from files"

  I'm OK with keeping the default as `BLOB` but I could add a `--text` option which stores the content as text instead.

  If the text can't be stored as `utf-8` I'll probably raise an error.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905003381
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/319
node_id: IC_kwDOCGYnMM418UF1
created_at: 2021-08-24T21:56:49Z
updated_at: 2021-08-24T21:56:49Z
author_association: NONE
body:
  I was thinking that an approach could be making FILE_COLUMNS a generator (_get_file_columns(mode)) or you can just have a different set of columns (is there something else that makes sense to be changed on the text scenario?).

  About UTF-8 I was referring to the encoding to use when reading files. This can be difficult to auto-detect but I believe that UTF-8 is pretty much the standard for text files.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905013162
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/319
node_id: IC_kwDOCGYnMM418Weq
created_at: 2021-08-24T22:15:31Z
updated_at: 2021-08-24T22:15:31Z
author_association: OWNER
body:
  I'm going to assume utf-8 but allow `--encoding` to be used to specify something different, since that option is already supported by other commands.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905013183
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/319
node_id: IC_kwDOCGYnMM418We_
created_at: 2021-08-24T22:15:34Z
updated_at: 2021-08-24T22:15:34Z
author_association: OWNER
body:
  Here's the error message I have working for invalid unicode:
  ```
  sqlite-utils insert-files /tmp/text.db files *.txt --text
    [------------------------------------]    0%
  Error: Could not read file '/Users/simon/Dropbox/Development/sqlite-utils/data.txt' as text

  'utf-8' codec can't decode byte 0xe3 in position 83: invalid continuation byte
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905021010
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/319
node_id: IC_kwDOCGYnMM418YZS
created_at: 2021-08-24T22:33:42Z
updated_at: 2021-08-24T22:33:42Z
author_association: NONE
body:
  Oh, I misread. Yes some files will not be valid UTF-8, I'd throw a warning and continue (not adding that file) but if you want to get more elaborate you could allow to define a policy on what to do. Not adding the file, index binary content or use a conversion policy like the ones available on Python's decode.

  From https://stackoverflow.com/questions/24616678/unicodedecodeerror-in-python-when-reading-a-file-how-to-ignore-the-error-and-ju :
   - 'ignore' ignores errors. Note that ignoring encoding errors can lead to data loss.
   - 'replace' causes a replacement marker (such as '?') to be inserted where there is malformed data.
   - 'surrogateescape' will represent any incorrect bytes as code points in the Unicode Private Use Area ranging from U+DC80 to U+DCFF. These private code points will then be turned back into the same bytes when the surrogateescape error handler is used when writing data. This is useful for processing files in an unknown encoding.
   - 'xmlcharrefreplace' is only supported when writing to a file. Characters not supported by the encoding are replaced with the appropriate XML character reference &#nnn;.
   - 'backslashreplace'  (also only supported when writing) replaces unsupported characters with Python’s backslashed escape sequences.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905021047
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/319
node_id: IC_kwDOCGYnMM418YZ3
created_at: 2021-08-24T22:33:48Z
updated_at: 2021-08-24T22:33:48Z
author_association: OWNER
body:
  I had a few doubts about the design just now. Since `content_text` is supported as a special argument, an alternative way of handling the above would be:

      sqlite-utils insert-files /tmp/text.db files *.txt -c path -c content_text -c size

  This does exactly the same thing as just using `--text` and not specifying any columns, because the actual implementation of `--text` is as follows:

  https://github.com/simonw/sqlite-utils/blob/0c796cd945b146b7395ff5f553861400be503867/sqlite_utils/cli.py#L1851-L1855

  But actually I think that's OK - ``--text`` is a useful shorthand that avoids you having to remember how to manually specify those columns with `-c`. So I'm going to leave the design as is.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905021933
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/319
node_id: IC_kwDOCGYnMM418Ynt
created_at: 2021-08-24T22:36:04Z
updated_at: 2021-08-24T22:36:04Z
author_association: OWNER
body:
  > Oh, I misread. Yes some files will not be valid UTF-8, I'd throw a warning and continue (not adding that file) but if you want to get more elaborate you could allow to define a policy on what to do. Not adding the file, index binary content or use a conversion policy like the ones available on Python's decode.

  I thought about supporting those different policies (with something like `--errors ignore`) but I feel like that's getting a little bit too deep into the weeds. Right now if you try to import an invalid file the behaviour is the same as for the `sqlite-utils insert` command (I added the same detailed error message):

  ```
  Error: Could not read file '/Users/simon/Dropbox/Development/sqlite-utils/data.txt' as text

  'utf-8' codec can't decode byte 0xe3 in position 83: invalid continuation byte

  The input you provided uses a character encoding other than utf-8.

  You can fix this by passing the --encoding= option with the encoding of the file.

  If you do not know the encoding, running 'file filename.csv' may tell you.

  It's often worth trying: --encoding=latin-1
  ```
  If someone has data that can't be translated to valid text using a known encoding, I'm happy leaving them to have to insert it into a `BLOB` column instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/321#issuecomment-905022931
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/321
node_id: IC_kwDOCGYnMM418Y3T
created_at: 2021-08-24T22:38:38Z
updated_at: 2021-08-24T23:27:26Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/321?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#321](https://codecov.io/gh/simonw/sqlite-utils/pull/321?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (db2dd6d) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/9258f4bd8450c951900de998a7bf81ca9b45a014?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (9258f4b) will **increase** coverage by `0.16%`.
  > The diff coverage is `100.00%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/321/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/321?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@            Coverage Diff             @@
  ##             main     #321      +/-   ##
  ==========================================
  + Coverage   96.41%   96.58%   +0.16%     
  ==========================================
    Files           5        5              
    Lines        2206     2223      +17     
  ==========================================
  + Hits         2127     2147      +20     
  + Misses         79       76       -3     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/321?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [sqlite\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/321/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.69% <100.00%> (+0.42%)` | :arrow_up: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/321?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/321?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [9258f4b...db2dd6d](https://codecov.io/gh/simonw/sqlite-utils/pull/321?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905024066
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/319
node_id: IC_kwDOCGYnMM418ZJC
created_at: 2021-08-24T22:41:39Z
updated_at: 2021-08-24T22:41:39Z
author_association: NONE
body:
  I'm happy with this functionality left the way you describe. In my case the data is homogeneous but other cases would work just by being consistent on the encoding. Thanks a lot, Simon!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/321#issuecomment-905039576
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/321
node_id: IC_kwDOCGYnMM418c7Y
created_at: 2021-08-24T23:21:29Z
updated_at: 2021-08-24T23:21:29Z
author_association: OWNER
body:
  Hah, the error here is actually:
  ```
  >           assert result.exit_code == 1, result.output
  E           AssertionError: 
  E             
  E           assert 0 == 1
  E            +  where 0 = <Result okay>.exit_code
  ```
  So I was expecting an error, but instead the command worked.

  I suspect this is because on Windows the default character set may not be UTF-8?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/321#issuecomment-905040307
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/321
node_id: IC_kwDOCGYnMM418dGz
created_at: 2021-08-24T23:23:36Z
updated_at: 2021-08-24T23:23:36Z
author_association: OWNER
body:
  https://discuss.python.org/t/pep-597-use-utf-8-for-default-text-file-encoding/1819 says:

  > Currently, `TextIOWrapper` uses `locale.getpreferredencoding(False)` (hereinafter called “locale encoding”) when encoding is not specified.
  > ...
  > Package authors using macOS or Linux may forget that the default encoding
  is not always UTF-8.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905043974
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/319
node_id: IC_kwDOCGYnMM418eAG
created_at: 2021-08-24T23:33:44Z
updated_at: 2021-08-24T23:33:44Z
author_association: OWNER
body:
  Updated documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#inserting-data-from-files

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/evernote-to-sqlite/issues/13#issuecomment-905203570
issue_url: https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13
node_id: IC_kwDOEhK-wc419E9y
created_at: 2021-08-25T05:51:22Z
updated_at: 2021-08-25T05:53:27Z
author_association: MEMBER
body:
  The debugger showed me that it broke on a string that looked like this:
  ```xml
  <?xml version="1.0" encoding="UTF-8" standalone="no"?>
  <!DOCTYPE en-note SYSTEM "http://xml.evernote.com/pub/enml2.dtd">
  <en-note>
    <h1 title="Q3 2018 Reflection & Development">
      <span title=Q3 2018 Reflection & Development">
        Q3 2018 Reflection & Development
      </span>
    </h1>
    ...
  ```
  Yeah that is not valid XML!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/evernote-to-sqlite/issues/13#issuecomment-905206234
issue_url: https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13
node_id: IC_kwDOEhK-wc419Fna
created_at: 2021-08-25T05:58:42Z
updated_at: 2021-08-25T05:58:42Z
author_association: MEMBER
body:
  https://github.com/dogsheep/evernote-to-sqlite/blob/36a466f142e5bad52719851c2fbda0c05cd35b99/evernote_to_sqlite/utils.py#L34-L42

  Not sure why I was round-tripping the `content_xml` like that - I will try not doing that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/323#issuecomment-905886797
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/323
node_id: IC_kwDOCGYnMM41_rxN
created_at: 2021-08-25T21:25:18Z
updated_at: 2021-08-25T21:25:18Z
author_association: OWNER
body:
  As far as I can tell the Python `sqlite3` module doesn't actually have a mechanism for de-registering a custom SQL function.

  This means that if I implement a mechanism whereby each call to `.convert()` registers a new SQL function with a random suffix (`convert_value_23424()` for example) those functions will stay registered - and if `.convert()` is called a large number of times the number of obsolete custom function registrations will grow without bounds.

  For that reason, I'm going to `wontfix` this issue.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/859#issuecomment-905899177
issue_url: https://api.github.com/repos/simonw/datasette/issues/859
node_id: IC_kwDOBm6k_c41_uyp
created_at: 2021-08-25T21:48:00Z
updated_at: 2021-08-25T21:48:00Z
author_association: CONTRIBUTOR
body:
  Upon first stab, there's two issues here:
  - DB/table/row counts (as discussed above). This isn't too bad if the DBs are actually above the MAX limit check.
  - Populating the internal DB. On first load of a giant set of DBs, it can take 10-20 mins to populate. By altering datasette and persisting the internal DB to disk, this problem is vastly improved, but I'm sure this will cause problems elsewhere.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/859#issuecomment-905900807
issue_url: https://api.github.com/repos/simonw/datasette/issues/859
node_id: IC_kwDOBm6k_c41_vMH
created_at: 2021-08-25T21:51:10Z
updated_at: 2021-08-25T21:51:10Z
author_association: OWNER
body:
  10-20 minutes to populate `_internal`! How many databases and tables is that for?

  I may have to rethink the `_internal` mechanism entirely. One possible alternative would be for the Datasette homepage to just show a list of available databases (maybe only if there are more than X connected) and then load in their metadata only the first time they are accessed.

  I need to get my own stress testing rig setup for this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/859#issuecomment-905904540
issue_url: https://api.github.com/repos/simonw/datasette/issues/859
node_id: IC_kwDOBm6k_c41_wGc
created_at: 2021-08-25T21:59:14Z
updated_at: 2021-08-25T21:59:55Z
author_association: CONTRIBUTOR
body:
  I did two tests: one with 1000 5-30mb DBs and a second with 20 multi gig DBs. For the second, I created them like so:
  `for i in {1..20}; do sqlite-generate db$i.db --tables ${i}00 --rows 100,2000 --columns 5,100 --pks 0 --fks 0; done`

  This was for deciding whether to use lots of small DBs or to group things into a smaller number of bigger DBs. The second strategy wins.

  By simply persisting the `_internal` DB to disk, I was able to avoid most of the performance issues I was experiencing previously. (To do this, I changed the `datasette/internal_db.py:init_internal_db` creates to if not exists, and changed the `_internal` DB instantiation in `datasette/app.py:Datasette.__init__` to a path with `is_mutable=True`.) Super rough, but the pages now load so I can continue testing ideas.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1448#issuecomment-906414811
issue_url: https://api.github.com/repos/simonw/datasette/issues/1448
node_id: IC_kwDOBm6k_c42Bsrb
created_at: 2021-08-26T13:29:02Z
updated_at: 2021-08-26T13:29:02Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1448?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1448](https://codecov.io/gh/simonw/datasette/pull/1448?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (a211747) into [main](https://codecov.io/gh/simonw/datasette/commit/a1a33bb5822214be1cebd98cd858b2058d91a4aa?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (a1a33bb) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1448/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1448?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1448   +/-   ##
  =======================================
    Coverage   91.82%   91.82%           
  =======================================
    Files          34       34           
    Lines        4418     4418           
  =======================================
    Hits         4057     4057           
    Misses        361      361           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1448?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1448?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [a1a33bb...a211747](https://codecov.io/gh/simonw/datasette/pull/1448?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/evernote-to-sqlite/issues/13#issuecomment-906635938
issue_url: https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13
node_id: IC_kwDOEhK-wc42Ciqi
created_at: 2021-08-26T18:18:27Z
updated_at: 2021-08-26T18:18:27Z
author_association: MEMBER
body:
  It looks like I was using the round-trip to dump the `<?xml version="1.0" encoding="UTF-8" standalone="no"?>` and `<!DOCTYPE` prefixes.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/evernote-to-sqlite/issues/13#issuecomment-906646452
issue_url: https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13
node_id: IC_kwDOEhK-wc42ClO0
created_at: 2021-08-26T18:34:34Z
updated_at: 2021-08-26T18:35:20Z
author_association: MEMBER
body:
  I tried this ampersand fix: https://regex101.com/r/ojU2H9/1
  ```python

  # https://regex101.com/r/ojU2H9/1
  _invalid_ampersand_re = re.compile(r'&(?![a-z0-9]+;)')

  def fix_bad_xml(xml):
      # More fixes for things like '&' not as part of an entity
      return _invalid_ampersand_re.sub('&amp;', xml)
  ```

  Even with that I'm still getting total garbage in the `<en-note>` content - it's just HTML, not even trying to be XML.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1449#issuecomment-907537366
issue_url: https://api.github.com/repos/simonw/datasette/issues/1449
node_id: IC_kwDOBm6k_c42F-vW
created_at: 2021-08-28T00:29:16Z
updated_at: 2021-08-28T00:29:29Z
author_association: OWNER
body:
  The closest plugin hook to this right now is [publish_subcommand](https://docs.datasette.io/en/stable/plugin_hooks.html#publish-subcommand-publish) - which looks like this:
  ```python
  @hookimpl
  def publish_subcommand(publish):
      @publish.command()
      @add_common_publish_arguments_and_options
      @click.option(
          "-k",
          "--api_key",
          help="API key for talking to my hosting provider",
      )
      def my_hosting_provider(...):
  ```
  But there are also several plugin hooks with `register_` prefixes, which may be a good naming convention to stick to here: `register_output_renderer(datasette)`, `register_routes(datasette)`, `register_facet_classes()`, `register_magic_parameters(datasette)`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1449#issuecomment-907537610
issue_url: https://api.github.com/repos/simonw/datasette/issues/1449
node_id: IC_kwDOBm6k_c42F-zK
created_at: 2021-08-28T00:30:51Z
updated_at: 2021-08-28T00:30:51Z
author_association: OWNER
body:
  There's also the option for plugins to muck around with existing registered commands - this could get a bit untidy if multiple plugins try to do it, but being able to replace `serve` with a fresh implementation that adds an additional command-line option before calling back to the original might open up some interesting possibilities.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1449#issuecomment-907537693
issue_url: https://api.github.com/repos/simonw/datasette/issues/1449
node_id: IC_kwDOBm6k_c42F-0d
created_at: 2021-08-28T00:31:26Z
updated_at: 2021-08-28T00:31:26Z
author_association: OWNER
body:
  Terminology question: is it correct to call these subcommands or should they be commands? `publish_subcommand()` adds subcommands of the format `datasette publish X` - but are we instead adding commands with this new one?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1449#issuecomment-907538940
issue_url: https://api.github.com/repos/simonw/datasette/issues/1449
node_id: IC_kwDOBm6k_c42F_H8
created_at: 2021-08-28T00:39:28Z
updated_at: 2021-08-28T00:39:28Z
author_association: OWNER
body:
  Here's the answer to that:
  ```
  ~ % datasette --help
  Usage: datasette [OPTIONS] COMMAND [ARGS]...

    Datasette!

  Options:
    --version  Show the version and exit.
    --help     Show this message and exit.

  Commands:
    serve*     Serve up specified SQLite database files with a web UI
    inspect
    install    Install Python packages - e.g.
    package    Package specified SQLite files into a new datasette Docker...
    plugins    List currently available plugins
    publish    Publish specified SQLite database files to the internet...
    uninstall  Uninstall Python packages (e.g.
  ```
  Since it's adding extra things that show up in `--help` under the "Commands:" heading, I should call them commands.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1449#issuecomment-907539065
issue_url: https://api.github.com/repos/simonw/datasette/issues/1449
node_id: IC_kwDOBm6k_c42F_J5
created_at: 2021-08-28T00:40:24Z
updated_at: 2021-08-28T00:40:24Z
author_association: OWNER
body:
  I'm going to call the new hook `register_commands` - since it will allow ambitious plugins to register more than one command if they want to. That's also pleasingly similar to `register_routes`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1449#issuecomment-907539251
issue_url: https://api.github.com/repos/simonw/datasette/issues/1449
node_id: IC_kwDOBm6k_c42F_Mz
created_at: 2021-08-28T00:41:37Z
updated_at: 2021-08-28T00:41:50Z
author_association: OWNER
body:
  The first example plugin I'm going to build for this will be `datasette verify file.db file2.db` - it will take one or more paths to SQLite files and verify if they can be opened by Datasette or not.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1449#issuecomment-907539668
issue_url: https://api.github.com/repos/simonw/datasette/issues/1449
node_id: IC_kwDOBm6k_c42F_TU
created_at: 2021-08-28T00:44:16Z
updated_at: 2021-08-28T00:44:16Z
author_association: OWNER
body:
  Considering this piece of code: https://github.com/simonw/datasette/blob/a1a33bb5822214be1cebd98cd858b2058d91a4aa/datasette/cli.py#L122-L142

  I think the hook itself gets called with a single argument, `cli`, which it can then use in the standard Click way to register extra stuff.

  I can't pass it `datasette` (like I do with `register_routes()`) because the Datasette object itself is instantiated by the `serve` command, which will not have been called.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1450#issuecomment-907540240
issue_url: https://api.github.com/repos/simonw/datasette/issues/1450
node_id: IC_kwDOBm6k_c42F_cQ
created_at: 2021-08-28T00:48:30Z
updated_at: 2021-08-28T00:48:30Z
author_association: OWNER
body:
  I'll go with this:
  ```
  % datasette --help
  Usage: datasette [OPTIONS] COMMAND [ARGS]...

    Datasette is an open source multi-tool for exploring and publishing data

    About Datasette: https://datasette.io/
    Full documentation: https://docs.datasette.io/

  Options:
    --version  Show the version and exit.
    --help     Show this message and exit.
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1449#issuecomment-907540790
issue_url: https://api.github.com/repos/simonw/datasette/issues/1449
node_id: IC_kwDOBm6k_c42F_k2
created_at: 2021-08-28T00:52:32Z
updated_at: 2021-08-28T00:52:32Z
author_association: OWNER
body:
  I don't think I can get this new hook to support the handy [--plugins-dir= mechanism](https://docs.datasette.io/en/stable/plugins.html#one-off-plugins-using-plugins-dir) for loading plugins from Python files as opposed to registering them with setuptools - that mechanism is itself implemented inside of code called by `datasette serve` so I don't have a way of taking advantage of it from outside that command.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1449#issuecomment-907540928
issue_url: https://api.github.com/repos/simonw/datasette/issues/1449
node_id: IC_kwDOBm6k_c42F_nA
created_at: 2021-08-28T00:53:37Z
updated_at: 2021-08-28T00:53:37Z
author_association: OWNER
body:
  I'll probably have to use this mechanism for the tests then: https://til.simonwillison.net/pytest/registering-plugins-in-tests

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1449#issuecomment-907542214
issue_url: https://api.github.com/repos/simonw/datasette/issues/1449
node_id: IC_kwDOBm6k_c42F_7G
created_at: 2021-08-28T01:02:38Z
updated_at: 2021-08-28T01:02:38Z
author_association: OWNER
body:
  Partial prototype of `datasette-verify`:
  ```python
  from datasette import hookimpl
  import click


  @hookimpl
  def register_commands(cli):
      from datasette.cli import sqlite_extensions
      @cli.command()
      @click.argument("files", type=click.Path(exists=True), nargs=-1)
      @sqlite_extensions
      def verify(files, sqlite_extensions):
          "Verify that files can be opened by Datasette"
          for file in files:
              print(file)
  ```
  I had to move the `from datasette.cli import sqlite_extensions` inside the hook function to avoid a circular import.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1449#issuecomment-907543982
issue_url: https://api.github.com/repos/simonw/datasette/issues/1449
node_id: IC_kwDOBm6k_c42GAWu
created_at: 2021-08-28T01:14:27Z
updated_at: 2021-08-28T01:14:27Z
author_association: OWNER
body:
  Writing the test for this is proving difficult, because the `cli` module has already been imported when I attempt to register a new plugin - so it doesn't pick up on the additional command registrations.

  Trying to work around that with `importlib.reload(cli)`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1453#issuecomment-908337614
issue_url: https://api.github.com/repos/simonw/datasette/issues/1453
node_id: IC_kwDOBm6k_c42JCHO
created_at: 2021-08-30T13:20:26Z
updated_at: 2021-08-30T13:20:26Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1453?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1453](https://codecov.io/gh/simonw/datasette/pull/1453?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (4f492a7) into [main](https://codecov.io/gh/simonw/datasette/commit/67cbf0ae7243431bf13702e6e3ba466b619c4d6f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (67cbf0a) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1453/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1453?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1453   +/-   ##
  =======================================
    Coverage   91.83%   91.83%           
  =======================================
    Files          34       34           
    Lines        4421     4421           
  =======================================
    Hits         4060     4060           
    Misses        361      361           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1453?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1453?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [67cbf0a...4f492a7](https://codecov.io/gh/simonw/datasette/pull/1453?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1446#issuecomment-908832938
issue_url: https://api.github.com/repos/simonw/datasette/issues/1446
node_id: IC_kwDOBm6k_c42K7Cq
created_at: 2021-08-31T01:54:59Z
updated_at: 2021-08-31T01:54:59Z
author_association: OWNER
body:
  I used the sticky footer mechanism in `datasette.app`: https://github.com/simonw/datasette.app/issues/3

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/evernote-to-sqlite/issues/14#issuecomment-911772943
issue_url: https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/14
node_id: IC_kwDOEhK-wc42WI0P
created_at: 2021-09-02T14:53:11Z
updated_at: 2021-09-02T14:53:11Z
author_association: NONE
body:
  Additionally, assuming the line numbers match up with the provided enenx file, the mentioned line plus one before and after is  as follows:
  ```
  <![CDATA[>]]>
  </span></div>
  <div style="padding: 0px; font-family: Arial, sans-serif; font-size: 12px; line-height: 16px; white-space: pre-wrap;"><br style=" padding: 0px;"/>
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1459#issuecomment-913218494
issue_url: https://api.github.com/repos/simonw/datasette/issues/1459
node_id: IC_kwDOBm6k_c42bpu-
created_at: 2021-09-05T19:58:51Z
updated_at: 2021-09-05T19:59:15Z
author_association: OWNER
body:
  This idea makes sense to me.

  There's actually an existing option that takes a path, called `--get` - it returns the HTML or JSON for that oath directly to the console, eg `datasette my.db --get /mydb/mytable.json`

  So... one option would be to allow combining that with `-o` to open that URL in the browser:

      datasette my.db -o --get /mydb

  So some options here are:

  - `datasette my.db --open-url /mydb`
  - `datasette my.db --open-path /mydb`
  - `datasette my.db --open --get /mydb`

  I quite like that last combination option, mainly to avoid adding even more command options.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1461#issuecomment-914439356
issue_url: https://api.github.com/repos/simonw/datasette/issues/1461
node_id: IC_kwDOBm6k_c42gTy8
created_at: 2021-09-07T16:11:37Z
updated_at: 2021-09-07T16:11:37Z
author_association: OWNER
body:
  ```
  (datasette) datasette % blacken-docs docs/*.rst
  docs/authentication.rst: Rewriting...
  docs/internals.rst:169: code block parse error Cannot parse: 14:0: <line number missing in source>
  docs/plugin_hooks.rst:251: code block parse error Cannot parse: 6:4:     ]
  docs/plugin_hooks.rst:312: code block parse error Cannot parse: 38:0: <line number missing in source>
  docs/spatialite.rst: Rewriting...
  docs/testing_plugins.rst:135: code block parse error Cannot parse: 5:0: <line number missing in source>
  docs/writing_plugins.rst: Rewriting...
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1461#issuecomment-914440282
issue_url: https://api.github.com/repos/simonw/datasette/issues/1461
node_id: IC_kwDOBm6k_c42gUBa
created_at: 2021-09-07T16:12:57Z
updated_at: 2021-09-07T16:12:57Z
author_association: OWNER
body:
  Here's the diff it produced from that first run:
  ```diff
  diff --git a/docs/authentication.rst b/docs/authentication.rst
  index 0d98cf8..8008023 100644
  --- a/docs/authentication.rst
  +++ b/docs/authentication.rst
  @@ -381,11 +381,7 @@ Authentication plugins can set signed ``ds_actor`` cookies themselves like so:
   .. code-block:: python
   
       response = Response.redirect("/")
  -    response.set_cookie("ds_actor", datasette.sign({
  -        "a": {
  -            "id": "cleopaws"
  -        }
  -    }, "actor"))
  +    response.set_cookie("ds_actor", datasette.sign({"a": {"id": "cleopaws"}}, "actor"))
   
   Note that you need to pass ``"actor"`` as the namespace to :ref:`datasette_sign`.
   
  @@ -412,12 +408,16 @@ To include an expiry, add a ``"e"`` key to the cookie value containing a `base62
       expires_at = int(time.time()) + (24 * 60 * 60)
   
       response = Response.redirect("/")
  -    response.set_cookie("ds_actor", datasette.sign({
  -        "a": {
  -            "id": "cleopaws"
  -        },
  -        "e": baseconv.base62.encode(expires_at),
  -    }, "actor"))
  +    response.set_cookie(
  +        "ds_actor",
  +        datasette.sign(
  +            {
  +                "a": {"id": "cleopaws"},
  +                "e": baseconv.base62.encode(expires_at),
  +            },
  +            "actor",
  +        ),
  +    )
   
   The resulting cookie will encode data that looks something like this:
   
  diff --git a/docs/spatialite.rst b/docs/spatialite.rst
  index d1b300b..556bad8 100644
  --- a/docs/spatialite.rst
  +++ b/docs/spatialite.rst
  @@ -58,19 +58,22 @@ Here's a recipe for taking a table with existing latitude and longitude columns,
   .. code-block:: python
   
       import sqlite3
  -    conn = sqlite3.connect('museums.db')
  +
  +    conn = sqlite3.connect("museums.db")
       # Lead the spatialite extension:
       conn.enable_load_extension(True)
  -    conn.load_extension('/usr/local/lib/mod_spatialite.dylib')
  +    conn.load_extension("/usr/local/lib/mod_spatialite.dylib")
       # Initialize spatial metadata for this database:
  -    conn.execute('select InitSpatialMetadata(1)')
  +    conn.execute("select InitSpatialMetadata(1)")
       # Add a geometry column called point_geom to our museums table:
       conn.execute("SELECT AddGeometryColumn('museums', 'point_geom', 4326, 'POINT', 2);")
       # Now update that geometry column with the lat/lon points
  -    conn.execute('''
  +    conn.execute(
  +        """
           UPDATE museums SET
           point_geom = GeomFromText('POINT('||"longitude"||' '||"latitude"||')',4326);
  -    ''')
  +    """
  +    )
       # Now add a spatial index to that column
       conn.execute('select CreateSpatialIndex("museums", "point_geom");')
       # If you don't commit your changes will not be persisted:
  @@ -186,13 +189,14 @@ Here's Python code to create a SQLite database, enable SpatiaLite, create a plac
   .. code-block:: python
   
       import sqlite3
  -    conn = sqlite3.connect('places.db')
  +
  +    conn = sqlite3.connect("places.db")
       # Enable SpatialLite extension
       conn.enable_load_extension(True)
  -    conn.load_extension('/usr/local/lib/mod_spatialite.dylib')
  +    conn.load_extension("/usr/local/lib/mod_spatialite.dylib")
       # Create the masic countries table
  -    conn.execute('select InitSpatialMetadata(1)')
  -    conn.execute('create table places (id integer primary key, name text);')
  +    conn.execute("select InitSpatialMetadata(1)")
  +    conn.execute("create table places (id integer primary key, name text);")
       # Add a MULTIPOLYGON Geometry column
       conn.execute("SELECT AddGeometryColumn('places', 'geom', 4326, 'MULTIPOLYGON', 2);")
       # Add a spatial index against the new column
  @@ -201,13 +205,17 @@ Here's Python code to create a SQLite database, enable SpatiaLite, create a plac
       from shapely.geometry.multipolygon import MultiPolygon
       from shapely.geometry import shape
       import requests
  -    geojson = requests.get('https://data.whosonfirst.org/404/227/475/404227475.geojson').json()
  +
  +    geojson = requests.get(
  +        "https://data.whosonfirst.org/404/227/475/404227475.geojson"
  +    ).json()
       # Convert to "Well Known Text" format
  -    wkt = shape(geojson['geometry']).wkt
  +    wkt = shape(geojson["geometry"]).wkt
       # Insert and commit the record
  -    conn.execute("INSERT INTO places (id, name, geom) VALUES(null, ?, GeomFromText(?, 4326))", (
  -       "Wales", wkt
  -    ))
  +    conn.execute(
  +        "INSERT INTO places (id, name, geom) VALUES(null, ?, GeomFromText(?, 4326))",
  +        ("Wales", wkt),
  +    )
       conn.commit()
   
   Querying polygons using within()
  diff --git a/docs/writing_plugins.rst b/docs/writing_plugins.rst
  index bd60a4b..5af01f6 100644
  --- a/docs/writing_plugins.rst
  +++ b/docs/writing_plugins.rst
  @@ -18,9 +18,10 @@ The quickest way to start writing a plugin is to create a ``my_plugin.py`` file
   
       from datasette import hookimpl
   
  +
       @hookimpl
       def prepare_connection(conn):
  -        conn.create_function('hello_world', 0, lambda: 'Hello world!')
  +        conn.create_function("hello_world", 0, lambda: "Hello world!")
   
   If you save this in ``plugins/my_plugin.py`` you can then start Datasette like this::
   
  @@ -60,22 +61,18 @@ The example consists of two files: a ``setup.py`` file that defines the plugin:
   
       from setuptools import setup
   
  -    VERSION = '0.1'
  +    VERSION = "0.1"
   
       setup(
  -        name='datasette-plugin-demos',
  -        description='Examples of plugins for Datasette',
  -        author='Simon Willison',
  -        url='https://github.com/simonw/datasette-plugin-demos',
  -        license='Apache License, Version 2.0',
  +        name="datasette-plugin-demos",
  +        description="Examples of plugins for Datasette",
  +        author="Simon Willison",
  +        url="https://github.com/simonw/datasette-plugin-demos",
  +        license="Apache License, Version 2.0",
           version=VERSION,
  -        py_modules=['datasette_plugin_demos'],
  -        entry_points={
  -            'datasette': [
  -                'plugin_demos = datasette_plugin_demos'
  -            ]
  -        },
  -        install_requires=['datasette']
  +        py_modules=["datasette_plugin_demos"],
  +        entry_points={"datasette": ["plugin_demos = datasette_plugin_demos"]},
  +        install_requires=["datasette"],
       )
   
   And a Python module file, ``datasette_plugin_demos.py``, that implements the plugin:
  @@ -88,12 +85,12 @@ And a Python module file, ``datasette_plugin_demos.py``, that implements the plu
   
       @hookimpl
       def prepare_jinja2_environment(env):
  -        env.filters['uppercase'] = lambda u: u.upper()
  +        env.filters["uppercase"] = lambda u: u.upper()
   
   
       @hookimpl
       def prepare_connection(conn):
  -        conn.create_function('random_integer', 2, random.randint)
  +        conn.create_function("random_integer", 2, random.randint)
   
   
   Having built a plugin in this way you can turn it into an installable package using the following command::
  @@ -123,11 +120,13 @@ To bundle the static assets for a plugin in the package that you publish to PyPI
   
   .. code-block:: python
   
  -        package_data={
  -            'datasette_plugin_name': [
  -                'static/plugin.js',
  -            ],
  -        },
  +        package_data = (
  +            {
  +                "datasette_plugin_name": [
  +                    "static/plugin.js",
  +                ],
  +            },
  +        )
   
   Where ``datasette_plugin_name`` is the name of the plugin package (note that it uses underscores, not hyphens) and ``static/plugin.js`` is the path within that package to the static file.
   
  @@ -152,11 +151,13 @@ Templates should be bundled for distribution using the same ``package_data`` mec
   
   .. code-block:: python
   
  -        package_data={
  -            'datasette_plugin_name': [
  -                'templates/my_template.html',
  -            ],
  -        },
  +        package_data = (
  +            {
  +                "datasette_plugin_name": [
  +                    "templates/my_template.html",
  +                ],
  +            },
  +        )
   
   You can also use wildcards here such as ``templates/*.html``. See `datasette-edit-schema <https://github.com/simonw/datasette-edit-schema>`__ for an example of this pattern.
   ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1461#issuecomment-914441037
issue_url: https://api.github.com/repos/simonw/datasette/issues/1461
node_id: IC_kwDOBm6k_c42gUNN
created_at: 2021-09-07T16:13:59Z
updated_at: 2021-09-07T16:13:59Z
author_association: OWNER
body:
  I don't think I'll adopt it for this project. For example, here:
  ```diff
       response = Response.redirect("/")
  -    response.set_cookie("ds_actor", datasette.sign({
  -        "a": {
  -            "id": "cleopaws"
  -        }
  -    }, "actor"))
  +    response.set_cookie("ds_actor", datasette.sign({"a": {"id": "cleopaws"}}, "actor"))
  ```
  I chose to use the multi-line version to help emphasize the structure - the single-line replacement loses that. I think I'll continue to make my own editorial choices about how the code examples are laid out.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1462#issuecomment-914644260
issue_url: https://api.github.com/repos/simonw/datasette/issues/1462
node_id: IC_kwDOBm6k_c42hF0k
created_at: 2021-09-07T21:34:32Z
updated_at: 2021-09-07T21:34:32Z
author_association: OWNER
body:
  I think this is a setting. There are two relevant settings at the moment:
  ```
      "template_debug": false,
      "trace_debug": false,
  ```
  For consistence then this should be called `something_debug` - but do I want a single setting that exposes the `_internal` database and adds those debug options to the menu, or do I want those as two separate settings?

  - `internal_debug` to enable access to that `_internal` database
  - `menu_debug` for those menu options?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1463#issuecomment-915229323
issue_url: https://api.github.com/repos/simonw/datasette/issues/1463
node_id: IC_kwDOBm6k_c42jUqL
created_at: 2021-09-08T13:15:26Z
updated_at: 2021-10-13T21:52:23Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1463?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1463](https://codecov.io/gh/simonw/datasette/pull/1463?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (2dd94c5) into [main](https://codecov.io/gh/simonw/datasette/commit/0d5cc20aeffa3537cfc9296d01ec24b9c6e23dcf?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0d5cc20) will **decrease** coverage by `0.00%`.
  > The diff coverage is `n/a`.

  > :exclamation: Current head 2dd94c5 differs from pull request most recent head 481212c. Consider uploading reports for the commit 481212c to get more accurate results
  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1463/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1463?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@            Coverage Diff             @@
  ##             main    #1463      +/-   ##
  ==========================================
  - Coverage   91.83%   91.83%   -0.01%     
  ==========================================
    Files          34       34              
    Lines        4422     4421       -1     
  ==========================================
  - Hits         4061     4060       -1     
    Misses        361      361              
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1463?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1463/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `96.00% <0.00%> (-0.01%)` | :arrow_down: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1463?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1463?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [e1012e7...481212c](https://codecov.io/gh/simonw/datasette/pull/1463?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1464#issuecomment-915299013
issue_url: https://api.github.com/repos/simonw/datasette/issues/1464
node_id: IC_kwDOBm6k_c42jlrF
created_at: 2021-09-08T14:40:28Z
updated_at: 2021-09-08T14:40:28Z
author_association: CONTRIBUTOR
body:
  What are the full errors you're getting?

  This *may* be the same issue as described in https://github.com/simonw/datasette/pull/1223 - essentially the test suite (and corresponding Datasette features I assume) are by default implicitly dependent on your Sqlite installation having been compiled with the `SQLITE_ENABLE_FTS3_PARENTHESIS` option. If this is the same issue then I think this can be fixed either by recompiling with that option or (probably more easily) by running `pip install pysqlite3-binary`, which will be used in preference to your system Sqlite installation and has this option enabled.

reactions:
[cell repeats an earlier one in this file: 112 characters]


