# issue_comments

html_url: https://github.com/simonw/datasette/issues/1619#issuecomment-1483009959
issue_url: https://api.github.com/repos/simonw/datasette/issues/1619
node_id: IC_kwDOBm6k_c5YZO-n
created_at: 2023-03-24T15:38:04Z
updated_at: 2023-03-24T15:38:04Z
author_association: NONE
body:
  I also have the same problem when running behind an apache proxy server with base_url. However, I have researched the problem a bit and have come to the conclusion that if you change _facet_date to _facet in the following https://github.com/simonw/datasette/blob/3feed1f66e2b746f349ee56970a62246a18bb164/datasette/facets.py#LL493C57-L493C68 , the facets work. But I'm not sure if this has other consequences?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1484269326
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/388
node_id: IC_kwDOCGYnMM5YeCcO
created_at: 2023-03-26T23:24:12Z
updated_at: 2023-03-26T23:28:14Z
author_association: OWNER
body:
  This broke on the upgrade to Furo.

  I fixed it in Datasette like this: https://github.com/simonw/datasette/commit/5c1cfa451d78e3935193f5e10eba59bf741241a1 and https://github.com/simonw/datasette/commit/db8cf899e286fbaa0a40f3a9ae8d5aaa1478822e

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1608#issuecomment-1484270067
issue_url: https://api.github.com/repos/simonw/datasette/issues/1608
node_id: IC_kwDOBm6k_c5YeCnz
created_at: 2023-03-26T23:26:10Z
updated_at: 2023-03-26T23:27:42Z
author_association: OWNER
body:
  That didn't work, because `jQuery` isn't yet defined that high up the template.

  I should use `{% block scripts %}` instead: https://github.com/pradyunsg/furo/blob/05350110cf02d033ebb57147eb54e9371a63545a/src/furo/theme/furo/base.html#L89-L102

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1608#issuecomment-1484272313
issue_url: https://api.github.com/repos/simonw/datasette/issues/1608
node_id: IC_kwDOBm6k_c5YeDK5
created_at: 2023-03-26T23:36:17Z
updated_at: 2023-03-26T23:36:17Z
author_association: OWNER
body:
  Still getting this error:

  <img width="537" alt="image" src="https://user-images.githubusercontent.com/9599/227811947-a35a6758-cf94-4196-89ce-85ae2f48ab06.png">

  Weird because `jQuery` is present in the devtools console.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1608#issuecomment-1484274796
issue_url: https://api.github.com/repos/simonw/datasette/issues/1608
node_id: IC_kwDOBm6k_c5YeDxs
created_at: 2023-03-26T23:38:18Z
updated_at: 2023-03-26T23:38:18Z
author_association: OWNER
body:
  ChatGPT prompt:

  > ```
  > jQuery(function ($) {
  >   // Show banner linking to /stable/ if this is a /latest/ page
  >   if (!/\/latest\//.test(location.pathname)) {
  >     return;
  >   }
  >   var stableUrl = location.pathname.replace("/latest/", "/stable/");
  >   // Check it's not a 404
  >   fetch(stableUrl, { method: "HEAD" }).then((response) => {
  >     if (response.status == 200) {
  >       var warning = $(
  >         `<div class="admonition warning">
  >            <p class="first admonition-title">Note</p>
  >            <p class="last">
  >              This documentation covers the <strong>development version</strong> of Datasette.</p>
  >              <p>See <a href="${stableUrl}">this page</a> for the current stable release.
  >            </p>
  >         </div>`
  >       );
  >       warning.find("a").attr("href", stableUrl);
  >       $("article[role=main]").prepend(warning);
  >     }
  >   });
  > });
  > 
  > Rewrite this to use vanilla JavaScript, no jQuery
  > ```

  Got back:

  ```javascript
  document.addEventListener("DOMContentLoaded", function() {
    // Show banner linking to /stable/ if this is a /latest/ page
    if (!/\/latest\//.test(location.pathname)) {
      return;
    }
    var stableUrl = location.pathname.replace("/latest/", "/stable/");
    // Check it's not a 404
    fetch(stableUrl, { method: "HEAD" }).then((response) => {
      if (response.status === 200) {
        var warning = document.createElement("div");
        warning.className = "admonition warning";
        warning.innerHTML = `
          <p class="first admonition-title">Note</p>
          <p class="last">
            This documentation covers the <strong>development version</strong> of Datasette.
          </p>
          <p>
            See <a href="${stableUrl}">this page</a> for the current stable release.
          </p>
        `;
        var mainArticle = document.querySelector("article[role=main]");
        mainArticle.insertBefore(warning, mainArticle.firstChild);
      }
    });
  });
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1484276302
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/388
node_id: IC_kwDOCGYnMM5YeEJO
created_at: 2023-03-26T23:39:27Z
updated_at: 2023-03-26T23:39:27Z
author_association: OWNER
body:
  Actually final Datasette fix was:
  ```html+jinja
  {% block scripts %}
  {{ super() }}
  <script>
  document.addEventListener("DOMContentLoaded", function() {
    // Show banner linking to /stable/ if this is a /latest/ page
    if (!/\/latest\//.test(location.pathname)) {
      return;
    }
    var stableUrl = location.pathname.replace("/latest/", "/stable/");
    // Check it's not a 404
    fetch(stableUrl, { method: "HEAD" }).then((response) => {
      if (response.status === 200) {
        var warning = document.createElement("div");
        warning.className = "admonition warning";
        warning.innerHTML = `
          <p class="first admonition-title">Note</p>
          <p class="last">
            This documentation covers the <strong>development version</strong> of Datasette.
          </p>
          <p>
            See <a href="${stableUrl}">this page</a> for the current stable release.
          </p>
        `;
        var mainArticle = document.querySelector("article[role=main]");
        mainArticle.insertBefore(warning, mainArticle.firstChild);
      }
    });
  });
  </script>
  {% endblock %}
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/262#issuecomment-1488010837
issue_url: https://api.github.com/repos/simonw/datasette/issues/262
node_id: IC_kwDOBm6k_c5YsT5V
created_at: 2023-03-29T06:22:21Z
updated_at: 2023-03-29T06:22:21Z
author_association: OWNER
body:
  I need to get the arbitrary query page to return the same format. It likely won't have nearly as many extras.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2049#issuecomment-1489526501
issue_url: https://api.github.com/repos/simonw/datasette/issues/2049
node_id: IC_kwDOBm6k_c5YyF7l
created_at: 2023-03-30T00:44:05Z
updated_at: 2023-03-30T00:44:05Z
author_association: OWNER
body:
  As part of this I should be able to figure out which bits of the new code I wrote for the table view should actually be shared with the query view. That stuff is mostly going to be from this commit: https://github.com/simonw/datasette/commit/d97e82df3c8a3f2e97038d7080167be9bb74a68d

  Here's the existing QueryView class I need to replace:

  https://github.com/simonw/datasette/blob/4c1e277edbd783d06840d3f9b20bf00783478ce4/datasette/views/database.py#L215-L532

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2049#issuecomment-1489530037
issue_url: https://api.github.com/repos/simonw/datasette/issues/2049
node_id: IC_kwDOBm6k_c5YyGy1
created_at: 2023-03-30T00:50:30Z
updated_at: 2023-03-30T00:50:30Z
author_association: OWNER
body:
  Two things to consider here: `_shape=` and `_extra=`.

  Most of the shapes make sense, with the exception of `?_shape=object` since we don't know which column we would use as a primary key.

  Looking at the (undocumented) list of extras from the table view, here are the ones I think make sense:

  - `count` - YES
  - `facet_results` - no
  - `facets_timed_out` - no
  - `suggested_facets` - no
  - `human_description_en` - no
  - `next_url` - MAYBE
  - `columns` - YES
  - `primary_keys` - no
  - `display_columns` - YES
  - `display_rows` - YES
  - `debug` - YES?
  - `request` - YES
  - `query` - YES
  - `metadata` - YES
  - `extras` - YES
  - `database` - YES
  - `table` - no
  - `database_color` - no?
  - `table_actions` - no
  - `filters` - no
  - `renderers` - YES
  - `custom_table_templates` - no
  - `sorted_facet_results` - no
  - `table_definition` - no
  - `view_definition` - no
  - `is_view` - no
  - `private` - YES
  - `expandable_columns` - no
  - `form_hidden_args` - no

  Just the YES ones:

  - `count` - this is new
  - `columns`
  - `display_columns`
  - `display_rows`
  - `debug`
  - `request`
  - `query`
  - `metadata`
  - `extras`
  - `database`
  - `renderers`
  - `private`

  The `count` one is interesting - I think I can provide that by optionally running `select count(*) from (inner query)`. It's a new feature though and not one I want to expose on the HTML view since it could result in poor performance - but having it as an extra that API users can opt into may make sense.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2049#issuecomment-1489530555
issue_url: https://api.github.com/repos/simonw/datasette/issues/2049
node_id: IC_kwDOBm6k_c5YyG67
created_at: 2023-03-30T00:51:27Z
updated_at: 2023-03-30T00:51:27Z
author_association: OWNER
body:
  I'd really like to refactor all of the extras functions into a `datasette/extras.py` module. The table ones currently rely a LOT on local variables in scope though, so I would need to rewrite those such that EVERY dependency they take is passed to `asyncinject` explicitly.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1989#issuecomment-1491357104
issue_url: https://api.github.com/repos/simonw/datasette/issues/1989
node_id: IC_kwDOBm6k_c5Y5E2w
created_at: 2023-03-31T06:17:23Z
updated_at: 2023-03-31T06:18:05Z
author_association: NONE
body:
  I'm running into a similar use case as pax above- I made a `nice` view that just has the data I'm interested in (which doesn't include the `id`, since it's not important in this case). But, by excluding `id` from the view, I can't do fts queries against it because the view has no `id` field to tie to `rowid`:

  ```
  ERROR: conn=<sqlite3.Connection object at 0x106521210>, 
  sql = 'select time, text, permalink, num_children from nice where id in 
  (select rowid from items_fts where items_fts match :search)  limit 101', 
  params = {'search': 'whatever'}: no such column: id
  ```

  It works fine when I include `id` in my view, but now my `nice` view is cluttered up. Would be great to hide it permanently in the `config.json`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2050#issuecomment-1492387771
issue_url: https://api.github.com/repos/simonw/datasette/issues/2050
node_id: IC_kwDOBm6k_c5Y9Ae7
created_at: 2023-03-31T17:59:48Z
updated_at: 2023-03-31T17:59:48Z
author_association: OWNER
body:
  Some of the extras from the table view make sense here. A few custom ones make sense too - including this already existing but undocumented one:

  https://github.com/simonw/datasette/blob/5890a20c374fb0812d88c9b0ef26a838bfa06c76/datasette/views/row.py#L86-L89

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1492777509
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/235
node_id: IC_kwDOCGYnMM5Y-fol
created_at: 2023-04-01T01:31:48Z
updated_at: 2023-04-01T01:31:48Z
author_association: NONE
body:
  My current workaround is to use this library from a python script instead of as a CLI tool.

  This lets me set the foreign key constraint at table creation time, instead of trying to modify an existing table. [docs](https://sqlite-utils.datasette.io/en/stable/python-api.html#specifying-foreign-keys)

  I found this [stackoverflow helpful](https://stackoverflow.com/a/1884841/5129731), as it explained that Sqlite doesn't support modifying existing tables directly.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/159#issuecomment-1493051222
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/159
node_id: IC_kwDOCGYnMM5Y_idW
created_at: 2023-04-01T17:21:05Z
updated_at: 2023-04-01T17:21:05Z
author_association: NONE
body:
  In a related issue, nearly a year later I just stumbled across this again, as I wondered why none of my rebuild-fts were rebuilding. It looks like: disable_fts in db.py commits; enable_fts partly commits except the last step (due to executescript committing a pending transaction); rebuild_fts won't commit unless manually done as above with e.g. a context manager.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/265#issuecomment-1493052396
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/265
node_id: IC_kwDOCGYnMM5Y_ivs
created_at: 2023-04-01T17:27:18Z
updated_at: 2023-04-01T17:27:18Z
author_association: NONE
body:
  `enable_fts` is a function in datasette, not in this repo, which doesn't do any escaping of search terms. It sounds like from https://docs.datasette.io/en/stable/full_text_search.html#advanced-sqlite-search-queries you might want to enable raw searching, as otherwise it's disabled and everything is escaped by default.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/apple-notes-to-sqlite/issues/6#issuecomment-1493442956
issue_url: https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/6
node_id: IC_kwDOJHON9s5ZBCGM
created_at: 2023-04-02T21:20:43Z
updated_at: 2023-04-02T21:25:37Z
author_association: NONE
body:
  I'm experiencing something similar. My apostrophes (') turn into (‚Äô) and the output is truncated. Hoping to debug next weekend 
  <img width="814" alt="Screenshot 2023-04-02 at 4 24 10 PM" src="https://user-images.githubusercontent.com/14314871/229379762-f622205a-7bba-4598-b0da-a6b86e5c0a44.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1495780111
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/235
node_id: IC_kwDOCGYnMM5ZJ8sP
created_at: 2023-04-04T11:09:56Z
updated_at: 2023-04-04T11:13:40Z
author_association: NONE
body:
  @wpears' workaround also worked for me, but also required me to manually set `PRAGMA writable_schema`.

  ```sql
  PRAGMA writable_schema = 1;
  UPDATE sqlite_master SET sql = 'CREATE TABLE [foos] (...)'
  PRAGMA writable_schema = 0;
  ```

  ```shell
  $ python --version
  Python 3.11.2

  $ sqlite3 --version
  3.41.2 2023-03-22 11:56:21 0d1fc92f94cb6b76bffe3ec34d69cffde2924203304e8ffc4155597af0c191da

  $ sqlite-utils --version
  sqlite-utils, version 3.30
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/79#issuecomment-1498167714
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/79
node_id: IC_kwDODFdgUs5ZTDmi
created_at: 2023-04-05T21:12:55Z
updated_at: 2023-04-05T21:12:55Z
author_association: MEMBER
body:
  It's a rate limiting problem: https://github.com/dogsheep/github-to-sqlite/actions/runs/4622674009/jobs/8175633155

  ```
    File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py", line 407, in commits
      utils.save_commits(db, commits, repo_full["id"])
    File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 635, in save_commits
      for commit in commits:
    File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 417, in fetch_commits
      for commits in paginate(url, headers):
    File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 478, in paginate
      raise GitHubError.from_response(response)
  github_to_sqlite.utils.GitHubError: ('API rate limit exceeded for user ID 9599.', 403)
  Error: Process completed with exit code 1.
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2049#issuecomment-1498275621
issue_url: https://api.github.com/repos/simonw/datasette/issues/2049
node_id: IC_kwDOBm6k_c5ZTd8l
created_at: 2023-04-05T23:23:01Z
updated_at: 2023-04-05T23:23:01Z
author_association: OWNER
body:
  The default representation here can be even smaller.

  For rows it's this:

  ```json
  {
    "ok": true,
    "next": "d,v",
    "rows": [...]
  }
  ```
  For SQL queries I'm considering this:
  ```json
  {
    "ok": true,
    "rows": [...]
  }
  ```
  I considered adding `"sql"` and `"params"` too, but on further thought those would be entirely a waste of bytes the majority of the time. If a user wants those they can request them with an `?_extra=query` as seen here:

  http://localhost:8001/content/releases.json?_size=0&_extra=query

  ```json
  {
    "ok": true,
    "next": null,
    "query": {
      "sql": "select html_url, id, author, node_id, tag_name, target_commitish, name, draft, prerelease, created_at, published_at, body, repo, reactions, mentions_count from releases order by id limit 1",
      "params": {}
    },
    "rows": []
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2053#issuecomment-1498279469
issue_url: https://api.github.com/repos/simonw/datasette/issues/2053
node_id: IC_kwDOBm6k_c5ZTe4t
created_at: 2023-04-05T23:28:53Z
updated_at: 2023-04-05T23:28:53Z
author_association: OWNER
body:
  Table errors page currently does this:
  ```json
  {
    "ok": false,
    "error": "no such column: blah",
    "status": 400,
    "title": "Invalid SQL"
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2054#issuecomment-1499452122
issue_url: https://api.github.com/repos/simonw/datasette/issues/2054
node_id: IC_kwDOBm6k_c5ZX9La
created_at: 2023-04-06T18:21:51Z
updated_at: 2023-04-06T18:21:51Z
author_association: OWNER
body:
  I'm going to make notes against the code in the most recent alpha release, ignoring the recent work I did to refactor `TableView`.

  https://github.com/simonw/datasette/tree/1.0a2/datasette/views

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2054#issuecomment-1499457201
issue_url: https://api.github.com/repos/simonw/datasette/issues/2054
node_id: IC_kwDOBm6k_c5ZX-ax
created_at: 2023-04-06T18:26:39Z
updated_at: 2023-04-06T18:26:39Z
author_association: OWNER
body:
  These classes - `TableView` and `RowView` and `QueryView` - all subclass `DataView` which subclasses` BaseView`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2054#issuecomment-1499457291
issue_url: https://api.github.com/repos/simonw/datasette/issues/2054
node_id: IC_kwDOBm6k_c5ZX-cL
created_at: 2023-04-06T18:26:45Z
updated_at: 2023-04-06T18:26:45Z
author_association: OWNER
body:
  Here's `BaseView`:

  https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L56-L145

  It has methods for the `options`, `get`, `post`, `delete`, `put`, `patch` and `head` HTTP verbs, most defaulting to returinng a 405 Method not allowed message in plain text or JSON, depending on this check:

  https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L71-L81

  Also adds CORS headers to anything if CORS mode is on:

  https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L106-L107

  And adds the `database_color` (weirdly) and the `select_templates` variables to the template context:

  https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L112-L122

  And has special code for setting the `Link: ...; rel="alternate"` HTTP header:

  https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L124-L136

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2054#issuecomment-1499462324
issue_url: https://api.github.com/repos/simonw/datasette/issues/2054
node_id: IC_kwDOBm6k_c5ZX_q0
created_at: 2023-04-06T18:31:56Z
updated_at: 2023-04-06T18:31:56Z
author_association: OWNER
body:
  The `DataView` class does a LOT of work - mostly involving CSV responses.

  https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L160-L544

  It has a `redirect()` method with some complex logic and CORS handling:

  https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L163-L172

  It uses this method a lot, which has to be over-ridden in the `TableView` and `RowView` and `QueryView` subclasses:

  https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L174-L175

  This method:

  https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L180

  Is the bulk of the complexity, because it knows how to both turn a list of SQLite rows into a CSV file but also knows how to call `.data()` repeatedly with different pagination arguments in order to stream CSV back for a large table.

  The `async def get()` method for GET requests is also very complicated. It mainly handles format stuff - knowing how to render HTML v.s. JSON v.s. CSV v.s. other formats specified using this plugin hook: https://docs.datasette.io/en/1.0a2/plugin_hooks.html#register-output-renderer-datasette

  Plus it catches interrupted queries and returns a special error page for those (and other error messages too): https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L381-L408

  It adds the time taken to execute the queries: https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L410-L411

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2054#issuecomment-1499465648
issue_url: https://api.github.com/repos/simonw/datasette/issues/2054
node_id: IC_kwDOBm6k_c5ZYAew
created_at: 2023-04-06T18:35:03Z
updated_at: 2023-04-06T18:35:03Z
author_association: OWNER
body:
  There are actually five classes that subclass `DataView`:

  https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/row.py#L16

  https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/database.py#L34

  https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/database.py#L172

  https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/database.py#L215

  https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/table.py#L72

  I don't think `DatabaseView` and `DatabaseDownload` should have subclassed that at all, since they don't return a table of data.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2054#issuecomment-1499467703
issue_url: https://api.github.com/repos/simonw/datasette/issues/2054
node_id: IC_kwDOBm6k_c5ZYA-3
created_at: 2023-04-06T18:37:22Z
updated_at: 2023-04-06T18:57:22Z
author_association: OWNER
body:
  I've been hoping to move entirely away from class-based views, but now I'm wondering if that's the right decision given the need to support HTTP verbs.

  It is cleaner to have a class that has methods for each verb, or to have a single function that can behave differently depending on the verb?

  Or should I have a mechanism for dispatching to separate functions based on the verbs that isn't implemented in a base class?

  The trouble with using base classes is that I've already shown that they tempt me to put logic in weird places, which makes code harder to modify later on.

  The thing here is that the ONLY thing I want to use the base class for is as an HTTP verb switcher - which makes me doubt that I should risk taking on the other temptations of having a base class.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2054#issuecomment-1499491258
issue_url: https://api.github.com/repos/simonw/datasette/issues/2054
node_id: IC_kwDOBm6k_c5ZYGu6
created_at: 2023-04-06T19:00:16Z
updated_at: 2023-04-06T20:40:16Z
author_association: OWNER
body:
  I asked GPT-4 to compare URL routing based on path and verb approaches in different languages and frameworks and spit out some examples. Full transcript here: https://gist.github.com/simonw/fa2379b97420404a81b0fcdb4db79657

  Initial prompt:

  > Show me code examples of different web frameworks in Python and JavaScript and Go illustrating how HTTP routing works - in particular the problem of mapping an incoming HTTP request to some code based on both the URL path and the HTTP verb
  >
  > Don't show installation instructions, just get straight to the code snippets

  Flask:
  ```python
  from flask import Flask

  app = Flask(__name__)

  @app.route('/example', methods=['GET'])
  def get_example():
      return "This is a GET request."

  @app.route('/example', methods=['POST'])
  def post_example():
      return "This is a POST request."

  if __name__ == '__main__':
      app.run()
  ```
  Express:
  ```javascript
  const express = require('express');
  const app = express();

  app.get('/example', (req, res) => {
    res.send('This is a GET request.');
  });

  app.post('/example', (req, res) => {
    res.send('This is a POST request.');
  });

  app.listen(3000, () => {
    console.log('Server is running on port 3000');
  });
  ```
  Gin:
  ```go
  package main

  import (
  	"github.com/gin-gonic/gin"
  )

  func main() {
  	r := gin.Default()

  	r.GET("/example", func(c *gin.Context) {
  		c.String(200, "This is a GET request.")
  	})

  	r.POST("/example", func(c *gin.Context) {
  		c.String(200, "This is a POST request.")
  	})

  	r.Run()
  }
  ```
  Actix-web:
  ```rust
  use actix_web::{web, App, HttpResponse, HttpServer, Responder};

  async fn get_example() -> impl Responder {
      HttpResponse::Ok().body("This is a GET request.")
  }

  async fn post_example() -> impl Responder {
      HttpResponse::Ok().body("This is a POST request.")
  }

  #[actix_web::main]
  async fn main() -> std::io::Result<()> {
      HttpServer::new(|| {
          App::new()
              .route("/example", web::get().to(get_example))
              .route("/example", web::post().to(post_example))
      })
      .bind("127.0.0.1:8080")?
      .run()
      .await
  }
  ```
  Koa:
  ```javascript
  const Koa = require('koa');
  const Router = require('koa-router');

  const app = new Koa();
  const router = new Router();

  router.get('/example', (ctx) => {
    ctx.body = 'This is a GET request.';
  });

  router.post('/example', (ctx) => {
    ctx.body = 'This is a POST request.';
  });

  app.use(router.routes());
  app.use(router.allowedMethods());

  app.listen(3000, () => {
    console.log('Server is running on port 3000');
  });
  ```
  Django:
  ```python
  from django.http import HttpResponse
  from django.views import View

  class ExampleView(View):
      def get(self, request):
          return HttpResponse("This is a GET request.")

      def post(self, request):
          return HttpResponse("This is a POST request.")
  ```
  ```python
  from django.urls import path
  from .views import ExampleView

  urlpatterns = [
      path('example/', ExampleView.as_view(), name='example'),
  ]
  ```
  Rocket:
  ```rust
  #![feature(proc_macro_hygiene, decl_macro)]

  #[macro_use] extern crate rocket;

  use rocket::response::content;

  #[get("/example")]
  fn get_example() -> content::Plain<&'static str> {
      content::Plain("This is a GET request.")
  }

  #[post("/example")]
  fn post_example() -> content::Plain<&'static str> {
      content::Plain("This is a POST request.")
  }

  #[launch]
  fn rocket() -> _ {
      rocket::build().mount("/", routes![get_example, post_example])
  }
  ```
  Elixir/Phoenix:
  ```elixir
  defmodule ExampleAppWeb.Router do
    use ExampleAppWeb, :router

    pipeline :api do
      plug :accepts, ["json"]
    end

    scope "/api", ExampleAppWeb do
      pipe_through :api

      get "/example", ExampleController, :get_example
      post "/example", ExampleController, :post_example
    end
  end
  ```
  Bottle:
  ```python
  from bottle import Bottle, run

  app = Bottle()

  @app.route('/example', method='GET')
  def get_example():
      return "This is a GET request."

  @app.route('/example', method='POST')
  def post_example():
      return "This is a POST request."

  if __name__ == '__main__':
      run(app, host='localhost', port=8080)
  ```
  FastAPI:
  ```python
  from fastapi import FastAPI

  app = FastAPI()

  @app.get("/example")
  async def get_example():
      return {"message": "This is a GET request."}

  @app.post("/example")
  async def post_example():
      return {"message": "This is a POST request."}

  if __name__ == "__main__":
      import uvicorn
      uvicorn.run(app, host="127.0.0.1", port=8000)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2054#issuecomment-1499596941
issue_url: https://api.github.com/repos/simonw/datasette/issues/2054
node_id: IC_kwDOBm6k_c5ZYgiN
created_at: 2023-04-06T20:39:35Z
updated_at: 2023-04-06T20:39:35Z
author_association: OWNER
body:
  I think there are two main approaches:

  - Map path to a function, that function then behaves differently on GET/POST/etc
  - Map method AND path to a function - the `app.get('/example')` v.s. `app.post('/example')` ones

  The decorator approach is calling out to me a bit now.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2054#issuecomment-1499604066
issue_url: https://api.github.com/repos/simonw/datasette/issues/2054
node_id: IC_kwDOBm6k_c5ZYiRi
created_at: 2023-04-06T20:47:30Z
updated_at: 2023-04-06T20:47:30Z
author_association: OWNER
body:
  I'm contemplating a new approach: using a class with static methods. Something like this:

  ```python
  class TableView(MethodRouter):
      @staticmethod
      async def get(request):
          return Response.text("GET")

      @staticmethod
      async def post(request):
          return Response.text("POST")
  ```
  So effectively the class is just there to bundle together verb implementations, and to provide a `route(request)` method which knows how to dispatch them to the right place.

  It can offer default HEAD and OPTIONS methods too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2054#issuecomment-1499604822
issue_url: https://api.github.com/repos/simonw/datasette/issues/2054
node_id: IC_kwDOBm6k_c5ZYidW
created_at: 2023-04-06T20:48:19Z
updated_at: 2023-04-06T20:48:55Z
author_association: OWNER
body:
  I actually quite like that. I could even use `@classmethod` and have utility methods defined on that class that both `get()` and `post()` could call.

  The crucial rule here is NO INSTANCE STATE - that's what makes routing to classes particularly damaging, and encourages code that's hard to maintain.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2054#issuecomment-1500608101
issue_url: https://api.github.com/repos/simonw/datasette/issues/2054
node_id: IC_kwDOBm6k_c5ZcXZl
created_at: 2023-04-07T20:14:38Z
updated_at: 2023-04-07T20:14:38Z
author_association: OWNER
body:
  Ooh that one's really interesting - very different from the others:
  ```ruby
  # app.rb
  require "roda"

  class App < Roda
    route do |r|
      r.root do
        "Home page"
      end

      r.on "pages" do
        r.get ":slug" do |slug|
          "Page: #{slug}"
        end
      end

      r.on "news" do
        r.get ":yyyy/:mm/:dd" do |yyyy, mm, dd|
          "News for #{yyyy}/#{mm}/#{dd}"
        end
      end
    end
  end

  # config.ru
  require_relative "app"
  run App.freeze.app
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/536#issuecomment-1500893216
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/536
node_id: IC_kwDOCGYnMM5ZddAg
created_at: 2023-04-08T13:35:42Z
updated_at: 2023-04-08T13:35:42Z
author_association: NONE
body:
  ## [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/536?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch and project coverage have no change.
  > Comparison is base [(`c0251cc`)](https://codecov.io/gh/simonw/sqlite-utils/commit/c0251cc9271260de73b4227859a51fab9b4cb745?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.25% compared to head [(`cea05dc`)](https://codecov.io/gh/simonw/sqlite-utils/pull/536?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.25%.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main     #536   +/-   ##
  =======================================
    Coverage   96.25%   96.25%           
  =======================================
    Files           6        6           
    Lines        2671     2671           
  =======================================
    Hits         2571     2571           
    Misses        100      100           
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/536?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [sqlite\_utils/utils.py](https://codecov.io/gh/simonw/sqlite-utils/pull/536?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `95.13% <ø> (ø)` | |

  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  </details>

  [:umbrella: View full report in Codecov by Sentry](https://codecov.io/gh/simonw/sqlite-utils/pull/536?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/531#issuecomment-1501017004
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/531
node_id: IC_kwDOCGYnMM5Zd7Os
created_at: 2023-04-09T01:49:43Z
updated_at: 2023-04-09T01:49:43Z
author_association: CONTRIBUTOR
body:
  I'm going to close this in favor of #536. Will try a cleaner approach to custom paths once that one is merge.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/swarm-to-sqlite/issues/13#issuecomment-1502543165
issue_url: https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/13
node_id: IC_kwDODD6af85Zjv09
created_at: 2023-04-11T01:10:36Z
updated_at: 2023-04-11T01:11:47Z
author_association: MEMBER
body:
  I just had that error myself on macOS while running the tests:
  ```
  ERROR tests/test_save_checkin.py::test_tables - sqlite3.OperationalError: table sqlite_master may not be modified
  ERROR tests/test_save_checkin.py::test_venue - sqlite3.OperationalError: table sqlite_master may not be modified
  ERROR tests/test_save_checkin.py::test_event - sqlite3.OperationalError: table sqlite_master may not be modified
  ERROR tests/test_save_checkin.py::test_sticker - sqlite3.OperationalError: table sqlite_master may not be modified
  ERROR tests/test_save_checkin.py::test_likes - sqlite3.OperationalError: table sqlite_master may not be modified
  ERROR tests/test_save_checkin.py::test_with_ - sqlite3.OperationalError: table sqlite_master may not be modified
  ERROR tests/test_save_checkin.py::test_users - sqlite3.OperationalError: table sqlite_master may not be modified
  ERROR tests/test_save_checkin.py::test_photos - sqlite3.OperationalError: table sqlite_master may not be modified
  ERROR tests/test_save_checkin.py::test_posts - sqlite3.OperationalError: table sqlite_master may not be modified
  ERROR tests/test_save_checkin.py::test_view - sqlite3.OperationalError: table sqlite_master may not be modified
  ```
  `pytest --pdb` shows it happening in the bit that adds foreign keys:
  ```
  > /Users/simon/.local/share/virtualenvs/swarm-to-sqlite-daPW7yIJ/lib/python3.9/site-packages/sqlite_utils/db.py(1096)add_foreign_keys()
  -> cursor.execute(
  (Pdb) list
  1096 >>	                cursor.execute(
  1097 	                    "UPDATE sqlite_master SET sql = ? WHERE name = ?",
  1098 	                    (new_sql, table_name),
  1099 	                )
  1100 	            cursor.execute("PRAGMA schema_version = %d" % (schema_version + 1))
  1101 ->	            cursor.execute("PRAGMA writable_schema = 0")
  1102 	        # Have to VACUUM outside the transaction to ensure .foreign_keys property
  1103 	        # can see the newly created foreign key.
  1104 	        self.vacuum()
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1502556111
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/235
node_id: IC_kwDOCGYnMM5Zjy_P
created_at: 2023-04-11T01:28:41Z
updated_at: 2023-04-11T01:28:41Z
author_association: OWNER
body:
  Investigating this one now.

  The `sqlite-utils` test suite passes without errors on my Python 3.11.2 installation... but it fails with this error on a Python 3.9.6 installation.

  In the broken version's virtual environment directory I ran this:
  ```
  cat pyvenv.cfg 
  home = /Applications/Xcode.app/Contents/Developer/usr/bin
  implementation = CPython
  version_info = 3.9.6.final.0
  virtualenv = 20.17.1
  include-system-site-packages = false
  base-prefix = /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9
  base-exec-prefix = /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9
  base-executable = /Applications/Xcode.app/Contents/Developer/usr/bin/python3
  ```
  So it looks like the Xcode `python3` has "defensive" mode turned on for SQLite.

  As far as I can tell there's no way to turn it OFF again in Python.

  My virtual environment that DOES work has this:

  ```
  home = /opt/homebrew/opt/python@3.11/bin
  implementation = CPython
  version_info = 3.11.2.final.0
  virtualenv = 20.17.1
  include-system-site-packages = false
  base-prefix = /opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11
  base-exec-prefix = /opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11
  base-executable = /opt/homebrew/opt/python@3.11/bin/python3.11
  ```
  So the Python 3.11 I installed through Homebrew doesn't exhibit this bug.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1502557629
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/235
node_id: IC_kwDOCGYnMM5ZjzW9
created_at: 2023-04-11T01:30:12Z
updated_at: 2023-04-11T01:30:12Z
author_association: OWNER
body:
  I'll ask on the SQLite forum if it's possible to toggle that mode on and off using regular SQL. My hunch is that it isn't.

  In which case `sqlite-utils` should at least know how to catch this error and display a much more readable error message, maybe with a link to further documentation.

  A utility function that can detect this mode would be really useful too. I'd probably have to do a test that tries to modify `sqlite_master` on a new in-memory database to catch if it's possible or not.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1502559442
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/235
node_id: IC_kwDOCGYnMM5ZjzzS
created_at: 2023-04-11T01:32:30Z
updated_at: 2023-04-11T01:33:27Z
author_association: OWNER
body:
  This seems to work:
  ```python
  import sqlite3
  db = sqlite3.connect(":memory:")
  db.executescript("""
  PRAGMA writable_schema = 1;
  UPDATE sqlite_master SET sql = 'CREATE TABLE [foos] (id integer primary key)';
  PRAGMA writable_schema = 0;
  """)
  ```
  It succeeds on my Python 3.11 and raises the following exception on my broken Python 3.9:
  ```
  sqlite3.OperationalError: table sqlite_master may not be modified
  ```
  Removing the `PRAGMA writable_schema = 1;` causes the same exception to be raised on both Pythons.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/swarm-to-sqlite/issues/13#issuecomment-1502629219
issue_url: https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/13
node_id: IC_kwDODD6af85ZkE1j
created_at: 2023-04-11T03:15:26Z
updated_at: 2023-04-11T03:15:26Z
author_association: MEMBER
body:
  OK, I figured this out. Unfortunately it's an error that occurs on Python versions that have defensive mode turned on, and it doesn't look like there's a way to turn that mode off. See notes above.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/swarm-to-sqlite/issues/13#issuecomment-1502629404
issue_url: https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/13
node_id: IC_kwDODD6af85ZkE4c
created_at: 2023-04-11T03:15:47Z
updated_at: 2023-04-11T03:46:17Z
author_association: MEMBER
body:
  I think `swarm-to-sqlite` needs to avoid this error, maybe by setting up foreign keys in another way - or even by skipping foreign keys entirely on databases that don't support this kind of operation.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2057#issuecomment-1503832422
issue_url: https://api.github.com/repos/simonw/datasette/issues/2057
node_id: IC_kwDOBm6k_c5Zoqlm
created_at: 2023-04-11T17:42:57Z
updated_at: 2023-04-11T17:46:42Z
author_association: OWNER
body:
  I ran this prompt against ChatGPT with the Browsing alpha:

  > ```python
  >                 if pkg_resources.resource_isdir(plugin.__name__, "static"):
  >                     static_path = pkg_resources.resource_filename(
  >                         plugin.__name__, "static"
  >                     )
  >                 if pkg_resources.resource_isdir(plugin.__name__, "templates"):
  >                     templates_path = pkg_resources.resource_filename(
  >                         plugin.__name__, "templates"
  >                     )
  > ```
  > This code gives a deprecation warning in Python 3.11 - fix it

  It looked up the fix for me:

  <img width="632" alt="image" src="https://user-images.githubusercontent.com/9599/231245259-af4d962c-e631-420b-8312-fd9657a23bbd.png">

  And suggested:

  ```python
  import importlib.resources

  # Replace pkg_resources.resource_isdir with importlib.resources.files().is_file()
  if importlib.resources.files(plugin.__name__).joinpath("static").is_file():
      static_path = importlib.resources.as_file(
          importlib.resources.files(plugin.__name__).joinpath("static")
      )
  if importlib.resources.files(plugin.__name__).joinpath("templates").is_file():
      templates_path = importlib.resources.as_file(
          importlib.resources.files(plugin.__name__).joinpath("templates")
      )
  ```
  This looks wrong to me - I would expect something like `is_directory()` not `is_file()` for telling if `static/` is a directory.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2057#issuecomment-1503833906
issue_url: https://api.github.com/repos/simonw/datasette/issues/2057
node_id: IC_kwDOBm6k_c5Zoq8y
created_at: 2023-04-11T17:44:16Z
updated_at: 2023-04-11T17:45:45Z
author_association: OWNER
body:
  Another prompt:

  > How to fix this:
  >
  > `pkg_resources.get_distribution(package).version`

  Response:

  <img width="641" alt="image" src="https://user-images.githubusercontent.com/9599/231245559-dd943527-818a-45aa-9833-b9db8c7ca87e.png">

  ```python
  import importlib.metadata

  # Get the version number of the specified package
  package_version = importlib.metadata.version(package)
  ```

  That seems to work:

  ```pycon
  >>> import importlib.metadata
  >>> importlib.metadata.version("datasette")
  '0.64.2'
  >>> importlib.metadata.version("pluggy")
  '1.0.0'
  >>> importlib.metadata.version("not-a-package")
  ...
  importlib.metadata.PackageNotFoundError: No package metadata was found for not-a-package
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2057#issuecomment-1503838640
issue_url: https://api.github.com/repos/simonw/datasette/issues/2057
node_id: IC_kwDOBm6k_c5ZosGw
created_at: 2023-04-11T17:48:23Z
updated_at: 2023-04-11T17:48:23Z
author_association: OWNER
body:
  > This looks wrong to me - I would expect something like `is_directory()` not `is_file()` for telling if `static/` is a directory.

  I was right about that:

  ```pycon
  >>> importlib.resources.files('datasette_graphql')
  PosixPath('/Users/simon/.local/share/virtualenvs/datasette-big-local-6Yn-280V/lib/python3.11/site-packages/datasette_graphql')
  >>> importlib.resources.files('datasette_graphql').joinpath("static")
  PosixPath('/Users/simon/.local/share/virtualenvs/datasette-big-local-6Yn-280V/lib/python3.11/site-packages/datasette_graphql/static')
  >>> p = importlib.resources.files('datasette_graphql').joinpath("static")
  >>> p
  PosixPath('/Users/simon/.local/share/virtualenvs/datasette-big-local-6Yn-280V/lib/python3.11/site-packages/datasette_graphql/static')
  >>> p.is_
  p.is_absolute()     p.is_char_device()  p.is_fifo()         p.is_mount()        p.is_reserved()     p.is_symlink()     
  p.is_block_device() p.is_dir()          p.is_file()         p.is_relative_to(   p.is_socket()      
  >>> p.is_dir()
  True
  >>> p.is_file()
  False
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1504245029
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/235
node_id: IC_kwDOCGYnMM5ZqPUl
created_at: 2023-04-11T23:13:41Z
updated_at: 2023-04-11T23:14:39Z
author_association: OWNER
body:
  I also tested this against the current `ubuntu:latest` Docker image (on an M2 Mac), in Python 3.10 and 3.11:
  ```
  docker run -it ubuntu:latest /bin/bash
  ```
  Then in the container:
  ```
  apt-get update
  apt-get install python3
  python3
  # pasted in the above recipe
  apt install software-properties-common
  add-apt-repository ppa:deadsnakes/ppa
  apt install python3.11
  python3.11 
  # pasted it in again
  ```
  In both cases the Python code did not raise an exception, which suggests that on Ubuntu those two Python versions do not have the defensive mode set.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1504288134
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/235
node_id: IC_kwDOCGYnMM5ZqZ2G
created_at: 2023-04-11T23:55:06Z
updated_at: 2023-04-12T03:34:32Z
author_association: OWNER
body:
  Also checked the official Datasette Docker image - I had to run that in Codespaces because it doesn't currently work on my M2 Mac:

  ```
  codespace@codespaces-112c61:/workspaces/sqlite-utils$ docker pull datasetteproject/datasette
  Using default tag: latest
  ...
  codespace@codespaces-112c61:/workspaces/sqlite-utils$ docker run -it datasetteproject/datasette /
  bin/bash
  root@75ba34f501ec:/# python
  Python 3.11.0 (main, Dec  6 2022, 13:31:55) [GCC 10.2.1 20210110] on linux
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import sqlite3
  .executescript("""
  PRAGMA writable_schema = 1;
  UPDATE sqlite_master SET sql = 'CREATE TABLE [foos] (id integer primary key)';
  PRAGMA writable_schema = 0;
  """)>>> db = sqlite3.connect(":memory:")
  >>> db.executescript("""
  ... PRAGMA writable_schema = 1;
  ... UPDATE sqlite_master SET sql = 'CREATE TABLE [foos] (id integer primary key)';
  ... PRAGMA writable_schema = 0;
  ... """)
  <sqlite3.Cursor object at 0x7fd9b0561140>
  >>>
  ```
  So that confirms that the official image also has a Python with a SQLite that's not in defensive mode.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2058#issuecomment-1504291892
issue_url: https://api.github.com/repos/simonw/datasette/issues/2058
node_id: IC_kwDOBm6k_c5Zqaw0
created_at: 2023-04-11T23:58:45Z
updated_at: 2023-04-11T23:58:45Z
author_association: OWNER
body:
  I thought it might relate to the "defensive mode" issue described here:

  - https://github.com/simonw/sqlite-utils/issues/235

  But I have since determined that the Datasette official Docker image does NOT run anything in defensive mode, so I don't think it's related to that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2058#issuecomment-1504292145
issue_url: https://api.github.com/repos/simonw/datasette/issues/2058
node_id: IC_kwDOBm6k_c5Zqa0x
created_at: 2023-04-11T23:58:59Z
updated_at: 2023-04-11T23:58:59Z
author_association: OWNER
body:
  Asked on the SQLite Forum if anyone has seen this before: https://sqlite.org/forum/forumpost/793a2ed75b

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2058#issuecomment-1504295345
issue_url: https://api.github.com/repos/simonw/datasette/issues/2058
node_id: IC_kwDOBm6k_c5Zqbmx
created_at: 2023-04-12T00:01:42Z
updated_at: 2023-04-12T00:02:26Z
author_association: OWNER
body:
  Here's the relevant code:

  https://github.com/simonw/datasette/blob/5890a20c374fb0812d88c9b0ef26a838bfa06c76/datasette/app.py#L421-L437

  This function is called on almost every request (everything that subclasses `BaseView` at least - need to remember that for the refactor in #2053 etc).

  https://github.com/simonw/datasette/blob/5890a20c374fb0812d88c9b0ef26a838bfa06c76/datasette/views/base.py#L101-L103

  It uses `PRAGMA schema_version` as a cheap way to determine if the schema has changed, in which case it needs to refresh the internal schema tables.

  This was already the cause of a subtle bug here:
  - #1231

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2058#issuecomment-1504298448
issue_url: https://api.github.com/repos/simonw/datasette/issues/2058
node_id: IC_kwDOBm6k_c5ZqcXQ
created_at: 2023-04-12T00:04:01Z
updated_at: 2023-04-12T00:04:01Z
author_association: OWNER
body:
  Here's a potential workaround: when I store the schema versions, I could also score an MD5 hash of the full schema (`select group_concat(sql) from sqlite_master`). When I read the schema version with `PRAGMA schema_version` I could catch that exception and, if I see it, I could calculate that MD5 hash again as a fallback and use that to determine if the schema has changed instead.

  The performance overhead of this needs investigating - how much more expensive is `md5(... that SQL query result)` compared to just `PRAGMA schema_version`, especially on a database with a lot of tables?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2058#issuecomment-1504315697
issue_url: https://api.github.com/repos/simonw/datasette/issues/2058
node_id: IC_kwDOBm6k_c5Zqgkx
created_at: 2023-04-12T00:16:22Z
updated_at: 2023-04-12T00:27:12Z
author_association: OWNER
body:
  I got ChatGPT (code execution alpha) to run a micro-benchmark for me. This was the conclusion:

  > The benchmark using `PRAGMA schema_version` is approximately 1.36 times faster than the benchmark using `hashlib.md5` for the case with 100 tables. For the case with 200 tables, the benchmark using `PRAGMA schema_version` is approximately 2.33 times faster than the benchmark using `hashlib.md5`.

  Here's the chart it drew me:

  ![image](https://user-images.githubusercontent.com/9599/231315366-3a12b6d3-08d7-419d-a1fd-36eb24da0d85.png)

  (It's a pretty rubbish chart though, it only took measurements at 100 and 200 and drew a line between the two, I should have told it to measure every 10 and plot that)

  And the full transcript: https://gist.github.com/simonw/2fc46effbfbe49e6de0bcfdc9e31b235

  The benchmark looks good enough on first glance that I don't feel the need to be more thorough with it. `PRAGMA schema_version` is faster, but not so fast that I feel like the MD5 hack is worth worrying about too much.

  I'm tempted to add something to the `/-/versions` page that tries to identify if this is a problem or not though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2058#issuecomment-1504328395
issue_url: https://api.github.com/repos/simonw/datasette/issues/2058
node_id: IC_kwDOBm6k_c5ZqjrL
created_at: 2023-04-12T00:28:38Z
updated_at: 2023-04-12T00:28:38Z
author_association: OWNER
body:
  Here's a much better chart, which shows that MD5 performance unsurprisingly gets worse as the number of tables increases while `schema_version` remains constant:

  ![image](https://user-images.githubusercontent.com/9599/231316778-513bd99f-5ea4-495c-b86d-c572a7106369.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2058#issuecomment-1504426792
issue_url: https://api.github.com/repos/simonw/datasette/issues/2058
node_id: IC_kwDOBm6k_c5Zq7so
created_at: 2023-04-12T02:02:42Z
updated_at: 2023-04-12T02:02:42Z
author_association: OWNER
body:
  I tightened up the benchmark (it was measuring the time taken to create the tables too) and got this:

  ![image](https://user-images.githubusercontent.com/9599/231328328-85ca35ac-a11b-46f4-b132-dae367103570.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2059#issuecomment-1506174353
issue_url: https://api.github.com/repos/simonw/datasette/issues/2059
node_id: IC_kwDOBm6k_c5ZxmWR
created_at: 2023-04-13T01:13:00Z
updated_at: 2023-04-13T01:13:00Z
author_association: OWNER
body:
  Can you provide a URL to an example, and/or a screenshot of this? Is it a browser warning or is it a warning from Heroku itself?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2056#issuecomment-1506175208
issue_url: https://api.github.com/repos/simonw/datasette/issues/2056
node_id: IC_kwDOBm6k_c5Zxmjo
created_at: 2023-04-13T01:14:13Z
updated_at: 2023-04-13T01:14:13Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/actions/runs/4664796647/jobs/8300596121?pr=2056 it's pretty fast - that finished in 9s.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2056#issuecomment-1506177115
issue_url: https://api.github.com/repos/simonw/datasette/issues/2056
node_id: IC_kwDOBm6k_c5ZxnBb
created_at: 2023-04-13T01:17:16Z
updated_at: 2023-04-13T01:17:16Z
author_association: OWNER
body:
  Here are the failures: https://github.com/simonw/datasette/actions/runs/4684460653/jobs/8300630794?pr=2056

  <img width="902" alt="CleanShot 2023-04-12 at 18 16 45@2x" src="https://user-images.githubusercontent.com/9599/231620533-bce286b7-8173-4a61-aac1-b772832fb893.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2056#issuecomment-1506177857
issue_url: https://api.github.com/repos/simonw/datasette/issues/2056
node_id: IC_kwDOBm6k_c5ZxnNB
created_at: 2023-04-13T01:18:18Z
updated_at: 2023-04-13T01:18:18Z
author_association: OWNER
body:
  Cool - and now https://github.com/simonw/datasette/pull/2056/files is showing me those inline annotations:

  <img width="868" alt="image" src="https://user-images.githubusercontent.com/9599/231620660-4843d113-b671-494e-8265-24226446abba.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2056#issuecomment-1506179555
issue_url: https://api.github.com/repos/simonw/datasette/issues/2056
node_id: IC_kwDOBm6k_c5Zxnnj
created_at: 2023-04-13T01:21:05Z
updated_at: 2023-04-13T01:22:08Z
author_association: OWNER
body:
  OK, I'm sold - this is a really neat improvement.

  One thing to change in the PR: right now it runs `pip install --user ruff` on every commit, which hits PyPI to install the package.

  I prefer to avoid hitting PyPI every time, so I like to use the GitHub Actions cache. My usual pattern for that looks like this:

  https://github.com/simonw/datasette/blob/5890a20c374fb0812d88c9b0ef26a838bfa06c76/.github/workflows/test-pyodide.yml#L16-L20

  Then a separate command that runs `pip install ...` will benefit from that cache.

  Are you OK to make that change?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/537#issuecomment-1506200813
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/537
node_id: IC_kwDOCGYnMM5Zxszt
created_at: 2023-04-13T01:45:22Z
updated_at: 2023-04-13T01:45:22Z
author_association: NONE
body:
  ## [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/537?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch coverage: **`100.00`**% and no project coverage change.
  > Comparison is base [(`c0251cc`)](https://codecov.io/gh/simonw/sqlite-utils/commit/c0251cc9271260de73b4227859a51fab9b4cb745?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.25% compared to head [(`a75abeb`)](https://codecov.io/gh/simonw/sqlite-utils/pull/537?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.25%.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main     #537   +/-   ##
  =======================================
    Coverage   96.25%   96.25%           
  =======================================
    Files           6        6           
    Lines        2671     2673    +2     
  =======================================
  + Hits         2571     2573    +2     
    Misses        100      100           
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/537?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/537?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.33% <100.00%> (+<0.01%)` | :arrow_up: |

  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  </details>

  [:umbrella: View full report in Codecov by Sentry](https://codecov.io/gh/simonw/sqlite-utils/pull/537?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/527#issuecomment-1506223848
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/527
node_id: IC_kwDOCGYnMM5Zxybo
created_at: 2023-04-13T02:08:16Z
updated_at: 2023-04-13T02:08:16Z
author_association: OWNER
body:
  I agree, this is a design flaw.

  It's technically a breaking change. As such, I would need to bump to v4 to responsibly release this.

  I'd rather bundle in a few more breaking changes before shipping that version.

  One thing we could do here is add an extra argument to `.convert()` - something like this:

  ```python
  table.convert(col, lambda x: x+1, skip_false=False)
  ```

  This would trigger the new, improved behaviour without breaking existing code that depends on how it works at the moment.

  Then in `sqlite-utils` 4 we can change the default of that option.

  What do you think?

  (I'm open to suggestions for better names for this parameter too)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2056#issuecomment-1506485287
issue_url: https://api.github.com/repos/simonw/datasette/issues/2056
node_id: IC_kwDOBm6k_c5ZyyQn
created_at: 2023-04-13T07:29:38Z
updated_at: 2023-04-13T07:41:55Z
author_association: NONE
body:
  Ruff (written in Rust, not Python) is a 23MB executable so the time to download and pip install it dwarfs its runtime.

  Let's run ruff with and without GitHub Actions pip cache side-by-side to see the relative performance.

  Once you approve the workflows below, `ruff_with_cache` should echo `cache-hit = false` but if you rerun that job hopefully it should echo `cache-hit = true`.  That will be the execution time that we are interested to compare.

  There are two great problems in computer science: ;-)
  1. Naming things
  2. Cache invalidation
  3. Off-by-one errors

  For 2., https://github.com/actions/setup-python#caching-packages-dependencies is vital reading.
  Only _exactly pinned requirements_ can be cached.  Currently in `setup.py` the only pinned dependencies are:
  1. Sphinx==6.1.3
  2. furo==2023.3.27
  3. black==23.3.0
  4. blacken-docs==1.13.0  # but unpinned elsewhere in `setup.py`

  This means that there will be very few cache hits in the current actions.  See the link below to print out cache hits:
  https://github.com/actions/setup-python/blob/main/docs/advanced-usage.md#cache-hit

  > PyPI uses Fastly's CDN to quickly serve content to end-users, allowing us to minimize our hosting infrastructure and obscure possible downtime. -- https://pypi.org/sponsors

  I would be shocked if Fastly does not have beefy CDN nodes in the same datacenters where GitHub Actions run so GHA requests to download `ruff` probably never hit a PyPI server.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2058#issuecomment-1507264934
issue_url: https://api.github.com/repos/simonw/datasette/issues/2058
node_id: IC_kwDOBm6k_c5Z1wmm
created_at: 2023-04-13T16:35:21Z
updated_at: 2023-04-13T16:35:21Z
author_association: NONE
body:
  I tried deploying the fix you submitted, but still getting the same errors. I can past the logs here if needed, but I really don't see anything new in them.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/apple-notes-to-sqlite/issues/6#issuecomment-1508784533
issue_url: https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/6
node_id: IC_kwDOJHON9s5Z7jmV
created_at: 2023-04-14T15:22:09Z
updated_at: 2023-04-14T15:22:09Z
author_association: NONE
body:
  Just changing the encoding in `extract_notes` to `utf8` seems to fix it for my titles that were messed up.

  ![Screen Shot 2023-04-14 at 5 14 18 PM](https://user-images.githubusercontent.com/579727/232086062-e7edc4d1-0880-417a-925b-fd6c65b05155.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2052#issuecomment-1509461324
issue_url: https://api.github.com/repos/simonw/datasette/issues/2052
node_id: IC_kwDOBm6k_c5Z-I1M
created_at: 2023-04-15T01:57:06Z
updated_at: 2023-04-15T01:57:06Z
author_association: CONTRIBUTOR
body:
  Notes from 1:1 - it _is_ possible to pass in URL params into a ObservableHQ notebook: https://observablehq.com/@bherbertlc/pass-values-as-url-parameters

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/425#issuecomment-1509951952
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/425
node_id: IC_kwDOCGYnMM5aAAnQ
created_at: 2023-04-15T20:14:58Z
updated_at: 2023-04-15T20:14:58Z
author_association: NONE
body:
  is this change released ? Because when we run docker containers issue still persists for production deployments.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2052#issuecomment-1510423051
issue_url: https://api.github.com/repos/simonw/datasette/issues/2052
node_id: IC_kwDOBm6k_c5aBzoL
created_at: 2023-04-16T16:12:14Z
updated_at: 2023-04-20T05:14:39Z
author_association: CONTRIBUTOR
body:
  # Javascript Plugin Docs (alpha)

  ## Motivation

  The Datasette JS Plugin API allows developers to add interactive features to the UI, without having to modify the Python source code. 

  ## Setup

  No external/NPM dependencies are needed.

  Plugin behavior is coordinated by the Datasette `manager`. Every page has 1 `manager`.

  There are 2 ways to add your plugin to the `manager`.

  1. Read `window.__DATASETTE__` if the manager was already loaded.

  ```js
  const manager = window.__DATASETTE__;
  ```

  2. Wait for the `datasette_init` event to fire if your code was loaded before the manager is ready. 

  ```js
  document.addEventListener("datasette_init", function (evt) {
    const { detail: manager } = evt;
    
    // register plugin here
  });
  ```

  3. Add plugin to the manager by calling `manager.registerPlugin` in a JS file. Each plugin will supply 1 or more hooks with

  - unique name (`YOUR_PLUGIN_NAME`)
  - a numeric version (starting at `0.1`), 
  - configuration value, the details vary by hook. (In this example, `getColumnActions` takes a function)

  ```js
  manager.registerPlugin("YOUR_PLUGIN_NAME", {
      version: 0.1,
      makeColumnActions: (columnMeta) => {
        return [
          {
            label: "Copy name to clipboard",
            // evt = native click event
            onClick: (evt) => copyToClipboard(columnMeta.column),
          }
        ];
      },
    });
  ```

  There are 2 plugin hooks available to `manager.registerPlugin`:

  - `makeColumnActions` - Add items to the cog menu for headers on datasette table pages
  - `makeAboveTablePanelConfigs` - Add items to "tabbed" panel above the `<table/>` on pages that use the Datasette table template.

  While there are additional properties on the `manager`, but it's not advised to depend on them directly as the shape is subject to change.

  4. To make your JS file available as a Datasette plugin from the Python side, you can add a python file resembling [this](https://github.com/simonw/datasette/pull/2052/files#diff-c5ecf3d22075a60d04a4e95da2e15c612cf1bc84e38d777b67ba60dbd156e293) to your plugins directory. Note that you could host your JS file anywhere, it doesn't have to be served from the Datasette statics folder.

  I welcome ideas for more hooks, or feedback on the current design!

  ## Examples

  See the [example plugins file](https://github.com/simonw/datasette/blob/2d92b9328022d86505261bcdac419b6ed9cb2236/datasette/static/table-example-plugins.js) for additional examples.

  ## Hooks API Guide

  ### `makeAboveTablePanelConfigs`

  Provide a function with a list of panel objects. Each panel object should contain

  1. A unique string `id`
  2. A string `label` for the tab
  3. A `render` function. The first argument is reference to an HTML [Element](https://developer.mozilla.org/en-US/docs/Web/API/Element). 

  Example:

  ```js
   manager.registerPlugin("panel-plugin-graphs", {
      version: 0.1,
      makeAboveTablePanelConfigs: () => {
        return [
          {
            id: 'first-panel',
            label: "My new panel",
            render: node => {
              const description = document.createElement('p');
              description.innerText = 'Hello world';
              node.appendChild(description);
            }
          }
        ];
      },
    });
  ```

  ### `makeColumnActions`

  Provide a function that returns a list of action objects. Each action object has

  1. A string `label` for the menu dropdown label
  2. An onClick `render` function.

  Example:

  ```js
    manager.registerPlugin("column-name-plugin", {
      version: 0.1,
      getColumnActions: (columnMeta) => {
        
        // Info about selected column. 
        const { columnName, columnNotNull, columnType, isPk } = columnMeta;

        return [
          {
            label: "Copy name to clipboard",
            onClick: (evt) => copyToClipboard(column),
          }
        ];
      },
    });
  ```

  The getColumnActions callback has access to an object with metadata about the clicked column. These fields include:

  - columnName: string (name of the column)
  - columnNotNull: boolean
  - columnType: sqlite datatype enum (text, number, etc)
  - isPk: Whether this is the primary key: boolean

  You can use this column metadata to customize the action config objects (for example, handling different summaries for text vs number columns).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2052#issuecomment-1510423215
issue_url: https://api.github.com/repos/simonw/datasette/issues/2052
node_id: IC_kwDOBm6k_c5aBzqv
created_at: 2023-04-16T16:12:59Z
updated_at: 2023-04-16T16:12:59Z
author_association: CONTRIBUTOR
body:
  ## Research notes

  - I stuck to the "minimal dependencies" ethos of datasette (no React, Typescript, JS linting, etc).
  - Main threads on JS plugin development
    - Main: sketch of pluggy-inspired system: https://github.com/simonw/datasette/issues/983
    - Main: provide locations in Datasette HTML that are designed for multiple plugins to safely cooperate with each other (starting with the panel, but eventually could extend to "search boxes"): https://github.com/simonw/datasette/issues/1191
    - Main: HTML hooks for JS plugin authors: https://github.com/simonw/datasette/issues/987
  - Prior threads on JS plugins in Datasette for future design directions
    - Idea: pass useful strings to JS plugins: https://github.com/simonw/datasette/issues/1565
    - Idea: help with plugin dependency loading: https://github.com/simonw/datasette/issues/1542 . (IMO - the plugin providing the dependency can emit an event once it's done.  Other plugins can listen for it, or ask the manager to inform them when the dependency is available). 
    - Idea: help plugins to manage state in shareable URLs (plugins shouldn't have to interact with the URL directly, should have some basic insulation from clobbering each others' keys): https://github.com/simonw/datasette/issues/1144
  - Articles on plugins reviewed
    - https://css-tricks.com/designing-a-javascript-plugin-system/
  - Plugin/Extension systems reviewed (mostly JS).
    - Yarn: https://yarnpkg.com/advanced/plugin-tutorial
    - Tappable https://github.com/webpack/tapable (used by Auto, webpack)
    - Pluggy: https://pluggy.readthedocs.io/en/stable/
    - VSCode: https://code.visualstudio.com/api/get-started/your-first-extension
    - Chrome: https://developer.chrome.com/docs/extensions/reference/
    - Figma/Figjam Widget: https://www.figma.com/widget-docs/
    - Datadog Apps: [Programming Model](https://github.com/DataDog/apps/blob/master/docs/en/programming-model.md)
    - Storybook: https://storybook.js.org/docs/react/addons/addons-api

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2052#issuecomment-1515694393
issue_url: https://api.github.com/repos/simonw/datasette/issues/2052
node_id: IC_kwDOBm6k_c5aV6k5
created_at: 2023-04-20T04:25:55Z
updated_at: 2023-04-20T04:25:55Z
author_association: CONTRIBUTOR
body:
  Thanks for the thoughtful review and generous examples @asg017 ! I'll make the changes you suggested soon. Bonus thoughts inlined below.

  > comments

  These were very much appreciated, it's important to a plugin system that details like this feel right! I'll address them in batch later in the week. 

  > I know TypeScript can be a little controversial

  FWIW I am in favor of doing Typescript - I just wanted to keep the initial set of files in this PR as simple as possible to review. Really appreciate you scaffolding this initial set of types + I think it would be a welcome addition to maintain a set of types.d.ts files. 

  I'm entertaining the idea of writing the actual source code in Typescript as long as the compiled output is readable b/c it can be tricky to keep the types and plain JS files in sync. Curious if you have encountered projects that are good at preventing drift.

  > Maybe they should have more "action-y" names

  This is a great observation. I'm inclined towards something like `make*` or `build*` since to me `add*` make me think the thing the method is attached to is being mutated, but I agree that any of these may be clearer than the current `get*` setup. I'll go through and update these. 

  > Maybe we can make it easier to do pure-js datasette plugins?

  I really like this idea! It'll be easier to get contributors if they don't have to touch the python side at _all_. 

  > And then do the PERMITTED_VIEWS filtering in JS rather than Python.

  One cost of doing this is that pages that won't use the JS would still have to load the unused code (given that I'm not sending up anything complex like lazy loading). But hopefully the manager core size is close to negligible, and it won't be a big deal. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2065#issuecomment-1524616740
issue_url: https://api.github.com/repos/simonw/datasette/issues/2065
node_id: IC_kwDOBm6k_c5a384k
created_at: 2023-04-27T03:38:21Z
updated_at: 2023-04-27T03:38:21Z
author_association: OWNER
body:
  Tried this:

      rye install https://github.com/simonw/datasette/archive/refs/heads/main.zip

  But got this error:

      Error: Expected one of `@`, `(`, `<`, `=`, `>`, `~`, `!`, `;`, found `:`

  I instead downloaded that file and ran:

      rye install main.zip

  This worked! And now:
  ```
  ~/.rye/tools/main-zip/bin/datasette --version
  ```
  ```
  datasette, version 1.0a2
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2066#issuecomment-1524637376
issue_url: https://api.github.com/repos/simonw/datasette/issues/2066
node_id: IC_kwDOBm6k_c5a4B7A
created_at: 2023-04-27T03:50:19Z
updated_at: 2023-04-27T03:50:19Z
author_association: OWNER
body:
  Having trouble replicating this on my laptop. I tried a fresh virtual environment with fresh packages (in case this is a `httpx` change) but this passed:

      pytest tests/test_csv.py

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2066#issuecomment-1524648995
issue_url: https://api.github.com/repos/simonw/datasette/issues/2066
node_id: IC_kwDOBm6k_c5a4Ewj
created_at: 2023-04-27T03:56:42Z
updated_at: 2023-04-27T03:57:11Z
author_association: OWNER
body:
  I don't have 3.7 locally. Trying to install it with `pyenv`.

      brew install pyenv

  Then:

      pyenv install --list | grep 3.7

  Installing:

      pyenv install 3.7.16

  Output:

      Installed Python-3.7.16 to /Users/simon/.pyenv/versions/3.7.16

  So the executable is `/Users/simon/.pyenv/versions/3.7.16/bin/python`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2066#issuecomment-1524655203
issue_url: https://api.github.com/repos/simonw/datasette/issues/2066
node_id: IC_kwDOBm6k_c5a4GRj
created_at: 2023-04-27T03:59:56Z
updated_at: 2023-04-27T03:59:56Z
author_association: OWNER
body:
  In a fresh Datasette checkout I ran:

      pipenv shell --python /Users/simon/.pyenv/versions/3.7.16/bin/python

  That gave me a virtual environment with 3.7.16 Python.

  Then I ran:

      pip install -e '.[test]'

  Weirdly that gave me a `which pytest` of `/opt/homebrew/bin/pytest` which ran the tests on 3.11.

  I figured out the location of the virtual environment with `which python` and then ran this:

  ```
  % /Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/bin/pytest tests/test_csv.py 
  ============================================================================================== test session starts ===============================================================================================
  platform darwin -- Python 3.7.16, pytest-7.3.1, pluggy-1.0.0
  SQLite: 3.39.5
  rootdir: /private/tmp/datasette
  configfile: pytest.ini
  plugins: asyncio-0.21.0, timeout-2.1.0, xdist-3.2.1, anyio-3.6.2
  asyncio: mode=strict
  collected 15 items                                                                                                                                                                                               

  tests/test_csv.py ..........F....                                                                                                                                                                          [100%]

  ==================================================================================================== FAILURES ====================================================================================================
  ________________________________________________________________________________________________ test_max_csv_mb _________________________________________________________________________________________________

  app_client_csv_max_mb_one = <datasette.utils.testing.TestClient object at 0x107ab5c08>

      def test_max_csv_mb(app_client_csv_max_mb_one):
          response = app_client_csv_max_mb_one.get(
              (
                  "/fixtures.csv?sql=select+'{}'+"
                  "from+compound_three_primary_keys&_stream=1&_size=max"
  >           ).format("abcdefg" * 10000)
          )

  /private/tmp/datasette/tests/test_csv.py:161: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
  /Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/asgiref/sync.py:240: in __call__
      return call_result.result()
  /Users/simon/.pyenv/versions/3.7.16/lib/python3.7/concurrent/futures/_base.py:428: in result
      return self.__get_result()
  /Users/simon/.pyenv/versions/3.7.16/lib/python3.7/concurrent/futures/_base.py:384: in __get_result
      raise self._exception
  /Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/asgiref/sync.py:306: in main_wrap
      result = await self.awaitable(*args, **kwargs)
  /private/tmp/datasette/datasette/utils/testing.py:76: in get
      headers=headers,
  /private/tmp/datasette/datasette/utils/testing.py:167: in _request
      content=post_body,
  /private/tmp/datasette/datasette/app.py:1787: in request
      method, self._fix(path, avoid_path_rewrites), **kwargs
  /Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/httpx/_client.py:1528: in request
      extensions=extensions,
  /Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/httpx/_client.py:346: in build_request
      url = self._merge_url(url)
  /Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/httpx/_client.py:376: in _merge_url
      merge_url = URL(url)
  /Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/httpx/_urls.py:113: in __init__
      self._uri_reference = urlparse(url, **kwargs)
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

  url = "http://localhost/fixtures.csv?sql=select+'abcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcde...gabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefg'+from+compound_three_primary_keys&_stream=1&_size=max"
  kwargs = {}

      def urlparse(url: str = "", **kwargs: typing.Optional[str]) -> ParseResult:
          # Initial basic checks on allowable URLs.
          # ---------------------------------------
      
          # Hard limit the maximum allowable URL length.
          if len(url) > MAX_URL_LENGTH:
  >           raise InvalidURL("URL too long")
  E           httpx.InvalidURL: URL too long

  /Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/httpx/_urlparse.py:155: InvalidURL
  ============================================================================================ short test summary info =============================================================================================
  FAILED tests/test_csv.py::test_max_csv_mb - httpx.InvalidURL: URL too long
  ========================================================================================== 1 failed, 14 passed in 1.83s ==========================================================================================
  (datasette) simon@Simons-MacBook-Pro datasette % 
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2066#issuecomment-1524659084
issue_url: https://api.github.com/repos/simonw/datasette/issues/2066
node_id: IC_kwDOBm6k_c5a4HOM
created_at: 2023-04-27T04:02:07Z
updated_at: 2023-04-27T04:02:07Z
author_association: OWNER
body:
  This is the failing test:

  https://github.com/simonw/datasette/blob/249fcf8e3e2a90e763f41b080c1b9ec8017f5005/tests/test_csv.py#L156-L167

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2066#issuecomment-1524660603
issue_url: https://api.github.com/repos/simonw/datasette/issues/2066
node_id: IC_kwDOBm6k_c5a4Hl7
created_at: 2023-04-27T04:02:55Z
updated_at: 2023-04-27T04:02:55Z
author_association: OWNER
body:
  In the debugger:
  ```
  (Pdb) MAX_URL_LENGTH
  65536
  ```
  Weird this only seems to be a problem with `httpx` on Python 3.7 though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2066#issuecomment-1524666049
issue_url: https://api.github.com/repos/simonw/datasette/issues/2066
node_id: IC_kwDOBm6k_c5a4I7B
created_at: 2023-04-27T04:06:18Z
updated_at: 2023-04-27T04:06:18Z
author_association: OWNER
body:
  Most recent `httpx` release is 0.24 a couple of weeks ago. Here's what changed in that version: https://github.com/encode/httpx/compare/0.23.3...0.24.0

  It looks like that URL limit is new: https://github.com/encode/httpx/commit/57daabf673705954afa94686c0002801c93d31f3#diff-78d8d93b5dd4c77d99c3e2b46b7286ba71a8fd60e92d8bd4eee5fb200b4f87bfR149-R155

  ```python
  def urlparse(url: str = "", **kwargs: typing.Optional[str]) -> ParseResult:
      # Initial basic checks on allowable URLs.
      # ---------------------------------------

      # Hard limit the maximum allowable URL length.
      if len(url) > MAX_URL_LENGTH:
          raise InvalidURL("URL too long")
  ```
  https://github.com/encode/httpx/blob/32e25497a36e6222cc64a758c98275b450dac28d/httpx/_urlparse.py#L153-L155

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2066#issuecomment-1524669124
issue_url: https://api.github.com/repos/simonw/datasette/issues/2066
node_id: IC_kwDOBm6k_c5a4JrE
created_at: 2023-04-27T04:10:44Z
updated_at: 2023-04-27T04:10:52Z
author_association: OWNER
body:
  I need an alternative way of generating a long string with a shorter URL.

  ```sql
  select group_concat('abcabcabc', '') from json_each(json_array(1, 2, 3, 4, 5, 6, 7, 8, 9, 10))
  ```
  https://latest.datasette.io/_memory?sql=select+group_concat%28%27abcabcabc%27%2C+%27%27%29+from+json_each%28json_array%281%2C+2%2C+3%2C+4%2C+5%2C+6%2C+7%2C+8%2C+9%2C+10%29%29

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2066#issuecomment-1524675817
issue_url: https://api.github.com/repos/simonw/datasette/issues/2066
node_id: IC_kwDOBm6k_c5a4LTp
created_at: 2023-04-27T04:21:03Z
updated_at: 2023-04-27T04:21:03Z
author_association: OWNER
body:
  I went with this to generate the long string: https://github.com/simonw/datasette/blob/0b0c5cd7a94fe3f151a3e10261b5c84ee64f2f18/tests/test_csv.py#L157-L176

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2065#issuecomment-1524707628
issue_url: https://api.github.com/repos/simonw/datasette/issues/2065
node_id: IC_kwDOBm6k_c5a4TEs
created_at: 2023-04-27T05:06:44Z
updated_at: 2023-04-27T05:06:44Z
author_association: OWNER
body:
  I need `pip` as a dependency too:

  ```
  % ~/.rye/tools/main-zip/bin/datasette install datasette-graphql
  Traceback (most recent call last):
    File "/Users/simon/.rye/tools/main-zip/bin/datasette", line 8, in <module>
      sys.exit(cli())
               ^^^^^
    File "/Users/simon/.rye/tools/main-zip/lib/python3.11/site-packages/click/core.py", line 1130, in __call__
      return self.main(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/simon/.rye/tools/main-zip/lib/python3.11/site-packages/click/core.py", line 1055, in main
      rv = self.invoke(ctx)
           ^^^^^^^^^^^^^^^^
    File "/Users/simon/.rye/tools/main-zip/lib/python3.11/site-packages/click/core.py", line 1657, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/simon/.rye/tools/main-zip/lib/python3.11/site-packages/click/core.py", line 1404, in invoke
      return ctx.invoke(self.callback, **ctx.params)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/simon/.rye/tools/main-zip/lib/python3.11/site-packages/click/core.py", line 760, in invoke
      return __callback(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/simon/.rye/tools/main-zip/lib/python3.11/site-packages/datasette/cli.py", line 365, in install
      run_module("pip", run_name="__main__")
    File "<frozen runpy>", line 222, in run_module
    File "<frozen runpy>", line 142, in _get_module_details
  ImportError: No module named pip
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2065#issuecomment-1524709988
issue_url: https://api.github.com/repos/simonw/datasette/issues/2065
node_id: IC_kwDOBm6k_c5a4Tpk
created_at: 2023-04-27T05:09:36Z
updated_at: 2023-04-27T05:09:36Z
author_association: OWNER
body:
  That fixed it - after installing `main.zip` again I ran this and it worked:

      ~/.rye/tools/main-zip/bin/datasette install datasette-graphql

  ```
  % ~/.rye/tools/main-zip/bin/datasette plugins                  
  [
      {
          "name": "datasette-graphql",
          "static": true,
          "templates": true,
          "version": "2.2",
          "hooks": [
              "database_actions",
              "extra_template_vars",
              "menu_links",
              "register_routes",
              "startup",
              "table_actions"
          ]
      }
  ]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2052#issuecomment-1530817667
issue_url: https://api.github.com/repos/simonw/datasette/issues/2052
node_id: IC_kwDOBm6k_c5bPmyD
created_at: 2023-05-02T03:24:53Z
updated_at: 2023-05-02T03:24:53Z
author_association: CONTRIBUTOR
body:
  Thanks for putting this together! I've been slammed with work/personal stuff so haven't been able to actually prototype anything with this. :(

  tl;dr: I think this would be useful immediately as is. It might also be nice if the plugins could return `Promise`s.

  The long version: I read the design notes and example plugin. I think I'd be able to use this in [datasette-ui-extras](https://github.com/cldellow/datasette-ui-extras) for my lazy-facets feature.

  The lazy-facets feature tries to provide a snappier user experience. It does this by altering how suggested facets work.

  First, at page render time:
  (A) it lies to Datasette and claims that no columns support facets, this avoids the lengthy delays/timeouts that can happen if the dataset is large.
  (B) there's a python plugin that implements the [extra_body_script](https://docs.datasette.io/en/stable/plugin_hooks.html#extra-body-script-template-database-table-columns-view-name-request-datasette) hook, to write out the list of column names for future use by JavaScript

  Second, at page load time: there is some JavaScript that:
  (C) makes AJAX requests to suggest facets for each column - it makes 1 request per column, using the data from (B)
  (D) wires up the column menus to add Facet-by-this options for each facet

  With the currently proposed plugin scheme, I think (D) could be moved into the plugin. I'd do the ajax requests, then register the plugin.

  If the plugin scheme also supported promises, I think (B) and (C) could also be moved into the plugin.

  Does that make sense? Sorry for the wall of text!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2052#issuecomment-1530822437
issue_url: https://api.github.com/repos/simonw/datasette/issues/2052
node_id: IC_kwDOBm6k_c5bPn8l
created_at: 2023-05-02T03:35:30Z
updated_at: 2023-05-02T16:02:38Z
author_association: CONTRIBUTOR
body:
  Also, just checking - is this how I'd write bulletproof plugin registration code that is robust against the order in which the script tags load (eg if both my code and the Datasette code are loaded via a `<script async src='...'/>` tag)?

  ```js
  if (window.__DATASETTE__)
    go(window.__DATASETTE__);
  else
    document.addEventListener("datasette_init", (evt) => go(evt.detail));

  function go(manager) {
    manager.registerPlugin(...)
  }
  ```

  I don't know if it'd make sense, but you could also consider the asynchronous queuing pattern that Google Analytics uses (see [this Stack Overflow post](https://stackoverflow.com/questions/6963779/whats-the-name-of-google-analytics-async-design-pattern-and-where-is-it-used) for more details):

  ```js
  __DATASETTE__ = __DATASETTE__ || [];
  __DATASETTE__.push(go);

  function go(manager) {
    manager.registerPlugin(...);
  }
  ```

reactions:
  {"total_count": 2, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 1}


html_url: https://github.com/simonw/datasette/issues/2067#issuecomment-1532304714
issue_url: https://api.github.com/repos/simonw/datasette/issues/2067
node_id: IC_kwDOBm6k_c5bVR1K
created_at: 2023-05-03T00:16:03Z
updated_at: 2023-05-03T00:16:03Z
author_association: NONE
body:
  Curiously, after running commands on the database that was litestream-restored, datasette starts to work again, e.g.

  ```sh
  litestream restore -o data/db.sqlite s3://mytestbucketxx/db
  datasette data/db.sqlite
  # fails (OperationalError described above)
  ```

  ```sh
  litestream restore -o data/db.sqlite s3://mytestbucketxx/db
  sqlite-utils enable-wal data/db.sqlite
  datasette data/db.sqlite
  # works
  ```

  ```sh
  litestream restore -o data/db.sqlite s3://mytestbucketxx/db
  sqlite-utils optimize data/db.sqlite
  datasette data/db.sqlite
  # works
  ```

  ```sh
  litestream restore -o data/db.sqlite s3://mytestbucketxx/db
  sqlite3 data/db.sqlite ".clone test.db"
  datasette test.db
  # works
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/496#issuecomment-1532481862
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/496
node_id: IC_kwDOCGYnMM5bV9FG
created_at: 2023-05-03T05:53:26Z
updated_at: 2023-05-03T05:53:26Z
author_association: NONE
body:
  Would love to put our heads together for improvements here. 

  I _think_ anything that is `argname=DEFAULT` needs to be typed as `argname: str | Default = DEFAULT` (replacing `str` with the appropriate type(s)). We may be able to get clever and tie the types to that key in the `_defaults` dict (definitely possible in Typescript, but I'm less familiar with advanced python types). 

  Right now, all args are typed as `Default`, which means all callers get type errors. 

  As for table/view, given that they don't have the same methods, it would be nice to be able to get one or the other specifically.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2069#issuecomment-1537277919
issue_url: https://api.github.com/repos/simonw/datasette/issues/2069
node_id: IC_kwDOBm6k_c5boP_f
created_at: 2023-05-07T03:17:35Z
updated_at: 2023-05-07T03:17:35Z
author_association: NONE
body:
  Some updates:

  I notice that there is an option in the CLI where we can explicitly set `immutable` mode when spinning up the server

  ```console
  Options:
    -i, --immutable PATH            Database files to open in immutable mode
  ```

  Then, the question is - how can I disable immutable mode in the deployed instance on Vercel?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/539#issuecomment-1537507394
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/539
node_id: IC_kwDOCGYnMM5bpIBC
created_at: 2023-05-07T18:08:44Z
updated_at: 2023-05-07T18:08:44Z
author_association: OWNER
body:
  Prototype:
  ```diff
  diff --git a/docs/cli-reference.rst b/docs/cli-reference.rst
  index 153e5f9..c830518 100644
  --- a/docs/cli-reference.rst
  +++ b/docs/cli-reference.rst
  @@ -124,6 +124,7 @@ See :ref:`cli_query`.
         --json-cols                 Detect JSON cols and output them as JSON, not
                                     escaped strings
         -r, --raw                   Raw output, first column of first row
  +      --raw-lines                 Raw output, first column of each row
         -p, --param <TEXT TEXT>...  Named :parameters for SQL query
         --functions TEXT            Python code defining one or more custom SQL
                                     functions
  @@ -192,6 +193,7 @@ See :ref:`cli_memory`.
         --json-cols                 Detect JSON cols and output them as JSON, not
                                     escaped strings
         -r, --raw                   Raw output, first column of first row
  +      --raw-lines                 Raw output, first column of each row
         -p, --param <TEXT TEXT>...  Named :parameters for SQL query
         --encoding TEXT             Character encoding for CSV input, defaults to
                                     utf-8
  diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py
  index d25b1df..da0e4b6 100644
  --- a/sqlite_utils/cli.py
  +++ b/sqlite_utils/cli.py
  @@ -1653,6 +1653,7 @@ def drop_view(path, view, ignore, load_extension):
   )
   @output_options
   @click.option("-r", "--raw", is_flag=True, help="Raw output, first column of first row")
  +@click.option("--raw-lines", is_flag=True, help="Raw output, first column of each row")
   @click.option(
       "-p",
       "--param",
  @@ -1677,6 +1678,7 @@ def query(
       fmt,
       json_cols,
       raw,
  +    raw_lines,
       param,
       load_extension,
       functions,
  @@ -1700,7 +1702,19 @@ def query(
           _register_functions(db, functions)
   
       _execute_query(
  -        db, sql, param, raw, table, csv, tsv, no_headers, fmt, nl, arrays, json_cols
  +        db,
  +        sql,
  +        param,
  +        raw,
  +        raw_lines,
  +        table,
  +        csv,
  +        tsv,
  +        no_headers,
  +        fmt,
  +        nl,
  +        arrays,
  +        json_cols,
       )
   
   
  @@ -1728,6 +1742,7 @@ def query(
   )
   @output_options
   @click.option("-r", "--raw", is_flag=True, help="Raw output, first column of first row")
  +@click.option("--raw-lines", is_flag=True, help="Raw output, first column of each row")
   @click.option(
       "-p",
       "--param",
  @@ -1773,6 +1788,7 @@ def memory(
       fmt,
       json_cols,
       raw,
  +    raw_lines,
       param,
       encoding,
       no_detect_types,
  @@ -1879,12 +1895,36 @@ def memory(
           _register_functions(db, functions)
   
       _execute_query(
  -        db, sql, param, raw, table, csv, tsv, no_headers, fmt, nl, arrays, json_cols
  +        db,
  +        sql,
  +        param,
  +        raw,
  +        raw_lines,
  +        table,
  +        csv,
  +        tsv,
  +        no_headers,
  +        fmt,
  +        nl,
  +        arrays,
  +        json_cols,
       )
   
   
   def _execute_query(
  -    db, sql, param, raw, table, csv, tsv, no_headers, fmt, nl, arrays, json_cols
  +    db,
  +    sql,
  +    param,
  +    raw,
  +    raw_lines,
  +    table,
  +    csv,
  +    tsv,
  +    no_headers,
  +    fmt,
  +    nl,
  +    arrays,
  +    json_cols,
   ):
       with db.conn:
           try:
  @@ -1903,6 +1943,13 @@ def _execute_query(
                   sys.stdout.buffer.write(data)
               else:
                   sys.stdout.write(str(data))
  +        elif raw_lines:
  +            for row in cursor:
  +                data = row[0]
  +                if isinstance(data, bytes):
  +                    sys.stdout.buffer.write(data + b"\n")
  +                else:
  +                    sys.stdout.write(str(data) + "\n")
           elif fmt or table:
               print(
                   tabulate.tabulate(
  ```
  Needs tests and more documentation.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/539#issuecomment-1537507676
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/539
node_id: IC_kwDOCGYnMM5bpIFc
created_at: 2023-05-07T18:09:43Z
updated_at: 2023-05-07T18:09:54Z
author_association: OWNER
body:
  This worked:

  ```bash
  sqlite-utils memory /tmp/books3.json:nl \
    'select name from books3' --raw-lines > titles.txt
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/540#issuecomment-1537513653
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/540
node_id: IC_kwDOCGYnMM5bpJi1
created_at: 2023-05-07T18:37:59Z
updated_at: 2023-05-07T18:38:19Z
author_association: OWNER
body:
  Useful comment here:

  - https://github.com/urllib3/urllib3/issues/2168#issuecomment-1537360928

  > I faced the same issue. I switched to a different Python kernel (3.11.2) and it worked.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/540#issuecomment-1537513912
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/540
node_id: IC_kwDOCGYnMM5bpJm4
created_at: 2023-05-07T18:39:29Z
updated_at: 2023-05-07T18:39:29Z
author_association: OWNER
body:
  https://readthedocs.org/projects/sqlite-utils/builds/20513034/ said:

  > Problem in your project's configuration. Invalid "python.version": expected one of (2, 2.7, 3, 3.5, 3.6, 3.7, 3.8, pypy3.5), got 3.11

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/540#issuecomment-1537514069
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/540
node_id: IC_kwDOCGYnMM5bpJpV
created_at: 2023-05-07T18:40:18Z
updated_at: 2023-05-07T18:40:18Z
author_association: OWNER
body:
  https://docs.readthedocs.io/en/stable/config-file/v2.html suggests:

  ```yaml
  build:
    os: ubuntu-22.04
    tools:
      python: "3.11"
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/539#issuecomment-1537514610
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/539
node_id: IC_kwDOCGYnMM5bpJxy
created_at: 2023-05-07T18:43:24Z
updated_at: 2023-05-07T18:43:24Z
author_association: OWNER
body:
  Documentation:
  - https://sqlite-utils.datasette.io/en/latest/cli.html#returning-raw-data-such-as-binary-content
  - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#query
  - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#memory

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/540#issuecomment-1537744000
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/540
node_id: IC_kwDOCGYnMM5bqByA
created_at: 2023-05-08T04:56:12Z
updated_at: 2023-05-08T04:56:12Z
author_association: NONE
body:
  Hey @simonw, urllib3 maintainer here :wave:

  Sorry for breaking your CI. I understand you may prefer to pin the Python version, but note that specifying just `python: "3"` will get you the latest. We use that in urllib3: https://github.com/urllib3/urllib3/blob/main/.readthedocs.yml

  I can open PRs to sqlite-utils / datasette if you're interested

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538793817
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/538
node_id: IC_kwDOCGYnMM5buCFZ
created_at: 2023-05-08T17:55:10Z
updated_at: 2023-05-08T17:55:10Z
author_association: OWNER
body:
  Confirmed - I added this test and it fails:
  ```python
  def test_upsert_all_not_null(fresh_db):
      # https://github.com/simonw/sqlite-utils/issues/538
      fresh_db["comments"].upsert_all(
          [{"id": 1, "name": "Cleo"}],
          pk="id",
          not_null=["name"],
      )
      assert list(fresh_db["comments"].rows) == [{"id": 1, "name": "Cleo"}]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538801855
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/538
node_id: IC_kwDOCGYnMM5buEC_
created_at: 2023-05-08T18:00:17Z
updated_at: 2023-05-08T18:00:17Z
author_association: OWNER
body:
  From time in the debugger, after creating the table it ends up doing this:
  ```
  (Pdb) queries_and_params
  [
    ('INSERT OR IGNORE INTO [comments]([id]) VALUES(?);', [1]),
    ('UPDATE [comments] SET [name] = ? WHERE [id] = ?', ['Cleo', 1])
  ]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538887361
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/538
node_id: IC_kwDOCGYnMM5buY7B
created_at: 2023-05-08T19:01:20Z
updated_at: 2023-05-08T19:01:20Z
author_association: OWNER
body:
  Here's the problem:
  ```python
  import sqlite3
  db = sqlite3.connect(":memory:")
  db.execute('create table foo (id integer primary key, name not null)')
  db.execute('insert into foo (id) values (1)')
  ```
  Produces:
  ```
  IntegrityError: NOT NULL constraint failed: foo.name
  ```
  But this:
  ```python
  db.execute('insert or ignore into foo (id) values (1)')
  ```
  Completes without an exception.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538889482
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/538
node_id: IC_kwDOCGYnMM5buZcK
created_at: 2023-05-08T19:02:38Z
updated_at: 2023-05-08T19:02:38Z
author_association: OWNER
body:
  Here's the code at fault:
  https://github.com/simonw/sqlite-utils/blob/80763edaa2bdaf1113717378b8d62075c4dcbcfb/sqlite_utils/db.py#L2774-L2788

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538893329
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/538
node_id: IC_kwDOCGYnMM5buaYR
created_at: 2023-05-08T19:04:47Z
updated_at: 2023-05-08T19:04:47Z
author_association: OWNER
body:
  This feels like a fundamental flaw in the way upserts are implemented by `sqlite-utils`.

  One fix would be to switch to using the `UPSERT` feature in SQLite: https://www.sqlite.org/lang_UPSERT.html

  But...

  > UPSERT syntax was added to SQLite with version 3.24.0 (2018-06-04).

  I still want to support SQLite versions earlier than that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538903556
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/538
node_id: IC_kwDOCGYnMM5buc4E
created_at: 2023-05-08T19:11:24Z
updated_at: 2023-05-08T19:13:23Z
author_association: OWNER
body:
  I could detect if this happens using `cursor.rowcount` - not sure how I would recover from it though.

  This would also require some major re-engineering, since currently it all works by generating a list of SQL queries in advance and applying them inside a loop in `.insert_chunk()`:

  https://github.com/simonw/sqlite-utils/blob/80763edaa2bdaf1113717378b8d62075c4dcbcfb/sqlite_utils/db.py#L2839-L2878

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538910894
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/538
node_id: IC_kwDOCGYnMM5buequ
created_at: 2023-05-08T19:16:52Z
updated_at: 2023-05-08T19:17:00Z
author_association: OWNER
body:
  How about if I had logic which checked that all not-null columns were provided in the call to `upsert_all()` - and if they were, modified the `INSERT OR IGNORE INTO` to include a placeholder value for those columns that would then be fixed by the later `UPDATE`?

  Something like this:

  ```python
  [
    ('INSERT OR IGNORE INTO [comments]([id], name) VALUES(?, ?);', [1, '']),
    ('UPDATE [comments] SET [name] = ? WHERE [id] = ?', ['Cleo', 1])
  ]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/534#issuecomment-1538933540
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/534
node_id: IC_kwDOCGYnMM5bukMk
created_at: 2023-05-08T19:34:37Z
updated_at: 2023-05-08T19:34:37Z
author_association: OWNER
body:
  On macOS this shows the same warning:
  ```
  % python -Wdefault $(which sqlite-utils) insert dogs.db dogs dogs.csv --csv
    [############------------------------]   35%
    [####################################]  100%/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py:1187: ResourceWarning: unclosed file <_io.TextIOWrapper name='dogs.csv' encoding='utf-8-sig'>
    insert_upsert_implementation(
  ResourceWarning: Enable tracemalloc to get the object allocation traceback
  ```
  The file itself is a `click.File` which is automatically closed - https://click.palletsprojects.com/en/8.1.x/api/#click.File - but it looks like it's the `_io.TextIOWrapper` which is not being closed:

  https://github.com/simonw/sqlite-utils/blob/2376c452a56b0c3e75e7ca698273434e32945304/sqlite_utils/cli.py#L949-L956

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/541#issuecomment-1538963959
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/541
node_id: IC_kwDOCGYnMM5burn3
created_at: 2023-05-08T19:59:34Z
updated_at: 2023-05-08T19:59:34Z
author_association: OWNER
body:
  There are 8 failing tests left:

  ```
  ==== short test summary info ====
  FAILED tests/test_cli_memory.py::test_memory_csv[False-test] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>
  FAILED tests/test_cli_memory.py::test_memory_csv[False-t] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>
  FAILED tests/test_cli_memory.py::test_memory_csv[False-t1] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>
  FAILED tests/test_cli_memory.py::test_memory_tsv[False] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>
  FAILED tests/test_cli_memory.py::test_memory_dump[extra_args0] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>
  FAILED tests/test_cli_memory.py::test_memory_two_files_with_same_stem - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>
  FAILED tests/test_recipes.py::test_dateparse_errors[None-parsedate] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <function Table.convert.<locals>.convert_value at 0x106bcaca0>
  FAILED tests/test_recipes.py::test_dateparse_errors[None-parsedatetime] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <function Table.convert.<locals>.convert_value at 0x106bc9620>
  ERROR tests/test_cli.py::test_upsert_analyze - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>
  ==== 8 failed, 894 passed, 4 skipped, 1 error in 6.27s ====
  ```
  Full traceback here: https://gist.github.com/simonw/b40b3e814729d6c02a0302a84ce54d9e

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/532#issuecomment-1539009453
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/532
node_id: IC_kwDOCGYnMM5bu2ut
created_at: 2023-05-08T20:30:29Z
updated_at: 2023-05-08T20:30:42Z
author_association: OWNER
body:
  Here's an improvement:
  ```
  % sqlite-utils insert /tmp/b.db blah /tmp/blah.txt
    [####################################]  100%
  Error: Invalid JSON - use --csv for CSV or --tsv for TSV files

  JSON error: Expecting value: line 1 column 1 (char 0)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/530#issuecomment-1539015064
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/530
node_id: IC_kwDOCGYnMM5bu4GY
created_at: 2023-05-08T20:35:07Z
updated_at: 2023-05-08T20:35:07Z
author_association: OWNER
body:
  Wow, this is a neat feature I didn't know about. Looks like there are a bunch of options:

  - NO ACTION (default)
  - RESTRICT: application is prohibited from deleting a parent key when there exists one or more child keys mapped to it
  - SET NULL: when a parent key is deleted the child key columns of all rows in the child table that mapped to the parent key are set to contain SQL NULL values
  - SET DEFAULT: set a specific default
  - CASCADE: propagates the delete or update operation on the parent key to each dependent child key

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/530#issuecomment-1539018912
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/530
node_id: IC_kwDOCGYnMM5bu5Cg
created_at: 2023-05-08T20:39:00Z
updated_at: 2023-05-08T20:39:00Z
author_association: OWNER
body:
  I think the natural place to add these in the Python library API would be https://sqlite-utils.datasette.io/en/stable/python-api.html#adding-foreign-key-constraints - maybe something like this:

  ```python
  db["books"].add_foreign_key("author_id", "authors", "id", on_delete=RESTRICT)
  ```

  Then for the CLI tool could be added to https://sqlite-utils.datasette.io/en/stable/cli-reference.html#add-foreign-key

  ```
  sqlite-utils add-foreign-key my.db books author_id authors id --on-update SET_NULL
  ```
  I wouldn't support these for the other methods of adding foreign keys - `foreign_keys(...)` for the various `.insert()` etc methods and the `add_foreign_keys(...)` bulk menthod.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/527#issuecomment-1539033736
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/527
node_id: IC_kwDOCGYnMM5bu8qI
created_at: 2023-05-08T20:52:51Z
updated_at: 2023-05-08T20:52:51Z
author_association: OWNER
body:
  OK, I implemented that at the Python API level. I need to decide how it should work for the `sqlite-utils convert` command too: https://sqlite-utils.datasette.io/en/stable/cli-reference.html#convert

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/527#issuecomment-1539035838
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/527
node_id: IC_kwDOCGYnMM5bu9K-
created_at: 2023-05-08T20:55:00Z
updated_at: 2023-05-08T20:55:00Z
author_association: OWNER
body:
  I'm going to go with `--no-skip-false` as the CLI option. It's ugly, but this whole thing is ugly. I'm going to make a note to remove this misfeature in `sqlite-utils` 4.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/527#issuecomment-1539051724
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/527
node_id: IC_kwDOCGYnMM5bvBDM
created_at: 2023-05-08T21:07:04Z
updated_at: 2023-05-08T21:07:04Z
author_association: OWNER
body:
  Updated documentation:
  - https://sqlite-utils.datasette.io/en/latest/python-api.html#converting-data-in-columns
  - https://sqlite-utils.datasette.io/en/latest/cli.html#converting-data-in-columns
  - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#convert

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/542#issuecomment-1539052467
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/542
node_id: IC_kwDOCGYnMM5bvBOz
created_at: 2023-05-08T21:07:41Z
updated_at: 2023-05-08T21:07:41Z
author_association: OWNER
body:
  Relevant commits (will mostly revert these):
  - https://github.com/simonw/sqlite-utils/commit/455c35b512895c19bf922c2b804d750d27cb8dbd
  - https://github.com/simonw/sqlite-utils/commit/e0ec4c345129996011951e400388fd74865f65a2

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/515#issuecomment-1539077777
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/515
node_id: IC_kwDOCGYnMM5bvHaR
created_at: 2023-05-08T21:27:10Z
updated_at: 2023-05-08T21:27:10Z
author_association: OWNER
body:
  I should have spotted this PR before I shipped my own fix! https://github.com/simonw/sqlite-utils/commit/2376c452a56b0c3e75e7ca698273434e32945304

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/514#issuecomment-1539079507
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/514
node_id: IC_kwDOCGYnMM5bvH1T
created_at: 2023-05-08T21:28:37Z
updated_at: 2023-05-08T21:28:37Z
author_association: OWNER
body:
  > This means that a table with NON NULL (or other constraint) columns that aren't part of the pkey can't have new rows upserted.

  Huh... on that basis, it's possible my fix in https://github.com/simonw/sqlite-utils/commit/2376c452a56b0c3e75e7ca698273434e32945304 is incomplete. I only covered the 'not null' case.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/514#issuecomment-1539094287
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/514
node_id: IC_kwDOCGYnMM5bvLcP
created_at: 2023-05-08T21:44:11Z
updated_at: 2023-05-08T21:44:11Z
author_association: OWNER
body:
  OK, this fails silently:
  ```python
  import sqlite_utils
  db = sqlite_utils.Database(memory=True)
  db.execute('''CREATE TABLE employees (
      id INTEGER PRIMARY KEY,
      name TEXT,
      age INTEGER,
      salary REAL,
      CHECK (salary is not null and salary > 0)
  );''')
  db["employees"].upsert({"id": 1, "name": "Bob"}, pk="id")
  list(db["employees"].rows)
  ````
  It outputs:
  ```python
  []
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/514#issuecomment-1539099703
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/514
node_id: IC_kwDOCGYnMM5bvMw3
created_at: 2023-05-08T21:50:06Z
updated_at: 2023-05-08T21:50:06Z
author_association: OWNER
body:
  Applying the fix from the PR here doesn't fix the above problem either:
  - https://github.com/simonw/sqlite-utils/pull/515

  So it looks like these kinds of `check` constraints currently aren't compatible with how `upsert()` works.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/514#issuecomment-1539100300
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/514
node_id: IC_kwDOCGYnMM5bvM6M
created_at: 2023-05-08T21:50:51Z
updated_at: 2023-05-08T21:50:51Z
author_association: OWNER
body:
  Seeing as `sqlite-utils` doesn't currently provide mechanisms for adding `check` constraints like this I'm going to leave this - I'm happy with the fix I put in for the `not null` constraints.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/537#issuecomment-1539157643
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/537
node_id: IC_kwDOCGYnMM5bva6L
created_at: 2023-05-08T22:45:09Z
updated_at: 2023-05-08T22:45:21Z
author_association: OWNER
body:
  Here's an example from the new tests:

  https://github.com/simonw/sqlite-utils/blob/a75abeb61b91a28650d3b9933e7ec80ad0d92529/tests/test_create.py#L291-L307

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2070#issuecomment-1540494121
issue_url: https://api.github.com/repos/simonw/datasette/issues/2070
node_id: IC_kwDOBm6k_c5b0hMp
created_at: 2023-05-09T16:25:00Z
updated_at: 2023-05-09T16:25:00Z
author_association: OWNER
body:
  Can now be used here: https://github.com/simonw/datasette/actions/workflows/deploy-branch-preview.yml

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/527#issuecomment-1540900733
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/527
node_id: IC_kwDOCGYnMM5b2Ed9
created_at: 2023-05-09T21:15:05Z
updated_at: 2023-05-09T21:15:05Z
author_association: CONTRIBUTOR
body:
  Sorry, I completely missed your first comment whilst on Easter break.

  This looks like a good practical compromise before v4. Thanks!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2073#issuecomment-1546117538
issue_url: https://api.github.com/repos/simonw/datasette/issues/2073
node_id: IC_kwDOBm6k_c5cJ-Gi
created_at: 2023-05-12T18:21:38Z
updated_at: 2023-05-12T18:21:38Z
author_association: OWNER
body:
  https://latest.datasette.io/fixtures doesn't currently have a view with any integer columns in it, making this bug harder to demonstrate there.

  I can't replicate the bug using https://datasette.io/content/plugins?_facet=stargazers_count&stargazers_count=3 - I would expect that not to work correctly.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2073#issuecomment-1546119773
issue_url: https://api.github.com/repos/simonw/datasette/issues/2073
node_id: IC_kwDOBm6k_c5cJ-pd
created_at: 2023-05-12T18:24:07Z
updated_at: 2023-05-12T18:24:07Z
author_association: OWNER
body:
  Here's a demo of this breaking in Datasette Lite:

  https://lite.datasette.io/?sql=https://gist.github.com/simonw/261564c0ca01567df6eeb9b222b8be84&json=https%3A%2F%2Fcdn.jsdelivr.net%2Fnpm%2Fweb-features%2Findex.json#/data/baseline?_filter_column_1=is_baseline&_filter_op_1=exact&_filter_value_1=1&_filter_column_2=&_filter_op_2=notnull__1&_filter_value_2=1&_filter_column=&_filter_op=exact&_filter_value=&_sort=&_facet=is_baseline

  <img width="609" alt="image" src="https://github.com/simonw/datasette/assets/9599/3c23ae6e-7fb0-41d4-92b8-421c400a731a">

  Here's a SQL query that demonstrates the underlying issue:

  ```sql
  select 'working', count(*) from baseline where is_baseline = 1
  union all
  select 'broken', count(*) from baseline where is_baseline = '1'
  ```
  https://lite.datasette.io/?sql=https://gist.github.com/simonw/261564c0ca01567df6eeb9b222b8be84&json=https%3A%2F%2Fcdn.jsdelivr.net%2Fnpm%2Fweb-features%2Findex.json#/data?sql=select+%27working%27%2C+count%28*%29+from+baseline+where+is_baseline+%3D+1%0Aunion+all%0Aselect+%27broken%27%2C+count%28*%29+from+baseline+where+is_baseline+%3D+%271%27

  <img width="673" alt="image" src="https://github.com/simonw/datasette/assets/9599/867914f8-8cef-432c-a518-2739d1caf964">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2052#issuecomment-1546362374
issue_url: https://api.github.com/repos/simonw/datasette/issues/2052
node_id: IC_kwDOBm6k_c5cK54G
created_at: 2023-05-12T22:09:03Z
updated_at: 2023-05-12T22:09:03Z
author_association: CONTRIBUTOR
body:
  Hey @cldellow , thanks for the thoughtful feedback and describing the "lazy facets" feature! 

  It sounds like the [postTask](https://developer.mozilla.org/en-US/docs/Web/API/Scheduler/postTask) API might be relevant for the types of network request scheduling you have in mind. 

  Addressing your points inline below:

  > It might also be nice if the plugins could return Promises.

  Were you picturing that the whole plugin config object could be returned as a promise, or that the individual hooks (like `makeColumnActions` or `makeAboveTablePanelConfigs` supported returning a promise of arrays instead only returning plain arrays?

  I think what you're describing can be achievable, but I want to make sure I do so in a way that addresses your need / keeps the complexity of the plugin core system at a level this is approachable . 

  I have a hunch that what you're describing might be achievable without adding Promises to the API with something like

  ```
  fetch('/api/with-custom-facets').then(myFacets => {
      // reusing the go() idiom
     go(manager, myFacets);
  })
  ```

  but I'd like to confirm if that's the case before investigating adding support.

  > bulletproof plugin registration code that is robust against the order in which the script tags load

  Yes, I think what you wrote looks right to me! While it looks a little bit verbose compared to the second example, I'm hoping we can mitigate the cost of that during this API incubation phase by making it an easy-to-copy paste code snippet.

  I haven't heard of the GA queing pattern before, thanks for the example. I won't have time to implement of proof of concept in the next few weeks, but I took some time to think through the pros/cons to decide whether we may want to add this in a future release:

  I can see that this approach brings advantages

  - Plugin developers don't need to know the name of the datasette initialization event to start their plugin
  - Pushing a function to an array probably is easier (definitely more concise) than adding a document event listener
  - One less event listener sitting in memory

  It also has some minor costs

  - A malicious plugin could choose to (or accidentally) mess with the order of the queue if multiple scripts are lined up
  - Some risk in encouraging people to mutate global state
  - (not a cost, more a moot point): changing this API may not make a meaningful difference if we're discussing whether people enter 2 vs 5 lines of code, especially if those lines are encapsulated by a function we provide (maybe something that's available on the `window` provided by Datasette as an inline script tag). 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2075#issuecomment-1547944971
issue_url: https://api.github.com/repos/simonw/datasette/issues/2075
node_id: IC_kwDOBm6k_c5cQ8QL
created_at: 2023-05-15T14:12:20Z
updated_at: 2023-05-15T14:12:20Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2075?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch and project coverage have no change.
  > Comparison is base [(`49184c5`)](https://app.codecov.io/gh/simonw/datasette/commit/49184c569cd70efbda4f3f062afef3a34401d8d5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.40% compared to head [(`b99e1d3`)](https://app.codecov.io/gh/simonw/datasette/pull/2075?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.40%.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main    #2075   +/-   ##
  =======================================
    Coverage   92.40%   92.40%           
  =======================================
    Files          38       38           
    Lines        5751     5751           
  =======================================
    Hits         5314     5314           
    Misses        437      437           
  ```




  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2075?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2052#issuecomment-1548617257
issue_url: https://api.github.com/repos/simonw/datasette/issues/2052
node_id: IC_kwDOBm6k_c5cTgYp
created_at: 2023-05-15T21:32:20Z
updated_at: 2023-05-15T21:32:20Z
author_association: CONTRIBUTOR
body:
  > Were you picturing that the whole plugin config object could be returned as a promise, or that the individual hooks (like makeColumnActions or makeAboveTablePanelConfigs supported returning a promise of arrays instead only returning plain arrays?

  The latter - that you could return a promise of arrays, so it parallels the ["await me maybe" pattern in Datasette](https://simonwillison.net/2020/Sep/2/await-me-maybe/), where you can return either a value, a callable or an awaitable.

  > I have a hunch that what you're describing might be achievable without adding Promises to the API with something

  Oops, I did a poor job explaining. Yes, this would work - but it requires me to continue to communicate the column names out of band (in order to fetch the facet data per-column before registering my plugin), vs being able to re-use them from the plugin implementation.

  This isn't that big of a deal - it'd be a nice ergonomic improvement, but nowhere near as a big of an improvement as having an officially sanctioned way to add stuff to the column menus in the first place.

  This could also be layered on in a future commit without breaking v1 users, too, so it's not at all urgent.

  > especially if those lines are encapsulated by a function we provide (maybe something that's available on the window provided by Datasette as an inline script tag

  Ah, this is maybe the the key point. Since it's all hosted inside Datasette, Datasette can provide some arbitrary sugar to make it easier to work with.

  My experience with async scripts in JS is that people sometimes don't understand the race conditions inherent to them. If they copy/paste from a tutorial, it does just work. But then they'll delete half the code, and by chance it still works on their machine/Datasette templates, and now someone's headed for an annoying debugging session -- maybe them, maybe someone else who tries to re-use their plugin.

  Again, a fairly minor thing, though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1548913065
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/399
node_id: IC_kwDOCGYnMM5cUomp
created_at: 2023-05-16T03:11:03Z
updated_at: 2023-05-16T03:11:52Z
author_association: NONE
body:
  Using this thread and some [other resources](https://sqlite-utils.datasette.io/en/stable/cli.html#spatialite-helpers) I managed to cobble together a couple of sqlite-utils lines to add a geometry column for a table that already has a lat/lng column.

  ```
  # add a geometry column
  sqlite-utils add-geometry-column [db name] [table name] geometry --type POINT --srid 4326

  # add a point for each row to geometry column
  sqlite-utils --load-extension=spatialite [db name] 'update [table name] SET Geometry=MakePoint(longitude, latitude, 4326);'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556189823
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/545
node_id: IC_kwDOCGYnMM5cwZJ_
created_at: 2023-05-21T14:09:59Z
updated_at: 2023-05-21T14:09:59Z
author_association: OWNER
body:
  I don't want to add `trogon` as a default dependency because it's a little heavy - it pulls in all of Rich and Textual as well. People who use `sqlite-utils` just for its Python API won't benefit from this - it's a CLI feature only.

  But I have a `sqlite-utils install ...` command for helping people to install packages into the same virtual environment as `sqlite-utils` no matter how they installed that tool: https://sqlite-utils.datasette.io/en/stable/cli.html#cli-install

  So I can treat Trogon as an optional dependency and add the `sqlite-utils tui` command only if that package is also installed.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556190531
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/545
node_id: IC_kwDOCGYnMM5cwZVD
created_at: 2023-05-21T14:13:43Z
updated_at: 2023-05-21T14:13:43Z
author_association: OWNER
body:
  OK, this works!

  ![trogon](https://github.com/simonw/sqlite-utils/assets/9599/2ae194c5-ec82-471a-9d1b-b01b3f2632f3)

  To try it out, install that branch from GitHub:

      pip install https://github.com/simonw/sqlite-utils/archive/refs/heads/trogon.zip

  Then run this:

      sqlite-utils install trogon

  And this:

      sqlite-utils tui

reactions:
  {"total_count": 5, "+1": 2, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 3, "eyes": 0}


html_url: https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556191894
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/545
node_id: IC_kwDOCGYnMM5cwZqW
created_at: 2023-05-21T14:20:14Z
updated_at: 2023-05-21T14:20:14Z
author_association: OWNER
body:
  Opened a feature request for customizing the help and command name:

  - https://github.com/Textualize/trogon/issues/2

reactions:
  {"total_count": 2, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 1, "rocket": 0, "eyes": 0}


html_url: https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556210844
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/545
node_id: IC_kwDOCGYnMM5cweSc
created_at: 2023-05-21T15:44:10Z
updated_at: 2023-05-21T15:44:10Z
author_association: OWNER
body:
  It looks like `nargs=-1` on a positional argument isn't yet supported - opened an issue here:
  - https://github.com/Textualize/trogon/issues/4

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/544#issuecomment-1556211643
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/544
node_id: IC_kwDOCGYnMM5cwee7
created_at: 2023-05-21T15:48:17Z
updated_at: 2023-05-21T15:48:17Z
author_association: OWNER
body:
  I generated the commit message in https://github.com/simonw/sqlite-utils/commit/1c1991b447a1ddd3d61d9d4a8a1d6a9da47ced20 using `git diff | llm --system 'describe this change'`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/546#issuecomment-1556213031
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/546
node_id: IC_kwDOCGYnMM5cwe0n
created_at: 2023-05-21T15:56:05Z
updated_at: 2023-05-21T16:18:03Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/546?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch coverage: **`93.75`**% and no project coverage change.
  > Comparison is base [(`b3b100d`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/b3b100d7f5b2a76ccd4bfe8b0301a29e321d0375?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.30% compared to head [(`9f23e68`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/546?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.31%.

  > :exclamation: Current head 9f23e68 differs from pull request most recent head 2eca17d. Consider uploading reports for the commit 2eca17d to get more accurate results

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main     #546   +/-   ##
  =======================================
    Coverage   96.30%   96.31%           
  =======================================
    Files           6        6           
    Lines        2707     2712    +5     
  =======================================
  + Hits         2607     2612    +5     
    Misses        100      100           
  ```


  | [Impacted Files](https://app.codecov.io/gh/simonw/sqlite-utils/pull/546?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [sqlite\_utils/db.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/546?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.37% <90.90%> (+<0.01%)` | :arrow_up: |
  | [sqlite\_utils/cli.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/546?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.26% <100.00%> (+0.01%)` | :arrow_up: |


  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/546?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/546#issuecomment-1556213396
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/546
node_id: IC_kwDOCGYnMM5cwe6U
created_at: 2023-05-21T15:58:12Z
updated_at: 2023-05-21T16:18:46Z
author_association: OWNER
body:
  Documentation preview:
  - https://sqlite-utils--546.org.readthedocs.build/en/546/cli.html#cli-analyze-tables
  - https://sqlite-utils--546.org.readthedocs.build/en/546/cli-reference.html#analyze-tables
  - https://sqlite-utils--546.org.readthedocs.build/en/546/python-api.html#analyzing-a-column
  - https://sqlite-utils--546.org.readthedocs.build/en/546/reference.html#sqlite_utils.db.Table.analyze_column

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/544#issuecomment-1556225788
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/544
node_id: IC_kwDOCGYnMM5cwh78
created_at: 2023-05-21T17:02:05Z
updated_at: 2023-05-21T17:02:05Z
author_association: OWNER
body:
  New docs:

  - https://sqlite-utils.datasette.io/en/latest/cli.html#cli-analyze-tables
  - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#analyze-tables
  - https://sqlite-utils.datasette.io/en/latest/python-api.html#analyzing-a-column
  - https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Table.analyze_column

  New help output:

  ```
   % sqlite-utils analyze-tables --help
  Usage: sqlite-utils analyze-tables [OPTIONS] PATH [TABLES]...

    Analyze the columns in one or more tables

    Example:

        sqlite-utils analyze-tables data.db trees

  Options:
    -c, --column TEXT       Specific columns to analyze
    --save                  Save results to _analyze_tables table
    --common-limit INTEGER  How many common values
    --no-most               Skip most common values
    --no-least              Skip least common values
    --load-extension TEXT   Path to SQLite extension, with optional :entrypoint
    -h, --help              Show this message and exit.
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/547#issuecomment-1556228395
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/547
node_id: IC_kwDOCGYnMM5cwikr
created_at: 2023-05-21T17:11:15Z
updated_at: 2023-05-21T17:11:15Z
author_association: OWNER
body:
  This will be a cosmetic change to the CLI output only - the options to save data to the database and the Python API function will continue to return `[(None, 158)]`.

  I can add an optimization though to avoid running the SQL count query if we know that it's all `null`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/548#issuecomment-1556231832
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/548
node_id: IC_kwDOCGYnMM5cwjaY
created_at: 2023-05-21T17:24:13Z
updated_at: 2023-05-21T17:24:13Z
author_association: OWNER
body:
  Oh, I see why that is now:

  https://github.com/simonw/sqlite-utils/blob/6027f3ea6939a399aeef2578fca17efec0e539df/sqlite_utils/cli.py#L2670-L2679

  This is because of the following command:

      sqlite-utils analyze-tables table1 table2 --column x

  Since you can pass multiple tables AND multiple columns, the tool currently assumes that the column(s) you specify may be available on a subset of the provided tables.

  I'm going to change this so if the column is not on ANY of those tables you get an error.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/549#issuecomment-1556241555
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/549
node_id: IC_kwDOCGYnMM5cwlyT
created_at: 2023-05-21T17:57:24Z
updated_at: 2023-05-21T18:28:44Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/549?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch coverage: **`83.33`**% and project coverage change: **`+0.06`** :tada:
  > Comparison is base [(`b3b100d`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/b3b100d7f5b2a76ccd4bfe8b0301a29e321d0375?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.30% compared to head [(`948692a`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/549?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.36%.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@            Coverage Diff             @@
  ##             main     #549      +/-   ##
  ==========================================
  + Coverage   96.30%   96.36%   +0.06%     
  ==========================================
    Files           6        6              
    Lines        2707     2726      +19     
  ==========================================
  + Hits         2607     2627      +20     
  + Misses        100       99       -1     
  ```


  | [Impacted Files](https://app.codecov.io/gh/simonw/sqlite-utils/pull/549?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [sqlite\_utils/cli.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/549?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.22% <83.33%> (-0.03%)` | :arrow_down: |

  ... and [1 file with indirect coverage changes](https://app.codecov.io/gh/simonw/sqlite-utils/pull/549/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)


  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/549?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/549#issuecomment-1556242262
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/549
node_id: IC_kwDOCGYnMM5cwl9W
created_at: 2023-05-21T18:00:05Z
updated_at: 2023-05-21T18:00:05Z
author_association: OWNER
body:
  Failing `mypy` test: https://github.com/simonw/sqlite-utils/actions/runs/5038983349/jobs/9036828465
  ```
  sqlite_utils/cli.py:37: error: Skipping analyzing "trogon": module is installed, but missing library stubs or py.typed marker  [import]
  sqlite_utils/cli.py:37: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
  Found 1 error in 1 file (checked 52 source files)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/550#issuecomment-1556250236
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/550
node_id: IC_kwDOCGYnMM5cwn58
created_at: 2023-05-21T18:25:26Z
updated_at: 2023-05-21T18:25:26Z
author_association: OWNER
body:
  Relevant issues:
  - https://github.com/python/importlib_metadata/issues/406
  - https://github.com/PyCQA/flake8/issues/1701

  It looks to me like this is only a problem for `flake8` on Python 3.7 - 3.8 and higher work OK.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556262574
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/551
node_id: IC_kwDOCGYnMM5cwq6u
created_at: 2023-05-21T19:04:59Z
updated_at: 2023-05-21T19:04:59Z
author_association: OWNER
body:
  I wrote the docs like this because early examples include both the command and its output:

  https://sqlite-utils.datasette.io/en/stable/cli.html#returning-json

  <img width="766" alt="image" src="https://github.com/simonw/sqlite-utils/assets/9599/a5d05f24-ca0f-4718-a0a8-d85adc736141">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556263182
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/551
node_id: IC_kwDOCGYnMM5cwrEO
created_at: 2023-05-21T19:06:48Z
updated_at: 2023-05-21T19:06:48Z
author_association: OWNER
body:
  I could split them up into two blocks like this:

  <img width="771" alt="image" src="https://github.com/simonw/sqlite-utils/assets/9599/9b042ac8-db30-49d5-8882-1ea7be272a49">

  I do miss the visual indication that one of these is the command and one is the output though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556265772
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/551
node_id: IC_kwDOCGYnMM5cwrss
created_at: 2023-05-21T19:16:15Z
updated_at: 2023-05-21T19:16:15Z
author_association: OWNER
body:
  Another option:

  <img width="593" alt="image" src="https://github.com/simonw/sqlite-utils/assets/9599/a9d819a0-89e3-4b67-b2f0-1052b0a7a7c0">

  That's using this markup:

  ```
  Newline-delimited JSON
  ~~~~~~~~~~~~~~~~~~~~~~

  Use ``--nl`` to get back newline-delimited JSON objects:

  .. code-block:: bash

      sqlite-utils dogs.db "select * from dogs" --nl

  .. code-block:: output

      {"id": 1, "age": 4, "name": "Cleo"}
      {"id": 2, "age": 2, "name": "Pancakes"}
  ```
  And this extra CSS:

  ```css
  .highlight-output .highlight {
    border-left: 9px solid #30c94f;
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/553#issuecomment-1556287870
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/553
node_id: IC_kwDOCGYnMM5cwxF-
created_at: 2023-05-21T20:45:58Z
updated_at: 2023-05-21T20:57:08Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/553?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch and project coverage have no change.
  > Comparison is base [(`e240133`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/e240133b11588d31dc22c632f7a7ca636c72978d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.36% compared to head [(`0b81794`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/553?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.36%.

  > :exclamation: Current head 0b81794 differs from pull request most recent head 21036a5. Consider uploading reports for the commit 21036a5 to get more accurate results

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main     #553   +/-   ##
  =======================================
    Coverage   96.36%   96.36%           
  =======================================
    Files           6        6           
    Lines        2726     2726           
  =======================================
    Hits         2627     2627           
    Misses         99       99           
  ```




  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/553?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556288270
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/551
node_id: IC_kwDOCGYnMM5cwxMO
created_at: 2023-05-21T20:47:51Z
updated_at: 2023-05-21T20:47:51Z
author_association: OWNER
body:
  This page has all of the changes: https://sqlite-utils--553.org.readthedocs.build/en/553/cli.html#running-sql-queries

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/553#issuecomment-1556288300
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/553
node_id: IC_kwDOCGYnMM5cwxMs
created_at: 2023-05-21T20:48:01Z
updated_at: 2023-05-21T20:48:01Z
author_association: OWNER
body:
  If https://sqlite-utils--553.org.readthedocs.build/en/553/cli.html#running-sql-queries looks good I can merge this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/552#issuecomment-1556292204
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/552
node_id: IC_kwDOCGYnMM5cwyJs
created_at: 2023-05-21T21:05:15Z
updated_at: 2023-05-21T21:05:15Z
author_association: OWNER
body:
  Now live at https://sqlite-utils.datasette.io/en/latest/installation.html#setting-up-shell-completion

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2077#issuecomment-1557289070
issue_url: https://api.github.com/repos/simonw/datasette/issues/2077
node_id: IC_kwDOBm6k_c5c0lhu
created_at: 2023-05-22T14:08:33Z
updated_at: 2023-06-29T14:40:35Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2077?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch and project coverage have no change.
  > Comparison is base [(`ede6203`)](https://app.codecov.io/gh/simonw/datasette/commit/ede62036180993dbd9d4e5d280fc21c183cda1c3?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.40% compared to head [(`9785c4f`)](https://app.codecov.io/gh/simonw/datasette/pull/2077?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.40%.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main    #2077   +/-   ##
  =======================================
    Coverage   92.40%   92.40%           
  =======================================
    Files          39       39           
    Lines        5803     5803           
  =======================================
    Hits         5362     5362           
    Misses        441      441           
  ```




  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2077?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/554#issuecomment-1557607516
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/554
node_id: IC_kwDOCGYnMM5c1zRc
created_at: 2023-05-22T17:18:33Z
updated_at: 2023-05-22T17:18:33Z
author_association: NONE
body:
  Oh and for context - this goes away if I use `.upsert` instead of `insert(..., ignore=True)`, but I don't want to update the value if it's written, just do an insert if it's new. The code is basically:

  ```py
  def save_items(table, items):
      db["users"].insert(build_user(items[0]), pk="id",ignore=True)
      db[table].insert_all(items)

  if comments := fetch_comments():
      save_items('comments', comments)

  if posts := fetch_posts():
      save_items('posts', posts)
  ```

  So either `comments` or `post` could create the relevant user if those items exist. In cases where they _both_ exist, I get this error. I need the `pk` because either call could create the table.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2078#issuecomment-1563282327
issue_url: https://api.github.com/repos/simonw/datasette/issues/2078
node_id: IC_kwDOBm6k_c5dLcuX
created_at: 2023-05-25T17:46:05Z
updated_at: 2023-05-25T17:46:05Z
author_association: OWNER
body:
  Here's what `wrap_view()` does:

  https://github.com/simonw/datasette/blob/49184c569cd70efbda4f3f062afef3a34401d8d5/datasette/app.py#L1676-L1700

  It's used e.g. here:

  https://github.com/simonw/datasette/blob/49184c569cd70efbda4f3f062afef3a34401d8d5/datasette/app.py#L1371-L1375

  The `BaseView` thing meanwhile works like this:

  https://github.com/simonw/datasette/blob/d97e82df3c8a3f2e97038d7080167be9bb74a68d/datasette/views/base.py#L56-L157

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2078#issuecomment-1563283939
issue_url: https://api.github.com/repos/simonw/datasette/issues/2078
node_id: IC_kwDOBm6k_c5dLdHj
created_at: 2023-05-25T17:47:38Z
updated_at: 2023-05-25T17:47:38Z
author_association: OWNER
body:
  The idea behind `wrap_view()` is dependency injection - it's mainly used by plugins:

  https://docs.datasette.io/en/0.64.3/plugin_hooks.html#register-routes-datasette

  But I like the pattern so I started using it for some of Datasette's own features.

  I should use it for _all_ of Datasette's own features.

  But I still like the way `BaseView` helps with running different code for GET/POST/etc verbs.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2053#issuecomment-1563285150
issue_url: https://api.github.com/repos/simonw/datasette/issues/2053
node_id: IC_kwDOBm6k_c5dLdae
created_at: 2023-05-25T17:48:50Z
updated_at: 2023-05-25T17:49:52Z
author_association: OWNER
body:
  Uncommitted experimental code:
  ```diff
  diff --git a/datasette/views/database.py b/datasette/views/database.py
  index 455ebd1f..85775433 100644
  --- a/datasette/views/database.py
  +++ b/datasette/views/database.py
  @@ -909,12 +909,13 @@ async def query_view(
       elif format_ in datasette.renderers.keys():
           # Dispatch request to the correct output format renderer
           # (CSV is not handled here due to streaming)
  +        print(data)
           result = call_with_supported_arguments(
               datasette.renderers[format_][0],
               datasette=datasette,
  -            columns=columns,
  -            rows=rows,
  -            sql=sql,
  +            columns=data["rows"][0].keys(),
  +            rows=data["rows"],
  +            sql='',
               query_name=None,
               database=db.name,
               table=None,
  @@ -923,7 +924,7 @@ async def query_view(
               # These will be deprecated in Datasette 1.0:
               args=request.args,
               data={
  -                "rows": rows,
  +                "rows": data["rows"],
               },  # TODO what should this be?
           )
           result = await await_me_maybe(result)
  diff --git a/docs/index.rst b/docs/index.rst
  index 5a9cc7ed..254ed3da 100644
  --- a/docs/index.rst
  +++ b/docs/index.rst
  @@ -57,6 +57,7 @@ Contents
      settings
      introspection
      custom_templates
  +   template_context
      plugins
      writing_plugins
      plugin_hooks
  ```
  Where `docs/template_context.rst` looked like this:
  ```rst
  .. _template_context:

  Template context
  ================

  .. currentmodule:: datasette.context

  This page describes the variables made available to templates used by Datasette to render different pages of the application.

  .. autoclass:: QueryContext
      :members:
  ```
  And `datasette/context.py` had this:
  ```python
  from dataclasses import dataclass

  @dataclass
  class QueryContext:
      """
      Used by the ``/database`` page when showing the results of a SQL query
      """
      id: int
      "Id is a thing"
      rows: list[dict]
      "Name is another thing"
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2078#issuecomment-1563292373
issue_url: https://api.github.com/repos/simonw/datasette/issues/2078
node_id: IC_kwDOBm6k_c5dLfLV
created_at: 2023-05-25T17:55:12Z
updated_at: 2023-05-25T17:55:30Z
author_association: OWNER
body:
  So I think subclasses of `BaseView` need to offer a callable which accepts all five of the DI arguments - `datasette`, `request`, `scope`, `send`, `receive` - and then makes a decision based on the HTTP verb as to which method of the class to call. Those methods themselves can accept a subset of those parameters and will only be sent on to them.

  Having two layers of parameter detection feels a little bit untidy, but I think it will work.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2078#issuecomment-1563294669
issue_url: https://api.github.com/repos/simonw/datasette/issues/2078
node_id: IC_kwDOBm6k_c5dLfvN
created_at: 2023-05-25T17:57:06Z
updated_at: 2023-05-25T17:57:06Z
author_association: OWNER
body:
  I may need to be able to detect if a class instance has an `async def __call__` method - I think I can do that like so:

  ```python
  def iscoroutinefunction(obj):
      if inspect.iscoroutinefunction(obj):
          return True
      if hasattr(obj, '__call__') and inspect.iscoroutinefunction(obj.__call__):
          return True
      return False
  ```
  From https://github.com/encode/starlette/issues/886#issuecomment-606585152

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2078#issuecomment-1563308919
issue_url: https://api.github.com/repos/simonw/datasette/issues/2078
node_id: IC_kwDOBm6k_c5dLjN3
created_at: 2023-05-25T18:08:34Z
updated_at: 2023-05-25T18:08:34Z
author_association: OWNER
body:
  After much fiddling this seems to work:
  ```python
  import asyncio, types

  def is_async_callable(obj):
      if not callable(obj):
          raise ValueError("Object is not callable")

      if isinstance(obj, types.FunctionType):
          return asyncio.iscoroutinefunction(obj)
          
      if hasattr(obj, '__call__'):
          return asyncio.iscoroutinefunction(obj.__call__)

      raise ValueError("Not a function and has no __call__ attribute")
  ```
  Tested like so:
  ```python
  class AsyncClass:
      async def __call__(self):
          pass

  class NotAsyncClass:
      def __call__(self):
          pass

  class ClassNoCall:
      pass
      
  async def async_func():
      pass

  def non_async_func():
      pass

  for thing in (AsyncClass(), NotAsyncClass(), ClassNoCall(), async_func, non_async_func):
      try:
          print(thing, is_async_callable(thing))
      except Exception as ex:
          print(thing, ex)
  ```
  ```
  <__main__.AsyncClass object at 0x106c32150> True
  <__main__.NotAsyncClass object at 0x106c32390> False
  <__main__.ClassNoCall object at 0x106c32750> Object is not callable
  <function async_func at 0x1073d5120> True
  <function non_async_func at 0x1073d5080> False
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2078#issuecomment-1563318598
issue_url: https://api.github.com/repos/simonw/datasette/issues/2078
node_id: IC_kwDOBm6k_c5dLllG
created_at: 2023-05-25T18:17:03Z
updated_at: 2023-05-25T18:21:25Z
author_association: OWNER
body:
  I think I want that to return `(is_callable, is_async)` - so I can both test if the thing can be called AND if it should be awaited in the same operation (without any exceptions).

  I tried this:
  ```python
  def is_callable(obj):
      "Returns (is_callable, is_async_callable)"
      if not callable(obj):
          return False, False

      if isinstance(obj, types.FunctionType):
          return True, asyncio.iscoroutinefunction(obj)
          
      if hasattr(obj, '__call__'):
          return True, asyncio.iscoroutinefunction(obj.__call__)

      return False, False
  ```
  ```python
  for thing in (
      async_func, non_async_func, AsyncClass(), NotAsyncClass(), ClassNoCall(),
      AsyncClass, NotAsyncClass, ClassNoCall
  ):
      print(thing, is_callable(thing))
  ```
  And got:
  ```
  <function async_func at 0x1073d5120> (True, True)
  <function non_async_func at 0x1073d5080> (True, False)
  <__main__.AsyncClass object at 0x106cce490> (True, True)
  <__main__.NotAsyncClass object at 0x106ccf710> (True, False)
  <__main__.ClassNoCall object at 0x106ccc810> (False, False)
  <class '__main__.AsyncClass'> (True, True)
  <class '__main__.NotAsyncClass'> (True, False)
  <class '__main__.ClassNoCall'> (True, False)
  ```
  Which is almost right, but I don't like that `AsyncClass` is shown as callable (which it is, since it's a class) and awaitable (which it is not - the `__call__` method may be async but calling the class constructor is not).

  So I'm going to detect classes using `isinstance(obj, type)`.

  ```python
  def is_callable(obj):
      "Returns (is_callable, is_async_callable)"
      if not callable(obj):
          return False, False

      if isinstance(obj, type):
          # It's a class
          return True, False
      
      if isinstance(obj, types.FunctionType):
          return True, asyncio.iscoroutinefunction(obj)
          
      if hasattr(obj, '__call__'):
          return True, asyncio.iscoroutinefunction(obj.__call__)

      assert False, "obj {} somehow is callable with no __call__ method".format(obj)
  ```
  I am reasonably confident the `AssertionError` can never be raised.

  And now:
  ```
  <function async_func at 0x1073d5120> (True, True)
  <function non_async_func at 0x1073d5080> (True, False)
  <__main__.AsyncClass object at 0x106ccfa50> (True, True)
  <__main__.NotAsyncClass object at 0x106ccc8d0> (True, False)
  <__main__.ClassNoCall object at 0x106cd7690> (False, False)
  <class '__main__.AsyncClass'> (True, False)
  <class '__main__.NotAsyncClass'> (True, False)
  <class '__main__.ClassNoCall'> (True, False)
  ```
  Which is what I wanted.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2078#issuecomment-1563326000
issue_url: https://api.github.com/repos/simonw/datasette/issues/2078
node_id: IC_kwDOBm6k_c5dLnYw
created_at: 2023-05-25T18:23:38Z
updated_at: 2023-05-25T18:23:38Z
author_association: OWNER
body:
  I don't like that `is_callable()` implies a single boolean result but actually returns a pair. I'll call it `check_callable(obj)` instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2078#issuecomment-1563329245
issue_url: https://api.github.com/repos/simonw/datasette/issues/2078
node_id: IC_kwDOBm6k_c5dLoLd
created_at: 2023-05-25T18:26:47Z
updated_at: 2023-05-25T18:28:08Z
author_association: OWNER
body:
  With type hints and a namedtuple:
  ```python
  import asyncio
  import types
  from typing import NamedTuple, Any


  class CallableStatus(NamedTuple):
      is_callable: bool
      is_async_callable: bool


  def check_callable(obj: Any) -> CallableStatus:
      if not callable(obj):
          return CallableStatus(False, False)

      if isinstance(obj, type):
          # It's a class
          return CallableStatus(True, False)

      if isinstance(obj, types.FunctionType):
          return CallableStatus(True, asyncio.iscoroutinefunction(obj))

      if hasattr(obj, "__call__"):
          return CallableStatus(True, asyncio.iscoroutinefunction(obj.__call__))

      assert False, "obj {} is somehow callable with no __call__ method".format(repr(obj))
  ```
  ```python
  for thing in (
      async_func,
      non_async_func,
      AsyncClass(),
      NotAsyncClass(),
      ClassNoCall(),
      AsyncClass,
      NotAsyncClass,
      ClassNoCall,
  ):
      print(thing, check_callable(thing))
  ```
  ```
  <function async_func at 0x1073d5120> CallableStatus(is_callable=True, is_async_callable=True)
  <function non_async_func at 0x1073d5080> CallableStatus(is_callable=True, is_async_callable=False)
  <__main__.AsyncClass object at 0x106ba7490> CallableStatus(is_callable=True, is_async_callable=True)
  <__main__.NotAsyncClass object at 0x106740150> CallableStatus(is_callable=True, is_async_callable=False)
  <__main__.ClassNoCall object at 0x10676d910> CallableStatus(is_callable=False, is_async_callable=False)
  <class '__main__.AsyncClass'> CallableStatus(is_callable=True, is_async_callable=False)
  <class '__main__.NotAsyncClass'> CallableStatus(is_callable=True, is_async_callable=False)
  <class '__main__.ClassNoCall'> CallableStatus(is_callable=True, is_async_callable=False)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2078#issuecomment-1563359114
issue_url: https://api.github.com/repos/simonw/datasette/issues/2078
node_id: IC_kwDOBm6k_c5dLveK
created_at: 2023-05-25T18:47:57Z
updated_at: 2023-05-25T18:47:57Z
author_association: OWNER
body:
  Oops, that broke everything:
  ```
      @documented
      async def await_me_maybe(value: typing.Any) -> typing.Any:
          "If value is callable, call it. If awaitable, await it. Otherwise return it."
  >       if callable(value):
  E       TypeError: 'module' object is not callable
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2078#issuecomment-1563419066
issue_url: https://api.github.com/repos/simonw/datasette/issues/2078
node_id: IC_kwDOBm6k_c5dL-G6
created_at: 2023-05-25T19:42:16Z
updated_at: 2023-05-25T19:43:08Z
author_association: OWNER
body:
  Maybe what I want here is the ability to register classes with the router - and have the router know that if it's a class it should instantiate it via its constructor and then await `__call__` it.

  The neat thing about it is that it can reduce the risk of having a class instance that accidentally shares state between requests.

  It also encourages that each class only responds based on the `datasette, request, ...` objects that are passed to its methods.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2078#issuecomment-1563444296
issue_url: https://api.github.com/repos/simonw/datasette/issues/2078
node_id: IC_kwDOBm6k_c5dMERI
created_at: 2023-05-25T20:06:08Z
updated_at: 2023-05-25T20:06:08Z
author_association: OWNER
body:
  This prototype seems to work well:

  ```diff
  diff --git a/datasette/app.py b/datasette/app.py
  index d7dace67..ed0edf28 100644
  --- a/datasette/app.py
  +++ b/datasette/app.py
  @@ -17,6 +17,7 @@ import secrets
   import sys
   import threading
   import time
  +import types
   import urllib.parse
   from concurrent import futures
   from pathlib import Path
  @@ -1266,6 +1267,8 @@ class Datasette:
           # TODO: /favicon.ico and /-/static/ deserve far-future cache expires
           add_route(favicon, "/favicon.ico")
   
  +        add_route(wrap_view(DemoView, self), '/demo')
  +
           add_route(
               asgi_static(app_root / "datasette" / "static"), r"/-/static/(?P<path>.*)$"
           )
  @@ -1673,8 +1676,46 @@ def _cleaner_task_str(task):
       return _cleaner_task_str_re.sub("", s)
   
   
  -def wrap_view(view_fn, datasette):
  -    @functools.wraps(view_fn)
  +class DemoView:
  +    async def __call__(self, datasette, request):
  +        return Response.text("Hello there! {} - {}".format(datasette, request))
  +
  +def wrap_view(view_fn_or_class, datasette):
  +    is_function = isinstance(view_fn_or_class, types.FunctionType)
  +    if is_function:
  +        return wrap_view_function(view_fn_or_class, datasette)
  +    else:
  +        if not isinstance(view_fn_or_class, type):
  +            raise ValueError("view_fn_or_class must be a function or a class")
  +        return wrap_view_class(view_fn_or_class, datasette)
  +
  +
  +def wrap_view_class(view_class, datasette):
  +    async def async_view_for_class(request, send):
  +        instance = view_class()
  +        if inspect.iscoroutinefunction(instance.__call__):
  +            return await async_call_with_supported_arguments(
  +                instance.__call__,
  +                scope=request.scope,
  +                receive=request.receive,
  +                send=send,
  +                request=request,
  +                datasette=datasette,
  +            )
  +        else:
  +            return call_with_supported_arguments(
  +                instance.__call__,
  +                scope=request.scope,
  +                receive=request.receive,
  +                send=send,
  +                request=request,
  +                datasette=datasette,
  +            )
  +
  +    return async_view_for_class
  +
  +
  +def wrap_view_function(view_fn, datasette):
       async def async_view_fn(request, send):
           if inspect.iscoroutinefunction(view_fn):
               response = await async_call_with_supported_arguments(
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2078#issuecomment-1563488929
issue_url: https://api.github.com/repos/simonw/datasette/issues/2078
node_id: IC_kwDOBm6k_c5dMPKh
created_at: 2023-05-25T20:48:12Z
updated_at: 2023-05-25T20:48:39Z
author_association: OWNER
body:
  Actually no need for that extra level of parameter detection: `BaseView.__call__` should _always_ take `datasette, request` - `scope` and `receive` are both available on `request`, and `send` is only needed if you're not planning on returning a `Response` object.

  So the `get` and `post` and suchlike methods should take `datasette` and `request` too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2078#issuecomment-1563498048
issue_url: https://api.github.com/repos/simonw/datasette/issues/2078
node_id: IC_kwDOBm6k_c5dMRZA
created_at: 2023-05-25T20:57:52Z
updated_at: 2023-05-25T20:58:13Z
author_association: OWNER
body:
  Here's a new `BaseView` class that automatically populates `OPTIONS` based on available methods:
  ```python
  class BaseView:
      async def head(self, *args, **kwargs):
          try:
              response = await self.get(*args, **kwargs)
              response.body = b""
              return response
          except AttributeError:
              raise

      async def method_not_allowed(self, request):
          if (
              request.path.endswith(".json")
              or request.headers.get("content-type") == "application/json"
          ):
              response = Response.json(
                  {"ok": False, "error": "Method not allowed"}, status=405
              )
          else:
              response = Response.text("Method not allowed", status=405)
          return response

      async def options(self, request, *args, **kwargs):
          response = Response.text("ok")
          response.headers["allow"] = ", ".join(
              method.upper()
              for method in ("head", "get", "post", "put", "patch", "delete")
              if hasattr(self, method)
          )
          return response

      async def __call__(self, request, datasette):
          try:
              handler = getattr(self, request.method.lower())
              return await handler(request, datasette)
          except AttributeError:
              return await self.method_not_allowed(request)


  class DemoView(BaseView):
      async def get(self, datasette, request):
          return Response.text("Hello there! {} - {}".format(datasette, request))

      post = get
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2078#issuecomment-1563522011
issue_url: https://api.github.com/repos/simonw/datasette/issues/2078
node_id: IC_kwDOBm6k_c5dMXPb
created_at: 2023-05-25T21:22:30Z
updated_at: 2023-05-25T21:22:30Z
author_association: OWNER
body:
  This is bad:
  ```python
      async def __call__(self, request, datasette):
          try:
              handler = getattr(self, request.method.lower())
              return await handler(request, datasette)
          except AttributeError:
              return await self.method_not_allowed(request)
  ```
  Because it hides any `AttributeError` exceptions that might occur in the view code.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2079#issuecomment-1563547097
issue_url: https://api.github.com/repos/simonw/datasette/issues/2079
node_id: IC_kwDOBm6k_c5dMdXZ
created_at: 2023-05-25T21:51:38Z
updated_at: 2023-05-25T21:51:38Z
author_association: OWNER
body:
  Also need to update this documentation:

  https://github.com/simonw/datasette/blob/9584879534ff0556e04e4c420262972884cac87b/docs/json_api.rst?plain=1#L453-L465

  Or maybe make that automated via `cog`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2079#issuecomment-1563563438
issue_url: https://api.github.com/repos/simonw/datasette/issues/2079
node_id: IC_kwDOBm6k_c5dMhWu
created_at: 2023-05-25T22:08:28Z
updated_at: 2023-05-25T22:08:28Z
author_association: OWNER
body:
  I ran this on https://www.example.com/ twice using the console:
  ```javascript
  fetch(
    `https://latest.datasette.io/ephemeral/foo/1/-/update`,
    {
      method: "POST",
      mode: "cors",
      headers: {
        Authorization: `Bearer tok`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({update: {blah: 1}}),
    }
  )
    .then((r) => r.json())
    .then((data) => {
      console.log(data);
    });
  ```
  And got this in the network pane:

  <img width="945" alt="image" src="https://github.com/simonw/datasette/assets/9599/cc767917-388b-4616-b35d-efaa1591948a">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2079#issuecomment-1563587230
issue_url: https://api.github.com/repos/simonw/datasette/issues/2079
node_id: IC_kwDOBm6k_c5dMnKe
created_at: 2023-05-25T22:28:20Z
updated_at: 2023-05-25T22:28:20Z
author_association: OWNER
body:
  Weird... after the deploy went out:

  <img width="945" alt="image" src="https://github.com/simonw/datasette/assets/9599/a8abb12a-446b-4970-8e76-3b218b6d02c7">

  But the request did indeed get the new header:

  <img width="951" alt="image" src="https://github.com/simonw/datasette/assets/9599/7c72578c-9250-47d3-a4d2-1ece8c07ef84">

  So I'm not sure why it's making multiple `POST` requests like that.

  Maybe it's because the attempted `POST` failed with a 404?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2079#issuecomment-1563588199
issue_url: https://api.github.com/repos/simonw/datasette/issues/2079
node_id: IC_kwDOBm6k_c5dMnZn
created_at: 2023-05-25T22:29:47Z
updated_at: 2023-05-25T22:30:12Z
author_association: OWNER
body:
  https://fetch.spec.whatwg.org/#http-access-control-max-age says:

  > Indicates the number of seconds (5 by default) the information provided by the [Access-Control-Allow-Methods](https://fetch.spec.whatwg.org/#http-access-control-allow-methods) and [Access-Control-Allow-Headers](https://fetch.spec.whatwg.org/#http-access-control-allow-headers) [headers](https://fetch.spec.whatwg.org/#concept-header) can be cached.

  So there was already a 5s cache anyway.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2079#issuecomment-1563597589
issue_url: https://api.github.com/repos/simonw/datasette/issues/2079
node_id: IC_kwDOBm6k_c5dMpsV
created_at: 2023-05-25T22:42:07Z
updated_at: 2023-05-25T22:42:07Z
author_association: OWNER
body:
  Mystery solved as to why I wasn't seeing this work:

  <img width="951" alt="CleanShot 2023-05-25 at 15 38 32@2x" src="https://github.com/simonw/datasette/assets/9599/140ee046-69c2-4a68-b7cb-a4603552477c">

  I had "Disable Cache" checked!

  I ran this experiment after un-checking that box:

  ```javascript
  fetch('https://latest.datasette.io/ephemeral/foo/1/-/update', {
      method: 'POST',
      headers: {
          'Content-Type': 'application/json'
      },
      body: JSON.stringify({ test: 'test' })
  });
  // And run it again
  fetch('https://latest.datasette.io/ephemeral/foo/1/-/update', {
      method: 'POST',
      headers: {
          'Content-Type': 'application/json'
      },
      body: JSON.stringify({ test: 'test' })
  });
  // Now try a thing that doesn't serve that max-age header yet:
  fetch('https://latest-with-plugins.datasette.io/ephemeral/foo/1/-/update', {
      method: 'POST',
      headers: {
          'Content-Type': 'application/json'
      },
      body: JSON.stringify({ test: 'test' })
  });
  // And a second time but within 5s
  fetch('https://latest-with-plugins.datasette.io/ephemeral/foo/1/-/update', {
      method: 'POST',
      headers: {
          'Content-Type': 'application/json'
      },
      body: JSON.stringify({ test: 'test' })
  });
  // Third time after waiting longer than 5s
  fetch('https://latest-with-plugins.datasette.io/ephemeral/foo/1/-/update', {
      method: 'POST',
      headers: {
          'Content-Type': 'application/json'
      },
      body: JSON.stringify({ test: 'test' })
  });
  // Try that original one again - still within the 1hr cache time
  fetch('https://latest.datasette.io/ephemeral/foo/1/-/update', {
      method: 'POST',
      headers: {
          'Content-Type': 'application/json'
      },
      body: JSON.stringify({ test: 'test' })
  });
  ```
  The results show that the cache of 1hr was being obeyed for `latest.datasette.io` while the `latest-with-plugins.datasette.io` default cache of 5s was being obeyed too.

  <img width="904" alt="image" src="https://github.com/simonw/datasette/assets/9599/39609d24-ef68-4db0-905a-451ed2462859">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2080#issuecomment-1563629348
issue_url: https://api.github.com/repos/simonw/datasette/issues/2080
node_id: IC_kwDOBm6k_c5dMxck
created_at: 2023-05-25T23:31:10Z
updated_at: 2023-05-26T00:07:34Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2080?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch coverage: **`95.45`**% and project coverage change: **`+0.01`** :tada:
  > Comparison is base [(`b49fa44`)](https://app.codecov.io/gh/simonw/datasette/commit/b49fa446d683ddcaf6faf2944dacc0d866bf2d70?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.40% compared to head [(`df5fd73`)](https://app.codecov.io/gh/simonw/datasette/pull/2080?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.41%.

  > :exclamation: Current head df5fd73 differs from pull request most recent head e990fbc. Consider uploading reports for the commit e990fbc to get more accurate results

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@            Coverage Diff             @@
  ##             main    #2080      +/-   ##
  ==========================================
  + Coverage   92.40%   92.41%   +0.01%     
  ==========================================
    Files          39       39              
    Lines        5768     5790      +22     
  ==========================================
  + Hits         5330     5351      +21     
  - Misses        438      439       +1     
  ```


  | [Impacted Files](https://app.codecov.io/gh/simonw/datasette/pull/2080?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/views/base.py](https://app.codecov.io/gh/simonw/datasette/pull/2080?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `92.97% <95.45%> (+0.18%)` | :arrow_up: |


  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2080?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2053#issuecomment-1563663616
issue_url: https://api.github.com/repos/simonw/datasette/issues/2053
node_id: IC_kwDOBm6k_c5dM50A
created_at: 2023-05-26T00:32:08Z
updated_at: 2023-05-26T00:32:08Z
author_association: OWNER
body:
  Now that I have the new `View` subclass from #2078 I want to use it to simplify this code.

  Challenge: there are several things to consider here:

  - The `/db` page without `?sql=` displays a list of tables in that database
  - With `?sql=` it shows the query results for that query (or an error)
  - If it's a `/db/name-of-canned-query` it works a bit like the query page, but executes a canned query instead of the `?sql=` query
  - POST `/db/name-of-canned-query` is support for writable canned queries

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2053#issuecomment-1563663925
issue_url: https://api.github.com/repos/simonw/datasette/issues/2053
node_id: IC_kwDOBm6k_c5dM541
created_at: 2023-05-26T00:32:47Z
updated_at: 2023-05-26T00:35:47Z
author_association: OWNER
body:
  I'm going to entirely split canned queries off from `?sql=` queries - they share a bunch of code right now which is just making everything much harder to follow.

  I'll refactor their shared bits into functions that they both call.

  Or _maybe_ I'll try having `CannedQueryView` as a subclass of `QueryView`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2053#issuecomment-1563667574
issue_url: https://api.github.com/repos/simonw/datasette/issues/2053
node_id: IC_kwDOBm6k_c5dM6x2
created_at: 2023-05-26T00:40:22Z
updated_at: 2023-05-26T00:40:22Z
author_association: OWNER
body:
  Or maybe...

  - `BaseQueryView(View)` - knows how to render the results of a SQL query
  - `QueryView(BaseQueryView)` - renders from `?sql=`
  - `CannedQueryView(BaseQueryView)` - renders for a named canned query

  And then later perhaps:

  - `RowQueryView(BaseQueryView)` - renders the `select * from t where pk = ?`
  - `TableQueryView(BaseQueryView)` - replaces the super complex existing `TableView`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2053#issuecomment-1563793781
issue_url: https://api.github.com/repos/simonw/datasette/issues/2053
node_id: IC_kwDOBm6k_c5dNZl1
created_at: 2023-05-26T04:27:55Z
updated_at: 2023-05-26T04:27:55Z
author_association: OWNER
body:
  I should split out a `canned_query.html` template too, as something that extends the `query.html` template.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2053#issuecomment-1565058994
issue_url: https://api.github.com/repos/simonw/datasette/issues/2053
node_id: IC_kwDOBm6k_c5dSOey
created_at: 2023-05-26T23:13:02Z
updated_at: 2023-05-26T23:13:02Z
author_association: OWNER
body:
  I should have an extra called `extra_html_context` which bundles together all of the weird extra stuff needed by the HTML template, and is then passed as the root context when the template is rendered (with the other stuff from extras patched into it).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/556#issuecomment-1575310378
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/556
node_id: IC_kwDOCGYnMM5d5VQq
created_at: 2023-06-04T01:21:15Z
updated_at: 2023-06-04T01:21:15Z
author_association: CONTRIBUTOR
body:
  I've resolved my use, with the line-buffered output and while read loop for line buffered input, but I leave this here so the incremental saving or line-buffered use-case can be explicitly handled or rejected (or deferred).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/557#issuecomment-1577355134
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/557
node_id: IC_kwDOCGYnMM5eBId-
created_at: 2023-06-05T19:26:26Z
updated_at: 2023-06-05T19:26:26Z
author_association: CONTRIBUTOR
body:
  this isn't really actionable... I'm just being a whiny baby. I have tasted the milk of being able to use `upsert_all`, `insert_all`, etc without having to write DDL to create tables. The meat of the issue is that SQLITE doesn't make rowid stable between vacuums so it is not possible to take shortcuts

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/433#issuecomment-1578840450
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/433
node_id: IC_kwDOCGYnMM5eGzGC
created_at: 2023-06-06T14:09:04Z
updated_at: 2023-06-06T14:09:04Z
author_association: NONE
body:
  I also ran into this recently. See below for a patch for one possible solution (tested via "it works on my machine", but I don't expect that this behavior would vary a whole lot across terminal emulators and shells). Another possible solution might be to subclass click's `ProgressBar` to keep the logic within the original context manager. Happy to send a PR or for this patch to serve as the basis for a fix that someone else authors.

  ```patch
  diff --git a/sqlite_utils/utils.py b/sqlite_utils/utils.py
  index 06c1a4c..530a3a3 100644
  --- a/sqlite_utils/utils.py
  +++ b/sqlite_utils/utils.py
  @@ -147,14 +147,23 @@ def decode_base64_values(doc):
   
   
   class UpdateWrapper:
  -    def __init__(self, wrapped, update):
  +    def __init__(self, wrapped, update, render_finish):
           self._wrapped = wrapped
           self._update = update
  +        self._render_finish = render_finish
   
       def __iter__(self):
  -        for line in self._wrapped:
  -            self._update(len(line))
  -            yield line
  +        return self
  +
  +    def __next__(self):
  +        try:
  +            line = next(self._wrapped)
  +        except StopIteration as e:
  +            self._render_finish()
  +            raise
  +
  +        self._update(len(line))
  +        return line
   
       def read(self, size=-1):
           data = self._wrapped.read(size)
  @@ -178,7 +187,7 @@ def file_progress(file, silent=False, **kwargs):
       else:
           file_length = os.path.getsize(file.name)
           with click.progressbar(length=file_length, **kwargs) as bar:
  -            yield UpdateWrapper(file, bar.update)
  +            yield UpdateWrapper(file, bar.update, bar.render_finish)
   
   
   class Format(enum.Enum):

  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2052#issuecomment-1585149909
issue_url: https://api.github.com/repos/simonw/datasette/issues/2052
node_id: IC_kwDOBm6k_c5ee3fV
created_at: 2023-06-09T21:35:00Z
updated_at: 2023-06-09T21:35:00Z
author_association: CONTRIBUTOR
body:
  Thanks @cldellow for the thoughtful comments! These are all things that I'll keep in mind as we figure out how/if this API is actually used by plugin authors once it's actually out in the world.

  > Yes, this would work - but it requires me to continue to communicate the column names out of band (in order to fetch the facet data per-column before registering my plugin), vs being able to re-use them from the plugin implementation.

  Ah, I understand now! Thanks for explaining. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/557#issuecomment-1590531892
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/557
node_id: IC_kwDOCGYnMM5ezZc0
created_at: 2023-06-14T06:09:21Z
updated_at: 2023-06-14T06:09:21Z
author_association: CONTRIBUTOR
body:
  I put together a [simple script](https://github.com/chapmanjacobd/library/blob/42129c5ebe15f9d74653c0f5ca4ed0c991d383e0/xklb/scripts/dedupe_db.py) to upsert and remove duplicate rows based on business keys. If anyone has similar problems with above this might help

  ```
  CREATE TABLE my_table (
      id INTEGER PRIMARY KEY,
      column1 TEXT,
      column2 TEXT,
      column3 TEXT
  );

  INSERT INTO my_table (column1, column2, column3)
  VALUES
      ('Value 1', 'Duplicate 1', 'Duplicate A'),
      ('Value 2', 'Duplicate 2', 'Duplicate B'),
      ('Value 3', 'Duplicate 2', 'Duplicate C'),
      ('Value 4', 'Duplicate 3', 'Duplicate D'),
      ('Value 5', 'Duplicate 3', 'Duplicate E'),
      ('Value 6', 'Duplicate 3', 'Duplicate F');
  ```

  ```
  library dedupe-db test.db my_table --bk column2
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/555#issuecomment-1592047502
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/555
node_id: IC_kwDOCGYnMM5e5LeO
created_at: 2023-06-14T22:00:10Z
updated_at: 2023-06-14T22:01:57Z
author_association: CONTRIBUTOR
body:
  You may want to try doing a performance comparison between this and just selecting all the ids with few constraints and then doing the filtering within python.

  That might seem like a lazy-programmer, inefficient way but queries with large resultsets are a different profile than what databases like SQLITE are designed for. That is not to say that SQLITE is slow or that python is always faster but when you start reading >20% of an index there is an equilibrium that is reached. Especially when adding in writing extra temp tables and stuff to memory/disk. And especially given the `NOT IN` style of query...

  You may also try chunking like this:

  ```py
  def chunks(lst, n) -> Generator:
      for i in range(0, len(lst), n):
          yield lst[i : i + n]

  SQLITE_PARAM_LIMIT = 32765

  data = []
  chunked = chunks(video_ids, consts.SQLITE_PARAM_LIMIT)
  for ids in chunked:
      data.expand(
          list(
              db.query(
                  f"""SELECT * from videos
                  WHERE id in ("""
                  + ",".join(["?"] * len(ids))
                  + ")",
                  (*ids,),
              )
          )
      )
  ```

  but that actually won't work with your `NOT IN` requirements. You need to query the full resultset to check any row.

  Since you are doing stuff with files/videos in SQLITE you might be interested in my side project: https://github.com/chapmanjacobd/library

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/529#issuecomment-1592110694
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/529
node_id: IC_kwDOCGYnMM5e5a5m
created_at: 2023-06-14T23:11:47Z
updated_at: 2023-06-14T23:12:12Z
author_association: CONTRIBUTOR
body:
  sorry i was wrong. `sqlite-utils --raw-lines` works correctly

  ```
  sqlite-utils --raw-lines :memory: "SELECT * FROM (VALUES ('test'), ('line2'))" | cat -A
  test$
  line2$

  sqlite-utils --csv --no-headers :memory: "SELECT * FROM (VALUES ('test'), ('line2'))" | cat -A
  test$
  line2$
  ```

  I think this was fixed somewhat recently

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/535#issuecomment-1592617271
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/535
node_id: IC_kwDOCGYnMM5e7Wk3
created_at: 2023-06-15T08:39:49Z
updated_at: 2023-06-15T08:39:49Z
author_association: NONE
body:
  > piping to `jq` is good enough usually

  ... or `python -m json.tool`[^1], if you don't have `jq` installed.

  [^1]: no fancy colouring, like `jq`; only pretty-printing

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/260#issuecomment-1600778057
issue_url: https://api.github.com/repos/simonw/datasette/issues/260
node_id: IC_kwDOBm6k_c5fae9J
created_at: 2023-06-21T12:51:22Z
updated_at: 2023-06-21T12:51:22Z
author_association: OWNER
body:
  Another example of confusion from this today: https://discord.com/channels/823971286308356157/823971286941302908/1121042411238457374

  See also https://gist.github.com/BinomeDeNewton/651ac8b50dd5420f8e54d1682eee5fed?permalink_comment_id=4605982#gistcomment-4605982

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1604379952
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/235
node_id: IC_kwDOCGYnMM5foOUw
created_at: 2023-06-23T14:39:55Z
updated_at: 2023-06-23T15:39:32Z
author_association: OWNER
body:
  Ideally a workaround for this right now would be to install `pysqlite3` in the same virtual environment:

      sqlite-utils install pysqlite3-binary

  But `pysqlite3-binary` doesn't yet ship a wheel for macOS so this probably won't work for most people.

  The "easiest" fix at the moment is to use Python from Homebrew - so `brew install sqlite-utils` for example won't suffer from this problem. Not a great solution for people who aren't using Homebrew though!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606237836
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/560
node_id: IC_kwDOCGYnMM5fvT6M
created_at: 2023-06-25T19:49:45Z
updated_at: 2023-06-26T08:20:59Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch coverage: **`80.55`**% and project coverage change: **`-0.15`** :warning:
  > Comparison is base [(`2747257`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/2747257a3334d55e890b40ec58fada57ae8cfbfd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.36% compared to head [(`5e7d27e`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.22%.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@            Coverage Diff             @@
  ##             main     #560      +/-   ##
  ==========================================
  - Coverage   96.36%   96.22%   -0.15%     
  ==========================================
    Files           6        6              
    Lines        2726     2752      +26     
  ==========================================
  + Hits         2627     2648      +21     
  - Misses         99      104       +5     
  ```


  | [Impacted Files](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [sqlite\_utils/utils.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `94.56% <63.63%> (-0.62%)` | :arrow_down: |
  | [sqlite\_utils/db.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.33% <86.36%> (-0.20%)` | :arrow_down: |
  | [sqlite\_utils/cli.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.22% <100.00%> (ø)` | |


  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606270055
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/560
node_id: IC_kwDOCGYnMM5fvbxn
created_at: 2023-06-25T21:31:56Z
updated_at: 2023-06-25T21:31:56Z
author_association: OWNER
body:
  Lots of failures now that I'm trying to run the tests against `sqlean.py` on macOS and Python 3.10: https://github.com/simonw/sqlite-utils/actions/runs/5371800108/jobs/9744802953

  A bunch of these, because `pysqlite3` chooses not to implement `.iterdump()`:
  ```
      @pytest.fixture
      def db_to_analyze_path(db_to_analyze, tmpdir):
          path = str(tmpdir / "test.db")
          db = sqlite3.connect(path)
  >       db.executescript("\n".join(db_to_analyze.conn.iterdump()))
  E       AttributeError: 'sqlean.dbapi2.Connection' object has no attribute 'iterdump'
  ```
  Also some of these:
  ```
      def test_analyze_whole_database(db):
          assert set(db.table_names()) == {"one_index", "two_indexes"}
          db.analyze()
  >       assert set(db.table_names()) == {"one_index", "two_indexes", "sqlite_stat1"}
  E       AssertionError: assert {'one_index',...'two_indexes'} == {'one_index',...'two_indexes'}
  E         Extra items in the left set:
  E         'sqlite_stat4'
  E         Full diff:
  E         - {'two_indexes', 'sqlite_stat1', 'one_index'}
  E         + {'two_indexes', 'sqlite_stat1', 'sqlite_stat4', 'one_index'}
  E         ?                                 ++++++++++++++++
  ```
  Apparently `sqlean.py` adds a `sqlite_stat4` table that the tests are not expecting.

  Plus some errors that look like this:
  ```
      def test_enable_wal():
          runner = CliRunner()
          dbs = ["test.db", "test2.db"]
          with runner.isolated_filesystem():
              for dbname in dbs:
                  db = Database(dbname)
                  db["t"].create({"pk": int}, pk="pk")
                  assert db.journal_mode == "delete"
              result = runner.invoke(cli.cli, ["enable-wal"] + dbs)
  >           assert 0 == result.exit_code
  E           AssertionError: assert 0 == 1
  E            +  where 1 = <Result OperationalError('cannot change into wal mode from within a transaction')>.exit_code
  ```
  Test summary:
  ```
  ============ 13 failed, 909 passed, 16 skipped, 2 errors in 19.29s =============
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606270887
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/560
node_id: IC_kwDOCGYnMM5fvb-n
created_at: 2023-06-25T21:37:12Z
updated_at: 2023-06-26T08:21:00Z
author_association: OWNER
body:
  On my own laptop I got a crash running the tests - details here:

  - https://github.com/nalgeon/sqlean.py/issues/3

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606273005
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/560
node_id: IC_kwDOCGYnMM5fvcft
created_at: 2023-06-25T21:47:47Z
updated_at: 2023-06-25T21:47:47Z
author_association: OWNER
body:
  I can use https://github.com/simonw/sqlite-dump as an optional dependency to handle the missing `.iterdump()` method.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606290917
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/560
node_id: IC_kwDOCGYnMM5fvg3l
created_at: 2023-06-25T22:32:28Z
updated_at: 2023-06-25T22:32:28Z
author_association: OWNER
body:
  I've fixed most of the test failures, but I still need to fix this one:

  > cannot change into wal mode from within a transaction

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606293382
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/560
node_id: IC_kwDOCGYnMM5fvheG
created_at: 2023-06-25T22:34:47Z
updated_at: 2023-06-25T22:34:47Z
author_association: OWNER
body:
  ```pycon
  >>> import sqlite3
  >>> db = sqlite3.connect("/tmp/1.db")
  >>> db.execute('PRAGMA journal_mode=wal;')
  <sqlite3.Cursor object at 0x102475ec0>
  >>> import sqlean
  >>> db2 = sqlean.connect("/tmp/2.db")
  >>> db2.execute('PRAGMA journal_mode=wal;')
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
  sqlean.dbapi2.OperationalError: cannot change into wal mode from within a transaction
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606294627
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/560
node_id: IC_kwDOCGYnMM5fvhxj
created_at: 2023-06-25T22:40:10Z
updated_at: 2023-06-25T22:40:10Z
author_association: OWNER
body:
  I suspect this has something to do with `autocommit` mode in `sqlite3` - which I may be able to turn off by setting `con.isolation_level = None`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606297356
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/560
node_id: IC_kwDOCGYnMM5fvicM
created_at: 2023-06-25T22:42:41Z
updated_at: 2023-06-25T22:42:41Z
author_association: OWNER
body:
  Yes that does seem to do the trick:
  ```pycon
  >>> import sqlean
  >>> db = sqlean.connect("/tmp/4.db")
  >>> db.execute('PRAGMA journal_mode;').fetchall()
  [('delete',)]
  >>> db.isolation_level
  ''
  >>> db.execute('PRAGMA journal_mode=wal;')
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
  sqlean.dbapi2.OperationalError: cannot change into wal mode from within a transaction
  >>> db.isolation_level = None
  >>> db.isolation_level
  >>> db.execute('PRAGMA journal_mode=wal;')
  <sqlean.dbapi2.Cursor object at 0x104437c20>
  ```
  Weird how `isolation_level` of empty string causes the error, but setting that to `None` fixes the error.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606310630
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/560
node_id: IC_kwDOCGYnMM5fvlrm
created_at: 2023-06-25T23:06:07Z
updated_at: 2023-06-25T23:06:07Z
author_association: OWNER
body:
  Filed an issue about the above with `pysqlite3` (which `sqlean.py` is based on) here:
  - https://github.com/coleifer/pysqlite3/issues/58

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606315321
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/560
node_id: IC_kwDOCGYnMM5fvm05
created_at: 2023-06-25T23:18:33Z
updated_at: 2023-06-25T23:18:33Z
author_association: OWNER
body:
  Documentation preview: https://sqlite-utils--560.org.readthedocs.build/en/560/installation.html#alternatives-to-sqlite3

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2052#issuecomment-1606352600
issue_url: https://api.github.com/repos/simonw/datasette/issues/2052
node_id: IC_kwDOBm6k_c5fvv7Y
created_at: 2023-06-26T00:17:04Z
updated_at: 2023-06-26T00:17:04Z
author_association: CONTRIBUTOR
body:
  :wave: would love to see this get merged soon! I want to make a javascript plugin on top of the code-mirror editor to make a few things nicer (function auto-complete, table/column descriptions, etc.), and this would help out a bunch

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1606411508
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/235
node_id: IC_kwDOCGYnMM5fv-T0
created_at: 2023-06-26T01:42:10Z
updated_at: 2023-06-26T01:42:22Z
author_association: OWNER
body:
  https://sqlite-utils.datasette.io/en/stable/changelog.html#v3-33 - upgrading to `sqlite-utils>=3.33` and then installing both `sqlean.py` and `sqlite-dump` in the same virtual environment as `sqlite-utils` should fix this issue.

reactions:
  {"total_count": 2, "+1": 0, "-1": 0, "laugh": 0, "hooray": 2, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


html_url: https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1606415188
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/235
node_id: IC_kwDOCGYnMM5fv_NU
created_at: 2023-06-26T01:46:47Z
updated_at: 2023-06-26T01:47:01Z
author_association: OWNER
body:
  I just tested this in a brand new virtual environment using the macOS Python 3:
  ```bash
  pipenv shell --python /Applications/Xcode.app/Contents/Developer/usr/bin/python3
  ```
  Then in that virtual environment I ran:
  ```bash
  pip install sqlite-utils
  # Confirm the right one is on the path:
  which sqlite-utils

  curl "https://data.nasa.gov/resource/y77d-th95.json" | \
      sqlite-utils insert meteorites.db meteorites - --pk=id
  sqlite-utils extract meteorites.db meteorites  recclass
  ```
  This threw the same error reported above. Then I did this:
  ```bash
  rm meteorites.db
  pip install sqlean.py

  curl "https://data.nasa.gov/resource/y77d-th95.json" | \
      sqlite-utils insert meteorites.db meteorites - --pk=id
  sqlite-utils extract meteorites.db meteorites  recclass
  ```
  And that second time it worked correctly.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/561#issuecomment-1610040517
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/561
node_id: IC_kwDOCGYnMM5f90TF
created_at: 2023-06-27T18:44:31Z
updated_at: 2023-06-27T18:44:38Z
author_association: OWNER
body:
  Got this working:
  ```bash
  sqlite-utils insert /tmp/playground.db Playground_Submission_Data \
    ~/Downloads/Playground_Submission_Data.csv --csv --stop-after 2000
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1510#issuecomment-1610512875
issue_url: https://api.github.com/repos/simonw/datasette/issues/1510
node_id: IC_kwDOBm6k_c5f_nnr
created_at: 2023-06-28T02:02:10Z
updated_at: 2023-06-28T02:05:21Z
author_association: OWNER
body:
  I prototyped an approach to this using dataclasses and a `cog` mechanism for turning those into rendered tables in Sphinx. Here's what that prototype looks like:

  <img width="1069" alt="image" src="https://github.com/simonw/datasette/assets/9599/79210a43-716c-4470-a62d-01aa871973ec">

  See https://github.com/simonw/datasette/commit/68223784167fdec4e7ebfca56002a6548ba7b423 for how it works.

  Here's the class that documented:

  https://github.com/simonw/datasette/blob/68223784167fdec4e7ebfca56002a6548ba7b423/datasette/context.py#L54-L68

  And the code that generates the rST: https://github.com/simonw/datasette/blob/68223784167fdec4e7ebfca56002a6548ba7b423/datasette/context.py#L19-L45

  And the bit that cog executes: https://github.com/simonw/datasette/blob/68223784167fdec4e7ebfca56002a6548ba7b423/docs/template_context.rst?plain=1#L9-L12

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2089#issuecomment-1613305070
issue_url: https://api.github.com/repos/simonw/datasette/issues/2089
node_id: IC_kwDOBm6k_c5gKRTu
created_at: 2023-06-29T14:40:44Z
updated_at: 2023-06-29T14:40:44Z
author_association: OWNER
body:
  I'm not sure why I can't duplicate this failure in my local development environment:
  ```
  % codespell docs/metadata.rst
  ```
  It finds no errors.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2089#issuecomment-1613306787
issue_url: https://api.github.com/repos/simonw/datasette/issues/2089
node_id: IC_kwDOBm6k_c5gKRuj
created_at: 2023-06-29T14:41:47Z
updated_at: 2023-06-29T14:41:47Z
author_association: OWNER
body:
  Looks like in CI it's running 2.2.5:
  ```
  Collecting codespell (from datasette==1.0a2)
    Downloading codespell-2.2.5-py3-none-any.whl (242 kB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 242.7/242.7 kB 31.1 MB/s eta 0:00:00
  ```
  But on my laptop it's 2.2.2:
  ```
  % codespell --version
  2.2.2
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2089#issuecomment-1613307716
issue_url: https://api.github.com/repos/simonw/datasette/issues/2089
node_id: IC_kwDOBm6k_c5gKR9E
created_at: 2023-06-29T14:42:23Z
updated_at: 2023-06-29T14:42:23Z
author_association: OWNER
body:
  Yes, upgrading locally got me the correct version and the test failure:
  ```
  % pip install -U codespell
  Requirement already satisfied: codespell in /Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages (2.2.2)
  Collecting codespell
    Downloading codespell-2.2.5-py3-none-any.whl (242 kB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 242.7/242.7 kB 4.9 MB/s eta 0:00:00
  Installing collected packages: codespell
    Attempting uninstall: codespell
      Found existing installation: codespell 2.2.2
      Uninstalling codespell-2.2.2:
        Successfully uninstalled codespell-2.2.2
  Successfully installed codespell-2.2.5
  % codespell docs/metadata.rst
  docs/metadata.rst:192: displaing ==> displaying
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2089#issuecomment-1613315851
issue_url: https://api.github.com/repos/simonw/datasette/issues/2089
node_id: IC_kwDOBm6k_c5gKT8L
created_at: 2023-06-29T14:47:38Z
updated_at: 2023-06-29T14:47:38Z
author_association: OWNER
body:
  Confirmed, this was a 2.2.5 change: https://github.com/codespell-project/codespell/releases/tag/v2.2.5

  > -   Add displaing->displaying by [@peternewman](https://github.com/peternewman) in [#2808](https://github.com/codespell-project/codespell/pull/2808)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2090#issuecomment-1613339404
issue_url: https://api.github.com/repos/simonw/datasette/issues/2090
node_id: IC_kwDOBm6k_c5gKZsM
created_at: 2023-06-29T15:01:01Z
updated_at: 2023-06-29T15:01:20Z
author_association: OWNER
body:
  I tried it just now and got some interesting results.

  I dropped in a `ruff.toml` file:
  ```toml
  line-length = 160
  ```
  Because the default line length limit of 88 was causing a lot of noisy errors.

  Then run:

  ```bash
  pip install ruff
  ruff check .
  ```
  Plenty of warnings about unused imports - running `ruff check . --fix` fixed those automatically, but I think I still need to manually review them as some might be imports which are deliberate and should be in `__all__` to ensure they are visible from that module as well.

  Some lines in tests are longer than even 160 chars, e.g.:

  https://github.com/simonw/datasette/blob/99ba05118891db9dc30f1dca22ad6709775560de/tests/test_html.py#L673-L681

  These can have `  # noqa: E501` added to the end of those lines to skip the check for them.

  That got it down to:

  ```
  % ruff check . 
  datasette/views/table.py:23:5: F811 Redefinition of unused `format_bytes` from line 19
  run_tests.py:2:5: E401 Multiple imports on one line
  tests/test_api.py:591:40: F811 Redefinition of unused `app_client_no_files` from line 7
  tests/test_api.py:629:35: F811 Redefinition of unused `app_client_no_files` from line 7
  tests/test_api.py:635:54: F811 Redefinition of unused `app_client_with_dot` from line 8
  tests/test_api.py:661:25: F811 Redefinition of unused `app_client_shorter_time_limit` from line 9
  tests/test_api.py:759:25: F811 Redefinition of unused `app_client_two_attached_databases_one_immutable` from line 10
  tests/test_api.py:892:28: F811 Redefinition of unused `app_client_larger_cache_size` from line 11
  tests/test_api.py:928:5: F811 Redefinition of unused `app_client_with_cors` from line 12
  tests/test_api.py:929:5: F811 Redefinition of unused `app_client_two_attached_databases_one_immutable` from line 10
  tests/test_api.py:969:38: F811 Redefinition of unused `app_client_two_attached_databases` from line 13
  tests/test_api.py:976:39: F811 Redefinition of unused `app_client_conflicting_database_names` from line 14
  tests/test_api.py:987:38: F811 Redefinition of unused `app_client_immutable_and_inspect_file` from line 15
  tests/test_api.py:1002:24: F811 Redefinition of unused `app_client` from line 6
  tests/test_csv.py:67:33: F811 Redefinition of unused `app_client_with_cors` from line 6
  tests/test_csv.py:157:21: F811 Redefinition of unused `app_client_csv_max_mb_one` from line 5
  tests/test_csv.py:198:20: F811 Redefinition of unused `app_client_with_trace` from line 7
  tests/test_csv.py:209:53: F811 Redefinition of unused `app_client_with_trace` from line 7
  tests/test_csv.py:215:53: F811 Redefinition of unused `app_client_with_trace` from line 7
  tests/test_filters.py:102:11: F811 Redefinition of unused `test_through_filters_from_request` from line 81
  tests/test_html.py:19:19: F811 Redefinition of unused `app_client_two_attached_databases` from line 7
  tests/test_html.py:175:25: F811 Redefinition of unused `app_client_shorter_time_limit` from line 6
  tests/test_html.py:469:51: F811 Redefinition of unused `app_client` from line 4
  tests/test_html.py:797:26: F811 Redefinition of unused `app_client_base_url_prefix` from line 5
  tests/test_html.py:840:44: F811 Redefinition of unused `app_client_base_url_prefix` from line 5
  tests/test_html.py:850:51: F811 Redefinition of unused `app_client_base_url_prefix` from line 5
  tests/test_pagination.py:50:43: F821 Undefined name `parse_next`
  tests/test_pagination.py:82:7: F811 Redefinition of unused `KeysetPaginator` from line 36
  tests/test_plugins.py:115:15: E741 Ambiguous variable name: `l`
  tests/test_plugins.py:482:161: E501 Line too long (170 > 160 characters)
  tests/test_plugins.py:543:29: E741 Ambiguous variable name: `l`
  tests/test_plugins.py:563:161: E501 Line too long (170 > 160 characters)
  tests/test_plugins.py:940:62: E741 Ambiguous variable name: `l`
  tests/test_table_api.py:739:5: F811 Redefinition of unused `app_client_returned_rows_matches_page_size` from line 6
  tests/test_table_api.py:1066:45: F811 Redefinition of unused `app_client_with_trace` from line 5
  tests/test_table_html.py:484:29: E741 Ambiguous variable name: `l`
  tests/test_table_html.py:524:29: E741 Ambiguous variable name: `l`
  tests/test_table_html.py:675:161: E501 Line too long (165 > 160 characters)
  tests/test_table_html.py:897:161: E501 Line too long (164 > 160 characters)
  tests/test_table_html.py:902:161: E501 Line too long (164 > 160 characters)
  tests/test_utils.py:141:161: E501 Line too long (176 > 160 characters)
  Found 41 errors.
  ```
  Those "Redefinition of unused `app_client_two_attached_databases`" lines are caused because of the fixtures pattern I'm using here:

  https://github.com/simonw/datasette/blob/99ba05118891db9dc30f1dca22ad6709775560de/tests/test_html.py#L3-L20

  I could fix that by getting rid of `fixtures.py` and moving those into `conftest.py`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2090#issuecomment-1613346412
issue_url: https://api.github.com/repos/simonw/datasette/issues/2090
node_id: IC_kwDOBm6k_c5gKbZs
created_at: 2023-06-29T15:05:04Z
updated_at: 2023-06-29T15:05:04Z
author_association: OWNER
body:
  Decided to fix just those "Ambiguous variable name" ones:

  ```bash
  ruff check . | grep E741
  ```
  Then iterated through and fixed them all.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2091#issuecomment-1613360413
issue_url: https://api.github.com/repos/simonw/datasette/issues/2091
node_id: IC_kwDOBm6k_c5gKe0d
created_at: 2023-06-29T15:13:04Z
updated_at: 2023-06-29T15:13:04Z
author_association: OWNER
body:
  One problem: https://glitch.com/ still provides 3.7:
  ```
  $ python3 --version
  Python 3.7.10
  ```
  <img width="956" alt="image" src="https://github.com/simonw/datasette/assets/9599/e1fa44c0-4362-40e6-a2c7-d5b09b784c66">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2091#issuecomment-1613369355
issue_url: https://api.github.com/repos/simonw/datasette/issues/2091
node_id: IC_kwDOBm6k_c5gKhAL
created_at: 2023-06-29T15:18:34Z
updated_at: 2023-06-29T15:18:34Z
author_association: OWNER
body:
  Posted on the Glitch feedback forum about this here: https://support.glitch.com/t/upgrade-python-version-from-3-7-which-is-now-eol-to-something-more-recent/63011

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2092#issuecomment-1613381990
issue_url: https://api.github.com/repos/simonw/datasette/issues/2092
node_id: IC_kwDOBm6k_c5gKkFm
created_at: 2023-06-29T15:23:04Z
updated_at: 2023-06-29T15:26:28Z
author_association: OWNER
body:
  Felt lazy:
  ```bash
  symbex test_homepage -f tests/test_api.py | \
    llm -m 4 --system 'Change all of the == some integer tests in this code to isinstance(thing, int) instead'
  ```
  Output:
  ```python
  # File: tests/test_api.py Line: 26
  @pytest.mark.asyncio
  async def test_homepage(ds_client):
      response = await ds_client.get("/.json")
      assert response.status_code == 200
      assert "application/json; charset=utf-8" == response.headers["content-type"]
      data = response.json()
      assert isinstance(data.keys(), int)
      d = data["fixtures"]
      assert d["name"] == "fixtures"
      assert isinstance(d["tables_count"], int)
      assert isinstance(len(d["tables_and_views_truncated"]), int)
      assert d["tables_and_views_more"] is True
      # 4 hidden FTS tables + no_primary_key (hidden in metadata)
      assert isinstance(d["hidden_tables_count"], int)
      # 201 in no_primary_key, plus 6 in other hidden tables:
      assert isinstance(d["hidden_table_rows_sum"], int), data
      assert isinstance(d["views_count"], int)
  ```
  I'll use most of that and delete the obsoleted comments.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2052#issuecomment-1613778296
issue_url: https://api.github.com/repos/simonw/datasette/issues/2052
node_id: IC_kwDOBm6k_c5gME14
created_at: 2023-06-29T20:36:09Z
updated_at: 2023-06-29T20:36:09Z
author_association: CONTRIBUTOR
body:
  Ok @hydrosquall a couple things before this PR should be good to go:

  - Can we move `datasette/static/table-example-plugins.js` into `demos/plugins/static`?
  - For `datasetteManager.VERSION`, can we fill that in or just comment it out for now? Not sure how difficult it'll be to inject it server-side. I imagine we could also have a small build process with esbuild/rollup that just injects a version string into `manager.js` directly, so we don't have to worry about server-rendering (but that can be a future PR)

  In terms of how to integrate this into Datasette, a few options I can see working:

  - Push this as-is and figure it out before the next release
  - Hide this feature behind a settings flag (`--setting unstable-js-plugins on`) and use that setting to hide/show `<script src="{{ urls.static('datasette-manager.js') }}" defer></script>` in `base.html`

  I'll let @simonw decide which one to work with. I kindof like the idea of having an "unstable" opt-in process to enable JS plugins, to give us time to try it out with a wide variety of plugins until we feel its ready.

  I'm also curious to see how "plugins for a plugin' would work, like #1542. For example, if the leaflet plugin showed default markers, but also included its own hook for other plugins to add more markers/styling. I'm imagine that the individual plugin would re-create their own plugin system compared to this, since handling "plugins of plugins" at the top with Datasette seems really convoluted. 

  Also for posterity, here's a list of Simon's Datasette plugins that use "extra_js_urls()", which probably means they can be ported/re-written to use this new plugin system:

  - [`datasette-vega`](https://github.com/simonw/datasette-vega/blob/00de059ab1ef77394ba9f9547abfacf966c479c4/datasette_vega/__init__.py#L25)
  - [`datasette-cluster-map`](https://github.com/simonw/datasette-cluster-map/blob/795d25ad9ff6cba0307191f44fecc8f8070bef5c/datasette_cluster_map/__init__.py#L14)
  - [`datasette-leaflet-geojson`](https://github.com/simonw/datasette-leaflet-geojson/blob/64713aa497750400b9ac2c12e8bb6ffab8eb77f3/datasette_leaflet_geojson/__init__.py#L47)
  - [`datasette-pretty-traces`](https://github.com/simonw/datasette-pretty-traces/blob/5219d65eca3d7d7a73bb9d3120df42fe046a1315/datasette_pretty_traces/__init__.py#L5)
  - [`datasette-youtube-embed`](https://github.com/simonw/datasette-youtube-embed/blob/4b4a0d7e58ebe15f47e9baf68beb9908c1d899da/datasette_youtube_embed/__init__.py#L55)
  - [`datasette-leaflet-freedraw`](https://github.com/simonw/datasette-leaflet-freedraw/blob/8f28c2c2080ec9d29f18386cc6a2573a1c8fbde7/datasette_leaflet_freedraw/__init__.py#L66)
  - [`datasette-hovercards`](https://github.com/simonw/datasette-hovercards/blob/9439ba46b7140fb03223faff0d21aeba5615a287/datasette_hovercards/__init__.py#L5)
  - [`datasette-mp3-audio`](https://github.com/simonw/datasette-mp3-audio/blob/4402168792f452a46ab7b488e40ec49cd4b12185/datasette_mp3_audio/__init__.py#L6)
  - [`datasette-geojson-map`](https://github.com/simonw/datasette-geojson-map/blob/32af5f1fd1a07278bbf8071fbb20a61e0f613246/datasette_geojson_map/__init__.py#L30)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2093#issuecomment-1613887492
issue_url: https://api.github.com/repos/simonw/datasette/issues/2093
node_id: IC_kwDOBm6k_c5gMfgE
created_at: 2023-06-29T22:40:25Z
updated_at: 2023-06-29T22:40:25Z
author_association: OWNER
body:
  I'm strongly in favour of combining settings, configuration and plugin configuration.

  I'm not keen on mixing in metadata as well - that feels like a different concept to me, and I'm unhappy with how that's already had things like plugin settings leak into it.

  I'm not yet sold on TOML - I actually find it less intuitive than YAML, surprisingly. They all have their warts I guess.

  Datasette already has the ability to consume JSON or YAML for metadata - maybe it could grow TOML support too? That way users could have a `datasette.json` or `datasette.yaml` or `datasette.toml` file depending on their preference.

  In terms of metadata: since that's means to be driven by a plugin hook anyway, maybe one of the potential sources of metadata is a `metadata` nested object in that `datasette.*` configuration file. Or you can have it in a separate `metadata.json` or bundled into the SQLite database or some other plugin-driven mechanism.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2093#issuecomment-1613889979
issue_url: https://api.github.com/repos/simonw/datasette/issues/2093
node_id: IC_kwDOBm6k_c5gMgG7
created_at: 2023-06-29T22:44:08Z
updated_at: 2023-06-30T13:25:39Z
author_association: OWNER
body:
  I do like also being able to set options using command line options though - for things like SQL time limits I'd much rather be able to throw on `--setting sql_time_limit_ms 10000` than have to save a config file to disk.

  So I'd want to support both. Which maybe means also having a way to set plugin options with CLI options. `datasette publish` kind of has that ability already:

  ```
  datasette publish heroku my_database.db \
      --name my-heroku-app-demo \
      --install=datasette-auth-github \
      --plugin-secret datasette-auth-github client_id your_client_id \
      --plugin-secret datasette-auth-github client_secret your_client_secret
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2093#issuecomment-1613895188
issue_url: https://api.github.com/repos/simonw/datasette/issues/2093
node_id: IC_kwDOBm6k_c5gMhYU
created_at: 2023-06-29T22:51:53Z
updated_at: 2023-06-29T22:51:53Z
author_association: CONTRIBUTOR
body:
  I agree with not liking `metadata.json` stuff in a `datasette.*` config file. Editing description of a table/column in a file like `datasette.*` seems odd to me. 

  Though since plugin configuration currently lives in `metadata.json`, I think it should be removed from there and placed in `datasette.*`, at least for top-level config like `datasette-auth-github`'s config. Keeping `metadata.json` strictly for documentation/licensing/column units makes sense to me, but anything plugin related should be in some config file, like `datasette.*`.

  And ya, supporting both `datasette.*` and CLI flags makes a lot of sense to me. Any `--setting` flag should override anything in `datasette.*` for easier debugging, with possibly a warning message so people don't get confused. Same with `--port` and a port defined in `datasette.*`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2093#issuecomment-1613896210
issue_url: https://api.github.com/repos/simonw/datasette/issues/2093
node_id: IC_kwDOBm6k_c5gMhoS
created_at: 2023-06-29T22:53:33Z
updated_at: 2023-06-29T22:53:33Z
author_association: CONTRIBUTOR
body:
  Maybe we can have a separate issue for revamping `metadata.json`? A `datasette_metadata` table or the `sqlite-docs` extension seem like two reasonable additions that we can work through. Storing metadata inside a SQLite database makes sense, but I don't think storing `datasette.*` style config (ex ports, settings, etc.) inside a SQLite DB makes sense, since it's very environment-dependent

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2093#issuecomment-1614652001
issue_url: https://api.github.com/repos/simonw/datasette/issues/2093
node_id: IC_kwDOBm6k_c5gPaJh
created_at: 2023-06-30T13:27:13Z
updated_at: 2023-06-30T13:27:13Z
author_association: OWNER
body:
  I agree, settings in the DB doesn't make sense but metadata does.

  On the JSON v YAML v TOML issue I just spotted Caddy has a concept of config adapters which they use to resolve exactly that problem: https://caddyserver.com/docs/config-adapters

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2052#issuecomment-1615997736
issue_url: https://api.github.com/repos/simonw/datasette/issues/2052
node_id: IC_kwDOBm6k_c5gUiso
created_at: 2023-07-01T16:55:24Z
updated_at: 2023-07-01T16:55:24Z
author_association: CONTRIBUTOR
body:
  > Ok @hydrosquall a couple things before this PR should be good to go:

  Thank you @asg017 ! I've pushed both suggested changes onto this branch.

  > Not sure how difficult it'll be to inject it server-side

  If we are OK with having a build system, it would free me up to do do many things! We could make datasette-manager.js a server-side rendered file as a "template" instead of having it as a static JS file, but I'm not sure it's worth the extra jump in complexity / loss of syntax highlighting in the JS file.

  In the short-term, I could see an intermediary solution where a unit test in the preferred language was able to read both `version.py` and `datasette-manager.js`, and make sure that the strings versions are in sync. (This assumes that we want the manager and datasette's versions to be synced, and not decoupled). Since the version is not changing very often, a "manual sync" might be good enough. 

  > In terms of how to integrate this into Datasette, a few options I can see working:

  This sounds good to me. I'm not sure how to add a settings flag, but will be interested to see the PR that adds support for it.

  > I'm also curious to see how "plugins for a plugin' would work

  I'm comfortable to wait until we have a realistic usecase for this. In the short term, I think we could give plugins a way to grant access to a "public API of other plugins", and also ask to be notified when plugins with other names have loaded, but don't picture the datasette manager getting more involved than that. 

  > here's a list of Simon's Datasette plugins that use "extra_js_urls()"

  Neat, thanks for compiling this list! Just curious, is there a query that can be used to compile this programmatically, or did you identify these through memory?

  > I want to make a javascript plugin on top of the code-mirror editor to make a few things nicer (function auto-complete, table/column descriptions, etc.)

  I look forward to trying this out 👍 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2052#issuecomment-1616095810
issue_url: https://api.github.com/repos/simonw/datasette/issues/2052
node_id: IC_kwDOBm6k_c5gU6pC
created_at: 2023-07-01T20:31:31Z
updated_at: 2023-07-01T20:31:31Z
author_association: CONTRIBUTOR
body:
  > Just curious, is there a query that can be used to compile this programmatically, or did you identify these through memory?

  I just did a github search for `user:simonw "def extra_js_urls("` ! Though I'm sure other plugins made by people other than Simon also exist out there https://github.com/search?q=user%3Asimonw+%22def+extra_js_urls%28%22&type=code

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2093#issuecomment-1616195496
issue_url: https://api.github.com/repos/simonw/datasette/issues/2093
node_id: IC_kwDOBm6k_c5gVS-o
created_at: 2023-07-02T00:06:54Z
updated_at: 2023-07-02T00:07:17Z
author_association: NONE
body:
  I'm not keen on requiring metadata to be within the database. I commonly have multiple DBs, from various sources, and having one config file to provide the metadata works out very well. I use Datasette with databases where I'm not the original source, needing to mutate them to add a metadata table or sqlite-docs makes me uncomfortable.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2093#issuecomment-1616286848
issue_url: https://api.github.com/repos/simonw/datasette/issues/2093
node_id: IC_kwDOBm6k_c5gVpSA
created_at: 2023-07-02T02:17:46Z
updated_at: 2023-07-02T02:17:46Z
author_association: CONTRIBUTOR
body:
  Storing metadata in the database won't be required. I imagine there'll be many different ways to store metadata, including any possible `datasette_metadata` or sqlite-docs, or the older metadata.json way. 

  The next question will be how precedence should work - i'd imagine metadata.json > plugins > datasette_metadata > sqlite-docs

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/562#issuecomment-1616782404
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/562
node_id: IC_kwDOCGYnMM5gXiRE
created_at: 2023-07-02T19:24:14Z
updated_at: 2023-07-02T19:26:39Z
author_association: OWNER
body:
  [Dataclasses](https://docs.python.org/3/library/dataclasses.html) were added in Python 3.7 and `sqlite-utils` was originally written for Python 3.6 - but both 3.6 and 3.7 are EOL now.

  The thing that makes Dataclasses particularly interesting is the potential to use type annotations with them to help specify the types of the related SQLite columns.

  Example for https://datasette.io/content/users

  ```sql
  CREATE TABLE [users] (
     [login] TEXT,
     [id] INTEGER PRIMARY KEY,
     [node_id] TEXT,
     [avatar_url] TEXT,
     [gravatar_id] TEXT,
     [html_url] TEXT,
     [type] TEXT,
     [site_admin] INTEGER,
     [name] TEXT
  );
  ```
  And the dataclass:
  ```python
  from dataclasses import dataclass

  @dataclass
  class User:
      id: int
      login: str
      node_id: str
      avatar_url: str
      gravatar_id: str
      html_url: str
      type: str
      site_admin: int
      name: str
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2087#issuecomment-1616853644
issue_url: https://api.github.com/repos/simonw/datasette/issues/2087
node_id: IC_kwDOBm6k_c5gXzqM
created_at: 2023-07-02T22:00:48Z
updated_at: 2023-07-02T22:00:48Z
author_association: CONTRIBUTOR
body:
  I just saw in the docs that Dasette auto-detects `settings.json`:

  > settings.json - settings that would normally be passed using --setting - here they should be stored as a JSON object of key/value pairs
  > [*Source*](https://docs.datasette.io/en/stable/settings.html#:~:text=settings.json%20%2D%20settings%20that%20would%20normally%20be%20passed%20using%20%2D%2Dsetting%20%2D%20here%20they%20should%20be%20stored%20as%20a%20JSON%20object%20of%20key/value%20pairs)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/563#issuecomment-1617395444
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/563
node_id: IC_kwDOCGYnMM5gZ370
created_at: 2023-07-03T05:44:43Z
updated_at: 2023-07-03T05:44:43Z
author_association: OWNER
body:
  Documentation at the bottom of this section: https://sqlite-utils.datasette.io/en/latest/cli.html#inserting-csv-or-tsv-data

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/565#issuecomment-1618380888
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/565
node_id: IC_kwDOCGYnMM5gdohY
created_at: 2023-07-03T14:09:11Z
updated_at: 2023-07-03T14:09:31Z
author_association: OWNER
body:
  For the CLI:

  ```bash
  sqlite-utils rename-table data.db old_table_name new_table_name
  ```

  For the Python code, should it go on Table or on Database?
  ```python
  db["foo"].rename_table("bar")

  db.rename_table("foo", "bar")
  ```
  I think I like the second better, it's slightly more clear.

  Also need a design for an option for the `.transform()` method to indicate that the new table should be created with a new name without dropping the old one.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1153#issuecomment-1627395947
issue_url: https://api.github.com/repos/simonw/datasette/issues/1153
node_id: IC_kwDOBm6k_c5hABdr
created_at: 2023-07-08T16:35:45Z
updated_at: 2023-07-08T16:35:45Z
author_association: OWNER
body:
  I was inspired to finally address this after seeing `sphinx-inline-tabs` at work in https://webcolors.readthedocs.io/en/latest/install.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1153#issuecomment-1627396450
issue_url: https://api.github.com/repos/simonw/datasette/issues/1153
node_id: IC_kwDOBm6k_c5hABli
created_at: 2023-07-08T16:38:58Z
updated_at: 2023-07-08T16:38:58Z
author_association: OWNER
body:
  I'm using `cog` and this utility function to generate the YAML/JSON tabs:

  https://github.com/simonw/datasette/blob/3b336d8071fb5707bd006de1d614f701d20246a3/docs/metadata_doc.py#L1-L13

  Example usage:

  https://github.com/simonw/datasette/blob/3b336d8071fb5707bd006de1d614f701d20246a3/docs/metadata.rst?plain=1#L17-L53

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1153#issuecomment-1627396658
issue_url: https://api.github.com/repos/simonw/datasette/issues/1153
node_id: IC_kwDOBm6k_c5hABoy
created_at: 2023-07-08T16:40:07Z
updated_at: 2023-07-08T16:40:07Z
author_association: OWNER
body:
  https://docs.datasette.io/en/latest/metadata.html

  ![inline-tabs](https://github.com/simonw/datasette/assets/9599/975bdff5-74ac-451e-92c3-a7dd05d4b862)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1153#issuecomment-1627447478
issue_url: https://api.github.com/repos/simonw/datasette/issues/1153
node_id: IC_kwDOBm6k_c5hAOC2
created_at: 2023-07-08T17:59:25Z
updated_at: 2023-07-08T17:59:25Z
author_association: OWNER
body:
  Hit a problem:
  ```
  Exception occurred:
    File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/docutils/nodes.py", line 2028, in unknown_visit
      raise NotImplementedError(
  NotImplementedError: <class 'docutils.writers.docutils_xml.XMLTranslator'> visiting unknown node type: TabContainer
  The full traceback has been saved in /tmp/sphinx-err-tfujyw1h.log, if you want to report the issue to the developers.
  Please also report this if it was a user error, so that a better error message can be provided next time.
  A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
  ```
  That's happening here: https://github.com/simonw/datasette/blob/0183e1a72d4d93b1d9a9363f4d47fcc0b5d5849c/.github/workflows/deploy-latest.yml#L42-L48

  My https://github.com/simonw/sphinx-to-sqlite tool can't handle the new `TabContainer` elements introduced by `sphinx-inline-tabs`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1153#issuecomment-1627447750
issue_url: https://api.github.com/repos/simonw/datasette/issues/1153
node_id: IC_kwDOBm6k_c5hAOHG
created_at: 2023-07-08T18:00:56Z
updated_at: 2023-07-08T18:00:56Z
author_association: OWNER
body:
  Actually no it's in `sphinx-build`:
  ```
  % sphinx-build -b xml . _build
  Running Sphinx v6.1.3
  building [mo]: targets for 0 po files that are out of date
  writing output... 
  building [xml]: targets for 28 source files that are out of date
  updating environment: [new config] 28 added, 0 changed, 0 removed
  reading sources... [100%] writing_plugins                                                                                                                          
  looking for now-outdated files... none found
  pickling environment... done
  checking consistency... done
  preparing documents... done
  writing output... [  3%] authentication                                                                                                                            
  Exception occurred:
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/docutils/nodes.py", line 2028, in unknown_visit
      raise NotImplementedError(
  NotImplementedError: <class 'docutils.writers.docutils_xml.XMLTranslator'> visiting unknown node type: TabContainer
  The full traceback has been saved in /var/folders/x6/31xf1vxj0nn9mxqq8z0mmcfw0000gn/T/sphinx-err-1wkxmkji.log, if you want to report the issue to the developers.
  Please also report this if it was a user error, so that a better error message can be provided next time.
  A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1153#issuecomment-1627448180
issue_url: https://api.github.com/repos/simonw/datasette/issues/1153
node_id: IC_kwDOBm6k_c5hAON0
created_at: 2023-07-08T18:03:31Z
updated_at: 2023-07-08T18:03:31Z
author_association: OWNER
body:
  Relevant code: https://github.com/docutils/docutils/blob/3b53ded52bc439d8068b6ecb20ea0a761247e479/docutils/docutils/nodes.py#L2021-L2031

  ```python
      def unknown_visit(self, node):
          """
          Called when entering unknown `Node` types.

          Raise an exception unless overridden.
          """
          if (self.document.settings.strict_visitor
              or node.__class__.__name__ not in self.optional):
              raise NotImplementedError(
                  '%s visiting unknown node type: %s'
                  % (self.__class__, node.__class__.__name__))
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1153#issuecomment-1627448542
issue_url: https://api.github.com/repos/simonw/datasette/issues/1153
node_id: IC_kwDOBm6k_c5hAOTe
created_at: 2023-07-08T18:05:44Z
updated_at: 2023-07-08T18:05:44Z
author_association: OWNER
body:
  Running with `-P` opens a debugger when it hits the error:
  ```bash
  sphinx-build -P -b xml . _build
  ```
  ```
  (Pdb) list
  2023 	
  2024 	        Raise an exception unless overridden.
  2025 	        """
  2026 	        if (self.document.settings.strict_visitor
  2027 	            or node.__class__.__name__ not in self.optional):
  2028 ->	            raise NotImplementedError(
  2029 	                '%s visiting unknown node type: %s'
  2030 	                % (self.__class__, node.__class__.__name__))
  2031 	
  2032 	    def unknown_departure(self, node):
  2033 	        """
  (Pdb) self.optional
  ('meta',)
  (Pdb) node.__class__.__name__
  'TabContainer'
  (Pdb) self.document.settings.strict_visitor
  (Pdb) type(self.document.settings.strict_visitor)
  <class 'NoneType'>
  ```
  So if I can get `TabContainer` into that `self.optional` list I'll have fixed this problem.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1153#issuecomment-1627450852
issue_url: https://api.github.com/repos/simonw/datasette/issues/1153
node_id: IC_kwDOBm6k_c5hAO3k
created_at: 2023-07-08T18:17:35Z
updated_at: 2023-07-08T18:17:35Z
author_association: OWNER
body:
  I figured out a workaround:
  ```python
  extensions = [
      "sphinx.ext.extlinks",
      "sphinx.ext.autodoc",
      "sphinx_copybutton",
  ]
  if not os.environ.get("DISABLE_SPHINX_INLINE_TABS"):
      extensions += ["sphinx_inline_tabs"]
  ```
  That way I can run `sphinx-build -b xml . _build` successfully if I set that environment variable.

  I get some noisy warnings, but it runs OK. And the resulting `docs.db` file has rows like this, which I think are fine:

  <img width="1533" alt="image" src="https://github.com/simonw/datasette/assets/9599/0a289f53-2c6c-43e9-bda1-d4380383a02f">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1153#issuecomment-1627451646
issue_url: https://api.github.com/repos/simonw/datasette/issues/1153
node_id: IC_kwDOBm6k_c5hAPD-
created_at: 2023-07-08T18:21:24Z
updated_at: 2023-07-08T18:21:24Z
author_association: OWNER
body:
  This one was tricky:

  <img width="764" alt="image" src="https://github.com/simonw/datasette/assets/9599/0fe2886b-4e07-4539-a254-4c47f02c3f65">

  I wanted complete control over the YAML example here, so I could ensure it used multi-line strings correctly.

  I ended up changing my cog helper function to this:

  ```python
  import json
  import textwrap
  from yaml import safe_dump
  from ruamel.yaml import round_trip_load


  def metadata_example(cog, data=None, yaml=None):
      assert data or yaml, "Must provide data= or yaml="
      assert not (data and yaml), "Cannot use data= and yaml="
      output_yaml = None
      if yaml:
          # dedent it first
          yaml = textwrap.dedent(yaml).strip()
          # round_trip_load to preserve key order:
          data = round_trip_load(yaml)
          output_yaml = yaml
      else:
          output_yaml = safe_dump(data, sort_keys=False)
      cog.out("\n.. tab:: YAML\n\n")
      cog.out("    .. code-block:: yaml\n\n")
      cog.out(textwrap.indent(output_yaml, "        "))
      cog.out("\n\n.. tab:: JSON\n\n")
      cog.out("    .. code-block:: json\n\n")
      cog.out(textwrap.indent(json.dumps(data, indent=2), "        "))
      cog.out("\n")
  ```
  This allows me to call it ith YAML in some places:
  ```
  .. [[[cog
      metadata_example(cog, yaml="""
      databases:
        fixtures:
          queries:
            neighborhood_search:
              fragment: fragment-goes-here
              hide_sql: true
              sql: |-
                select neighborhood, facet_cities.name, state
                from facetable join facet_cities on facetable.city_id = facet_cities.id
                where neighborhood like '%' || :text || '%' order by neighborhood;
      """)
  .. ]]]
  ```
  I had to introduce https://pypi.org/project/ruamel.yaml/ as a dependency here in order to load YAML from disk while maintaining key order.

  I'm still using `safe_dump(data, sort_keys=False)` from PyYAML as I couldn't get the result I wanted for outputting YAML from an input of JSON using PyYAML.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1153#issuecomment-1627455892
issue_url: https://api.github.com/repos/simonw/datasette/issues/1153
node_id: IC_kwDOBm6k_c5hAQGU
created_at: 2023-07-08T18:39:19Z
updated_at: 2023-07-08T18:39:19Z
author_association: OWNER
body:
  ```
  ERROR: Could not find a version that satisfies the requirement Sphinx==6.1.3; extra == "docs" (from datasette[docs,test]) (from versions: 0.1.61611, 0.1.61798, 0.1.61843, 0.1.61945, 0.1.61950, 0.2, 0.3, 0.4, 0.4.1, 0.4.2, 0.4.3, 0.5, 0.5.1, 0.5.2b1, 0.5.2, 0.6b1, 0.6, 0.6.1, 0.6.2, 0.6.3, 0.6.4, 0.6.5, 0.6.6, 0.6.7, 1.0b1, 1.0b2, 1.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.0.5, 1.0.6, 1.0.7, 1.0.8, 1.1, 1.1.1, 1.1.2, 1.1.3, 1.2b1, 1.2b2, 1.2b3, 1.2, 1.2.1, 1.2.2, 1.2.3, 1.3b1, 1.3b2, 1.3b3, 1.3, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.3.5, 1.3.6, 1.4a1, 1.4b1, 1.4, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 1.4.5, 1.4.6, 1.4.7, 1.4.8, 1.4.9, 1.5a1, 1.5a2, 1.5b1, 1.5, 1.5.1, 1.5.2, 1.5.3, 1.5.4, 1.5.5, 1.5.6, 1.6b1, 1.6b2, 1.6b3, 1.6.1, 1.6.2, 1.6.3, 1.6.4, 1.6.5, 1.6.6, 1.6.7, 1.7.0b1, 1.7.0b2, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6, 1.7.7, 1.7.8, 1.7.9, 1.8.0b1, 1.8.0, 1.8.1, 1.8.2, 1.8.3, 1.8.4, 1.8.5, 1.8.6, 2.0.0b1, 2.0.0b2, 2.0.0, 2.0.1, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.2.1, 2.2.2, 2.3.0, 2.3.1, 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.4.4, 2.4.5, 3.0.0b1, 3.0.0, 3.0.1, 3.0.2, 3.0.3, 3.0.4, 3.1.0, 3.1.1, 3.1.2, 3.2.0, 3.2.1, 3.3.0, 3.3.1, 3.4.0, 3.4.1, 3.4.2, 3.4.3, 3.5.0, 3.5.1, 3.5.2, 3.5.3, 3.5.4, 4.0.0b1, 4.0.0b2, 4.0.0, 4.0.1, 4.0.2, 4.0.3, 4.1.0, 4.1.1, 4.1.2, 4.2.0, 4.3.0, 4.3.1, 4.3.2, 4.4.0, 4.5.0, 5.0.0b1, 5.0.0, 5.0.1, 5.0.2, 5.1.0, 5.1.1, 5.2.0, 5.2.0.post0, 5.2.1, 5.2.2, 5.2.3, 5.3.0)
  ERROR: No matching distribution found for Sphinx==6.1.3; extra == "docs"
  ```
  I'm going to drop Python 3.7.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1153#issuecomment-1627478910
issue_url: https://api.github.com/repos/simonw/datasette/issues/1153
node_id: IC_kwDOBm6k_c5hAVt-
created_at: 2023-07-08T20:01:19Z
updated_at: 2023-07-08T20:01:19Z
author_association: OWNER
body:
  Some examples:

  -  https://docs.datasette.io/en/latest/sql_queries.html#canned-queries
  - https://docs.datasette.io/en/latest/sql_queries.html#canned-query-parameters
  - https://docs.datasette.io/en/latest/authentication.html#access-to-an-instance
  - https://docs.datasette.io/en/latest/facets.html#facets-in-metadata
  - https://docs.datasette.io/en/latest/full_text_search.html#configuring-full-text-search-for-a-table-or-view
  - https://docs.datasette.io/en/latest/metadata.html
  - https://docs.datasette.io/en/latest/custom_templates.html#custom-css-and-javascript
  - https://docs.datasette.io/en/latest/plugins.html#plugin-configuration

  I need to fix this section: https://docs.datasette.io/en/latest/writing_plugins.html#writing-plugins-that-accept-configuration

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1153#issuecomment-1627480353
issue_url: https://api.github.com/repos/simonw/datasette/issues/1153
node_id: IC_kwDOBm6k_c5hAWEh
created_at: 2023-07-08T20:09:48Z
updated_at: 2023-07-08T20:09:48Z
author_association: OWNER
body:
  https://docs.datasette.io/en/latest/writing_plugins.html#writing-plugins-that-accept-configuration is fixed now.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/pocket-to-sqlite/issues/12#issuecomment-1627563202
issue_url: https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/12
node_id: IC_kwDODLZ_YM5hAqTC
created_at: 2023-07-09T01:14:27Z
updated_at: 2023-07-09T01:14:27Z
author_association: MEMBER
body:
  I tested this locally with `python -m build` and then `pip install ...whl` in a fresh virtual environment.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/pocket-to-sqlite/issues/12#issuecomment-1627564127
issue_url: https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/12
node_id: IC_kwDODLZ_YM5hAqhf
created_at: 2023-07-09T01:19:42Z
updated_at: 2023-07-09T01:19:42Z
author_association: MEMBER
body:
  https://github.com/dogsheep/pocket-to-sqlite/tree/0.2.3 and https://pypi.org/project/pocket-to-sqlite/0.2.3/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/566#issuecomment-1627598570
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/566
node_id: IC_kwDOCGYnMM5hAy7q
created_at: 2023-07-09T04:13:34Z
updated_at: 2023-07-09T04:13:34Z
author_association: OWNER
body:
  On consulting https://pypi.org/project/tabulate/ it looks like most of those formats don't actually makes sense without headers - so the right thing here might be to raise an error if `--fmt` and `--no-headers` are used at the same time.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2052#issuecomment-1629337927
issue_url: https://api.github.com/repos/simonw/datasette/issues/2052
node_id: IC_kwDOBm6k_c5hHblH
created_at: 2023-07-10T16:43:38Z
updated_at: 2023-07-10T16:44:23Z
author_association: OWNER
body:
  I tried running this locally just now. I made one edit:
  ```diff
  diff --git a/demos/plugins/example_js_manager_plugins.py b/demos/plugins/example_js_manager_plugins.py
  index 7bdb9f3f..f9dfa8e6 100644
  --- a/demos/plugins/example_js_manager_plugins.py
  +++ b/demos/plugins/example_js_manager_plugins.py
  @@ -15,6 +15,6 @@ def extra_js_urls(view_name):
       if view_name in PERMITTED_VIEWS:
           return [
               {
  -                "url": f"/-/demos/plugins/static/table-example-plugins.js",
  +                "url": f"/static/table-example-plugins.js",
               }
           ]
  ```
  And then started it running like this:
  ```bash
  wget https://datasette.io/content.db
  ```
  ```bash
  datasette content.db \
    --plugins-dir demos/plugins \
    --static static:datasette/demos/plugins/static
  ```

  It didn't quite work for me - I got this error on a table page:

  <img width="681" alt="image" src="https://github.com/simonw/datasette/assets/9599/02f5c16d-ea43-4c36-920d-4b14406645b5">

  And this error on a query page:

  <img width="689" alt="image" src="https://github.com/simonw/datasette/assets/9599/994db3a0-502a-4f36-8581-c47556600d63">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2052#issuecomment-1630776144
issue_url: https://api.github.com/repos/simonw/datasette/issues/2052
node_id: IC_kwDOBm6k_c5hM6tQ
created_at: 2023-07-11T12:54:03Z
updated_at: 2023-07-11T12:54:03Z
author_association: CONTRIBUTOR
body:
  Thanks for the review and the code pointers @simonw - I've made the suggested edits, fixed the renamed variable, and confirmed that the panels still render on the `table` and `database` views. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2052#issuecomment-1632867333
issue_url: https://api.github.com/repos/simonw/datasette/issues/2052
node_id: IC_kwDOBm6k_c5hU5QF
created_at: 2023-07-12T16:38:27Z
updated_at: 2023-10-12T23:52:24Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2052?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  All modified lines are covered by tests :white_check_mark:
  > Comparison is base [(`3feed1f`)](https://app.codecov.io/gh/simonw/datasette/commit/3feed1f66e2b746f349ee56970a62246a18bb164?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.46% compared to head [(`8ae479c`)](https://app.codecov.io/gh/simonw/datasette/pull/2052?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69%.
  > Report is 112 commits behind head on main.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@            Coverage Diff             @@
  ##             main    #2052      +/-   ##
  ==========================================
  + Coverage   92.46%   92.69%   +0.22%     
  ==========================================
    Files          38       40       +2     
    Lines        5750     6047     +297     
  ==========================================
  + Hits         5317     5605     +288     
  - Misses        433      442       +9     
  ```


  [see 19 files with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2052/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)


  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2052?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2101#issuecomment-1634443907
issue_url: https://api.github.com/repos/simonw/datasette/issues/2101
node_id: IC_kwDOBm6k_c5ha6KD
created_at: 2023-07-13T15:24:17Z
updated_at: 2023-07-13T15:24:17Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/blob/0f7192b6154edb576c41b55bd3f2a3f53e5f436a/datasette/views/table.py#L486-L506

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2102#issuecomment-1636036312
issue_url: https://api.github.com/repos/simonw/datasette/issues/2102
node_id: IC_kwDOBm6k_c5hg-7Y
created_at: 2023-07-14T15:37:14Z
updated_at: 2023-07-14T15:37:14Z
author_association: OWNER
body:
  I think I made this decision because I was thinking about default deny: obviously if a user has been denied access to a database. It doesn't make sense that they could access tables within it.

  But now that I am spending more time with authentication tokens, which default to denying everything, except for the things that you have explicitly listed, this policy, no longer makes as much sense.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2102#issuecomment-1636040164
issue_url: https://api.github.com/repos/simonw/datasette/issues/2102
node_id: IC_kwDOBm6k_c5hg_3k
created_at: 2023-07-14T15:40:21Z
updated_at: 2023-07-14T15:40:21Z
author_association: OWNER
body:
  Relevant code: 
  https://github.com/simonw/datasette/blob/0f7192b6154edb576c41b55bd3f2a3f53e5f436a/datasette/app.py#L822-L855

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2102#issuecomment-1636042066
issue_url: https://api.github.com/repos/simonw/datasette/issues/2102
node_id: IC_kwDOBm6k_c5hhAVS
created_at: 2023-07-14T15:41:54Z
updated_at: 2023-07-14T15:42:32Z
author_association: OWNER
body:
  I tried some code spelunking and came across https://github.com/simonw/datasette/commit/d6e03b04302a0852e7133dc030eab50177c37be7 which says:

  > - If you have table permission but not database permission you can now view the table page

  Refs:
  - #832 

  Which suggests that my initial design decision wasn't what appears to be implemented today.

  Needs more investigation.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2102#issuecomment-1636053060
issue_url: https://api.github.com/repos/simonw/datasette/issues/2102
node_id: IC_kwDOBm6k_c5hhDBE
created_at: 2023-07-14T15:51:36Z
updated_at: 2023-07-14T16:14:05Z
author_association: OWNER
body:
  This might only be an issue with the code that checks `_r` on actors.

  https://github.com/simonw/datasette/blob/0f7192b6154edb576c41b55bd3f2a3f53e5f436a/datasette/default_permissions.py#L185-L222

  Added in https://github.com/simonw/datasette/commit/bcc781f4c50a8870e3389c4e60acb625c34b0317 - refs:

  - #1855 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2102#issuecomment-1636093730
issue_url: https://api.github.com/repos/simonw/datasette/issues/2102
node_id: IC_kwDOBm6k_c5hhM8i
created_at: 2023-07-14T16:26:27Z
updated_at: 2023-07-14T16:32:49Z
author_association: OWNER
body:
  Here's that crucial comment:

  >  If _r is defined then we use those to further restrict the actor.
  >
  >Crucially, we only use this to say NO (return False) - we never use it to return YES (True) because that might over-ride other restrictions placed on this actor

  So that's why I implemented it like this.

  The goal here is to be able to issue a token which can't do anything _more_ than the actor it is associated with, but CAN be configured to do less.

  So I think the solution here is for the `_r` checking code to perhaps implement its own view cascade logic - it notices if you have `view-table` and consequently fails to block `view-table` and `view-instance`.

  I'm not sure that's going to work though - would that mean that granting `view-table` grants `view-database` in a surprising and harmful way?

  Maybe that's OK: if you have `view-database` but permission checks fail for individual tables and queries you shouldn't be able to see a thing that you shouldn't. Need to verify that though.

  Also, do `Permission` instances have enough information to implement this kind of cascade without hard-coding anything? 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2087#issuecomment-1636134091
issue_url: https://api.github.com/repos/simonw/datasette/issues/2087
node_id: IC_kwDOBm6k_c5hhWzL
created_at: 2023-07-14T17:02:03Z
updated_at: 2023-07-14T17:02:03Z
author_association: NONE
body:
  @asg017  - the docs say that the autodetection only occurs in configuration directory mode. I for one would also be interested in the `--settings settings.json` feature. 

  For context, I am developing a large database for use with Datasette, but the database lives in a different network volume than my source code, since the volume in which my source code lives is aggressively backed up, while the location where the database lives is meant for temporary files and is not as aggressively backed up (since the backups would get unreasonably large).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2104#issuecomment-1638552567
issue_url: https://api.github.com/repos/simonw/datasette/issues/2104
node_id: IC_kwDOBm6k_c5hqlP3
created_at: 2023-07-17T17:14:20Z
updated_at: 2023-07-17T17:14:20Z
author_association: OWNER
body:
  Relevant code: https://github.com/simonw/datasette/blob/0f7192b6154edb576c41b55bd3f2a3f53e5f436a/datasette/database.py#L391-L451

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2102#issuecomment-1638567228
issue_url: https://api.github.com/repos/simonw/datasette/issues/2102
node_id: IC_kwDOBm6k_c5hqo08
created_at: 2023-07-17T17:24:19Z
updated_at: 2023-07-17T17:25:12Z
author_association: OWNER
body:
  Confirmed that this is an issue with regular Datasette signed tokens as well. I created one on https://latest.datasette.io/-/create-token with these details:
  ```json
  {
      "_r": {
          "r": {
              "fixtures": {
                  "sortable": [
                      "vt"
                  ]
              }
          }
      },
      "a": "root",
      "d": 3600,
      "t": 1689614483
  }
  ```
  Run like this:
  ```
  curl -H 'Authorization: Bearer dstok_eyJhIjoicm9vdCIsInQiOjE2ODk2MTQ0ODMsImQiOjM2MDAsIl9yIjp7InIiOnsiZml4dHVyZXMiOnsic29ydGFibGUiOlsidnQiXX19fX0.n-VGxxawz1Q0WK7sqLfhXUgcvY0' \
    https://latest.datasette.io/fixtures/sortable.json
  ```
  Returned an HTML Forbidden page:
  ```html
  <!DOCTYPE html>
  <html>
  <head>
      <title>Forbidden</title>
      <link rel="stylesheet" href="/-/static/app.css?d59929">
  ...
  ```
  Same token againts `/-/actor.json` returns:
  ```json
  {
    "actor": {
      "id": "root",
      "token": "dstok",
      "_r": {
        "r": {
          "fixtures": {
            "sortable": [
              "vt"
            ]
          }
        }
      },
      "token_expires": 1689618083
    }
  }
  ```
  Reminder - `"_r"` means restrict, `"r"` means resource.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1638910473
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/567
node_id: IC_kwDOCGYnMM5hr8oJ
created_at: 2023-07-17T21:27:41Z
updated_at: 2023-07-17T21:27:41Z
author_association: CONTRIBUTOR
body:
  Another use-case: I want to make a `sqlite-utils` plugin that'll help me insert data into Datasette. 

  ```bash
  sqlite-utils insert-datasette \
    --token $DATASETTE_API_KEY \
    https://latest.datasette.io/fixtures/my-table \
    'select ...'
  ```


  This could also be a datasette plugin (ex `datasette upload-data ...`, but you can also think of `sqlite-utils` plugins that upload to S3, a postgres table, other DBMS's, etc.)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1638926655
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/567
node_id: IC_kwDOCGYnMM5hsAk_
created_at: 2023-07-17T21:42:37Z
updated_at: 2023-07-17T21:42:37Z
author_association: OWNER
body:
  I really like this. I'm also interested in:
  - Plugins that make new custom SQL functions available - similar to this Datasette hook: https://docs.datasette.io/en/stable/plugin_hooks.html#prepare-connection-conn-database-datasette
  - Plugins that register functions that can be used as recipes for `sqlite-utils convert` https://sqlite-utils.datasette.io/en/stable/cli.html#sqlite-utils-convert-recipes

  The upload-data-to-Datasette problem is planned to be solved by a future version of https://github.com/simonw/dclient 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2102#issuecomment-1640064620
issue_url: https://api.github.com/repos/simonw/datasette/issues/2102
node_id: IC_kwDOBm6k_c5hwWZs
created_at: 2023-07-18T11:47:21Z
updated_at: 2023-07-18T11:47:21Z
author_association: OWNER
body:
  I think I've figured out the problem here.

  The question being asked is "can this actor access this resource, which is within this database within this instance".

  The answer to this question needs to consider the full set of questions at once - yes they can access within this instance IF they have access to the specified table and that's the table being asked about.

  But the questions are currently being asked independently, which means the plugin hook acting on `view-instance` can't see that the answer here should be yes because it's actually about a table that the actor has explicit permission to view.

  So I think I may need to redesign the plugin hook to always see the full hierarchy of checks, not just a single check at a time.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/433#issuecomment-1640826795
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/433
node_id: IC_kwDOCGYnMM5hzQer
created_at: 2023-07-18T19:08:50Z
updated_at: 2023-07-18T19:08:50Z
author_association: NONE
body:
  Came here to report this, but instead I'll confirm the issue across two terminal emulators (Gnome Terminal and Alacritty) on Pop_OS! 22.04 (currently based on Ubuntu/Gnome). Also messes up the formatting of the terminal. Can also confirm that reset fixes it until the next sqlite-utils command. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2104#issuecomment-1641082395
issue_url: https://api.github.com/repos/simonw/datasette/issues/2104
node_id: IC_kwDOBm6k_c5h0O4b
created_at: 2023-07-18T22:41:37Z
updated_at: 2023-07-18T22:41:37Z
author_association: CONTRIBUTOR
body:
  For filtering virtual table's "shadow tables" (ex the FTS5 _content and most the spatialite tables),  you can use `pragma_table_list` (first appeared in SQLite 3.37 (2021-11-27), which has a `type` column that calls out `type="shadow"` tables https://www.sqlite.org/pragma.html#pragma_table_list

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2105#issuecomment-1642013043
issue_url: https://api.github.com/repos/simonw/datasette/issues/2105
node_id: IC_kwDOBm6k_c5h3yFz
created_at: 2023-07-19T12:41:36Z
updated_at: 2023-07-19T12:41:36Z
author_association: NONE
body:
  The same problem can be seen in the links:

  Advanced export

  JSON shape: default, array, newline-delimited, object

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2105#issuecomment-1643873232
issue_url: https://api.github.com/repos/simonw/datasette/issues/2105
node_id: IC_kwDOBm6k_c5h-4PQ
created_at: 2023-07-20T12:53:00Z
updated_at: 2023-07-20T12:53:34Z
author_association: NONE
body:
  I forgot to add that I followed these instructions to set up the python llm:

  https://simonwillison.net/2023/Jul/18/accessing-llama-2/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2105#issuecomment-1646247246
issue_url: https://api.github.com/repos/simonw/datasette/issues/2105
node_id: IC_kwDOBm6k_c5iH71O
created_at: 2023-07-21T21:16:37Z
updated_at: 2023-07-21T21:17:09Z
author_association: NONE
body:
  I must be doing something wrong. On page https://192.168.1.3:5432/datasette-llm/logs/_llm_migrations there's the option to filter the results, there's an Apply button. It also tries to load an URL with an extra URL element in it:

  https://192.168.1.3:5432/datasette-llm/datasette-llm/logs/_llm_migrations?_sort=name&name__contains=initial

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2098#issuecomment-1646250291
issue_url: https://api.github.com/repos/simonw/datasette/issues/2098
node_id: IC_kwDOBm6k_c5iH8kz
created_at: 2023-07-21T21:21:00Z
updated_at: 2023-07-21T21:27:25Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2098?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch and project coverage have no change.
  > Comparison is base [(`3a51ca9`)](https://app.codecov.io/gh/simonw/datasette/commit/3a51ca901427ac93b0ed0638fd1f82fb647a299a?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.39% compared to head [(`3d45262`)](https://app.codecov.io/gh/simonw/datasette/pull/2098?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.39%.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main    #2098   +/-   ##
  =======================================
    Coverage   92.39%   92.39%           
  =======================================
    Files          39       39           
    Lines        5802     5802           
  =======================================
    Hits         5361     5361           
    Misses        441      441           
  ```




  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2098?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/568#issuecomment-1646642666
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/568
node_id: IC_kwDOCGYnMM5iJcXq
created_at: 2023-07-22T18:13:19Z
updated_at: 2023-07-22T18:13:19Z
author_association: OWNER
body:
  https://sqlite-utils.datasette.io/en/stable/cli-reference.html#create-table

  ```bash
  sqlite-utils create-table ... --replace
  ```
  That also has `--ignore`:

  ```
    --ignore                  If table already exists, do nothing
    --replace                 If table already exists, replace it
    --transform               If table already exists, try to transform the schema
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/568#issuecomment-1646642959
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/568
node_id: IC_kwDOCGYnMM5iJccP
created_at: 2023-07-22T18:14:49Z
updated_at: 2023-07-22T18:14:49Z
author_association: OWNER
body:
  Here's where those are implemented for the `create-table` CLI command: https://github.com/simonw/sqlite-utils/blob/f7af23837deab5c98dae9441d1f68318065d7d8c/sqlite_utils/cli.py#L1543-L1564

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646643379
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/567
node_id: IC_kwDOCGYnMM5iJciz
created_at: 2023-07-22T18:16:54Z
updated_at: 2023-07-22T18:16:54Z
author_association: OWNER
body:
  > Would this possibly make a bunch of `x-to-sqlite` tools obsolete? Or nudge some to become plugins?

  Yeah, it could do! That's not a terrible idea to be honest, those things have really been proliferating.

  Alternatively, they could each register themselves as plugins in addition - so if you install e.g. `pocket-to-sqlite` you could then optionally also run it as `sqlite-utils pocket-to-sqlite ...`

  The benefit there is for people who install `sqlite-utils` from Homebrew, where it gets its own virtual environment. They could run:

  ```bash
  brew install sqlite-utils
  sqlite-utils install pocket-to-sqlite

  sqlite-utils pocket-to-sqlite ...
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646643450
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/567
node_id: IC_kwDOCGYnMM5iJcj6
created_at: 2023-07-22T18:17:18Z
updated_at: 2023-07-22T18:17:18Z
author_association: OWNER
body:
  I'm going to start by adding the `register_command` hook using the exact same pattern as Datasette and LLM.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/569#issuecomment-1646643676
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/569
node_id: IC_kwDOCGYnMM5iJcnc
created_at: 2023-07-22T18:18:24Z
updated_at: 2023-07-22T18:18:24Z
author_association: OWNER
body:
  Here's where I added that to LLM: https://github.com/simonw/llm/commit/a396950f7934e82a9968703bb3ce9ab7ab62f7f8

  - https://github.com/simonw/llm/issues/49

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/569#issuecomment-1646645990
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/569
node_id: IC_kwDOCGYnMM5iJdLm
created_at: 2023-07-22T18:29:40Z
updated_at: 2023-07-22T18:30:16Z
author_association: OWNER
body:
  Example plugin: https://gist.github.com/simonw/ccfbbf9b384a38ac7535b01849f57daf

  ```bash
  sqlite-utils install https://gist.github.com/simonw/ccfbbf9b384a38ac7535b01849f57daf/archive/cd7960e476c441a3c5f619e2a44a641a39b91467.zip
  ```
  `pyproject.toml`:
  ```toml
  [project]
  name = "sqlite-utils-hello-world"
  version = "0.1"

  [project.entry-points.sqlite_utils]
  hello_world = "sqlite_utils_hello_world"
  ```
  `sqlite_utils_hello_world.py`:
  ```python
  import click
  import sqlite_utils


  @sqlite_utils.hookimpl
  def register_commands(cli):
      @cli.command()
      def hello_world():
          "Say hello world"
          click.echo("Hello world!")
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2106#issuecomment-1646646931
issue_url: https://api.github.com/repos/simonw/datasette/issues/2106
node_id: IC_kwDOBm6k_c5iJdaT
created_at: 2023-07-22T18:34:04Z
updated_at: 2023-07-22T18:34:04Z
author_association: OWNER
body:
  Here's the diff for adding it to `sqlite-utils`: https://github.com/simonw/sqlite-utils/commit/ef31210bf06f920e0890e171c3198f0b0dc8d72d

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2106#issuecomment-1646648262
issue_url: https://api.github.com/repos/simonw/datasette/issues/2106
node_id: IC_kwDOBm6k_c5iJdvG
created_at: 2023-07-22T18:42:20Z
updated_at: 2023-07-22T18:42:20Z
author_association: OWNER
body:
  Tested this locally with:
  ```bash
  datasette install -e ../datasette-graphql
  ```
  Running `datasette plugins` confirmed that the plugin had been installed.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/568#issuecomment-1646652105
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/568
node_id: IC_kwDOCGYnMM5iJerJ
created_at: 2023-07-22T19:05:13Z
updated_at: 2023-07-22T19:05:13Z
author_association: OWNER
body:
  I think this is `replace=True` and `ignore=True` to match the CLI. And refactoring the CLI to use them.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/568#issuecomment-1646653382
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/568
node_id: IC_kwDOCGYnMM5iJe_G
created_at: 2023-07-22T19:13:20Z
updated_at: 2023-07-22T19:13:20Z
author_association: OWNER
body:
  Demo:
  ```pycon
  >>> from sqlite_utils import Database
  >>> db = Database(memory=True)
  >>> db["foo"].create({"id": int})
  <Table foo (id)>
  >>> db["foo"].create({"id": int})
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py", line 1647, in create
      self.db.create_table(
    File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py", line 1030, in create_table
      self.execute(sql)
    File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py", line 510, in execute
      return self.conn.execute(sql)
             ^^^^^^^^^^^^^^^^^^^^^^
  sqlean.dbapi2.OperationalError: table [foo] already exists
  >>> db["foo"].create({"id": int}, ignore=True)
  <Table foo (id)>
  >>> db["foo"].create({"id": int, "name": str}, replace=True)
  <Table foo (id, name)>
  >>> db["foo"].create({"id": int, "name": str, "age": int}, transform=True)
  <Table foo (id, name, age)>
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/568#issuecomment-1646653610
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/568
node_id: IC_kwDOCGYnMM5iJfCq
created_at: 2023-07-22T19:14:56Z
updated_at: 2023-07-22T19:14:56Z
author_association: OWNER
body:
  Manual testing of CLI command as well:
  ```
  $ sqlite-utils create-table /tmp/f.db foo id integer                         
  $ sqlite-utils create-table /tmp/f.db foo id integer
  Error: Table "foo" already exists. Use --replace to delete and replace it.
  $ sqlite-utils create-table /tmp/f.db foo id integer --replace
  $ sqlite-utils create-table /tmp/f.db foo id                  
  $ sqlite-utils schema /tmp/f.db
  CREATE TABLE [foo] (
     [id] INTEGER
  );
  $ sqlite-utils create-table /tmp/f.db foo id integer name str --transform
  Error: column types must be one of ('INTEGER', 'TEXT', 'FLOAT', 'BLOB')
  $ sqlite-utils create-table /tmp/f.db foo id integer name text --transform
  $ sqlite-utils schema /tmp/f.db                                          
  CREATE TABLE "foo" (
     [id] INTEGER,
     [name] TEXT
  );
  $ sqlite-utils create-table /tmp/f.db foo id integer name text --ignore   
  $ sqlite-utils create-table /tmp/f.db foo id integer name text --replace
  $ sqlite-utils schema /tmp/f.db                                        
  CREATE TABLE [foo] (
     [id] INTEGER,
     [name] TEXT
  );                                                                    
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646654383
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/567
node_id: IC_kwDOCGYnMM5iJfOv
created_at: 2023-07-22T19:20:16Z
updated_at: 2023-07-22T19:20:16Z
author_association: OWNER
body:
  Here's documentation for the new plugins mechanism, including a very short tutorial on writing a new plugin (inspired by https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html):

  https://sqlite-utils.datasette.io/en/latest/plugins.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/568#issuecomment-1646654818
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/568
node_id: IC_kwDOCGYnMM5iJfVi
created_at: 2023-07-22T19:22:40Z
updated_at: 2023-07-22T19:22:40Z
author_association: OWNER
body:
  I think this broke a test:
  ```
  $ pytest tests/test_tracer.py                           
  =============================================== test session starts ================================================
  platform darwin -- Python 3.11.4, pytest-7.2.2, pluggy-1.0.0
  rootdir: /Users/simon/Dropbox/Development/sqlite-utils
  plugins: icdiff-0.6, hypothesis-6.68.2
  collected 2 items                                                                                                  

  tests/test_tracer.py F.                                                                                      [100%]

  ===================================================== FAILURES =====================================================
  ___________________________________________________ test_tracer ____________________________________________________

      def test_tracer():
          collected = []
          db = Database(
              memory=True, tracer=lambda sql, params: collected.append((sql, params))
          )
          db["dogs"].insert({"name": "Cleopaws"})
          db["dogs"].enable_fts(["name"])
          db["dogs"].search("Cleopaws")
  >       assert collected == [
              ("PRAGMA recursive_triggers=on;", None),
              ("select name from sqlite_master where type = 'view'", None),
              ("select name from sqlite_master where type = 'table'", None),
              ("select name from sqlite_master where type = 'view'", None),
              ("CREATE TABLE [dogs] (\n   [name] TEXT\n);\n        ", None),
              ("select name from sqlite_master where type = 'view'", None),
              ("INSERT INTO [dogs] ([name]) VALUES (?);", ["Cleopaws"]),
              ("select name from sqlite_master where type = 'view'", None),
              (
                  "CREATE VIRTUAL TABLE [dogs_fts] USING FTS5 (\n    [name],\n    content=[dogs]\n)",
                  None,
              ),
              (
                  "INSERT INTO [dogs_fts] (rowid, [name])\n    SELECT rowid, [name] FROM [dogs];",
                  None,
              ),
              ("select name from sqlite_master where type = 'view'", None),
          ]
  E       assert equals failed
  E         [                                                  [                                                 
  E           ('PRAGMA recursive_triggers=on;', None),           ('PRAGMA recursive_triggers=on;', None),        
  E           (                                                                                                  
  E             "select name from sqlite_master where type =                                                     
  E         'view'",                                                                                             
  E             None,                                                       ...
  E         
  E         ...Full output truncated (13 lines hidden), use '-vv' to show

  tests/test_tracer.py:12: AssertionError
  ============================================= short test summary info ==============================================
  FAILED tests/test_tracer.py::test_tracer - assert equals failed
  =========================================== 1 failed, 1 passed in 0.05s ============================================
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/568#issuecomment-1646655272
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/568
node_id: IC_kwDOCGYnMM5iJfco
created_at: 2023-07-22T19:25:35Z
updated_at: 2023-07-22T19:25:35Z
author_association: OWNER
body:
  Here's why that test broke: https://github.com/simonw/sqlite-utils/blob/58b577279fcd5ef6ce88f88b28668dffebfe7f44/sqlite_utils/db.py#L960-L964

  I added an extra `if self[name].exists()` check to the `db.create_table()` method.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/565#issuecomment-1646656246
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/565
node_id: IC_kwDOCGYnMM5iJfr2
created_at: 2023-07-22T19:32:07Z
updated_at: 2023-07-22T19:32:07Z
author_association: OWNER
body:
  Trying out a simple first implementation:
  ```pycon
  >>> from sqlite_utils import Database
  >>> db = Database(memory=True, tracer=print)
  PRAGMA recursive_triggers=on; None
  >>> db["foo"].insert({"id": 1})
  select name from sqlite_master where type = 'view' None
  select name from sqlite_master where type = 'table' None
  select name from sqlite_master where type = 'view' None
  select name from sqlite_master where type = 'table' None
  select name from sqlite_master where type = 'view' None
  CREATE TABLE [foo] (
     [id] INTEGER
  );
           None
  select name from sqlite_master where type = 'view' None
  INSERT INTO [foo] ([id]) VALUES (?); [1]
  select name from sqlite_master where type = 'table' None
  select name from sqlite_master where type = 'table' None
  PRAGMA table_info([foo]) None
  <Table foo (id)>
  >>> db.rename_table("foo", "baz")
  ALTER TABLE [foo] RENAME TO [baz] None
  >>> print(db.schema)
  select sql from sqlite_master where sql is not null None
  CREATE TABLE "baz" (
     [id] INTEGER
  );
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646656283
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/567
node_id: IC_kwDOCGYnMM5iJfsb
created_at: 2023-07-22T19:32:24Z
updated_at: 2023-07-22T19:32:24Z
author_association: CONTRIBUTOR
body:
  Cool. I might try to add a geojson plugin that handles both input and output. That would help me out a lot. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/565#issuecomment-1646657324
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/565
node_id: IC_kwDOCGYnMM5iJf8s
created_at: 2023-07-22T19:39:06Z
updated_at: 2023-07-22T19:39:06Z
author_association: OWNER
body:
  > Also need a design for an option for the `.transform()` method to indicate that the new table should be created with a new name without dropping the old one.

  I think `keep_table="name_of_table"` is good for this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/565#issuecomment-1646657849
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/565
node_id: IC_kwDOCGYnMM5iJgE5
created_at: 2023-07-22T19:42:02Z
updated_at: 2023-07-22T19:42:02Z
author_association: OWNER
body:
  Manually testing new `rename-table` command:
  ```
  $ sqlite-utils schema /tmp/f.db 
  CREATE TABLE [foo] (
     [id] INTEGER,
     [name] TEXT
  );
  $ sqlite-utils rename-table /tmp/f.db bad-table hi
  Error: Table "bad-table" could not be renamed. no such table: bad-table
  $ sqlite-utils rename-table /tmp/f.db foo foo     
  Error: Table "foo" could not be renamed. there is already another table or index with this name: foo
  $ sqlite-utils rename-table /tmp/f.db foo bar
  $ sqlite-utils schema /tmp/f.db                   
  CREATE TABLE "bar" (
     [id] INTEGER,
     [name] TEXT
  );
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/565#issuecomment-1646658978
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/565
node_id: IC_kwDOCGYnMM5iJgWi
created_at: 2023-07-22T19:48:54Z
updated_at: 2023-07-22T19:48:54Z
author_association: OWNER
body:
  Python method documentation:
  - https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Database.rename_table
  - https://sqlite-utils.datasette.io/en/latest/python-api.html#renaming-a-table

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/565#issuecomment-1646659809
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/565
node_id: IC_kwDOCGYnMM5iJgjh
created_at: 2023-07-22T19:53:56Z
updated_at: 2023-07-22T19:53:56Z
author_association: OWNER
body:
  CLI documentation:

  - https://sqlite-utils.datasette.io/en/latest/cli.html#renaming-a-table
  - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#rename-table

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/572#issuecomment-1646660777
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/572
node_id: IC_kwDOCGYnMM5iJgyp
created_at: 2023-07-22T19:59:13Z
updated_at: 2023-07-22T19:59:13Z
author_association: OWNER
body:
  I don't know where that is coming from! I can't see `textual` as a dependency we are pulling in anywhere.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/572#issuecomment-1646681192
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/572
node_id: IC_kwDOCGYnMM5iJlxo
created_at: 2023-07-22T22:12:08Z
updated_at: 2023-07-22T22:12:08Z
author_association: OWNER
body:
  Found it: https://github.com/simonw/sqlite-utils/blob/18f190e28334d821be78a1dbbf31d7610fc1f9c1/.github/workflows/test.yml#L29-L31

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/571#issuecomment-1646682686
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/571
node_id: IC_kwDOCGYnMM5iJmI-
created_at: 2023-07-22T22:23:47Z
updated_at: 2023-07-22T22:23:47Z
author_association: OWNER
body:
  Demo:
  ```pycon
  >>> from sqlite_utils import Database
  >>> db = Database(memory=True)
  >>> db["foo"].insert({"name": "Cleo"})
  <Table foo (name)>
  >>> print(db["foo"].schema)
  CREATE TABLE [foo] (
     [name] TEXT
  )
  >>> db["foo"].transform(not_null={"name"})
  <Table foo (name)>
  >>> print(db["foo"].schema)
  CREATE TABLE "foo" (
     [name] TEXT NOT NULL
  )
  >>> db["foo"].transform(types={"name": int}, keep_table="kept")
  <Table foo (name)>
  >>> print(db.schema)
  CREATE TABLE "kept" (
     [name] TEXT NOT NULL
  );
  CREATE TABLE "foo" (
     [name] INTEGER NOT NULL
  );
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/573#issuecomment-1646686332
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/573
node_id: IC_kwDOCGYnMM5iJnB8
created_at: 2023-07-22T22:52:01Z
updated_at: 2023-07-22T22:52:01Z
author_association: OWNER
body:
  I was literally seconds away from shipping version 3.34 but I this looks good so I'm going to try and get it in there.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/573#issuecomment-1646686382
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/573
node_id: IC_kwDOCGYnMM5iJnCu
created_at: 2023-07-22T22:52:22Z
updated_at: 2023-07-22T22:56:49Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/573?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch coverage: **`100.00`**% and project coverage change: **`+0.03`** :tada:
  > Comparison is base [(`86a352f`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/86a352f8b713ca30a65a2048170bd510d529d8c4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.09% compared to head [(`faf398f`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/573?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.12%.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@            Coverage Diff             @@
  ##             main     #573      +/-   ##
  ==========================================
  + Coverage   96.09%   96.12%   +0.03%     
  ==========================================
    Files           8        8              
    Lines        2794     2816      +22     
  ==========================================
  + Hits         2685     2707      +22     
    Misses        109      109              
  ```


  | [Impacted Files](https://app.codecov.io/gh/simonw/sqlite-utils/pull/573?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [sqlite\_utils/\_\_init\_\_.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/573?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL19faW5pdF9fLnB5) | `100.00% <100.00%> (ø)` | |
  | [sqlite\_utils/db.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/573?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.36% <100.00%> (+0.01%)` | :arrow_up: |
  | [sqlite\_utils/hookspecs.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/573?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2hvb2tzcGVjcy5weQ==) | `100.00% <100.00%> (ø)` | |

  ... and [1 file with indirect coverage changes](https://app.codecov.io/gh/simonw/sqlite-utils/pull/573/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)


  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/573?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646686424
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/567
node_id: IC_kwDOCGYnMM5iJnDY
created_at: 2023-07-22T22:52:34Z
updated_at: 2023-07-22T22:52:34Z
author_association: OWNER
body:
  Splitting off an issue for `prepare_connection()` since Alex got the PR in seconds before I shipped 3.34!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/573#issuecomment-1646686675
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/573
node_id: IC_kwDOCGYnMM5iJnHT
created_at: 2023-07-22T22:54:38Z
updated_at: 2023-07-22T22:54:38Z
author_association: OWNER
body:
  <img width="779" alt="image" src="https://github.com/simonw/sqlite-utils/assets/9599/c4e5769d-f1a3-427b-b260-882fce5c7205">

  Glitch in the rendered documentation from https://sqlite-utils--573.org.readthedocs.build/en/573/plugins.html#prepare-connection-conn

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/573#issuecomment-1646687103
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/573
node_id: IC_kwDOCGYnMM5iJnN_
created_at: 2023-07-22T22:58:35Z
updated_at: 2023-07-22T22:58:35Z
author_association: OWNER
body:
  https://sqlite-utils--573.org.readthedocs.build/en/573/plugins.html#prepare-connection-conn

  <img width="773" alt="image" src="https://github.com/simonw/sqlite-utils/assets/9599/3211a37a-c30d-47f9-bc10-c2c3240b6f6e">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646687219
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/567
node_id: IC_kwDOCGYnMM5iJnPz
created_at: 2023-07-22T22:59:36Z
updated_at: 2023-07-22T22:59:36Z
author_association: OWNER
body:
  Now that we have two plugin hooks I'm closing this issue (we can open other issues for further hooks).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/575#issuecomment-1646687461
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/575
node_id: IC_kwDOCGYnMM5iJnTl
created_at: 2023-07-22T23:01:44Z
updated_at: 2023-07-22T23:01:44Z
author_association: OWNER
body:
  Relevant code: https://github.com/simonw/sqlite-utils/blob/3f80a026983d3e634f05a46f2a6da162b5139dd9/sqlite_utils/db.py#L346

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/574#issuecomment-1646688748
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/574
node_id: IC_kwDOCGYnMM5iJnns
created_at: 2023-07-22T23:12:03Z
updated_at: 2023-07-22T23:13:14Z
author_association: OWNER
body:
  Just tried this out by adding the example from the docs to my `sqlite-utils-hello-world` plugin and running:
  ```bash
  sqlite-utils install -e ../sqlite-utils-hello-world 
  ```
  ```
  Obtaining file:///Users/simon/Dropbox/Development/sqlite-utils-hello-world
  ...
  ```
  Then:
  ```bash
  sqlite-utils memory "select hello('simon')"
  ```
  ```json
  [{"hello('simon')": "Hello, simon!"}]
  ```
  Also:
  ```pycon
  >>> import sqlite_utils
  >>> db = sqlite_utils.Database(memory=True)
  >>> list(db.query("select hello('simon')"))
  [{"hello('simon')": 'Hello, simon!'}]
  >>> db2 = sqlite_utils.Database(memory=True, execute_plugins=False)
  >>> list(db2.query("select hello('simon')"))
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py", line 494, in query
      cursor = self.execute(sql, params or tuple())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py", line 512, in execute
      return self.conn.execute(sql, parameters)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  sqlean.dbapi2.OperationalError: no such function: hello
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/576#issuecomment-1646752814
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/576
node_id: IC_kwDOCGYnMM5iJ3Qu
created_at: 2023-07-23T05:43:13Z
updated_at: 2023-07-23T05:43:13Z
author_association: OWNER
body:
  Using `llm` in my `sqlite-utils` local checkout:
  ```bash
  git diff 0.2 0.3 | llm -s 'release notes as markdown bullet points'
  ```
  > - Added a new `Table` class in the `db` module which represents a table in a SQLite database
  > - The `Table` class has properties that provide information about the table, such as column names, foreign keys, and schema
  > - The `Table` class also has a `count` property that returns the number of rows in the table
  > - The `Table` class has a `create` method that can be used to create a new table with specified columns, primary key, and foreign keys
  > - Updated the `Database` class in the `__init__` module to include the `Table` class and expose it via the `__all__` variable
  > - Added new tests for creating tables and introspecting tables

  ```bash
  git diff 0.3 0.3.1 | llm -s 'release notes as markdown bullet points'
  ```

  > - Updated PyPI version badge to display the latest version
  > - Updated Travis CI badge to link to the master branch
  > - Added a badge for documentation status
  > - Added a license badge
  > - Added a description of the package
  > - Added installation instructions
  > - Added a link to the documentation
  > - Added a "Related projects" section and linked to Datasette and csvs-to-sqlite
  > - Updated the version in setup.py to 0.3.1

  ```bash
  git diff 0.3.1 0.4 | llm -s 'release notes as markdown bullet points'
  ```

  > - Added an `enable_fts` method to the `Table` class that enables full-text search on specified columns
  > - Added a `populate_fts` method to the `Table` class that refreshes the search index for full-text search
  > - Added a `search` method to the `Table` class that allows searching the table using full-text search
  > - Updated the `setup.py` script to version 0.4
  > - Updated the `db.py` file to include the `enable_fts`, `populate_fts`, and `search` methods
  > - Added test cases for the `enable_fts`, `populate_fts`, and `search` methods in the `test_enable_fts.py` file
  > - Added test cases for the `count`, `columns`, and `schema` properties in the `test_introspect.py` file
  > 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/37#issuecomment-1646950438
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/37
node_id: IC_kwDOC8SPRc5iKngm
created_at: 2023-07-23T20:18:26Z
updated_at: 2023-07-23T20:18:26Z
author_association: NONE
body:
  My bad, although I could not find how to use `url`s, I was using a function call syntax for the filters instead of jinja's pipe syntax; closing this issue.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/578#issuecomment-1648323482
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/578
node_id: IC_kwDOCGYnMM5iP2ua
created_at: 2023-07-24T17:31:56Z
updated_at: 2023-07-24T17:31:56Z
author_association: OWNER
body:
  The main blocker here is coming up with a design. The challenge is cleanly integrating it with the existing format options:

  https://github.com/simonw/sqlite-utils/blob/8bee14588687b66c54c7a3dfae5de2f9cc7cac3f/docs/cli-reference.rst#L114-L132

  I'd like to avoid breaking backwards compatibility.

  The most obvious solution is to add a `--format x` option. This is slightly confusing as `--fmt` already exists.

  Or... `--fmt` could be enhanced to handle plugins too - and the existing set of formats (from the `tabulate` integration) could be refactored to use the new hook.

  If I ever do release `sqlite-utils 4` I'm tempted to move that tabulate stuff into a plugin.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/578#issuecomment-1648324312
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/578
node_id: IC_kwDOCGYnMM5iP27Y
created_at: 2023-07-24T17:32:21Z
updated_at: 2023-07-24T17:32:21Z
author_association: OWNER
body:
  I'm currently leaning very slightly towards `--format geojson` - and explaining in the docs that `--format` is for formats added by plugins.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/578#issuecomment-1648325682
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/578
node_id: IC_kwDOCGYnMM5iP3Qy
created_at: 2023-07-24T17:33:10Z
updated_at: 2023-07-24T17:33:10Z
author_association: OWNER
body:
  A related feature would be support for plugins to add new ways of _ingesting_ data - currently `sqlite-utils insert` works against JSON, newline-JSON, CSV and TSV.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/578#issuecomment-1648339661
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/578
node_id: IC_kwDOCGYnMM5iP6rN
created_at: 2023-07-24T17:44:30Z
updated_at: 2023-07-24T17:44:30Z
author_association: CONTRIBUTOR
body:
  > A related feature would be support for plugins to add new ways of ingesting data - currently sqlite-utils insert works against JSON, newline-JSON, CSV and TSV.

  This is my goal, to have one plugin that handles input and output symmetrically. I'd like to be able to do something like this:

  ```sh
  sqlite-utils insert data.db table file.geojson --format geojson
  # ... explore and manipulate in Datasette
  sqlite-utils query data.db ... --format geojson > output.geojson
  ```

  This would work especially well with [datasette-query-files](https://github.com/eyeseast/datasette-query-files), since I already have the queries I need saved in standalone SQL files.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2107#issuecomment-1649863533
issue_url: https://api.github.com/repos/simonw/datasette/issues/2107
node_id: IC_kwDOBm6k_c5iVutt
created_at: 2023-07-25T13:37:07Z
updated_at: 2023-07-25T13:37:07Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2107?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch and project coverage have no change.
  > Comparison is base [(`278ac91`)](https://app.codecov.io/gh/simonw/datasette/commit/278ac91a4d68da333a6ed74a2c75b3ec9db8bd19?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38% compared to head [(`7fc79ba`)](https://app.codecov.io/gh/simonw/datasette/pull/2107?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38%.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main    #2107   +/-   ##
  =======================================
    Coverage   92.38%   92.38%           
  =======================================
    Files          39       39           
    Lines        5805     5805           
  =======================================
    Hits         5363     5363           
    Misses        442      442           
  ```




  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2107?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2053#issuecomment-1651874649
issue_url: https://api.github.com/repos/simonw/datasette/issues/2053
node_id: IC_kwDOBm6k_c5idZtZ
created_at: 2023-07-26T14:03:37Z
updated_at: 2023-07-26T14:03:37Z
author_association: OWNER
body:
  Big chunk of commented-out code I just removed:
  ```python

      import pdb

      pdb.set_trace()

      if isinstance(output, dict) and output.get("ok") is False:
          # TODO: Other error codes?

          response.status_code = 400

      if datasette.cors:
          add_cors_headers(response.headers)

      return response

      # registry = Registry(
      #     extra_count,
      #     extra_facet_results,
      #     extra_facets_timed_out,
      #     extra_suggested_facets,
      #     facet_instances,
      #     extra_human_description_en,
      #     extra_next_url,
      #     extra_columns,
      #     extra_primary_keys,
      #     run_display_columns_and_rows,
      #     extra_display_columns,
      #     extra_display_rows,
      #     extra_debug,
      #     extra_request,
      #     extra_query,
      #     extra_metadata,
      #     extra_extras,
      #     extra_database,
      #     extra_table,
      #     extra_database_color,
      #     extra_table_actions,
      #     extra_filters,
      #     extra_renderers,
      #     extra_custom_table_templates,
      #     extra_sorted_facet_results,
      #     extra_table_definition,
      #     extra_view_definition,
      #     extra_is_view,
      #     extra_private,
      #     extra_expandable_columns,
      #     extra_form_hidden_args,
      # )

      results = await registry.resolve_multi(
          ["extra_{}".format(extra) for extra in extras]
      )
      data = {
          "ok": True,
          "next": next_value and str(next_value) or None,
      }
      data.update(
          {
              key.replace("extra_", ""): value
              for key, value in results.items()
              if key.startswith("extra_") and key.replace("extra_", "") in extras
          }
      )
      raw_sqlite_rows = rows[:page_size]
      data["rows"] = [dict(r) for r in raw_sqlite_rows]

      private = False
      if canned_query:
          # Respect canned query permissions
          visible, private = await datasette.check_visibility(
              request.actor,
              permissions=[
                  ("view-query", (database, canned_query)),
                  ("view-database", database),
                  "view-instance",
              ],
          )
          if not visible:
              raise Forbidden("You do not have permission to view this query")

      else:
          await datasette.ensure_permissions(request.actor, [("execute-sql", database)])

      # If there's no sql, show the database index page
      if not sql:
          return await database_index_view(request, datasette, db)

      validate_sql_select(sql)

      # Extract any :named parameters
      named_parameters = named_parameters or await derive_named_parameters(db, sql)
      named_parameter_values = {
          named_parameter: params.get(named_parameter) or ""
          for named_parameter in named_parameters
          if not named_parameter.startswith("_")
      }

      # Set to blank string if missing from params
      for named_parameter in named_parameters:
          if named_parameter not in params and not named_parameter.startswith("_"):
              params[named_parameter] = ""

      extra_args = {}
      if params.get("_timelimit"):
          extra_args["custom_time_limit"] = int(params["_timelimit"])
      if _size:
          extra_args["page_size"] = _size

      templates = [f"query-{to_css_class(database)}.html", "query.html"]
      if canned_query:
          templates.insert(
              0,
              f"query-{to_css_class(database)}-{to_css_class(canned_query)}.html",
          )

      query_error = None

      # Execute query - as write or as read
      if write:
          raise NotImplementedError("Write queries not yet implemented")
          # if request.method == "POST":
          #     # If database is immutable, return an error
          #     if not db.is_mutable:
          #         raise Forbidden("Database is immutable")
          #     body = await request.post_body()
          #     body = body.decode("utf-8").strip()
          #     if body.startswith("{") and body.endswith("}"):
          #         params = json.loads(body)
          #         # But we want key=value strings
          #         for key, value in params.items():
          #             params[key] = str(value)
          #     else:
          #         params = dict(parse_qsl(body, keep_blank_values=True))
          #     # Should we return JSON?
          #     should_return_json = (
          #         request.headers.get("accept") == "application/json"
          #         or request.args.get("_json")
          #         or params.get("_json")
          #     )
          #     if canned_query:
          #         params_for_query = MagicParameters(params, request, self.ds)
          #     else:
          #         params_for_query = params
          #     ok = None
          #     try:
          #         cursor = await self.ds.databases[database].execute_write(
          #             sql, params_for_query
          #         )
          #         message = metadata.get(
          #             "on_success_message"
          #         ) or "Query executed, {} row{} affected".format(
          #             cursor.rowcount, "" if cursor.rowcount == 1 else "s"
          #         )
          #         message_type = self.ds.INFO
          #         redirect_url = metadata.get("on_success_redirect")
          #         ok = True
          #     except Exception as e:
          #         message = metadata.get("on_error_message") or str(e)
          #         message_type = self.ds.ERROR
          #         redirect_url = metadata.get("on_error_redirect")
          #         ok = False
          #     if should_return_json:
          #         return Response.json(
          #             {
          #                 "ok": ok,
          #                 "message": message,
          #                 "redirect": redirect_url,
          #             }
          #         )
          #     else:
          #         self.ds.add_message(request, message, message_type)
          #         return self.redirect(request, redirect_url or request.path)
          # else:

          #     async def extra_template():
          #         return {
          #             "request": request,
          #             "db_is_immutable": not db.is_mutable,
          #             "path_with_added_args": path_with_added_args,
          #             "path_with_removed_args": path_with_removed_args,
          #             "named_parameter_values": named_parameter_values,
          #             "canned_query": canned_query,
          #             "success_message": request.args.get("_success") or "",
          #             "canned_write": True,
          #         }

          #     return (
          #         {
          #             "database": database,
          #             "rows": [],
          #             "truncated": False,
          #             "columns": [],
          #             "query": {"sql": sql, "params": params},
          #             "private": private,
          #         },
          #         extra_template,
          #         templates,
          #     )

      # Not a write
      rows = []
      if canned_query:
          params_for_query = MagicParameters(params, request, datasette)
      else:
          params_for_query = params
      try:
          results = await datasette.execute(
              database, sql, params_for_query, truncate=True, **extra_args
          )
          columns = [r[0] for r in results.description]
          rows = list(results.rows)
      except sqlite3.DatabaseError as e:
          query_error = e
          results = None
          columns = []

      allow_execute_sql = await datasette.permission_allowed(
          request.actor, "execute-sql", database
      )

      format_ = request.url_vars.get("format") or "html"

      if format_ == "csv":
          raise NotImplementedError("CSV format not yet implemented")
      elif format_ in datasette.renderers.keys():
          # Dispatch request to the correct output format renderer
          # (CSV is not handled here due to streaming)
          result = call_with_supported_arguments(
              datasette.renderers[format_][0],
              datasette=datasette,
              columns=columns,
              rows=rows,
              sql=sql,
              query_name=None,
              database=db.name,
              table=None,
              request=request,
              view_name="table",  # TODO: should this be "query"?
              # These will be deprecated in Datasette 1.0:
              args=request.args,
              data={
                  "rows": rows,
              },  # TODO what should this be?
          )
          result = await await_me_maybe(result)
          if result is None:
              raise NotFound("No data")
          if isinstance(result, dict):
              r = Response(
                  body=result.get("body"),
                  status=result.get("status_code") or 200,
                  content_type=result.get("content_type", "text/plain"),
                  headers=result.get("headers"),
              )
          elif isinstance(result, Response):
              r = result
              # if status_code is not None:
              #     # Over-ride the status code
              #     r.status = status_code
          else:
              assert False, f"{result} should be dict or Response"
      elif format_ == "html":
          headers = {}
          templates = [f"query-{to_css_class(database)}.html", "query.html"]
          template = datasette.jinja_env.select_template(templates)
          alternate_url_json = datasette.absolute_url(
              request,
              datasette.urls.path(path_with_format(request=request, format="json")),
          )
          headers.update(
              {
                  "Link": '{}; rel="alternate"; type="application/json+datasette"'.format(
                      alternate_url_json
                  )
              }
          )
          r = Response.html(
              await datasette.render_template(
                  template,
                  dict(
                      data,
                      append_querystring=append_querystring,
                      path_with_replaced_args=path_with_replaced_args,
                      fix_path=datasette.urls.path,
                      settings=datasette.settings_dict(),
                      # TODO: review up all of these hacks:
                      alternate_url_json=alternate_url_json,
                      datasette_allow_facet=(
                          "true" if datasette.setting("allow_facet") else "false"
                      ),
                      is_sortable=any(c["sortable"] for c in data["display_columns"]),
                      allow_execute_sql=await datasette.permission_allowed(
                          request.actor, "execute-sql", resolved.db.name
                      ),
                      query_ms=1.2,
                      select_templates=[
                          f"{'*' if template_name == template.name else ''}{template_name}"
                          for template_name in templates
                      ],
                  ),
                  request=request,
                  view_name="table",
              ),
              headers=headers,
          )
      else:
          assert False, "Invalid format: {}".format(format_)
      # if next_url:
      #     r.headers["link"] = f'<{next_url}>; rel="next"'
      return r
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2053#issuecomment-1651883505
issue_url: https://api.github.com/repos/simonw/datasette/issues/2053
node_id: IC_kwDOBm6k_c5idb3x
created_at: 2023-07-26T14:08:20Z
updated_at: 2023-07-26T14:08:20Z
author_association: OWNER
body:
  I think the hardest part of getting this working is dealing with the different formats.

  Idea: refactor `.html` as a format (since it's by far the most complex) and tweak the plugin hook a bit as part of that, then use what I learn from that to get the other formats working.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2053#issuecomment-1651894668
issue_url: https://api.github.com/repos/simonw/datasette/issues/2053
node_id: IC_kwDOBm6k_c5idemM
created_at: 2023-07-26T14:14:34Z
updated_at: 2023-07-26T14:14:34Z
author_association: OWNER
body:
  Another point of confusion is how `/content` sometimes serves the database index page (with a list of tables) and sometimes solves the results of a query.

  I could resolve this by turning the information on the index page into extras, which can optionally be requested any time a query is run but default to being shown if there is no query.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2053#issuecomment-1651904060
issue_url: https://api.github.com/repos/simonw/datasette/issues/2053
node_id: IC_kwDOBm6k_c5idg48
created_at: 2023-07-26T14:19:00Z
updated_at: 2023-07-26T15:25:15Z
author_association: OWNER
body:
  Worth noting that the `register_output_renderer()` is actually pretty easy to extend, because it returns a dictionary which could have more keys (like the required set of extras) added to it:
  ```python
  @hookimpl
  def register_output_renderer(datasette):
      return {
          "extension": "test",
          "render": render_demo,
          "can_render": can_render_demo,  # Optional
      }
  ```
  https://docs.datasette.io/en/0.64.3/plugin_hooks.html#register-output-renderer-datasette

  One slight hiccup with that plugin hook is this:

  > rows - list of `sqlite3.Row` objects

  I could turn that into a Datasette defined object that behaves like a [sqlite3.Row](https://docs.python.org/3/library/sqlite3.html#sqlite3.Row) though, which would give me extra flexibility in the future.

  A bit tricky though since it's implemented in C for performance: https://github.com/python/cpython/blob/b0202a4e5d6b629ba5acbc703e950f08ebaf07df/Modules/_sqlite/row.c

  Pasted that into Claude for the following explanation:

  > -   pysqlite_Row is the structure defining the Row object. It contains the tuple of data (self->data) and description of columns (self->description).
  > -   pysqlite_row_new() is the constructor which creates a new Row object given a cursor and tuple of data.
  > -   pysqlite_row_dealloc() frees the memory when Row object is deleted.
  > -   pysqlite_row_keys() returns the column names of the row.
  > -   pysqlite_row_length() and pysqlite_row_subscript() implement sequence like behavior to access row elements by index.
  > -   pysqlite_row_subscript() also allows accessing by column name by doing a lookup in description.
  > -   pysqlite_row_hash() and pysqlite_row_richcompare() implement equality checks and hash function.

  I could use protocols in Python to make my own `DatasetteRow` which can be used interchangeably with `sqlite3.Row` - https://docs.python.org/3/library/typing.html#typing.Protocol

  Turned this into a TIL: https://til.simonwillison.net/python/protocols

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2109#issuecomment-1652295866
issue_url: https://api.github.com/repos/simonw/datasette/issues/2109
node_id: IC_kwDOBm6k_c5ifAi6
created_at: 2023-07-26T18:26:18Z
updated_at: 2023-07-26T18:26:18Z
author_association: OWNER
body:
  I'm going to do this work in a fresh branch, copying bits over from here as needed:
  - https://github.com/simonw/datasette/pull/2053

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2053#issuecomment-1652296467
issue_url: https://api.github.com/repos/simonw/datasette/issues/2053
node_id: IC_kwDOBm6k_c5ifAsT
created_at: 2023-07-26T18:26:44Z
updated_at: 2023-07-26T18:26:44Z
author_association: OWNER
body:
  I'm abandoning this branch in favour of a fresh attempt, described here:
  - https://github.com/simonw/datasette/issues/2109

  I'll copy bits and pieces of this branch across as-needed.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2116#issuecomment-1652303471
issue_url: https://api.github.com/repos/simonw/datasette/issues/2116
node_id: IC_kwDOBm6k_c5ifCZv
created_at: 2023-07-26T18:32:24Z
updated_at: 2023-07-26T18:32:24Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/blob/dc5171eb1b1d9f1d55e367f8a4d93edb55a43351/datasette/views/database.py#L172-L212

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2114#issuecomment-1652321419
issue_url: https://api.github.com/repos/simonw/datasette/issues/2114
node_id: IC_kwDOBm6k_c5ifGyL
created_at: 2023-07-26T18:48:03Z
updated_at: 2023-07-26T18:48:03Z
author_association: OWNER
body:
  This is also where I'll bring back writable canned queries:

  https://github.com/simonw/datasette/blob/18dd88ee4d78fe9d760e9da96028ae06d938a85c/datasette/views/database.py#L281-L334

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2109#issuecomment-1652325193
issue_url: https://api.github.com/repos/simonw/datasette/issues/2109
node_id: IC_kwDOBm6k_c5ifHtJ
created_at: 2023-07-26T18:51:15Z
updated_at: 2023-07-26T18:51:15Z
author_association: OWNER
body:
  Here's the code I'm going to be entirely replacing:

  https://github.com/simonw/datasette/blob/18dd88ee4d78fe9d760e9da96028ae06d938a85c/datasette/views/database.py#L213-L530

  Plus this weird class in `views/table.py`:

  https://github.com/simonw/datasette/blob/18dd88ee4d78fe9d760e9da96028ae06d938a85c/datasette/views/table.py#L701-L749

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2110#issuecomment-1652330111
issue_url: https://api.github.com/repos/simonw/datasette/issues/2110
node_id: IC_kwDOBm6k_c5ifI5_
created_at: 2023-07-26T18:55:31Z
updated_at: 2023-07-26T18:55:31Z
author_association: OWNER
body:
  Changed my mind on this - I'm going to have the `query_view` mapped to `/db` but have the first code on there notice if `?sql=` is missing and return a `database_view()` function instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2111#issuecomment-1652354606
issue_url: https://api.github.com/repos/simonw/datasette/issues/2111
node_id: IC_kwDOBm6k_c5ifO4u
created_at: 2023-07-26T19:16:35Z
updated_at: 2023-07-26T19:17:45Z
author_association: OWNER
body:
  I just noticed that this URL: https://latest.datasette.io/fixtures.csv

  Returns a 500 error right now!

  <img width="275" alt="image" src="https://github.com/simonw/datasette/assets/9599/bd560821-07ba-4e8f-8472-9e677294687d">

  It's fine with a `?sql=` query: https://latest.datasette.io/fixtures.csv?sql=select+*+from+facetable

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2111#issuecomment-1652407208
issue_url: https://api.github.com/repos/simonw/datasette/issues/2111
node_id: IC_kwDOBm6k_c5ifbuo
created_at: 2023-07-26T19:54:20Z
updated_at: 2023-07-26T19:54:20Z
author_association: OWNER
body:
  I implemented `/content` and `/content.json` but I left `/content?sql=...` as this:

  https://github.com/simonw/datasette/blob/2e40a506ad45b44fd7642474f630a31ef18b5911/datasette/views/database.py#L220-L221

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2118#issuecomment-1652681136
issue_url: https://api.github.com/repos/simonw/datasette/issues/2118
node_id: IC_kwDOBm6k_c5igemw
created_at: 2023-07-26T23:30:44Z
updated_at: 2023-07-26T23:30:44Z
author_association: OWNER
body:
  The `_shape=` stuff should use `json_renderer` instead - that's how the table view did it:
  https://github.com/simonw/datasette/commit/d97e82df3c8a3f2e97038d7080167be9bb74a68d#diff-5c9ef29c33ed0fde413565b23fa258d60fc3a2bb205b016db9e915c9bd5ecfb3

  https://github.com/simonw/datasette/blob/c3e3ecf590ca5fa61b00aba4c78599e33d370b60/datasette/views/table.py#L822-L850

  Instead of:

  https://github.com/simonw/datasette/blob/c3e3ecf590ca5fa61b00aba4c78599e33d370b60/datasette/views/database.py#L239-L288

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2109#issuecomment-1652715823
issue_url: https://api.github.com/repos/simonw/datasette/issues/2109
node_id: IC_kwDOBm6k_c5ignEv
created_at: 2023-07-27T00:17:04Z
updated_at: 2023-07-27T00:18:07Z
author_association: OWNER
body:
  New decision: I had originally decided that the HTML view would just use data that could otherwise be extracted from the JSON view if you fed in enough extras.

  I've changed my mind. I'm OK with the HTML view getting a few bonus things available in its context, provided those are clearly documented for template authors.

  This should help avoid me having to make many changes to the templates themselves.

  HTML-specific stuff will be things like `csrftoken()` and `edit_sql_url` and `show_hide_link` and `database_color` and suchlike.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2109#issuecomment-1652721111
issue_url: https://api.github.com/repos/simonw/datasette/issues/2109
node_id: IC_kwDOBm6k_c5igoXX
created_at: 2023-07-27T00:23:28Z
updated_at: 2023-07-27T00:23:37Z
author_association: OWNER
body:
  Once again I'm tempted to formalize the extra HTML context as a `dataclass` so I can ensure it is documented correctly.

  Here's an example of Hugging Face doing that: https://github.com/huggingface/transformers/blob/1689aea73346816b936b84932e12b774974e61a6/src/transformers/training_args.py#L622C1-L624

  ```python
  from dataclasses import dataclass, field

  @dataclass
  class TrainingArguments:
      output_dir: str = field(
          metadata={"help": "The output directory where the model predictions and checkpoints will be written."},
      )
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2112#issuecomment-1652722208
issue_url: https://api.github.com/repos/simonw/datasette/issues/2112
node_id: IC_kwDOBm6k_c5igoog
created_at: 2023-07-27T00:24:50Z
updated_at: 2023-07-27T00:25:16Z
author_association: OWNER
body:
  I changed my mind, I'm not going to do this with a format hook.

  Also: https://github.com/simonw/datasette/issues/2109#issuecomment-1652715823

  I've decided to let HTML have its own special context. I'm going to try formalizing that as a `@dataclass` with fields with `metadata={"help": ...}` to see what that looks like.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2112#issuecomment-1652723029
issue_url: https://api.github.com/repos/simonw/datasette/issues/2112
node_id: IC_kwDOBm6k_c5igo1V
created_at: 2023-07-27T00:25:49Z
updated_at: 2023-07-27T00:25:49Z
author_association: OWNER
body:
  Getting the query page working with all of the necessary HTML context variables is a slog.

  I uploaded `query.html` to Claude and asked it "List of every template context variable with a detailed description of each one":

  <img width="518" alt="image" src="https://github.com/simonw/datasette/assets/9599/9c7f3292-d3f3-49bc-9263-dc39bed5aa58">

  > Based on the template code provided, here is a list of the template context variables and a description of each one:
  > 
  > - `database` - The name of the database being queried.
  > - `query` - The SQL query object containing the `sql` string.
  > - `canned_query` - The name of the canned query if this is a canned query. 
  > - `private` - Boolean indicating if this is a private database.
  > - `urls` - Object containing URL helpers like `database()`.
  > - `canned_write` - Boolean indicating if this canned query allows writes.
  > - `db_is_immutable` - Boolean indicating if this database is immutable. 
  > - `error` - Any query error message.
  > - `hide_sql` - Boolean indicating if the SQL should be hidden. 
  > - `show_hide_link` - The URL to toggle showing/hiding the SQL.
  > - `show_hide_text` - The text for the show/hide SQL link. 
  > - `editable` - Boolean indicating if the SQL can be edited.
  > - `allow_execute_sql` - Boolean indicating if custom SQL can be executed.
  > - `tables` - List of table objects in the database.
  > - `named_parameter_values` - Dictionary of parameter names/values.
  > - `csrftoken()` - Function to generate a CSRF token.
  > - `edit_sql_url` - URL to edit the SQL for a canned query.
  > - `display_rows` - List of result rows to display.
  > - `columns` - List of column names. 
  > - `renderers` - Dictionary of renderer name to URL.
  > - `url_csv` - URL for CSV export.
  > - `metadata` - Metadata about the query/database.
  > - `crumbs` - Breadcrumb navigation object.
  > 
  > So in summary, it includes variables for the database, query, results, URLs, permissions, and metadata to power the template.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2112#issuecomment-1652746185
issue_url: https://api.github.com/repos/simonw/datasette/issues/2112
node_id: IC_kwDOBm6k_c5igufJ
created_at: 2023-07-27T00:46:37Z
updated_at: 2023-07-27T00:46:37Z
author_association: OWNER
body:
  Got Claude to generate this code:
  ```python
  from dataclasses import dataclass, field

  @dataclass
  class QueryContext:
      database: str = field(metadata={"help": "The name of the database being queried"})
      query: dict = field(metadata={"help": "The SQL query object containing the `sql` string"})
      canned_query: str = field(metadata={"help": "The name of the canned query if this is a canned query"})
      private: bool = field(metadata={"help": "Boolean indicating if this is a private database"})
      urls: dict = field(metadata={"help": "Object containing URL helpers like `database()`"})
      canned_write: bool = field(metadata={"help": "Boolean indicating if this canned query allows writes"})
      db_is_immutable: bool = field(metadata={"help": "Boolean indicating if this database is immutable"})
      error: str = field(metadata={"help": "Any query error message"})
      hide_sql: bool = field(metadata={"help": "Boolean indicating if the SQL should be hidden"})
      show_hide_link: str = field(metadata={"help": "The URL to toggle showing/hiding the SQL"})
      show_hide_text: str = field(metadata={"help": "The text for the show/hide SQL link"})
      editable: bool = field(metadata={"help": "Boolean indicating if the SQL can be edited"})
      allow_execute_sql: bool = field(metadata={"help": "Boolean indicating if custom SQL can be executed"})
      tables: list = field(metadata={"help": "List of table objects in the database"})
      named_parameter_values: dict = field(metadata={"help": "Dictionary of parameter names/values"})
      csrftoken: callable = field(metadata={"help": "Function to generate a CSRF token"})
      edit_sql_url: str = field(metadata={"help": "URL to edit the SQL for a canned query"})
      display_rows: list = field(metadata={"help": "List of result rows to display"})
      columns: list = field(metadata={"help": "List of column names"})
      renderers: dict = field(metadata={"help": "Dictionary of renderer name to URL"})
      url_csv: str = field(metadata={"help": "URL for CSV export"})
      metadata: dict = field(metadata={"help": "Metadata about the query/database"})
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2112#issuecomment-1652751140
issue_url: https://api.github.com/repos/simonw/datasette/issues/2112
node_id: IC_kwDOBm6k_c5igvsk
created_at: 2023-07-27T00:52:53Z
updated_at: 2023-07-27T00:52:53Z
author_association: OWNER
body:
  Maybe I teach `datasette.render_template()` to take a dataclass instance as an optional alternative to a dictionary, and if it gets one it turns it into something that works well in the template context.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2121#issuecomment-1653666188
issue_url: https://api.github.com/repos/simonw/datasette/issues/2121
node_id: IC_kwDOBm6k_c5ikPGM
created_at: 2023-07-27T13:48:43Z
updated_at: 2023-07-27T13:48:43Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2121?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch and project coverage have no change.
  > Comparison is base [(`0818182`)](https://app.codecov.io/gh/simonw/datasette/commit/08181823990a71ffa5a1b57b37259198eaa43e06?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38% compared to head [(`3aca01f`)](https://app.codecov.io/gh/simonw/datasette/pull/2121?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38%.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main    #2121   +/-   ##
  =======================================
    Coverage   92.38%   92.38%           
  =======================================
    Files          39       39           
    Lines        5803     5803           
  =======================================
    Hits         5361     5361           
    Misses        442      442           
  ```




  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2121?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2124#issuecomment-1655691977
issue_url: https://api.github.com/repos/simonw/datasette/issues/2124
node_id: IC_kwDOBm6k_c5ir9rJ
created_at: 2023-07-28T13:33:54Z
updated_at: 2023-07-28T13:33:54Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2124?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch and project coverage have no change.
  > Comparison is base [(`0818182`)](https://app.codecov.io/gh/simonw/datasette/commit/08181823990a71ffa5a1b57b37259198eaa43e06?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38% compared to head [(`9388102`)](https://app.codecov.io/gh/simonw/datasette/pull/2124?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38%.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main    #2124   +/-   ##
  =======================================
    Coverage   92.38%   92.38%           
  =======================================
    Files          39       39           
    Lines        5803     5803           
  =======================================
    Hits         5361     5361           
    Misses        442      442           
  ```




  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2124?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/pull/38#issuecomment-1656694854
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/38
node_id: IC_kwDOD079W85ivyhG
created_at: 2023-07-29T10:00:45Z
updated_at: 2023-07-29T10:00:45Z
author_association: NONE
body:
  Ran across https://github.com/dogsheep/dogsheep-photos/issues/33 which is the same subject.  My PR just fixes docs

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/pull/31#issuecomment-1656696679
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/31
node_id: IC_kwDOD079W85ivy9n
created_at: 2023-07-29T10:10:29Z
updated_at: 2023-07-29T10:10:29Z
author_association: NONE
body:
  +1 to getting this merged down.

  For future googlers, I installed by...
  ```
  git clone git@github.com:RhetTbull/dogsheep-photos.git
  cd dogsheep-photos
  git checkout update_for_bigsur
  python setup.py install
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1159#issuecomment-1658968449
issue_url: https://api.github.com/repos/simonw/datasette/issues/1159
node_id: IC_kwDOBm6k_c5i4dmB
created_at: 2023-07-31T18:57:58Z
updated_at: 2023-07-31T18:57:58Z
author_association: OWNER
body:
  Added to the 1.0 milestone because: https://news.ycombinator.com/item?id=36932876#36939734

  >  Hah, wow, it looks like I've been procrastinating on making a decision if I like that or not for three years!
  >
  > I'll add it to the Datasette 1.0 milestone so it definitely gets my attention before shipping that release. 

reactions:
  {"total_count": 2, "+1": 0, "-1": 0, "laugh": 1, "hooray": 0, "confused": 0, "heart": 0, "rocket": 1, "eyes": 0}


html_url: https://github.com/simonw/datasette/pull/2125#issuecomment-1662228666
issue_url: https://api.github.com/repos/simonw/datasette/issues/2125
node_id: IC_kwDOBm6k_c5jE5i6
created_at: 2023-08-02T13:36:22Z
updated_at: 2023-08-02T13:36:22Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2125?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch and project coverage have no change.
  > Comparison is base [(`0818182`)](https://app.codecov.io/gh/simonw/datasette/commit/08181823990a71ffa5a1b57b37259198eaa43e06?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38% compared to head [(`730f554`)](https://app.codecov.io/gh/simonw/datasette/pull/2125?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38%.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main    #2125   +/-   ##
  =======================================
    Coverage   92.38%   92.38%           
  =======================================
    Files          39       39           
    Lines        5803     5803           
  =======================================
    Hits         5361     5361           
    Misses        442      442           
  ```




  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2125?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2126#issuecomment-1666912107
issue_url: https://api.github.com/repos/simonw/datasette/issues/2126
node_id: IC_kwDOBm6k_c5jWw9r
created_at: 2023-08-06T16:27:34Z
updated_at: 2023-08-06T16:27:34Z
author_association: NONE
body:
  And in similar fashion, how can I assign the `edit-tiddlywiki` permission to my user `myuser` in `metadata.yml` / `metadata.json`?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2127#issuecomment-1667011607
issue_url: https://api.github.com/repos/simonw/datasette/issues/2127
node_id: IC_kwDOBm6k_c5jXJQX
created_at: 2023-08-07T00:02:49Z
updated_at: 2023-08-07T00:02:49Z
author_association: OWNER
body:
  This is pretty neat:

  https://github.com/simonw/datasette/blob/2f9038a831a3510d4c9ab39a12d96259b3a55bc7/datasette/views/database.py#L157-L206

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/578#issuecomment-1668113177
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/578
node_id: IC_kwDOCGYnMM5jbWMZ
created_at: 2023-08-07T15:41:49Z
updated_at: 2023-08-07T15:41:49Z
author_association: CONTRIBUTOR
body:
  I wonder if this should be two hooks: input and output. The current `--csv` (and `--tsv`) options apply to both. Haven't looked at how it's implemented. Or maybe it's one hook that returns a format for reading and for writing.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2128#issuecomment-1668143992
issue_url: https://api.github.com/repos/simonw/datasette/issues/2128
node_id: IC_kwDOBm6k_c5jbdt4
created_at: 2023-08-07T15:58:08Z
updated_at: 2023-08-07T15:58:08Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2128?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch and project coverage have no change.
  > Comparison is base [(`adf54f5`)](https://app.codecov.io/gh/simonw/datasette/commit/adf54f5c808c16e9ad812c3a4376f35cf829a93c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38% compared to head [(`935d336`)](https://app.codecov.io/gh/simonw/datasette/pull/2128?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38%.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main    #2128   +/-   ##
  =======================================
    Coverage   92.38%   92.38%           
  =======================================
    Files          39       39           
    Lines        5803     5803           
  =======================================
    Hits         5361     5361           
    Misses        442      442           
  ```




  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2128?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2118#issuecomment-1668578107
issue_url: https://api.github.com/repos/simonw/datasette/issues/2118
node_id: IC_kwDOBm6k_c5jdHs7
created_at: 2023-08-07T21:06:50Z
updated_at: 2023-08-07T21:06:50Z
author_association: OWNER
body:
  15:
  ```
  =========================================================================================== short test summary info ============================================================================================
  FAILED tests/test_cli.py::test_sql_errors_logged_to_stderr - assert 0 == 1
  FAILED tests/test_html.py::test_show_hide_sql_query - AssertionError: assert False
  FAILED tests/test_canned_queries.py::test_magic_parameters_cannot_be_used_in_arbitrary_queries - assert 400 == 200
  FAILED tests/test_html.py::test_templates_considered[/fixtures-database-fixtures.html, *database.html] - assert '<!-- Templates considered: database-fixtures.html, *database.html -->' in '<!DOCTYPE html>\n<html>\n<head>\n    <title>fixtures</title>\n    <link rel="stylesheet" href="/-/static/app.css?d599......
  FAILED tests/test_html.py::test_query_json_csv_export_links - assert '<a href="/fixtures.csv?sql=select+1&amp;_size=max">CSV</a>' in '<!DOCTYPE html>\n<html>\n<head>\n    <title>fixtures: select 1</title>\n    <link rel="stylesheet" href="/-/static/ap...se"}, "view...
  FAILED tests/test_html.py::test_base_url_config[True-/fixtures?sql=select+1] - AssertionError: {
  FAILED tests/test_api.py::test_cors[/fixtures.json?sql=select+blah-400] - assert 200 == 400
  FAILED tests/test_html.py::test_alternate_url_json[/fixtures?sql=select+*+from+facetable-http://localhost/fixtures.json?sql=select+*+from+facetable] - assert '<link rel="alternate" type="application/json+datasette" href="http://localhost/fixtures.json?sql=select+*+from+facetable">' in '<!DOCTYPE html>\n<html>\n<head>\n    <title>fixtures: select * from...
  FAILED tests/test_plugins.py::test_hook_extra_css_urls[/fixtures-expected_decoded_object1] - AssertionError: assert equals failed
  FAILED tests/test_plugins.py::test_view_names[/fixtures-database] - AssertionError: assert equals failed
  FAILED tests/test_plugins.py::test_view_names[/fixtures?sql=select+1-database] - AssertionError: assert equals failed
  FAILED tests/test_plugins.py::test_hook_extra_body_script[/fixtures-expected_extra_body_script1] - AssertionError: assert equals failed
  FAILED tests/test_html.py::test_base_url_config[False-/fixtures?sql=select+1] - AssertionError: {
  FAILED tests/test_table_api.py::test_max_returned_rows - KeyError: 'query'
  FAILED tests/test_html.py::test_alternate_url_json[/fixtures-http://localhost/fixtures.json] - KeyError: 'link'
  ============================================================================ 15 failed, 1297 passed, 2 skipped, 1 xfailed in 58.15s ============================================================================
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2118#issuecomment-1668687891
issue_url: https://api.github.com/repos/simonw/datasette/issues/2118
node_id: IC_kwDOBm6k_c5jdigT
created_at: 2023-08-07T23:11:04Z
updated_at: 2023-08-07T23:14:34Z
author_association: OWNER
body:
  One of the failing tests is because this URL on my local environment returns this:

  http://127.0.0.1:8001/_memory.json?sql=select+blah
  ```json
  {"rows": [], "columns": [], "truncated": false}
  ```
  With a 200 status code. It should be an error.

  https://latest.datasette.io/_memory.json?sql=select+blah returns a 400 status code with the following JSON:

  ```json
  {
    "ok": false,
    "database": "_memory",
    "query_name": null,
    "rows": [],
    "truncated": false,
    "columns": [],
    "query": {
      "sql": "select blah",
      "params": {}
    },
    "error": "no such column: blah",
    "private": false,
    "allow_execute_sql": true,
    "query_ms": 2.68310698447749,
    "source": "tests/fixtures.py",
    "source_url": "https://github.com/simonw/datasette/blob/main/tests/fixtures.py",
    "license": "Apache License 2.0",
    "license_url": "https://github.com/simonw/datasette/blob/main/LICENSE"
  }
  ```
  So I need a design for what this should look like for the error case.

  I should consider what errors look like for alternative formats too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2118#issuecomment-1668688947
issue_url: https://api.github.com/repos/simonw/datasette/issues/2118
node_id: IC_kwDOBm6k_c5jdiwz
created_at: 2023-08-07T23:12:37Z
updated_at: 2023-08-07T23:12:37Z
author_association: OWNER
body:
  https://latest.datasette.io/_memory.csv?sql=select+blah is a blank page right now:

  ```bash
  curl -I 'https://latest.datasette.io/_memory.csv?sql=select+blah'
  ```
  ```
  HTTP/2 200 
  access-control-allow-origin: *
  access-control-allow-headers: Authorization, Content-Type
  access-control-expose-headers: Link
  access-control-allow-methods: GET, POST, HEAD, OPTIONS
  access-control-max-age: 3600
  content-type: text/plain; charset=utf-8
  x-databases: _memory, _internal, fixtures, fixtures2, extra_database, ephemeral
  date: Mon, 07 Aug 2023 23:12:15 GMT
  server: Google Frontend
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2118#issuecomment-1668691898
issue_url: https://api.github.com/repos/simonw/datasette/issues/2118
node_id: IC_kwDOBm6k_c5jdje6
created_at: 2023-08-07T23:15:14Z
updated_at: 2023-08-07T23:16:28Z
author_association: OWNER
body:
  New error design:
  ```json
  {"ok": false, "error": "Error message here", "rows": [], "columns": [], "truncated": false}
  ```
  I guess that means I need to add `"ok": true` to the default format as well - which makes sense because table JSON has that already:

  https://latest.datasette.io/fixtures/facetable.json

  ```json
  {
    "ok": true,
    "next": null,
    "rows": [...]
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2130#issuecomment-1668695958
issue_url: https://api.github.com/repos/simonw/datasette/issues/2130
node_id: IC_kwDOBm6k_c5jdkeW
created_at: 2023-08-07T23:21:07Z
updated_at: 2023-08-07T23:21:16Z
author_association: OWNER
body:
  Main place the code needs updating is here: https://github.com/simonw/datasette/blob/5139c0886a7f6bb94d317ba0665aa6e728716028/datasette/views/base.py#L301-L310

  Plus the documentation.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2118#issuecomment-1668715803
issue_url: https://api.github.com/repos/simonw/datasette/issues/2118
node_id: IC_kwDOBm6k_c5jdpUb
created_at: 2023-08-07T23:50:26Z
updated_at: 2023-08-08T01:41:53Z
author_association: OWNER
body:
  This bit needs some more work: https://github.com/simonw/datasette/blob/a791115e31e64e9bbd1d36f4d46887b3e42e752a/datasette/views/database.py#L514-L518

  The original logic for that is from here, I need to figure out how to replicate it:

  https://github.com/simonw/datasette/blob/5139c0886a7f6bb94d317ba0665aa6e728716028/datasette/views/base.py#L368

  https://github.com/simonw/datasette/blob/5139c0886a7f6bb94d317ba0665aa6e728716028/datasette/views/base.py#L342-L344

  UPDATE: Fixed the `_size=max` bit, but I've not replicated the behaviour where it adds `?_labels=on` if there are expandable columns yet.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2118#issuecomment-1668720671
issue_url: https://api.github.com/repos/simonw/datasette/issues/2118
node_id: IC_kwDOBm6k_c5jdqgf
created_at: 2023-08-07T23:58:21Z
updated_at: 2023-08-07T23:58:21Z
author_association: OWNER
body:
  11 tests left:
  ```
  FAILED tests/test_html.py::test_alternate_url_json[/fixtures-http://localhost/fixtures.json] - KeyError: 'link'
  FAILED tests/test_html.py::test_alternate_url_json[/fixtures?sql=select+*+from+facetable-http://localhost/fixtures.json?sql=select+*+from+facetable] - assert '<link rel="alternate" type="application/json+datasette" href="http://localhost/fixtures.json?sql=select+*+from+facetable">' in '<!DOCTYPE html>\n<html>\n<head>\n    <title>fixtures: select * from...
  FAILED tests/test_html.py::test_query_page_truncates - assert equals failed
  FAILED tests/test_html.py::test_templates_considered[/fixtures-database-fixtures.html, *database.html] - assert '<!-- Templates considered: database-fixtures.html, *database.html -->' in '<!DOCTYPE html>\n<html>\n<head>\n    <title>fixtures</title>\n    <link rel="stylesheet" href="/-/static/app.css?d599......
  FAILED tests/test_html.py::test_query_json_csv_export_links - assert '<a href="/fixtures.csv?sql=select+1&amp;_size=max">CSV</a>' in '<!DOCTYPE html>\n<html>\n<head>\n    <title>fixtures: select 1</title>\n    <link rel="stylesheet" href="/-/static/ap...se"}, "view...
  FAILED tests/test_plugins.py::test_hook_extra_css_urls[/fixtures-expected_decoded_object1] - AssertionError: assert equals failed
  FAILED tests/test_api.py::test_cors[/fixtures.json?sql=select+blah-400] - KeyError: 'Access-Control-Allow-Origin'
  FAILED tests/test_plugins.py::test_hook_extra_body_script[/fixtures-expected_extra_body_script1] - AssertionError: assert equals failed
  FAILED tests/test_table_api.py::test_max_returned_rows - KeyError: 'query'
  FAILED tests/test_plugins.py::test_view_names[/fixtures-database] - AssertionError: assert equals failed
  FAILED tests/test_plugins.py::test_view_names[/fixtures?sql=select+1-database] - AssertionError: assert equals failed
  ======================================================================= 11 failed, 1301 passed, 2 skipped, 1 xfailed in 64.10s (0:01:04) =======================================================================
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2118#issuecomment-1668781558
issue_url: https://api.github.com/repos/simonw/datasette/issues/2118
node_id: IC_kwDOBm6k_c5jd5X2
created_at: 2023-08-08T01:34:42Z
updated_at: 2023-08-08T01:34:42Z
author_association: OWNER
body:
  3 failures left:
  ```
  =================================================================== short test summary info ====================================================================
  FAILED tests/test_html.py::test_query_json_csv_export_links - assert '<a href="/fixtures.csv?sql=select+1&amp;_size=max">CSV</a>' in '<!DOCTYPE html>\n<html>\n<head>\n    <title>fixtures: select 1</title>\n    <link r...
  FAILED tests/test_api.py::test_cors[/fixtures.json?sql=select+blah-400] - KeyError: 'Access-Control-Allow-Origin'
  FAILED tests/test_table_api.py::test_max_returned_rows - KeyError: 'query'
  =============================================== 3 failed, 1309 passed, 2 skipped, 1 xfailed in 61.34s (0:01:01) ================================================
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2118#issuecomment-1668786770
issue_url: https://api.github.com/repos/simonw/datasette/issues/2118
node_id: IC_kwDOBm6k_c5jd6pS
created_at: 2023-08-08T01:42:55Z
updated_at: 2023-08-08T01:42:55Z
author_association: OWNER
body:
  > UPDATE: Fixed the `_size=max` bit, but I've not replicated the behaviour where it adds `?_labels=on` if there are expandable columns yet.

  It looks like that behaviour is only relevant to table views, and it's already implemented - https://latest.datasette.io/fixtures/roadside_attraction_characteristics links to https://latest.datasette.io/fixtures/roadside_attraction_characteristics.csv?_labels=on&_size=max

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2118#issuecomment-1668789134
issue_url: https://api.github.com/repos/simonw/datasette/issues/2118
node_id: IC_kwDOBm6k_c5jd7OO
created_at: 2023-08-08T01:46:12Z
updated_at: 2023-08-08T01:46:12Z
author_association: OWNER
body:
  I'm going to squash merge this even though it's a bit big, because I want a clean passing commit in the `main` branch.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2127#issuecomment-1668792576
issue_url: https://api.github.com/repos/simonw/datasette/issues/2127
node_id: IC_kwDOBm6k_c5jd8EA
created_at: 2023-08-08T01:51:14Z
updated_at: 2023-08-08T01:51:14Z
author_association: OWNER
body:
  OK, I've landed the first example of this now. I'm cautiously optimistic about it, but I think I need to use it in a few more places AND add the actual code that uses it for documentation before I'm completely happy and close this issue.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2130#issuecomment-1668793100
issue_url: https://api.github.com/repos/simonw/datasette/issues/2130
node_id: IC_kwDOBm6k_c5jd8MM
created_at: 2023-08-08T01:51:54Z
updated_at: 2023-08-08T01:51:54Z
author_association: OWNER
body:
  Updated documentation is here: https://docs.datasette.io/en/latest/plugin_hooks.html#register-output-renderer-datasette

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2111#issuecomment-1668798261
issue_url: https://api.github.com/repos/simonw/datasette/issues/2111
node_id: IC_kwDOBm6k_c5jd9c1
created_at: 2023-08-08T02:00:22Z
updated_at: 2023-08-08T02:00:37Z
author_association: OWNER
body:
  Shipped: https://latest.datasette.io/_memory.json?sql=select+sqlite_version()
  ```json
  {
    "rows": [
      {
        "sqlite_version()": "3.34.1"
      }
    ],
    "columns": [
      "sqlite_version()"
    ],
    "ok": true,
    "truncated": false
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2113#issuecomment-1668983779
issue_url: https://api.github.com/repos/simonw/datasette/issues/2113
node_id: IC_kwDOBm6k_c5jeqvj
created_at: 2023-08-08T06:27:42Z
updated_at: 2023-08-08T06:27:42Z
author_association: OWNER
body:
  I've now shipped the new query JSON without any extras: https://latest.datasette.io/fixtures.json?sql=select+*+from+facetable

  There aren't actually that many extras that make sense for the query page. These are the most obvious:

  - `query` - adding a `{"sql": "...", "params": ".."}` block, as seen on https://latest.datasette.io/fixtures/facetable.json?_extra=query&_size=0
  - `duration_ms` to show the duration - which isn't an extra on table right now but should be.
  - `metadata` - https://latest.datasette.io/fixtures/facetable.json?_extra=metadata&_size=0

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2132#issuecomment-1669640626
issue_url: https://api.github.com/repos/simonw/datasette/issues/2132
node_id: IC_kwDOBm6k_c5jhLGy
created_at: 2023-08-08T13:39:54Z
updated_at: 2023-08-08T13:39:54Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/blob/1377a290cd85ba8d3338b1da47c4665ed4c6c625/datasette/views/database.py#L509-L517

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/pull/31#issuecomment-1669877769
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/31
node_id: IC_kwDOD079W85jiFAJ
created_at: 2023-08-08T15:52:52Z
updated_at: 2023-08-08T15:52:52Z
author_association: NONE
body:
  You can also install this with pip using this oneliner:

  ```
  pip install git+https://github.com/RhetTbull/dogsheep-photos.git@update_for_bigsur
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2114#issuecomment-1670454222
issue_url: https://api.github.com/repos/simonw/datasette/issues/2114
node_id: IC_kwDOBm6k_c5jkRvO
created_at: 2023-08-08T23:44:42Z
updated_at: 2023-08-08T23:44:42Z
author_association: OWNER
body:
  The tests for this are still passing:

  ```bash
  pytest tests/test_canned_queries.py
  ```
  Because I left in the old code:

  https://github.com/simonw/datasette/blob/cd57b0f71234273156cb1eba3f9153b9e27ac14d/datasette/views/table.py#L752-L767

  https://github.com/simonw/datasette/blob/cd57b0f71234273156cb1eba3f9153b9e27ac14d/datasette/views/table.py#L701-L727

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2114#issuecomment-1670470516
issue_url: https://api.github.com/repos/simonw/datasette/issues/2114
node_id: IC_kwDOBm6k_c5jkVt0
created_at: 2023-08-09T00:09:55Z
updated_at: 2023-08-09T00:10:32Z
author_association: OWNER
body:
  I decided to figure out where the `_size=` parameter here comes from: https://github.com/simonw/datasette/blob/cd57b0f71234273156cb1eba3f9153b9e27ac14d/datasette/views/database.py#L576-L588

  It took some spelunking, but it looks like I added that five years ago as part of CSV streaming in https://github.com/simonw/datasette/commit/fc3660cfad7668dbce6ead12766e048fc1f78b11

  It's there purely to allow CSV streaming to set `_size="max"`.

  And since CSV streaming is still just a table thing, not a query thing, I can drop it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2134#issuecomment-1671500965
issue_url: https://api.github.com/repos/simonw/datasette/issues/2134
node_id: IC_kwDOBm6k_c5joRSl
created_at: 2023-08-09T14:31:43Z
updated_at: 2023-08-09T14:31:43Z
author_association: OWNER
body:
  Here's a simple plugin that can do this:
  ```python
  from datasette import hookimpl

  @hookimpl
  def startup(datasette):
      db = datasette.add_memory_database("counters")

      async def inner():
          await db.execute_write("create table if not exists counters (name text primary key, value integer)")
          await db.execute_write("insert or ignore into counters (name, value) values ('counter', 0)")

      return inner


  @hookimpl
  def canned_queries(database):
      if database == "counters":
          return {
              "increment": {
                  "sql": "update counters set value = value + 1 where name = 'counter'",
                  "write": True,
              },
              "decrement": {
                  "sql": "update counters set value = value - 1 where name = 'counter'",
                  "write": True,
              },
          }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2134#issuecomment-1671503163
issue_url: https://api.github.com/repos/simonw/datasette/issues/2134
node_id: IC_kwDOBm6k_c5joR07
created_at: 2023-08-09T14:32:30Z
updated_at: 2023-08-09T14:32:30Z
author_association: OWNER
body:
  Annoying thing about this plugin is that you don't see the new counter value when you submit the increment or decrement query.

  Maybe canned queries should support SQL multiple statements? Could return the result of the last one.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2134#issuecomment-1671526372
issue_url: https://api.github.com/repos/simonw/datasette/issues/2134
node_id: IC_kwDOBm6k_c5joXfk
created_at: 2023-08-09T14:39:54Z
updated_at: 2023-08-09T14:39:54Z
author_association: OWNER
body:
  Alternatively, what about if there was a custom SQL function available during canned write queries for setting the output message? Then the fact that canned queries don't return a table view wouldn't be a problem. Bit weird though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2134#issuecomment-1671581610
issue_url: https://api.github.com/repos/simonw/datasette/issues/2134
node_id: IC_kwDOBm6k_c5jok-q
created_at: 2023-08-09T15:01:14Z
updated_at: 2023-08-10T00:19:56Z
author_association: OWNER
body:
  Or... how about if the `on_success_message` option could define a SQL query to be executed to generate that message? Maybe `on_success_message_sql`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2131#issuecomment-1671626876
issue_url: https://api.github.com/repos/simonw/datasette/issues/2131
node_id: IC_kwDOBm6k_c5jowB8
created_at: 2023-08-09T15:27:40Z
updated_at: 2023-08-09T15:27:40Z
author_association: OWNER
body:
  Another place that needs fixing: https://github.com/simonw/datasette/blob/26be9f0445b753fb84c802c356b0791a72269f25/datasette/views/database.py#L244-L249

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2133#issuecomment-1671628602
issue_url: https://api.github.com/repos/simonw/datasette/issues/2133
node_id: IC_kwDOBm6k_c5jowc6
created_at: 2023-08-09T15:28:42Z
updated_at: 2023-08-09T15:28:42Z
author_association: OWNER
body:
  Neat idea! We already have this command:

      datasette install -r requirements.txt

  So maybe `datasetet plugins --requirements` could generate a `requirements.txt` file which could be used there.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2133#issuecomment-1671634654
issue_url: https://api.github.com/repos/simonw/datasette/issues/2133
node_id: IC_kwDOBm6k_c5jox7e
created_at: 2023-08-09T15:32:14Z
updated_at: 2023-08-09T15:32:14Z
author_association: OWNER
body:
  ```diff
  diff --git a/datasette/cli.py b/datasette/cli.py
  index 32266888..370b722c 100644
  --- a/datasette/cli.py
  +++ b/datasette/cli.py
  @@ -223,15 +223,22 @@ pm.hook.publish_subcommand(publish=publish)
   
   @cli.command()
   @click.option("--all", help="Include built-in default plugins", is_flag=True)
  +@click.option(
  +    "--requirements", help="Output requirements.txt of installed plugins", is_flag=True
  +)
   @click.option(
       "--plugins-dir",
       type=click.Path(exists=True, file_okay=False, dir_okay=True),
       help="Path to directory containing custom plugins",
   )
  -def plugins(all, plugins_dir):
  +def plugins(all, requirements, plugins_dir):
       """List currently installed plugins"""
       app = Datasette([], plugins_dir=plugins_dir)
  -    click.echo(json.dumps(app._plugins(all=all), indent=4))
  +    if requirements:
  +        for plugin in app._plugins():
  +            click.echo("{}=={}".format(plugin["name"], plugin["version"]))
  +    else:
  +        click.echo(json.dumps(app._plugins(all=all), indent=4))
   
   
   @cli.command()
  ```
  Output:
  ```
  % datasette plugins --requirements  
  datasette-codespaces==0.1.1
  datasette-graphql==2.2
  datasette-json-html==1.0.1
  datasette-pretty-json==0.2.2
  datasette-x-forwarded-host==0.1
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2133#issuecomment-1671649530
issue_url: https://api.github.com/repos/simonw/datasette/issues/2133
node_id: IC_kwDOBm6k_c5jo1j6
created_at: 2023-08-09T15:41:14Z
updated_at: 2023-08-09T15:41:14Z
author_association: NONE
body:
  Yes,  using this approach(`datasette install -r requirements.txt`) will result in more consistency. 

  I'm curious about the results of the `datasette plugins --all` command. Where will we use the output of this command? Will it include configuration information for these plugins in the future? If so, will we need to consider the configuration of these plugins in addition to installing them on different computers?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2133#issuecomment-1671720761
issue_url: https://api.github.com/repos/simonw/datasette/issues/2133
node_id: IC_kwDOBm6k_c5jpG85
created_at: 2023-08-09T16:05:33Z
updated_at: 2023-08-09T16:05:33Z
author_association: OWNER
body:
  Honestly the `--all` option isn't particularly useful - it's really more of a debugging tool for myself than something that I expect anyone else to use.

  The purpose of `datasette plugins` generally is to help people confirm that plugins installed correctly. It's effectively the CLI alternative to hitting https://datasette.io/-/plugins

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2133#issuecomment-1671722482
issue_url: https://api.github.com/repos/simonw/datasette/issues/2133
node_id: IC_kwDOBm6k_c5jpHXy
created_at: 2023-08-09T16:06:28Z
updated_at: 2023-08-09T16:06:28Z
author_association: OWNER
body:
  Plugin configuration itself currently lives in `metadata.json` which I don't like - that file was supposed to be for metadata about your databases, it's a bit of an accident that it grew configuration options as well.

  I'm hoping to fix that for Datasette 1.0, but I'm still figuring out how to do that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2135#issuecomment-1671750913
issue_url: https://api.github.com/repos/simonw/datasette/issues/2135
node_id: IC_kwDOBm6k_c5jpOUB
created_at: 2023-08-09T16:24:26Z
updated_at: 2023-08-09T16:24:26Z
author_association: OWNER
body:
  Remember not to include changes from this:

  https://github.com/simonw/datasette/blob/26be9f0445b753fb84c802c356b0791a72269f25/docs/changelog.rst?plain=1#L9-L31

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2135#issuecomment-1671753753
issue_url: https://api.github.com/repos/simonw/datasette/issues/2135
node_id: IC_kwDOBm6k_c5jpPAZ
created_at: 2023-08-09T16:26:17Z
updated_at: 2023-08-09T16:28:19Z
author_association: OWNER
body:
  - Dropped support for Python 3.7
  - New `Justfile`
  - #1970
  - `datasette.render_template()` now accepts a `Context` subclass as an alternative to a dictionary
  - #2106
  - #2007
  - https://github.com/simonw/datasette/issues/2130
  - https://github.com/simonw/datasette/issues/2079
  - https://github.com/simonw/datasette/issues/1153

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2007#issuecomment-1671836347
issue_url: https://api.github.com/repos/simonw/datasette/issues/2007
node_id: IC_kwDOBm6k_c5jpjK7
created_at: 2023-08-09T17:15:03Z
updated_at: 2023-08-09T17:15:03Z
author_association: OWNER
body:
  Updated documentation: https://docs.datasette.io/en/latest/plugin_hooks.html#render-cell-row-value-column-table-database-datasette-request

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2135#issuecomment-1671844664
issue_url: https://api.github.com/repos/simonw/datasette/issues/2135
node_id: IC_kwDOBm6k_c5jplM4
created_at: 2023-08-09T17:21:22Z
updated_at: 2023-08-09T17:21:22Z
author_association: OWNER
body:
  OK, I've done the small changes - still need to document the big JSON changes though.

  Those aren't covered by the regular documentation either so I need to update that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2136#issuecomment-1671848664
issue_url: https://api.github.com/repos/simonw/datasette/issues/2136
node_id: IC_kwDOBm6k_c5jpmLY
created_at: 2023-08-09T17:24:37Z
updated_at: 2023-08-09T17:24:37Z
author_association: OWNER
body:
  It also bothers me that `ok` comes first in one but not first in the other - I'd like the order of keys in the JSON to be consistent, even though order isn't meant to mean anything.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2113#issuecomment-1671862711
issue_url: https://api.github.com/repos/simonw/datasette/issues/2113
node_id: IC_kwDOBm6k_c5jppm3
created_at: 2023-08-09T17:35:22Z
updated_at: 2023-08-09T17:35:22Z
author_association: OWNER
body:
  Need `?_extra=columns` too since I'm removing `"columns"` from the default query JSON in https://github.com/simonw/datasette/issues/2136

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2136#issuecomment-1671964003
issue_url: https://api.github.com/repos/simonw/datasette/issues/2136
node_id: IC_kwDOBm6k_c5jqCVj
created_at: 2023-08-09T18:53:10Z
updated_at: 2023-08-09T18:53:18Z
author_association: OWNER
body:
  I made an initial attempt at this in a branch, but it broke too much other stuff.  https://github.com/simonw/datasette/commit/f3944608cc000a2542b4fe9a7a89c866391924f2

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2136#issuecomment-1671982215
issue_url: https://api.github.com/repos/simonw/datasette/issues/2136
node_id: IC_kwDOBm6k_c5jqGyH
created_at: 2023-08-09T19:02:19Z
updated_at: 2023-08-09T19:02:19Z
author_association: OWNER
body:
  I'm going with a bit of a hacky workaround for the moment, until I get a chance to refactor this:
  - #2137 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2137#issuecomment-1671983228
issue_url: https://api.github.com/repos/simonw/datasette/issues/2137
node_id: IC_kwDOBm6k_c5jqHB8
created_at: 2023-08-09T19:02:47Z
updated_at: 2023-08-09T19:02:47Z
author_association: OWNER
body:
  As part of this I'd like to refactor the JSON renderer code - I started attempting that here: https://github.com/simonw/datasette/commit/f3944608cc000a2542b4fe9a7a89c866391924f2

  In particular I want to get rid of that `data` argument.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2133#issuecomment-1672076284
issue_url: https://api.github.com/repos/simonw/datasette/issues/2133
node_id: IC_kwDOBm6k_c5jqdv8
created_at: 2023-08-09T20:09:12Z
updated_at: 2023-08-09T20:09:12Z
author_association: OWNER
body:
  Problem - if there are plugins in the plugins directory it was returning this:
  ```
  messages_output_renderer.py==None
  my_plugin.py==None
  my_plugin_2.py==None
  register_output_renderer.py==None
  sleep_sql_function.py==None
  view_name.py==None
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2138#issuecomment-1672353428
issue_url: https://api.github.com/repos/simonw/datasette/issues/2138
node_id: IC_kwDOBm6k_c5jrhaU
created_at: 2023-08-10T00:20:56Z
updated_at: 2023-08-10T00:20:56Z
author_association: OWNER
body:
  Goal here is to customize the message displayed after a query has succeeded. e.g.

  ```yaml
  databases:
    mydatabase:
      queries:
        add_name:
          sql: INSERT INTO names (name) VALUES (:name)
          write: true
          on_success_message_sql: 'select ''Name inserted: '' || :name'
          on_success_redirect: /mydatabase/names
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2133#issuecomment-1672360472
issue_url: https://api.github.com/repos/simonw/datasette/issues/2133
node_id: IC_kwDOBm6k_c5jrjIY
created_at: 2023-08-10T00:31:24Z
updated_at: 2023-08-10T00:31:24Z
author_association: NONE
body:
  It looks very nice now.
  Finally, no more manual installation of plugins one by one. Thank you, Simon!  ❤️ 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2138#issuecomment-1672366410
issue_url: https://api.github.com/repos/simonw/datasette/issues/2138
node_id: IC_kwDOBm6k_c5jrklK
created_at: 2023-08-10T00:39:16Z
updated_at: 2023-08-10T00:39:16Z
author_association: OWNER
body:
  Here's what it looks like when it works:

  <img width="715" alt="image" src="https://github.com/simonw/datasette/assets/9599/a53efc81-8ceb-44a2-b083-2efb27d20cf7">

  And here's what you get if there's a SQL error in the `on_success_message_sql` query itself:

  <img width="715" alt="CleanShot 2023-08-09 at 17 38 12@2x" src="https://github.com/simonw/datasette/assets/9599/e211cee2-3edb-475a-906d-b0bb5a118d77">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2134#issuecomment-1672384439
issue_url: https://api.github.com/repos/simonw/datasette/issues/2134
node_id: IC_kwDOBm6k_c5jro-3
created_at: 2023-08-10T01:05:56Z
updated_at: 2023-08-10T01:22:46Z
author_association: OWNER
body:
  Now live at https://latest.datasette.io/counters

  ![increment](https://github.com/simonw/datasette/assets/9599/ffb3625b-b61b-4189-80e3-7b4e8b411f14)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2126#issuecomment-1672385674
issue_url: https://api.github.com/repos/simonw/datasette/issues/2126
node_id: IC_kwDOBm6k_c5jrpSK
created_at: 2023-08-10T01:07:43Z
updated_at: 2023-08-10T01:07:43Z
author_association: OWNER
body:
  What version of Datasette are you running?

  That feature was added in Datasette 1.0a2, so if you're on the current stable release you won't have it yet.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2127#issuecomment-1672404276
issue_url: https://api.github.com/repos/simonw/datasette/issues/2127
node_id: IC_kwDOBm6k_c5jrt00
created_at: 2023-08-10T01:30:25Z
updated_at: 2023-08-10T01:30:25Z
author_association: OWNER
body:
  Here's the prototype of adding documentation for these using Cog: https://github.com/simonw/datasette/issues/1510#issuecomment-1610512875

  ![](https://user-images.githubusercontent.com/9599/249324964-79210a43-716c-4470-a62d-01aa871973ec.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2139#issuecomment-1674115279
issue_url: https://api.github.com/repos/simonw/datasette/issues/2139
node_id: IC_kwDOBm6k_c5jyPjP
created_at: 2023-08-11T01:27:30Z
updated_at: 2023-08-11T01:27:30Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/blob/33251d04e78d575cca62bb59069bb43a7d924746/datasette/templates/database.html#L13

  Looks like I should be doing this instead:

  https://github.com/simonw/datasette/blob/33251d04e78d575cca62bb59069bb43a7d924746/datasette/templates/index.html#L13

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2139#issuecomment-1674210009
issue_url: https://api.github.com/repos/simonw/datasette/issues/2139
node_id: IC_kwDOBm6k_c5jymrZ
created_at: 2023-08-11T04:52:44Z
updated_at: 2023-08-11T04:52:44Z
author_association: OWNER
body:
  I'll fix these too:
  - https://github.com/simonw/datasette/issues/2119
  - https://github.com/simonw/datasette/issues/568

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2119#issuecomment-1674210215
issue_url: https://api.github.com/repos/simonw/datasette/issues/2119
node_id: IC_kwDOBm6k_c5jymun
created_at: 2023-08-11T04:53:11Z
updated_at: 2023-08-11T04:53:11Z
author_association: OWNER
body:
  Relevant template snippet:

  https://github.com/simonw/datasette/blob/33251d04e78d575cca62bb59069bb43a7d924746/datasette/templates/index.html#L13

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2119#issuecomment-1674211804
issue_url: https://api.github.com/repos/simonw/datasette/issues/2119
node_id: IC_kwDOBm6k_c5jynHc
created_at: 2023-08-11T04:56:28Z
updated_at: 2023-08-11T04:56:28Z
author_association: OWNER
body:
  That `database.color` is only available in `index.html` at the moment thanks to this:

  https://github.com/simonw/datasette/blob/33251d04e78d575cca62bb59069bb43a7d924746/datasette/views/index.py#L108-L110

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/568#issuecomment-1674223722
issue_url: https://api.github.com/repos/simonw/datasette/issues/568
node_id: IC_kwDOBm6k_c5jyqBq
created_at: 2023-08-11T05:19:42Z
updated_at: 2023-08-11T05:19:42Z
author_association: OWNER
body:
  I decided not to implement this yet, because making `database.color` dependent on metadata interferes with my goal to eventually make metadata lookups `async` in order to serve them better from plugins.

  I'll leave this issue open though as it may get solved in future redesigns of the metadata system, refs:

  - #1384

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2119#issuecomment-1674227353
issue_url: https://api.github.com/repos/simonw/datasette/issues/2119
node_id: IC_kwDOBm6k_c5jyq6Z
created_at: 2023-08-11T05:25:45Z
updated_at: 2023-08-11T05:25:45Z
author_association: OWNER
body:
  Demo:

  - https://latest.datasette.io/
  - https://latest.datasette.io/fixtures
  - https://latest.datasette.io/fixtures/facetable
  - https://latest.datasette.io/fixtures/pragma_cache_size
  - https://latest.datasette.io/fixtures/attraction_characteristic/2

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2140#issuecomment-1675157445
issue_url: https://api.github.com/repos/simonw/datasette/issues/2140
node_id: IC_kwDOBm6k_c5j2N_F
created_at: 2023-08-11T17:45:54Z
updated_at: 2023-08-11T17:51:59Z
author_association: OWNER
body:
  Affected pages:

  - https://docs.datasette.io/en/latest/authentication.html
  - https://docs.datasette.io/en/latest/changelog.html
  - https://docs.datasette.io/en/latest/cli-reference.html
  - https://docs.datasette.io/en/latest/contributing.html
  - https://docs.datasette.io/en/latest/custom_templates.html
  - https://docs.datasette.io/en/latest/deploying.html
  - https://docs.datasette.io/en/latest/facets.html
  - https://docs.datasette.io/en/latest/full_text_search.html
  - https://docs.datasette.io/en/latest/installation.html
  - https://docs.datasette.io/en/latest/plugin_hooks.html
  - https://docs.datasette.io/en/latest/publish.html
  - https://docs.datasette.io/en/latest/settings.html
  - https://docs.datasette.io/en/latest/spatialite.html
  - https://docs.datasette.io/en/latest/sql_queries.html

  e.g. on https://docs.datasette.io/en/latest/authentication.html#using-the-root-actor

  <img width="759" alt="image" src="https://github.com/simonw/datasette/assets/9599/aa3ea5bd-1b4b-4ecf-a5ed-a62508dcde31">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2141#issuecomment-1680660343
issue_url: https://api.github.com/repos/simonw/datasette/issues/2141
node_id: IC_kwDOBm6k_c5kLNd3
created_at: 2023-08-16T13:55:22Z
updated_at: 2023-08-16T13:55:22Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2141?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch and project coverage have no change.
  > Comparison is base [(`943df09`)](https://app.codecov.io/gh/simonw/datasette/commit/943df09dcca93c3b9861b8c96277a01320db8662?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.06% compared to head [(`e2be94a`)](https://app.codecov.io/gh/simonw/datasette/pull/2141?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.06%.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main    #2141   +/-   ##
  =======================================
    Coverage   92.06%   92.06%           
  =======================================
    Files          40       40           
    Lines        5937     5937           
  =======================================
    Hits         5466     5466           
    Misses        471      471           
  ```




  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2141?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2142#issuecomment-1682272492
issue_url: https://api.github.com/repos/simonw/datasette/issues/2142
node_id: IC_kwDOBm6k_c5kRXDs
created_at: 2023-08-17T13:18:08Z
updated_at: 2023-08-17T13:18:08Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2142?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch and project coverage have no change.
  > Comparison is base [(`943df09`)](https://app.codecov.io/gh/simonw/datasette/commit/943df09dcca93c3b9861b8c96277a01320db8662?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.06% compared to head [(`6b617f7`)](https://app.codecov.io/gh/simonw/datasette/pull/2142?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.06%.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main    #2142   +/-   ##
  =======================================
    Coverage   92.06%   92.06%           
  =======================================
    Files          40       40           
    Lines        5937     5937           
  =======================================
    Hits         5466     5466           
    Misses        471      471           
  ```




  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2142?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683066934
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/577
node_id: IC_kwDOCGYnMM5kUZA2
created_at: 2023-08-17T22:37:18Z
updated_at: 2023-08-17T22:37:18Z
author_association: OWNER
body:
  I'm certain this could work.

  It turns out the `.transform()` method already has code that creates the new table with a copy of foreign keys from the old one - dropping any foreign keys that were specified in the `drop_foreign_keys=` parameter:

  https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/sqlite_utils/db.py#L1850-L1872

  Improving this code to support adding foreign keys as well would be pretty simple.

  And then the `.add_foreign_keys()` and `.add_foreign_key()` methods could be updated to use `.transform(...)` under the hood instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683068505
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/577
node_id: IC_kwDOCGYnMM5kUZZZ
created_at: 2023-08-17T22:39:17Z
updated_at: 2023-08-17T22:39:38Z
author_association: OWNER
body:
  This would help address these issues, among potentially many others:
  - https://github.com/simonw/llm/issues/60
  - https://github.com/simonw/llm/issues/116
  - https://github.com/simonw/llm/issues/123

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683071519
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/577
node_id: IC_kwDOCGYnMM5kUaIf
created_at: 2023-08-17T22:42:28Z
updated_at: 2023-08-17T22:42:28Z
author_association: OWNER
body:
  Looking at the whole of the `.add_foreign_keys()` method, the first section of it can remain unchanged - it's just a bunch of validation:

  https://github.com/simonw/sqlite-utils/blob/13ebcc575d2547c45e8d31288b71a3242c16b886/sqlite_utils/db.py#L1106-L1149

  At that point we have `foreign_keys_to_create` as the ones that are new, but we should instead try to build up a `foreign_keys` which is both new and old, ready to be passed to `.transform()`.

  Here's the rest of that function, which will be replaced by a called to `.transform(foreign_keys=foreign_keys)`:

  https://github.com/simonw/sqlite-utils/blob/13ebcc575d2547c45e8d31288b71a3242c16b886/sqlite_utils/db.py#L1151-L1177

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683074009
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/577
node_id: IC_kwDOCGYnMM5kUavZ
created_at: 2023-08-17T22:45:29Z
updated_at: 2023-08-17T22:47:08Z
author_association: OWNER
body:
  Actually I think `table.transform()` might get the following optional arguments:
  ```python
      def transform(
          self,
          *,
          # ...
          # This one exists already:
          drop_foreign_keys: Optional[Iterable] = None,
          # These two are new. This one specifies keys to add:
          add_foreign_keys: Optional[ForeignKeysType] = None,
          # Or this one causes them all to be replaced with the new definitions:
          foreign_keys: Optional[ForeignKeysType] = None,
  ```
  There should be validation that forbids you from using `foreign_keys=` at the same time as either `drop_foreign_keys=` or `add_foreign_keys=` because the point of `foreign_keys=` is to define the keys for the new table all in one go.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683074546
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/577
node_id: IC_kwDOCGYnMM5kUa3y
created_at: 2023-08-17T22:46:18Z
updated_at: 2023-08-17T22:46:18Z
author_association: OWNER
body:
  Maybe this:
  ```python
          drop_foreign_keys: Optional[Iterable] = None,
  ```
  Should be this:
  ```python
          drop_foreign_keys: Optional[Iterable[str]] = None,
  ```
  Because it takes a list of column names that should have their foreign keys dropped.

reactions:
[cell repeats an earlier one in this file: 112 characters]


