# issue_comments

html_url: https://github.com/simonw/sqlite-utils/pull/508#issuecomment-1297754631
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/508
node_id: IC_kwDOCGYnMM5NWioH
created_at: 2022-10-31T22:14:48Z
updated_at: 2022-10-31T22:53:59Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/508?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Base: **96.25**% // Head: **96.09**% // Decreases project coverage by **`-0.15%`** :warning:
  > Coverage data is based on head [(`2d6a149`)](https://codecov.io/gh/simonw/sqlite-utils/pull/508?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`529110e`)](https://codecov.io/gh/simonw/sqlite-utils/commit/529110e7d8c4a6b1bbf5fb61f2e29d72aa95a611?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > Patch coverage: 63.63% of modified lines in pull request are covered.

  > :exclamation: Current head 2d6a149 differs from pull request most recent head 43a8c4c. Consider uploading reports for the commit 43a8c4c to get more accurate results

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@            Coverage Diff             @@
  ##             main     #508      +/-   ##
  ==========================================
  - Coverage   96.25%   96.09%   -0.16%     
  ==========================================
    Files           4        4              
    Lines        2401     2407       +6     
  ==========================================
  + Hits         2311     2313       +2     
  - Misses         90       94       +4     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/508?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/508/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `96.79% <63.63%> (-0.30%)` | :arrow_down: |

  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  </details>

  [:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/508?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/508#issuecomment-1297788531
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/508
node_id: IC_kwDOCGYnMM5NWq5z
created_at: 2022-10-31T22:54:33Z
updated_at: 2022-11-17T15:11:16Z
author_association: CONTRIBUTOR
body:
  Maybe this is actually a problem in the python sqlite bindings. Given [SQLITE's stance on this](https://www.sqlite.org/invalidutf.html) they should probably use `encode('utf-8', 'surrogatepass')`. As far as I understand the error here won't actually be resolved by this PR as-is. We would need to modify the data with `surrogateescape`... :/ or modify the sqlite3 module to use `surrogatepass`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/507#issuecomment-1297859539
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/507
node_id: IC_kwDOCGYnMM5NW8PT
created_at: 2022-11-01T00:40:16Z
updated_at: 2022-11-01T00:40:16Z
author_association: CONTRIBUTOR
body:
  Ideally people could fix their data if they run into this issue.

  If you are using filenames try [convmv](https://linux.die.net/man/1/convmv)

  ```
  convmv --preserve-mtimes -f utf8 -t utf8 --notest -i -r .
  ```

  maybe this script will also help: 

  ```py
  import argparse, shutil
  from pathlib import Path

  import ftfy

  from xklb import utils
  from xklb.utils import log


  def parse_args() -> argparse.Namespace:
      parser = argparse.ArgumentParser()
      parser.add_argument("paths", nargs='*')
      parser.add_argument("--verbose", "-v", action="count", default=0)
      args = parser.parse_args()

      log.info(utils.dict_filter_bool(args.__dict__))
      return args


  def rename_invalid_paths() -> None:
      args = parse_args()

      for path in args.paths:
          log.info(path)
          for p in sorted([str(p) for p in Path(path).rglob("*")], key=len):
              fixed = ftfy.fix_text(p, uncurl_quotes=False).replace("\r\n", "\n").replace("\r", "\n").replace("\n", "")
              if p != fixed:
                  try:
                      shutil.move(p, fixed)
                  except FileNotFoundError:
                      log.warning("FileNotFound. %s", p)
                  else:
                      log.info(fixed)


  if __name__ == "__main__":
      rename_invalid_paths()
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1876#issuecomment-1298854321
issue_url: https://api.github.com/repos/simonw/datasette/issues/1876
node_id: IC_kwDOBm6k_c5NavGx
created_at: 2022-11-01T17:14:33Z
updated_at: 2022-11-01T17:14:33Z
author_association: OWNER
body:
  I could use a `textarea` here (would need to figure out a neat pattern to expand it to fit the query):

  <img width="426" alt="image" src="https://user-images.githubusercontent.com/9599/199295041-25abe0b9-f825-43a2-ae5a-face622e08bc.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/506#issuecomment-1298877872
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/506
node_id: IC_kwDOCGYnMM5Na02w
created_at: 2022-11-01T17:35:30Z
updated_at: 2022-11-01T17:35:30Z
author_association: OWNER
body:
  This may not make sense.

  First, `.last_rowid` is a property on table - but that doesn't make sense for `rowcount` since it should clearly be a property on the database itself (you can run a query directly using `db.execute()` without going through a `Table` object).

  So I tried this prototype:

  ```diff
  diff --git a/docs/python-api.rst b/docs/python-api.rst
  index 206e5e6..78d3a8d 100644
  --- a/docs/python-api.rst
  +++ b/docs/python-api.rst
  @@ -186,6 +186,15 @@ The ``db.query(sql)`` function executes a SQL query and returns an iterator over
       # {'name': 'Cleo'}
       # {'name': 'Pancakes'}
   
  +After executing a query the ``db.rowcount`` property on that database instance will reflect the number of rows affected by any insert, update or delete operations performed by that query:
  +
  +.. code-block:: python
  +
  +    db = Database(memory=True)
  +    db["dogs"].insert_all([{"name": "Cleo"}, {"name": "Pancakes"}])
  +    print(db.rowcount)
  +    # Outputs: 2
  +
   .. _python_api_execute:
   
   db.execute(sql, params)
  diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py
  index a06f4b7..c19c2dd 100644
  --- a/sqlite_utils/db.py
  +++ b/sqlite_utils/db.py
  @@ -294,6 +294,8 @@ class Database:
   
       _counts_table_name = "_counts"
       use_counts_table = False
  +    # Number of rows inserted, updated or deleted
  +    rowcount: Optional[int] = None
   
       def __init__(
           self,
  @@ -480,9 +482,11 @@ class Database:
           if self._tracer:
               self._tracer(sql, parameters)
           if parameters is not None:
  -            return self.conn.execute(sql, parameters)
  +            cursor = self.conn.execute(sql, parameters)
           else:
  -            return self.conn.execute(sql)
  +            cursor = self.conn.execute(sql)
  +        self.rowcount = cursor.rowcount
  +        return cursor
   
       def executescript(self, sql: str) -> sqlite3.Cursor:
           """
  ```
  But this happens:
  ```pycon
  >>> from sqlite_utils import Database
  >>> db = Database(memory=True)
  >>> db["dogs"].insert_all([{"name": "Cleo"}, {"name": "Pancakes"}])
  <Table dogs (name)>
  >>> db.rowcount
  -1
  ```
  Turning on query tracing demonstrates why:
  ```pycon
  >>> db = Database(memory=True, tracer=print)
  PRAGMA recursive_triggers=on; None
  >>> db["dogs"].insert_all([{"name": "Cleo"}, {"name": "Pancakes"}])
  select name from sqlite_master where type = 'view' None
  select name from sqlite_master where type = 'table' None
  select name from sqlite_master where type = 'view' None
  CREATE TABLE [dogs] (
     [name] TEXT
  );
           None
  select name from sqlite_master where type = 'view' None
  INSERT INTO [dogs] ([name]) VALUES (?), (?); ['Cleo', 'Pancakes']
  select name from sqlite_master where type = 'table' None
  select name from sqlite_master where type = 'table' None
  PRAGMA table_info([dogs]) None
  <Table dogs (name)>
  >>>
  ```
  The `.insert_all()` function does a bunch of other queries too, so `.rowcount` is quickly over-ridden by the same result from extra queries that it executed.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/506#issuecomment-1298879701
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/506
node_id: IC_kwDOCGYnMM5Na1TV
created_at: 2022-11-01T17:37:13Z
updated_at: 2022-11-01T17:37:13Z
author_association: OWNER
body:
  The question I was originally trying to answer here was this: how many rows were actually inserted by that call to `.insert_all()`?

  I don't know that `.rowcount` would ever be useful here, since the "correct" answer depends on other factors - had I determined to ignore or replace records with a primary key that matches an existing record for example?

  So I think if people need `rowcount` they can get it by using a `cursor` directly.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1873#issuecomment-1298885451
issue_url: https://api.github.com/repos/simonw/datasette/issues/1873
node_id: IC_kwDOBm6k_c5Na2tL
created_at: 2022-11-01T17:42:20Z
updated_at: 2022-11-01T17:42:20Z
author_association: OWNER
body:
  Design decision:
  ```json
  {
    "rows": [{"id": 1, "title": "The title"}],
    "ignore": true
  }
  ```
  Or `"replace": true`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1873#issuecomment-1298905135
issue_url: https://api.github.com/repos/simonw/datasette/issues/1873
node_id: IC_kwDOBm6k_c5Na7gv
created_at: 2022-11-01T17:59:59Z
updated_at: 2022-11-01T17:59:59Z
author_association: OWNER
body:
  It's a bit surprising that you can send `"ignore": true, "return_rows": true` and the returned `"inserted"` key will list rows that were NOT inserted (since they were ignored).

  Three options:

  1. Ignore that and document it
  2. Fix it so `"inserted"` only returns rows that were actually inserted (bit tricky)
  3. Change the name of `"inserted"` to something else

  I'm picking 3 - I'm going to change it to be called `"rows"` instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1873#issuecomment-1298919552
issue_url: https://api.github.com/repos/simonw/datasette/issues/1873
node_id: IC_kwDOBm6k_c5Na_CA
created_at: 2022-11-01T18:11:27Z
updated_at: 2022-11-01T18:11:27Z
author_association: OWNER
body:
  I forgot to document `ignore` and `replace`. Also I need to add tests that cover:

  - Forgetting to include a primary key on a non-autoincrement table
  - Compound primary keys
  - Rowid only tables with and without rowid specified

  I think my validation logic here will get caught out by the fact that `rowid` does not show up as a valid column name: https://github.com/simonw/datasette/blob/9bec7c38eb93cde5afb16df9bdd96aea2a5b0459/datasette/views/table.py#L1151-L1160

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1878#issuecomment-1299071456
issue_url: https://api.github.com/repos/simonw/datasette/issues/1878
node_id: IC_kwDOBm6k_c5NbkHg
created_at: 2022-11-01T20:02:43Z
updated_at: 2022-11-01T20:02:43Z
author_association: OWNER
body:
  Note that "update" is partially covered by the `replace` option to `/-/insert`, added here:
  - https://github.com/simonw/datasette/issues/1873#issuecomment-1298885451

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1862#issuecomment-1299073433
issue_url: https://api.github.com/repos/simonw/datasette/issues/1862
node_id: IC_kwDOBm6k_c5NbkmZ
created_at: 2022-11-01T20:04:31Z
updated_at: 2022-11-01T20:04:31Z
author_association: OWNER
body:
  It really feels like this should be accompanied by a `/db/-/create` API for creating tables. I had to add that to `sqlite-utils` eventually (initially it only supported creating by passing in an example document):

  https://sqlite-utils.datasette.io/en/stable/cli.html#cli-create-table

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1879#issuecomment-1299090678
issue_url: https://api.github.com/repos/simonw/datasette/issues/1879
node_id: IC_kwDOBm6k_c5Nboz2
created_at: 2022-11-01T20:20:28Z
updated_at: 2022-11-01T20:20:28Z
author_association: OWNER
body:
  My first step in debugging these is to install https://datasette.io/plugins/datasette-debug-asgi - but now I'm thinking maybe something like that should be part of core.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1879#issuecomment-1299096850
issue_url: https://api.github.com/repos/simonw/datasette/issues/1879
node_id: IC_kwDOBm6k_c5NbqUS
created_at: 2022-11-01T20:26:12Z
updated_at: 2022-11-01T20:26:12Z
author_association: OWNER
body:
  The other relevant plugin here is https://datasette.io/plugins/datasette-x-forwarded-host

  Maybe that should be rolled into core too?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1879#issuecomment-1299098458
issue_url: https://api.github.com/repos/simonw/datasette/issues/1879
node_id: IC_kwDOBm6k_c5Nbqta
created_at: 2022-11-01T20:27:40Z
updated_at: 2022-11-01T20:33:52Z
author_association: OWNER
body:
  https://github.com/simonw/datasette-x-forwarded-host/blob/main/datasette_x_forwarded_host/__init__.py could happen in core controlled by:

  `--setting trust_forwarded_host 1`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1879#issuecomment-1299102108
issue_url: https://api.github.com/repos/simonw/datasette/issues/1879
node_id: IC_kwDOBm6k_c5Nbrmc
created_at: 2022-11-01T20:30:54Z
updated_at: 2022-11-01T20:33:06Z
author_association: OWNER
body:
  One idea: add a `/-/debug` page (or `/-/tips` or `/-/checks`) which shows the incoming requests headers and could even detect if there's an `x-forwarded-host` header that isn't being repeated and show a tip on how to fix that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1879#issuecomment-1299102755
issue_url: https://api.github.com/repos/simonw/datasette/issues/1879
node_id: IC_kwDOBm6k_c5Nbrwj
created_at: 2022-11-01T20:31:37Z
updated_at: 2022-11-01T20:31:37Z
author_association: OWNER
body:
  And some JavaScript that can spot if Datasette thinks it is being served over HTTP when it's actually being served over HTTPS.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1871#issuecomment-1299349741
issue_url: https://api.github.com/repos/simonw/datasette/issues/1871
node_id: IC_kwDOBm6k_c5NcoDt
created_at: 2022-11-01T23:22:55Z
updated_at: 2022-11-01T23:22:55Z
author_association: OWNER
body:
  It's weird that the API explorer only lets you explore POST APIs. It should probably also let you explore GET APIs, or be renamed.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1871#issuecomment-1299388341
issue_url: https://api.github.com/repos/simonw/datasette/issues/1871
node_id: IC_kwDOBm6k_c5Ncxe1
created_at: 2022-11-02T00:24:28Z
updated_at: 2022-11-02T00:25:00Z
author_association: OWNER
body:
  I want JSON syntax highlighting.

  https://github.com/luyilin/json-format-highlight is an MIT licensed tiny highlighter that looks decent for this.

  https://unpkg.com/json-format-highlight@1.0.1/dist/json-format-highlight.js

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1871#issuecomment-1299597066
issue_url: https://api.github.com/repos/simonw/datasette/issues/1871
node_id: IC_kwDOBm6k_c5NdkcK
created_at: 2022-11-02T05:32:22Z
updated_at: 2022-11-02T05:32:22Z
author_association: OWNER
body:
  Demo of the latest API explorer:

  ![explorer](https://user-images.githubusercontent.com/9599/199406184-1292df42-25ea-4daf-8b54-ca26170ec1ea.gif)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1871#issuecomment-1299598570
issue_url: https://api.github.com/repos/simonw/datasette/issues/1871
node_id: IC_kwDOBm6k_c5Ndkzq
created_at: 2022-11-02T05:34:28Z
updated_at: 2022-11-02T05:34:28Z
author_association: OWNER
body:
  This is pretty useful now. Two features I still want to add:

  - The ability to link to the API explorer such that the form is pre-filled with material from the URL. Need to guard against clickjacking first though, so no-one can link to it in an invisible iframe and trick the user into hitting POST.
  - Some kind of list of endpoints so people can click links to start using the API explorer. A list of every table the user can write to with each of their `/db/table/-/insert` endpoints for example.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1871#issuecomment-1299599461
issue_url: https://api.github.com/repos/simonw/datasette/issues/1871
node_id: IC_kwDOBm6k_c5NdlBl
created_at: 2022-11-02T05:35:36Z
updated_at: 2022-11-02T05:36:15Z
author_association: OWNER
body:
  Here's a slightly wild idea: what if there was a button on `/-/api` that you could click to turn on "API explorer mode" for the rest of the Datasette interface - which sets a cookie, and that cookie means you then see "API explorer" links in all sorts of other relevant places in the Datasette UI (maybe tucked away in cog menus).

  Only reason I don't want to show these to everyone is that I don't think this is a very user-friendly feature: if you don't know what an API is I don't want to expose you to it unnecessarily.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1871#issuecomment-1299600257
issue_url: https://api.github.com/repos/simonw/datasette/issues/1871
node_id: IC_kwDOBm6k_c5NdlOB
created_at: 2022-11-02T05:36:40Z
updated_at: 2022-11-02T05:36:40Z
author_association: OWNER
body:
  The API Explorer should definitely link to the `/-/create-token` page for users who have permission though.

  And it should probably go in the Datasette application menu?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1871#issuecomment-1299607082
issue_url: https://api.github.com/repos/simonw/datasette/issues/1871
node_id: IC_kwDOBm6k_c5Ndm4q
created_at: 2022-11-02T05:45:31Z
updated_at: 2022-11-02T05:45:31Z
author_association: OWNER
body:
  I'm going to add a link to the Datasette API docs for the current running version of Datasette, e.g. to https://docs.datasette.io/en/0.63/json_api.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1880#issuecomment-1301043042
issue_url: https://api.github.com/repos/simonw/datasette/issues/1880
node_id: IC_kwDOBm6k_c5NjFdi
created_at: 2022-11-02T18:20:14Z
updated_at: 2022-11-02T18:20:14Z
author_association: NONE
body:
  Follow on question - is all memory use @simonw - for both datasette and SQLlite confined to the "query time" itself i.e. the memory use is relevant only to a particular transaction or query - and then subsequently released?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1855#issuecomment-1301594495
issue_url: https://api.github.com/repos/simonw/datasette/issues/1855
node_id: IC_kwDOBm6k_c5NlMF_
created_at: 2022-11-03T03:11:17Z
updated_at: 2022-11-03T03:11:17Z
author_association: OWNER
body:
  Maybe the way to do this is through a new standard mechanism on the actor: a set of additional restrictions, e.g.:

  ```
  {
    "id": "root",
    "_r": {
      "a": ["ir", "ur", "dr"],
      "d": {
          "fixtures": ["ir", "ur", "dr"]
      },
      "t": {
          "fixtures": {
              "searchable": ["ir"]
          }
      }
  }
  ```
  `"a"` is "all permissions" - these apply to everything.
  `"d"` permissions only apply to the specified database
  `"t"` permissions only apply to the specified table

  The way this works is there's a default [permission_allowed(datasette, actor, action, resource)](https://docs.datasette.io/en/stable/plugin_hooks.html#id25) hook which only consults these, and crucially just says NO if those rules do not match.

  In this way it would apply as an extra layer of permission rules over the defaults (which for this `root` instance would all return yes).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1881#issuecomment-1301635340
issue_url: https://api.github.com/repos/simonw/datasette/issues/1881
node_id: IC_kwDOBm6k_c5NlWEM
created_at: 2022-11-03T04:46:41Z
updated_at: 2022-11-03T04:46:41Z
author_association: OWNER
body:
  Built this prototype:

  ![prototype](https://user-images.githubusercontent.com/9599/199649219-f146e43b-bfb5-45e6-9777-956f21a79887.gif)

  In building it I realized I needed to know which permissions took a table, a database, both or neither. So I had to bake that into the code.

  Here's the prototype so far (which includes a prototype of the logic for the `_r` field on actor, see #1855):

  ```diff
  diff --git a/datasette/default_permissions.py b/datasette/default_permissions.py
  index 32b0c758..f68aa38f 100644
  --- a/datasette/default_permissions.py
  +++ b/datasette/default_permissions.py
  @@ -6,8 +6,8 @@ import json
   import time
   
   
  -@hookimpl(tryfirst=True)
  -def permission_allowed(datasette, actor, action, resource):
  +@hookimpl(tryfirst=True, specname="permission_allowed")
  +def permission_allowed_default(datasette, actor, action, resource):
       async def inner():
           if action in (
               "permissions-debug",
  @@ -57,6 +57,44 @@ def permission_allowed(datasette, actor, action, resource):
       return inner
   
   
  +@hookimpl(specname="permission_allowed")
  +def permission_allowed_actor_restrictions(actor, action, resource):
  +    if actor is None:
  +        return None
  +    _r = actor.get("_r")
  +    if not _r:
  +        # No restrictions, so we have no opinion
  +        return None
  +    action_initials = "".join([word[0] for word in action.split("-")])
  +    # If _r is defined then we use those to further restrict the actor
  +    # Crucially, we only use this to say NO (return False) - we never
  +    # use it to return YES (True) because that might over-ride other
  +    # restrictions placed on this actor
  +    all_allowed = _r.get("a")
  +    if all_allowed is not None:
  +        assert isinstance(all_allowed, list)
  +        if action_initials in all_allowed:
  +            return None
  +    # How about for the current database?
  +    if action in ("view-database", "view-database-download", "execute-sql"):
  +        database_allowed = _r.get("d", {}).get(resource)
  +        if database_allowed is not None:
  +            assert isinstance(database_allowed, list)
  +            if action_initials in database_allowed:
  +                return None
  +    # Or the current table? That's any time the resource is (database, table)
  +    if not isinstance(resource, str) and len(resource) == 2:
  +        database, table = resource
  +        table_allowed = _r.get("t", {}).get(database, {}).get(table)
  +        # TODO: What should this do for canned queries?
  +        if table_allowed is not None:
  +            assert isinstance(table_allowed, list)
  +            if action_initials in table_allowed:
  +                return None
  +    # This action is not specifically allowed, so reject it
  +    return False
  +
  +
   @hookimpl
   def actor_from_request(datasette, request):
       prefix = "dstok_"
  diff --git a/datasette/templates/allow_debug.html b/datasette/templates/allow_debug.html
  index 0f1b30f0..ae43f0f5 100644
  --- a/datasette/templates/allow_debug.html
  +++ b/datasette/templates/allow_debug.html
  @@ -35,7 +35,7 @@ p.message-warning {
   
   <p>Use this tool to try out different actor and allow combinations. See <a href="https://docs.datasette.io/en/stable/authentication.html#defining-permissions-with-allow-blocks">Defining permissions with "allow" blocks</a> for documentation.</p>
   
  -<form action="{{ urls.path('-/allow-debug') }}" method="get">
  +<form action="{{ urls.path('-/allow-debug') }}" method="get" style="margin-bottom: 1em">
       <div class="two-col">
           <p><label>Allow block</label></p>
           <textarea name="allow">{{ allow_input }}</textarea>
  @@ -55,4 +55,82 @@ p.message-warning {
   
   {% if result == "False" %}<p class="message-error">Result: deny</p>{% endif %}
   
  +<h2>Test permission check</h2>
  +
  +<p>This tool lets you simulate an actor and a permission check for that actor.</p>
  +
  +<form action="{{ urls.path('-/allow-debug') }}" id="debug-post" method="post" style="margin-bottom: 1em">
  +    <input type="hidden" name="csrftoken" value="{{ csrftoken() }}">
  +    <div class="two-col">
  +        <p><label>Actor</label></p>
  +        <textarea name="actor">{% if actor_input %}{{ actor_input }}{% else %}{"id": "root"}{% endif %}</textarea>
  +    </div>
  +    <div class="two-col" style="vertical-align: top">
  +        <p><label>Permission check</label></p>
  +        <p><label for="permission" style="display:block">Permission</label>
  +        <select name="permission" id="permission">
  +            {% for permission in [
  +                "view-instance",
  +                "view-database",
  +                "view-database-download",
  +                "view-table",
  +                "view-query",
  +                "insert-row",
  +                "delete-row",
  +                "drop-table",
  +                "execute-sql",
  +                "permissions-debug",
  +                "debug-menu"] %}
  +            <option value="{{ permission }}">{{ permission }}</option>
  +            {% endfor %}
  +        </select>
  +        <p><label for="resource_1">Database name</label><input type="text" id="resource_1" name="resource_1"></p>
  +        <p><label for="resource_2">Table or query name</label><input type="text" id="resource_2" name="resource_2"></p>
  +    </div>
  +    <div style="margin-top: 1em;">
  +        <input type="submit" value="Simulate permission check">
  +    </div>
  +</form>
  +
  +<script>
  +var rawPerms = {{ permissions|tojson }};
  +var permissions = Object.fromEntries(rawPerms.map(([label, abbr, needs_resource_1, needs_resource_2, def]) => [label, {needs_resource_1, needs_resource_2, def}]))
  +var permissionSelect = document.getElementById('permission');
  +var resource1 = document.getElementById('resource_1');
  +var resource2 = document.getElementById('resource_2');
  +function updateResourceVisibility() {
  +    var permission = permissionSelect.value;
  +    var {needs_resource_1, needs_resource_2} = permissions[permission];
  +    if (needs_resource_1) {
  +        resource1.closest('p').style.display = 'block';
  +    } else {
  +        resource1.closest('p').style.display = 'none';
  +    }
  +    if (needs_resource_2) {
  +        resource2.closest('p').style.display = 'block';
  +    } else {
  +        resource2.closest('p').style.display = 'none';
  +    }
  +}
  +permissionSelect.addEventListener('change', updateResourceVisibility);
  +updateResourceVisibility();
  +
  +// When #debug-post form is submitted, use fetch() to POST data
  +var debugPost = document.getElementById('debug-post');
  +debugPost.addEventListener('submit', function(ev) {
  +    ev.preventDefault();
  +    var formData = new FormData(debugPost);
  +    console.log(formData);
  +    fetch(debugPost.action, {
  +        method: 'POST',
  +        body: new URLSearchParams(formData),
  +    }).then(function(response) {
  +        return response.json();
  +    }).then(function(data) {
  +        alert(JSON.stringify(data, null, 4));
  +    });
  +});
  +</script>
  +    
  +
   {% endblock %}
  diff --git a/datasette/views/special.py b/datasette/views/special.py
  index 9922a621..d46fc280 100644
  --- a/datasette/views/special.py
  +++ b/datasette/views/special.py
  @@ -1,6 +1,8 @@
   import json
  +from datasette.permissions import PERMISSIONS
   from datasette.utils.asgi import Response, Forbidden
   from datasette.utils import actor_matches_allow, add_cors_headers
  +from datasette.permissions import PERMISSIONS
   from .base import BaseView
   import secrets
   import time
  @@ -138,9 +140,34 @@ class AllowDebugView(BaseView):
                   "error": "\n\n".join(errors) if errors else "",
                   "actor_input": actor_input,
                   "allow_input": allow_input,
  +                "permissions": PERMISSIONS,
               },
           )
   
  +    async def post(self, request):
  +        vars = await request.post_vars()
  +        actor = json.loads(vars["actor"])
  +        permission = vars["permission"]
  +        resource_1 = vars["resource_1"]
  +        resource_2 = vars["resource_2"]
  +        resource = []
  +        if resource_1:
  +            resource.append(resource_1)
  +        if resource_2:
  +            resource.append(resource_2)
  +        resource = tuple(resource)
  +        result = await self.ds.permission_allowed(
  +            actor, permission, resource, default="USE_DEFAULT"
  +        )
  +        return Response.json(
  +            {
  +                "actor": actor,
  +                "permission": permission,
  +                "resource": resource,
  +                "result": result,
  +            }
  +        )
  +
   
   class MessagesDebugView(BaseView):
       name = "messages_debug"
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1881#issuecomment-1301635906
issue_url: https://api.github.com/repos/simonw/datasette/issues/1881
node_id: IC_kwDOBm6k_c5NlWNC
created_at: 2022-11-03T04:48:09Z
updated_at: 2022-11-03T04:48:09Z
author_association: OWNER
body:
  I built this prototype on the http://127.0.0.1:8001/-/allow-debug page, which is open to anyone to visit.

  But... I just realized that using this tool can leak information - you can use it to guess the names of invisible databases and tables and run theoretical permission checks against them.

  Using the tool also pollutes the list of permission checks that show up on the root-anlo `/-/permissions` page.

  So.... I'm going to restrict the usage of this tool to users with access to `/-/permissions` and put it on that page instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1881#issuecomment-1301638156
issue_url: https://api.github.com/repos/simonw/datasette/issues/1881
node_id: IC_kwDOBm6k_c5NlWwM
created_at: 2022-11-03T04:54:00Z
updated_at: 2022-11-03T04:54:00Z
author_association: OWNER
body:
  If I have the permissions defined like this:
  ```python
  PERMISSIONS = (
      Permission("view-instance", "vi", False, False, True),
      Permission("view-database", "vd", True, False, True),
      Permission("view-database-download", "vdd", True, False, True),
      Permission("view-table", "vt", True, True, True),
      Permission("view-query", "vq", True, True, True),
      Permission("insert-row", "ir", True, True, False),
      Permission("delete-row", "dr", True, True, False),
      Permission("drop-table", "dt", True, True, False),
      Permission("execute-sql", "es", True, False, True),
      Permission("permissions-debug", "pd", False, False, False),
      Permission("debug-menu", "dm", False, False, False),
  )
  ```
  Instead of just calling them by their undeclared names in places like this:
  ```python
  await self.ds.permission_allowed(
      request.actor, "execute-sql", database, default=True
  )
  ```
  On the one hand I can ditch that confusing `default=True` option - whether a permission is on by default becomes a characteristic of that `Permission()` itself, which feels much neater.

  On the other hand though, plugins that introduce their own permissions - like https://datasette.io/plugins/datasette-edit-schema - will need a way to register those permissions with Datasette core. Probably another plugin hook.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1881#issuecomment-1301638918
issue_url: https://api.github.com/repos/simonw/datasette/issues/1881
node_id: IC_kwDOBm6k_c5NlW8G
created_at: 2022-11-03T04:56:06Z
updated_at: 2022-11-03T04:56:06Z
author_association: OWNER
body:
  I've also introduced a new concept of a permission abbreviation, which like the permission name needs to be globally unique.

  That's a problem for plugins - they might just be able to guarantee that their permission long-form name is unique among other plugins (through sensible naming conventions) but the thing where they declare a initial-letters-only abbreviation is far more risky.

  I think abbreviations are optional - they are provided for core permissions but plugins are advised not to use them.

  Also Datasette could check that the installed plugins do not provide conflicting permissions on startup and refuse to start if they do.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1881#issuecomment-1301639370
issue_url: https://api.github.com/repos/simonw/datasette/issues/1881
node_id: IC_kwDOBm6k_c5NlXDK
created_at: 2022-11-03T04:57:21Z
updated_at: 2022-11-03T04:57:21Z
author_association: OWNER
body:
  The plugin hook would be called `register_permissions()`, for consistency with `register_routes()` and `register_commands()`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1881#issuecomment-1301639741
issue_url: https://api.github.com/repos/simonw/datasette/issues/1881
node_id: IC_kwDOBm6k_c5NlXI9
created_at: 2022-11-03T04:58:21Z
updated_at: 2022-11-03T04:58:21Z
author_association: OWNER
body:
  The whole `database_name` or `(database_name, table_name)` tuple for resource is a bit of a code smell. Maybe this is a chance to tidy that up too?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1881#issuecomment-1301645921
issue_url: https://api.github.com/repos/simonw/datasette/issues/1881
node_id: IC_kwDOBm6k_c5NlYph
created_at: 2022-11-03T05:10:05Z
updated_at: 2022-12-09T01:38:21Z
author_association: OWNER
body:
  I'd love to come up with a good short name for the second part of the resource two-tuple, the thing which is usually the name of a table but could also be the name of a SQL view or the name of a canned query.

  Idea 8th December: why not call it resource? A resource could be a thing that lives inside a database.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1855#issuecomment-1301646493
issue_url: https://api.github.com/repos/simonw/datasette/issues/1855
node_id: IC_kwDOBm6k_c5NlYyd
created_at: 2022-11-03T05:11:06Z
updated_at: 2022-11-03T05:11:06Z
author_association: OWNER
body:
  Built a prototype of the above:

  ```diff
  diff --git a/datasette/default_permissions.py b/datasette/default_permissions.py
  index 32b0c758..f68aa38f 100644
  --- a/datasette/default_permissions.py
  +++ b/datasette/default_permissions.py
  @@ -6,8 +6,8 @@ import json
   import time
   
   
  -@hookimpl(tryfirst=True)
  -def permission_allowed(datasette, actor, action, resource):
  +@hookimpl(tryfirst=True, specname="permission_allowed")
  +def permission_allowed_default(datasette, actor, action, resource):
       async def inner():
           if action in (
               "permissions-debug",
  @@ -57,6 +57,44 @@ def permission_allowed(datasette, actor, action, resource):
       return inner
   
   
  +@hookimpl(specname="permission_allowed")
  +def permission_allowed_actor_restrictions(actor, action, resource):
  +    if actor is None:
  +        return None
  +    _r = actor.get("_r")
  +    if not _r:
  +        # No restrictions, so we have no opinion
  +        return None
  +    action_initials = "".join([word[0] for word in action.split("-")])
  +    # If _r is defined then we use those to further restrict the actor
  +    # Crucially, we only use this to say NO (return False) - we never
  +    # use it to return YES (True) because that might over-ride other
  +    # restrictions placed on this actor
  +    all_allowed = _r.get("a")
  +    if all_allowed is not None:
  +        assert isinstance(all_allowed, list)
  +        if action_initials in all_allowed:
  +            return None
  +    # How about for the current database?
  +    if action in ("view-database", "view-database-download", "execute-sql"):
  +        database_allowed = _r.get("d", {}).get(resource)
  +        if database_allowed is not None:
  +            assert isinstance(database_allowed, list)
  +            if action_initials in database_allowed:
  +                return None
  +    # Or the current table? That's any time the resource is (database, table)
  +    if not isinstance(resource, str) and len(resource) == 2:
  +        database, table = resource
  +        table_allowed = _r.get("t", {}).get(database, {}).get(table)
  +        # TODO: What should this do for canned queries?
  +        if table_allowed is not None:
  +            assert isinstance(table_allowed, list)
  +            if action_initials in table_allowed:
  +                return None
  +    # This action is not specifically allowed, so reject it
  +    return False
  +
  +
   @hookimpl
   def actor_from_request(datasette, request):
       prefix = "dstok_"

  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1843#issuecomment-1302574330
issue_url: https://api.github.com/repos/simonw/datasette/issues/1843
node_id: IC_kwDOBm6k_c5No7T6
created_at: 2022-11-03T19:30:22Z
updated_at: 2022-11-03T19:30:22Z
author_association: OWNER
body:
  This is affecting me a lot at the moment, on my laptop (runs fine in CI).

  Here's a change to `conftest.py` which highlights the problem - it cause a failure the moment there are more than 5 open files according to `psutil`:

  ```diff
  diff --git a/tests/conftest.py b/tests/conftest.py
  index f4638a14..21d433c1 100644
  --- a/tests/conftest.py
  +++ b/tests/conftest.py
  @@ -1,6 +1,7 @@
   import httpx
   import os
   import pathlib
  +import psutil
   import pytest
   import re
   import subprocess
  @@ -192,3 +193,8 @@ def ds_unix_domain_socket_server(tmp_path_factory):
       yield ds_proc, uds
       # Shut it down at the end of the pytest session
       ds_proc.terminate()
  +
  +
  +def pytest_runtest_teardown(item: pytest.Item) -> None:
  +    open_files = psutil.Process().open_files()
  +    assert len(open_files) < 5
  ```
  The first error I get from this with `pytest --pdb -x` is here:

  ```
  tests/test_api.py ............E
  >>>>> traceback >>>>>

  item = <Function test_sql_time_limit>

      def pytest_runtest_teardown(item: pytest.Item) -> None:
          open_files = psutil.Process().open_files()
  >       assert len(open_files) < 5
  E       AssertionError: assert 5 < 5
  E        +  where 5 = len([popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfglrt4p2/fixtures.db', fd=14), popenfile(... fd=19), popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmphdi5b250/fixtures.dot.db', fd=20)])

  /Users/simon/Dropbox/Development/datasette/tests/conftest.py:200: AssertionError
  >>>>> entering PDB >>>>>

  >>>>> PDB post_mortem (IO-capturing turned off) >>>>>
  > /Users/simon/Dropbox/Development/datasette/tests/conftest.py(200)pytest_runtest_teardown()
  -> assert len(open_files) < 5
  ```
  That's this test:

  https://github.com/simonw/datasette/blob/2ec5583629005b32cb0877786f9681c5d43ca33f/tests/test_api.py#L656-L673

  Which uses this fixture:

  https://github.com/simonw/datasette/blob/2ec5583629005b32cb0877786f9681c5d43ca33f/tests/fixtures.py#L228-L231

  Which calls this function:

  https://github.com/simonw/datasette/blob/2ec5583629005b32cb0877786f9681c5d43ca33f/tests/fixtures.py#L105-L122

  So now I'm suspicious that, even though the fixture is meant to be session scoped, the way I'm using `with tempfile.TemporaryDirectory() as tmpdir:` is causing a whole load of files to be created and held open which are not later closed.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1843#issuecomment-1302634332
issue_url: https://api.github.com/repos/simonw/datasette/issues/1843
node_id: IC_kwDOBm6k_c5NpJ9c
created_at: 2022-11-03T20:34:56Z
updated_at: 2022-11-03T20:34:56Z
author_association: OWNER
body:
  Confirmed that calling `conn.close()` on each SQLite file-based connection is the way to fix this problem.

  I'm adding a `db.close()` method (sync, not async - I tried async first but it was really hard to cause every thread in the pool to close its threadlocal database connection) which loops through all known open file-based connections and closes them.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1843#issuecomment-1302678384
issue_url: https://api.github.com/repos/simonw/datasette/issues/1843
node_id: IC_kwDOBm6k_c5NpUtw
created_at: 2022-11-03T21:21:59Z
updated_at: 2022-11-03T21:21:59Z
author_association: OWNER
body:
  I added extra debug info to `/-/threads` to see this for myself:

  ```diff
  diff --git a/datasette/app.py b/datasette/app.py
  index 02bd38f1..16579e28 100644
  --- a/datasette/app.py
  +++ b/datasette/app.py
  @@ -969,6 +969,13 @@ class Datasette:
               "threads": [
                   {"name": t.name, "ident": t.ident, "daemon": t.daemon} for t in threads
               ],
  +            "file_connections": {
  +                db.name: [
  +                    [dict(r) for r in conn.execute("pragma database_list").fetchall()]
  +                    for conn in db._all_file_connections
  +                ]
  +                for db in self.databases.values()
  +            },
           }
           # Only available in Python 3.7+
           if hasattr(asyncio, "all_tasks"):
  ```
  Output after hitting refresh on a few `/fixtures` tables to ensure more threads started:

  ```
      "file_connections": {
          "_internal": [],
          "fixtures": [
              [
                  {
                      "seq": 0,
                      "name": "main",
                      "file": "/Users/simon/Dropbox/Development/datasette/fixtures.db"
                  }
              ],
              [
                  {
                      "seq": 0,
                      "name": "main",
                      "file": "/Users/simon/Dropbox/Development/datasette/fixtures.db"
                  }
              ],
              [
                  {
                      "seq": 0,
                      "name": "main",
                      "file": "/Users/simon/Dropbox/Development/datasette/fixtures.db"
                  }
              ]
          ]
      },
  ```
  I decided not to ship this feature though as it leaks the names of internal database files.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1882#issuecomment-1302715662
issue_url: https://api.github.com/repos/simonw/datasette/issues/1882
node_id: IC_kwDOBm6k_c5Npd0O
created_at: 2022-11-03T21:50:27Z
updated_at: 2022-11-03T21:50:27Z
author_association: OWNER
body:
  API design for this:
  ```
  POST /db/-/create
  Authorization: Bearer xxx
  Content-Type: application/json
  {
      "table": {
          "name": "my new table",
          "columns": [
              {
                  "name": "id",
                  "type": "integer"
              },
              {
                  "name": "title",
                  "type": "text"
              }
          ]
         "pk": "id"
      }
  }
  ```
  Supported column types are:

  - `integer`
  - `text`
  - `float` (even though SQLite calls it a "real")
  - `blob`

  This matches my design for `sqlite-utils`: https://sqlite-utils.datasette.io/en/stable/cli.html#cli-create-table

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1882#issuecomment-1302716350
issue_url: https://api.github.com/repos/simonw/datasette/issues/1882
node_id: IC_kwDOBm6k_c5Npd--
created_at: 2022-11-03T21:51:14Z
updated_at: 2022-11-03T22:35:54Z
author_association: OWNER
body:
  Validating this JSON object is getting a tiny bit complex. I'm tempted to adopt https://pydantic-docs.helpmanual.io/ at this point.

  The `create_model` example on https://stackoverflow.com/questions/66168517/generate-dynamic-model-using-pydantic/66168682#66168682 is particularly relevant, especially when I work on this issue:

  - #1863

  ```python
  from pydantic import create_model

  d = {"strategy": {"name": "test_strat2", "periods": 10}}

  Strategy = create_model("Strategy", **d["strategy"])

  print(Strategy.schema_json(indent=2))
  ```
  `create_model()`: https://pydantic-docs.helpmanual.io/usage/models/#dynamic-model-creation

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1882#issuecomment-1302721916
issue_url: https://api.github.com/repos/simonw/datasette/issues/1882
node_id: IC_kwDOBm6k_c5NpfV8
created_at: 2022-11-03T21:58:50Z
updated_at: 2022-11-03T21:59:17Z
author_association: OWNER
body:
  Mocked up a quick HTML+JavaScript form for creating that JSON structure using some iteration against Copilot prompts:
  ```html
  <pre>
  /* JSON format:
  {
    "table": {
        "name": "my new table",
        "columns": [
            {
                "name": "id",
                "type": "integer"
            },
            {
                "name": "title",
                "type": "text"
            }
        ]
       "pk": "id"
    }
  }

  HTML form with Javascript for creating this JSON:
  */</pre>
  <form id="create-table-form">
    <label for="table-name">Table name</label>
    <input type="text" id="table-name" name="table-name" required><br>
    <label for="table-pk">Primary key</label>
    <input type="text" id="table-pk" name="table-pk" required><br>
    <label for="column-name">Column name</label>
    <input type="text" id="column-name" name="column-name" required>
    <label for="column-type">Column type</label>
    <input type="text" id="column-type" name="column-type" required>
    <button type="button" id="add-column">Add column</button>
    <p>Current columns:</p>
    <ul id="columns"></ul>
    <button type="button" id="create-table">Create table</button>
  </form>
  <script>
  var form = document.getElementById('create-table-form');
  var tableName = document.getElementById('table-name');
  var tablePk = document.getElementById('table-pk');
  var columnName = document.getElementById('column-name');
  var columnType = document.getElementById('column-type');
  var addColumn = document.getElementById('add-column');
  var createTable = document.getElementById('create-table');
  var columns = [];
  addColumn.addEventListener('click', () => {
    columns.push({
      name: columnName.value,
      type: columnType.value
    });
    var li = document.createElement('li');
    li.textContent = columnName.value + ' (' + columnType.value + ')';
    // Add a delete button to each column
    var deleteButton = document.createElement('button');
    deleteButton.textContent = 'Delete';
    deleteButton.addEventListener('click', () => {
      columns.splice(columns.indexOf(li), 1);
      li.remove();
    });
    li.appendChild(deleteButton);
    document.getElementById('columns').appendChild(li);
  });
  createTable.addEventListener('click', () => {
    var table = {
      name: tableName.value,
      pk: tablePk.value,
      columns: columns
    };
    console.log(JSON.stringify(table, null, 2));
  });
  </script>
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1863#issuecomment-1302759174
issue_url: https://api.github.com/repos/simonw/datasette/issues/1863
node_id: IC_kwDOBm6k_c5NpocG
created_at: 2022-11-03T22:40:47Z
updated_at: 2022-11-03T22:40:47Z
author_association: OWNER
body:
  I'm considering Pydantic for this, see:
  - https://github.com/simonw/datasette/issues/1882#issuecomment-1302716350

  In particular the `create_model()` method: https://pydantic-docs.helpmanual.io/usage/models/#dynamic-model-creation

  This would give me good validation. It would also, weirdly, give me the ability to output JSON schema. Maybe I could have this as the JSON schema for a row?

  `/db/table/-/json-schema`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1863#issuecomment-1302760382
issue_url: https://api.github.com/repos/simonw/datasette/issues/1863
node_id: IC_kwDOBm6k_c5Npou-
created_at: 2022-11-03T22:42:47Z
updated_at: 2022-11-03T22:42:47Z
author_association: OWNER
body:
  ```python
  print(create_model('document', id=(int, ...), title=(str, None)).schema_json(indent=2))
  ```
  ```json
  {
    "title": "document",
    "type": "object",
    "properties": {
      "id": {
        "title": "Id",
        "type": "integer"
      },
      "title": {
        "title": "Title",
        "type": "string"
      }
    },
    "required": [
      "id"
    ]
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1863#issuecomment-1302760549
issue_url: https://api.github.com/repos/simonw/datasette/issues/1863
node_id: IC_kwDOBm6k_c5Npoxl
created_at: 2022-11-03T22:43:04Z
updated_at: 2022-11-03T23:21:31Z
author_association: OWNER
body:
  The `id=(int, ...)` thing is weird, but is apparently Pydantic syntax for a required field?

  https://cs.github.com/starlite-api/starlite/blob/28ddc847c4cb072f0d5d21a9ecd5259711f12ec9/docs/usage/11-data-transfer-objects.md#L161 confirms:

  > 1. For required fields use a tuple of type + ellipsis, for example `(str, ...)`.
  > 2. For optional fields use a tuple of type + `None`, for example `(str, None)`
  > 3. To set a default value use a tuple of type + default value, for example `(str, "Hello World")`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1863#issuecomment-1302785086
issue_url: https://api.github.com/repos/simonw/datasette/issues/1863
node_id: IC_kwDOBm6k_c5Npuw-
created_at: 2022-11-03T23:24:33Z
updated_at: 2022-11-03T23:24:56Z
author_association: OWNER
body:
  Thinking more about validation: I'm considering if this should validate that columns which are defined as SQLite foreign keys are being updated to values that exist in those other tables.

  I like the sound of this. It seems like a sensible default behaviour for Datasette. And it fits with the fact that Datasette treats foreign keys specially elsewhere in the interface.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1863#issuecomment-1302790013
issue_url: https://api.github.com/repos/simonw/datasette/issues/1863
node_id: IC_kwDOBm6k_c5Npv99
created_at: 2022-11-03T23:32:30Z
updated_at: 2022-11-03T23:32:30Z
author_association: OWNER
body:
  I'm not going to allow updates to primary keys. If you need to do that, you can instead delete the record and then insert a new one with the new primary keys you wanted - or maybe use a custom SQL query.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1881#issuecomment-1302813449
issue_url: https://api.github.com/repos/simonw/datasette/issues/1881
node_id: IC_kwDOBm6k_c5Np1sJ
created_at: 2022-11-04T00:14:07Z
updated_at: 2022-11-04T00:14:07Z
author_association: OWNER
body:
  Tool is now live here: https://latest-1-0-dev.datasette.io/-/permissions

  Needs root perms, so access this first: https://latest-1-0-dev.datasette.io/login-as-root

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1871#issuecomment-1302814693
issue_url: https://api.github.com/repos/simonw/datasette/issues/1871
node_id: IC_kwDOBm6k_c5Np1_l
created_at: 2022-11-04T00:16:36Z
updated_at: 2022-11-04T00:16:36Z
author_association: OWNER
body:
  I can close this issue once I fix it so it no longer hard-codes a potentially invalid example endpoint:

  https://github.com/simonw/datasette/blob/bcc781f4c50a8870e3389c4e60acb625c34b0317/datasette/templates/api_explorer.html#L24-L26

  https://github.com/simonw/datasette/blob/bcc781f4c50a8870e3389c4e60acb625c34b0317/datasette/templates/api_explorer.html#L34-L35

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1871#issuecomment-1302815105
issue_url: https://api.github.com/repos/simonw/datasette/issues/1871
node_id: IC_kwDOBm6k_c5Np2GB
created_at: 2022-11-04T00:17:23Z
updated_at: 2022-11-04T00:17:23Z
author_association: OWNER
body:
  I'll probably enhance it a bit more though, I want to provide a UI that lists all the tables you can explore and lets you click to pre-fill the forms with them.

  Though at that point what should I do about the other endpoints? Probably list those too. Gets a bit complex, especially with the row-level update and delete endpoints.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1862#issuecomment-1302817500
issue_url: https://api.github.com/repos/simonw/datasette/issues/1862
node_id: IC_kwDOBm6k_c5Np2rc
created_at: 2022-11-04T00:22:31Z
updated_at: 2022-11-04T00:22:31Z
author_association: OWNER
body:
  Maybe this is a feature added to the existing `/db/table/-/insert` endpoint?

  Bit weird that you can call that endpoint for a table that doesn't exist yet, but it fits the `sqlite-utils` way of creating tables which I've found very pleasant over the past few years.

  So perhaps the API looks like this:

  ```
  POST /<database>/<table>/-/insert
  Content-Type: application/json
  Authorization: Bearer dstok_<rest-of-token>
  {
      "create_table": true,
      "rows": [
          {
              "column1": "value1",
              "column2": "value2"
          },
          {
              "column1": "value3",
              "column2": "value4"
          }
      ]
  }
  ```
  The `create_table` option will cause the table to be created if it doesn't already exist.

  That means I probably also need a `"pk": "..."` column for setting a primary key if the table is being created ... and maybe other options that I invent for this other feature too?
  - #1882

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1862#issuecomment-1302817807
issue_url: https://api.github.com/repos/simonw/datasette/issues/1862
node_id: IC_kwDOBm6k_c5Np2wP
created_at: 2022-11-04T00:23:13Z
updated_at: 2022-11-04T00:23:13Z
author_association: OWNER
body:
  I don't like this on `/db/table/-/insert` - I think it makes more sense to optionally pass a `"rows"` key to the `/db/-/create` endpoint instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1882#issuecomment-1302818153
issue_url: https://api.github.com/repos/simonw/datasette/issues/1882
node_id: IC_kwDOBm6k_c5Np21p
created_at: 2022-11-04T00:23:58Z
updated_at: 2022-11-04T00:23:58Z
author_association: OWNER
body:
  I made a decision here that this endpoint should also accept an optional `"rows": [...]` list which is used to automatically create the table using a schema derived from those example rows (which then get inserted):

  - https://github.com/simonw/datasette/issues/1862#issuecomment-1302817807

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1882#issuecomment-1302818784
issue_url: https://api.github.com/repos/simonw/datasette/issues/1882
node_id: IC_kwDOBm6k_c5Np2_g
created_at: 2022-11-04T00:25:18Z
updated_at: 2022-11-04T16:12:39Z
author_association: OWNER
body:
  On that basis I think the core API design should change to this:
  ```
  POST /db/-/create
  Authorization: Bearer xxx
  Content-Type: application/json
  {
      "name": "my new table",
      "columns": [
          {
              "name": "id",
              "type": "integer"
          },
          {
              "name": "title",
              "type": "text"
          }
      ]
     "pk": "id"
  }
  ```
  This leaves room for a `"rows": []` key at the root too. Having that as a child of `"table"` felt unintuitive to me, and I didn't like the way this looked either:

  ```json
  {
    "table": {
      "name": "my_new_table"
    },
    "rows": [
      {"id": 1, "title": "Title"}
    ]
  }
  ```
  Weird to have the table `name` nested inside `table` when `rows` wasn't.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1217#issuecomment-1303299509
issue_url: https://api.github.com/repos/simonw/datasette/issues/1217
node_id: IC_kwDOBm6k_c5NrsW1
created_at: 2022-11-04T11:35:13Z
updated_at: 2022-11-04T11:35:13Z
author_association: NONE
body:
  The following worked for deployment to RStudio / Posit Connect

  An app.py along the lines of:

  ```python
  from pathlib import Path

  from datasette.app import Datasette

  example_db = Path(__file__).parent / "data" / "example.db"

  # use connect 'Content URL' setting here to set app to /datasette/
  ds = Datasette(files=[example_db], settings={"base_url": "/datasette/"})

  ds._startup_invoked = True
  ds_app = ds.app()
  ```
  Then to deploy, from within a virtualenv with `rsconnect-python`
  ```sh
  rsconnect write-manifest fastapi -p $VIRTUAL_ENV/bin/python -e app:ds_app -o .
  rsconnect deploy manifest manifest.json -n <name of connect server> -t "Example Datasette"
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1217#issuecomment-1303301786
issue_url: https://api.github.com/repos/simonw/datasette/issues/1217
node_id: IC_kwDOBm6k_c5Nrs6a
created_at: 2022-11-04T11:37:52Z
updated_at: 2022-11-04T11:37:52Z
author_association: NONE
body:
  All seems to work well, but there are some glitches to do with proxies, see #1883 .

  Excited to use this :)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/50#issuecomment-1303660293
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/50
node_id: IC_kwDOCGYnMM5NtEcF
created_at: 2022-11-04T14:38:36Z
updated_at: 2022-11-04T14:38:36Z
author_association: CONTRIBUTOR
body:
  where did you see the limit as 999? I believe the limit has been 32766 for quite some time. If you could detect which one this could speed up batch insert of some types of data significantly

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/511#issuecomment-1304078945
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/511
node_id: IC_kwDOCGYnMM5Nuqph
created_at: 2022-11-04T19:38:36Z
updated_at: 2022-11-04T20:13:17Z
author_association: CONTRIBUTOR
body:
  Even more bizarre, the source db only has one record and the target table has no conflicting record:

  ```
  875 0.3s lb:/ (main|✚2) [0|0]🌺 sqlite-utils tube_71.db 'select * from media where path = "https://archive.org/details/088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz"' | jq
  [
    {
      "size": null,
      "time_created": null,
      "play_count": 1,
      "language": null,
      "view_count": null,
      "width": null,
      "height": null,
      "fps": null,
      "average_rating": null,
      "live_status": null,
      "age_limit": null,
      "uploader": null,
      "time_played": 0,
      "path": "https://archive.org/details/088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz",
      "id": "088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz/074 - Home Away from Home, Rainy Day Robot, Odie the Amazing DVDRip XviD [PhZ].mkv",
      "ie_key": "ArchiveOrg",
      "playlist_path": "https://archive.org/details/088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz",
      "duration": 1424.05,
      "tags": null,
      "title": "074 - Home Away from Home, Rainy Day Robot, Odie the Amazing DVDRip XviD [PhZ].mkv"
    }
  ]
  875 0.3s lb:/ (main|✚2) [0|0]🥧 sqlite-utils video.db 'select * from media where path = "https://archive.org/details/088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz"' | jq
  []
  ```

  I've been able to use this code successfully several times before so not sure what's causing the issue.

  I guess the way that I'm handling multiple databases is an issue, though it hasn't ever inserted into the source db, not sure what's different. The only reasonable explanation is that it is trying to insert into the source db from the source db for some reason? Or maybe sqlite3 is checking the source db for primary key violation because the table name is the same

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/511#issuecomment-1304320521
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/511
node_id: IC_kwDOCGYnMM5NvloJ
created_at: 2022-11-04T22:54:09Z
updated_at: 2022-11-04T22:59:54Z
author_association: CONTRIBUTOR
body:
  I ran `PRAGMA integrity_check` and it returned `ok`. but then I tried restoring from a backup and I didn't get this `IntegrityError: constraint failed` error. So I think it was just something wrong with my database. If it happens again I will first try to reindex and see if that fixes the issue

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1304539296
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/235
node_id: IC_kwDOCGYnMM5NwbCg
created_at: 2022-11-05T12:40:12Z
updated_at: 2022-11-05T12:40:12Z
author_association: NONE
body:
  I had the problem this morning when running:

  `Python==3.9.6
   sqlite3.sqlite_version==3.37.0
    sqlite-utils==3.30
  `

  I upgraded to:
  `Python ==3.10.8 
   sqlite3.sqlite_version==3.37.2
   sqlite-utils==3.30
  `

  and the error did not appear anymore.

  Hope this helps
  Ryan

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1871#issuecomment-1309650806
issue_url: https://api.github.com/repos/simonw/datasette/issues/1871
node_id: IC_kwDOBm6k_c5OD692
created_at: 2022-11-10T01:38:58Z
updated_at: 2022-11-10T01:38:58Z
author_association: CONTRIBUTOR
body:
  > Realized the API explorer doesn't need the API key piece at all - it can work with standard cookie-based auth.
  > 
  > This also reflects how most plugins are likely to use this API, where they'll be adding JavaScript that uses `fetch()` to call the write API directly.

  I agree (that's what I did with the previous insert plugin), maybe a complete example using `fetch()` in the documentation would be valuable as a “Getting started with the API” or similar?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1884#issuecomment-1309735529
issue_url: https://api.github.com/repos/simonw/datasette/issues/1884
node_id: IC_kwDOBm6k_c5OEPpp
created_at: 2022-11-10T03:57:23Z
updated_at: 2022-11-10T03:57:23Z
author_association: CONTRIBUTOR
body:
  Here's how to get a list of virtual tables: https://stackoverflow.com/questions/46617118/how-to-fetch-names-of-virtual-tables

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1880#issuecomment-1311271298
issue_url: https://api.github.com/repos/simonw/datasette/issues/1880
node_id: IC_kwDOBm6k_c5OKGmC
created_at: 2022-11-11T06:12:29Z
updated_at: 2022-11-11T06:12:29Z
author_association: OWNER
body:
  I think you may have misunderstood this feature. This is talking about the `_internal` in-memory database, which maintains a set of tables that list the databases and tables that are attached to Datasette.

  They're not a copy of the data itself - just a list of table names, column names and database names.

  You can see what that database looks like by signing in as root - running `datasette --root` and clicking the link. Or you can see an example here:

  - Click the button on https://latest.datasette.io/login-as-root
  - Now visit https://latest.datasette.io/_internal

  For the example instance that looks like this:

  <img width="697" alt="image" src="https://user-images.githubusercontent.com/9599/201275230-b6375574-17f0-4cd8-b363-0c69a5907080.png">

  The two most interesting tables in there are these ones:

  <img width="1703" alt="image" src="https://user-images.githubusercontent.com/9599/201275343-589eb6d7-6a9f-403c-b84a-07455d936a85.png">

  <img width="1670" alt="CleanShot 2022-11-10 at 22 11 23@2x" src="https://user-images.githubusercontent.com/9599/201275287-04e6c57d-3ef9-4987-9a8b-5d421a48c452.png">

  As you can see, it's just the table schema itself and the columns that make up the tables. Even if you have hundreds of databases connected each with hundreds of tables this should still only add up to a few MB of RAM.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1880#issuecomment-1311273063
issue_url: https://api.github.com/repos/simonw/datasette/issues/1880
node_id: IC_kwDOBm6k_c5OKHBn
created_at: 2022-11-11T06:15:28Z
updated_at: 2022-11-11T06:15:28Z
author_association: OWNER
body:
  The `_internal` database is intended to help Datasette handle much larger attached databases. Right now Datasette attempts to show every database on the https://latest.datasette.io/ index page and every table on the https://latest.datasette.io/fixtures database index page - but these are not paginated. If you had a database containing 1,000 tables the database index page would get pretty slow.

  So I want to be able to paginate (and search) those. But to paginate them it's useful to have them in a database table itself, since then I can paginate using SQL.

  My plan for `_internal` is to use it to implement those advanced browsing features. I've not completed this work yet though. See this issue for more details on that:

  - #417

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1883#issuecomment-1311278678
issue_url: https://api.github.com/repos/simonw/datasette/issues/1883
node_id: IC_kwDOBm6k_c5OKIZW
created_at: 2022-11-11T06:22:37Z
updated_at: 2022-11-11T06:22:37Z
author_association: OWNER
body:
  If you view source on that page the HTML looks correct:
  ```html
  <form class="filters" action="/prefix/fixtures/binary_data" method="get">
  ```
  (I just added a test that confirms this too.)

  But... it looks like the bug is in the redirection code. 

  https://datasette-apache-proxy-demo.datasette.io/prefix/fixtures/binary_data?_filter_column=rowid&_filter_op=exact&_filter_value=1&_sort=rowid returns the following:

      location: /fixtures/binary_data?_sort=rowid&rowid__exact=1

  Which is incorrect.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1883#issuecomment-1311280709
issue_url: https://api.github.com/repos/simonw/datasette/issues/1883
node_id: IC_kwDOBm6k_c5OKI5F
created_at: 2022-11-11T06:25:27Z
updated_at: 2022-11-11T06:25:27Z
author_association: OWNER
body:
  I tried adding this test but it passed! I expected it to fail:

  ```python
  def test_base_url_affects_filter_redirects(app_client_base_url_prefix):
      response = app_client_base_url_prefix.get(
          "/prefix/fixtures/binary_data?_filter_column=rowid&_filter_op=exact&_filter_value=1&_sort=rowid"
      )
      assert response.status == 302
      assert (
          response.headers["location"]
          == "/prefix/fixtures/binary_data?_sort=rowid&rowid__exact=1"
      )
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1883#issuecomment-1311282970
issue_url: https://api.github.com/repos/simonw/datasette/issues/1883
node_id: IC_kwDOBm6k_c5OKJca
created_at: 2022-11-11T06:28:05Z
updated_at: 2022-11-11T06:28:05Z
author_association: OWNER
body:
  Relevant code: https://github.com/simonw/datasette/blob/bbaab3b38ec2ce5944239ffbe2dd53328df40fff/datasette/views/table.py#L227-L249

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1883#issuecomment-1311283301
issue_url: https://api.github.com/repos/simonw/datasette/issues/1883
node_id: IC_kwDOBm6k_c5OKJhl
created_at: 2022-11-11T06:28:38Z
updated_at: 2022-11-11T06:29:33Z
author_association: OWNER
body:
  `path_with_added_args(request, redirect_params)` should be preserving the current path from the request.

  https://github.com/simonw/datasette/blob/bbaab3b38ec2ce5944239ffbe2dd53328df40fff/datasette/utils/__init__.py#L273-L286

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1883#issuecomment-1311284537
issue_url: https://api.github.com/repos/simonw/datasette/issues/1883
node_id: IC_kwDOBm6k_c5OKJ05
created_at: 2022-11-11T06:30:38Z
updated_at: 2022-11-11T06:30:38Z
author_association: OWNER
body:
  Is there a chance that it's Apache that's messing with that `location:` header here, not Datasette?

  https://github.com/simonw/datasette/blob/bbaab3b38ec2ce5944239ffbe2dd53328df40fff/demos/apache-proxy/000-default.conf#L7-L13

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1883#issuecomment-1311286593
issue_url: https://api.github.com/repos/simonw/datasette/issues/1883
node_id: IC_kwDOBm6k_c5OKKVB
created_at: 2022-11-11T06:34:09Z
updated_at: 2022-11-11T06:34:09Z
author_association: OWNER
body:
  https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypass includes this note:

  > Normally, mod_proxy will canonicalise ProxyPassed URLs. But this may be incompatible with some backends, particularly those that make use of *PATH_INFO*. The optional *nocanon* keyword suppresses this and passes the URL path "raw" to the backend.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1883#issuecomment-1311290115
issue_url: https://api.github.com/repos/simonw/datasette/issues/1883
node_id: IC_kwDOBm6k_c5OKLMD
created_at: 2022-11-11T06:40:14Z
updated_at: 2022-11-11T06:41:56Z
author_association: OWNER
body:
  I modified that config file to have this line instead:
  ```
      ProxyPass /prefix/ http://127.0.0.1:8001/ nocanon
  ```
  And then deployed it by running:

      flyctl deploy --build-arg DATASETTE_REF=main

  This does NOT seem to have fixed the bug:

  ```
  ~ % curl -i 'https://datasette-apache-proxy-demo.datasette.io/prefix/fixtures/binary_data?_filter_column=rowid&_filter_op=exact&_filter_value=1&_sort=rowid'
  HTTP/2 302 
  date: Fri, 11 Nov 2022 06:40:01 GMT
  server: Fly/b1863e2e7 (2022-11-09)
  location: /fixtures/binary_data?_sort=rowid&rowid__exact=1
  link: </fixtures/binary_data?_sort=rowid&rowid__exact=1>; rel=preload
  content-type: text/plain
  x-proxied-by: Apache2 Debian
  via: 2 fly.io
  fly-request-id: 01GHJPNCF51CJ626EWZEHK2CH9-sjc
  ```

  https://datasette-apache-proxy-demo.datasette.io/prefix/-/versions seems to confirm that this is the latest deployed version (0.63), so it looks like the deploy worked.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1883#issuecomment-1311291632
issue_url: https://api.github.com/repos/simonw/datasette/issues/1883
node_id: IC_kwDOBm6k_c5OKLjw
created_at: 2022-11-11T06:43:00Z
updated_at: 2022-11-11T06:43:00Z
author_association: OWNER
body:
  https://datasette-apache-proxy-demo.datasette.io/prefix/-/asgi-scope is useful:

  It confirms that `/prefix/` is nowhere to be seen in the incoming request data:

  ```
   'path': '/-/asgi-scope',
   'query_string': b'',
   'raw_path': b'/-/asgi-scope',
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1883#issuecomment-1311292463
issue_url: https://api.github.com/repos/simonw/datasette/issues/1883
node_id: IC_kwDOBm6k_c5OKLwv
created_at: 2022-11-11T06:44:24Z
updated_at: 2022-11-11T06:44:24Z
author_association: OWNER
body:
  Modifying that test to the following does indeed cause a failure:
  ```python
  def test_base_url_affects_filter_redirects(app_client_base_url_prefix):
      response = app_client_base_url_prefix.get(
          "/fixtures/binary_data?_filter_column=rowid&_filter_op=exact&_filter_value=1&_sort=rowid"
      )
      assert response.status == 302
      assert (
          response.headers["location"]
          == "/prefix/fixtures/binary_data?_sort=rowid&rowid__exact=1"
      )
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1883#issuecomment-1311299535
issue_url: https://api.github.com/repos/simonw/datasette/issues/1883
node_id: IC_kwDOBm6k_c5OKNfP
created_at: 2022-11-11T06:54:58Z
updated_at: 2022-11-11T06:54:58Z
author_association: OWNER
body:
  This time deployed with:
  ```
  cd demos/apache-proxy
  fly deploy --build-arg DATASETTE_REF=8d9a957c6329d26cc1e417b5d6911640d74765eb
  ```
  To ensure the exact commit with the fix.

  And that fixed it!
  ```
  % curl -i 'https://datasette-apache-proxy-demo.datasette.io/prefix/fixtures/binary_data?_filter_column=rowid&_filter_op=exact&_filter_value=1&_sort=rowid'
  HTTP/2 302 
  date: Fri, 11 Nov 2022 06:54:45 GMT
  server: Fly/b1863e2e7 (2022-11-09)
  location: /prefix/fixtures/binary_data?_sort=rowid&rowid__exact=1
  link: </prefix/fixtures/binary_data?_sort=rowid&rowid__exact=1>; rel=preload
  content-type: text/plain
  x-proxied-by: Apache2 Debian
  via: 2 fly.io
  fly-request-id: 01GHJQGBSXBR7E53TY0EKMQ9PA-sjc
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1878#issuecomment-1312534826
issue_url: https://api.github.com/repos/simonw/datasette/issues/1878
node_id: IC_kwDOBm6k_c5OO7Eq
created_at: 2022-11-12T17:34:58Z
updated_at: 2022-11-12T17:34:58Z
author_association: NONE
body:
  Hi Simon. I have just started experimenting with datasette in earnest, looking at it's suitability for air quality open data. A bulk upsert \ upsert_all would be very useful for me in enabling real time data to be pushed from a sql server  database with FME server to a datasette db.  

  An hourly process queries the last 2 hours of data and pushes that to my database,  inserting new data and updating existing combinations of pk siteid and date_time. This is already implemented on our current [open data portal](https://opendata.bristol.gov.uk/explore/dataset/air-quality-data-continuous/table/?disjunctive.location&sort=date_time). Excited to see your progress with this! Thank you for this amazing software.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1882#issuecomment-1312556044
issue_url: https://api.github.com/repos/simonw/datasette/issues/1882
node_id: IC_kwDOBm6k_c5OPAQM
created_at: 2022-11-12T19:29:11Z
updated_at: 2022-11-12T19:29:11Z
author_association: OWNER
body:
  Thought of an edge-case: with `sqlite-utils` one feature I really like is that I can pipe data into it without caring if the table already exists or not:

      cat data.json | sqlite-utils insert my.db mytable -

  How could this new API support that?

  I thought about adding a `"create": true` option to `/db/table/-/insert` which creates the table if it doesn't already exist, but if I do that I'll need to start adding other options to that endpoint - to set the primary key, add foreign keys and suchlike - which would be ignored except for the cases where the table was being created from scratch.

  This doesn't feel right to me - I want to keep those options here, on `/db/-/create`.

  One idea I had was to implement it such that you can call `/db/-/create` multiple times for the same table, but only if you are using the `"rows"` option. If so, and if the rows can be safely inserted, it would let you do that.

  But instead, I'm going to outsource this to the CLI tool I plan to write that feeds data into this API. I'm already planning to use that tool for CSV inserts (so the API doesn't need to accept CSV directly). I think it's a good place for other usability enhancements like "insert this, creating the table if it does not exist" as well.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1882#issuecomment-1312575048
issue_url: https://api.github.com/repos/simonw/datasette/issues/1882
node_id: IC_kwDOBm6k_c5OPE5I
created_at: 2022-11-12T21:22:58Z
updated_at: 2022-11-12T21:22:58Z
author_association: OWNER
body:
  Need to validate the table name. SQLite supports almost any table name - but they can't contain a newline character and cannot start with `sqlite_` - according to https://stackoverflow.com/questions/3694276/what-are-valid-table-names-in-sqlite/43049720#43049720

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1882#issuecomment-1312580348
issue_url: https://api.github.com/repos/simonw/datasette/issues/1882
node_id: IC_kwDOBm6k_c5OPGL8
created_at: 2022-11-12T21:55:54Z
updated_at: 2022-11-12T21:56:45Z
author_association: OWNER
body:
  What should this API return?

  I think the name of the table (`name`), the URL to that table (`table_url` - for consistency with how faceting API works already) and the schema of the table (`schema`).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1882#issuecomment-1312581008
issue_url: https://api.github.com/repos/simonw/datasette/issues/1882
node_id: IC_kwDOBm6k_c5OPGWQ
created_at: 2022-11-12T22:00:52Z
updated_at: 2022-11-12T22:00:52Z
author_association: OWNER
body:
  Tried out my prototype in the API explorer:

  <img width="1229" alt="image" src="https://user-images.githubusercontent.com/9599/201496194-a6ebcefe-2be8-47d7-8f15-8c6345cba7cf.png">

  The `"name"` on the output is bothering me a bit - should it be `table_name` or `table` instead?

  Problem is I really like `name` for the input, so should it be consistent to have the same name on the output here, or should I aim for consistency with other endpoints that might return `table` rather than the ambiguous `name` elsewhere?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1882#issuecomment-1312582512
issue_url: https://api.github.com/repos/simonw/datasette/issues/1882
node_id: IC_kwDOBm6k_c5OPGtw
created_at: 2022-11-12T22:11:18Z
updated_at: 2022-11-12T22:11:18Z
author_association: OWNER
body:
  I like this:

  <img width="1080" alt="image" src="https://user-images.githubusercontent.com/9599/201496508-2fe102e8-f0de-4dd8-87c5-40b6aae97413.png">

  ```json
  {
    "ok": true,
    "database": "data",
    "table": "agai2n",
    "table_url": "http://127.0.0.1:8001/data/agai2n",
    "schema": "CREATE TABLE [agai2n] (\n   [hello] INTEGER\n)"
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1886#issuecomment-1312814245
issue_url: https://api.github.com/repos/simonw/datasette/issues/1886
node_id: IC_kwDOBm6k_c5OP_Sl
created_at: 2022-11-13T20:28:26Z
updated_at: 2022-11-13T20:28:26Z
author_association: NONE
body:
  I work at The Wall Street Journal as a computational journalist and serve as our self-appointed Datasette evangelist. They say that to a hammer everything looks like a nail, but the reality is newsrooms find themselves in a sea of nails!

  I've only got a couple public projects that I can share, but happy to offer you a look at some of the internal projects. 

  More often than not the internal projects stay internal because the reporting doesn't lead anywhere or I can't convince an editor to greenlight it. But imho that's the beauty of datasette: a (relatively) painless mechanism to see if there's any there there. 

  - [WSJ Inflation Tracker](wsj.com/inflationtracker)
  - I scraped the oscars website and turned it into a datasette instance and ran the numbers on [best actress/best picture overlap ](https://www.wsj.com/livecoverage/oscars-academy-awards-2022/card/the-best-actress-nominees-aren-t-in-any-best-pictures-contenders-when-is-the-last-time-that-happened--mDxvbLug3rq84pxLE8gY)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1871#issuecomment-1312816292
issue_url: https://api.github.com/repos/simonw/datasette/issues/1871
node_id: IC_kwDOBm6k_c5OP_yk
created_at: 2022-11-13T20:38:42Z
updated_at: 2022-11-13T20:38:42Z
author_association: OWNER
body:
  The current API explorer uses details/summary elements for the GET and POST dialogs.

  I only want one of these to be open at a time, to reflect that you can make either a GET or a POST.

  I just noticed that clicking anywhere else on the page closes both elements, which isn't what I want to happen. Turns out that's because of this code I added as part of Datasette's menu implementation!

  https://github.com/simonw/datasette/blob/9f54f00a50a4d950cfd69a0ff3526ae82c858826/datasette/templates/_close_open_menus.html#L2-L15

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1871#issuecomment-1312821031
issue_url: https://api.github.com/repos/simonw/datasette/issues/1871
node_id: IC_kwDOBm6k_c5OQA8n
created_at: 2022-11-13T21:02:06Z
updated_at: 2022-11-13T21:03:11Z
author_association: OWNER
body:
  Actually no, I'm going to add a class of `details-menu` to the other details elements that SHOULD be closed. That way custom templates using `<details>` won't close in a surprising way.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1886#issuecomment-1312898318
issue_url: https://api.github.com/repos/simonw/datasette/issues/1886
node_id: IC_kwDOBm6k_c5OQT0O
created_at: 2022-11-14T00:52:16Z
updated_at: 2022-11-14T00:52:16Z
author_association: NONE
body:
  I'm a cryptic crossword enthusiast and have spent a lot of time scraping and parsing cryptic crossword clues from various blogs, forums and publications. The result is over **half a million clues from cryptic crosswords over the past twelve years**, including the clue, answer, puzzle date, puzzle name and a link to the original source. This is all hosted using Datasette, which has been a delight to use: https://cryptics.georgeho.org/

  This dataset is a significant work of crossword archivism and scholarship, as acquiring historical crosswords and structuring their contents require focused effort and tedious cleaning that few are willing to do for such trivial data - for example, according to [this 2004 selection guide](https://cryptics.georgeho.org/static/documents/Selection_AppendixE_v2.pdf), the Library of Congress explicitly does not collect crossword puzzles. Anecdotally, I know that many constructors/setters of cryptic crosswords use this dataset as a resource, and some even simply call it "the database" - this is probably one of the most impactful data projects I've worked on!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1886#issuecomment-1313052863
issue_url: https://api.github.com/repos/simonw/datasette/issues/1886
node_id: IC_kwDOBm6k_c5OQ5i_
created_at: 2022-11-14T03:40:50Z
updated_at: 2022-11-14T03:40:50Z
author_association: OWNER
body:
  Tim Sherratt on Twitter: https://twitter.com/wragge/status/1591930345469153282

  > Where do I start? The [#GLAMWorkbench](https://twitter.com/hashtag/GLAMWorkbench?src=hashtag_click) now includes a number of examples where GLAM data is harvested, processed, and then made available for exploration via Datasette.
  >
  > https://glam-workbench.net/
  >
  > For example the GLAM Name Index Search brings together 10+ million entries from 240 indexes and provides an aggregated search using the Datasette search-all plugin:
  >
  > https://glam-workbench.net/name-search/
  >
  > Most recently I converted PDFs of the Tasmanian Postal Directories to a big Datasette instance:  https://updates.timsherratt.org/2022/09/15/from-pdfs-to.html the process is documented and reusable.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1871#issuecomment-1313062699
issue_url: https://api.github.com/repos/simonw/datasette/issues/1871
node_id: IC_kwDOBm6k_c5OQ78r
created_at: 2022-11-14T04:03:29Z
updated_at: 2022-11-14T04:12:41Z
author_association: OWNER
body:
  Two things left before I close this issue:

  - [x] I want to preserve the state of the forms in the URL - probably after a `#`
  - [ ] Instead of hard-coding the current examples, I want to provide a list of links which populate the forms

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1871#issuecomment-1313072900
issue_url: https://api.github.com/repos/simonw/datasette/issues/1871
node_id: IC_kwDOBm6k_c5OQ-cE
created_at: 2022-11-14T04:15:50Z
updated_at: 2022-11-14T04:15:50Z
author_association: OWNER
body:
  For the example links - I'm going to have these at the bottom of the page so you don't have to scroll past them.

  Ideally these would take the user's permissions into account. This could make the page expensive to load, but I'm going to risk it for the moment.

  Something like this then:

  > - data
  >   - /data/-/create - create table
  >   - /data/table1/-/insert - insert into table1
  >   - /data/table1/-/drop - drop table1

  I won't bother with per-row demo links (for update and delete) because there could be thousands of them for each table.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1871#issuecomment-1313097057
issue_url: https://api.github.com/repos/simonw/datasette/issues/1871
node_id: IC_kwDOBm6k_c5OREVh
created_at: 2022-11-14T04:59:28Z
updated_at: 2022-11-14T04:59:28Z
author_association: OWNER
body:
  In playing with the API explorer just now I realized it's way too easy to accidentally drop a table using it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1887#issuecomment-1313097713
issue_url: https://api.github.com/repos/simonw/datasette/issues/1887
node_id: IC_kwDOBm6k_c5OREfx
created_at: 2022-11-14T05:00:54Z
updated_at: 2022-11-14T05:00:54Z
author_association: OWNER
body:
  I'm going to add a `"confirm": true` option to the API. Without that, it returns a note about how many rows will be deleted.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1875#issuecomment-1313114283
issue_url: https://api.github.com/repos/simonw/datasette/issues/1875
node_id: IC_kwDOBm6k_c5ORIir
created_at: 2022-11-14T05:20:00Z
updated_at: 2022-11-14T05:20:00Z
author_association: OWNER
body:
  I started a conversation about JSON error standards on Mastodon here: https://fedi.simonwillison.net/web/@simon/109338725610487457

  Quite a few people pointed to this RFC independently.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1850#issuecomment-1313115059
issue_url: https://api.github.com/repos/simonw/datasette/issues/1850
node_id: IC_kwDOBm6k_c5ORIuz
created_at: 2022-11-14T05:21:30Z
updated_at: 2022-11-14T05:21:30Z
author_association: OWNER
body:
  New documentation for these features currently lives here: https://docs.datasette.io/en/1.0-dev/json_api.html#the-json-write-api

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1874#issuecomment-1313125123
issue_url: https://api.github.com/repos/simonw/datasette/issues/1874
node_id: IC_kwDOBm6k_c5ORLMD
created_at: 2022-11-14T05:41:20Z
updated_at: 2022-11-14T05:42:23Z
author_association: OWNER
body:
  I also changed the confirmation JSON returned by this endpoint to add the `database` and `table` like so:
  ```json
  {
      "ok": true,
      "database": "data",
      "table": "docs",
      "row_count": 1,
      "message": "Pass \"confirm\": true to confirm"
  }
  ```
  Updated docs: https://docs.datasette.io/en/1.0-dev/json_api.html#dropping-tables

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1871#issuecomment-1313125870
issue_url: https://api.github.com/repos/simonw/datasette/issues/1871
node_id: IC_kwDOBm6k_c5ORLXu
created_at: 2022-11-14T05:42:50Z
updated_at: 2022-11-14T05:42:50Z
author_association: OWNER
body:
  Demo: https://latest-1-0-dev.datasette.io/-/api#path=%2Ffixtures%2Ffacetable%2F-%2Fdrop&json=%7B%22confirm%22%3A+true%7D&method=POST

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1874#issuecomment-1313127054
issue_url: https://api.github.com/repos/simonw/datasette/issues/1874
node_id: IC_kwDOBm6k_c5ORLqO
created_at: 2022-11-14T05:45:00Z
updated_at: 2022-11-14T05:45:00Z
author_association: OWNER
body:
  Demo: https://latest-1-0-dev.datasette.io/-/api#path=%2Ffixtures%2Ffacetable%2F-%2Fdrop&json=&method=POST

  <img width="974" alt="image" src="https://user-images.githubusercontent.com/9599/201584446-fd9d3978-d37d-4e81-8179-cd587027df46.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1866#issuecomment-1313128913
issue_url: https://api.github.com/repos/simonw/datasette/issues/1866
node_id: IC_kwDOBm6k_c5ORMHR
created_at: 2022-11-14T05:48:22Z
updated_at: 2022-11-14T05:48:22Z
author_association: OWNER
body:
  I changed my mind about the `"return_rows": true` option - I'm going to rename it to `"return": true`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1855#issuecomment-1313148519
issue_url: https://api.github.com/repos/simonw/datasette/issues/1855
node_id: IC_kwDOBm6k_c5ORQ5n
created_at: 2022-11-14T06:13:43Z
updated_at: 2022-12-13T02:46:51Z
author_association: OWNER
body:
  The `datasette create-token` command will need to be able to do this too.

  Right now that command looks like this:
  ```
  % datasette create-token --help
  Usage: datasette create-token [OPTIONS] ID

    Create a signed API token for the specified actor ID

  Options:
    --secret TEXT                Secret used for signing the API tokens
                                 [required]
    -e, --expires-after INTEGER  Token should expire after this many seconds
    --debug                      Show decoded token
    --help                       Show this message and exit.
  ```
  ```
  % datasette create-token root --secret sec --debug -e 445
  dstok_eyJhIjoicm9vdCIsInRva2VuIjoiZHN0b2siLCJ0IjoxNjY4NDA2MjEzLCJkIjo0NDV9.Hd6qRli6xRKkOIRQgZkPO5iN1wM

  Decoded:

  {
    "a": "root",
    "token": "dstok",
    "t": 1668406213,
    "d": 445
  }
  ```
  (The `--debug` bit adds the decoded token.)

  Syntax for adding "insert row" for everything, "update row" for all in the "data" database and "delete row" just for the docs / titles table:
  ```
  datasette create-token root --secret sec \
    --all insert-row \
    --database data update-row \
    --table docs titles delete-row
  ```
  The `ir` / `ur` / `dr` options would work too. To add multiple permissions use these options multiple times:
  ```
  datasette create-token root --secret sec \
    --all insert-row \
    --all delete-row
  ```
  Short versions: `-a` and `-d` and `-t`.

  UPDATE: I have decided to use the term `resource` in the user-facing elements of this feature instead of `table`, since that can refer to a SQL view and a canned query as well.

  So `--resource` and `-r`, not `-t`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1850#issuecomment-1313155712
issue_url: https://api.github.com/repos/simonw/datasette/issues/1850
node_id: IC_kwDOBm6k_c5ORSqA
created_at: 2022-11-14T06:22:57Z
updated_at: 2022-11-14T06:22:57Z
author_association: OWNER
body:
  I think the ability to create tokens should be protected by a `create-tokens` permission, not just a global setting.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1886#issuecomment-1313252879
issue_url: https://api.github.com/repos/simonw/datasette/issues/1886
node_id: IC_kwDOBm6k_c5ORqYP
created_at: 2022-11-14T08:10:23Z
updated_at: 2022-11-14T08:10:23Z
author_association: CONTRIBUTOR
body:
  Hi @simonw and thanks for the great tools you're publishing, your dedication is inspiring!

  I work for the French Ministry of Culture on a surveying tool for objects protected for their historical value. It is part of a program building modern public services called [beta.gouv.fr](https://beta.gouv.fr/).

  In that context I'm using data published by the Ministry that I have ingested into datasette and published on a free Fly instance : https://collectif-objets-datasette.fly.dev . I have also ingested another data set with infos about french cities on this instance so that I can perform joined queries.

  The surveying tool synchronizes its data regularly from this datasette instance, and I also use it to perform queries when asked generic questions about the distribution of objects. (The data is not very accessible as it's undocumented and for internal usage mostly)

reactions:
  {"total_count": 3, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 3, "rocket": 0, "eyes": 0}


html_url: https://github.com/simonw/datasette/issues/1886#issuecomment-1313271719
issue_url: https://api.github.com/repos/simonw/datasette/issues/1886
node_id: IC_kwDOBm6k_c5ORu-n
created_at: 2022-11-14T08:25:12Z
updated_at: 2022-11-14T08:25:12Z
author_association: NONE
body:
  Nothing spectacular yet but I think this falls under "cool/cute application of datasette": [improving fakedata performance for fun](https://lucapette.me/writing/improving-fakedata-performance-for-fun/). tl;dr I used datasette to visualize benchmarking data.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1884#issuecomment-1313962183
issue_url: https://api.github.com/repos/simonw/datasette/issues/1884
node_id: IC_kwDOBm6k_c5OUXjH
created_at: 2022-11-14T15:46:32Z
updated_at: 2022-11-14T15:46:32Z
author_association: CONTRIBUTOR
body:
  It does work, though I think it's probably still worth excluding virtual tables that will always be zero. Here's the same inspection as before, now with `--load-extension spatialite`:

  ```json
  {
    "alltheplaces": {
      "hash": "0843cfe414439ab903c22d1121b7ddbc643418c35c7f0edbcec82ef1452411df",
      "size": 963375104,
      "file": "alltheplaces.db",
      "tables": {
        "spatial_ref_sys": {
          "count": 6215
        },
        "spatialite_history": {
          "count": 18
        },
        "sqlite_sequence": {
          "count": 2
        },
        "geometry_columns": {
          "count": 3
        },
        "spatial_ref_sys_aux": {
          "count": 6164
        },
        "views_geometry_columns": {
          "count": 0
        },
        "virts_geometry_columns": {
          "count": 0
        },
        "geometry_columns_statistics": {
          "count": 3
        },
        "views_geometry_columns_statistics": {
          "count": 0
        },
        "virts_geometry_columns_statistics": {
          "count": 0
        },
        "geometry_columns_field_infos": {
          "count": 0
        },
        "views_geometry_columns_field_infos": {
          "count": 0
        },
        "virts_geometry_columns_field_infos": {
          "count": 0
        },
        "geometry_columns_time": {
          "count": 3
        },
        "geometry_columns_auth": {
          "count": 3
        },
        "views_geometry_columns_auth": {
          "count": 0
        },
        "virts_geometry_columns_auth": {
          "count": 0
        },
        "data_licenses": {
          "count": 10
        },
        "sql_statements_log": {
          "count": 0
        },
        "states": {
          "count": 56
        },
        "counties": {
          "count": 3234
        },
        "idx_states_geometry_rowid": {
          "count": 56
        },
        "idx_states_geometry_node": {
          "count": 3
        },
        "idx_states_geometry_parent": {
          "count": 2
        },
        "idx_counties_geometry_rowid": {
          "count": 3234
        },
        "idx_counties_geometry_node": {
          "count": 98
        },
        "idx_counties_geometry_parent": {
          "count": 97
        },
        "idx_places_geometry_rowid": {
          "count": 1236796
        },
        "idx_places_geometry_node": {
          "count": 38163
        },
        "idx_places_geometry_parent": {
          "count": 38162
        },
        "places": {
          "count": 1332609
        },
        "SpatialIndex": {
          "count": 0
        },
        "ElementaryGeometries": {
          "count": 0
        },
        "KNN": {
          "count": 0
        },
        "idx_states_geometry": {
          "count": 56
        },
        "idx_counties_geometry": {
          "count": 3234
        },
        "idx_places_geometry": {
          "count": 1236796
        }
      }
    }
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1884#issuecomment-1314054300
issue_url: https://api.github.com/repos/simonw/datasette/issues/1884
node_id: IC_kwDOBm6k_c5OUuCc
created_at: 2022-11-14T16:40:06Z
updated_at: 2022-11-14T16:40:06Z
author_association: OWNER
body:
  I wonder if there are any reasons that inspect SHOULD try to count virtual tables? Like are there any likely uses for a cirial table where the count is both interesting and likely to be accessed often enough that it's worth caching?

  I have an issue open to add a setting to disable table counts entirely:

  - #1818 

  Maybe that should be expanded to automatically disable row counts for virtual tables entirely? Which would mean no count would be shown for them in the UI.

  If you desperately wanted a count you would  then have to run a count(*) query against them explicitly.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1884#issuecomment-1314066229
issue_url: https://api.github.com/repos/simonw/datasette/issues/1884
node_id: IC_kwDOBm6k_c5OUw81
created_at: 2022-11-14T16:48:35Z
updated_at: 2022-11-14T16:48:35Z
author_association: CONTRIBUTOR
body:
  I'm realizing I don't know if a virtual table will ever return a count. Maybe it depends on the implementation. For these three, just checking now, it'll always return zero.

  That said, I'm not sure there's any downside to having them return zero and caching that. (They're hidden, too.) 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1886#issuecomment-1314223118
issue_url: https://api.github.com/repos/simonw/datasette/issues/1886
node_id: IC_kwDOBm6k_c5OVXQO
created_at: 2022-11-14T18:51:20Z
updated_at: 2022-11-14T18:51:20Z
author_association: NONE
body:
  I use Datasette to analyze blocklists by using csv-to-sqlite to pull their contents into a database and Datasette to look around through them.  I also use its REST API to query said database as part of filtering out garbage from domains found in those blocklists.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1886#issuecomment-1314241058
issue_url: https://api.github.com/repos/simonw/datasette/issues/1886
node_id: IC_kwDOBm6k_c5OVboi
created_at: 2022-11-14T19:06:35Z
updated_at: 2022-11-14T19:06:35Z
author_association: CONTRIBUTOR
body:
  This probably counts as a case study: https://github.com/eyeseast/spatial-data-cooking-show. Even has video.

  Seriously, though, this workflow has become integral to my work with reporters and editors across USA TODAY Network. Very often, I get sent a folder of data in mixed formats, with a vague ask of how we should communicate some part of it to users. Datasette and its constellation of tools makes it easy to get a quick look at that data, run exploratory queries, map it and ask questions to figure out what's important to show. And then I export a version of the data that's exactly what I need for display.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1886#issuecomment-1314455003
issue_url: https://api.github.com/repos/simonw/datasette/issues/1886
node_id: IC_kwDOBm6k_c5OWP3b
created_at: 2022-11-14T21:51:11Z
updated_at: 2022-11-14T21:51:11Z
author_association: NONE
body:
  Happy Birthday Datasette!

  I am a librarian at the Université du Québec à Montréal (UQAM) and I've been using Datasette to publish excerpts of our library data.
  There are several use cases I'm working with as a proof of concept : 
  1. New titles list : based on reports of recent acquisitions by subject, discipline, etc.
  2. List of all UQAM theses and dissertations : based on an extract of bibliographic records
  3. List of all publications by UQAM Authors : based on an extract of bibliographic records

  See our prototype under construction here : https://datasette-bib.uqam.ca/
  (some bits and pieces have been translated into French)

  Datasette is amazing, there is so much potential here for libraries. Thanks to Simon and all the contributors for this outstanding effort.
  Also sqlite-utils deserves special mention as incredibly handy and useful.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1875#issuecomment-1314488010
issue_url: https://api.github.com/repos/simonw/datasette/issues/1875
node_id: IC_kwDOBm6k_c5OWX7K
created_at: 2022-11-14T22:21:43Z
updated_at: 2022-11-14T22:21:43Z
author_association: OWNER
body:
  Here's the most relevant example from the RFC spec:
  ```
     POST /details HTTP/1.1
     Host: account.example.com
     Accept: application/json
  ```
  ```json
     {
       "age": 42.3,
       "profile": {
         "color": "yellow"
       }
     }
  ```
  ```
     HTTP/1.1 400 Bad Request
     Content-Type: application/problem+json
     Content-Language: en
  ```
  ```json
  {
      "type": "https://example.net/validation-error",
      "title": "Your request is not valid.",
      "errors": [
          {
              "detail": "must be a positive integer",
              "pointer": "#/age"
          },
          {
              "detail": "must be 'green', 'red' or 'blue'",
              "pointer": "#/profile/color"
          }
      ]
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1875#issuecomment-1314491150
issue_url: https://api.github.com/repos/simonw/datasette/issues/1875
node_id: IC_kwDOBm6k_c5OWYsO
created_at: 2022-11-14T22:25:20Z
updated_at: 2022-11-14T22:25:20Z
author_association: OWNER
body:
  That's using JSON Pointer: https://www.rfc-editor.org/rfc/rfc6901

  There's a Python library for that here https://github.com/stefankoegl/python-json-pointer/blob/master/jsonpointer.py - which looks simple and clean and well maintained and documented, but it only handles the "what is at this pointer within this JSON object" case - I need to generate the correct JSON pointer to explain where my error is.

  So I think I'll end up hand-rolling this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1875#issuecomment-1314491884
issue_url: https://api.github.com/repos/simonw/datasette/issues/1875
node_id: IC_kwDOBm6k_c5OWY3s
created_at: 2022-11-14T22:26:11Z
updated_at: 2022-11-14T22:26:54Z
author_association: OWNER
body:
  Spec looks pretty simple:

  > A JSON Pointer is a Unicode string (see [RFC4627], Section 3)
  > containing a sequence of zero or more reference tokens, each prefixed
  > by a `/` (%x2F) character.
  > 
  > Because the characters `~` (%x7E) and `/` (%x2F) have special
  > meanings in JSON Pointer, `~` needs to be encoded as `~0` and `/`
  > needs to be encoded as `~1` when these characters appear in a
  > reference token.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1875#issuecomment-1314615592
issue_url: https://api.github.com/repos/simonw/datasette/issues/1875
node_id: IC_kwDOBm6k_c5OW3Eo
created_at: 2022-11-15T01:04:28Z
updated_at: 2022-11-15T01:04:28Z
author_association: OWNER
body:
  Worth noting this bit in RFC 7807:

  >    The fictional problem type here defines the "errors" extension, an
  >    array that describes the details of each validation error.  Each
  >    member is an object containing "detail" to describe the issue, and
  >    "pointer" to locate the problem within the request's content using a
  >    JSON Pointer [JSON-POINTER].

  So the list of `"errors"` with JSON Pointer isn't technically part of the spec, it's an imaginary extension.

  It fits what I need to do though, so I'm inclined to stick with it anyway.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1875#issuecomment-1314620086
issue_url: https://api.github.com/repos/simonw/datasette/issues/1875
node_id: IC_kwDOBm6k_c5OW4K2
created_at: 2022-11-15T01:09:56Z
updated_at: 2022-11-15T01:09:56Z
author_association: OWNER
body:
  Rough initial prototype:
  ```diff
  diff --git a/datasette/views/table.py b/datasette/views/table.py
  index 8b987221..518ac578 100644
  --- a/datasette/views/table.py
  +++ b/datasette/views/table.py
  @@ -1103,19 +1103,30 @@ class TableInsertView(BaseView):
           except json.JSONDecodeError as e:
               return _errors(["Invalid JSON: {}".format(e)])
           if not isinstance(data, dict):
  -            return _errors(["JSON must be a dictionary"])
  +            return _errors([{"detail": "JSON must be a dictionary", "pointer": "#/"}])
           keys = data.keys()
   
           # keys must contain "row" or "rows"
           if "row" not in keys and "rows" not in keys:
               return _errors(['JSON must have one or other of "row" or "rows"'])
           rows = []
  +        was_single_row = False
           if "row" in keys:
               if "rows" in keys:
  -                return _errors(['Cannot use "row" and "rows" at the same time'])
  +                return _errors(
  +                    [
  +                        {
  +                            "detail": 'Cannot use "row" and "rows" at the same time',
  +                            "pointer": "#/row",
  +                        }
  +                    ]
  +                )
  +            was_single_row = True
               row = data["row"]
               if not isinstance(row, dict):
  -                return _errors(['"row" must be a dictionary'])
  +                return _errors(
  +                    [{"detail": '"row" must be a dictionary', "pointer": "#/row"}]
  +                )
               rows = [row]
               data["return"] = True
           else:
  @@ -1152,9 +1163,12 @@ class TableInsertView(BaseView):
               invalid_columns = set(row.keys()) - columns
               if invalid_columns:
                   errors.append(
  -                    "Row {} has invalid columns: {}".format(
  -                        i, ", ".join(sorted(invalid_columns))
  -                    )
  +                    {
  +                        "detail": "Invalid columns: {}".format(
  +                            ", ".join(sorted(invalid_columns))
  +                        ),
  +                        "pointer": "#/blah/",
  +                    }
                   )
           if errors:
               return _errors(errors)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1886#issuecomment-1314627077
issue_url: https://api.github.com/repos/simonw/datasette/issues/1886
node_id: IC_kwDOBm6k_c5OW54F
created_at: 2022-11-15T01:19:54Z
updated_at: 2022-11-15T01:19:54Z
author_association: NONE
body:
  Datasette usage comments for its 5th anniversary celebration:

  I use Datasette and related tools for a Cosmology Researcher Talks database app project, which is [described in the github Readme](https://github.com/jrdmb/cosmotalks-datasette#readme)  

  The app hosted on the Google Cloud Run service also uses other Datasette-related tools developed by Simon - datasette-render-markdown, csvs-to-sqlite, datasette-template-sql, and datasette-block-robots. This is one of two apps used for querying the talks database, each has it pros/cons as described in the github Readme.

  At present, over 170 different sites that host cosmology talks are scraped to collect new talks for import into the sqlite database. The shot-scraper and sqlite-utils tools are a major help for this.

  I also use the Mastodon API to get my favorites, toots, and boosts into a local database so I can do searches on the data. This was done on Twitter and was then extended to the Mastodon data. Again, sqlite-utils is an important tool for this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1882#issuecomment-1314813205
issue_url: https://api.github.com/repos/simonw/datasette/issues/1882
node_id: IC_kwDOBm6k_c5OXnUV
created_at: 2022-11-15T06:00:41Z
updated_at: 2022-11-15T06:00:41Z
author_association: OWNER
body:
  Documentation:

  - https://docs.datasette.io/en/1.0-dev/json_api.html#creating-a-table
  - https://docs.datasette.io/en/1.0-dev/json_api.html#creating-a-table-from-example-data

  Wrote a TIL about how I wrote some of those tests with Copilot: https://til.simonwillison.net/gpt3/writing-test-with-copilot

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1890#issuecomment-1314821337
issue_url: https://api.github.com/repos/simonw/datasette/issues/1890
node_id: IC_kwDOBm6k_c5OXpTZ
created_at: 2022-11-15T06:08:19Z
updated_at: 2022-11-15T06:08:19Z
author_association: OWNER
body:
  Oh interesting... this doesn't even need to be attached to the visible faceting feature, necessarily: Datasette could try to detect when a column has a limited number of options (which the faceting code handles already) and could turn those into an auto-complete interface.

  There's actually a native HTML element for this these days: the `<datalist>` https://developer.mozilla.org/en-US/docs/Web/HTML/Element/datalist

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1890#issuecomment-1314823752
issue_url: https://api.github.com/repos/simonw/datasette/issues/1890
node_id: IC_kwDOBm6k_c5OXp5I
created_at: 2022-11-15T06:11:49Z
updated_at: 2022-11-15T06:11:49Z
author_association: OWNER
body:
  I tried this out on https://congress-legislators.datasettes.com/legislators/legislator_terms for the `party` column - here's the demo:

  ![datalist](https://user-images.githubusercontent.com/9599/201839812-db887ce0-c4b9-432c-8620-5ac73f222a63.gif)

  I made this work by dropping the following HTML into the page in the browser DevTools:
  ```html
  <datalist id="party">
  <option value="Anti-Administration">
  <option value="Pro-Administration">
  <option value="Republican">
  <option value="Federalist">
  <option value="Democratic Republican">
  <option value="Pro-administration">
  <option value="Anti-administration">
  <option value="Unknown">
  <option value="Adams">
  <option value="Jackson">
  <option value="Jackson Republican">
  <option value="Crawford Republican">
  <option value="Whig">
  <option value="Jacksonian Republican">
  <option value="Jacksonian">
  <option value="Anti-Jacksonian">
  <option value="Adams Democrat">
  <option value="Nullifier">
  <option value="Anti Mason">
  <option value="Anti Masonic">
  <option value="Anti Jacksonian">
  <option value="Democrat">
  <option value="Anti Jackson">
  <option value="Union Democrat">
  <option value="Conservative">
  <option value="Ind. Democrat">
  <option value="Independent">
  <option value="Law and Order">
  <option value="American">
  <option value="Liberty">
  <option value="Free Soil">
  <option value="Ind. Republican-Democrat">
  <option value="Ind. Whig">
  <option value="Unionist">
  <option value="States Rights">
  <option value="Anti-Lecompton Democrat">
  <option value="Constitutional Unionist">
  <option value="Independent Democrat">
  <option value="Unconditional Unionist">
  <option value="Conservative Republican">
  <option value="Ind. Republican">
  <option value="Liberal Republican">
  <option value="National Greenbacker">
  <option value="Readjuster Democrat">
  <option value="Readjuster">
  <option value="Union">
  <option value="Union Labor">
  <option value="Populist">
  <option value="Silver Republican">
  <option value="Free Silver">
  <option value="Silver">
  <option value="Democratic and Union Labor">
  <option value="Progressive Republican">
  <option value="Progressive">
  <option value="Prohibitionist">
  <option value="Socialist">
  <option value="Farmer-Labor">
  <option value="American Labor">
  <option value="Nonpartisan">
  <option value="Coalitionist">
  <option value="Popular Democrat">
  <option value="Liberal">
  <option value="New Progressive">
  <option value="Republican-Conservative">
  <option value="Democrat-Liberal">
  <option value="AL">
  <option value="Libertarian">
  </datalist>
  ```
  And then adding `list="party"` to the input element in the filter form.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1890#issuecomment-1314825019
issue_url: https://api.github.com/repos/simonw/datasette/issues/1890
node_id: IC_kwDOBm6k_c5OXqM7
created_at: 2022-11-15T06:13:36Z
updated_at: 2022-11-15T06:13:36Z
author_association: OWNER
body:
  This could start out as a purely JavaScript enhancement for pages that already figured out the available values through faceting, like you suggested.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1890#issuecomment-1314829751
issue_url: https://api.github.com/repos/simonw/datasette/issues/1890
node_id: IC_kwDOBm6k_c5OXrW3
created_at: 2022-11-15T06:20:50Z
updated_at: 2022-11-15T06:20:50Z
author_association: OWNER
body:
  This finds the right links on the page:

      document.querySelectorAll('.facet-results [data-column] li:not(.facet-truncated) a')

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1890#issuecomment-1314833881
issue_url: https://api.github.com/repos/simonw/datasette/issues/1890
node_id: IC_kwDOBm6k_c5OXsXZ
created_at: 2022-11-15T06:27:21Z
updated_at: 2022-11-15T06:27:21Z
author_association: OWNER
body:
  Here's a prototype:
  ```javascript
  function createDataLists() {
    var facetResults = document.querySelectorAll(".facet-results [data-column]");
    Array.from(facetResults).forEach(function (facetResult) {
      // Use link text from all links in the facet result
      var linkTexts = Array.from(
        facetResult.querySelectorAll("li:not(.facet-truncated) a")
      ).map(function (link) {
        return link.textContent;
      });
      // Create a datalist element
      var datalist = document.createElement("datalist");
      datalist.id = "datalist-" + facetResult.dataset.column;
      // Create an option element for each link text
      linkTexts.forEach(function (linkText) {
        var option = document.createElement("option");
        option.value = linkText;
        datalist.appendChild(option);
      });
      // Add the datalist to the facet result
      facetResult.appendChild(datalist);
    });
  }
  createDataLists();

  // When any select with name=_filter_column changes, update the datalist
  document.body.addEventListener("change", function (event) {
    if (event.target.name === "_filter_column") {
      event.target
        .closest(".filter-row")
        .querySelector(".filter-value")
        .setAttribute("list", "datalist-" + event.target.value);
    }
  });
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1862#issuecomment-1314845667
issue_url: https://api.github.com/repos/simonw/datasette/issues/1862
node_id: IC_kwDOBm6k_c5OXvPj
created_at: 2022-11-15T06:42:09Z
updated_at: 2022-11-15T06:42:32Z
author_association: OWNER
body:
  I implemented this as part of `/db/-/create`.

  https://docs.datasette.io/en/1.0-dev/json_api.html#creating-a-table-from-example-data

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1890#issuecomment-1314849867
issue_url: https://api.github.com/repos/simonw/datasette/issues/1890
node_id: IC_kwDOBm6k_c5OXwRL
created_at: 2022-11-15T06:47:51Z
updated_at: 2022-11-15T06:47:51Z
author_association: OWNER
body:
  Demo now live here: https://congress-legislators.datasettes.com/legislators/legislator_terms?_facet=party - select `party` and start typing.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1890#issuecomment-1314850524
issue_url: https://api.github.com/repos/simonw/datasette/issues/1890
node_id: IC_kwDOBm6k_c5OXwbc
created_at: 2022-11-15T06:48:37Z
updated_at: 2022-11-15T06:48:37Z
author_association: OWNER
body:
  Spotted a bug with this on https://latest.datasette.io/fixtures/facetable?_facet=_city_id - the `_city_id` column is a foreign key, so you need to type `1` or `2` - but the autocomplete list shows the full text names for the cities.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1890#issuecomment-1314856513
issue_url: https://api.github.com/repos/simonw/datasette/issues/1890
node_id: IC_kwDOBm6k_c5OXx5B
created_at: 2022-11-15T06:56:29Z
updated_at: 2022-11-15T06:56:29Z
author_association: OWNER
body:
  Looks like I can fix that like so:
  ```html
  <datalist id="datalist-_city_id">
      <option label="San Francisco" value="1"></option>
      <option label="Los Angeles" value="2"></option>
      <option label="Detroit" value="3"></option>
      <option label="Memnonia" value="4"></option>
  </datalist>
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1890#issuecomment-1314891228
issue_url: https://api.github.com/repos/simonw/datasette/issues/1890
node_id: IC_kwDOBm6k_c5OX6Xc
created_at: 2022-11-15T07:23:01Z
updated_at: 2022-11-15T07:23:01Z
author_association: OWNER
body:
  Annoying: Mobile Safari doesn't seem to support separate labels and values. I should probably disable this feature on that browser, at least for foreign key facets (for the moment).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1892#issuecomment-1315805498
issue_url: https://api.github.com/repos/simonw/datasette/issues/1892
node_id: IC_kwDOBm6k_c5ObZk6
created_at: 2022-11-15T20:05:30Z
updated_at: 2022-11-15T20:05:30Z
author_association: OWNER
body:
  One slight concern: https://latest.datasette.io/ will increasingly reflect a version that isn't the most recent production release.

  I might setup https://stable.datasette.io/ as a demo instance of the most recent non-alpha release to compensate for that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1863#issuecomment-1315808062
issue_url: https://api.github.com/repos/simonw/datasette/issues/1863
node_id: IC_kwDOBm6k_c5ObaM-
created_at: 2022-11-15T20:08:04Z
updated_at: 2022-11-15T20:08:04Z
author_association: OWNER
body:
  The initial design I'm going to implement will look like this:

  ```
  POST /db/table/1/-/update
  Authorization: Bearer xxx
  Content-Type: application/json
  ```
  ```json
  {
    "update": {
      "name": "New name"
    }
  }
  ```
  Any fields that are not yet columns will return an error.

  Should it enforce types, in as much as an integer column should have a JSON integer passed to it, or should it allow strings containing valid integers?

  I'm going to allow strings, mainly as a workaround for the fact that JavaScript integers have a maximum size.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1863#issuecomment-1315809260
issue_url: https://api.github.com/repos/simonw/datasette/issues/1863
node_id: IC_kwDOBm6k_c5Obafs
created_at: 2022-11-15T20:09:11Z
updated_at: 2022-11-15T20:09:11Z
author_association: OWNER
body:
  I'm going to use the error format I've been experimenting with here:
  - #1875

  ```json
  {
      "type": "https://example.net/validation-error",
      "title": "Your request is not valid.",
      "errors": [
          {
              "detail": "must be a positive integer",
              "pointer": "#/age"
          },
          {
              "detail": "must be 'green', 'red' or 'blue'",
              "pointer": "#/profile/color"
          }
      ]
  }
  ```
  I'm not quite ready to commit to a `type` URL though, so I'll leave that to be solved later should I fully embrace that RFC.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1863#issuecomment-1315809867
issue_url: https://api.github.com/repos/simonw/datasette/issues/1863
node_id: IC_kwDOBm6k_c5ObapL
created_at: 2022-11-15T20:09:44Z
updated_at: 2022-11-15T20:09:44Z
author_association: OWNER
body:
  I'm also not going to implement `"alter": true` yet (which would add any missing columns based on the update) - I'll hold that off for a later feature.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1863#issuecomment-1315812212
issue_url: https://api.github.com/repos/simonw/datasette/issues/1863
node_id: IC_kwDOBm6k_c5ObbN0
created_at: 2022-11-15T20:12:02Z
updated_at: 2022-11-15T20:12:02Z
author_association: OWNER
body:
  If the update succeeds it will return `{"ok": true}`.

  For consistency with `/db/table/-/insert` you can pass `"return": true` and it will return a `"row"` key with the now-updated full row.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1892#issuecomment-1315814786
issue_url: https://api.github.com/repos/simonw/datasette/issues/1892
node_id: IC_kwDOBm6k_c5Obb2C
created_at: 2022-11-15T20:14:38Z
updated_at: 2022-11-15T20:14:38Z
author_association: NONE
body:
  I have no particular point, I just want to say being around for the 1.0 release of Datasette seems historic and legendary to witness.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1315869040
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5ObpFw
created_at: 2022-11-15T21:11:42Z
updated_at: 2022-11-15T21:11:42Z
author_association: CONTRIBUTOR
body:
  extraKeys is done - Shift+Enter is added in the helper function, and it appears that the Tab behavior now defaults to what the `Tab: false` setting was doing (allowing it to escape to the form)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1315869946
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5ObpT6
created_at: 2022-11-15T21:12:38Z
updated_at: 2022-11-15T21:12:38Z
author_association: CONTRIBUTOR
body:
  https://github.com/Sphinxxxx/cm-resize isn't compatible with 6. There's a suggestion to try using CSS resize in https://discuss.codemirror.net/t/resizing-codemirror-6/3265/2

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1316041828
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5OcTRk
created_at: 2022-11-15T23:51:35Z
updated_at: 2022-11-15T23:51:35Z
author_association: CONTRIBUTOR
body:
  I experimented with autocompleting the actual schema in https://github.com/bgrins/datasette/commit/8431c98850c7a552dbcde2a4dd0c3dc942a97d25, but it would need some work (current problems with it listed in the commit message there)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1316135244
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5OcqFM
created_at: 2022-11-16T01:21:41Z
updated_at: 2022-11-16T01:21:41Z
author_association: OWNER
body:
  I just deployed a demo instance like this (using the commit hash from this PR):

  ```bash
  datasette publish vercel fixtures.db \
    --branch 544f7025900b78f63c34b9985522271ba5fd9c0f \
    --project datasette-pr-1893 \
    --scope  datasette \
    --about 'PR 1893' \
    --about_url https://github.com/simonw/datasette/pull/1893
  ```
  Here's the result: https://datasette-pr-1893.vercel.app/fixtures

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1316137982
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5Ocqv-
created_at: 2022-11-16T01:23:47Z
updated_at: 2022-11-16T01:23:47Z
author_association: OWNER
body:
  Autocomplete here looks promising (I've wanted that to work for years!), but it does currently show a whole bunch of suggestions which aren't part of the SQLite SQL dialect:

  ![autocomplete](https://user-images.githubusercontent.com/9599/202060211-51ec9f45-bc52-459a-a729-27fc2faadff9.gif)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1316141764
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5OcrrE
created_at: 2022-11-16T01:26:59Z
updated_at: 2022-11-16T01:26:59Z
author_association: OWNER
body:
  Resizing works great for me - and the page automatically sizes the editor to fit an existing query, e.g. on https://datasette-pr-1893.vercel.app/fixtures?sql=select+id%2C+content%2C+content2%0D%0A++from+primary_key_multiple_columns_explicit_label%0D%0A++order+by+id%0D%0A++limit+101

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1316227073
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5OdAgB
created_at: 2022-11-16T02:54:22Z
updated_at: 2022-11-16T02:54:32Z
author_association: OWNER
body:
  If you can get a version of this working with table and column autocompletion just using a static JavaScript object in the source code with the right tables and columns, I'm happy to take on the work of turning that static object into something that Datasette includes in the page itself with all of the correct values.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1316231560
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5OdBmI
created_at: 2022-11-16T02:59:00Z
updated_at: 2022-11-16T02:59:00Z
author_association: OWNER
body:
  The resize handle doesn't appear on Mobile Safari on iPhone - I don't think that particularly matters though.

  The textarea does get a weird border around it when focused on iPhone though.

  Focused:

  ![BF34E8FB-E35C-4CAB-9BFB-8EEF7E29B16C_1_201_a](https://user-images.githubusercontent.com/9599/202072748-c85bab94-a039-4ed6-8185-3cac25c78ed3.jpeg)

  Not focused:

  ![31A5CF38-D540-4A1A-8A7D-E29453D150F4_1_201_a](https://user-images.githubusercontent.com/9599/202072744-d9f0ea62-13b7-46ff-afe1-6d88d7fb8b53.jpeg)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1890#issuecomment-1316233532
issue_url: https://api.github.com/repos/simonw/datasette/issues/1890
node_id: IC_kwDOBm6k_c5OdCE8
created_at: 2022-11-16T03:00:58Z
updated_at: 2022-11-16T03:00:58Z
author_association: OWNER
body:
  Oops, introduced a test failure: 

  ```
      def test_table_html_foreign_key_facets(app_client):
          response = app_client.get(
              "/fixtures/foreign_key_references?_facet=foreign_key_with_blank_label"
          )
          assert response.status == 200
  >       assert (
              '<li><a href="http://localhost/fixtures/foreign_key_references?_facet=foreign_key_with_blank_label&amp;foreign_key_with_blank_label=3">'
              "-</a> 1</li>"
          ) in response.text
  E       assert '<li><a href="http://localhost/fixtures/foreign_key_references?_facet=foreign_key_with_blank_label&amp;foreign_key_with_blank_label=3">-</a> 1</li>' in '<!DOCTYPE html>\n<html>\n<head>\n    <title>fixtures: foreign_key_references: 2 rows</title>\n    <link rel="styleshe.../script>\n\n\n<!-- Templates considered: table-fixtures-foreign_key_references.html, *table.html -->\n</body>\n</html>'
  E        +  where '<!DOCTYPE html>\n<html>\n<head>\n    <title>fixtures: foreign_key_references: 2 rows</title>\n    <link rel="styleshe.../script>\n\n\n<!-- Templates considered: table-fixtures-foreign_key_references.html, *table.html -->\n</body>\n</html>' = <datasette.utils.testing.TestResponse object at 0x7fd1b0080640>.text
  ```
  Need to fix this test:

  https://github.com/simonw/datasette/blob/eac028d3f77aa5473a5fcf59240635a1bca80f7d/tests/test_table_html.py#L616-L624

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1890#issuecomment-1316240839
issue_url: https://api.github.com/repos/simonw/datasette/issues/1890
node_id: IC_kwDOBm6k_c5OdD3H
created_at: 2022-11-16T03:09:11Z
updated_at: 2022-11-16T03:09:11Z
author_association: OWNER
body:
  Here's a polyfill for `<datalist>`: https://github.com/mfranzke/datalist-polyfill

  It shouldn't be necessary now that Safari has shipped support (apparently added in https://developer.apple.com/documentation/safari-release-notes/safari-12_1-release-notes#3130314 Safari 12.1 in March 2019).

  But it does look like Safari doesn't support differing `label` and `value` attributes, though documentation about this is hard to come by.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1890#issuecomment-1316242752
issue_url: https://api.github.com/repos/simonw/datasette/issues/1890
node_id: IC_kwDOBm6k_c5OdEVA
created_at: 2022-11-16T03:10:52Z
updated_at: 2022-11-16T03:12:47Z
author_association: OWNER
body:
  https://bugs.webkit.org/show_bug.cgi?id=201768 - " Datalist option's label not used" - marked as RESOLVED FIXED on March 31st 2020.

  The commit: https://trac.webkit.org/changeset/259330/webkit

  And here's the test mirrored on GitHub: https://cs.github.com/qtwebkit/webkit-mirror/blob/cc3fcd0b4bad1f7cf77c26e34aa01d16618d6d5e/LayoutTests/fast/forms/datalist/datalist-option-labels.html?q=datalist-option-labels.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1316243602
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5OdEiS
created_at: 2022-11-16T03:11:46Z
updated_at: 2022-11-16T03:11:46Z
author_association: CONTRIBUTOR
body:
  Was just reviewing the SQL options and there's an [upperCaseKeywords](https://github.com/codemirror/lang-sql#user-content-sqlconfig.uppercasekeywords) if we'd rather have SELECT vs select. Datasette seems to prefer lowercase so probably best to keep it as-is

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1316253186
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5OdG4C
created_at: 2022-11-16T03:16:36Z
updated_at: 2022-11-16T03:16:36Z
author_association: OWNER
body:
  Yeah I haven't written this down anywhere but Datasette definitely has an undocumented preference for lower-case SQL.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1316256386
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5OdHqC
created_at: 2022-11-16T03:18:06Z
updated_at: 2022-11-16T03:18:06Z
author_association: CONTRIBUTOR
body:
  > If you can get a version of this working with table and column autocompletion just using a static JavaScript object in the source code with the right tables and columns, I'm happy to take on the work of turning that static object into something that Datasette includes in the page itself with all of the correct values.

  This version "sort of" works when on the main database page where the template passes the relevant data https://github.com/bgrins/datasette/commit/8431c98850c7a552dbcde2a4dd0c3dc942a97d25 by doing this and passing that into the `schema` object:

  ```
    let TABLES_DATA = [];
    {% if tables is defined %}   
    TABLES_DATA = {{ tables | tojson(indent=2) }};
    {% endif %}

    // Turn into an object, shaped like https://github.com/codemirror/lang-sql/blob/ebf115fffdbe07f91465ccbd82868c587f8182bc/test/test-complete.ts#L27.
    const TABLES_SCHEMA = Object.fromEntries(
      new Map(
        TABLES_DATA.map((table) => {
          return [table.name, table.columns];
        })
      ).entries()
    );
  ```

  But there are a number of papercuts with it - it's not escaping table names with spaces (likely be fixable from the data being passed into the view) but mainly it doesn't seem to autocomplete columns. I think it might only want to do it when you first type the table name from my read of https://github.com/codemirror/lang-sql/blob/ebf115fffdbe07f91465ccbd82868c587f8182bc/test/test-complete.ts#L37. It's possible I'm just passing something wrong, but it may end up being something that needs feature work upstream.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1890#issuecomment-1316262169
issue_url: https://api.github.com/repos/simonw/datasette/issues/1890
node_id: IC_kwDOBm6k_c5OdJEZ
created_at: 2022-11-16T03:22:40Z
updated_at: 2022-11-16T03:22:40Z
author_association: OWNER
body:
  Actually this works as it should in desktop Safari:

  ![autocomplete-safari](https://user-images.githubusercontent.com/9599/202075764-fbc4b4c8-c92f-4f69-81fd-84002de5aea7.gif)

  I'm going to just put up with the weird behaviour in Mobile Safari.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1886#issuecomment-1316289392
issue_url: https://api.github.com/repos/simonw/datasette/issues/1886
node_id: IC_kwDOBm6k_c5OdPtw
created_at: 2022-11-16T03:54:17Z
updated_at: 2022-11-16T03:58:56Z
author_association: NONE
body:
  Happy Birthday Datasette!

  Thanks Simon!!

  I use datasette on everything most notably [my flickr metadata SQLite DB](https://www.dropbox.com/s/6j10e2vohp2j5kf/roland2019-2020.db?dl=0) to make art.

  Datasette lite on my 2019 flickr metadata is super helpful too:
  https://lite.datasette.io/?csv=https%3A%2F%2Fraw.githubusercontent.com%2Frtanglao%2Frt-flickr-sqlite-csv%2Fmain%2F2019-roland-flickr-metadata.csv

  Even better datasette lite on all firefox support questions from 2021: https://lite.datasette.io/?url=https%3A%2F%2Fraw.githubusercontent.com%2Frtanglao%2Frt-kits-api3%2Fmain%2FYEARLY_CSV_FILES%2F2021-firefox-sumo-questions.db

  Thanks again Simon! So great! What a gift to the world!!!!!!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1316293353
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5OdQrp
created_at: 2022-11-16T03:59:03Z
updated_at: 2022-11-16T03:59:03Z
author_association: OWNER
body:
  Deployed a fresh copy:
  ```
  datasette publish vercel fixtures.db \
    --branch b7b2942b13f9ea09cfa9f8c73e2869b9bd2349ae \
    --project datasette-pr-1893 \
    --about 'PR 1893' \
    --about_url https://github.com/simonw/datasette/pull/1893 \
    --scope  datasette
  ```
  https://datasette-pr-1893.vercel.app/fixtures

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1316294156
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5OdQ4M
created_at: 2022-11-16T04:00:12Z
updated_at: 2022-11-16T04:00:12Z
author_association: OWNER
body:
  Have you ever seen CodeMirror correctly auto-completing columns? I'm not entirely sure I believe that the feature works anywhere else.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1316297347
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5OdRqD
created_at: 2022-11-16T04:05:12Z
updated_at: 2022-11-16T23:27:45Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1893?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Base: **92.55**% // Head: **92.55**% // No change to project coverage :thumbsup:
  > Coverage data is based on head [(`f254be4`)](https://codecov.io/gh/simonw/datasette/pull/1893?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`6f610e1`)](https://codecov.io/gh/simonw/datasette/commit/6f610e1d94b7b8ec605b5b7fcb01537f6adf9c5b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > Patch has no changes to coverable lines.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main    #1893   +/-   ##
  =======================================
    Coverage   92.55%   92.55%           
  =======================================
    Files          35       35           
    Lines        4432     4432           
  =======================================
    Hits         4102     4102           
    Misses        330      330           
  ```



  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  </details>

  [:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1893?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1316318961
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5OdW7x
created_at: 2022-11-16T04:27:51Z
updated_at: 2022-11-16T04:27:51Z
author_association: CONTRIBUTOR
body:
  > The resize handle doesn't appear on Mobile Safari on iPhone - I don't think that particularly matters though.
  > 
  > The textarea does get a weird border around it when focused on iPhone though.

  The default focus styles appear to be

  ```
  .c1.cm-editor.cm-focused {
    outline: 1px dotted #212121;
  }
  ```

  Which I also see on desktop. Would be nice to changed to whatever the default UA textarea styles are to blend in better but I wouldn't recommend removing it entirely - just to keep the visual indication that the element is focused. Maybe followup material to have a theming pass

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1316320521
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5OdXUJ
created_at: 2022-11-16T04:29:23Z
updated_at: 2022-11-16T04:29:23Z
author_association: CONTRIBUTOR
body:
  <img width="276" alt="Screenshot 2022-11-15 at 8 27 17 PM" src="https://user-images.githubusercontent.com/95570/202083682-dab271f7-cb7b-44dd-8266-70b1eba265ee.png">

  UI issue I see on the autocomplete popup with overlapping icon & text. Screenshot's from Firefox, it seems even a little more pronounced on Safari

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1316339035
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5Odb1b
created_at: 2022-11-16T04:47:11Z
updated_at: 2022-11-16T04:47:11Z
author_association: CONTRIBUTOR
body:
  > Have you ever seen CodeMirror correctly auto-completing columns? I'm not entirely sure I believe that the feature works anywhere else.

  I was thinking of the BigQuery console, like 

  <img width="516" alt="Screenshot 2022-11-15 at 8 31 10 PM" src="https://user-images.githubusercontent.com/95570/202084210-4d23d916-6862-4fd2-8f41-087d6355921a.png">

  But they must be doing something pretty custom & appears to be using Monaco anyway. I suspect some kind of lower level autocomplete integration could make this work, but if the table completion is a good-enough starting point I think it's not too hard. The main issue is that we don't pass the relevant table data down to QueryView.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1316340865
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5OdcSB
created_at: 2022-11-16T04:49:30Z
updated_at: 2022-11-16T04:49:43Z
author_association: OWNER
body:
  > The main issue is that we don't pass the relevant table data down to QueryView.

  If you can come up with a static example JSON data structure example that does the right thing, I'm happy to refactor QueryView to make that available to the template - or even have a separate `fetch()` that grabs just the data needed for the autocomplete as a separate hit when the page loads (whichever has better performance implications). I'm working a fair amount in the view classes at the moment so adding this to that work would make sense.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1316387382
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5Odno2
created_at: 2022-11-16T05:33:55Z
updated_at: 2022-11-16T05:33:55Z
author_association: CONTRIBUTOR
body:
  I added a commit to make our own dialect at https://github.com/simonw/datasette/pull/1893/commits/e273fc8ed5341bdf0b622e722d761bd2acc30a90. Pulled in the full list of keywords from https://www.sqlite.org/lang_keywords.html but haven't gone through and pruned it to only include common select keywords. @simonw you'll have better knowledge than me on that - do you want to take a first shot at narrowing that down to the set that people will be using in the editor?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1316400688
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5Odq4w
created_at: 2022-11-16T05:49:31Z
updated_at: 2022-11-16T05:49:31Z
author_association: OWNER
body:
  OK, I decided to play around with GPT-3 for this. https://beta.openai.com/playground

  I prompted:

  ```
  For each keyword say it it can be used in a SQLite select statement: and as asc between by case cast 
  cross current current_date current_time current_timestamp desc detach distinct each else end escape 
  except exclude exclusive exists explain fail filter first following for foreign from full generated glob 
  group groups having if ignore immediate in index indexed initially inner insert instead intersect into 
  is isnull join key last left like limit match materialized natural no not nothing notnull null nulls of 
  offset on or order others outer over partition plan pragma preceding primary query raise range recursive 
  references regexp reindex release rename replace restrict returning right rollback row rows savepoint 
  select set table temp temporary then ties to transaction trigger unbounded union unique update using 
  vacuum values view virtual when where window with without

  - and: yes
  - as: yes
  - asc: yes
  - between: yes
  - by: yes
  - case: yes
  - cast: yes
  - current: yes
  - current_date: yes
  - current_time: yes
  - current_timestamp: yes
  - desc: yes
  - detach: no
  - distinct: yes
  - each: yes
  - else: yes
  - end: yes
  - escape: yes
  - except: no
  - exclude: yes
  - exclusive: no
  ```
  Having given it these examples I ran it a few times:
  ```
  - exists: yes
  - explain: no
  - fail: no
  - filter: yes
  - first: yes
  - following: no
  - for: yes
  - foreign: no
  - from: yes
  - full: no
  - generated: no
  - glob: yes
  - group: yes
  - groups: no
  - having: yes
  - if: yes
  - ignore: no
  - immediate: no
  - in: yes
  - index: yes
  - indexed: no
  - initially: no
  - inner: no
  - insert: no
  - instead: no
  - intersect: no
  - into: yes
  - is: yes
  - isnull: yes
  - join: yes
  - key: no
  - last: yes
  - left: yes
  - like: yes
  - limit: yes
  - match: no
  - materialized: no
  -natural: yes
  - no: no
  - not: no
  - nothing: no
  - notnull: yes
  - null: yes
  - nulls: yes
  - of: no
  - offset: yes
  - on: yes
  - or: yes
  - order: yes
  - others: yes
  - outer: no
  - over: no
  - partition: no
  - pragma: no
  - preceding: no
  - primary: no
  - query: no
  - raise: no
  - range: no
  - recursive: no
  - references: no
  - regexp: yes
  - reindex: no
  - release: no
  - rename: no
  - replace: no
  - restrict: no
  - returning: yes
  - right: yes
  - rollback: no
  - row: yes
  - rows: no
  - savepoint: no
  - select: yes
  - set: no
  - table: yes
  - temp: yes
  - temporary: yes
  - then: yes
  - ties: no
  - to: yes
  - transaction: no
  - trigger: no
  - unbounded: no
  - union: yes
  - unique: yes
  - update: no
  - using: yes
  - vacuum: no
  - values: yes
  - view: yes
  - virtual: no
  - when: yes
  - where: yes
  - window: no
  - with: yes
  - without: no
  ```
  I added this:
  ```
  A space separated list of all of the keywords for which the answer was yes: 
  ```
  It replied:
  ```
  and as asc between by case cast current current_date current_time current_timestamp desc distinct each else escape except exists explain filter first for from full generated group having if in index inner intersect into isnull join last left like limit not null or order outer over pragma primary query raise range regexp right rollback row select set table temp temporary then to union unique using values view virtual when where
  ```
  Then I tried doing this:
  ```
  Produce a short summary of how each of those are used:

  - and:
  ```
  This kind of worked too! It ended up defining every keyword, not just the ones for which the "is it used in select" was "yes", but it gave me this list:

  > - and: Used to combine multiple conditions in a WHERE clause
  > - as: Rename a column or table in the output of a query
  > - asc: Sort results in ascending order
  > - between: Used in a WHERE clause to specify a range
  > - by: Specifies how to sort the results of a query
  > - case: Used to create different outputs based on conditions
  > - cast: Converts a value from one data type to another
  > - current: Returns the current date, time, or timestamp
  > - current_date: Returns the current date
  > - current_time: Returns the current time
  > - current_timestamp: Returns the current timestamp
  > - desc: Sort results in descending order
  > - distinct: Selects only unique values
  > - each: Used with aggregate functions
  > - else: Used with the CASE statement
  > - escape: Used to escape special characters in a string
  > - except: Returns all rows from the first query that are not in the second query
  > - exists: Used with subqueries
  > - explain: Analyzes a query and returns information about it
  > - fail: Used with the ON CONFLICT clause
  > - filter: Used to specify conditions for aggregate functions
  > - first: Used to specify conditions for aggregate functions
  > - following: Used with window functions
  > - for: Used to specify conditions for aggregate functions
  > - foreign: Creates a foreign key
  > - from: Specifies the table(s) to select data from
  > - full: Used with outer joins
  > - generated: Used with generated columns
  > - glob: Used to match a string with a pattern
  > - group: Groups together rows that have the same values
  > - groups: Used with aggregate functions
  > - having: Specifies conditions for groups
  > - if: Used with the ON CONFLICT clause
  > - ignore: Used with the ON CONFLICT clause
  > - immediate: Used with the ON CONFLICT clause
  > - in: Used to check if a value is in a list
  > - index: Creates an index on a column or columns
  > - indexed: Used with the ON CONFLICT clause
  > - initially: Used with the ON CONFLICT clause
  > - inner: Creates an inner join
  > - insert: Inserts new data into a table
  > - instead: Used with triggers
  > - intersect: Returns all rows that are in both queries
  > - into: Specifies the table to insert data into
  > - is: Used to compare values
  > - isnull: Checks if a value is null
  > - join: Joins two or more tables together
  > - key: Creates a primary key or unique key
  > - last: Used to specify conditions for aggregate functions
  > - left: Creates a left join
  > - like: Checks if a value matches a pattern
  > - limit: Specifies the maximum number of rows to return
  > - not: Negates a condition
  > - null: Specifies a null value
  > - nulls: Used with the ORDER BY clause
  > - of: Used with window functions
  > - offset: Specifies the number of rows to skip
  > - on: Specifies conditions for a join
  > - or: Checks if either condition is true
  > - order: Specifies how to sort the results of a query
  > - others: Used with the CASE statement
  > - outer: Creates an outer join
  > - over: Used with window functions
  > - partition: Used with window functions
  > - pragma: Used to query or change various internal settings
  > - preceding: Used with window functions
  > - primary: Creates a primary key
  > - query: Used with the ON CONFLICT clause
  > - raise: Used with the ON CONFLICT clause
  > - range: Used with window functions
  > - recursive: Used with common table expressions
  > - references: Creates a foreign key
  > - regexp: Checks if a value matches a regular expression
  > - reindex: Recreates an index
  > - release: Used with savepoints
  > - rename: Changes the name of a table or column
  > - replace: Inserts new data into a table or updates existing data
  > - restrict: Used with savepoints
  > - returning: Returns data from inserted or updated rows
  > - right: Creates a right join
  > - rollback: Rolls back a transaction
  > - row: Used with aggregate functions
  > - select: Selects data from a table
  > - set: Changes the value of a column
  > - table: Specifies the table(s) to select data from
  > - temp: Creates a temporary table
  > - temporary: Creates a temporary table
  > - then: Used with the CASE statement
  > - ties: Used with the ORDER BY clause
  > - to: Used with common table expressions
  > - transaction: Starts a transaction
  > - trigger: Creates a trigger
  > - unbounded: Used with window functions
  > - union: Returns all rows from both queries
  > - unique: Creates a unique constraint
  > - update: Updates existing data in a table
  > - using: Specifies the table(s) to join
  > - vacuum: Optimizes a database
  > - values: Specifies the values to insert into a table
  > - view: Creates a view
  > - virtual: Creates a virtual table
  > - when: Used with the CASE statement
  > - where: Specifies conditions for a query
  > - window: Used with window functions
  > - with: Specifies common table expressions used in a query
  > - without: Used with indexes

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1316401895
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5OdrLn
created_at: 2022-11-16T05:50:40Z
updated_at: 2022-11-16T05:50:40Z
author_association: OWNER
body:
  So I think our dialect (at least to start with) should be:

  ```
    keywords:
      "and as asc between by case cast current current_date current_time current_timestamp desc distinct each else escape except exists explain filter first for from full generated group having if in index inner intersect into isnull join last left like limit not null or order outer over pragma primary query raise range regexp right rollback row select set table temp temporary then to union unique using values view virtual when where",
    // https://www.sqlite.org/datatype3.html
    types: "null integer real text blob",
    builtin:
      ""
  ```
  I left `builtin` blank here because I don't think we need any of those things at all.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1316412234
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5OdttK
created_at: 2022-11-16T06:00:39Z
updated_at: 2022-11-16T06:01:36Z
author_association: OWNER
body:
  Should note though that this is a classic example of GPT-3 making stuff up in places.

  > current: Returns the current date, time, or timestamp

  `select current` throws an error for me: https://latest.datasette.io/_memory?sql=select+current

  `select current_date, current_time, current_timestamp` works though: https://latest.datasette.io/_memory?sql=select+current_date%2C+current_time%2C+current_timestamp

  So let's drop `current` from the list. I'm OK with it though, I think it's likely good enough for the first attempt at this.

  We should drop `temp` and `temporary` too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/512#issuecomment-1316437748
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/512
node_id: IC_kwDOCGYnMM5Odz70
created_at: 2022-11-16T06:24:31Z
updated_at: 2022-11-16T06:24:31Z
author_association: OWNER
body:
  ```
  sqlite-utils % pipx run no_implicit_optional .
  Calculating full-repo metadata...
  Executing codemod...
  11.43s 98% complete, 0.24s estimated for 5 files to go...
  ```
  Then:
  ```
  Finished codemodding 239 files!
   - Transformed 239 files successfully.
   - Skipped 0 files.
   - Failed to codemod 0 files.
   - 0 warnings were generated.
  ```
  Here's the diff:
  ```diff
   diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py
  index a06f4b7..e819d17 100644
  --- a/sqlite_utils/db.py
  +++ b/sqlite_utils/db.py
  @@ -297,12 +297,12 @@ class Database:
   
       def __init__(
           self,
  -        filename_or_conn: Union[str, pathlib.Path, sqlite3.Connection] = None,
  +        filename_or_conn: Optional[Union[str, pathlib.Path, sqlite3.Connection]] = None,
           memory: bool = False,
  -        memory_name: str = None,
  +        memory_name: Optional[str] = None,
           recreate: bool = False,
           recursive_triggers: bool = True,
  -        tracer: Callable = None,
  +        tracer: Optional[Callable] = None,
           use_counts_table: bool = False,
       ):
           assert (filename_or_conn is not None and (not memory and not memory_name)) or (
  @@ -341,7 +341,7 @@ class Database:
           self.conn.close()
   
       @contextlib.contextmanager
  -    def tracer(self, tracer: Callable = None):
  +    def tracer(self, tracer: Optional[Callable] = None):
           """
           Context manager to temporarily set a tracer function - all executed SQL queries will
           be passed to this.
  @@ -378,7 +378,7 @@ class Database:
   
       def register_function(
           self,
  -        fn: Callable = None,
  +        fn: Optional[Callable] = None,
           deterministic: bool = False,
           replace: bool = False,
           name: Optional[str] = None,
  @@ -879,7 +879,7 @@ class Database:
           pk: Optional[Any] = None,
           foreign_keys: Optional[ForeignKeysType] = None,
           column_order: Optional[List[str]] = None,
  -        not_null: Iterable[str] = None,
  +        not_null: Optional[Iterable[str]] = None,
           defaults: Optional[Dict[str, Any]] = None,
           hash_id: Optional[str] = None,
           hash_id_columns: Optional[Iterable[str]] = None,
  @@ -1129,7 +1129,7 @@ class Database:
               sql += " [{}]".format(name)
           self.execute(sql)
   
  -    def init_spatialite(self, path: str = None) -> bool:
  +    def init_spatialite(self, path: Optional[str] = None) -> bool:
           """
           The ``init_spatialite`` method will load and initialize the SpatiaLite extension.
           The ``path`` argument should be an absolute path to the compiled extension, which
  @@ -1182,7 +1182,7 @@ class Queryable:
   
       def count_where(
           self,
  -        where: str = None,
  +        where: Optional[str] = None,
           where_args: Optional[Union[Iterable, dict]] = None,
       ) -> int:
           """
  @@ -1213,12 +1213,12 @@ class Queryable:
   
       def rows_where(
           self,
  -        where: str = None,
  +        where: Optional[str] = None,
           where_args: Optional[Union[Iterable, dict]] = None,
  -        order_by: str = None,
  +        order_by: Optional[str] = None,
           select: str = "*",
  -        limit: int = None,
  -        offset: int = None,
  +        limit: Optional[int] = None,
  +        offset: Optional[int] = None,
       ) -> Generator[dict, None, None]:
           """
           Iterate over every row in this table or view that matches the specified where clause.
  @@ -1251,11 +1251,11 @@ class Queryable:
   
       def pks_and_rows_where(
           self,
  -        where: str = None,
  +        where: Optional[str] = None,
           where_args: Optional[Union[Iterable, dict]] = None,
  -        order_by: str = None,
  -        limit: int = None,
  -        offset: int = None,
  +        order_by: Optional[str] = None,
  +        limit: Optional[int] = None,
  +        offset: Optional[int] = None,
       ) -> Generator[Tuple[Any, Dict], None, None]:
           """
           Like ``.rows_where()`` but returns ``(pk, row)`` pairs - ``pk`` can be a single value or tuple.
  @@ -1345,7 +1345,7 @@ class Table(Queryable):
           pk: Optional[Any] = None,
           foreign_keys: Optional[ForeignKeysType] = None,
           column_order: Optional[List[str]] = None,
  -        not_null: Iterable[str] = None,
  +        not_null: Optional[Iterable[str]] = None,
           defaults: Optional[Dict[str, Any]] = None,
           batch_size: int = 100,
           hash_id: Optional[str] = None,
  @@ -1545,7 +1545,7 @@ class Table(Queryable):
           pk: Optional[Any] = None,
           foreign_keys: Optional[ForeignKeysType] = None,
           column_order: Optional[List[str]] = None,
  -        not_null: Iterable[str] = None,
  +        not_null: Optional[Iterable[str]] = None,
           defaults: Optional[Dict[str, Any]] = None,
           hash_id: Optional[str] = None,
           hash_id_columns: Optional[Iterable[str]] = None,
  @@ -2464,7 +2464,7 @@ class Table(Queryable):
           columns: Optional[Iterable[str]] = None,
           limit: Optional[int] = None,
           offset: Optional[int] = None,
  -        where: str = None,
  +        where: Optional[str] = None,
           where_args: Optional[Union[Iterable, dict]] = None,
           quote: bool = False,
       ) -> Generator[dict, None, None]:
  @@ -2527,7 +2527,7 @@ class Table(Queryable):
   
       def delete_where(
           self,
  -        where: str = None,
  +        where: Optional[str] = None,
           where_args: Optional[Union[Iterable, dict]] = None,
           analyze: bool = False,
       ) -> "Table":
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/512#issuecomment-1316447182
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/512
node_id: IC_kwDOCGYnMM5Od2PO
created_at: 2022-11-16T06:32:31Z
updated_at: 2022-11-16T06:32:31Z
author_association: OWNER
body:
  Test failed again: https://github.com/simonw/sqlite-utils/actions/runs/3476950474/jobs/5812663096

  `E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/universe/s/spatialite/libsqlite3-mod-spatialite_4.3.0a-6build1_amd64.deb  Unable to connect to azure.archive.ubuntu.com:http:`

  That looks like an intermittent error. I'll try running it again in the morning.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1317281292
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5OhB4M
created_at: 2022-11-16T16:19:16Z
updated_at: 2022-11-16T16:19:16Z
author_association: CONTRIBUTOR
body:
  Ha, nice idea! Updating the dialect with that list.

  I'm thinking of also adding `count` to the list since that's a common thing people would want to autocomplete. I notice BQ console highlights `count` in the same manner as other keywords like `select` as well.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1317314064
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5OhJ4Q
created_at: 2022-11-16T16:36:46Z
updated_at: 2022-11-16T16:36:46Z
author_association: CONTRIBUTOR
body:
  With

  ```patch
  diff --git a/datasette/templates/_codemirror_foot.html b/datasette/templates/_codemirror_foot.html
  index ed709b3..74fe18e 100644
  --- a/datasette/templates/_codemirror_foot.html
  +++ b/datasette/templates/_codemirror_foot.html
  @@ -7,7 +7,11 @@
         sqlFormat.hidden = false;
       }
       if (sqlInput) {
  -      var editor = (window.editor = cm.editorFromTextArea(sqlInput));
  +      var editor = (window.editor = cm.editorFromTextArea(sqlInput, {
  +        schema: {
  +          compound_three_primary_keys: ["pk1", "pk2", "pk3", "content"],
  +        },
  +      }));
  ```

  we get table autocompletion and column completion if you name the table in the query (see screencast). I do see bugs with escaped table names like `"'123_starts_with_digits'": ["col1", "col2"]` or `"[123_starts_with_digits]": ["col1", "col2"]` where it doesn't seem to pick up the column names though. I think it needs some further testing and debugging. 

  https://user-images.githubusercontent.com/95570/202238521-e613b4e2-ba92-4418-9068-fc022edaee93.mp4

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1317326406
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5OhM5G
created_at: 2022-11-16T16:45:09Z
updated_at: 2022-11-16T16:45:09Z
author_association: CONTRIBUTOR
body:
  For escaped table names it looks like we could pass a Completion object (https://codemirror.net/docs/ref/#autocomplete) instead of a string which would allow the non escaped name to be a label and then the escaped name to actually complete in the editor, which might help with some of the funkiness I was seeing w/ completion

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1317329157
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5OhNkF
created_at: 2022-11-16T16:46:52Z
updated_at: 2022-11-16T16:46:52Z
author_association: CONTRIBUTOR
body:
  > <img alt="Screenshot 2022-11-15 at 8 27 17 PM" width="276" src="https://user-images.githubusercontent.com/95570/202083682-dab271f7-cb7b-44dd-8266-70b1eba265ee.png">
  > 
  > UI issue I see on the autocomplete popup with overlapping icon & text. Screenshot's from Firefox, it seems even a little more pronounced on Safari

  I checked and if I empty out app.css the bug goes away, so there's some kind of inheritance issue there. It's hard to debug bc the autocomplete popup goes away on blur (i.e. when trying to inspect it in devtools), but at least it's narrowed down a bit.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1880#issuecomment-1317420812
issue_url: https://api.github.com/repos/simonw/datasette/issues/1880
node_id: IC_kwDOBm6k_c5Ohj8M
created_at: 2022-11-16T17:50:29Z
updated_at: 2022-11-16T17:50:29Z
author_association: NONE
body:
  I appreciate your response @simonw - thanks!

  I'll clarify what we need further - let's imagine we have 2000 SQLLite databases (for 2000 tenants), but we only want to run _one_ datasette instance for each of those tenants to query/use datasette against their _own_ database only. This means the "connection" between datasette and the SQLLite database would be dynamic, based on the tenantID that's required on an incoming request.  

  Is there any specific config or other considerations in this use case, to minimize memory use on a single, efficient VM and serve queries to all these tenants? 

  cc @muadham

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1317449610
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5Ohq-K
created_at: 2022-11-16T18:14:28Z
updated_at: 2022-11-16T18:14:28Z
author_association: OWNER
body:
  > I'm thinking of also adding `count` to the list since that's a common thing people would want to autocomplete. I notice BQ console highlights `count` in the same manner as other keywords like `select` as well.

  Huh, yeah we should definitely have `count` - surprised it's not on the list on https://www.sqlite.org/lang_keywords.html which is why we didn't get it from the GPT-3 generated schema.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1317452541
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5Ohrr9
created_at: 2022-11-16T18:15:52Z
updated_at: 2022-11-16T18:15:52Z
author_association: OWNER
body:
  Deployed latest copy with:
  ```
  datasette publish vercel fixtures.db \
    --project datasette-pr-1893 \
    --about 'PR 1893' \
    --about_url https://github.com/simonw/datasette/pull/1893 \
    --scope datasette \
    --branch eccb1c6c781d69d8ec3c542ef65c78a4a0927a7c
  ```
  https://datasette-pr-1893.vercel.app/fixtures

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1317456909
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5OhswN
created_at: 2022-11-16T18:17:39Z
updated_at: 2022-11-16T18:17:39Z
author_association: OWNER
body:
  Tiny feature request (since you're in this code already) - I keep hitting Command+Enter on my macOS keyboard to submit the query, but the correct shortcut is Shift+Enter. Would be great if both worked!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1317465874
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5Ohu8S
created_at: 2022-11-16T18:21:17Z
updated_at: 2022-11-16T18:21:54Z
author_association: OWNER
body:
  I was worrying about the server-side overhead of collecting together all of the tables and column names for databases that might have hundreds of tables... but then I remember that I built the `_internal` table precisely for this kind of thing - so gathering all of that data should still only be a single SQL query against an in-memory database.

  https://latest.datasette.io/login-as-root and then visit this page for an example query: https://latest.datasette.io/_internal?sql=select%0D%0A++database_name%2C%0D%0A++table_name%2C%0D%0A++json_group_array%28name%29%0D%0Afrom%0D%0A++columns%0D%0Awhere%0D%0A++database_name+%21%3D+%27_internal%27%0D%0Agroup+by%0D%0A++database_name%2C%0D%0A++table_name

  ```sql
  select
    database_name,
    table_name,
    json_group_array(name)
  from
    columns
  where
    database_name != '_internal'
  group by
    database_name,
    table_name
  ```

  database_name | table_name | json_group_array(name)
  -- | -- | --
  extra_database | searchable | ["pk","text1","text2"]
  extra_database | searchable_fts | ["__langid","content","docid","searchable_fts","text1","text2"]
  extra_database | searchable_fts_content | ["c0text1","c1text2","c2content","docid"]
  extra_database | searchable_fts_segdir | ["end_block","idx","leaves_end_block","level","root","start_block"]
  extra_database | searchable_fts_segments | ["block","blockid"]
  fixtures | 123_starts_with_digits | ["content"]
  fixtures | Table With Space In Name | ["content","pk"]
  fixtures | attraction_characteristic | ["name","pk"]
  fixtures | binary_data | ["data"]
  fixtures | complex_foreign_keys | ["f1","f2","f3","pk"]
  fixtures | compound_primary_key | ["content","pk1","pk2"]

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1317475720
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5OhxWI
created_at: 2022-11-16T18:25:16Z
updated_at: 2022-11-16T18:25:16Z
author_association: OWNER
body:
  Here's a query that returns the exact JSON we need to pass to the schema:

  https://latest.datasette.io/_internal?sql=with+inner+as+%28%0D%0A++select%0D%0A++++table_name%2C%0D%0A++++json_group_array%28name%29+as+table_columns%0D%0A++from%0D%0A++++columns%0D%0A++where%0D%0A++++database_name+%3D+%3Adatabase%0D%0A++group+by%0D%0A++++table_name%0D%0A%29%0D%0Aselect%0D%0A++json_group_object%28table_name%2C+table_columns%29%0D%0Afrom%0D%0A++inner&database=fixtures

  ```sql
  with inner as (
    select
      table_name,
      json_group_array(name) as table_columns
    from
      columns
    where
      database_name = :database
    group by
      table_name
  )
  select
    json_group_object(table_name, table_columns)
  from
    inner
  ```
  Returns (after pretty-printing):
  ```json
  {
      "123_starts_with_digits": [
          "content"
      ],
      "Table With Space In Name": [
          "content",
          "pk"
      ],
      "attraction_characteristic": [
          "name",
          "pk"
      ],
      "binary_data": [
          "data"
      ],
      "complex_foreign_keys": [
          "f1",
          "f2",
          "f3",
          "pk"
      ],
      "compound_primary_key": [
          "content",
          "pk1",
          "pk2"
      ],
      "compound_three_primary_keys": [
          "content",
          "pk1",
          "pk2",
          "pk3"
      ],
      "custom_foreign_key_label": [
          "foreign_key_with_custom_label",
          "pk"
      ],
      "facet_cities": [
          "id",
          "name"
      ],
      "facetable": [
          "_city_id",
          "_neighborhood",
          "complex_array",
          "created",
          "distinct_some_null",
          "n",
          "on_earth",
          "pk",
          "planet_int",
          "state",
          "tags"
      ],
      "foreign_key_references": [
          "foreign_key_compound_pk1",
          "foreign_key_compound_pk2",
          "foreign_key_with_blank_label",
          "foreign_key_with_label",
          "foreign_key_with_no_label",
          "pk"
      ],
      "infinity": [
          "value"
      ],
      "no_primary_key": [
          "a",
          "b",
          "c",
          "content"
      ],
      "primary_key_multiple_columns": [
          "content",
          "content2",
          "id"
      ],
      "primary_key_multiple_columns_explicit_label": [
          "content",
          "content2",
          "id"
      ],
      "roadside_attraction_characteristics": [
          "attraction_id",
          "characteristic_id"
      ],
      "roadside_attractions": [
          "address",
          "latitude",
          "longitude",
          "name",
          "pk",
          "url"
      ],
      "searchable": [
          "name with . and spaces",
          "pk",
          "text1",
          "text2"
      ],
      "searchable_fts": [
          "__langid",
          "docid",
          "name with . and spaces",
          "searchable_fts",
          "text1",
          "text2"
      ],
      "searchable_fts_docsize": [
          "docid",
          "size"
      ],
      "searchable_fts_segdir": [
          "end_block",
          "idx",
          "leaves_end_block",
          "level",
          "root",
          "start_block"
      ],
      "searchable_fts_segments": [
          "block",
          "blockid"
      ],
      "searchable_fts_stat": [
          "id",
          "value"
      ],
      "searchable_tags": [
          "searchable_id",
          "tag"
      ],
      "select": [
          "and",
          "group",
          "having",
          "json"
      ],
      "simple_primary_key": [
          "content",
          "id"
      ],
      "sortable": [
          "content",
          "pk1",
          "pk2",
          "sortable",
          "sortable_with_nulls",
          "sortable_with_nulls_2",
          "text"
      ],
      "table/with/slashes.csv": [
          "content",
          "pk"
      ],
      "tags": [
          "tag"
      ],
      "units": [
          "distance",
          "frequency",
          "pk"
      ]
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1317520304
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5Oh8Ow
created_at: 2022-11-16T18:58:43Z
updated_at: 2022-11-16T18:58:43Z
author_association: CONTRIBUTOR
body:
  Nice. And is it possible to include another field which is an escaped table name (only when necessary) - i.e. `[123_starts_with_digits]`. Or is that easy enough to derive on the client? I'm thinking we'd map those to Completion objects so that CM would show the non escaped text but complete to escaped.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1317522323
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5Oh8uT
created_at: 2022-11-16T18:59:49Z
updated_at: 2022-11-16T18:59:49Z
author_association: CONTRIBUTOR
body:
  Or I guess you could return only the escaped table name and then we could derive the unescaped from the client side (removing the outer `[]` when present)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1317681193
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5Oijgp
created_at: 2022-11-16T21:19:13Z
updated_at: 2022-11-16T21:19:13Z
author_association: CONTRIBUTOR
body:
  Alright, added Cmd+Enter to submit (Ctrl+Enter on Windows as well bc of using Meta-Enter on codemirror). We can make that MacOS only by changing the combo to Cmd+Enter specifically but I think it's probably fine to have both.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1317715580
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5Oir58
created_at: 2022-11-16T21:49:51Z
updated_at: 2022-11-16T21:49:51Z
author_association: CONTRIBUTOR
body:
  I think the table completion still has some quirks to work out. Something like

  ```
         schema: {
           "[123_starts_with_digits]": ["content"],
         }
  ```

  Seems to work alright, although it will append it after any other numbers you've started typing - so you end up with `select * from 12[123_starts_with_digits]` if you typed "12" to get the completion to appear. This might just be an issue with numeric names, I haven't tested it in a lot of detail.

  You can do 

  ```
            searchable: [
              {
                label: "name with . and spaces",
                apply: "[name with . and spaces]",
              },
              "pk",
              "text1",
              "text2",
            ],
  ```

  Which is pretty neat and will show the non-escaped string but complete to the escaped one. You can't easily do that with the table names themselves (you can pass a `tables` array like so https://github.com/codemirror/lang-sql/blob/ebf115fffdbe07f91465ccbd82868c587f8182bc/src/sql.ts#L121 but it will overwrite the columns from the schema ).

  It's buggy enough (bad output for these unusual table names) that I'd suggest that work gets moved into a follow up to the upgrade to 6. That would give space to sort out how to deliver that to the view directly, figure out where name escaping should happen, and have overall testing to uncover bugs and fix papercuts before enabling it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1317744563
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5Oiy-z
created_at: 2022-11-16T22:16:03Z
updated_at: 2022-11-16T22:16:03Z
author_association: OWNER
body:
  Honestly I'm not too bothered if table names with weird characters don't work correctly here - I care about those in the Datasette `fixtures.db` database because Datasette aims to support ANY valid SQLite database, so I need stuff in the test suite that includes weird edge cases like this. But I would hope very few people actually create tables with spaces in their names, so it's not a huge concern to me if autocompletion doesn't work properly for those.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1863#issuecomment-1317755263
issue_url: https://api.github.com/repos/simonw/datasette/issues/1863
node_id: IC_kwDOBm6k_c5Oi1l_
created_at: 2022-11-16T22:24:59Z
updated_at: 2022-11-16T22:24:59Z
author_association: OWNER
body:
  In trying to write this I realize that there's a lot of duplicated code with delete row, specifically around resolving the incoming URL into a row (or a database or a table).

  Since this is so common, I think it's worth extracting the logic out first.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1896#issuecomment-1317757112
issue_url: https://api.github.com/repos/simonw/datasette/issues/1896
node_id: IC_kwDOBm6k_c5Oi2C4
created_at: 2022-11-16T22:26:52Z
updated_at: 2022-11-16T22:33:13Z
author_association: OWNER
body:
  Some ideas from walking the dog:

  Challenge: standard URL routing of request to database/table/row

  Standardize on the named components of the URL patterns - `database`, `table`, `pks`

  Async function that takes the request and the Datasette instance and returns a Resolved instance with:
  ```
  .level - database or table or row (better name?)
  .database - the name of the database
  .db - the database object
  .table - the name of the table (or view)
  .is_view perhaps?
  .pk_values if it's a row
  ```
  Should this attempt to resolve names queries too?
  ```
  .where_sql - the where fragment you use
  .where_params - accompanying dictionary

  await datasette.resolve_request(request)
  ```
  Or even better three methods: 
  ```python
  datasette.resolve_database(request)
  datasette.resolve_table(request)
  datasette.resolve_row(request)
  ```
  These can be typed correctly

  Methods raise `NotFound` if not found

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1896#issuecomment-1317757949
issue_url: https://api.github.com/repos/simonw/datasette/issues/1896
node_id: IC_kwDOBm6k_c5Oi2P9
created_at: 2022-11-16T22:27:47Z
updated_at: 2022-11-18T21:48:29Z
author_association: OWNER
body:
  Open question: should `resolve_table()` know how to identify named canned queries too?

  I think not, at least for the moment. Feels a bit too specialist to expose in a documented API.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1317789308
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5Oi958
created_at: 2022-11-16T22:59:57Z
updated_at: 2022-11-16T22:59:57Z
author_association: CONTRIBUTOR
body:
  I can push up a commit that uses the static fixtures schema for testing, but given that the query used to generate it is authed we would still need some work to make that work on live data, right? Ideally it could come down to db and query views directly to avoid waiting on an extra xhr and managing that state change.On Nov 16, 2022, at 2:16 PM, Simon Willison ***@***.***> wrote:﻿
  Honestly I'm not too bothered if table names with weird characters don't work correctly here - I care about those in the Datasette fixtures.db database because Datasette aims to support ANY valid SQLite database, so I need stuff in the test suite that includes weird edge cases like this. But I would hope very few people actually create tables with spaces in their names, so it's not a huge concern to me if autocompletion doesn't work properly for those.

  —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1317797044
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5Oi_y0
created_at: 2022-11-16T23:08:34Z
updated_at: 2022-11-16T23:08:34Z
author_association: OWNER
body:
  > I can push up a commit that uses the static fixtures schema for testing, but given that the query used to generate it is authed we would still need some work to make that work on live data, right?

  Yeah, push that up. I'm happy to wire in the query right after we land this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1317805482
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5OjB2q
created_at: 2022-11-16T23:18:17Z
updated_at: 2022-11-16T23:18:17Z
author_association: CONTRIBUTOR
body:
  Alright with https://github.com/simonw/datasette/pull/1893/commits/f254be4b38936e95e7a7f25866e7c6b0520db96f we should be getting autocomplete on fixture data. Give that a test and see what you think

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1317834838
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5OjJBW
created_at: 2022-11-16T23:50:58Z
updated_at: 2022-11-16T23:50:58Z
author_association: CONTRIBUTOR
body:
  Should we empty out the fixture schema to avoid fixture autocomplete showing up on live databases in the interim, or are you planning to tackle #1897 shortly?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1893#issuecomment-1317837416
issue_url: https://api.github.com/repos/simonw/datasette/issues/1893
node_id: IC_kwDOBm6k_c5OjJpo
created_at: 2022-11-16T23:54:02Z
updated_at: 2022-11-16T23:54:02Z
author_association: OWNER
body:
  I'm going to tackle #1897 in the next few minutes.

  Tests failed due to Prettier check, just pushed a fix so it would ignore `.bundle.js` too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1897#issuecomment-1317838892
issue_url: https://api.github.com/repos/simonw/datasette/issues/1897
node_id: IC_kwDOBm6k_c5OjKAs
created_at: 2022-11-16T23:55:42Z
updated_at: 2022-11-16T23:55:42Z
author_association: OWNER
body:
  Here's where the schema is hard-coded at the moment: https://github.com/simonw/datasette/blob/00e233d7a7f6443cb95fb5227c23580c48551cad/datasette/templates/_codemirror_foot.html#L2-L7

  I figured out how to extract that data from the `_internal` table in this comment: https://github.com/simonw/datasette/pull/1893#issuecomment-1317475720

  Although that used JSON functions which may (in a real edge-case) not be available in the version of SQLite that Datasette is running on, so probably going to use a regular SQL query and then assemble the JSON separately.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1897#issuecomment-1317839781
issue_url: https://api.github.com/repos/simonw/datasette/issues/1897
node_id: IC_kwDOBm6k_c5OjKOl
created_at: 2022-11-16T23:56:47Z
updated_at: 2022-11-16T23:56:47Z
author_association: OWNER
body:
  I'm going to call this `table_columns` in the template context (because `schema` might mean `CREATE TABLE ...`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1897#issuecomment-1317840727
issue_url: https://api.github.com/repos/simonw/datasette/issues/1897
node_id: IC_kwDOBm6k_c5OjKdX
created_at: 2022-11-16T23:57:52Z
updated_at: 2022-11-16T23:57:52Z
author_association: OWNER
body:
  In terms of permissions: if you have `execute-sql` permission for a database then it's OK for you to see the table columns for that database.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1898#issuecomment-1317870550
issue_url: https://api.github.com/repos/simonw/datasette/issues/1898
node_id: IC_kwDOBm6k_c5OjRvW
created_at: 2022-11-17T00:27:55Z
updated_at: 2022-11-17T00:27:55Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1898?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Base: **92.55**% // Head: **92.55**% // No change to project coverage :thumbsup:
  > Coverage data is based on head [(`5eb4ea4`)](https://codecov.io/gh/simonw/datasette/pull/1898?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`00e233d`)](https://codecov.io/gh/simonw/datasette/commit/00e233d7a7f6443cb95fb5227c23580c48551cad?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > Patch has no changes to coverable lines.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main    #1898   +/-   ##
  =======================================
    Coverage   92.55%   92.55%           
  =======================================
    Files          35       35           
    Lines        4432     4432           
  =======================================
    Hits         4102     4102           
    Misses        330      330           
  ```



  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  </details>

  [:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1898?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1899#issuecomment-1317873458
issue_url: https://api.github.com/repos/simonw/datasette/issues/1899
node_id: IC_kwDOBm6k_c5OjScy
created_at: 2022-11-17T00:31:07Z
updated_at: 2022-11-17T00:31:07Z
author_association: CONTRIBUTOR
body:
  This is one way to fix it

  ```patch
  r.html
  diff --git a/datasette/static/cm-editor-6.0.1.js b/datasette/static/cm-editor-6.0.1.js
  index c1fd2ab..68cf398 100644
  --- a/datasette/static/cm-editor-6.0.1.js
  +++ b/datasette/static/cm-editor-6.0.1.js
  @@ -22,7 +22,14 @@ export function editorFromTextArea(textarea, conf = {}) {
     // https://github.com/codemirror/lang-sql#user-content-sqlconfig.tables
     let view = new EditorView({
       doc: textarea.value,
  +
       extensions: [
  +      EditorView.theme({
  +        ".cm-content": {
  +          // Height on cm-content ensures the editor is focusable by clicking beyond the height of the text
  +          minHeight: "70px",
  +        },
  +      }),
         keymap.of([
           {
             key: "Shift-Enter",
  diff --git a/datasette/templates/_codemirror.html b/datasette/templates/_codemirror.html
  index dea4710..c4629ae 100644
  --- a/datasette/templates/_codemirror.html
  +++ b/datasette/templates/_codemirror.html
  @@ -4,7 +4,6 @@
     .cm-editor {
       resize: both;
       overflow: hidden;
  -    min-height: 70px;
       width: 80%;
       border: 1px solid #ddd;
     }
  ```

  I don't love it but it seems to work for the default case. You can still retrigger the bug by resizing the editor to be > 70px high.

  The other approach would be to listen for a click on that empty region and move focus to the editor, or something

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/510#issuecomment-1318777114
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/510
node_id: IC_kwDOCGYnMM5OmvEa
created_at: 2022-11-17T15:09:47Z
updated_at: 2022-11-17T15:09:47Z
author_association: CONTRIBUTOR
body:
  why close? is the only problem that the _config table that incorrectly says 4 for fts5? if so, that's still something that should be fixed

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1897#issuecomment-1319401843
issue_url: https://api.github.com/repos/simonw/datasette/issues/1897
node_id: IC_kwDOBm6k_c5OpHlz
created_at: 2022-11-18T00:42:03Z
updated_at: 2022-11-18T00:42:23Z
author_association: OWNER
body:
  This function works even if the SQLite JSON functions are not available:

  ```python
  async def _table_columns(datasette, database_name):
      internal = datasette.get_database("_internal")
      result = await internal.execute(
          "select table_name, name from columns where database_name = ?",
          [database_name],
      )
      table_columns = {}
      for row in result.rows:
          table_columns.setdefault(row["table_name"], []).append(row["name"])
      return table_columns
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1901#issuecomment-1319482791
issue_url: https://api.github.com/repos/simonw/datasette/issues/1901
node_id: IC_kwDOBm6k_c5OpbWn
created_at: 2022-11-18T03:01:36Z
updated_at: 2022-11-18T03:01:36Z
author_association: OWNER
body:
  Good catch. Looks like that bug was introduced by this change: https://github.com/simonw/datasette/commit/1a5e5f2aa951e5bd731067a49819efba68fbe8ef

  From:

  - https://github.com/simonw/datasette/issues/1831

  The search all plugin includes this code which interacts poorly with that refactor:

  https://github.com/simonw/datasette-search-all/blob/847b55c368a285e4567627029624d7872ee75cac/datasette_search_all/templates/search_all.html#L31-L36

  ```html+jinja
  {% block nav %}
      <p class="crumbs">
          <a href="{{ urls.instance() }}">home</a>
      </p>
      {{ super() }}
  {% endblock %}
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1901#issuecomment-1319483555
issue_url: https://api.github.com/repos/simonw/datasette/issues/1901
node_id: IC_kwDOBm6k_c5Opbij
created_at: 2022-11-18T03:02:35Z
updated_at: 2022-11-18T03:02:35Z
author_association: OWNER
body:
  Looks like this issue could affect a bunch of other plugins too: https://cs.github.com/?scopeName=All+repos&scope=&q=%3Cp+class%3D%22crumbs%22%3E+user%3Asimonw

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1901#issuecomment-1319493475
issue_url: https://api.github.com/repos/simonw/datasette/issues/1901
node_id: IC_kwDOBm6k_c5Opd9j
created_at: 2022-11-18T03:19:25Z
updated_at: 2022-11-18T07:03:03Z
author_association: OWNER
body:
  Other plugins this looks like it will affect:

  - [x] `datasette-ripgrep` https://github.com/simonw/datasette-ripgrep/blob/03446464420130368582022eeb5944993f64ec8f/datasette_ripgrep/templates/ripgrep.html#L37-L42
  - [x] `datasette-socrata` https://github.com/simonw/datasette-socrata/blob/32fb256a461bf0e790eca10bdc7dd9d96c20f7c4/datasette_socrata/templates/datasette_socrata_error.html#L5-L10
  - [x] `datasette-configure-fts` https://github.com/simonw/datasette-configure-fts/blob/eca742e5d4b9190fc22d68bc0a406c575e6d09a0/datasette_configure_fts/templates/configure_fts_database.html#L9-L14
  - [x] `datasette-edit-templates` https://github.com/simonw/datasette-edit-templates/blob/f772aff4a2a4080c949746668a8ec6302dbeb0d9/datasette_edit_templates/templates/edit_template.html#L17-L23
  - [x] `datasette-copyable` https://github.com/simonw/datasette-copyable/blob/204d5c912a8d48c49155c67fba7339d4bb26ab9a/datasette_copyable/templates/copyable.html#L36-L43
  - [x] `datasette-public` https://github.com/simonw/datasette-public/blob/32b6a0ba53bd5714b6b41eddd8705b213c105efc/datasette_public/templates/public_table_change_privacy.html#L5-L11
  - [x] `datasette-import-table` https://github.com/simonw/datasettecloud-datasette/blob/37d0fe525c6649c1aec3d1ee8bc35a684570e87f/templates/import_data.html#L5-L10
  - [x] `datasette-edit-schema` (three places)
  - [x] `datasette-indieauth` https://github.com/simonw/datasette-indieauth/blob/a08ce67ddad6098b1240adbeff37d040e4df53b1/datasette_indieauth/templates/indieauth.html#L5-L10

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1901#issuecomment-1319525520
issue_url: https://api.github.com/repos/simonw/datasette/issues/1901
node_id: IC_kwDOBm6k_c5OplyQ
created_at: 2022-11-18T04:21:14Z
updated_at: 2022-11-18T07:22:37Z
author_association: OWNER
body:
  This search helps too: [https://ripgrep.datasette.io/-/ripgrep?pattern=%7B%25+block+nav&literal=on&ignore=on&glob=%21datasette%2F**](https://ripgrep.datasette.io/-/ripgrep?pattern=%7B%25+block+nav&literal=on&ignore=on&glob=%21datasette%2F**)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1901#issuecomment-1319528359
issue_url: https://api.github.com/repos/simonw/datasette/issues/1901
node_id: IC_kwDOBm6k_c5Opmen
created_at: 2022-11-18T04:27:00Z
updated_at: 2022-11-18T04:27:00Z
author_association: OWNER
body:
  Also `datasette-indieauth` https://github.com/simonw/datasette-indieauth/blob/a08ce67ddad6098b1240adbeff37d040e4df53b1/datasette_indieauth/templates/indieauth.html#L5-L10

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1897#issuecomment-1319533445
issue_url: https://api.github.com/repos/simonw/datasette/issues/1897
node_id: IC_kwDOBm6k_c5OpnuF
created_at: 2022-11-18T04:38:03Z
updated_at: 2022-11-18T04:38:03Z
author_association: CONTRIBUTOR
body:
  Are you tracking the change to send the JSON over to the frontend separately or was that part of this? Something like this is probably pretty close https://github.com/bgrins/datasette/commit/8431c98850c7a552dbcde2a4dd0c3dc942a97d25#diff-0c93232bfd5477eeac96382e52769108b41433d960d5277ffcccf2f464e60abdR9

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1897#issuecomment-1319570586
issue_url: https://api.github.com/repos/simonw/datasette/issues/1897
node_id: IC_kwDOBm6k_c5Opwya
created_at: 2022-11-18T05:33:20Z
updated_at: 2022-11-18T05:33:20Z
author_association: OWNER
body:
  One of the big changes still left to do for Datasette 1.0 is to unify the JSON representation with the context psssed to the templates (via an `?_extra=` mechanism to add extra context needed by the HTML templates), because a goal for 1.0 is for the template context to be a documented API contract such that custom templates won't break with future releases.

  As such I expect to do quite a bit of refactoring and cleanup on how the template context works later on.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1900#issuecomment-1319582239
issue_url: https://api.github.com/repos/simonw/datasette/issues/1900
node_id: IC_kwDOBm6k_c5Opzof
created_at: 2022-11-18T05:55:38Z
updated_at: 2022-11-18T05:55:38Z
author_association: OWNER
body:
  Trying this out locally with this 69MB SpatiaLite file I happened to have lying around (from testing `shapefile-to-sqlite` a while ago): https://static.simonwillison.net/static/2022/nps-spatialite.db
  ```
  % datasette package nps-spatialite.db --spatialite
  ...
   => [2/6] COPY . /app                                                                                                                    0.4s
   => [3/6] WORKDIR /app                                                                                                                   0.0s
   => [4/6] RUN apt-get update &&     apt-get install -y python3-dev gcc libsqlite3-mod-spatialite &&     rm -rf /var/lib/apt/lists/*     29.6s
   => [5/6] RUN pip install -U datasette                                                                                                  12.0s
   => [6/6] RUN datasette inspect nps-spatialite.db --inspect-file inspect-data.json                                                       2.6s 
   => exporting to image                                                                                                                   3.0s 
   => => exporting layers                                                                                                                  3.0s 
   => => writing image sha256:4dfef1c373c5c057ef7ac22344f834d522acef24313a1b25d2eba9e500066b8f                                             0.0s 
  ```
  And then:

      docker run -p 8001:8001 4dfef1c373c5

  This worked fine for me. I ran `datasette package` using Datasette 0.63.1.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1900#issuecomment-1319583281
issue_url: https://api.github.com/repos/simonw/datasette/issues/1900
node_id: IC_kwDOBm6k_c5Opz4x
created_at: 2022-11-18T05:57:44Z
updated_at: 2022-11-18T05:57:44Z
author_association: OWNER
body:
  Did you use the `--spatialite` option?

  I just tried this:

      datasette package nps-spatialite.db

  It built the image OK (I didn't see the error you reported), but running the container failed with an error:

  ```
  /tmp % docker run -p 8001:8001 7298e8e6bbfb
  Usage: datasette serve [OPTIONS] [FILES]...
  Try 'datasette serve --help' for help.

  Error: It looks like you're trying to load a SpatiaLite database without first loading the SpatiaLite module.

  Read more: https://docs.datasette.io/en/stable/spatialite.html
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1900#issuecomment-1319583703
issue_url: https://api.github.com/repos/simonw/datasette/issues/1900
node_id: IC_kwDOBm6k_c5Opz_X
created_at: 2022-11-18T05:58:31Z
updated_at: 2022-11-18T05:58:31Z
author_association: OWNER
body:
  Could you provide full steps to reproduce plus a SpatiaLite database file that triggered this for you? I'm not able to recreate the problem.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1899#issuecomment-1319584553
issue_url: https://api.github.com/repos/simonw/datasette/issues/1899
node_id: IC_kwDOBm6k_c5Op0Mp
created_at: 2022-11-18T06:00:10Z
updated_at: 2022-11-18T06:01:50Z
author_association: OWNER
body:
  I can't actually remember where that `min-height: 70px` came from. I just tried without it and it seems fine - especially since any time you add a newline in the editor it increases its height to fit.

  I ran this in the DevTools console:

  ```javascript
  document.querySelector('.cm-editor').style.minHeight = 'none';
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1901#issuecomment-1319588163
issue_url: https://api.github.com/repos/simonw/datasette/issues/1901
node_id: IC_kwDOBm6k_c5Op1FD
created_at: 2022-11-18T06:05:11Z
updated_at: 2022-11-18T06:05:11Z
author_association: OWNER
body:
  For `datasette-copyable` I want to show breadcrumbs that take database/instance permissions into account, so I'm removing `{% block nav %}` entirely and replacing it with this:

  ```html+jinja
  {% block crumbs %}
  {{ crumbs.nav(request=request, database=database, table=table) }}
  {% endblock %}
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1900#issuecomment-1319596087
issue_url: https://api.github.com/repos/simonw/datasette/issues/1900
node_id: IC_kwDOBm6k_c5Op3A3
created_at: 2022-11-18T06:16:33Z
updated_at: 2022-11-18T06:16:33Z
author_association: NONE
body:
  Interesting! So I tried this locally using your copy of `nps-spatialite.db` and I got the same error. 🤔

  ```
  ❯ datasette package nps-spatialite.db --spatialite
  [+] Building 27.5s (10/10) FINISHED                                                                                                                                                        
   => [internal] load build definition from Dockerfile                                                                                                                                  0.0s
   => => transferring dockerfile: 622B                                                                                                                                                  0.0s
   => [internal] load .dockerignore                                                                                                                                                     0.0s
   => => transferring context: 2B                                                                                                                                                       0.0s
   => [internal] load metadata for docker.io/library/python:3.11.0-slim-bullseye                                                                                                        0.9s
   => [internal] load build context                                                                                                                                                     2.3s
   => => transferring context: 72.38MB                                                                                                                                                  2.3s
   => CACHED [1/6] FROM docker.io/library/python:3.11.0-slim-bullseye@sha256:1cd45c5dad845af18d71745c017325725dc979571c1bbe625b67e6051533716c                                           0.0s
   => [2/6] COPY . /app                                                                                                                                                                 0.1s
   => [3/6] WORKDIR /app                                                                                                                                                                0.0s
   => [4/6] RUN apt-get update &&     apt-get install -y python3-dev gcc libsqlite3-mod-spatialite &&     rm -rf /var/lib/apt/lists/*                                                  18.5s
   => [5/6] RUN pip install -U datasette                                                                                                                                                4.9s
   => ERROR [6/6] RUN datasette inspect nps-spatialite.db --inspect-file inspect-data.json                                                                                              0.7s 
  ------                                                                                                                                                                                     
   > [6/6] RUN datasette inspect nps-spatialite.db --inspect-file inspect-data.json:                                                                                                         
  #10 0.681 Traceback (most recent call last):                                                                                                                                               
  #10 0.681   File "/usr/local/bin/datasette", line 8, in <module>                                                                                                                           
  #10 0.681     sys.exit(cli())                                                                                                                                                              
  #10 0.681              ^^^^^
  #10 0.681   File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1130, in __call__
  #10 0.682     return self.main(*args, **kwargs)
  #10 0.682            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  #10 0.682   File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1055, in main
  #10 0.682     rv = self.invoke(ctx)
  #10 0.682          ^^^^^^^^^^^^^^^^
  #10 0.682   File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1657, in invoke
  #10 0.682     return _process_result(sub_ctx.command.invoke(sub_ctx))
  #10 0.682                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  #10 0.682   File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1404, in invoke
  #10 0.682     return ctx.invoke(self.callback, **ctx.params)
  #10 0.682            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  #10 0.682   File "/usr/local/lib/python3.11/site-packages/click/core.py", line 760, in invoke
  #10 0.682     return __callback(*args, **kwargs)
  #10 0.682            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  #10 0.683   File "/usr/local/lib/python3.11/site-packages/datasette/cli.py", line 164, in inspect
  #10 0.683     inspect_data = loop.run_until_complete(inspect_(files, sqlite_extensions))
  #10 0.683                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  #10 0.683   File "/usr/local/lib/python3.11/asyncio/base_events.py", line 650, in run_until_complete
  #10 0.683     return future.result()
  #10 0.683            ^^^^^^^^^^^^^^^
  #10 0.683   File "/usr/local/lib/python3.11/site-packages/datasette/cli.py", line 179, in inspect_
  #10 0.683     counts = await database.table_counts(limit=3600 * 1000)
  #10 0.683              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  #10 0.683   File "/usr/local/lib/python3.11/site-packages/datasette/database.py", line 304, in table_counts
  #10 0.683     for table in await self.table_names():
  #10 0.683                  ^^^^^^^^^^^^^^^^^^^^^^^^
  #10 0.683   File "/usr/local/lib/python3.11/site-packages/datasette/database.py", line 342, in table_names
  #10 0.683     results = await self.execute(
  #10 0.683               ^^^^^^^^^^^^^^^^^^^
  #10 0.683   File "/usr/local/lib/python3.11/site-packages/datasette/database.py", line 267, in execute
  #10 0.683     results = await self.execute_fn(sql_operation_in_thread)
  #10 0.683               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  #10 0.683   File "/usr/local/lib/python3.11/site-packages/datasette/database.py", line 213, in execute_fn
  #10 0.683     return await asyncio.get_event_loop().run_in_executor(
  #10 0.683            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  #10 0.683   File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
  #10 0.683     result = self.fn(*self.args, **self.kwargs)
  #10 0.683              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  #10 0.683   File "/usr/local/lib/python3.11/site-packages/datasette/database.py", line 209, in in_thread
  #10 0.683     self.ds._prepare_connection(conn, self.name)
  #10 0.683   File "/usr/local/lib/python3.11/site-packages/datasette/app.py", line 593, in _prepare_connection
  #10 0.683     conn.execute("SELECT load_extension(?)", [extension])
  #10 0.683 sqlite3.OperationalError: /usr/lib/x86_64-linux-gnu/mod_spatialite.so.so: cannot open shared object file: No such file or directory
  ------
  executor failed running [/bin/sh -c datasette inspect nps-spatialite.db --inspect-file inspect-data.json]: exit code: 1
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1901#issuecomment-1319627012
issue_url: https://api.github.com/repos/simonw/datasette/issues/1901
node_id: IC_kwDOBm6k_c5Op-kE
created_at: 2022-11-18T07:07:03Z
updated_at: 2022-11-18T07:07:03Z
author_association: OWNER
body:
  Here's the full list of 10 plugin releases for this issue:

  * [datasette-search-all 1.1.1](https://github.com/simonw/datasette-search-all/releases/tag/1.1.1)
  * [datasette-ripgrep 0.7.1](https://github.com/simonw/datasette-ripgrep/releases/tag/0.7.1)
  * [datasette-socrata 0.3.1](https://github.com/simonw/datasette-socrata/releases/tag/0.3.1)
  * [datasette-configure-fts 1.1.1](https://github.com/simonw/datasette-configure-fts/releases/tag/1.1.1)
  * [datasette-edit-templates 0.2](https://github.com/simonw/datasette-edit-templates/releases/tag/0.2)
  * [datasette-copyable 0.3.2](https://github.com/simonw/datasette-copyable/releases/tag/0.3.2)
  * [datasette-public 0.2.1](https://github.com/simonw/datasette-public/releases/tag/0.2.1)
  * [datasette-import-table 0.3.1](https://github.com/simonw/datasette-import-table/releases/tag/0.3.1)
  * [datasette-indieauth 1.2.2](https://github.com/simonw/datasette-indieauth/releases/tag/1.2.2)
  * [datasette-edit-schema 0.5.2](https://github.com/simonw/datasette-edit-schema/releases/tag/0.5.2)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1900#issuecomment-1319629469
issue_url: https://api.github.com/repos/simonw/datasette/issues/1900
node_id: IC_kwDOBm6k_c5Op_Kd
created_at: 2022-11-18T07:10:17Z
updated_at: 2022-11-18T07:10:17Z
author_association: OWNER
body:
  This is so weird! What version of Datasette do you get from `datasette --version` there - and what's your Docker version / operating system version?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1900#issuecomment-1319631421
issue_url: https://api.github.com/repos/simonw/datasette/issues/1900
node_id: IC_kwDOBm6k_c5Op_o9
created_at: 2022-11-18T07:13:00Z
updated_at: 2022-11-18T07:13:00Z
author_association: OWNER
body:
  You get:

  ```
   => [internal] load metadata for docker.io/library/python:3.11.0-slim-bullseye                                                                                                        0.9s
   => [internal] load build context                                                                                                                                                     2.3s
   => => transferring context: 72.38MB                                                                                                                                                  2.3s
   => CACHED [1/6] FROM docker.io/library/python:3.11.0-slim-bullseye@sha256:1cd45c5dad845af18d71745c017325725dc979571c1bbe625b67e6051533716c                                           0.0s
  ```

  I get:

  ```
   => [internal] load metadata for docker.io/library/python:3.11.0-slim-bullseye                                                                                                                         1.0s
   => [internal] load build context                                                                                                                                                                      0.0s
   => => transferring context: 705B                                                                                                                                                                      0.0s
   => CACHED [1/6] FROM docker.io/library/python:3.11.0-slim-bullseye@sha256:1cd45c5dad845af18d71745c017325725dc979571c1bbe625b67e6051533716c                                                            0.0s
  ```
  Both the image name and the hash are _exactly_ the same. So why are you getting an error while mine works OK?

  For my machine:
  ```
  ~ % docker --version
  Docker version 20.10.12, build e91ed57

  ~ % uname -a
  Darwin Simons-MacBook-Pro-2.local 22.1.0 Darwin Kernel Version 22.1.0: Sun Oct  9 20:14:54 PDT 2022; root:xnu-8792.41.9~2/RELEASE_X86_64 x86_64
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1900#issuecomment-1319639462
issue_url: https://api.github.com/repos/simonw/datasette/issues/1900
node_id: IC_kwDOBm6k_c5OqBmm
created_at: 2022-11-18T07:24:19Z
updated_at: 2022-11-18T07:24:19Z
author_association: NONE
body:
  Is it, uh, possible we are on different architectures? 😅 I'm using an Apple M1 Pro.

  I jumped into a bash shell of an unmodified `python:3.11.0-slim-bullseye` container and manually ran `apt-get update` and installed `libsqlite3-mod-spatialite`. I don't end up with with `mod_spatialite.so` in `/usr/lib/x86_64-linux-gnu/` — _mine_ is in `/usr/lib/aarch64-linux-gnu/`.

  [I swapped that directory in here](https://github.com/simonw/datasette/blob/3db37e9a21f774d6c387fd04bf1e4c870554209e/datasette/utils/__init__.py#L407) in a local copy of `datasette` and poof — it worked!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1900#issuecomment-1319641636
issue_url: https://api.github.com/repos/simonw/datasette/issues/1900
node_id: IC_kwDOBm6k_c5OqCIk
created_at: 2022-11-18T07:27:26Z
updated_at: 2022-11-18T07:27:26Z
author_association: NONE
body:
  Can confirm that my `uname -a` returns something different at the end:

  ```
  root:xnu-8792.41.9~2/RELEASE_ARM64_T6000 arm64
  ```

  I'm in `arm64` land, you're in `x86_64`. I am admittedly very fuzzy on how this factors into Docker these days. Honestly thought this was one of the things Docker was suppose to help address. 🤔

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1900#issuecomment-1319664697
issue_url: https://api.github.com/repos/simonw/datasette/issues/1900
node_id: IC_kwDOBm6k_c5OqHw5
created_at: 2022-11-18T07:59:36Z
updated_at: 2022-11-18T08:00:38Z
author_association: NONE
body:
  Okay, my final observations for the night! I've been pushing and pulling the various levers in `utils/__init__.py` to see what makes this work without hard-coding in something for `arm64` and it seems that if I change `/usr/lib/x86_64-linux-gnu/mod_spatialite.so` [here](https://github.com/simonw/datasette/blob/3ecd131e57add427d847b614c920c9624bb2e66b/datasette/utils/__init__.py#L407) to just `mod_spatialite` it's happy.

  Unfortunately cannot audit that for `x86_64`, but maybe that's a solution that'd be cross-arch compatible? It seems like it's the hard-coding of that path that's tripping it up.

  (It was actually [this comment from back in 2018 in an entirely unrelated repo](https://github.com/pelias/docker/pull/28#issuecomment-433168462) that nudged me to try this, ha.)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/510#issuecomment-1320394127
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/510
node_id: IC_kwDOCGYnMM5Os52P
created_at: 2022-11-18T18:37:51Z
updated_at: 2022-11-18T18:37:51Z
author_association: NONE
body:
  I guess it is not incorrect when it says the version is `4`, though it is confusing. Maybe it doesn't even refer to FTS4/FTS5 versions, but something else? In any case, it's not related to sqlite-utils, but SQLite itself.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1863#issuecomment-1320563197
issue_url: https://api.github.com/repos/simonw/datasette/issues/1863
node_id: IC_kwDOBm6k_c5OtjH9
created_at: 2022-11-18T21:47:35Z
updated_at: 2022-11-18T21:48:07Z
author_association: OWNER
body:
  Incomplete implementation of this view:
  ```python
  class RowUpdateView(BaseView):
      name = "row-update"

      def __init__(self, datasette):
          self.ds = datasette

      async def post(self, request):
          database_route = tilde_decode(request.url_vars["database"])
          table = tilde_decode(request.url_vars["table"])
          try:
              db = self.ds.get_database(route=database_route)
          except KeyError:
              return _error(["Database not found: {}".format(database_route)], 404)

          database_name = db.name
          if not await db.table_exists(table):
              return _error(["Table not found: {}".format(table)], 404)

          pk_values = urlsafe_components(request.url_vars["pks"])

          sql, params, pks = await row_sql_params_pks(db, table, pk_values)
          results = await db.execute(sql, params, truncate=True)
          rows = list(results.rows)
          if not rows:
              return _error([f"Record not found: {pk_values}"], 404)

          # Ensure user has permission to update this row
          if not await self.ds.permission_allowed(
              request.actor, "update-row", resource=(database_name, table)
          ):
              return _error(["Permission denied"], 403)

          body = await request.post_body()
          try:
              data = json.loads(body)
          except json.JSONDecodeError as e:
              return _error(["Invalid JSON: {}".format(e)])
          if not isinstance(data, dict):
              return _error(["JSON must be a dictionary"])

          def update_row(conn):
              sqlite_utils.Database(conn)[table].update(pk_values, updates)

          await db.execute_write_fn(update_row)
          result = {"ok": True}
          if data.get("return"):
              result["row"] = {"row-here": "TODO"}
          return Response.json(result, status=200)
  ```
  This is before the refactor in:
  - #1896

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1896#issuecomment-1320588299
issue_url: https://api.github.com/repos/simonw/datasette/issues/1896
node_id: IC_kwDOBm6k_c5OtpQL
created_at: 2022-11-18T22:16:59Z
updated_at: 2022-11-18T22:17:06Z
author_association: OWNER
body:
  Found myself needing an `await db.view_exists()` method for this, similar to the existing `await db.table_exists()` one.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1903#issuecomment-1320614541
issue_url: https://api.github.com/repos/simonw/datasette/issues/1903
node_id: IC_kwDOBm6k_c5OtvqN
created_at: 2022-11-18T22:47:41Z
updated_at: 2022-11-18T22:47:41Z
author_association: OWNER
body:
  When I do this it's important to update the documentation for `resolve_database()` and the like:

  https://github.com/simonw/datasette/blob/ee64130fa8a5ff4a24791916c696e10cf2375102/docs/internals.rst#L594

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1896#issuecomment-1320616559
issue_url: https://api.github.com/repos/simonw/datasette/issues/1896
node_id: IC_kwDOBm6k_c5OtwJv
created_at: 2022-11-18T22:51:14Z
updated_at: 2022-11-18T22:51:14Z
author_association: OWNER
body:
  New methods are documented here: https://docs.datasette.io/en/1.0-dev/internals.html#resolve-database-request

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1891#issuecomment-1320625260
issue_url: https://api.github.com/repos/simonw/datasette/issues/1891
node_id: IC_kwDOBm6k_c5OtyRs
created_at: 2022-11-18T23:01:03Z
updated_at: 2022-11-18T23:01:48Z
author_association: OWNER
body:
  I think this actually needs to include a whole section of the documentation about the road to 1.0 - what to expect (planned breaking changes) etc. I can add that to the https://docs.datasette.io/en/stable/contributing.html page perhaps - or even create a Roadmap page.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1905#issuecomment-1320678715
issue_url: https://api.github.com/repos/simonw/datasette/issues/1905
node_id: IC_kwDOBm6k_c5Ot_U7
created_at: 2022-11-19T00:02:28Z
updated_at: 2022-11-19T00:02:28Z
author_association: OWNER
body:
  This is a strong argument for extracting the Heroku support out to a plugin - it would allow this to be fixed with a plugin release without needing to push a full release of Datasette itself.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1905#issuecomment-1320689643
issue_url: https://api.github.com/repos/simonw/datasette/issues/1905
node_id: IC_kwDOBm6k_c5OuB_r
created_at: 2022-11-19T00:17:19Z
updated_at: 2022-11-19T00:41:54Z
author_association: OWNER
body:
  The tests don't cover this bit at the moment.

  Would be easier to write tests if there was a `--generate-dir` option as seen in https://datasette.io/plugins/datasette-publish-vercel

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1905#issuecomment-1320706596
issue_url: https://api.github.com/repos/simonw/datasette/issues/1905
node_id: IC_kwDOBm6k_c5OuGIk
created_at: 2022-11-19T00:41:44Z
updated_at: 2022-11-19T00:41:44Z
author_association: OWNER
body:
  Tested a deploy:

  ```
   % datasette publish heroku fixtures.db -n datasette-issue-1905
   ›   Warning: heroku update available from 7.63.0 to 7.66.4.
   ›   Warning: heroku update available from 7.63.0 to 7.66.4.
   ›   Warning: heroku update available from 7.63.0 to 7.66.4.
  Creating datasette-issue-1905... done
   ›   Warning: heroku update available from 7.63.0 to 7.66.4.
   ▸    Couldn't detect GNU tar. Builds could fail due to decompression errors
   ▸    See https://devcenter.heroku.com/articles/platform-api-deploying-slugs#create-slug-archive
   ▸    Please install it, or specify the '--tar' option
   ▸    Falling back to node's built-in compressor

  -----> Building on the Heroku-22 stack
  -----> Determining which buildpack to use for this app
  -----> Python app detected
  -----> Using Python version specified in runtime.txt
  -----> Installing python-3.11.0
  -----> Installing pip 22.3.1, setuptools 63.4.3 and wheel 0.37.1
  -----> Installing SQLite3
  -----> Installing requirements with pip
         Collecting datasette
           Downloading datasette-0.63.1-py3-none-any.whl (231 kB)
  ...
  -----> Running post-compile hook
  -----> Discovering process types
         Procfile declares types -> web

  -----> Compressing...
         Done: 28M
  -----> Launching...
         Released v3
         https://datasette-issue-1905.herokuapp.com/ deployed to Heroku

  Starting November 28th, 2022, free Heroku Dynos, free Heroku Postgres, and free Heroku Data for Redis® will no longer be available.

  If you have apps using any of these resources, you must upgrade to paid plans by this date to ensure your apps continue to run and to retain your data. For students, we will announce a new program by the end of September. Learn more at https://blog.heroku.com/next-chapter
  ```
  I had to then pay for the dino because I'd run out of free hours.

  https://datasette-issue-1905.herokuapp.com/-/versions shows:

  ```json
  {
      "python": {
          "version": "3.11.0",
          "full": "3.11.0 (main, Oct 24 2022, 21:34:02) [GCC 11.2.0]"
      },
      "datasette": {
          "version": "0.63.1"
      }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1886#issuecomment-1321003094
issue_url: https://api.github.com/repos/simonw/datasette/issues/1886
node_id: IC_kwDOBm6k_c5OvOhW
created_at: 2022-11-20T00:52:05Z
updated_at: 2022-11-20T00:52:05Z
author_association: CONTRIBUTOR
body:
  Happy birthday to datasette and thank you Simon for your continued effort on this project! 

  I use datasette (python) as a fast layer on top of search for github projects using https://github.com/dogsheep/github-to-sqlite , and use the JSON API it provides to serve sample data to make Vega-Lite graphing workshop examples that don't require authentication/API keys. It's awesome to have a full SQL API support working without needing to develop any custom API middleware for both filtering and grouping.

  I've also enjoyed using it as a teaching tool for working with public dataset in [civic data workshops](https://2022.open-data.nyc/event/low-code-visual-data-exploration-with-nyc-public-data/) and as a platform for making visualization [plugins](https://github.com/hydrosquall/datasette-nteract-data-explorer) . I

  I'm especially excited about datasette-lite, as it will let people participate in future editions of this workshop without having to install anything to make use of their own tables :)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1903#issuecomment-1321262142
issue_url: https://api.github.com/repos/simonw/datasette/issues/1903
node_id: IC_kwDOBm6k_c5OwNw-
created_at: 2022-11-20T22:35:01Z
updated_at: 2022-11-20T22:35:01Z
author_association: OWNER
body:
  A want to call this `datasette/exceptions.py` inspired by Takahē: https://github.com/andrewgodwin/takahe/blob/f491fdb56e2de9200e14b855b5576009ca99dfa5/core/exceptions.py

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1884#issuecomment-1321460293
issue_url: https://api.github.com/repos/simonw/datasette/issues/1884
node_id: IC_kwDOBm6k_c5Ow-JF
created_at: 2022-11-21T04:40:55Z
updated_at: 2022-11-21T04:40:55Z
author_association: CONTRIBUTOR
body:
  Counting any virtual tables can be pretty tricky. On one hand, counting a [CSV virtual table](https://www.sqlite.org/csv.html) would return the number of rows in the CSV, which is helpful (but can be I/O intensive). Counting a [FTS5 virtual table](https://www.sqlite.org/fts5.html) would return the number of entries in the FTS index, which is kindof helpful, but can be misleading in some cases.

  On the other hand, arbitrarily running `COUNT(*)` on some virtual tables can be incredibly expensive. SQLite offers new shortcuts/pushdowns on `COUNT(*)` queries for virtual tables, and instead calls the underlying vtab implementation and iterates through all rows in the table without discretion. For example, a virtual table that's backed by a Postgres table would call `select * from pg_table`, which would use up a lot of network and CPU calls. Or a virtual table backed by a [google sheet](https://github.com/0x6b/libgsqlite) would make network/API requests to get all the rows from the sheet just to make a count.

  The [`pragma_table_list`](https://www.sqlite.org/pragma.html#pragma_table_list) pragma tells you when a table is a regular table or virtual (in the `type` column), but was only added in version 3.37.0 (2021-11-27). 


  Personally, I wouldnt try to `COUNT(*)` virtual tables - it depends on how the virtual table is implemented, it requires that the connection has the proper extensions loaded, and it may accientally cause perf issues for new-age extensions. A few extensions that I'm writing have virtual tables that wouldn't benefit much from `COUNT(*)`, and the fact that SQLite iterates through all rows in a table to count just makes things worse. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1863#issuecomment-1324531085
issue_url: https://api.github.com/repos/simonw/datasette/issues/1863
node_id: IC_kwDOBm6k_c5O8r2N
created_at: 2022-11-23T04:19:28Z
updated_at: 2022-11-23T04:19:28Z
author_association: OWNER
body:
  Had a design conversation with myself in https://github.com/simonw/dclient/issues/6 where I decided that the API should allow string values to be sent to integer columns which would be automatically converted *if possible to do so* - as an API usability feature.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1863#issuecomment-1324531750
issue_url: https://api.github.com/repos/simonw/datasette/issues/1863
node_id: IC_kwDOBm6k_c5O8sAm
created_at: 2022-11-23T04:20:47Z
updated_at: 2022-11-23T04:20:47Z
author_association: OWNER
body:
  ... which does imply that I'm going to do an extra layer of validation over what SQLite provides. SQLite will happily allow a text string to be added to a supposedly integer column. I'm not going to allow that - I'll return a validation error instead, unless the string can be safely coerced to the correct type.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1909#issuecomment-1329624931
issue_url: https://api.github.com/repos/simonw/datasette/issues/1909
node_id: IC_kwDOBm6k_c5PQHdj
created_at: 2022-11-28T19:19:26Z
updated_at: 2022-11-28T19:19:26Z
author_association: OWNER
body:
  The list of states here is a good example of somewhere this might be useful: https://congress-legislators.datasettes.com/legislators/legislator_terms?_facet=state&_facet_size=max

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1863#issuecomment-1330974099
issue_url: https://api.github.com/repos/simonw/datasette/issues/1863
node_id: IC_kwDOBm6k_c5PVQ2T
created_at: 2022-11-29T17:03:00Z
updated_at: 2022-11-29T17:11:05Z
author_association: OWNER
body:
  I've decided that I won't do that validation for the first version of this - I'm going to teach `dclient` to send the correct types instead: https://github.com/simonw/dclient/issues/6#issuecomment-1330963953

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1863#issuecomment-1331089156
issue_url: https://api.github.com/repos/simonw/datasette/issues/1863
node_id: IC_kwDOBm6k_c5PVs8E
created_at: 2022-11-29T18:08:53Z
updated_at: 2022-11-29T18:08:53Z
author_association: OWNER
body:
  I do think this needs type checking - I just tried and you really can send a string to an integer column and have it work, which feels bad.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1911#issuecomment-1331120055
issue_url: https://api.github.com/repos/simonw/datasette/issues/1911
node_id: IC_kwDOBm6k_c5PV0e3
created_at: 2022-11-29T18:36:01Z
updated_at: 2022-11-29T18:36:01Z
author_association: OWNER
body:
  Current API design:
  ```
  POST /<database>/-/create
  ```
  ```json
  {
      "table": "name_of_new_table",
      "columns": [
          {
              "name": "id",
              "type": "integer"
          },
          {
              "name": "title",
              "type": "text"
          }
      ],
      "pk": "id"
  }
  ```
  I'm going to add a new `"pks"` key which is a list, and can be used in place of `"pk"`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1891#issuecomment-1331140747
issue_url: https://api.github.com/repos/simonw/datasette/issues/1891
node_id: IC_kwDOBm6k_c5PV5iL
created_at: 2022-11-29T18:55:42Z
updated_at: 2022-11-29T18:55:42Z
author_association: OWNER
body:
  All features for the alpha are complete now. Release notes should be based on these commits:

  https://github.com/simonw/datasette/compare/0.63.2...6bda2257868a2cbd70b84b7a86a5bcb47dcc4874

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1891#issuecomment-1331181922
issue_url: https://api.github.com/repos/simonw/datasette/issues/1891
node_id: IC_kwDOBm6k_c5PWDli
created_at: 2022-11-29T19:23:41Z
updated_at: 2022-11-29T19:23:41Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/blob/4d49a5a39739476e1ada43f70a0029abcef07977/docs/changelog.rst#10a0-2022-11-29

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1605#issuecomment-1331187551
issue_url: https://api.github.com/repos/simonw/datasette/issues/1605
node_id: IC_kwDOBm6k_c5PWE9f
created_at: 2022-11-29T19:29:42Z
updated_at: 2022-11-29T19:29:42Z
author_association: CONTRIBUTOR
body:
  Interesting. I started a version using metadata like I outlined up top, but I realized that there's no documented way for a plugin to access either metadata or canned queries. Or at least, I couldn't find a way.

  There is this method: https://github.com/simonw/datasette/blob/main/datasette/app.py#L472 but I don't want to rely on it if it's not documented. Same with this: https://github.com/simonw/datasette/blob/main/datasette/app.py#L544

  If those are safe, I'll build on them. I'm also happy to document them, if that greases the wheels.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1912#issuecomment-1331196531
issue_url: https://api.github.com/repos/simonw/datasette/issues/1912
node_id: IC_kwDOBm6k_c5PWHJz
created_at: 2022-11-29T19:39:10Z
updated_at: 2022-11-29T19:39:10Z
author_association: OWNER
body:
  Annoyingly it looks like I can't rebase this one, and I don't want to squash-merge and lose the commits, so I'm going to do a regular merge instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1877#issuecomment-1331201207
issue_url: https://api.github.com/repos/simonw/datasette/issues/1877
node_id: IC_kwDOBm6k_c5PWIS3
created_at: 2022-11-29T19:44:07Z
updated_at: 2022-11-29T19:44:07Z
author_association: OWNER
body:
  I fixed the duplicate logic issue here: https://github.com/simonw/datasette/commit/ee64130fa8a5ff4a24791916c696e10cf2375102

  - #1896

  Decided not to address `views/table.py`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1913#issuecomment-1331203997
issue_url: https://api.github.com/repos/simonw/datasette/issues/1913
node_id: IC_kwDOBm6k_c5PWI-d
created_at: 2022-11-29T19:47:13Z
updated_at: 2022-11-29T19:47:13Z
author_association: OWNER
body:
  Weird, retrying the tests DID get them to pass. https://github.com/simonw/datasette/actions/runs/3577355358/jobs/6016518244

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1913#issuecomment-1331204360
issue_url: https://api.github.com/repos/simonw/datasette/issues/1913
node_id: IC_kwDOBm6k_c5PWJEI
created_at: 2022-11-29T19:47:40Z
updated_at: 2022-11-29T19:47:40Z
author_association: OWNER
body:
  ... but the last step of the deploy failed, when it was meant to push to PyPI!
  ```
  Uploading distributions to https://upload.pypi.org/legacy/
  Traceback (most recent call last):
    File "/opt/hostedtoolcache/Python/3.11.0/x64/bin/twine", line 8, in <module>
      sys.exit(main())
               ^^^^^^
    File "/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/twine/__main__.py", line 33, in main
      error = cli.dispatch(sys.argv[1:])
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/twine/cli.py", line 123, in dispatch
      return main(args.args)
             ^^^^^^^^^^^^^^^
    File "/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/twine/commands/upload.py", line 198, in main
      return upload(upload_settings, parsed_args.dists)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/twine/commands/upload.py", line 123, in upload
      packages_to_upload = [
                           ^
    File "/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/twine/commands/upload.py", line 124, in <listcomp>
      _make_package(filename, signatures, upload_settings) for filename in uploads
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/twine/commands/upload.py", line 77, in _make_package
      package = package_file.PackageFile.from_filename(filename, upload_settings.comment)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/twine/package.py", line 96, in from_filename
      meta = DIST_TYPES[dtype](filename)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/twine/wheel.py", line 42, in __init__
      self.extractMetadata()
    File "/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/pkginfo/distribution.py", line 121, in extractMetadata
      self.parse(data)
    File "/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/twine/wheel.py", line 89, in parse
      fp = io.StringIO(distribution.must_decode(data))
                       ^^^^^^^^^^^^^^^^^^^^^^^^
  AttributeError: module 'pkginfo.distribution' has no attribute 'must_decode'. Did you mean: '_must_decode'?
  Error: Process completed with exit code 1.
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1913#issuecomment-1331205613
issue_url: https://api.github.com/repos/simonw/datasette/issues/1913
node_id: IC_kwDOBm6k_c5PWJXt
created_at: 2022-11-29T19:48:52Z
updated_at: 2022-11-29T19:48:52Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/blob/07aad511769da9242260c850e8d975cbd8c29552/.github/workflows/publish.yml#L52-L61

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1913#issuecomment-1331207334
issue_url: https://api.github.com/repos/simonw/datasette/issues/1913
node_id: IC_kwDOBm6k_c5PWJym
created_at: 2022-11-29T19:50:37Z
updated_at: 2022-11-29T19:50:37Z
author_association: OWNER
body:
  https://pypi.org/project/setuptools/65.6.3/ came out most recently - 23rd November (wheel and twine are older).

  No search results at all for that error message. This is very weird, I would have expected it to have been reported by now.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1913#issuecomment-1331225277
issue_url: https://api.github.com/repos/simonw/datasette/issues/1913
node_id: IC_kwDOBm6k_c5PWOK9
created_at: 2022-11-29T19:59:14Z
updated_at: 2022-11-29T19:59:34Z
author_association: OWNER
body:
  I deleted the tag and tried creating a new release. Now running here: https://github.com/simonw/datasette/actions/runs/3577554546

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1913#issuecomment-1331226346
issue_url: https://api.github.com/repos/simonw/datasette/issues/1913
node_id: IC_kwDOBm6k_c5PWObq
created_at: 2022-11-29T20:00:16Z
updated_at: 2022-11-29T20:00:36Z
author_association: OWNER
body:
  Looks like a fix is coming: https://github.com/pypa/twine/issues/940#issuecomment-1331225509

  > > Note that `must_decode` was defined in `pkg_info/_compat.py`, and was thus never an API: before 1.9.0, it was only imported and used in `pkginfo/distribution.py'.
  > 
  > Nevertheless, I will push out a 1.9.1 release of `pkginfo` which restores a deprecated compatibility alias.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1913#issuecomment-1331238029
issue_url: https://api.github.com/repos/simonw/datasette/issues/1913
node_id: IC_kwDOBm6k_c5PWRSN
created_at: 2022-11-29T20:10:35Z
updated_at: 2022-11-29T20:10:35Z
author_association: OWNER
body:
  Released:

  - https://pypi.org/project/datasette/1.0a0/
  - https://docs.datasette.io/en/latest/changelog.html#a0-2022-11-29

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1915#issuecomment-1331331082
issue_url: https://api.github.com/repos/simonw/datasette/issues/1915
node_id: IC_kwDOBm6k_c5PWoAK
created_at: 2022-11-29T21:24:59Z
updated_at: 2022-11-29T21:34:53Z
author_association: OWNER
body:
  Maybe a plugin called `datasette-temporary-tables` or `datasette-demo-tables` or `datasette-demo-database`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1915#issuecomment-1331432223
issue_url: https://api.github.com/repos/simonw/datasette/issues/1915
node_id: IC_kwDOBm6k_c5PXAsf
created_at: 2022-11-29T23:06:17Z
updated_at: 2022-11-29T23:06:17Z
author_association: OWNER
body:
  To (slightly) discourage abuse I'm going to make the demo database only visible to the root user - so people can't create tables with rude names and have them show to the public on https://latest.datasette.io/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1915#issuecomment-1331476246
issue_url: https://api.github.com/repos/simonw/datasette/issues/1915
node_id: IC_kwDOBm6k_c5PXLcW
created_at: 2022-11-30T00:04:35Z
updated_at: 2022-11-30T00:08:24Z
author_association: OWNER
body:
  The new https://github.com/simonw/datasette-ephemeral-tables plugin is live now: https://latest.datasette.io/ephemeral - you have to navigate through https://latest.datasette.io/login-as-root first

  It work! I created a table using https://latest.datasette.io/-/api#path=%2Fephemeral%2F-%2Fcreate&json=%7B%0A++%22table%22%3A+%22new_table%22%2C%0A++%22columns%22%3A+%5B%0A++++%7B%0A++++++%22name%22%3A+%22id%22%2C%0A++++++%22type%22%3A+%22integer%22%0A++++%7D%2C%0A++++%7B%0A++++++%22name%22%3A+%22name%22%2C%0A++++++%22type%22%3A+%22text%22%0A++++%7D%0A++%5D%2C%0A++%22pk%22%3A+%22id%22%0A%7D&method=POST

  The table should vanish in a few minutes.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1915#issuecomment-1331478611
issue_url: https://api.github.com/repos/simonw/datasette/issues/1915
node_id: IC_kwDOBm6k_c5PXMBT
created_at: 2022-11-30T00:07:37Z
updated_at: 2022-11-30T00:07:37Z
author_association: OWNER
body:
  Then I created an API token at https://latest.datasette.io/-/create-token and ran this:

  ```
  curl -XPOST 'https://latest.datasette.io/ephemeral/new_table/-/insert' \
    -H 'Authorization: Bearer xxx' \
    -H 'Content-Type: application/json' \
    -d '{"row": {"name": "NAME"}}'
  ```
  And it inserted a row into https://latest.datasette.io/ephemeral/new_table

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1915#issuecomment-1331479606
issue_url: https://api.github.com/repos/simonw/datasette/issues/1915
node_id: IC_kwDOBm6k_c5PXMQ2
created_at: 2022-11-30T00:09:06Z
updated_at: 2022-11-30T00:09:06Z
author_association: OWNER
body:
  One last feature: I want to show an indication on the table page that the table has X seconds left to live.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1917#issuecomment-1331644078
issue_url: https://api.github.com/repos/simonw/datasette/issues/1917
node_id: IC_kwDOBm6k_c5PX0au
created_at: 2022-11-30T04:58:06Z
updated_at: 2022-11-30T04:58:06Z
author_association: OWNER
body:
  The problem might actually be here:

  https://github.com/simonw/datasette/blob/9f5321ff1eca58c469a45cc406d7eb5ad05accbd/datasette/app.py#L280-L281

  `is_mutable` defaults to `True`, so this line should probably be:

  ```python
       self.add_database(Database(self, is_mutable=False, is_memory=True), name="_memory")
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1917#issuecomment-1331644751
issue_url: https://api.github.com/repos/simonw/datasette/issues/1917
node_id: IC_kwDOBm6k_c5PX0lP
created_at: 2022-11-30T04:59:22Z
updated_at: 2022-11-30T04:59:22Z
author_association: OWNER
body:
  Yeah it looks like I introduced this bug here:

  https://github.com/simonw/datasette/commit/fb7e70d5e72a951efe4b29ad999d8915c032d021

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1916#issuecomment-1331651721
issue_url: https://api.github.com/repos/simonw/datasette/issues/1916
node_id: IC_kwDOBm6k_c5PX2SJ
created_at: 2022-11-30T05:10:27Z
updated_at: 2022-11-30T05:10:27Z
author_association: OWNER
body:
  They should return 405 method not allowed with an `{"ok":false, "error": "Method not allowed"}` body.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1919#issuecomment-1331657404
issue_url: https://api.github.com/repos/simonw/datasette/issues/1919
node_id: IC_kwDOBm6k_c5PX3q8
created_at: 2022-11-30T05:19:43Z
updated_at: 2022-11-30T05:19:43Z
author_association: OWNER
body:
  This is the test: https://github.com/simonw/datasette/blob/8404b21556d133c89eda4bd1bf5335ed9a0785d6/tests/test_api_write.py#L342-L401

  I'm suspicious that there's a timing error of some sort but I can't think what it might be.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1918#issuecomment-1331658629
issue_url: https://api.github.com/repos/simonw/datasette/issues/1918
node_id: IC_kwDOBm6k_c5PX3-F
created_at: 2022-11-30T05:21:51Z
updated_at: 2022-11-30T05:21:51Z
author_association: OWNER
body:
  Much better:

  <img width="1147" alt="image" src="https://user-images.githubusercontent.com/9599/204714148-f21cd802-c268-41cf-bfc1-e9c2f4e48bf9.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1922#issuecomment-1332492092
issue_url: https://api.github.com/repos/simonw/datasette/issues/1922
node_id: IC_kwDOBm6k_c5PbDc8
created_at: 2022-11-30T17:17:21Z
updated_at: 2022-11-30T17:17:21Z
author_association: OWNER
body:
  I tried running `fetch()` with a POST from a separate domain and got a browser error because it did a GET against the `/db/-/create` endpoint and  the 405 method not supported response did not include the CORS headers.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1922#issuecomment-1332493004
issue_url: https://api.github.com/repos/simonw/datasette/issues/1922
node_id: IC_kwDOBm6k_c5PbDrM
created_at: 2022-11-30T17:18:10Z
updated_at: 2022-11-30T17:18:10Z
author_association: OWNER
body:
  Here's why:

  https://github.com/simonw/datasette/blob/4ddd77e51254bda3bac990ea662bac2e6b29c5e0/datasette/views/base.py#L71-L79

  That's code in `BaseView` - but it turns out the code that adds CORS headers is in the `DataView` subclass of that (which the various write API endpoints do not use).

  https://github.com/simonw/datasette/blob/4ddd77e51254bda3bac990ea662bac2e6b29c5e0/datasette/views/base.py#L158-L162

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1922#issuecomment-1332561059
issue_url: https://api.github.com/repos/simonw/datasette/issues/1922
node_id: IC_kwDOBm6k_c5PbUSj
created_at: 2022-11-30T18:19:20Z
updated_at: 2022-11-30T18:19:20Z
author_association: OWNER
body:
  Two test failures:
  ```
  ____________________________ test_homepage_options _____________________________
  [gw0] linux -- Python 3.11.0 /opt/hostedtoolcache/Python/3.11.0/x64/bin/python

  app_client = <datasette.utils.testing.TestClient object at 0x7f4c489269d0>

      def test_homepage_options(app_client):
          response = app_client.get("/", method="OPTIONS")
  >       assert response.status == 405
  E       assert 200 == 405
  E        +  where 200 = <datasette.utils.testing.TestResponse object at 0x7f4c4892f4d0>.status

  /home/runner/work/datasette/datasette/tests/test_html.py:58: AssertionError
  ______________________ test_client_methods[options-/-405] ______________________
  [gw1] linux -- Python 3.11.0 /opt/hostedtoolcache/Python/3.11.0/x64/bin/python

  datasette = <datasette.app.Datasette object at 0x7fc33c227550>
  method = 'options', path = '/', expected_status = 405

      @pytest.mark.asyncio
      @pytest.mark.parametrize(
          "method,path,expected_status",
          [
              ("get", "/", 200),
              ("options", "/", 405),
              ("head", "/", 200),
              ("put", "/", 405),
              ("patch", "/", 405),
              ("delete", "/", 405),
          ],
      )
      async def test_client_methods(datasette, method, path, expected_status):
          client_method = getattr(datasette.client, method)
          response = await client_method(path)
          assert isinstance(response, httpx.Response)
  >       assert response.status_code == expected_status
  E       assert 200 == 405
  E        +  where 200 = <Response [200 OK]>.status_code

  /home/runner/work/datasette/datasette/tests/test_internals_datasette_client.py:29: AssertionError
  =============================== warnings summary ===============================
  tests/test_cli.py::test_inspect_cli_writes_to_file
  tests/test_cli.py::test_inspect_cli
    /home/runner/work/datasette/datasette/datasette/cli.py:163: DeprecationWarning: There is no current event loop
      loop = asyncio.get_event_loop()

  tests/test_cli_serve_get.py: 2 warnings
  tests/test_cli.py: 12 warnings
  tests/test_crossdb.py: 1 warning
    /home/runner/work/datasette/datasette/datasette/cli.py:591: DeprecationWarning: There is no current event loop
      asyncio.get_event_loop().run_until_complete(ds.invoke_startup())

  tests/test_cli_serve_get.py: 2 warnings
  tests/test_cli.py: 12 warnings
  tests/test_crossdb.py: 1 warning
    /home/runner/work/datasette/datasette/datasette/cli.py:594: DeprecationWarning: There is no current event loop
      asyncio.get_event_loop().run_until_complete(check_databases(ds))

  -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
  =========================== short test summary info ============================
  FAILED tests/test_html.py::test_homepage_options - assert 200 == 405
   +  where 200 = <datasette.utils.testing.TestResponse object at 0x7f4c4892f4d0>.status
  FAILED tests/test_internals_datasette_client.py::test_client_methods[options-/-405] - assert 200 == 405
   +  where 200 = <Response [200 OK]>.status_code
  ====== 2 failed, 1195 passed, 1 skipped, 32 warnings in 191.06s (0:03:11) ======
  Error: Process completed with exit code 1.
  ```
  On reading https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/OPTIONS I feel like I should be a bit more thoughtful about how I treat OPTIONS - maybe it should work for every URL on the site, but return a `204 No Content` header?

  Comparing a few different sites:

  ```
  ~ % curl -X OPTIONS https://www.google.com/ -i
  HTTP/2 405 
  allow: GET, HEAD
  date: Wed, 30 Nov 2022 18:18:15 GMT
  content-type: text/html; charset=UTF-8
  server: gws
  content-length: 1592
  x-xss-protection: 0
  x-frame-options: SAMEORIGIN
  alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"

  <!DOCTYPE html>
  <html lang=en>
    <meta charset=utf-8>
    <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
    <title>Error 405 (Method Not Allowed)!!1</title>
    <style>
      *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
    </style>
    <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
    <p><b>405.</b> <ins>That’s an error.</ins>
    <p>The request method <code>OPTIONS</code> is inappropriate for the URL <code>/</code>.  <ins>That’s all we know.</ins>
  ~ % curl -X OPTIONS https://www.mozilla.org/ -i
  HTTP/2 405 
  content-type: text/html; charset=utf-8
  content-length: 0
  server: meinheld/1.0.2
  date: Wed, 30 Nov 2022 18:18:38 GMT
  allow: GET, HEAD
  x-frame-options: DENY
  content-security-policy: child-src 'self' *.mozilla.net *.mozilla.org *.mozilla.com www.googletagmanager.com www.google-analytics.com www.youtube-nocookie.com trackertest.org www.surveygizmo.com accounts.firefox.com accounts.firefox.com.cn www.youtube.com; connect-src 'self' *.mozilla.net *.mozilla.org *.mozilla.com www.googletagmanager.com www.google-analytics.com region1.google-analytics.com logs.convertexperiments.com 1003350.metrics.convertexperiments.com 1003343.metrics.convertexperiments.com sentry.prod.mozaws.net o1069899.sentry.io o1069899.ingest.sentry.io https://accounts.firefox.com/ stage.cjms.nonprod.cloudops.mozgcp.net cjms.services.mozilla.com; frame-src 'self' *.mozilla.net *.mozilla.org *.mozilla.com www.googletagmanager.com www.google-analytics.com www.youtube-nocookie.com trackertest.org www.surveygizmo.com accounts.firefox.com accounts.firefox.com.cn www.youtube.com; script-src 'self' *.mozilla.net *.mozilla.org *.mozilla.com 'unsafe-inline' 'unsafe-eval' www.googletagmanager.com www.google-analytics.com tagmanager.google.com www.youtube.com s.ytimg.com cdn-3.convertexperiments.com app.convert.com data.track.convertexperiments.com 1003350.track.convertexperiments.com 1003343.track.convertexperiments.com; img-src 'self' *.mozilla.net *.mozilla.org *.mozilla.com data: mozilla.org www.googletagmanager.com www.google-analytics.com adservice.google.com adservice.google.de adservice.google.dk creativecommons.org cdn-3.convertexperiments.com logs.convertexperiments.com images.ctfassets.net ad.doubleclick.net; style-src 'self' *.mozilla.net *.mozilla.org *.mozilla.com 'unsafe-inline' app.convert.com; default-src 'self' *.mozilla.net *.mozilla.org *.mozilla.com; font-src 'self'
  cache-control: max-age=600
  expires: Wed, 30 Nov 2022 18:28:38 GMT
  x-backend-server: bedrock-prod-web-b95bc569d-grd25.iowa-a
  strict-transport-security: max-age=31536000
  x-content-type-options: nosniff
  x-xss-protection: 1; mode=block
  referrer-policy: strict-origin-when-cross-origin
  via: 1.1 google, 1.1 6c90b631453c435bd0022caa657b67e8.cloudfront.net (CloudFront)
  x-cache: Error from cloudfront
  x-amz-cf-pop: SFO5-P2
  x-amz-cf-id: A6-9mLztaE2tz840CbV9bXYiBMZRKEamDj6jGGEl1U7sg8egWfsDqg==

  ~ % curl -X OPTIONS https://example.com -i     
  HTTP/2 200 
  allow: OPTIONS, GET, HEAD, POST
  cache-control: max-age=604800
  content-type: text/html; charset=UTF-8
  date: Wed, 30 Nov 2022 18:18:59 GMT
  expires: Wed, 07 Dec 2022 18:18:59 GMT
  server: EOS (vny/0451)
  content-length: 0
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1922#issuecomment-1332561813
issue_url: https://api.github.com/repos/simonw/datasette/issues/1922
node_id: IC_kwDOBm6k_c5PbUeV
created_at: 2022-11-30T18:20:05Z
updated_at: 2022-11-30T18:20:05Z
author_association: OWNER
body:
  Weird, GitHub reply with a 404!
  ```
  ~ % curl -X OPTIONS https://github.com/ -i
  HTTP/2 404 
  server: GitHub.com
  date: Wed, 30 Nov 2022 18:19:39 GMT
  content-type: text/html; charset=utf-8
  content-length: 0
  strict-transport-security: max-age=31536000; includeSubdomains; preload
  x-frame-options: deny
  x-content-type-options: nosniff
  x-xss-protection: 0
  referrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin
  content-security-policy: default-src 'none'; base-uri 'self'; block-all-mixed-content; child-src github.com/assets-cdn/worker/ gist.github.com/assets-cdn/worker/; connect-src 'self' uploads.github.com objects-origin.githubusercontent.com www.githubstatus.com collector.github.com raw.githubusercontent.com api.github.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com cdn.optimizely.com logx.optimizely.com/v1/events; font-src github.githubassets.com; form-action 'self' github.com gist.github.com objects-origin.githubusercontent.com; frame-ancestors 'none'; frame-src viewscreen.githubusercontent.com notebooks.githubusercontent.com; img-src 'self' data: github.githubassets.com media.githubusercontent.com camo.githubusercontent.com identicons.github.com avatars.githubusercontent.com github-cloud.s3.amazonaws.com objects.githubusercontent.com objects-origin.githubusercontent.com secured-user-images.githubusercontent.com/ opengraph.githubassets.com github-production-user-asset-6210df.s3.amazonaws.com customer-stories-feed.github.com spotlights-feed.github.com; manifest-src 'self'; media-src github.com user-images.githubusercontent.com/ secured-user-images.githubusercontent.com/; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com; worker-src github.com/assets-cdn/worker/ gist.github.com/assets-cdn/worker/
  vary: Accept-Encoding, Accept, X-Requested-With
  x-github-request-id: DD6B:5ACA:102E8A6:1164A99:63879EBB
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1922#issuecomment-1332572453
issue_url: https://api.github.com/repos/simonw/datasette/issues/1922
node_id: IC_kwDOBm6k_c5PbXEl
created_at: 2022-11-30T18:30:38Z
updated_at: 2022-11-30T18:30:54Z
author_association: OWNER
body:
  Started a conversation about how OPTIONS should work on Mastodon: https://fedi.simonwillison.net/@simon/109434148676475291

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1922#issuecomment-1332580395
issue_url: https://api.github.com/repos/simonw/datasette/issues/1922
node_id: IC_kwDOBm6k_c5PbZAr
created_at: 2022-11-30T18:38:22Z
updated_at: 2022-11-30T18:38:22Z
author_association: OWNER
body:
  > [@simon](https://fedi.simonwillison.net/@simon) IMO, it should always be a 2XX series response, typically with no content & an extra `Allow` header with a list of HTTP verbs it responds to.

  https://mastodon.social/@daniellindsley/109434186252099323

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1922#issuecomment-1332585861
issue_url: https://api.github.com/repos/simonw/datasette/issues/1922
node_id: IC_kwDOBm6k_c5PbaWF
created_at: 2022-11-30T18:43:46Z
updated_at: 2022-11-30T18:43:46Z
author_association: OWNER
body:
  Here's what Django Rest Framework does: https://github.com/encode/django-rest-framework/blob/1ae812ea209392ad76cc5d2f35f9f7fb337f63e4/rest_framework/views.py#L514-L521

  ```python
      def options(self, request, *args, **kwargs):
          """
          Handler method for HTTP 'OPTIONS' request.
          """
          if self.metadata_class is None:
              return self.http_method_not_allowed(request, *args, **kwargs)
          data = self.metadata_class().determine_metadata(request, self)
          return Response(data, status=status.HTTP_200_OK)
  ```
  That default `determine_metadata` method looks like this: https://github.com/encode/django-rest-framework/blob/1ae812ea209392ad76cc5d2f35f9f7fb337f63e4/rest_framework/metadata.py#L61-L71

  ```python
      def determine_metadata(self, request, view):
          metadata = OrderedDict()
          metadata['name'] = view.get_view_name()
          metadata['description'] = view.get_view_description()
          metadata['renders'] = [renderer.media_type for renderer in view.renderer_classes]
          metadata['parses'] = [parser.media_type for parser in view.parser_classes]
          if hasattr(view, 'get_serializer'):
              actions = self.determine_actions(request, view)
              if actions:
                  metadata['actions'] = actions
          return metadata
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1922#issuecomment-1332688245
issue_url: https://api.github.com/repos/simonw/datasette/issues/1922
node_id: IC_kwDOBm6k_c5PbzV1
created_at: 2022-11-30T20:15:08Z
updated_at: 2022-11-30T20:15:08Z
author_association: OWNER
body:
  Still getting a CORS error:

  <img width="1367" alt="image" src="https://user-images.githubusercontent.com/9599/204898990-b1370950-8ae5-4c95-8f7f-42b4a7a717e7.png">

  My hunch is this is because I'm not sending `Access-Control-Allow-Methods: GET,HEAD,POST`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1922#issuecomment-1332689547
issue_url: https://api.github.com/repos/simonw/datasette/issues/1922
node_id: IC_kwDOBm6k_c5PbzqL
created_at: 2022-11-30T20:16:21Z
updated_at: 2022-11-30T20:16:46Z
author_association: OWNER
body:
  That notebook:
  ```javascript
  viewof token = Inputs.text({
    label: "Your API token"
  })
  ```
  ```javascript
  viewof createResponse = Inputs.button("Create table", {
    value: null,
    reduce: async () => {
      const response = await fetch(
        "https://latest.datasette.io/ephemeral/-/create",
        {
          method: "POST",
          mode: "cors",
          headers: {
            Authorization: `Bearer {$token}`,
            "Content-Type": "application/json"
          },
          body: JSON.stringify({
            table: "my_new_table",
            row: {
              task: "Demonstrate a JSON creation from another domain"
            }
          })
        }
      );
      return await response.json();
    }
  })
  ```
  Based on this tip: https://talk.observablehq.com/t/best-pattern-for-click-here-to-submit-your-results-to-an-api-backend/7353/3

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1922#issuecomment-1332698636
issue_url: https://api.github.com/repos/simonw/datasette/issues/1922
node_id: IC_kwDOBm6k_c5Pb14M
created_at: 2022-11-30T20:25:50Z
updated_at: 2022-11-30T20:25:50Z
author_association: OWNER
body:
  I just shipped this:

      Access-Control-Allow-Methods: GET, POST, HEAD, OPTIONS

  I'll try this out on `latest.datasette.io` - but I need to research more to check if this is a safe thing to do or not.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1923#issuecomment-1332835664
issue_url: https://api.github.com/repos/simonw/datasette/issues/1923
node_id: IC_kwDOBm6k_c5PcXVQ
created_at: 2022-11-30T22:50:10Z
updated_at: 2022-11-30T22:51:25Z
author_association: OWNER
body:
  https://stackoverflow.com/questions/74490465/github-actions-failing-for-setup-gcloud/74562526#74562526 suggests setting `version: '318.0.0'` of `google-github-actions/setup-gcloud`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1923#issuecomment-1332842435
issue_url: https://api.github.com/repos/simonw/datasette/issues/1923
node_id: IC_kwDOBm6k_c5PcY_D
created_at: 2022-11-30T22:58:33Z
updated_at: 2022-11-30T22:58:33Z
author_association: OWNER
body:
  https://stackoverflow.com/questions/74490465/github-actions-failing-for-setup-gcloud/74562740#74562740 suggests trying Python 3.9.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1922#issuecomment-1332855687
issue_url: https://api.github.com/repos/simonw/datasette/issues/1922
node_id: IC_kwDOBm6k_c5PccOH
created_at: 2022-11-30T23:09:31Z
updated_at: 2022-11-30T23:09:31Z
author_association: OWNER
body:
  Still getting a CORS header.

  I tried it in Chrome, which outputs helpful messages to the console:

  <img width="671" alt="image" src="https://user-images.githubusercontent.com/9599/204927523-4b9860f5-4400-4dd5-a036-de9f07f31c5f.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1922#issuecomment-1332903011
issue_url: https://api.github.com/repos/simonw/datasette/issues/1922
node_id: IC_kwDOBm6k_c5Pcnxj
created_at: 2022-11-30T23:45:29Z
updated_at: 2022-11-30T23:45:29Z
author_association: OWNER
body:
  That worked for the preflight request - got this now:

  <img width="1377" alt="image" src="https://user-images.githubusercontent.com/9599/204931998-d70df6c0-723b-4543-b8c4-601c00fe4b10.png">

  So it looks like error responses (in this case for permission denied) are missing CORS headers.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1924#issuecomment-1333023273
issue_url: https://api.github.com/repos/simonw/datasette/issues/1924
node_id: IC_kwDOBm6k_c5PdFIp
created_at: 2022-12-01T01:34:48Z
updated_at: 2022-12-01T01:34:48Z
author_association: OWNER
body:
  This will enable some very cool Datasette write API demos - for example, Git scrapers that insert-replace the most recent copy of the scraped data to a table somewhere (which can then produce an atom feed with https://datasette.io/plugins/datasette-atom)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1924#issuecomment-1333025076
issue_url: https://api.github.com/repos/simonw/datasette/issues/1924
node_id: IC_kwDOBm6k_c5PdFk0
created_at: 2022-12-01T01:37:09Z
updated_at: 2022-12-01T01:37:09Z
author_association: OWNER
body:
  Related question: what happens if you attempt to insert rows without either of these settings and 1:10 of them is an integrity error due to an existing primary key?

  Does the entire batch fail to be inserted? Should it?

  This may be the point that I need to think hard about how to use transactions here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1924#issuecomment-1333042785
issue_url: https://api.github.com/repos/simonw/datasette/issues/1924
node_id: IC_kwDOBm6k_c5PdJ5h
created_at: 2022-12-01T02:00:06Z
updated_at: 2022-12-01T02:00:06Z
author_association: OWNER
body:
  Looks like I did implement these already:

  https://github.com/simonw/datasette/blob/9a1536b52a07e32da5900652da1bd7894c58fa9f/tests/test_api_write.py#L270-L273

  But forgot to document them!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1924#issuecomment-1334165225
issue_url: https://api.github.com/repos/simonw/datasette/issues/1924
node_id: IC_kwDOBm6k_c5Phb7p
created_at: 2022-12-01T18:15:15Z
updated_at: 2022-12-01T18:15:15Z
author_association: OWNER
body:
  Updated docs at the bottom of this section: https://docs.datasette.io/en/latest/json_api.html#inserting-rows

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1636#issuecomment-1334666806
issue_url: https://api.github.com/repos/simonw/datasette/issues/1636
node_id: IC_kwDOBm6k_c5PjWY2
created_at: 2022-12-02T01:58:40Z
updated_at: 2022-12-02T02:00:53Z
author_association: OWNER
body:
  Current design:

  ```json
  {
      "databases": {
          "private": {
              "allow": {
                  "id": "*"
              }
          }
      }
  }
  ```
  This can be applied at the instance, database, table or query level within the nested JSON.

  https://docs.datasette.io/en/stable/authentication.html#controlling-access-to-specific-databases

  It's actually controlling the following permissions:

  - `view-instance`
  - `view-database`
  - `view-table`
  - `view-query`

  There's also a special case for allowing SQL queries,at the instance and database level:

  ```json
  {
      "databases": {
          "mydatabase": {
              "allow_sql": {
                  "id": "root"
              }
          }
      }
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1636#issuecomment-1334673179
issue_url: https://api.github.com/repos/simonw/datasette/issues/1636
node_id: IC_kwDOBm6k_c5PjX8b
created_at: 2022-12-02T02:07:20Z
updated_at: 2022-12-02T04:27:07Z
author_association: OWNER
body:
  So the new mechanism needs to extend that to handle all of the other permissions as well.

  The simplest design I can think of is this (here illustrated using YAML):

  ```yaml
  # instance-level permissions - give every logged in user the debug menu:
  permissions:
    debug-menu:
      id: *
  databases:
    content:
      # Allow bob to create-table in the content database
      permissions:
        create-table:
          id: bob
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1636#issuecomment-1334757597
issue_url: https://api.github.com/repos/simonw/datasette/issues/1636
node_id: IC_kwDOBm6k_c5Pjsjd
created_at: 2022-12-02T04:42:35Z
updated_at: 2022-12-02T04:42:35Z
author_association: OWNER
body:
  Should I call this key `permissions` or something else?

  Some options:

  - `permissions`
  - `perms` - shorter to type
  - `allow` - I like the word, but might be confusing to change its meaning since we use it already

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1636#issuecomment-1334758766
issue_url: https://api.github.com/repos/simonw/datasette/issues/1636
node_id: IC_kwDOBm6k_c5Pjs1u
created_at: 2022-12-02T04:45:16Z
updated_at: 2022-12-02T04:45:16Z
author_association: OWNER
body:
  Also, this is another thing which should live in `config.yml` rather than being crammed into `metadata.yml` - but I can fix that when I address:
  - #493

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1636#issuecomment-1334759315
issue_url: https://api.github.com/repos/simonw/datasette/issues/1636
node_id: IC_kwDOBm6k_c5Pjs-T
created_at: 2022-12-02T04:46:32Z
updated_at: 2022-12-02T04:46:32Z
author_association: OWNER
body:
  Thankfully all of the logic for this already lives in just one place:

  https://github.com/simonw/datasette/blob/d7e5e3c9f98d194fdfb12f1ecc60ed5b3afbc464/datasette/default_permissions.py#L23-L59

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1928#issuecomment-1335870879
issue_url: https://api.github.com/repos/simonw/datasette/issues/1928
node_id: IC_kwDOBm6k_c5Pn8Wf
created_at: 2022-12-02T21:18:25Z
updated_at: 2022-12-02T21:18:25Z
author_association: OWNER
body:
  This is the SQL view for the atom feed:
  ```sql
  CREATE VIEW hacker_news_posts_atom as select
    id as atom_id,
    title as atom_title,
    url,
    commentsUrl as atom_link,
    dt || 'Z' as atom_updated,
    'Submitter: ' || submitter || ' - ' || points || ' points, ' || numComments || ' comments' as atom_content
  from
    hacker_news_posts
  order by
    dt desc
  limit
    100;
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1928#issuecomment-1335870883
issue_url: https://api.github.com/repos/simonw/datasette/issues/1928
node_id: IC_kwDOBm6k_c5Pn8Wj
created_at: 2022-12-02T21:19:10Z
updated_at: 2022-12-02T21:19:10Z
author_association: OWNER
body:
  I created the `hacker_news_posts` table like this:

  ```bash
  export ROWS=$(
    jq -n --argjson rows "$(cat simonwillison-net.json)" \
    '{ "table": "hacker_news_posts", "rows": $rows, "pk": "id", "replace": true }'
  )
  # Use curl to POST some JSON to a URL
  curl -X POST \
    https://simon.datasette.cloud/data/-/create \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $DS_TOKEN" \
    -d $ROWS
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1928#issuecomment-1335870884
issue_url: https://api.github.com/repos/simonw/datasette/issues/1928
node_id: IC_kwDOBm6k_c5Pn8Wk
created_at: 2022-12-02T21:19:58Z
updated_at: 2022-12-02T21:19:58Z
author_association: OWNER
body:
  But until I fix this issue:
  - https://github.com/simonw/datasette/issues/1927

  I need to insert freshly scraped data like this:

  ```bash
  export ROWS=$(
    jq -n --argjson rows "$(cat simonwillison-net.json)" \
    '{ "rows": $rows, "replace": true }'
  )

  curl -X POST \
    https://simon.datasette.cloud/data/hacker_news_posts/-/insert \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $DS_TOKEN" \
    -d $ROWS
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1928#issuecomment-1335870887
issue_url: https://api.github.com/repos/simonw/datasette/issues/1928
node_id: IC_kwDOBm6k_c5Pn8Wn
created_at: 2022-12-02T21:25:16Z
updated_at: 2022-12-02T21:25:16Z
author_association: OWNER
body:
  Here's the change that should submit data to Datasette Cloud: https://github.com/simonw/scrape-hacker-news-by-domain/commit/848bb7e835a9fb87cd656362591835179cd1dc1b

  I ran `delete from hacker_news_posts` against my instance so https://simon.datasette.cloud/data/hacker_news_posts is now empty.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1928#issuecomment-1335870889
issue_url: https://api.github.com/repos/simonw/datasette/issues/1928
node_id: IC_kwDOBm6k_c5Pn8Wp
created_at: 2022-12-02T21:41:09Z
updated_at: 2022-12-02T21:41:09Z
author_association: OWNER
body:
  Got it working!

  https://simon.datasette.cloud/data/hacker_news_posts

  https://github.com/simonw/scrape-hacker-news-by-domain/blob/main/submit-to-datasette-cloud.sh

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1927#issuecomment-1335984268
issue_url: https://api.github.com/repos/simonw/datasette/issues/1927
node_id: IC_kwDOBm6k_c5PoYCM
created_at: 2022-12-03T00:26:26Z
updated_at: 2022-12-03T00:26:26Z
author_association: OWNER
body:
  Also: the documentation should clarify that you can call this API multiple times when using the `rows` option.

  (It will probably grow `"alter": true` soon too).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1930#issuecomment-1336017976
issue_url: https://api.github.com/repos/simonw/datasette/issues/1930
node_id: IC_kwDOBm6k_c5PogQ4
created_at: 2022-12-03T02:30:21Z
updated_at: 2022-12-03T02:30:21Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1930?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Base: **90.42**% // Head: **90.42**% // No change to project coverage :thumbsup:
  > Coverage data is based on head [(`9928ff1`)](https://codecov.io/gh/simonw/datasette/pull/1930?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`cab5b60`)](https://codecov.io/gh/simonw/datasette/commit/cab5b60e09e94aca820dbec5308446a88c99ea3d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > Patch has no changes to coverable lines.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main    #1930   +/-   ##
  =======================================
    Coverage   90.42%   90.42%           
  =======================================
    Files          36       36           
    Lines        5057     5057           
  =======================================
    Hits         4573     4573           
    Misses        484      484           
  ```



  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  </details>

  [:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1930?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1878#issuecomment-1336073212
issue_url: https://api.github.com/repos/simonw/datasette/issues/1878
node_id: IC_kwDOBm6k_c5Potv8
created_at: 2022-12-03T05:38:49Z
updated_at: 2022-12-03T05:38:49Z
author_association: OWNER
body:
  And on Discord today: https://discord.com/channels/823971286308356157/823971286941302908/1048426072066236536

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1878#issuecomment-1336094381
issue_url: https://api.github.com/repos/simonw/datasette/issues/1878
node_id: IC_kwDOBm6k_c5Poy6t
created_at: 2022-12-03T06:26:25Z
updated_at: 2022-12-03T06:26:25Z
author_association: OWNER
body:
  Initial prototype:
  ```diff
  diff --git a/datasette/app.py b/datasette/app.py
  index 125b4969..282c0984 100644
  --- a/datasette/app.py
  +++ b/datasette/app.py
  @@ -40,7 +40,7 @@ from .views.special import (
       PermissionsDebugView,
       MessagesDebugView,
   )
  -from .views.table import TableView, TableInsertView, TableDropView
  +from .views.table import TableView, TableInsertView, TableUpsertView, TableDropView
   from .views.row import RowView, RowDeleteView, RowUpdateView
   from .renderer import json_renderer
   from .url_builder import Urls
  @@ -1292,6 +1292,10 @@ class Datasette:
               TableInsertView.as_view(self),
               r"/(?P<database>[^\/\.]+)/(?P<table>[^\/\.]+)/-/insert$",
           )
  +        add_route(
  +            TableUpsertView.as_view(self),
  +            r"/(?P<database>[^\/\.]+)/(?P<table>[^\/\.]+)/-/upsert$",
  +        )
           add_route(
               TableDropView.as_view(self),
               r"/(?P<database>[^\/\.]+)/(?P<table>[^\/\.]+)/-/drop$",
  diff --git a/datasette/views/table.py b/datasette/views/table.py
  index 7ba78c11..ae0d6366 100644
  --- a/datasette/views/table.py
  +++ b/datasette/views/table.py
  @@ -1074,9 +1074,15 @@ class TableInsertView(BaseView):
       def __init__(self, datasette):
           self.ds = datasette
   
  -    async def _validate_data(self, request, db, table_name):
  +    async def _validate_data(self, request, db, table_name, pks, upsert):
           errors = []
   
  +        pks_list = []
  +        if isinstance(pks, str):
  +            pks_list = [pks]
  +        else:
  +            pks_list = list(pks)
  +
           def _errors(errors):
               return None, errors, {}
   
  @@ -1135,6 +1141,15 @@ class TableInsertView(BaseView):
           # Validate columns of each row
           columns = set(await db.table_columns(table_name))
           for i, row in enumerate(rows):
  +            if upsert:
  +                # It MUST have the primary key
  +                missing_pks = [pk for pk in pks_list if pk not in row]
  +                if missing_pks:
  +                    errors.append(
  +                        'Row {} is missing primary key column(s): "{}"'.format(
  +                            i, '", "'.join(missing_pks)
  +                        )
  +                    )
               invalid_columns = set(row.keys()) - columns
               if invalid_columns:
                   errors.append(
  @@ -1146,7 +1161,7 @@ class TableInsertView(BaseView):
               return _errors(errors)
           return rows, errors, extras
   
  -    async def post(self, request):
  +    async def post(self, request, upsert=False):
           try:
               resolved = await self.ds.resolve_table(request)
           except NotFound as e:
  @@ -1164,7 +1179,12 @@ class TableInsertView(BaseView):
               request.actor, "insert-row", resource=(database_name, table_name)
           ):
               return _error(["Permission denied"], 403)
  -        rows, errors, extras = await self._validate_data(request, db, table_name)
  +
  +        pks = await db.primary_keys(table_name)
  +
  +        rows, errors, extras = await self._validate_data(
  +            request, db, table_name, pks, upsert
  +        )
           if errors:
               return _error(errors, 400)
   
  @@ -1172,15 +1192,19 @@ class TableInsertView(BaseView):
           replace = extras.get("replace")
   
           should_return = bool(extras.get("return", False))
  -        # Insert rows
  -        def insert_rows(conn):
  +
  +        def insert_or_upsert_rows(conn):
               table = sqlite_utils.Database(conn)[table_name]
  +            kwargs = {}
  +            if upsert:
  +                kwargs["pk"] = pks[0] if len(pks) == 1 else pks
  +            else:
  +                kwargs = {"ignore": ignore, "replace": replace}
               if should_return:
                   rowids = []
  +                method = table.upsert if upsert else table.insert
                   for row in rows:
  -                    rowids.append(
  -                        table.insert(row, ignore=ignore, replace=replace).last_rowid
  -                    )
  +                    rowids.append(method(row, **kwargs).last_rowid)
                   return list(
                       table.rows_where(
                           "rowid in ({})".format(",".join("?" for _ in rowids)),
  @@ -1188,10 +1212,11 @@ class TableInsertView(BaseView):
                       )
                   )
               else:
  -                table.insert_all(rows, ignore=ignore, replace=replace)
  +                method_all = table.upsert_all if upsert else table.insert_all
  +                method_all(rows, **kwargs)
   
           try:
  -            rows = await db.execute_write_fn(insert_rows)
  +            rows = await db.execute_write_fn(insert_or_upsert_rows)
           except Exception as e:
               return _error([str(e)])
           result = {"ok": True}
  @@ -1200,6 +1225,13 @@ class TableInsertView(BaseView):
           return Response.json(result, status=201)
   
   
  +class TableUpsertView(TableInsertView):
  +    name = "table-upsert"
  +
  +    async def post(self, request):
  +        return await super().post(request, upsert=True)
  +
  +
   class TableDropView(BaseView):
       name = "table-drop"
   ```
  Manual testing reveals that this mostly works... but it's not doing the right thing for `"return": true` - it always returns an empty list.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1878#issuecomment-1336094470
issue_url: https://api.github.com/repos/simonw/datasette/issues/1878
node_id: IC_kwDOBm6k_c5Poy8G
created_at: 2022-12-03T06:27:13Z
updated_at: 2022-12-03T06:27:13Z
author_association: OWNER
body:
  Tests are going to need to cover both rowid-only and compound primary key tables, including all of the error states.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1878#issuecomment-1336094562
issue_url: https://api.github.com/repos/simonw/datasette/issues/1878
node_id: IC_kwDOBm6k_c5Poy9i
created_at: 2022-12-03T06:27:50Z
updated_at: 2022-12-03T06:29:06Z
author_association: OWNER
body:
  This adds it to the API explorer:

  ```diff
  diff --git a/datasette/views/special.py b/datasette/views/special.py
  index 1f84b094..1b4a9d3c 100644
  --- a/datasette/views/special.py
  +++ b/datasette/views/special.py
  @@ -316,21 +316,37 @@ class ApiExplorerView(BaseView):
                       request.actor, "insert-row", (name, table)
                   ):
                       pks = await db.primary_keys(table)
  -                    table_links.append(
  -                        {
  -                            "path": self.ds.urls.table(name, table) + "/-/insert",
  -                            "method": "POST",
  -                            "label": "Insert rows into {}".format(table),
  -                            "json": {
  -                                "rows": [
  -                                    {
  -                                        column: None
  -                                        for column in await db.table_columns(table)
  -                                        if column not in pks
  -                                    }
  -                                ]
  +                    table_links.extend(
  +                        [
  +                            {
  +                                "path": self.ds.urls.table(name, table) + "/-/insert",
  +                                "method": "POST",
  +                                "label": "Insert rows into {}".format(table),
  +                                "json": {
  +                                    "rows": [
  +                                        {
  +                                            column: None
  +                                            for column in await db.table_columns(table)
  +                                            if column not in pks
  +                                        }
  +                                    ]
  +                                },
                               },
  -                        }
  +                            {
  +                                "path": self.ds.urls.table(name, table) + "/-/upsert",
  +                                "method": "POST",
  +                                "label": "Upsert rows into {}".format(table),
  +                                "json": {
  +                                    "rows": [
  +                                        {
  +                                            column: None
  +                                            for column in await db.table_columns(table)
  +                                            if column not in pks
  +                                        }
  +                                    ]
  +                                },
  +                            },
  +                        ]
                       )
                   if await self.ds.permission_allowed(
                       request.actor, "drop-table", (name, table)
  ```
  Except it doesn't quite, because the example JSON this generates is invalid as it does not include the primary key column.

  (Made me notice that the way example columns are created for `/-/insert` will fail for tables that don't have an auto-incrementing primary key)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1927#issuecomment-1336099368
issue_url: https://api.github.com/repos/simonw/datasette/issues/1927
node_id: IC_kwDOBm6k_c5Po0Io
created_at: 2022-12-03T06:56:36Z
updated_at: 2022-12-03T06:56:36Z
author_association: OWNER
body:
  Neither of these options make sense if you didn't pass a `"pk"`.

  My initial implementation spotted if the `pk` was missing and looked it up from the table, but actually I don't think that makes sense - if you know the table exists and hence don't pass the `pk` you should be using `/-/insert` or `/-/upsert` instead.

  So maybe this work should expanded to include validation that checks if the table exists already - and if it does, confirms that the primary key (and maybe even the columns) are the same as for that existing table.

  Of course if you only send `row` or `rows` then checking `columns` doesn't completely make sense - but we could check that the rows you have sent are equal to or a subset of the columns in the table. We could even check the column types as well, as seen in:
  - #1910 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1927#issuecomment-1336099533
issue_url: https://api.github.com/repos/simonw/datasette/issues/1927
node_id: IC_kwDOBm6k_c5Po0LN
created_at: 2022-12-03T06:57:52Z
updated_at: 2022-12-03T06:57:52Z
author_association: OWNER
body:
  I'm going to push what I have anyway. I'll keep this issue open while I think through the above comment.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1931#issuecomment-1339768422
issue_url: https://api.github.com/repos/simonw/datasette/issues/1931
node_id: IC_kwDOBm6k_c5P2z5m
created_at: 2022-12-06T18:04:59Z
updated_at: 2022-12-06T18:04:59Z
author_association: OWNER
body:
  I realized this API should require both the `insert-row` AND the `update-row` permissions, since calls to it could do either one.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1931#issuecomment-1339784569
issue_url: https://api.github.com/repos/simonw/datasette/issues/1931
node_id: IC_kwDOBm6k_c5P2315
created_at: 2022-12-06T18:16:15Z
updated_at: 2022-12-06T18:17:56Z
author_association: OWNER
body:
  Just noticed the insert API returns `{}` when it should return `{"ok": true}` - will fix that here too.

  UPDATE: no it did that already, it was just the documentation that was wrong.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/516#issuecomment-1339834918
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/516
node_id: IC_kwDOCGYnMM5P3EIm
created_at: 2022-12-06T19:00:18Z
updated_at: 2022-12-06T19:00:35Z
author_association: OWNER
body:
  Right now the command produces no output at all.

  Maybe a `--verbose` mode that writes these numbers to standard error (or even standard output since it's an option)?

  Is there a better name than `--verbose` for this? `--summary` perhaps?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/516#issuecomment-1339837520
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/516
node_id: IC_kwDOCGYnMM5P3ExQ
created_at: 2022-12-06T19:02:30Z
updated_at: 2022-12-06T19:02:30Z
author_association: NONE
body:
  `--verbose`  or `--verbosity=ABC` were the flags I looked for.    Expected to see them at a global level near `--version`.  But only sharing because that's where I looked first, I don't have a strong opinion on the exact wording/location. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/516#issuecomment-1339839767
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/516
node_id: IC_kwDOCGYnMM5P3FUX
created_at: 2022-12-06T19:04:17Z
updated_at: 2022-12-06T19:04:17Z
author_association: NONE
body:
  Current behavior is different when importing via stdin vs. a file. Imports from a file give a progress bar.  For this new request, I'd love to see total imported and total ignored/replaced in both cases. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/516#issuecomment-1339844639
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/516
node_id: IC_kwDOCGYnMM5P3Ggf
created_at: 2022-12-06T19:08:13Z
updated_at: 2022-12-06T19:08:13Z
author_association: NONE
body:
  Reference: tqdm (https://tqdm.github.io/) shows a progress bar when total is known, and falls back to counting units of work done for streams. File input vs. stdin seems like a similar situation. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1929#issuecomment-1339867570
issue_url: https://api.github.com/repos/simonw/datasette/issues/1929
node_id: IC_kwDOBm6k_c5P3MGy
created_at: 2022-12-06T19:22:47Z
updated_at: 2022-12-06T19:22:47Z
author_association: OWNER
body:
  Yeah I should deploy the docs for the alpha versions (the intention is that the docs exactly match the release you are using), thanks for spotting that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1929#issuecomment-1339871933
issue_url: https://api.github.com/repos/simonw/datasette/issues/1929
node_id: IC_kwDOBm6k_c5P3NK9
created_at: 2022-12-06T19:23:48Z
updated_at: 2022-12-06T19:24:17Z
author_association: OWNER
body:
  I can do that on this page: https://readthedocs.org/projects/datasette/versions/?version_filter=1.0

  <img width="861" alt="image" src="https://user-images.githubusercontent.com/9599/206003351-1a480f9c-ffc3-413a-a7c8-ad138c25a2f6.png">

  I'm making them both active and hidden:

  <img width="843" alt="CleanShot 2022-12-06 at 11 23 56@2x" src="https://user-images.githubusercontent.com/9599/206003466-3825bf2f-d58d-4eff-8848-8a28e1d5e19f.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1931#issuecomment-1339906241
issue_url: https://api.github.com/repos/simonw/datasette/issues/1931
node_id: IC_kwDOBm6k_c5P3VjB
created_at: 2022-12-06T19:33:32Z
updated_at: 2022-12-08T01:04:56Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1931?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Base: **90.42**% // Head: **91.77**% // Increases project coverage by **`+1.34%`** :tada:
  > Coverage data is based on head [(`645be0f`)](https://codecov.io/gh/simonw/datasette/pull/1931?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`cab5b60`)](https://codecov.io/gh/simonw/datasette/commit/cab5b60e09e94aca820dbec5308446a88c99ea3d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > Patch coverage: 95.55% of modified lines in pull request are covered.

  > :exclamation: Current head 645be0f differs from pull request most recent head 7cd6fd9. Consider uploading reports for the commit 7cd6fd9 to get more accurate results

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@            Coverage Diff             @@
  ##             main    #1931      +/-   ##
  ==========================================
  + Coverage   90.42%   91.77%   +1.34%     
  ==========================================
    Files          36       36              
    Lines        5057     5019      -38     
  ==========================================
  + Hits         4573     4606      +33     
  + Misses        484      413      -71     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1931?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/views/special.py](https://codecov.io/gh/simonw/datasette/pull/1931/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3NwZWNpYWwucHk=) | `79.41% <0.00%> (ø)` | |
  | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1931/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `92.44% <97.43%> (+0.20%)` | :arrow_up: |
  | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1931/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.42% <100.00%> (+<0.01%)` | :arrow_up: |
  | [datasette/default\_permissions.py](https://codecov.io/gh/simonw/datasette/pull/1931/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RlZmF1bHRfcGVybWlzc2lvbnMucHk=) | `94.81% <100.00%> (+0.07%)` | :arrow_up: |
  | [datasette/views/database.py](https://codecov.io/gh/simonw/datasette/pull/1931/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `95.83% <0.00%> (+17.01%)` | :arrow_up: |

  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  </details>

  [:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1931?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1855#issuecomment-1339952692
issue_url: https://api.github.com/repos/simonw/datasette/issues/1855
node_id: IC_kwDOBm6k_c5P3g40
created_at: 2022-12-06T20:15:50Z
updated_at: 2022-12-06T20:16:00Z
author_association: OWNER
body:
  That commit there https://github.com/simonw/datasette/commit/6da17d5529773dfe41b53ed4ce5a6ecb46ed2457 (which will be squash-merged in a PR later on) made it so that `_r` was correctly copied across from the token to the created actor, and fixed a bug in the code that checks permissions against it for resources.

  I needed that mechanism to write a test that exercised different API permissions.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1931#issuecomment-1339968514
issue_url: https://api.github.com/repos/simonw/datasette/issues/1931
node_id: IC_kwDOBm6k_c5P3kwC
created_at: 2022-12-06T20:28:47Z
updated_at: 2022-12-06T20:28:47Z
author_association: OWNER
body:
  Should the `"return": true` mode reflect the order in which the rows were provided when the API was called?

  I think it should. Since this is small enough to happily fit in Python memory (thanks to the `max_insert_rows` setting) I can load the fresh data from the database and then sort it in Python space before returning it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1935#issuecomment-1340950566
issue_url: https://api.github.com/repos/simonw/datasette/issues/1935
node_id: IC_kwDOBm6k_c5P7Ugm
created_at: 2022-12-07T13:14:41Z
updated_at: 2022-12-07T13:14:41Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1935?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Base: **91.73**% // Head: **91.49**% // Decreases project coverage by **`-0.24%`** :warning:
  > Coverage data is based on head [(`e8ae41e`)](https://codecov.io/gh/simonw/datasette/pull/1935?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`93ababe`)](https://codecov.io/gh/simonw/datasette/commit/93ababe6f7150454d2cf278dae08569e505d2a5b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > Patch has no changes to coverable lines.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@            Coverage Diff             @@
  ##             main    #1935      +/-   ##
  ==========================================
  - Coverage   91.73%   91.49%   -0.25%     
  ==========================================
    Files          36       37       +1     
    Lines        4987     5031      +44     
  ==========================================
  + Hits         4575     4603      +28     
  - Misses        412      428      +16     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1935?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/utils/shutil\_backport.py](https://codecov.io/gh/simonw/datasette/pull/1935/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL3NodXRpbF9iYWNrcG9ydC5weQ==) | `9.09% <0.00%> (ø)` | |
  | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1935/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.78% <0.00%> (+0.36%)` | :arrow_up: |
  | [datasette/plugins.py](https://codecov.io/gh/simonw/datasette/pull/1935/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3BsdWdpbnMucHk=) | `85.29% <0.00%> (+2.94%)` | :arrow_up: |
  | [datasette/utils/asgi.py](https://codecov.io/gh/simonw/datasette/pull/1935/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL2FzZ2kucHk=) | `93.60% <0.00%> (+3.59%)` | :arrow_up: |
  | [datasette/cli.py](https://codecov.io/gh/simonw/datasette/pull/1935/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `82.18% <0.00%> (+4.00%)` | :arrow_up: |

  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  </details>

  [:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1935?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1931#issuecomment-1341825314
issue_url: https://api.github.com/repos/simonw/datasette/issues/1931
node_id: IC_kwDOBm6k_c5P-qEi
created_at: 2022-12-08T01:03:18Z
updated_at: 2022-12-08T01:03:18Z
author_association: OWNER
body:
  I broke this test:
  ```
  ds_write = <datasette.app.Datasette object at 0x7f0965858700>

      @pytest.mark.asyncio
      async def test_insert_row(ds_write):
          token = write_token(ds_write)
          response = await ds_write.client.post(
              "/data/docs/-/insert",
              json={"row": {"title": "Test", "score": 1.2, "age": 5}},
              headers={
                  "Authorization": "***".format(token),
                  "Content-Type": "application/json",
              },
          )
          expected_row = {"id": 1, "title": "Test", "score": 1.2, "age": 5}
  >       assert response.status_code == 201
  E       assert 500 == 201
  E        +  where 500 = <Response [500 Internal Server Error]>.status_code

  /home/runner/work/datasette/datasette/tests/test_api_write.py:43: AssertionError
  ----------------------------- Captured stderr call -----------------------------
  Traceback (most recent call last):
    File "/home/runner/work/datasette/datasette/datasette/app.py", line 1447, in route_path
      response = await view(request, send)
    File "/home/runner/work/datasette/datasette/datasette/views/base.py", line 151, in view
      return await self.dispatch_request(request)
    File "/home/runner/work/datasette/datasette/datasette/views/base.py", line 105, in dispatch_request
      response = await handler(request)
    File "/home/runner/work/datasette/datasette/datasette/views/table.py", line 1228, in post
      row_pk_values_for_later = [tuple(row[pk] for pk in pks) for row in rows]
    File "/home/runner/work/datasette/datasette/datasette/views/table.py", line 1228, in <listcomp>
      row_pk_values_for_later = [tuple(row[pk] for pk in pks) for row in rows]
    File "/home/runner/work/datasette/datasette/datasette/views/table.py", line 1228, in <genexpr>
      row_pk_values_for_later = [tuple(row[pk] for pk in pks) for row in rows]
  KeyError: 'id'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1937#issuecomment-1341848525
issue_url: https://api.github.com/repos/simonw/datasette/issues/1937
node_id: IC_kwDOBm6k_c5P-vvN
created_at: 2022-12-08T01:34:03Z
updated_at: 2022-12-08T01:34:03Z
author_association: OWNER
body:
  Check should go somewhere around here: https://github.com/simonw/datasette/blob/dee18ed8ce7af2ab8699bcb5a51a99f48301bc42/datasette/views/database.py#L625-L631

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1936#issuecomment-1341849496
issue_url: https://api.github.com/repos/simonw/datasette/issues/1936
node_id: IC_kwDOBm6k_c5P-v-Y
created_at: 2022-12-08T01:35:35Z
updated_at: 2022-12-08T01:35:35Z
author_association: OWNER
body:
  Related bug: you can send `"id": null` and it works (it should throw an error):

  <img width="1006" alt="image" src="https://user-images.githubusercontent.com/9599/206334362-5727f3e8-8b6b-4f05-b876-47b060b81248.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1936#issuecomment-1341849735
issue_url: https://api.github.com/repos/simonw/datasette/issues/1936
node_id: IC_kwDOBm6k_c5P-wCH
created_at: 2022-12-08T01:35:54Z
updated_at: 2022-12-08T01:35:54Z
author_association: OWNER
body:
  Running that twice produced this:

  <img width="182" alt="image" src="https://user-images.githubusercontent.com/9599/206334432-b577fcee-f44b-440c-9f10-a5f5808e91cf.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1867#issuecomment-1341874378
issue_url: https://api.github.com/repos/simonw/datasette/issues/1867
node_id: IC_kwDOBm6k_c5P-2DK
created_at: 2022-12-08T02:07:06Z
updated_at: 2022-12-08T02:28:02Z
author_association: OWNER
body:
  Basic version of this allows you to rename a table:

  ```
  POST /db/table/-/rename
  {
      "name": "new_table_name"
  }
  ```
  If a table with that new name already exists you will get an error - unless you pass `"replace": true` in which case that table will be dropped and replaced by the new one.


  ```
  POST /db/table/-/rename
  {
      "name": "new_table_name",
      "replace": true
  }
  ```

  This is useful because it allows for that atomic replacement operation: upload brand new data into a `_tmp_name` table, then atomic rename and replace after the upload has completed.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1636#issuecomment-1343440504
issue_url: https://api.github.com/repos/simonw/datasette/issues/1636
node_id: IC_kwDOBm6k_c5QE0Z4
created_at: 2022-12-08T22:10:28Z
updated_at: 2022-12-08T22:10:48Z
author_association: OWNER
body:
  What if you want to grant `insert-row` to a user for ALL tables in a database, or even for all tables in all databases?

  You should be able to do that by putting that in the root `permissions:` block. Need to figure out how the implementation will handle that.

  Also: there are some permissions like `view-instance` or `debug-menu` for which putting them at the `database` or `table` or `query` level doesn't actually make any sense.

  Ideally the implementation would spot those on startup and refuse to start the server, with a helpful error message.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1938#issuecomment-1343445885
issue_url: https://api.github.com/repos/simonw/datasette/issues/1938
node_id: IC_kwDOBm6k_c5QE1t9
created_at: 2022-12-08T22:16:03Z
updated_at: 2022-12-08T22:16:03Z
author_association: OWNER
body:
  Docs: https://datasette--1938.org.readthedocs.build/en/1938/authentication.html#other-permissions-in-metadata

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1636#issuecomment-1343446071
issue_url: https://api.github.com/repos/simonw/datasette/issues/1636
node_id: IC_kwDOBm6k_c5QE1w3
created_at: 2022-12-08T22:16:17Z
updated_at: 2022-12-08T22:16:17Z
author_association: OWNER
body:
  First draft of documentation: https://datasette--1938.org.readthedocs.build/en/1938/authentication.html#other-permissions-in-metadata

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1938#issuecomment-1343449918
issue_url: https://api.github.com/repos/simonw/datasette/issues/1938
node_id: IC_kwDOBm6k_c5QE2s-
created_at: 2022-12-08T22:20:10Z
updated_at: 2022-12-08T22:54:08Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1938?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Base: **92.00**% // Head: **92.01**% // Increases project coverage by **`+0.01%`** :tada:
  > Coverage data is based on head [(`6e35a6b`)](https://codecov.io/gh/simonw/datasette/pull/1938?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`e539c1c`)](https://codecov.io/gh/simonw/datasette/commit/e539c1c024bc62d88df91d9107cbe37e7f0fe55f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > Patch coverage: 100.00% of modified lines in pull request are covered.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@            Coverage Diff             @@
  ##             main    #1938      +/-   ##
  ==========================================
  + Coverage   92.00%   92.01%   +0.01%     
  ==========================================
    Files          38       38              
    Lines        5378     5386       +8     
  ==========================================
  + Hits         4948     4956       +8     
    Misses        430      430              
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1938?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/default\_permissions.py](https://codecov.io/gh/simonw/datasette/pull/1938/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RlZmF1bHRfcGVybWlzc2lvbnMucHk=) | `95.10% <100.00%> (+0.29%)` | :arrow_up: |

  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  </details>

  [:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1938?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1636#issuecomment-1343715746
issue_url: https://api.github.com/repos/simonw/datasette/issues/1636
node_id: IC_kwDOBm6k_c5QF3mi
created_at: 2022-12-09T01:27:41Z
updated_at: 2022-12-09T01:27:58Z
author_association: OWNER
body:
  I may need to consult this file to figure out if the permission that is being checked can act at the database/table/instance level:

  https://github.com/simonw/datasette/blob/e539c1c024bc62d88df91d9107cbe37e7f0fe55f/datasette/permissions.py#L1-L19

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1939#issuecomment-1343721522
issue_url: https://api.github.com/repos/simonw/datasette/issues/1939
node_id: IC_kwDOBm6k_c5QF5Ay
created_at: 2022-12-09T01:35:15Z
updated_at: 2022-12-09T01:35:15Z
author_association: OWNER
body:
  One concern I have about this: there are a bunch of existing plugins that do stuff with permissions that won't currently be using this hook.

  Do I break those plugins, forcing new releases of them for compatibility with Datasette 1.0?

  Or maybe I keep them working, but until they've upgraded to register their permissions there are things about them that won't work - e.g. you won't be able to configure their permissions in `metadata.yml` until they release something that does this hook.

  Best thing is probably for me to get this working in core first and then evaluate the impact it would have on existing plugins once I have some running code.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1939#issuecomment-1343722020
issue_url: https://api.github.com/repos/simonw/datasette/issues/1939
node_id: IC_kwDOBm6k_c5QF5Ik
created_at: 2022-12-09T01:36:05Z
updated_at: 2022-12-09T01:36:16Z
author_association: OWNER
body:
  I originally added `permissions.py` for the permission debug tool in https://github.com/simonw/datasette/commit/c51d9246b996a2831c9bd6a1e205f6cb48b9a5f3 - I don't think anything else uses it yet.

  - #1881

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1939#issuecomment-1343724732
issue_url: https://api.github.com/repos/simonw/datasette/issues/1939
node_id: IC_kwDOBm6k_c5QF5y8
created_at: 2022-12-09T01:40:44Z
updated_at: 2022-12-09T01:43:25Z
author_association: OWNER
body:
  ```python
  Permission = collections.namedtuple( 
      "Permission", ("name", "abbr", "takes_database", "takes_table", "default") 
  ) 
  ```
  I don`t think that design is quite right.

  - Elsewhere in the code the concept is called an "action" rather than a "permission" - I think I can stick with the `Permission` name here though, it's pretty clear
  - `takes_database` - is `takes_` the right verb here?
  - `takes_table` can also refer to a SQL view or a canned named query

  A question that was raised by the work in #1938 is whether you should be able to grant a permission like `insert-row` at the instance or database level - and if so, what does that look like? I think you should be able to do that, it doesn't make sense to have to grant it explicitly for every single table.

  So maybe `takes_table` and `takes_database` are the right names here? But `table` is still bad because it doesn't reflect views and canned queries.

  One thought is to use `resource` - but that will require a bunch of breaking changes to the existing APIs which treat resource as a tuple. Now's the best time to do that though before Datasette 1.0.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1939#issuecomment-1343727184
issue_url: https://api.github.com/repos/simonw/datasette/issues/1939
node_id: IC_kwDOBm6k_c5QF6ZQ
created_at: 2022-12-09T01:45:15Z
updated_at: 2022-12-09T01:45:15Z
author_association: OWNER
body:
  Moving the concept of the default for the permission into this registry warrants a redesign of this method anyway:

  https://github.com/simonw/datasette/blob/e539c1c024bc62d88df91d9107cbe37e7f0fe55f/datasette/app.py#L706

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1939#issuecomment-1343728929
issue_url: https://api.github.com/repos/simonw/datasette/issues/1939
node_id: IC_kwDOBm6k_c5QF60h
created_at: 2022-12-09T01:48:11Z
updated_at: 2022-12-09T01:52:33Z
author_association: OWNER
body:
  This code search shows a bunch of repos I don't know about that would be affected by this change: https://cs.github.com/?scopeName=All+repos&scope=&q=datasette+permission_allowed+-user%3Asimonw#

  These (and likely more):

  Repositories

  - [20after4/ddd](https://github.com/20after4/ddd)
  - [next-LI/datasette-csv-importer](https://github.com/next-LI/datasette-csv-importer)
  - [digital-land/datasette](https://github.com/digital-land/datasette)
  - [mroswell/datasette](https://github.com/mroswell/datasette)
  - [next-LI/datasette-live-config](https://github.com/next-LI/datasette-live-config)
  - [keladhruv/datasette](https://github.com/keladhruv/datasette)
  - [RhetTbull/datasette](https://github.com/RhetTbull/datasette)
  - [chriswedgwood/datasette](https://github.com/chriswedgwood/datasette)
  - [boan-anbo/datasette](https://github.com/boan-anbo/datasette)
  - [MattTriano/datasette](https://github.com/MattTriano/datasette)
  - [incadenza/datasette](https://github.com/incadenza/datasette)
  - [robdyke/datasette](https://github.com/robdyke/datasette)
  - [ctb/datasette](https://github.com/ctb/datasette)
  - [eyeseast/datasette](https://github.com/eyeseast/datasette)
  - [symbol-management/api-match-audit](https://github.com/symbol-management/api-match-audit)

  Actually a lot of those are forks of Datasette itself - so maybe this is manageable?

  Would be nice if I could come up with a GitHub search that excluded any repos with "datasette" as their exact name.

  https://docs.github.com/en/search-github/github-code-search/understanding-github-code-search-syntax#using-qualifiers says:

  > **Note:** The new code search beta does not currently support regular expressions or partial matching for repository names, so you will have to type the entire repository name (including the user prefix) for the `repo:` qualifier to work.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1939#issuecomment-1343734812
issue_url: https://api.github.com/repos/simonw/datasette/issues/1939
node_id: IC_kwDOBm6k_c5QF8Qc
created_at: 2022-12-09T01:57:07Z
updated_at: 2022-12-09T01:57:07Z
author_association: OWNER
body:
  This search is better:

      datasette permission_allowed -user:simonw -path:datasette/** -path:docs/** -path:tests/** language:python

  That returns 11 results: https://cs.github.com/?scopeName=All+repos&scope=&q=datasette+permission_allowed+-user%3Asimonw+-path%3Adatasette%2F**+-path%3Adocs%2F**+-path%3Atests%2F**+language%3Apython

  3 are forks of my repos. The rest are all by four users:

  - [20after4/ddd](https://github.com/20after4/ddd)
  - [emg110/datasette-graphql](https://github.com/emg110/datasette-graphql)
  - [next-LI/datasette-csv-importer](https://github.com/next-LI/datasette-csv-importer)
  - [next-LI/datasette-demo](https://github.com/next-LI/datasette-demo)
  - [next-LI/datasette-live-config](https://github.com/next-LI/datasette-live-config)
  - [next-LI/datasette-live-permissions](https://github.com/next-LI/datasette-live-permissions)
  - [next-LI/datasette-search-all](https://github.com/next-LI/datasette-search-all)
  - [next-LI/datasette-surveys](https://github.com/next-LI/datasette-surveys)
  - [next-LI/datasette-write](https://github.com/next-LI/datasette-write)
  - [rclement/datasette-dashboards](https://github.com/rclement/datasette-dashboards)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1939#issuecomment-1343749617
issue_url: https://api.github.com/repos/simonw/datasette/issues/1939
node_id: IC_kwDOBm6k_c5QF_3x
created_at: 2022-12-09T02:15:54Z
updated_at: 2022-12-09T02:15:54Z
author_association: OWNER
body:
  What if I came up with a new method name for this, which could co-exist with the old one while that old one was deprecated?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1939#issuecomment-1343751261
issue_url: https://api.github.com/repos/simonw/datasette/issues/1939
node_id: IC_kwDOBm6k_c5QGARd
created_at: 2022-12-09T02:17:14Z
updated_at: 2022-12-09T02:17:14Z
author_association: OWNER
body:
  One option:
  ```python
  async def action_allowed(self, actor, action, database=None, resource=None):
  ```
  `action_allowed` fixes the `permission` v.s. `action` thing a bit, and is a new name that doesn't clash with the existing method. I dropped `default` because that's now a property of the permission itself. `table` is now called `resource` and `database` is a separate parameter.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1939#issuecomment-1343751860
issue_url: https://api.github.com/repos/simonw/datasette/issues/1939
node_id: IC_kwDOBm6k_c5QGAa0
created_at: 2022-12-09T02:18:11Z
updated_at: 2022-12-09T02:19:23Z
author_association: OWNER
body:
  Should I rename "permission" to "action" elsewhere too? Maybe have a `register_actions(...)` plugin hook instead of adding `register_permissions(...)`? 

  What else could the word "action" mean?

  Currently it's used in the codebase to refer to GitHub Actions, and for code like this:
  ```python
          if await self.permission_allowed(
              actor=actor, action="view-instance", default=True
          ):
  ```
  Which is already revealing the confusion between "permission" and "action".

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1939#issuecomment-1343753386
issue_url: https://api.github.com/repos/simonw/datasette/issues/1939
node_id: IC_kwDOBm6k_c5QGAyq
created_at: 2022-12-09T02:20:20Z
updated_at: 2022-12-09T02:21:01Z
author_association: OWNER
body:
  It's also referenced in this plugin hook:
  ```python
  @hookspec
  def permission_allowed(datasette, actor, action, resource):
      """Check if actor is allowed to perform this action - return True, False or None"""
  ```
  But more importantly, in these ones:
  ```python
  @hookspec
  def table_actions(datasette, actor, database, table, request):
      """Links for the table actions menu"""


  @hookspec
  def database_actions(datasette, actor, database, request):
      """Links for the database actions menu"""
  ```
  So the word "action" is already used within Datasette to refer to those things - which are _almost_ but not quite the same as actions-as-permissions: many of the things that show up in those menus relate to permissions the user has, but not necessarily all of them.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1939#issuecomment-1343842362
issue_url: https://api.github.com/repos/simonw/datasette/issues/1939
node_id: IC_kwDOBm6k_c5QGWg6
created_at: 2022-12-09T04:43:38Z
updated_at: 2022-12-09T04:43:38Z
author_association: OWNER
body:
  Asked ChatGPT for some alternative names, I didn't like any of them:

      is_permission_granted
      has_permission
      check_permission
      is_action_allowed
      check_access_permission
      permission_check
      validate_permission
      check_actor_permission
      verify_permission
      check_authorization

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1939#issuecomment-1343844112
issue_url: https://api.github.com/repos/simonw/datasette/issues/1939
node_id: IC_kwDOBm6k_c5QGW8Q
created_at: 2022-12-09T04:47:28Z
updated_at: 2022-12-09T04:47:28Z
author_association: OWNER
body:
  I think `action_allowed` is my favourite, even though there's a little bit of concept overlap with `table_actions` and `database_actions`.

  I never really liked those plugin hook names much to be honest, especially since they are inconsistent with `menu_links`:

  https://github.com/simonw/datasette/blob/d67f812b7327c7075732688f3df728807503dc58/datasette/hookspecs.py#L123-L135

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1939#issuecomment-1343844555
issue_url: https://api.github.com/repos/simonw/datasette/issues/1939
node_id: IC_kwDOBm6k_c5QGXDL
created_at: 2022-12-09T04:48:28Z
updated_at: 2022-12-09T04:48:28Z
author_association: OWNER
body:
  I'm going to try a spike in a branch with `datasette.action_allowed(...)` and a `register_permissions()` plugin hook, to see what they look like.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1939#issuecomment-1343858998
issue_url: https://api.github.com/repos/simonw/datasette/issues/1939
node_id: IC_kwDOBm6k_c5QGak2
created_at: 2022-12-09T05:12:17Z
updated_at: 2022-12-09T05:12:17Z
author_association: OWNER
body:
  Draft docs for the new plugin hook: https://datasette--1940.org.readthedocs.build/en/1940/plugin_hooks.html#register-permissions-datasette

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1941#issuecomment-1343869900
issue_url: https://api.github.com/repos/simonw/datasette/issues/1941
node_id: IC_kwDOBm6k_c5QGdPM
created_at: 2022-12-09T05:25:19Z
updated_at: 2022-12-09T05:25:19Z
author_association: OWNER
body:
  I don't plan to implement this for Datasette 1.0, but it should be something that can be added later on in a minor version bump.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1939#issuecomment-1343872168
issue_url: https://api.github.com/repos/simonw/datasette/issues/1939
node_id: IC_kwDOBm6k_c5QGdyo
created_at: 2022-12-09T05:29:53Z
updated_at: 2022-12-09T05:29:53Z
author_association: OWNER
body:
  I'm going to address those ideas for changes to the `permission_allowed()` in a separate issue.

  What would it take for the `register_permissions()` hook to be something I'm comfortable landing?

  I think it's mainly that the list of permissions it provides should Do More Stuff:

  - Participate in unit tests, in particular this one:
     
     https://github.com/simonw/datasette/blob/e539c1c024bc62d88df91d9107cbe37e7f0fe55f/tests/conftest.py#L79-L102

  - That new `default` option should be respected - maybe if you omit `default=` from a call to `permission_allowed()` it could fall back on the default from there?
  - Log a warning if you attempt to check a permission that wasn't registered

  Then I can use the permissions - in particular their metadata - to help implement his:

  - #1636

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1942#issuecomment-1344959032
issue_url: https://api.github.com/repos/simonw/datasette/issues/1942
node_id: IC_kwDOBm6k_c5QKnI4
created_at: 2022-12-10T01:11:26Z
updated_at: 2022-12-10T01:11:26Z
author_association: OWNER
body:
  One way this could work: if plugins request it, a block like this is added to the page:

  ```html
  <script type="application/json+datasette" id="datasette-json">
  {
    "...": "..."
  }
  </script>
  ```
  Then a function could be provided which extracts and parses that data:
  ```javascript
  var data = await datasette.jsonData();
  ```
  Why an `await`? Because then I could have it work the exact same way if the data is NOT available on the page - it could trigger a `fetch()` call for the same stuff. So loading it on the page becomes an optional performance optimization.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/517#issuecomment-1344965367
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/517
node_id: IC_kwDOCGYnMM5QKor3
created_at: 2022-12-10T01:26:31Z
updated_at: 2022-12-10T01:26:31Z
author_association: OWNER
body:
  At some point I should drop it from all of these other projects too: https://cs.github.com/?scopeName=All+repos&scope=&q=user%3Asimonw+%223.6%22+path%3A.github%2Fworkflows%2F*

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1940#issuecomment-1345701316
issue_url: https://api.github.com/repos/simonw/datasette/issues/1940
node_id: IC_kwDOBm6k_c5QNcXE
created_at: 2022-12-12T00:10:59Z
updated_at: 2022-12-12T00:10:59Z
author_association: OWNER
body:
  Here's my first test failure:
  ```
  tests/test_permissions.py .......F
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

  allow = {}, expected_anon = 403, expected_auth = 403, path = '/fixtures/compound_three_primary_keys'
  padlock_client = <datasette.utils.testing.TestClient object at 0x107c09d20>

      @pytest.mark.parametrize(
          "allow,expected_anon,expected_auth",
          [
              (None, 200, 200),
              ({}, 403, 403),
              ({"id": "root"}, 403, 200),
          ],
      )
      @pytest.mark.parametrize(
          "path",
          (
              "/",
              "/fixtures",
              "/fixtures/compound_three_primary_keys",
              "/fixtures/compound_three_primary_keys/a,a,a",
              "/fixtures/two",  # Query
          ),
      )
      def test_view_padlock(allow, expected_anon, expected_auth, path, padlock_client):
          padlock_client.ds._metadata_local["allow"] = allow
          fragment = "🔒</h1>"
          anon_response = padlock_client.get(path)
  >       assert expected_anon == anon_response.status
  E       assert 403 == 200
  E        +  where 200 = <datasette.utils.testing.TestResponse object at 0x107aea680>.status

  /Users/simon/Dropbox/Development/datasette/tests/test_permissions.py:61: AssertionError
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  > /Users/simon/Dropbox/Development/datasette/tests/test_permissions.py(61)test_view_padlock()
  -> assert expected_anon == anon_response.status
  (Pdb) anon_response
  <datasette.utils.testing.TestResponse object at 0x107aea680>
  (Pdb) anon_response.status
  200
  (Pdb) path
  '/fixtures/compound_three_primary_keys'
  (Pdb) padlock_client.ds._metadata_
  *** AttributeError: 'Datasette' object has no attribute '_metadata_'
  (Pdb) padlock_client.ds._metadata_local
  {'databases': {'fixtures': {'queries': {'two': {'sql': 'select 1 + 1', 'name': 'two'}, 'from_async_hook': {'sql': 'select 2', 'name': 'from_async_hook'}, 'from_hook': {'sql': "select 1, 'null' as actor_id", 'name': 'from_hook'}}, 'source': None, 'source_url': None, 'license': None, 'license_url': None, 'about': None, 'about_url': None}}, 'allow': {}}
  (Pdb) allow
  {}
  ```
  It looks like I've broken the `allow` logic that notices that if there's an `"allow": {}` on the root then anonymous users should not be allowed to view any pages.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1940#issuecomment-1347616055
issue_url: https://api.github.com/repos/simonw/datasette/issues/1940
node_id: IC_kwDOBm6k_c5QUv03
created_at: 2022-12-13T01:27:03Z
updated_at: 2022-12-13T01:27:03Z
author_association: OWNER
body:
  I'm going to revert that last commit, see if I can get the tests running again and then apply the changes a line at a time to figure out which ones broke things.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1940#issuecomment-1347620733
issue_url: https://api.github.com/repos/simonw/datasette/issues/1940
node_id: IC_kwDOBm6k_c5QUw99
created_at: 2022-12-13T01:33:06Z
updated_at: 2022-12-13T01:33:06Z
author_association: OWNER
body:
  It's this change which triggers the failures:
  ```diff
  diff --git a/datasette/app.py b/datasette/app.py
  index 760063d5..defa9688 100644
  --- a/datasette/app.py
  +++ b/datasette/app.py
  @@ -707,9 +707,12 @@ class Datasette:
                   )
           return crumbs
   
  -    async def permission_allowed(self, actor, action, resource=None, default=False):
  +    async def permission_allowed(self, actor, action, resource=None, default=None):
           """Check permissions using the permissions_allowed plugin hook"""
           result = None
  +        # Use default from registered permission, if available
  +        if default is None and action in self.permissions:
  +            default = self.permissions[action].default
           for check in pm.hook.permission_allowed(
               datasette=self,
               actor=actor,
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1940#issuecomment-1347632350
issue_url: https://api.github.com/repos/simonw/datasette/issues/1940
node_id: IC_kwDOBm6k_c5QUzze
created_at: 2022-12-13T01:48:40Z
updated_at: 2022-12-13T02:00:52Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1940?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Base: **92.00**% // Head: **92.03**% // Increases project coverage by **`+0.02%`** :tada:
  > Coverage data is based on head [(`a1317ab`)](https://codecov.io/gh/simonw/datasette/pull/1940?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`e539c1c`)](https://codecov.io/gh/simonw/datasette/commit/e539c1c024bc62d88df91d9107cbe37e7f0fe55f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > Patch coverage: 100.00% of modified lines in pull request are covered.

  > :exclamation: Current head a1317ab differs from pull request most recent head 94e5c75. Consider uploading reports for the commit 94e5c75 to get more accurate results

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@            Coverage Diff             @@
  ##             main    #1940      +/-   ##
  ==========================================
  + Coverage   92.00%   92.03%   +0.02%     
  ==========================================
    Files          38       38              
    Lines        5378     5396      +18     
  ==========================================
  + Hits         4948     4966      +18     
    Misses        430      430              
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1940?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/permissions.py](https://codecov.io/gh/simonw/datasette/pull/1940/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3Blcm1pc3Npb25zLnB5) | `100.00% <ø> (ø)` | |
  | [datasette/views/database.py](https://codecov.io/gh/simonw/datasette/pull/1940/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `96.26% <ø> (ø)` | |
  | [datasette/views/index.py](https://codecov.io/gh/simonw/datasette/pull/1940/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2luZGV4LnB5) | `96.49% <ø> (ø)` | |
  | [datasette/views/special.py](https://codecov.io/gh/simonw/datasette/pull/1940/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3NwZWNpYWwucHk=) | `79.20% <ø> (-0.21%)` | :arrow_down: |
  | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1940/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `92.57% <ø> (ø)` | |
  | [datasette/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1940/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL19faW5pdF9fLnB5) | `100.00% <100.00%> (ø)` | |
  | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1940/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.47% <100.00%> (+0.04%)` | :arrow_up: |
  | [datasette/default\_permissions.py](https://codecov.io/gh/simonw/datasette/pull/1940/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RlZmF1bHRfcGVybWlzc2lvbnMucHk=) | `95.20% <100.00%> (+0.39%)` | :arrow_up: |
  | [datasette/hookspecs.py](https://codecov.io/gh/simonw/datasette/pull/1940/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2hvb2tzcGVjcy5weQ==) | `100.00% <100.00%> (ø)` | |

  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  </details>

  [:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1940?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1940#issuecomment-1347634128
issue_url: https://api.github.com/repos/simonw/datasette/issues/1940
node_id: IC_kwDOBm6k_c5QU0PQ
created_at: 2022-12-13T01:51:56Z
updated_at: 2022-12-13T01:51:56Z
author_association: OWNER
body:
  Actually one last thing: I said that the error would only occur if the permissions differed in some way.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1940#issuecomment-1347640542
issue_url: https://api.github.com/repos/simonw/datasette/issues/1940
node_id: IC_kwDOBm6k_c5QU1ze
created_at: 2022-12-13T02:02:10Z
updated_at: 2022-12-13T02:02:10Z
author_association: OWNER
body:
  This PR ended up bundling part of the implementation of:
  - #1636

  I'm going to be bad an NOT untangle that from this before I merge it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1943#issuecomment-1347645615
issue_url: https://api.github.com/repos/simonw/datasette/issues/1943
node_id: IC_kwDOBm6k_c5QU3Cv
created_at: 2022-12-13T02:06:47Z
updated_at: 2022-12-13T02:06:47Z
author_association: OWNER
body:
  This URL is already used for the https://latest.datasette.io/-/permissions tool - but it could include a block on that page that tells you what permissions are available.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1939#issuecomment-1347646516
issue_url: https://api.github.com/repos/simonw/datasette/issues/1939
node_id: IC_kwDOBm6k_c5QU3Q0
created_at: 2022-12-13T02:07:50Z
updated_at: 2022-12-13T02:07:50Z
author_association: OWNER
body:
  Documentation for the new hook: https://docs.datasette.io/en/latest/plugin_hooks.html#register-permissions-datasette

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1636#issuecomment-1347647298
issue_url: https://api.github.com/repos/simonw/datasette/issues/1636
node_id: IC_kwDOBm6k_c5QU3dC
created_at: 2022-12-13T02:08:46Z
updated_at: 2022-12-13T02:08:46Z
author_association: OWNER
body:
  A bunch of the work for this just landed - in particular the new scheme is now documented (even though it doesn't work yet):

  https://docs.datasette.io/en/latest/authentication.html#other-permissions-in-metadata

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1636#issuecomment-1347648326
issue_url: https://api.github.com/repos/simonw/datasette/issues/1636
node_id: IC_kwDOBm6k_c5QU3tG
created_at: 2022-12-13T02:10:02Z
updated_at: 2022-12-13T02:10:02Z
author_association: OWNER
body:
  The implementation for this will go here: https://github.com/simonw/datasette/blob/8bf06a76b51bc9ace7cf72cf0cca8f1da7704ea7/datasette/default_permissions.py#L81-L83

  Here's the start of the tests (currently marked as `xfail`):

  https://github.com/simonw/datasette/blob/8bf06a76b51bc9ace7cf72cf0cca8f1da7704ea7/tests/test_permissions.py#L652-L689

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1636#issuecomment-1347655074
issue_url: https://api.github.com/repos/simonw/datasette/issues/1636
node_id: IC_kwDOBm6k_c5QU5Wi
created_at: 2022-12-13T02:21:04Z
updated_at: 2022-12-13T02:21:23Z
author_association: OWNER
body:
  The thing I'm stuck on at the moment is how to implement it such that an `allow` block for `create-table` at the root of the metadata will be checked correctly.

  Maybe the algorithm when `_resolve_metadata_permissions_blocks(datasette, actor, action, resource)` is called should do this:

  1. If a root permission block matching that action exists, test with that
  2. Next, if resource has been passed, check at the database level
  3. If the resource included a table/query, check at that level too

  So everything is keyed off the incoming `action` name.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1855#issuecomment-1347675456
issue_url: https://api.github.com/repos/simonw/datasette/issues/1855
node_id: IC_kwDOBm6k_c5QU-VA
created_at: 2022-12-13T02:57:46Z
updated_at: 2022-12-13T02:57:46Z
author_association: OWNER
body:
  I was going to have the CLI command throw an error if you attempt to use a permission that isn't registered with Datasette, but then I remembered that one of the uses for the CLI tool is to create signed tokens that will work against other Datasette instances (via the `--secret` option) that might have different plugins installed that register different permission names.

  So I might have it output warnings instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1855#issuecomment-1347694871
issue_url: https://api.github.com/repos/simonw/datasette/issues/1855
node_id: IC_kwDOBm6k_c5QVDEX
created_at: 2022-12-13T03:28:15Z
updated_at: 2022-12-13T03:28:15Z
author_association: OWNER
body:
  Initial prototype of the `create-token` command changes:

  ```diff
  diff --git a/datasette/default_permissions.py b/datasette/default_permissions.py
  index 406dae40..bbe1247e 100644
  --- a/datasette/default_permissions.py
  +++ b/datasette/default_permissions.py
  @@ -278,17 +278,55 @@ def register_commands(cli):
           help="Token should expire after this many seconds",
           type=int,
       )
  +    @click.option(
  +        "alls",
  +        "-a",
  +        "--all",
  +        type=str,
  +        multiple=True,
  +        help="Restrict token to this permission",
  +    )
  +    @click.option(
  +        "databases",
  +        "-d",
  +        "--database",
  +        type=(str, str),
  +        multiple=True,
  +        help="Restrict token to this permission on this database",
  +    )
  +    @click.option(
  +        "resources",
  +        "-r",
  +        "--resource",
  +        type=(str, str, str),
  +        multiple=True,
  +        help="Restrict token to this permission on this database resource (a table, SQL view or named query)",
  +    )
       @click.option(
           "--debug",
           help="Show decoded token",
           is_flag=True,
       )
  -    def create_token(id, secret, expires_after, debug):
  +    def create_token(id, secret, expires_after, alls, databases, resources, debug):
           "Create a signed API token for the specified actor ID"
           ds = Datasette(secret=secret)
           bits = {"a": id, "token": "dstok", "t": int(time.time())}
           if expires_after:
               bits["d"] = expires_after
  +        if alls or databases or resources:
  +            bits["_r"] = {}
  +            if alls:
  +                bits["_r"]["a"] = list(alls)
  +            if databases:
  +                bits["_r"]["d"] = {}
  +                for database, action in databases:
  +                    bits["_r"]["d"].setdefault(database, []).append(action)
  +            if resources:
  +                bits["_r"]["r"] = {}
  +                for database, table, action in resources:
  +                    bits["_r"]["r"].setdefault(database, {}).setdefault(
  +                        table, []
  +                    ).append(action)
           token = ds.sign(bits, namespace="token")
           click.echo("dstok_{}".format(token))
           if debug:
  ```
  Still needs tests, plus I'd like it to use abbreviations if available to keep the token length shorter.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1855#issuecomment-1347695728
issue_url: https://api.github.com/repos/simonw/datasette/issues/1855
node_id: IC_kwDOBm6k_c5QVDRw
created_at: 2022-12-13T03:30:09Z
updated_at: 2022-12-13T03:30:09Z
author_association: OWNER
body:
  I just noticed this in the existing code:

  https://github.com/simonw/datasette/blob/c5d30b58a1cd1c66bbddcf3561db005543ecaf25/datasette/default_permissions.py#L195-L203

  Hard-coding those action names should not be necessary any more, especially now we have `datasette.permissions` for looking up metadata about the permissions.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1855#issuecomment-1347707683
issue_url: https://api.github.com/repos/simonw/datasette/issues/1855
node_id: IC_kwDOBm6k_c5QVGMj
created_at: 2022-12-13T03:55:35Z
updated_at: 2022-12-13T04:15:27Z
author_association: OWNER
body:
  Help looks like this:

  ```
  Usage: datasette create-token [OPTIONS] ID

    Create a signed API token for the specified actor ID

    Example:

        datasette create-token root --secret mysecret

    To only allow create-table:

        datasette create-token root --secret mysecret \
            --all create-table

    Or to only allow insert-row against a specific table:

        datasette create-token root --secret myscret \
            --resource mydb mytable insert-row

    Restricted actions can be specified multiple times using multiple --all,
    --database, and --resource options.

    Add --debug to see a decoded version of the token.

  Options:
    --secret TEXT                   Secret used for signing the API tokens
                                    [required]
    -e, --expires-after INTEGER     Token should expire after this many seconds
    -a, --all ACTION                Restrict token to this action
    -d, --database DB ACTION        Restrict token to this action on this
                                    database
    -r, --resource DB RESOURCE ACTION
                                    Restrict token to this action on this
                                    database resource (a table, SQL view or
                                    named query)
    --debug                         Show decoded token
    --plugins-dir DIRECTORY         Path to directory containing custom plugins
    --help                          Show this message and exit.
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1855#issuecomment-1347726302
issue_url: https://api.github.com/repos/simonw/datasette/issues/1855
node_id: IC_kwDOBm6k_c5QVKve
created_at: 2022-12-13T04:16:26Z
updated_at: 2022-12-13T04:16:26Z
author_association: OWNER
body:
  I'm going to move this code into `datasette/cli.py` - it's a bit unexpected having it live in `default_permissions.py` like this (I couldn't find the code when I went looking for it earlier).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1855#issuecomment-1347731288
issue_url: https://api.github.com/repos/simonw/datasette/issues/1855
node_id: IC_kwDOBm6k_c5QVL9Y
created_at: 2022-12-13T04:24:50Z
updated_at: 2022-12-13T04:24:50Z
author_association: OWNER
body:
  For the tests for `datasette create-token` it would be useful if `datasette --get` had a mechanism for sending an `Authorization: Bearer X` header.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1946#issuecomment-1347732039
issue_url: https://api.github.com/repos/simonw/datasette/issues/1946
node_id: IC_kwDOBm6k_c5QVMJH
created_at: 2022-12-13T04:26:20Z
updated_at: 2022-12-13T04:26:20Z
author_association: OWNER
body:
  Two options:

  - `--header "Authorization: Bearer XXX"` which can be used to send any headers
  - `--token XXX` to specify the token, which is then sent using that header

  I like the second option more, simply because there are currently no other headers that affect how Datasette works. `--token` feels obvious and easy to use.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1946#issuecomment-1347733217
issue_url: https://api.github.com/repos/simonw/datasette/issues/1946
node_id: IC_kwDOBm6k_c5QVMbh
created_at: 2022-12-13T04:28:45Z
updated_at: 2022-12-13T04:28:45Z
author_association: OWNER
body:
  Demo of the new feature:
  ```
  % datasette create-token --secret s root
  dstok_eyJhIjoicm9vdCIsInRva2VuIjoiZHN0b2siLCJ0IjoxNjcwOTA1NjgwfQ.pqSWOwCSNp678hEWl9l5o7m1GaM
  % datasette --get /-/actor.json
  {"actor": null}
  % DATASETTE_SECRET=s datasette --get /-/actor.json --token dstok_eyJhIjoicm9vdCIsInRva2VuIjoiZHN0b2siLCJ0IjoxNjcwOTA1NjgwfQ.pqSWOwCSNp678hEWl9l5o7m1GaM
  {"actor": {"id": "root", "token": "dstok"}}
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1947#issuecomment-1347760109
issue_url: https://api.github.com/repos/simonw/datasette/issues/1947
node_id: IC_kwDOBm6k_c5QVS_t
created_at: 2022-12-13T05:12:00Z
updated_at: 2022-12-13T05:12:00Z
author_association: OWNER
body:
  For the UI: I think I'm going to dump a whole bunch of form elements on the page (so you can set up to 3 of each category of limit without any JavaScript), then add JavaScript that hides all but one of the options and gives you a "add another" widget that adds multiple more.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1855#issuecomment-1347761892
issue_url: https://api.github.com/repos/simonw/datasette/issues/1855
node_id: IC_kwDOBm6k_c5QVTbk
created_at: 2022-12-13T05:14:25Z
updated_at: 2022-12-13T05:14:25Z
author_association: OWNER
body:
  New documentation: https://docs.datasette.io/en/latest/authentication.html#restricting-the-actions-that-a-token-can-perform

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1947#issuecomment-1347768328
issue_url: https://api.github.com/repos/simonw/datasette/issues/1947
node_id: IC_kwDOBm6k_c5QVVAI
created_at: 2022-12-13T05:25:31Z
updated_at: 2022-12-13T22:25:46Z
author_association: OWNER
body:
  https://latest.datasette.io/-/create-token currently looks like this:


  ![Image](https://user-images.githubusercontent.com/9599/207458002-7c46940b-22c0-45d3-a668-ec7f1082588c.png)


  As a reminder, the CLI options that this needs to provide an alternative to are:

  https://github.com/simonw/datasette/blob/d4b98d3924dec625a99236e65b1b169ff957381f/docs/cli-reference.rst#L619-L638

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1937#issuecomment-1347770871
issue_url: https://api.github.com/repos/simonw/datasette/issues/1937
node_id: IC_kwDOBm6k_c5QVVn3
created_at: 2022-12-13T05:30:43Z
updated_at: 2022-12-13T05:30:43Z
author_association: OWNER
body:
  Also you should need `update-row` permission to use the `"replace": true` option - I should add that rule to `/-/insert` add well.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1914#issuecomment-1347801679
issue_url: https://api.github.com/repos/simonw/datasette/issues/1914
node_id: IC_kwDOBm6k_c5QVdJP
created_at: 2022-12-13T06:15:54Z
updated_at: 2022-12-13T06:15:54Z
author_association: OWNER
body:
  Should make sure that every API that returns an object as the top level (that's almost all of them) includes `"ok": true` to indicate no errors.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1950#issuecomment-1349855620
issue_url: https://api.github.com/repos/simonw/datasette/issues/1950
node_id: IC_kwDOBm6k_c5QdSmE
created_at: 2022-12-13T22:08:50Z
updated_at: 2022-12-13T22:08:50Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/blob/d4b98d3924dec625a99236e65b1b169ff957381f/datasette/views/table.py#L392-L400

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1950#issuecomment-1349864950
issue_url: https://api.github.com/repos/simonw/datasette/issues/1950
node_id: IC_kwDOBm6k_c5QdU32
created_at: 2022-12-13T22:11:15Z
updated_at: 2022-12-13T22:11:15Z
author_association: OWNER
body:
  Most places I use that exception at the moment set their own non-500 status error:
  ```
  datasette % rg DatasetteError
  datasette/handle_exception.py
  7:from .views.base import DatasetteError
  33:        elif isinstance(exception, DatasetteError):

  datasette/filters.py
  2:from datasette.views.base import DatasetteError
  22:                raise DatasetteError("_where= is not allowed", status=403)
  141:                    raise DatasetteError(

  datasette/views/table.py
  34:from .base import BaseView, DataView, DatasetteError, ureg, _error
  178:                raise DatasetteError(
  192:                raise DatasetteError(
  390:            raise DatasetteError("Cannot use _sort and _sort_desc at the same time")
  394:                raise DatasetteError(f"Cannot sort table by {sort}")
  400:                raise DatasetteError(f"Cannot sort table by {sort_desc}")

  datasette/views/base.py
  39:class DatasetteError(Exception):
  219:            raise DatasetteError(str(e), title="Invalid SQL", status=400)
  222:            raise DatasetteError(str(e))
  224:        except DatasetteError:
  382:            raise DatasetteError(
  402:            raise DatasetteError(str(e), title="Invalid SQL", status=400)
  405:            raise DatasetteError(str(e))
  407:        except DatasetteError:

  datasette/views/table2.py
  28:from .base import DataView, DatasetteError, ureg
  296:                raise DatasetteError(
  310:                raise DatasetteError(
  472:            raise DatasetteError("Cannot use _sort and _sort_desc at the same time")
  476:                raise DatasetteError(f"Cannot sort table by {sort}")
  482:                raise DatasetteError(f"Cannot sort table by {sort_desc}")

  datasette/views/database.py
  31:from .base import BaseView, DatasetteError, DataView, _error
  188:            raise DatasetteError("Invalid database", status=404)
  190:            raise DatasetteError("Cannot download in-memory databases", status=404)
  194:            raise DatasetteError("Cannot download database", status=404)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1947#issuecomment-1349972480
issue_url: https://api.github.com/repos/simonw/datasette/issues/1947
node_id: IC_kwDOBm6k_c5QdvIA
created_at: 2022-12-13T22:58:51Z
updated_at: 2022-12-13T22:58:51Z
author_association: OWNER
body:
  I'm experimenting with a `<select multiple>` for this.

  The usability for keyboards is still pretty awful, but it's a niche enough feature that maybe that's OK for the moment?

  ```javascript
  var select = document.querySelector('select');
  var selected = Array.from(temp0.options).filter(o => o.selected).map(o => o.value) 
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1947#issuecomment-1349974287
issue_url: https://api.github.com/repos/simonw/datasette/issues/1947
node_id: IC_kwDOBm6k_c5QdvkP
created_at: 2022-12-13T22:59:44Z
updated_at: 2022-12-13T22:59:44Z
author_association: OWNER
body:
  Got an option group thing working:

  <img width="445" alt="image" src="https://user-images.githubusercontent.com/9599/207463308-15268283-a0e9-4c05-b5e5-952def0f3ef1.png">

  But... it strikes me that any time you're considering a `<select multiple>` like this a nested list of checkboxes would actually be better - easier for people to use.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1947#issuecomment-1349975255
issue_url: https://api.github.com/repos/simonw/datasette/issues/1947
node_id: IC_kwDOBm6k_c5QdvzX
created_at: 2022-12-13T23:00:11Z
updated_at: 2022-12-13T23:00:11Z
author_association: OWNER
body:
  My `<select multiple>` prototype:
  ```diff
  diff --git a/datasette/templates/create_token.html b/datasette/templates/create_token.html
  index a94881ed..5bd641cc 100644
  --- a/datasette/templates/create_token.html
  +++ b/datasette/templates/create_token.html
  @@ -6,7 +6,7 @@
   
   <h1>Create an API token</h1>
   
  -<p>This token will allow API access with the same abilities as your current user.</p>
  +<p>This token will allow API access with the same abilities as your current user, <strong>{{ request.actor.id }}</strong></p>
   
   {% if errors %}
     {% for error in errors %}
  @@ -28,6 +28,36 @@
       <input type="hidden" name="csrftoken" value="{{ csrftoken() }}">
       <input type="submit" value="Create token">
     </div>
  +
  +  <details style="margin-top: 1em" open>
  +    <summary style="cursor: pointer;">Restrict actions that can be performed using this token</summary>
  +    <p style="margin-top: 1em">Restrict actions that can be performed using this token:</p>
  +    <p><label="all_permissions">All databases and tables:</label></p>
  +    <p><select multiple id="all_permissions" size="{{ all_permissions|length * 4 }}">
  +      <optgroup label="All databases and tables">
  +      {% for permission in all_permissions %}
  +        <option value="all:{{ permission }}">{{ permission }}</option>
  +      {% endfor %}
  +    </optgroup>
  +    {% for database in databases %}
  +      <optgroup label="All tables in database: {{ database }}">
  +        {% for permission in database_permissions %}
  +          <option value="db:{{ database }}:{{ permission }}">{{ permission }}</option>
  +        {% endfor %}
  +      </optgroup>
  +    {% endfor %}
  +    {% for dbt in database_with_tables %}
  +      {% for table in dbt.tables %}
  +        <optgroup label="Table {{ dbt.database }}.{{ table }}">
  +          {% for permission in table_permissions %}
  +            <option value="table:{{ dbt.database }}:{{ permission }}">{{ permission }}</option>
  +          {% endfor %}
  +        </optgroup>
  +      {% endfor %}
  +    {% endfor %}
  +  </select></p>
  +  </details>
  +
   </form>
   
   {% if token %}
  diff --git a/datasette/views/special.py b/datasette/views/special.py
  index 30345d14..9d0fcd31 100644
  --- a/datasette/views/special.py
  +++ b/datasette/views/special.py
  @@ -231,7 +231,17 @@ class CreateTokenView(BaseView):
           return await self.render(
               ["create_token.html"],
               request,
  -            {"actor": request.actor},
  +            {
  +                "actor": request.actor,
  +                "all_permissions": self.ds.permissions.keys(),
  +                "database_permissions": [key for key, value in self.ds.permissions.items() if value.takes_database],
  +                "table_permissions": [key for key, value in self.ds.permissions.items() if value.takes_resource],
  +                "databases": self.ds.databases.keys(),
  +                "database_with_tables": [{
  +                    "database": db.name,
  +                    "tables": await db.table_names(),
  +                } for db in self.ds.databases.values()],
  +            },
           )
   
       async def post(self, request):
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1947#issuecomment-1350002434
issue_url: https://api.github.com/repos/simonw/datasette/issues/1947
node_id: IC_kwDOBm6k_c5Qd2cC
created_at: 2022-12-13T23:11:50Z
updated_at: 2022-12-13T23:11:59Z
author_association: OWNER
body:
  I think checkboxes will work well.

  Here's the data I get back from them (as `post_vars()`):

  ```
  {'all:debug-menu': 'on',
   'all:insert-row': 'on',
   'expire_duration': '',
   'expire_type': '',
   'table:fixtures:delete-row': 'on',
   'table:fixtures:drop-table': 'on',
   'table:fixtures:view-query': 'on'}
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1947#issuecomment-1350008636
issue_url: https://api.github.com/repos/simonw/datasette/issues/1947
node_id: IC_kwDOBm6k_c5Qd388
created_at: 2022-12-13T23:14:33Z
updated_at: 2022-12-13T23:14:33Z
author_association: OWNER
body:
  Checkbox interface looks like this. It's not beautiful but it's good enough for the moment:

  <img width="598" alt="image" src="https://user-images.githubusercontent.com/9599/207465391-78b6e440-65b0-4a02-9911-0b948e65c357.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1947#issuecomment-1350013016
issue_url: https://api.github.com/repos/simonw/datasette/issues/1947
node_id: IC_kwDOBm6k_c5Qd5BY
created_at: 2022-12-13T23:16:24Z
updated_at: 2022-12-13T23:17:17Z
author_association: OWNER
body:
  Slightly tricky thing here is that it should only show permissions that the user themselves has - on databases and tables that they have permission to access.

  I have a nasty feeling this may require looping through _everything_ and running every permission check, which could get very expensive if there are plugins involved that do their own storage check to resolve a permission.

  It's that classic permission system problem: how to efficiently iterate through everything the user has permission to do in one go?

  Might be that I have to punt on that, and show the user a list of permissions to select that they might not actually have ability for.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1947#issuecomment-1350019528
issue_url: https://api.github.com/repos/simonw/datasette/issues/1947
node_id: IC_kwDOBm6k_c5Qd6nI
created_at: 2022-12-13T23:19:16Z
updated_at: 2022-12-13T23:19:16Z
author_association: OWNER
body:
  Here's the checkbox prototype:
  ```diff
  diff --git a/datasette/templates/create_token.html b/datasette/templates/create_token.html
  index a94881ed..1795ebaf 100644
  --- a/datasette/templates/create_token.html
  +++ b/datasette/templates/create_token.html
  @@ -2,11 +2,20 @@
   
   {% block title %}Create an API token{% endblock %}
   
  +{% block extra_head %}
  +<style type="text/css">
  +#restrict-permissions label {
  +  display: inline;
  +  width: 90%;
  +}
  +</style>
  +{% endblock %}
  +
   {% block content %}
   
   <h1>Create an API token</h1>
   
  -<p>This token will allow API access with the same abilities as your current user.</p>
  +<p>This token will allow API access with the same abilities as your current user, <strong>{{ request.actor.id }}</strong></p>
   
   {% if errors %}
     {% for error in errors %}
  @@ -27,8 +36,39 @@
       <input type="text" name="expire_duration" style="width: 10%">
       <input type="hidden" name="csrftoken" value="{{ csrftoken() }}">
       <input type="submit" value="Create token">
  -  </div>
  +
  +  <details style="margin-top: 1em" open id="restrict-permissions">
  +    <summary style="cursor: pointer;">Restrict actions that can be performed using this token</summary>
  +    <h2>All databases and tables</h2>
  +    <ul>
  +      {% for permission in all_permissions %}
  +        <li><label><input type="checkbox" name="all:{{ permission }}"> {{ permission }}</label></li>
  +      {% endfor %}
  +    </ul>
  +
  +    {% for database in databases %}
  +      <h2>All tables in database: {{ database }}</h2>
  +      <ul>
  +        {% for permission in database_permissions %}
  +          <li><label><input type="checkbox" name="db:{{ database }}:{{ permission }}"> {{ permission }}</label></li>
  +        {% endfor %}
  +      </ul>
  +    {% endfor %}
  +    <h2>Specific tables</h2>
  +    {% for dbt in database_with_tables %}
  +      {% for table in dbt.tables %}
  +        <h3>{{ dbt.database }}: {{ table }}</h3>
  +        <ul>
  +          {% for permission in table_permissions %}
  +            <li><label><input type="checkbox" name="table:{{ dbt.database }}:{{ permission }}"> {{ permission }}</label></li>
  +          {% endfor %}
  +        </ul>
  +      {% endfor %}
  +    {% endfor %}
  +  </details>
  +
   </form>
  +</div>
   
   {% if token %}
     <div>
  diff --git a/datasette/views/special.py b/datasette/views/special.py
  index 30345d14..48357f87 100644
  --- a/datasette/views/special.py
  +++ b/datasette/views/special.py
  @@ -231,12 +231,37 @@ class CreateTokenView(BaseView):
           return await self.render(
               ["create_token.html"],
               request,
  -            {"actor": request.actor},
  +            {
  +                "actor": request.actor,
  +                "all_permissions": self.ds.permissions.keys(),
  +                "database_permissions": [
  +                    key
  +                    for key, value in self.ds.permissions.items()
  +                    if value.takes_database
  +                ],
  +                "table_permissions": [
  +                    key
  +                    for key, value in self.ds.permissions.items()
  +                    if value.takes_resource
  +                ],
  +                "databases": [k for k in self.ds.databases.keys() if k != "_internal"],
  +                "database_with_tables": [
  +                    {
  +                        "database": db.name,
  +                        "tables": await db.table_names(),
  +                    }
  +                    for db in self.ds.databases.values()
  +                    if db.name != "_internal"
  +                ],
  +            },
           )
   
       async def post(self, request):
           self.check_permission(request)
           post = await request.post_vars()
  +        from pprint import pprint
  +
  +        pprint(post)
           errors = []
           duration = None
           if post.get("expire_type"):
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1947#issuecomment-1350037572
issue_url: https://api.github.com/repos/simonw/datasette/issues/1947
node_id: IC_kwDOBm6k_c5Qd_BE
created_at: 2022-12-13T23:27:32Z
updated_at: 2022-12-13T23:27:32Z
author_association: OWNER
body:
  I'm going to ignore the permissions issue for the moment - I'll allow people to select any permissions they like in any of the databases or tables that are visible to them (don't want to leak the existence of databases/tables to users who shouldn't be able to see them).

  I think the value of getting this working outweights any potential confusion from not using finely grained permission checks to decide if the user should be able to apply a permission or not.

  The tokens themselves won't be able to perform `insert-row` or similar if the user doesn't have the ability to do that, even if they selected that checkbox.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1947#issuecomment-1350124381
issue_url: https://api.github.com/repos/simonw/datasette/issues/1947
node_id: IC_kwDOBm6k_c5QeUNd
created_at: 2022-12-14T00:07:51Z
updated_at: 2022-12-14T00:07:51Z
author_association: OWNER
body:
  Another thing to consider in the future: once Datasette can support thousands of tables (see #417) the list on this page will turn into multiple MBs of HTML, which may cause all kinds of problems - not to mention the overhead of all of those table visibility permission checks. 

  Hopefully by then I'll have a good fix for the permission listings problem:
  - #1152

  And I can apply the same mechanism here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1947#issuecomment-1350148192
issue_url: https://api.github.com/repos/simonw/datasette/issues/1947
node_id: IC_kwDOBm6k_c5QeaBg
created_at: 2022-12-14T00:19:06Z
updated_at: 2022-12-14T00:19:06Z
author_association: OWNER
body:
  Another option: I could set a time limit - say 200ms - on how long I'm willing to spend calculating permissions before displaying this form

  First calculate view permissions for tables and databases (and maybe views and canned queries too).

  Then see if I can check every permission that I'm going to show as a checkbox on this page. If I get that done within the time limit use that to show the options.

  If I run out of time show all options and maybe include a note saying that some of them may not actually be available.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1947#issuecomment-1350215936
issue_url: https://api.github.com/repos/simonw/datasette/issues/1947
node_id: IC_kwDOBm6k_c5QeqkA
created_at: 2022-12-14T01:23:42Z
updated_at: 2022-12-14T01:23:42Z
author_association: OWNER
body:
  With tilde-encoding for database and table names the HTML looks like this:

  ```html
  <input type="checkbox" name="table:weird:foo~3Abar:view-table">
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1951#issuecomment-1350217380
issue_url: https://api.github.com/repos/simonw/datasette/issues/1951
node_id: IC_kwDOBm6k_c5Qeq6k
created_at: 2022-12-14T01:26:22Z
updated_at: 2022-12-14T01:26:22Z
author_association: OWNER
body:
  It's going to look very similar to the CLI tool, at least in terms of capabilities:
  ```
  Usage: datasette create-token [OPTIONS] ID

    Create a signed API token for the specified actor ID

    Example:

        datasette create-token root --secret mysecret

    To allow only "view-database-download" for all databases:

        datasette create-token root --secret mysecret \
            --all view-database-download

    To allow "create-table" against a specific database:

        datasette create-token root --secret mysecret \
            --database mydb create-table

    To allow "insert-row" against a specific table:

        datasette create-token root --secret myscret \
            --resource mydb mytable insert-row

    Restricted actions can be specified multiple times using multiple --all,
    --database, and --resource options.

    Add --debug to see a decoded version of the token.
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1951#issuecomment-1350218177
issue_url: https://api.github.com/repos/simonw/datasette/issues/1951
node_id: IC_kwDOBm6k_c5QerHB
created_at: 2022-12-14T01:27:45Z
updated_at: 2022-12-14T01:30:41Z
author_association: OWNER
body:
  Some sketches:
  ```python
  # Token for root user
  token = datasette.create_token("root")

  # Expiring in an hour
  token = datasette.create_token("root", expires_after=3600)
  ```
  More complicated is when you want to restrict to specific permissions:
  ```python
  # Limited to view-query and view-table
  token = datasette.create_token("root", expires_after=3600, all=("view-query", "view-table"))

  # I'm not sure about that all= name

  # Limits within a specific database:
  token = datasette.create_token("root", expires_after=3600, databases={
    "fixtures": ("view-query",)
  })

  # And specific tables:
  token = datasette.create_token("root", expires_after=3600, tables={
    "fixtures": {
      "facetable": ("insert-row", "update-row")
    }
  })
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1951#issuecomment-1350220579
issue_url: https://api.github.com/repos/simonw/datasette/issues/1951
node_id: IC_kwDOBm6k_c5Qersj
created_at: 2022-12-14T01:31:38Z
updated_at: 2022-12-14T01:31:38Z
author_association: OWNER
body:
  The problem with `all=(..)` is it feels misleading - it's actually restricting the permissions made available to the token.

  Likewise, `databases=` being a dict of restricted permissions isn't completely obvious.

  And the nested `tables=` dictionary feels a bit odd too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1951#issuecomment-1350222701
issue_url: https://api.github.com/repos/simonw/datasette/issues/1951
node_id: IC_kwDOBm6k_c5QesNt
created_at: 2022-12-14T01:35:05Z
updated_at: 2022-12-14T01:35:22Z
author_association: OWNER
body:
  Maybe this:

  ```python
  datasette.create_token("root", expires_after=3600, restrict_all=("view-query", "view-table"))

  token = datasette.create_token("root", expires_after=3600, restrict_database={
    "fixtures": ("view-query",)
  })

  token = datasette.create_token("root", expires_after=3600, restrict_resource={
    "fixtures": {
      "facetable": ("insert-row", "update-row")
    }
  })
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1951#issuecomment-1350293098
issue_url: https://api.github.com/repos/simonw/datasette/issues/1951
node_id: IC_kwDOBm6k_c5Qe9Zq
created_at: 2022-12-14T02:43:44Z
updated_at: 2022-12-14T02:43:44Z
author_association: OWNER
body:
  Documentation for the new method: https://docs.datasette.io/en/latest/internals.html#create-token-actor-id-expires-after-none-restrict-all-none-restrict-database-none-restrict-resource-none

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1947#issuecomment-1350402667
issue_url: https://api.github.com/repos/simonw/datasette/issues/1947
node_id: IC_kwDOBm6k_c5QfYJr
created_at: 2022-12-14T05:05:10Z
updated_at: 2022-12-14T05:05:10Z
author_association: OWNER
body:
  Tests can go here:

  https://github.com/simonw/datasette/blob/d98a8effb10ce8fe04a03eae42baa8a9cb0ca3f7/tests/test_auth.py#L143-L160

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1947#issuecomment-1350409537
issue_url: https://api.github.com/repos/simonw/datasette/issues/1947
node_id: IC_kwDOBm6k_c5QfZ1B
created_at: 2022-12-14T05:14:16Z
updated_at: 2022-12-14T05:14:16Z
author_association: OWNER
body:
  New interface now live at https://latest.datasette.io/-/create-token

  <img width="957" alt="image" src="https://user-images.githubusercontent.com/9599/207511688-f5366522-ff45-4175-82db-ed48474c3ac6.png">

  It shouldn't be showing `_memory` though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1947#issuecomment-1350413555
issue_url: https://api.github.com/repos/simonw/datasette/issues/1947
node_id: IC_kwDOBm6k_c5Qfazz
created_at: 2022-12-14T05:19:52Z
updated_at: 2022-12-14T05:19:52Z
author_association: OWNER
body:
  Maybe I should have kept `_memory` listed for instances that are running with `--crossdb` enabled?

  Yeah I think I should.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1947#issuecomment-1350414402
issue_url: https://api.github.com/repos/simonw/datasette/issues/1947
node_id: IC_kwDOBm6k_c5QfbBC
created_at: 2022-12-14T05:21:07Z
updated_at: 2022-12-14T05:21:07Z
author_association: OWNER
body:
  It would be neat not to show write permissions against immutable databases too - and not hard from a performance perspective since it doesn't involve hundreds more permission checks.

  That will need permissions to grow a flag for if they need a mutable database though, which is a bigger job.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1956#issuecomment-1352075845
issue_url: https://api.github.com/repos/simonw/datasette/issues/1956
node_id: IC_kwDOBm6k_c5QlwpF
created_at: 2022-12-14T19:57:17Z
updated_at: 2022-12-14T19:58:22Z
author_association: OWNER
body:
  I'm going to test this using calls to `ds.permission_allowed()` with an actor with `_r` block.

  I can add extra tests to https://github.com/simonw/datasette/blob/1a3dcf494376e32f7cff110c86a88e5b0a3f3924/tests/test_permissions.py#L605-L616

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1949#issuecomment-1352329027
issue_url: https://api.github.com/repos/simonw/datasette/issues/1949
node_id: IC_kwDOBm6k_c5QmudD
created_at: 2022-12-14T22:56:24Z
updated_at: 2022-12-14T22:57:19Z
author_association: OWNER
body:
  Most `.json` errors DO return as JSON, thanks to this:

  https://github.com/simonw/datasette/blob/c094dde3ff2bae030f261e6440d4fb082eb860a9/datasette/handle_exception.py#L19-L24

  https://github.com/simonw/datasette/blob/c094dde3ff2bae030f261e6440d4fb082eb860a9/datasette/handle_exception.py#L57-L58

  But that code triggers when the URL ends with `.json` - and none of the JSON write API endpoints (things like `/db/-/create` and `/db/table/-/insert`) follow that convention.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1949#issuecomment-1352330825
issue_url: https://api.github.com/repos/simonw/datasette/issues/1949
node_id: IC_kwDOBm6k_c5Qmu5J
created_at: 2022-12-14T22:58:51Z
updated_at: 2022-12-14T22:59:27Z
author_association: OWNER
body:
  I need a way for those JSON endpoints to communicate back to the `handle_exception` handler that they are returning JSON, so it knows to behave differently.

  Since it gets the `request` object, one way could be to have view code set `request.should_return_json = True` so that the handler knows to do something different.

  It's a bit of a cludge though!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1949#issuecomment-1352335503
issue_url: https://api.github.com/repos/simonw/datasette/issues/1949
node_id: IC_kwDOBm6k_c5QmwCP
created_at: 2022-12-14T23:03:28Z
updated_at: 2022-12-14T23:03:28Z
author_association: OWNER
body:
  This raises a more complicated issue

   At some point I'm likely to want to add an HTML interface for creating tables and inserting and updating rows.

  The obvious URLs for that are the same as for the JSON API: `/db/table/-/insert` and suchlike.

  Those endpoints are currently POST only - and can return JSON all the time.

  If they start accepting form POSTs too they'll need to be able to accept form-encoded data and return HTML instead. That's OK - they can detect incoming JSON thanks to the `content-type` header an the fact that the request body starts with `{` - but the `should_return_json` fix described above could intefere with how errors are returned if I'm not careful.

  I think it can still work though: I'll only set `should_return_json = True` if the endpoint gets a POST with a body starting `{`, or a content-type JSON header.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1949#issuecomment-1352338620
issue_url: https://api.github.com/repos/simonw/datasette/issues/1949
node_id: IC_kwDOBm6k_c5Qmwy8
created_at: 2022-12-14T23:05:17Z
updated_at: 2022-12-14T23:05:17Z
author_association: OWNER
body:
  Sniffing for a `{` is a little bit tricky though, as the post body is lazily loaded on request here:

  https://github.com/simonw/datasette/blob/9ad76d279e2c3874ca5070626a25458ce129f126/datasette/utils/asgi.py#L127-L135

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1949#issuecomment-1352340518
issue_url: https://api.github.com/repos/simonw/datasette/issues/1949
node_id: IC_kwDOBm6k_c5QmxQm
created_at: 2022-12-14T23:07:01Z
updated_at: 2022-12-14T23:07:01Z
author_association: OWNER
body:
  Easiest fix would be to look for `accept: application/json` and/or `content-type: application/json` headers.

  Not bullet-proof, so people might occasionally make JSON requests and get back an HTML error - but the documentation can tell people that they need to send those headers if they want to reliably get back JSON error messages.

  I'm happy with this as a solution.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1949#issuecomment-1352354927
issue_url: https://api.github.com/repos/simonw/datasette/issues/1949
node_id: IC_kwDOBm6k_c5Qm0xv
created_at: 2022-12-14T23:25:06Z
updated_at: 2022-12-14T23:25:14Z
author_association: OWNER
body:
  Looks like the code I've written for permission checking on `TableCreateView` and friends doesn't use the regular `raise Forbidden` or `raise DatasetteError` mechanisms - it does its own thing here:

  https://github.com/simonw/datasette/blob/9ad76d279e2c3874ca5070626a25458ce129f126/datasette/views/database.py#L580-L584

  Which uses this:

  https://github.com/simonw/datasette/blob/9ad76d279e2c3874ca5070626a25458ce129f126/datasette/views/base.py#L547-L548

  Having two different patterns to return errors is bad, I should fix that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1949#issuecomment-1352356356
issue_url: https://api.github.com/repos/simonw/datasette/issues/1949
node_id: IC_kwDOBm6k_c5Qm1IE
created_at: 2022-12-14T23:27:25Z
updated_at: 2022-12-14T23:28:16Z
author_association: OWNER
body:
  Also weird: errors returned by that mechanism look like this:
  ```json
  {
    "ok": false,
    "errors": ["list of error messages"]
  }
  ```
  While errors returned by the rest of Datasette look like this: https://latest.datasette.io/fixtures/no_table.json

  ```json
  {
    "ok": false,
    "error": "Table not found: no_table",
    "status": 404,
    "title": null
  }
  ```
  Related:
  - #1875

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1957#issuecomment-1352371019
issue_url: https://api.github.com/repos/simonw/datasette/issues/1957
node_id: IC_kwDOBm6k_c5Qm4tL
created_at: 2022-12-14T23:50:50Z
updated_at: 2022-12-14T23:50:50Z
author_association: OWNER
body:
  One option: if any rows were truncated, show a button you can click to run the query again with truncation disabled - maybe with `?_truncate=off` in the URL or similar.

  But... still want to truncate if the user runs a query that would return multiple MBs of HTML (assuming the other Datasette query limits don't stop that from happening).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1949#issuecomment-1352378370
issue_url: https://api.github.com/repos/simonw/datasette/issues/1949
node_id: IC_kwDOBm6k_c5Qm6gC
created_at: 2022-12-15T00:02:08Z
updated_at: 2022-12-15T00:04:54Z
author_association: OWNER
body:
  I fixed this issue to help research this further:
  - https://github.com/simonw/datasette-ripgrep/issues/26

  Now this search works:

  <https://ripgrep.datasette.io/-/ripgrep?pattern=return+_error&literal=on&glob=datasette%2F**>

  I wish I had this feature!
  - https://github.com/simonw/datasette-ripgrep/issues/24

  Looks like I have both `_error()` and `_errors()` functions in there!

  <img width="872" alt="image" src="https://user-images.githubusercontent.com/9599/207741634-56667a10-ba70-42e0-98ca-ca1444a86e57.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1953#issuecomment-1352410078
issue_url: https://api.github.com/repos/simonw/datasette/issues/1953
node_id: IC_kwDOBm6k_c5QnCPe
created_at: 2022-12-15T00:44:56Z
updated_at: 2022-12-15T00:44:56Z
author_association: OWNER
body:
  Highlights:

  - `/db/table/-/upsert`
  - ignore and replace for `/db/-/create`
  - `register_permissions()` plugin hook
  - `datasette create-token` can create restricted tokens
  - `/-/create-token` can too
  - `datasette --get --token` option
  - `datasette.create_token()` API method

  Plus some smaller things.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1949#issuecomment-1352411327
issue_url: https://api.github.com/repos/simonw/datasette/issues/1949
node_id: IC_kwDOBm6k_c5QnCi_
created_at: 2022-12-15T00:46:27Z
updated_at: 2022-12-15T00:46:27Z
author_association: OWNER
body:
  I got this far:
  ```diff
  diff --git a/datasette/handle_exception.py b/datasette/handle_exception.py
  index 8b7e83e3..31d41e00 100644
  --- a/datasette/handle_exception.py
  +++ b/datasette/handle_exception.py
  @@ -54,7 +54,17 @@ def handle_exception(datasette, request, exception):
           headers = {}
           if datasette.cors:
               add_cors_headers(headers)
  -        if request.path.split("?")[0].endswith(".json"):
  +        # Return JSON error under certain conditions
  +        should_return_json = (
  +            # URL ends in .json
  +            request.path.split("?")[0].endswith(".json")
  +            or
  +            # Hints from incoming request headers
  +            request.headers.get("content-type") == "application/json"
  +            or "application/json" in request.headers.get("accept", "")
  +        )
  +        breakpoint()
  +        if should_return_json:
               return Response.json(info, status=status, headers=headers)
           else:
               template = datasette.jinja_env.select_template(templates)
  diff --git a/tests/test_api_write.py b/tests/test_api_write.py
  index f27d143f..982543a6 100644
  --- a/tests/test_api_write.py
  +++ b/tests/test_api_write.py
  @@ -1140,6 +1140,38 @@ async def test_create_table_permissions(
           assert data["errors"] == expected_errors
   
   
  +@pytest.mark.asyncio
  +@pytest.mark.parametrize(
  +    "headers,expect_json",
  +    (
  +        ({}, False),
  +        ({"Accept": "text/html"}, True),
  +        ({"Accept": "application/json"}, True),
  +        ({"Content-Type": "application/json"}, True),
  +        ({"Accept": "application/json, text/plain, */*"}, True),
  +        ({"Content-Type": "application/json"}, True),
  +        ({"accept": "application/json, text/plain, */*"}, True),
  +        ({"content-type": "application/json"}, True),
  +    ),
  +)
  +async def test_permission_errors_html_and_json(ds_write, headers, expect_json):
  +    request_headers = {"Authorization": "Bearer bad_token"}
  +    request_headers.update(headers)
  +    response = await ds_write.client.post(
  +        "/data/-/create",
  +        json={},
  +        headers=request_headers,
  +    )
  +    assert response.status_code == 403
  +    if expect_json:
  +        data = response.json()
  +        assert data["ok"] is False
  +        assert data["errors"] == ["Permission denied"]
  +    else:
  +        assert response.headers["Content-Type"] == "text/html; charset=utf-8"
  +        assert "Permission denied" in response.text
  +
  +
   @pytest.mark.asyncio
   @pytest.mark.parametrize(
       "input,expected_rows_after",
  ```
  Then decided I would punt this until the next milestone.

reactions:
[cell repeats an earlier one in this file: 112 characters]


