# issue_comments

html_url: https://github.com/simonw/datasette/issues/260#issuecomment-1234927627
issue_url: https://api.github.com/repos/simonw/datasette/issues/260
node_id: IC_kwDOBm6k_c5Jm4AL
created_at: 2022-09-02T00:05:43Z
updated_at: 2022-09-02T00:05:43Z
author_association: OWNER
body:
  I'm inclined to consider [Pydantic](https://pydantic-docs.helpmanual.io/) for this, since it is widely used now and can generate really good error messages.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/260#issuecomment-1235079469
issue_url: https://api.github.com/repos/simonw/datasette/issues/260
node_id: IC_kwDOBm6k_c5JndEt
created_at: 2022-09-02T05:24:59Z
updated_at: 2022-09-02T05:24:59Z
author_association: NONE
body:
  @zschira is working with Pydantic while converting between and validating JSON frictionless datapackage descriptors that annotate an SQLite DB ([extracted from FERC's XBRL data](https://github.com/catalyst-cooperative/ferc-xbrl-extractor)) and the Datasette YAML metadata [so we can publish them with Datasette](https://github.com/catalyst-cooperative/pudl/pull/1831). Maybe there's some overlap? We've been loving Pydantic.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/239#issuecomment-1236200834
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/239
node_id: IC_kwDOCGYnMM5Jru2C
created_at: 2022-09-03T21:26:32Z
updated_at: 2022-09-03T21:26:32Z
author_association: NONE
body:
  I was looking for something like this today, for extracting columns containing objects (and arrays of objects) into separate tables. 

  Would it make sense (especially for the fields containing arrays of objects) to create a one-to-many relationship, where each row of the newly created table would contain the id of the row that originally contained it?

  If the extracted objects have a unique id and are repeated, it could even create a many-to-many relationship, with a third table for the joins.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/482#issuecomment-1236511153
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/482
node_id: IC_kwDOCGYnMM5Js6mx
created_at: 2022-09-05T03:38:29Z
updated_at: 2022-09-05T03:38:29Z
author_association: OWNER
body:
  This is an interesting research area! Running the same experiment as the one in https://www.cybertec-postgresql.com/en/column-order-in-postgresql-does-matter/ could demonstrate one way or another if column ordering makes a difference with regards to performance or not.

  My hunch is that even if it does this wouldn't be a feature to bake into `sqlite-utils` itself, beyond the existing features for changing the order of columns in a table (https://sqlite-utils.datasette.io/en/stable/python-api.html#changing-column-order).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1800#issuecomment-1237375000
issue_url: https://api.github.com/repos/simonw/datasette/issues/1800
node_id: IC_kwDOBm6k_c5JwNgY
created_at: 2022-09-05T18:24:34Z
updated_at: 2022-09-05T18:24:34Z
author_association: OWNER
body:
  I'm going to leave `black` pinned to the exact version simply because updates to Black can cause my tests to fail due to formatting style decisions they have made - and it's just a test dependency, not a install requires dependency.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1801#issuecomment-1237380827
issue_url: https://api.github.com/repos/simonw/datasette/issues/1801
node_id: IC_kwDOBm6k_c5JwO7b
created_at: 2022-09-05T18:35:24Z
updated_at: 2022-09-05T18:35:24Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1801?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Base: **91.66**% // Head: **91.66**% // No change to project coverage :thumbsup:
  > Coverage data is based on head [(`3f24edd`)](https://codecov.io/gh/simonw/datasette/pull/1801?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`ba35105`)](https://codecov.io/gh/simonw/datasette/commit/ba35105eee2d3ba620e4f230028a02b2e2571df2?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > Patch has no changes to coverable lines.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main    #1801   +/-   ##
  =======================================
    Coverage   91.66%   91.66%           
  =======================================
    Files          38       38           
    Lines        4705     4705           
  =======================================
    Hits         4313     4313           
    Misses        392      392           
  ```



  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  </details>

  [:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1801?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1800#issuecomment-1237383202
issue_url: https://api.github.com/repos/simonw/datasette/issues/1800
node_id: IC_kwDOBm6k_c5JwPgi
created_at: 2022-09-05T18:39:43Z
updated_at: 2022-09-05T18:39:52Z
author_association: OWNER
body:
  Dependabot noticed: https://github.com/simonw/datasette/pull/1799#issuecomment-1237381569

  > Looks like aiofiles is no longer updatable, so this is no longer needed.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1802#issuecomment-1237519153
issue_url: https://api.github.com/repos/simonw/datasette/issues/1802
node_id: IC_kwDOBm6k_c5Jwwsx
created_at: 2022-09-05T23:17:03Z
updated_at: 2022-09-05T23:17:03Z
author_association: OWNER
body:
  Some previous issues where I've run Python in Docker:

  - https://github.com/simonw/datasette/issues/1344#issuecomment-849805778
  - https://github.com/simonw/datasette/issues/1647#issuecomment-1059819628

  For this one I'm going to try this image: `python:3.7-buster` - from https://hub.docker.com/layers/library/python/3.7-buster/images/sha256-8a79c5ddabd9e3bceb7d274e515d85a3f5898bd983bb5761ec94186bc54ae968?context=explore

  ```
  docker run -it -v `pwd`:/mnt \
    python:3.7-buster \
    bash -c "cd /mnt && pip install -e '.[test]' && pytest"
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1802#issuecomment-1237522793
issue_url: https://api.github.com/repos/simonw/datasette/issues/1802
node_id: IC_kwDOBm6k_c5Jwxlp
created_at: 2022-09-05T23:27:08Z
updated_at: 2022-09-05T23:27:08Z
author_association: OWNER
body:
  Here's how GitHub Actions runs them: https://github.com/simonw/datasette/blob/294ecd45f7801971dbeef383d0c5456ee95ab839/.github/workflows/test.yml#L27-L37

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1802#issuecomment-1237523178
issue_url: https://api.github.com/repos/simonw/datasette/issues/1802
node_id: IC_kwDOBm6k_c5Jwxrq
created_at: 2022-09-05T23:28:11Z
updated_at: 2022-09-05T23:28:11Z
author_association: OWNER
body:
  Trying this instead:
  ```
  docker run -it -v `pwd`:/mnt \
    python:3.7-buster \
    bash -c "cd /mnt && pip install -e '.[test]' && pytest -n auto -m 'not serial' && pytest -m 'serial'"
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1802#issuecomment-1237527575
issue_url: https://api.github.com/repos/simonw/datasette/issues/1802
node_id: IC_kwDOBm6k_c5JwywX
created_at: 2022-09-05T23:40:32Z
updated_at: 2022-09-05T23:40:32Z
author_association: OWNER
body:
  That passed too:

  ```
  ========================================================= test session starts ==========================================================
  platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0
  SQLite: 3.27.2
  rootdir: /mnt, configfile: pytest.ini
  plugins: asyncio-0.19.0, anyio-3.6.1, timeout-2.1.0, xdist-2.5.0, forked-1.4.0
  asyncio: mode=strict
  gw0 [1024] / gw1 [1024] / gw2 [1024] / gw3 [1024] / gw4 [1024] / gw5 [1024]
  ................................................................................................................................ [ 12%]
  ................................................................................................................................ [ 25%]
  ...............................................................................................................ss............... [ 37%]
  ................................................................................................................................ [ 50%]
  ................................................................................................................................ [ 62%]
  .................................................s..........................s................................................... [ 75%]
  ................................................................................................................................ [ 87%]
  ................................................................................................................................ [100%]
  ============================================= 1020 passed, 4 skipped in 294.43s (0:04:54) ==============================================
  ========================================================= test session starts ==========================================================
  platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0
  SQLite: 3.27.2
  rootdir: /mnt, configfile: pytest.ini
  plugins: asyncio-0.19.0, anyio-3.6.1, timeout-2.1.0, xdist-2.5.0, forked-1.4.0
  asyncio: mode=strict
  collected 1054 items / 1024 deselected / 30 selected                                                                                   

  tests/test_package.py .                                                                                                          [  3%]
  tests/test_cli_serve_server.py ...                                                                                               [ 13%]
  tests/test_publish_cloudrun.py ......................                                                                            [ 86%]
  tests/test_publish_heroku.py ....                                                                                                [100%]

  ================================================= 30 passed, 1024 deselected in 7.62s ==================================================
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1802#issuecomment-1237530726
issue_url: https://api.github.com/repos/simonw/datasette/issues/1802
node_id: IC_kwDOBm6k_c5Jwzhm
created_at: 2022-09-05T23:50:18Z
updated_at: 2022-09-05T23:50:18Z
author_association: OWNER
body:
  Tried again, this time it failed with 3.8: https://github.com/simonw/datasette/runs/8197344556?check_suite_focus=true

  I'm going to try ditching the parallel test execution.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1802#issuecomment-1237532571
issue_url: https://api.github.com/repos/simonw/datasette/issues/1802
node_id: IC_kwDOBm6k_c5Jwz-b
created_at: 2022-09-05T23:56:28Z
updated_at: 2022-09-05T23:56:28Z
author_association: OWNER
body:
  That didn't help. This time it was 3.9 that failed first: https://github.com/simonw/datasette/runs/8197403744?check_suite_focus=true - same error though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1802#issuecomment-1237533471
issue_url: https://api.github.com/repos/simonw/datasette/issues/1802
node_id: IC_kwDOBm6k_c5Jw0Mf
created_at: 2022-09-05T23:59:26Z
updated_at: 2022-09-05T23:59:26Z
author_association: OWNER
body:
  https://docs.python.org/3/library/asyncio-task.html#asyncio.run notes that `asyncio.run()` was added in Python 3.7. I dropped support for 3.6 a while ago.

  https://stackoverflow.com/a/56446830/6083 suggests that switching to `asyncio.run()` has helped other people dealing with the `RuntimeError: There is no current event loop in thread` error in the past.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1802#issuecomment-1237537205
issue_url: https://api.github.com/repos/simonw/datasette/issues/1802
node_id: IC_kwDOBm6k_c5Jw1G1
created_at: 2022-09-06T00:08:03Z
updated_at: 2022-09-06T00:08:46Z
author_association: OWNER
body:
  I'm suspicious that this hack here may be hurting me:

  https://github.com/simonw/datasette/blob/b91e17280c05bbb9cf97432081bdcea8665879f9/tests/test_cli.py#L25-L29

  I added that two years ago in https://github.com/simonw/datasette/commit/3159263f05ac4baf968929d59384d9223a539071

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1802#issuecomment-1237540231
issue_url: https://api.github.com/repos/simonw/datasette/issues/1802
node_id: IC_kwDOBm6k_c5Jw12H
created_at: 2022-09-06T00:16:10Z
updated_at: 2022-09-06T00:16:10Z
author_association: OWNER
body:
  Removing the `ensure_eventloop()` hack produced the exact same test failure: https://github.com/simonw/datasette/runs/8197570109?check_suite_focus=true

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1802#issuecomment-1237542508
issue_url: https://api.github.com/repos/simonw/datasette/issues/1802
node_id: IC_kwDOBm6k_c5Jw2Zs
created_at: 2022-09-06T00:22:37Z
updated_at: 2022-09-06T00:22:37Z
author_association: OWNER
body:
  The code that I control that shows up in the trace is this:

  https://github.com/simonw/datasette/blob/fd1086c6867f3e3582b1eca456e4ea95f6cecf8b/datasette/app.py#L234

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1802#issuecomment-1237542803
issue_url: https://api.github.com/repos/simonw/datasette/issues/1802
node_id: IC_kwDOBm6k_c5Jw2eT
created_at: 2022-09-06T00:23:27Z
updated_at: 2022-09-06T00:23:27Z
author_association: OWNER
body:
  Trying this fix:
  ```diff
  diff --git a/datasette/app.py b/datasette/app.py
  index f2a6763a5..b17888a07 100644
  --- a/datasette/app.py
  +++ b/datasette/app.py
  @@ -231,7 +231,13 @@ def __init__(
           self.inspect_data = inspect_data
           self.immutables = set(immutables or [])
           self.databases = collections.OrderedDict()
  -        self._refresh_schemas_lock = asyncio.Lock()
  +        try:
  +            self._refresh_schemas_lock = asyncio.Lock()
  +        except RuntimeError as rex:
  +            if "There is no current event loop in thread" in str(rex):
  +                loop = asyncio.new_event_loop()
  +                asyncio.set_event_loop(loop)
  +                self._refresh_schemas_lock = asyncio.Lock()
           self.crossdb = crossdb
           self.nolock = nolock
           if memory or crossdb or not self.files:
  ```
  This is mainly superstition at this point! I found that recipe on https://techoverflow.net/2020/10/01/how-to-fix-python-asyncio-runtimeerror-there-is-no-current-event-loop-in-thread/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1803#issuecomment-1237548400
issue_url: https://api.github.com/repos/simonw/datasette/issues/1803
node_id: IC_kwDOBm6k_c5Jw31w
created_at: 2022-09-06T00:38:18Z
updated_at: 2022-09-06T00:38:18Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1803?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Base: **91.66**% // Head: **91.67**% // Increases project coverage by **`+0.01%`** :tada:
  > Coverage data is based on head [(`2db6f9c`)](https://codecov.io/gh/simonw/datasette/pull/1803?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`51030df`)](https://codecov.io/gh/simonw/datasette/commit/51030df1869b3b574dd3584d1563415776b9cd4e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > Patch coverage: 100.00% of modified lines in pull request are covered.

  > :exclamation: Current head 2db6f9c differs from pull request most recent head 7b6a2b4. Consider uploading reports for the commit 7b6a2b4 to get more accurate results

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@            Coverage Diff             @@
  ##             main    #1803      +/-   ##
  ==========================================
  + Coverage   91.66%   91.67%   +0.01%     
  ==========================================
    Files          38       38              
    Lines        4705     4711       +6     
  ==========================================
  + Hits         4313     4319       +6     
    Misses        392      392              
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1803?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1803/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.23% <100.00%> (+0.04%)` | :arrow_up: |

  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  </details>

  [:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1803?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1757#issuecomment-1237553237
issue_url: https://api.github.com/repos/simonw/datasette/issues/1757
node_id: IC_kwDOBm6k_c5Jw5BV
created_at: 2022-09-06T00:51:21Z
updated_at: 2022-09-06T00:51:21Z
author_association: OWNER
body:
  Interesting feature idea.

  It's a weird edge-case, but it's the kind of thing I worry about: what if the user has a column called `*`? This is technically allowed in SQLite, even though it's vastly unlikely.

  On that basis, I think `?_json_all=1` would be a better design here than `?_json=*` - since it removes any possible ambiguity.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/pull/76#issuecomment-1238190601
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/76
node_id: IC_kwDODFdgUs5JzUoJ
created_at: 2022-09-06T13:58:20Z
updated_at: 2022-09-06T13:59:08Z
author_association: NONE
body:
  Tested PR just now in private org, fetched >2k repos infos flawlessly!

      poetry run github-to-sqlite repos --organization github.db MYORG

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1804#issuecomment-1238308213
issue_url: https://api.github.com/repos/simonw/datasette/issues/1804
node_id: IC_kwDOBm6k_c5JzxV1
created_at: 2022-09-06T15:27:31Z
updated_at: 2022-09-06T15:27:31Z
author_association: OWNER
body:
  Relevant example from https://docs.datasette.io/en/stable/metadata.html#per-database-and-per-table-metadata
  ```json
  {
      "databases": {
          "database1": {
              "tables": {
                  "example_table": {
                      "sortable_columns": [
                          "height",
                          "weight"
                      ]
                  }
              }
          }
      }
  }
  ```
  I think this becomes `"facet_size": "..."` at the table level, to match the name of the `?_facet_size=100` querystring parameter.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1804#issuecomment-1238309481
issue_url: https://api.github.com/repos/simonw/datasette/issues/1804
node_id: IC_kwDOBm6k_c5Jzxpp
created_at: 2022-09-06T15:28:35Z
updated_at: 2022-09-06T15:28:35Z
author_association: OWNER
body:
  I'm going to do this just at the table level - if you want it at the database level I would imagine usually you would be OK running `--setting default_facet_size 10` instead, since most Datasette instances only expose a single database.

  I may reconsider this in the future.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1804#issuecomment-1238311565
issue_url: https://api.github.com/repos/simonw/datasette/issues/1804
node_id: IC_kwDOBm6k_c5JzyKN
created_at: 2022-09-06T15:30:19Z
updated_at: 2022-09-06T15:30:19Z
author_association: OWNER
body:
  Relevant code: https://github.com/simonw/datasette/blob/c9d1943aede436fa3413fd49bc56335cbda4ad07/datasette/facets.py#L102-L110

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1804#issuecomment-1238313314
issue_url: https://api.github.com/repos/simonw/datasette/issues/1804
node_id: IC_kwDOBm6k_c5Jzyli
created_at: 2022-09-06T15:31:36Z
updated_at: 2022-09-06T15:31:36Z
author_association: OWNER
body:
  In that code `self.table` is the name of the current table, but can also be `None` if the faceting is being run against a custom SQL query instead (a future feature). `self.database` is the name of the current database.

  So reading metadata via `self.ds` should work fine.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/596#issuecomment-1238383171
issue_url: https://api.github.com/repos/simonw/datasette/issues/596
node_id: IC_kwDOBm6k_c5J0DpD
created_at: 2022-09-06T16:27:25Z
updated_at: 2022-09-06T16:27:25Z
author_association: NONE
body:
  Perhaps some ways to address this.

  1. Add a **horizontal scrollbar at the top of the table**. There are some solutions here: https://stackoverflow.com/questions/3934271/horizontal-scrollbar-on-top-and-bottom-of-table
  2. Use a **fixed table header**. It would be useful when you're lost in the middle of a very big table. Pure CSS solutions seem to exist: https://stackoverflow.com/questions/21168521/table-fixed-header-and-scrollable-body
  3. Maybe a possibility to resize columns. Not sure about that because it would more work not to lose it after each reload.
  4. A way to keep **favorite views for each user**. The process would be: I select the column I want or not (with existing "settings" icon of each column); then I select a "favoritize view" option somewhere; then I can recall all my favorite views from a menu. These data could be hosted on the browser.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1805#issuecomment-1238491632
issue_url: https://api.github.com/repos/simonw/datasette/issues/1805
node_id: IC_kwDOBm6k_c5J0eHw
created_at: 2022-09-06T18:10:14Z
updated_at: 2022-09-06T18:10:14Z
author_association: OWNER
body:
  I like the idea of defaulting this to the `truncate_cells_html` setting. It could be alternatively customized using a custom Render Cell plugin, but having Datasette do this by default makes sense to me.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/484#issuecomment-1238607591
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/484
node_id: IC_kwDOCGYnMM5J06bn
created_at: 2022-09-06T20:16:39Z
updated_at: 2022-09-06T20:16:39Z
author_association: OWNER
body:
  Here's the implementation for recipes at the moment: https://github.com/simonw/sqlite-utils/blob/5b969273f1244b1bcf3e4dc071cdf17dab35d5f8/sqlite_utils/utils.py#L434-L441

  And here's the `--functions` implementation that doesn't expose them: https://github.com/simonw/sqlite-utils/blob/5b969273f1244b1bcf3e4dc071cdf17dab35d5f8/sqlite_utils/cli.py#L3029-L3040

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/477#issuecomment-1238608921
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/477
node_id: IC_kwDOCGYnMM5J06wZ
created_at: 2022-09-06T20:18:12Z
updated_at: 2022-09-06T20:18:12Z
author_association: OWNER
body:
  Hi @thewchan - thanks for that! Yes I'm interested, can you invite `swillison` @ Google's email provider please?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1805#issuecomment-1238760255
issue_url: https://api.github.com/repos/simonw/datasette/issues/1805
node_id: IC_kwDOBm6k_c5J1fs_
created_at: 2022-09-06T23:55:37Z
updated_at: 2022-09-06T23:55:37Z
author_association: OWNER
body:
  While working on this I noticed that the arbitrary query page doesn't currently obey truncation, when it should do.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1804#issuecomment-1238773577
issue_url: https://api.github.com/repos/simonw/datasette/issues/1804
node_id: IC_kwDOBm6k_c5J1i9J
created_at: 2022-09-07T00:21:56Z
updated_at: 2022-09-07T00:21:56Z
author_association: OWNER
body:
  `facet_size` is now documented here: https://docs.datasette.io/en/latest/facets.html#facets-in-metadata-json

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/596#issuecomment-1238790158
issue_url: https://api.github.com/repos/simonw/datasette/issues/596
node_id: IC_kwDOBm6k_c5J1nAO
created_at: 2022-09-07T00:54:55Z
updated_at: 2022-09-07T00:54:55Z
author_association: OWNER
body:
  The jqDoubleScroll plugin is interesting: https://github.com/avianey/jqDoubleScroll/blob/master/jquery.doubleScroll.js

  Here's a demo: https://6317eba21040d42c0228aab9--tiny-tarsier-c1f294.netlify.app/example.html

  Wouldn't be too hard to port that to work without jQuery.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/596#issuecomment-1238790634
issue_url: https://api.github.com/repos/simonw/datasette/issues/596
node_id: IC_kwDOBm6k_c5J1nHq
created_at: 2022-09-07T00:55:52Z
updated_at: 2022-09-07T00:58:41Z
author_association: OWNER
body:
  Or there's this recipe on StackOverflow that does the same thing without jQuery - there's an interactive demo on that page if you click "Run code snippet": https://stackoverflow.com/a/47038108/6083

  All sorts of other potential solutions in that thread too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1805#issuecomment-1238868091
issue_url: https://api.github.com/repos/simonw/datasette/issues/1805
node_id: IC_kwDOBm6k_c5J16B7
created_at: 2022-09-07T03:33:15Z
updated_at: 2022-09-07T03:34:04Z
author_association: OWNER
body:
  Here's an example (showing the default 2048 truncation length, so it's still pretty long):

  https://latest.datasette.io/_memory?sql=select+%27https%3A%2F%2Fexample.com%2Faaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.jpg%27+as+truncated

  <img width="1289" alt="image" src="https://user-images.githubusercontent.com/9599/188782889-72b7ecad-54c9-4e7f-b911-08136896c573.png">

  I think that would work better with `word-wrap: anywhere`:

  <img width="601" alt="image" src="https://user-images.githubusercontent.com/9599/188782928-669201c6-f5b0-4696-8065-2487ee3ef6ab.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1238873948
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/471
node_id: IC_kwDOCGYnMM5J17dc
created_at: 2022-09-07T03:46:26Z
updated_at: 2022-09-07T03:46:26Z
author_association: OWNER
body:
  > Is it still nfortunately slow and tricky when playing with floats ?

  Not sure what you mean here?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/484#issuecomment-1239697643
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/484
node_id: IC_kwDOCGYnMM5J5Ejr
created_at: 2022-09-07T17:48:00Z
updated_at: 2022-09-07T17:48:00Z
author_association: OWNER
body:
  Will also need to update documentation here: https://sqlite-utils.datasette.io/en/stable/cli.html#defining-custom-sql-functions

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/484#issuecomment-1239699276
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/484
node_id: IC_kwDOCGYnMM5J5E9M
created_at: 2022-09-07T17:49:49Z
updated_at: 2022-09-07T17:49:49Z
author_association: OWNER
body:
  This feature is a tiny bit weird though: the recipe functions are not exposed to SQL by default, they are instead designed to be used with `sqlite-utils convert`. I guess with `--functions` support you could do something like this:

      sqlite-utils data.db "update mytable set col1 = parsedate(col1)" --functions "parsedate = r.parsedate"

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/484#issuecomment-1239759022
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/484
node_id: IC_kwDOCGYnMM5J5Tiu
created_at: 2022-09-07T18:52:08Z
updated_at: 2022-09-07T18:52:08Z
author_association: OWNER
body:
  It's not quite that simple. I tried applying this patch:

  ```diff
  diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py
  index c51b101..33e4d90 100644
  --- a/sqlite_utils/cli.py
  +++ b/sqlite_utils/cli.py
  @@ -30,6 +30,7 @@ from .utils import (
       Format,
       TypeTracker,
   )
  +from . import recipes
   
   
   CONTEXT_SETTINGS = dict(help_option_names=["-h", "--help"])
  @@ -3029,7 +3030,7 @@ def _load_extensions(db, load_extension):
   def _register_functions(db, functions):
       # Register any Python functions as SQL functions:
       sqlite3.enable_callback_tracebacks(True)
  -    globals = {}
  +    globals = {"r": recipes, "recipes": recipes}
       try:
           exec(functions, globals)
       except SyntaxError as ex:
  ```

  Then got this:

  ```
  % sqlite-utils memory --functions 'parsedate = r.parsedate' 'select parsedate("1st jan")'
  Error: wrong number of arguments to function parsedate()
  % sqlite-utils memory --functions 'parsedate = r.parsedate' 'select parsedate("1st jan", 0, 0, 0)' 
  [{"parsedate(\"1st jan\", 0, 0, 0)": "2022-01-01"}]
  ```
  The problem here is that the `parsedate` function signature looks like this:

  https://github.com/simonw/sqlite-utils/blob/d9b9e075f07a20f1137cd2e34ed5d3f1a3db4ad8/sqlite_utils/recipes.py#L8

  But the code that register SQL functions introspects that signature, so creates a SQL function that requires four arguments.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/484#issuecomment-1239760001
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/484
node_id: IC_kwDOCGYnMM5J5TyB
created_at: 2022-09-07T18:53:17Z
updated_at: 2022-09-07T18:53:17Z
author_association: OWNER
body:
  So you would need to do this instead:
  ```
  sqlite-utils memory 'select parsedate("1st jan")' --functions '
  def parsedate(s):
      return r.parsedate(s)
  '
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/484#issuecomment-1239761280
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/484
node_id: IC_kwDOCGYnMM5J5UGA
created_at: 2022-09-07T18:54:51Z
updated_at: 2022-09-07T18:54:51Z
author_association: OWNER
body:
  I could teach this code here to only register the function using arguments that don't have default parameters:

  https://github.com/simonw/sqlite-utils/blob/d9b9e075f07a20f1137cd2e34ed5d3f1a3db4ad8/sqlite_utils/cli.py#L3037-L3040

  Or even this code here:

  https://github.com/simonw/sqlite-utils/blob/d9b9e075f07a20f1137cd2e34ed5d3f1a3db4ad8/sqlite_utils/db.py#L398-L418

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/484#issuecomment-1239762031
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/484
node_id: IC_kwDOCGYnMM5J5URv
created_at: 2022-09-07T18:55:30Z
updated_at: 2022-09-07T18:55:30Z
author_association: OWNER
body:
  That would be a breaking change though - existing code that registers functions with default parameters should continue to work unchanged (unless I want to ship `sqlite-utils` 4.0).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/484#issuecomment-1239762561
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/484
node_id: IC_kwDOCGYnMM5J5UaB
created_at: 2022-09-07T18:56:13Z
updated_at: 2022-09-07T18:56:13Z
author_association: OWNER
body:
  I could do this:
  ```python
   # Register all callables in the locals dict: 
   for name, value in globals.items(): 
       if callable(value) and not name.startswith("_"): 
           db.register_function(value, name=name, ignore_params_with_defaults=True) 
  ```
  Introducing a new `ignore_params_with_defaults` option.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/484#issuecomment-1239763997
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/484
node_id: IC_kwDOCGYnMM5J5Uwd
created_at: 2022-09-07T18:57:59Z
updated_at: 2022-09-07T18:58:10Z
author_association: OWNER
body:
  Here's how to detect defaults in the function signature:
  ```pycon
  >>> import inspect
  >>> def foo(a, b, c=1, d=2):
  ...      pass
  ... 
  >>> inspect.signature(foo)
  <Signature (a, b, c=1, d=2)>
  >>> inspect.signature(foo).parameters
  mappingproxy(OrderedDict([('a', <Parameter "a">), ('b', <Parameter "b">), ('c', <Parameter "c=1">), ('d', <Parameter "d=2">)]))
  >>> inspect.signature(foo).parameters['c']
  <Parameter "c=1">
  >>> dir(inspect.signature(foo).parameters['c'])
  ['KEYWORD_ONLY', 'POSITIONAL_ONLY', 'POSITIONAL_OR_KEYWORD', 'VAR_KEYWORD', 'VAR_POSITIONAL', '__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__sizeof__', '__slots__', '__str__', '__subclasshook__', '_annotation', '_default', '_kind', '_name', 'annotation', 'default', 'empty', 'kind', 'name', 'replace']
  >>> inspect.signature(foo).parameters['c'].default
  1
  >>> inspect.signature(foo).parameters['a'].default
  <class 'inspect._empty'>
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/484#issuecomment-1239766987
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/484
node_id: IC_kwDOCGYnMM5J5VfL
created_at: 2022-09-07T19:01:49Z
updated_at: 2022-09-07T19:01:49Z
author_association: OWNER
body:
  OK with this:
  ```diff
  diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py
  index c51b101..93d82a9 100644
  --- a/sqlite_utils/cli.py
  +++ b/sqlite_utils/cli.py
  @@ -30,6 +30,7 @@ from .utils import (
       Format,
       TypeTracker,
   )
  +from . import recipes
   
   
   CONTEXT_SETTINGS = dict(help_option_names=["-h", "--help"])
  @@ -3029,7 +3030,7 @@ def _load_extensions(db, load_extension):
   def _register_functions(db, functions):
       # Register any Python functions as SQL functions:
       sqlite3.enable_callback_tracebacks(True)
  -    globals = {}
  +    globals = {"r": recipes, "recipes": recipes}
       try:
           exec(functions, globals)
       except SyntaxError as ex:
  @@ -3037,4 +3038,4 @@ def _register_functions(db, functions):
       # Register all callables in the locals dict:
       for name, value in globals.items():
           if callable(value) and not name.startswith("_"):
  -            db.register_function(value, name=name)
  +            db.register_function(value, name=name, ignore_defaults=True)
  diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py
  index 27c46b0..1407d23 100644
  --- a/sqlite_utils/db.py
  +++ b/sqlite_utils/db.py
  @@ -370,6 +370,7 @@ class Database:
           self,
           fn: Callable = None,
           deterministic: bool = False,
  +        ignore_defaults: bool = False,
           replace: bool = False,
           name: Optional[str] = None,
       ):
  @@ -397,7 +398,10 @@ class Database:
   
           def register(fn):
               fn_name = name or fn.__name__
  -            arity = len(inspect.signature(fn).parameters)
  +            params = inspect.signature(fn).parameters
  +            if ignore_defaults:
  +                params = [p for p in params if params[p].default is inspect._empty]
  +            arity = len(params)
               if not replace and (fn_name, arity) in self._registered_functions:
                   return fn
               kwargs = {}
  ```
  I can now do this:
  ```
  % sqlite-utils memory --functions 'parsedate = r.parsedate' 'select parsedate("1st jan")'
  [{"parsedate(\"1st jan\")": "2022-01-01"}]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/484#issuecomment-1239772256
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/484
node_id: IC_kwDOCGYnMM5J5Wxg
created_at: 2022-09-07T19:07:51Z
updated_at: 2022-09-07T19:09:52Z
author_association: OWNER
body:
  Or... I could automatically register multiple copies of the function of different arities!

  If I'm going to do something like that though I need to think carefully about how functions that have keyword-only arguments should work: https://peps.python.org/pep-3102/

  ```python
  def compare(a, b, *ignore, key=None):
      ...
  ```
  I should think about how these work with `db.register_function()` anyway, since SQL functions cannot support keyword arguments.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1240882245
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/297
node_id: IC_kwDOCGYnMM5J9lxF
created_at: 2022-09-08T15:33:11Z
updated_at: 2022-09-08T15:33:11Z
author_association: OWNER
body:
  The more I think about this the more I like it - particularly for `sqlite-utils fast-insert` where differences in features aren't a problem.

  I used a variant of this trick with parquet files here: https://simonwillison.net/2022/Sep/5/laion-aesthetics-weeknotes/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1807#issuecomment-1241185851
issue_url: https://api.github.com/repos/simonw/datasette/issues/1807
node_id: IC_kwDOBm6k_c5J-v47
created_at: 2022-09-08T20:14:05Z
updated_at: 2022-09-08T20:14:05Z
author_association: OWNER
body:
  The entire concept of a "writable database" is something that's becoming increasingly important to be honest. Maybe Datasette should have that as a core concept which plugins can interact with?

  I had to write [this code](https://github.com/simonw/datasette-upload-csvs/blob/6b7581f5d5363bd17632ebbf5a1e7fe12beddab6/datasette_upload_csvs/__init__.py#L38) in `datasette-upload-csvs` which is a bit nasty: 

  ```python
          if await datasette.permission_allowed(
              actor, "upload-csvs", default=False
          ) and any(
              db.is_mutable and db.name not in ("_memory", "_internal")
              for db in datasette.databases.values()
          ):
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1808#issuecomment-1242164484
issue_url: https://api.github.com/repos/simonw/datasette/issues/1808
node_id: IC_kwDOBm6k_c5KCe0E
created_at: 2022-09-09T16:03:36Z
updated_at: 2022-09-09T16:05:08Z
author_association: OWNER
body:
  Fixing this is technically a breaking change, but to my knowledge there are VERY few plugins out there that use this particular API directly - and I would be extremely surprised if any of them actually relied on it using this default.

  (Good thing I've not shipped 1.0 yet though!)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1808#issuecomment-1242181891
issue_url: https://api.github.com/repos/simonw/datasette/issues/1808
node_id: IC_kwDOBm6k_c5KCjED
created_at: 2022-09-09T16:18:54Z
updated_at: 2022-09-09T16:18:54Z
author_association: OWNER
body:
  Weirdly `_memory` was being treated as immutable too - which doesn't really make sense, since it's not a file on disk that can be opened in immutable mode. Fixing that too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1808#issuecomment-1242203798
issue_url: https://api.github.com/repos/simonw/datasette/issues/1808
node_id: IC_kwDOBm6k_c5KCoaW
created_at: 2022-09-09T16:36:23Z
updated_at: 2022-09-09T16:37:57Z
author_association: OWNER
body:
  Yes, it should do - actually this fix to `datasette-upload-dbs` should sort that out even if you don't upgrade to the (unreleased) latest Datasette:
  - https://github.com/simonw/datasette-upload-dbs/issues/6

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1808#issuecomment-1242204717
issue_url: https://api.github.com/repos/simonw/datasette/issues/1808
node_id: IC_kwDOBm6k_c5KCoot
created_at: 2022-09-09T16:37:12Z
updated_at: 2022-09-09T16:37:12Z
author_association: OWNER
body:
  Updated documentation: https://docs.datasette.io/en/latest/internals.html#database-ds-path-none-is-mutable-true-is-memory-false-memory-name-none

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/487#issuecomment-1242513223
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/487
node_id: IC_kwDOCGYnMM5KDz9H
created_at: 2022-09-09T22:01:10Z
updated_at: 2022-09-09T22:01:10Z
author_association: NONE
body:
  Perfect. Apologies, I searched but didn’t see that one for some reason.
  Thank you.

  On Fri, Sep 9, 2022 at 3:04 PM Simon Willison ***@***.***>
  wrote:

  > This isn't supported yet - there's an older issue for that here:
  >
  >    - #117 <https://github.com/simonw/sqlite-utils/issues/117>
  >
  > —
  > Reply to this email directly, view it on GitHub
  > <https://github.com/simonw/sqlite-utils/issues/487#issuecomment-1242409766>,
  > or unsubscribe
  > <https://github.com/notifications/unsubscribe-auth/AAEECKBODHZCKWTN5OO6RI3V5OJ5RANCNFSM6AAAAAAQIV3YIU>
  > .
  > You are receiving this because you authored the thread.Message ID:
  > ***@***.***>
  >

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/488#issuecomment-1246971764
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/488
node_id: IC_kwDOCGYnMM5KU0d0
created_at: 2022-09-14T15:52:14Z
updated_at: 2022-09-14T15:52:14Z
author_association: OWNER
body:
  Frustratingly I think this counts as a backwards-incompatible change.

  Could have it be opt-in with a new option / method parameter, and then change it to the default if I release a `sqlite-utils 4`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1246977989
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/297
node_id: IC_kwDOCGYnMM5KU1_F
created_at: 2022-09-14T15:57:09Z
updated_at: 2022-09-14T15:57:09Z
author_association: OWNER
body:
  Should consider how this could best handle creating columns that are integer and float as opposed to just text.

  https://discord.com/channels/823971286308356157/823971286941302908/1019630014544748584 is a relevant discussion on Discord. Even if you create the schema in advance with the correct column types, this import mechanism can put empty strings in blank float/integer columns when ideally you would want to have nulls.

  Related feature idea for `sqlite-utils transform`:
  - #488

  Not sure how best to handle this for `sqlite3 .import` imports.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1246978641
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/297
node_id: IC_kwDOCGYnMM5KU2JR
created_at: 2022-09-14T15:57:41Z
updated_at: 2022-09-14T15:57:41Z
author_association: OWNER
body:
  One solution suggested on Discord:
  ```
  wget https://static.openfoodfacts.org/data/en.openfoodfacts.org.products.csv
  CREATE=`curl -s -L https://gist.githubusercontent.com/CharlesNepote/80fb813a416ad445fdd6e4738b4c8156/raw/032af70de631ff1c4dd09d55360f242949dcc24f/create.sql`
  INDEX=`curl -s -L https://gist.githubusercontent.com/CharlesNepote/80fb813a416ad445fdd6e4738b4c8156/raw/032af70de631ff1c4dd09d55360f242949dcc24f/index.sql`
  time sqlite3 products_new.db <<EOS
  /* Optimisations. See: https://avi.im/blag/2021/fast-sqlite-inserts/ */;
  PRAGMA page_size = 32768;
  $CREATE
  .mode ascii
  .separator "\t" "\n"
  .import --skip 1 en.openfoodfacts.org.products.csv all
  $INDEX
  EOS
  # Converting empty to NULL for columns which are either FLOAT or INTEGER
  time sqlite3 products.db ".schema all" | sed -nr 's/.*\[(.*)\] (INTEGER|FLOAT).*/\1/gp' | xargs -I % sqlite3 products.db -cmd "PRAGMA journal_mode=OFF;" "UPDATE [all] SET [%] = NULL WHERE [%] = '';"
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1247149969
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/297
node_id: IC_kwDOCGYnMM5KVf-R
created_at: 2022-09-14T18:28:53Z
updated_at: 2022-09-14T18:29:34Z
author_association: OWNER
body:
  As an aside, https://avi.im/blag/2021/fast-sqlite-inserts/ inspired my to try pypy since that article claimed to get a 2.5x speedup using pypy compared to regular Python for a CSV import script.

  Setup:
  ```
  brew install pypy3
  cd /tmp
  pypy3 -m venv venv
  source venv/bin/activate
  pip install sqlite-utils
  ```
  I grabbed the first 760M of that `https://static.openfoodfacts.org/data/en.openfoodfacts.org.products.csv` file (didn't wait for the whole thing to download).

  Then:
  ```
  time sqlite-utils insert pypy.db t en.openfoodfacts.org.products.csv --csv
    [------------------------------------]    0%
    [###################################-]   99% 
  11.76s user 2.26s system 93% cpu 14.981 total
  ```
  Compared to regular Python `sqlite-utils` doing the same thing:
  ```
  time sqlite-utils insert py.db t en.openfoodfacts.org.products.csv --csv
    [------------------------------------]    0%
    [###################################-]   99%  
  11.36s user 2.06s system 93% cpu 14.341 total
  ```
  So no perceivable performance difference.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1809#issuecomment-1247311275
issue_url: https://api.github.com/repos/simonw/datasette/issues/1809
node_id: IC_kwDOBm6k_c5KWHWr
created_at: 2022-09-14T21:16:32Z
updated_at: 2022-09-14T21:16:32Z
author_association: OWNER
body:
  It should also implement the `await_me_maybe` pattern so you can return an `async` function from it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1809#issuecomment-1247313134
issue_url: https://api.github.com/repos/simonw/datasette/issues/1809
node_id: IC_kwDOBm6k_c5KWHzu
created_at: 2022-09-14T21:18:46Z
updated_at: 2022-09-14T21:18:46Z
author_association: OWNER
body:
  `await_me_maybe` might be tricky though, because the only place that hook is executed is here:

  https://github.com/simonw/datasette/blob/8430c3bc7dd22b173c1a8c6cd7180e3b31240cd1/datasette/app.py#L348

  Which is inside the `Datasette.__init__` method.

  To implement an `await` on that would need to move it somewhere else - probably here: https://github.com/simonw/datasette/blob/8430c3bc7dd22b173c1a8c6cd7180e3b31240cd1/datasette/app.py#L391-L393

  But I'm not 100% confident that is always executed at the right time to ensure the Jinja environment is properly configured? I think it is, but would need to make sure.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1809#issuecomment-1247314352
issue_url: https://api.github.com/repos/simonw/datasette/issues/1809
node_id: IC_kwDOBm6k_c5KWIGw
created_at: 2022-09-14T21:20:12Z
updated_at: 2022-09-14T21:20:12Z
author_association: OWNER
body:
  The reason to support `await_me_maybe` is in case a hook wants to execute a SQL query as part of configuring the Jinja template loader.

  That's exactly what `datasette-edit-templates` needs to do, though it's currently achieving that in a `startup()` hook instead: https://github.com/simonw/datasette-edit-templates/blob/087f6a6cabc20020f2b0524f11aa3a7836320848/datasette_edit_templates/__init__.py#L32-L48

  ```python
  @hookimpl
  def startup(datasette):
      datasette._edit_templates = {}

      async def inner():
          db = get_database(datasette)
          # Does the table exist?
          if not await db.table_exists(TABLE):
              for sql in CREATE_TABLE:
                  await db.execute_write(sql, block=True)
          else:
              # Load all templates from that table
              rows = await db.execute("select template, body FROM {}".format(TABLE))
              for name, content in rows:
                  datasette._edit_templates[name] = content

      return inner
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1809#issuecomment-1247316097
issue_url: https://api.github.com/repos/simonw/datasette/issues/1809
node_id: IC_kwDOBm6k_c5KWIiB
created_at: 2022-09-14T21:22:24Z
updated_at: 2022-09-14T21:22:24Z
author_association: OWNER
body:
  It looks like this is the only place that calls `invoke_startup()`: https://github.com/simonw/datasette/blob/1d64c9a8dac45b9a3452acf8e76dfadea2b0bc49/datasette/cli.py#L590-L591

  `datasette-publish-vercel` is the one deployment mechanism that skips running Uvicorn, and it calls that method separately here: https://github.com/simonw/datasette-publish-vercel/blob/1559d979b4e3b1f2f83c51c3c0c10192ff9a6d0c/datasette_publish_vercel/__init__.py#L42-L52

  ```python
  ds = Datasette(
      [],
      {database_files},
      static_mounts=static_mounts,
      metadata=metadata{extras},
      secret=secret,
      cors=True,
      settings={settings}{crossdb}
  )
  asyncio.run(ds.invoke_startup())
  app = ds.app()
  ```
  So preparing the Jinja environment inside that function would work fine.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1809#issuecomment-1247316715
issue_url: https://api.github.com/repos/simonw/datasette/issues/1809
node_id: IC_kwDOBm6k_c5KWIrr
created_at: 2022-09-14T21:23:10Z
updated_at: 2022-09-14T21:23:10Z
author_association: OWNER
body:
  It might be good to have Datasette LOUDLY fail if you attempt to use it without calling `await ds.invoke_startup()`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1809#issuecomment-1247317941
issue_url: https://api.github.com/repos/simonw/datasette/issues/1809
node_id: IC_kwDOBm6k_c5KWI-1
created_at: 2022-09-14T21:24:43Z
updated_at: 2022-09-14T21:24:43Z
author_association: OWNER
body:
  It looks like Datasette Lite does NOT invoke that method, which is likely a bug: https://github.com/simonw/datasette-lite/blob/e7ccaf621b3cdf613ebaf544304d387f2af32edf/webworker.js#L103-L110

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1810#issuecomment-1248187089
issue_url: https://api.github.com/repos/simonw/datasette/issues/1810
node_id: IC_kwDOBm6k_c5KZdLR
created_at: 2022-09-15T14:31:36Z
updated_at: 2022-09-15T14:31:36Z
author_association: OWNER
body:
  Twitter conversation that inspired this issue: https://twitter.com/psychemedia/status/1570410108785684481

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1810#issuecomment-1248204219
issue_url: https://api.github.com/repos/simonw/datasette/issues/1810
node_id: IC_kwDOBm6k_c5KZhW7
created_at: 2022-09-15T14:44:47Z
updated_at: 2022-09-15T14:46:26Z
author_association: CONTRIBUTOR
body:
  A couple+ of possible use case examples:

  - someone has a collection of articles indexed with FTS; they want to publish a simple search tool over the results;
  - someone has an image collection and they want to be able to search over description text to return images;
  - someone has a set of locations with descriptions, and wants to run a query over places and descriptions and get results as a listing or on a map;
  - someone has a set of audio or video files with titles, descriptions and/or transcripts, and wants to be able to search over them and return playable versions of returned items.

  In many cases, I suspect the raw content will be in one table, but the search table will be a second (eg FTS) table. Generally, the search may be over one or more joined tables, and the results constructed from one or more tables (which may or may not be distinct from the search tables).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1810#issuecomment-1248289857
issue_url: https://api.github.com/repos/simonw/datasette/issues/1810
node_id: IC_kwDOBm6k_c5KZ2RB
created_at: 2022-09-15T15:50:46Z
updated_at: 2022-09-15T15:50:46Z
author_association: OWNER
body:
  Idea: allow the user to specify one or more featured tables. Each table is then shown as a summary on the homepage - with the total number of rows and the first 5 rows. If the table has search configured there's a search box too.

  If the instance has only one database with only one table (excluding hidden tables) it gets featured automatically perhaps (maybe with a way to opt-out of that if you want to).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1810#issuecomment-1248290151
issue_url: https://api.github.com/repos/simonw/datasette/issues/1810
node_id: IC_kwDOBm6k_c5KZ2Vn
created_at: 2022-09-15T15:51:04Z
updated_at: 2022-09-15T15:51:25Z
author_association: OWNER
body:
  I could prototype this idea as a `datasette-featured-tables` plugin that delivers its own custom `index.html` template.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/406#issuecomment-1248440137
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/406
node_id: IC_kwDOCGYnMM5Kaa9J
created_at: 2022-09-15T18:13:50Z
updated_at: 2022-09-15T18:13:50Z
author_association: NONE
body:
  I was wondering if you have any more thoughts on this? I have a tangible use case now: adding a "vector" column to a database to support semantic search using doc2vec embeddings ([example](https://psychemedia.github.io/storynotes/Lang_Doc2Vec.html); note that the `vtfunc` package may no longer be reliable...).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/489#issuecomment-1248474806
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/489
node_id: IC_kwDOCGYnMM5Kaja2
created_at: 2022-09-15T18:48:09Z
updated_at: 2022-09-15T18:48:09Z
author_association: OWNER
body:
  Built a prototype of this that works really well:
  ```diff
   diff --git a/sqlite_utils/utils.py b/sqlite_utils/utils.py
  index c0b7bf1..f9a482c 100644
  --- a/sqlite_utils/utils.py
  +++ b/sqlite_utils/utils.py
  @@ -272,7 +272,19 @@ def rows_from_file(
       if format == Format.JSON:
           decoded = json.load(fp)
           if isinstance(decoded, dict):
  -            decoded = [decoded]
  +            # TODO: Solve for if this isn't what people want
  +            # Does it have just one key that is a list of dicts?
  +            list_keys = [
  +                k
  +                for k in decoded
  +                if isinstance(decoded[k], list)
  +                and decoded[k]
  +                and all(isinstance(o, dict) for o in decoded[k])
  +            ]
  +            if len(list_keys) == 1:
  +                decoded = decoded[list_keys[0]]
  +            else:
  +                decoded = [decoded]
           if not isinstance(decoded, list):
               raise RowsFromFileBadJSON("JSON must be a list or a dictionary")
           return decoded, Format.JSON
  ```
  I used that to build this: https://gist.github.com/simonw/0e6901974a14ab7d56c2746a04d72c8c

  One problem though: right now, if you do this `sqlite-utils` treats it as a single object and adds a `tags` column with JSON in it:
  ```
  echo '{"title": "Hi", "tags": [{"t": "one"}]}` | sqlite-utils insert db.db t -
  ```
  If I implement this new mechanism the above line would behave differently - which would be a backwards incompatible change.

  So I probably need some kind of opt-in mechanism for this. And I need a good name for it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/489#issuecomment-1248475718
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/489
node_id: IC_kwDOCGYnMM5KajpG
created_at: 2022-09-15T18:49:05Z
updated_at: 2022-09-15T18:49:53Z
author_association: OWNER
body:
  Here's how I used my prototype to build [that Gist](https://gist.github.com/simonw/0e6901974a14ab7d56c2746a04d72c8c):

      sqlite-utils memory ~/Downloads/CVR_Export_20220908084311/*.json --schema > database.sql

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/489#issuecomment-1248479485
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/489
node_id: IC_kwDOCGYnMM5Kakj9
created_at: 2022-09-15T18:52:52Z
updated_at: 2022-09-15T18:53:45Z
author_association: OWNER
body:
  The most similar option I have at the moment is probably `--flatten`. What would good names for this option be?

  - `--auto-list`
  - `--auto-key`
  - `--inner-key`
  - `--auto-json`
  - `--find-list`
  - `--find-key`

  Those are all bad.

  Another option: introduce a new explicit format for it. Right now the explicit formats you can use are:

  https://github.com/simonw/sqlite-utils/blob/d9b9e075f07a20f1137cd2e34ed5d3f1a3db4ad8/docs/cli-reference.rst#L153-L158

  So I could add a `:autojson` format.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/489#issuecomment-1248481303
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/489
node_id: IC_kwDOCGYnMM5KalAX
created_at: 2022-09-15T18:54:30Z
updated_at: 2022-09-15T18:55:14Z
author_association: OWNER
body:
  Maybe this would make more sense as a mechanism where you can say "Use the data in the key called X" - but there's a special option for "figure out that key automatically".

  The syntax then could be:

  `--list-key List`

  Or for automatic detection:

  `--list-key-auto`

  Could also go with `--key List` and `--key-auto` - but would that be as obvious as `--list-key`?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/489#issuecomment-1248484094
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/489
node_id: IC_kwDOCGYnMM5Kalr-
created_at: 2022-09-15T18:56:31Z
updated_at: 2022-09-15T18:56:31Z
author_association: OWNER
body:
  Actually I quite like `--key X` - it could work for single nested objects too. You could insert a single record like this:

  ```json
  {
    "record" {
      "id": 1
    }
  }
  ```
  ```
  sqlite-utils insert db.db records record.json --key record
  ``` 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/489#issuecomment-1248501824
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/489
node_id: IC_kwDOCGYnMM5KaqBA
created_at: 2022-09-15T19:10:48Z
updated_at: 2022-09-15T19:10:48Z
author_association: OWNER
body:
  This feels pretty good:
  ```
  % sqlite-utils memory ~/Downloads/CVR_Export_20220908084311/*.json --schema --auto-key
  CREATE TABLE [BallotTypeContestManifest] (
     [BallotTypeId] INTEGER,
     [ContestId] INTEGER
  );
  CREATE VIEW t1 AS select * from [BallotTypeContestManifest];
  CREATE VIEW t AS select * from [BallotTypeContestManifest];
  CREATE TABLE [BallotTypeManifest] (
     [Description] TEXT,
     [Id] INTEGER,
     [ExternalId] TEXT
  );
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/489#issuecomment-1248512739
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/489
node_id: IC_kwDOCGYnMM5Kasrj
created_at: 2022-09-15T19:18:24Z
updated_at: 2022-09-15T19:21:01Z
author_association: OWNER
body:
  Why doesn't `sqlite-utils insert` use the `rows_from_file` function I wonder?

  https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864207841 says:

  > I can refactor `sqlite-utils insert` to use this new code too.

  Maybe I forgot to do that?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/489#issuecomment-1248522618
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/489
node_id: IC_kwDOCGYnMM5KavF6
created_at: 2022-09-15T19:29:20Z
updated_at: 2022-09-15T19:29:20Z
author_association: OWNER
body:
  I think refactoring `sqlite-utils insert` to use `rows_from_file` needs to happen as part of this work.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248527646
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/486
node_id: IC_kwDOCGYnMM5KawUe
created_at: 2022-09-15T19:34:59Z
updated_at: 2022-09-15T20:23:12Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/486?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Base: **96.47**% // Head: **96.52**% // Increases project coverage by **`+0.04%`** :tada:
  > Coverage data is based on head [(`0acbc68`)](https://codecov.io/gh/simonw/sqlite-utils/pull/486?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`d9b9e07`)](https://codecov.io/gh/simonw/sqlite-utils/commit/d9b9e075f07a20f1137cd2e34ed5d3f1a3db4ad8?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > Patch coverage: 100.00% of modified lines in pull request are covered.

  > :exclamation: Current head 0acbc68 differs from pull request most recent head d5db749. Consider uploading reports for the commit d5db749 to get more accurate results

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@            Coverage Diff             @@
  ##             main     #486      +/-   ##
  ==========================================
  + Coverage   96.47%   96.52%   +0.04%     
  ==========================================
    Files           6        6              
    Lines        2642     2646       +4     
  ==========================================
  + Hits         2549     2554       +5     
  + Misses         93       92       -1     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/486?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [sqlite\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/486/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.86% <100.00%> (ø)` | |
  | [sqlite\_utils/utils.py](https://codecov.io/gh/simonw/sqlite-utils/pull/486/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `94.98% <100.00%> (+0.47%)` | :arrow_up: |

  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  </details>

  [:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/486?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248565396
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/486
node_id: IC_kwDOCGYnMM5Ka5iU
created_at: 2022-09-15T20:12:50Z
updated_at: 2022-09-15T20:12:50Z
author_association: OWNER
body:
  Annoying `mypy` test failure:

  ```
  /Users/runner/hostedtoolcache/Python/3.10.7/x64/lib/python3.10/site-packages/numpy/__init__.pyi:636:
  error: Positional-only parameters are only supported in Python 3.8 and greater
  ```
  Looks like this:
  - https://github.com/python/mypy/issues/13627

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248567323
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/486
node_id: IC_kwDOCGYnMM5Ka6Ab
created_at: 2022-09-15T20:14:45Z
updated_at: 2022-09-15T20:14:45Z
author_association: OWNER
body:
  There's a fix for `mypy` that has landed but isn't out in a release yet:
  - https://github.com/python/mypy/issues/13385

  For the moment looks like pinning to Python 3.10.6 could help. Need to figure out how to do that in GitHub Actions though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248568775
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/486
node_id: IC_kwDOCGYnMM5Ka6XH
created_at: 2022-09-15T20:16:14Z
updated_at: 2022-09-15T20:16:14Z
author_association: OWNER
body:
  https://github.com/actions/setup-python/blob/main/docs/advanced-usage.md#using-the-python-version-input says can set the full version:

  ```
  - uses: actions/setup-python@v4
    with:
      python-version: "3.10.6" 
  ```
  I'll try that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248582147
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/486
node_id: IC_kwDOCGYnMM5Ka9oD
created_at: 2022-09-15T20:29:17Z
updated_at: 2022-09-15T20:29:17Z
author_association: OWNER
body:
  This looks good to me. I need to run some manual tests before merging (it's a good sign that the automated tests pass though).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248591268
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/486
node_id: IC_kwDOCGYnMM5Ka_2k
created_at: 2022-09-15T20:36:02Z
updated_at: 2022-09-15T20:40:03Z
author_association: OWNER
body:
  I had a big CSV file lying around, I converted it to other formats like this:

      sqlite-utils insert /tmp/t.db t /tmp/en.openfoodfacts.org.products.csv --csv
      sqlite-utils rows /tmp/t.db t --nl > /tmp/big.nl
      sqlite-utils rows /tmp/t.db t > /tmp/big.json

  Then tested the progress bar like this:

      sqlite-utils insert /tmp/t2.db t /tmp/big.nl --nl

  Output:

  ```
  sqlite-utils insert /tmp/t2.db t /tmp/big.nl --nl
    [------------------------------------]    0%
    [#######-----------------------------]   20%  00:00:20
  ```
  With `--silent` it is silent.

  And for regular JSON:

  ```
  sqlite-utils insert /tmp/t3.db t /tmp/big.json 
    [####################################]  100%
  ```
  This is actually not doing the right thing. The problem is that `sqlite-utils` doesn't include a streaming JSON parser, so it instead reads that entire JSON file into memory first (exhausting the progress bar to 100% instantly) and then does the rest of the work in-memory while the bar sticks at 100%.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248593835
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/486
node_id: IC_kwDOCGYnMM5KbAer
created_at: 2022-09-15T20:37:14Z
updated_at: 2022-09-15T20:37:14Z
author_association: OWNER
body:
  I'm going to land this anyway. The lack of a streaming JSON parser is a separate issue, I don't think it should block landing this improvement.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/485#issuecomment-1248597643
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/485
node_id: IC_kwDOCGYnMM5KbBaL
created_at: 2022-09-15T20:39:39Z
updated_at: 2022-09-15T20:39:52Z
author_association: OWNER
body:
  A note from PR #486: https://github.com/simonw/sqlite-utils/issues/486#issuecomment-1248591268_

  > ```
  > sqlite-utils insert /tmp/t3.db t /tmp/big.json 
  >   [####################################]  100%
  > ```
  > This is actually not doing the right thing. The problem is that `sqlite-utils` doesn't include a streaming JSON parser, so it instead reads that entire JSON file into memory first (exhausting the progress bar to 100% instantly) and then does the rest of the work in-memory while the bar sticks at 100%.

  I decided to land this anyway. If a streaming JSON parser is added later it will start to work.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/489#issuecomment-1248621072
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/489
node_id: IC_kwDOCGYnMM5KbHIQ
created_at: 2022-09-15T20:56:09Z
updated_at: 2022-09-15T20:56:09Z
author_association: OWNER
body:
  Prototype so far:
  ```diff
  diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py
  index 767b170..d96c507 100644
  --- a/sqlite_utils/cli.py
  +++ b/sqlite_utils/cli.py
  @@ -1762,6 +1762,17 @@ def query(
       is_flag=True,
       help="Analyze resulting tables and output results",
   )
  +@click.option("--key", help="read data from this key of the root object")
  +@click.option(
  +    "--auto-key",
  +    is_flag=True,
  +    help="Find a key in the root object that is a list of objects",
  +)
  +@click.option(
  +    "--analyze",
  +    is_flag=True,
  +    help="Analyze resulting tables and output results",
  +)
   @load_extension_option
   def memory(
       paths,
  @@ -1784,6 +1795,8 @@ def memory(
       schema,
       dump,
       save,
  +    key,
  +    auto_key,
       analyze,
       load_extension,
   ):
  @@ -1838,7 +1851,9 @@ def memory(
                   csv_table = stem
               stem_counts[stem] = stem_counts.get(stem, 1) + 1
               csv_fp = csv_path.open("rb")
  -        rows, format_used = rows_from_file(csv_fp, format=format, encoding=encoding)
  +        rows, format_used = rows_from_file(
  +            csv_fp, format=format, encoding=encoding, key=key, auto_key=auto_key
  +        )
           tracker = None
           if format_used in (Format.CSV, Format.TSV) and not no_detect_types:
               tracker = TypeTracker()
  diff --git a/sqlite_utils/utils.py b/sqlite_utils/utils.py
  index 8754554..2e69c26 100644
  --- a/sqlite_utils/utils.py
  +++ b/sqlite_utils/utils.py
  @@ -231,6 +231,8 @@ def rows_from_file(
       encoding: Optional[str] = None,
       ignore_extras: Optional[bool] = False,
       extras_key: Optional[str] = None,
  +    key: Optional[str] = None,
  +    auto_key: Optional[bool] = False,
   ) -> Tuple[Iterable[dict], Format]:
       """
       Load a sequence of dictionaries from a file-like object containing one of four different formats.
  @@ -271,13 +273,31 @@ def rows_from_file(
       :param encoding: the character encoding to use when reading CSV/TSV data
       :param ignore_extras: ignore any extra fields on rows
       :param extras_key: put any extra fields in a list with this key
  +    :param key: read data from this key of the root object
  +    :param auto_key: find a key in the root object that is a list of objects
       """
       if ignore_extras and extras_key:
           raise ValueError("Cannot use ignore_extras= and extras_key= together")
  +    if key and auto_key:
  +        raise ValueError("Cannot use key= and auto_key= together")
       if format == Format.JSON:
           decoded = json.load(fp)
           if isinstance(decoded, dict):
  -            decoded = [decoded]
  +            if auto_key:
  +                list_keys = [
  +                    k
  +                    for k in decoded
  +                    if isinstance(decoded[k], list)
  +                    and decoded[k]
  +                    and all(isinstance(o, dict) for o in decoded[k])
  +                ]
  +                if len(list_keys) == 1:
  +                    decoded = decoded[list_keys[0]]
  +            elif key:
  +                # Raises KeyError, I think that's OK
  +                decoded = decoded[key]
  +            if not isinstance(decoded, list):
  +                decoded = [decoded]
           if not isinstance(decoded, list):
               raise RowsFromFileBadJSON("JSON must be a list or a dictionary")
           return decoded, Format.JSON
  @@ -305,7 +325,9 @@ def rows_from_file(
           first_bytes = buffered.peek(2048).strip()
           if first_bytes.startswith(b"[") or first_bytes.startswith(b"{"):
               # TODO: Detect newline-JSON
  -            return rows_from_file(buffered, format=Format.JSON)
  +            return rows_from_file(
  +                buffered, format=Format.JSON, key=key, auto_key=auto_key
  +            )
           else:
               dialect = csv.Sniffer().sniff(
                   first_bytes.decode(encoding or "utf-8-sig", "ignore")
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1812#issuecomment-1249355888
issue_url: https://api.github.com/repos/simonw/datasette/issues/1812
node_id: IC_kwDOBm6k_c5Kd6hw
created_at: 2022-09-16T13:18:37Z
updated_at: 2022-09-16T13:18:37Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1812?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Base: **91.70**% // Head: **91.70**% // No change to project coverage :thumbsup:
  > Coverage data is based on head [(`b3855e7`)](https://codecov.io/gh/simonw/datasette/pull/1812?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`b40872f`)](https://codecov.io/gh/simonw/datasette/commit/b40872f5e5ae5dad331c58f75451e2d206565196?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > Patch has no changes to coverable lines.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main    #1812   +/-   ##
  =======================================
    Coverage   91.70%   91.70%           
  =======================================
    Files          38       38           
    Lines        4735     4735           
  =======================================
    Hits         4342     4342           
    Misses        393      393           
  ```



  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  </details>

  [:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1812?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1812#issuecomment-1249746777
issue_url: https://api.github.com/repos/simonw/datasette/issues/1812
node_id: IC_kwDOBm6k_c5KfZ9Z
created_at: 2022-09-16T19:50:45Z
updated_at: 2022-09-16T19:50:45Z
author_association: OWNER
body:
  Main difference I can see:

  ![CleanShot 2022-09-16 at 12 49 47@2x](https://user-images.githubusercontent.com/9599/190719563-a7b1bcc7-bfdc-4759-95c1-e19bcd0217c3.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1809#issuecomment-1249985741
issue_url: https://api.github.com/repos/simonw/datasette/issues/1809
node_id: IC_kwDOBm6k_c5KgUTN
created_at: 2022-09-17T03:04:51Z
updated_at: 2022-09-17T03:04:51Z
author_association: OWNER
body:
  I'm going to throw an error in `ds.render_template()` if you haven't previously called `await ds.invoke_startup()`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1809#issuecomment-1249985971
issue_url: https://api.github.com/repos/simonw/datasette/issues/1809
node_id: IC_kwDOBm6k_c5KgUWz
created_at: 2022-09-17T03:06:32Z
updated_at: 2022-09-17T03:06:32Z
author_association: OWNER
body:
  This is likely going to cause some tests in plugins to break, but I'm OK with that - I'll fix them as I find them once this release is out.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1809#issuecomment-1249986079
issue_url: https://api.github.com/repos/simonw/datasette/issues/1809
node_id: IC_kwDOBm6k_c5KgUYf
created_at: 2022-09-17T03:07:24Z
updated_at: 2022-09-17T03:07:24Z
author_association: OWNER
body:
  Datasette's own tests started to break because calls to the `TestClient` were performed without awaiting that method. I fixed that by adding this to `_request()` inside that class:

  ```python
      async def _request(
          self,
          path,
          follow_redirects=True,
          redirect_count=0,
          method="GET",
          cookies=None,
          headers=None,
          post_body=None,
          content_type=None,
          if_none_match=None,
      ):
          if not self.ds._startup_invoked:
              await self.ds.invoke_startup()
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1809#issuecomment-1249987643
issue_url: https://api.github.com/repos/simonw/datasette/issues/1809
node_id: IC_kwDOBm6k_c5KgUw7
created_at: 2022-09-17T03:19:24Z
updated_at: 2022-09-17T03:19:24Z
author_association: OWNER
body:
  In looking at the documentation on [writing tests](https://docs.datasette.io/en/latest/testing_plugins.html), there are a lot of examples like this:
  ```python
      def test_that_opens_the_debugger_or_errors():
          ds = Datasette([db_path], pdb=True)
          response = await ds.client.get("/")
  ```
  I really don't like having to tell people to add `await ds.invoke_startup()` to every test that might look like this.

  Since it's safe to call that function multiple times, I'm going to have `ds.client.get()` and friends call it for you too - so if you forget in a plugin test it won't matter.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1809#issuecomment-1249990033
issue_url: https://api.github.com/repos/simonw/datasette/issues/1809
node_id: IC_kwDOBm6k_c5KgVWR
created_at: 2022-09-17T03:39:05Z
updated_at: 2022-09-17T03:39:05Z
author_association: OWNER
body:
  New docs section on the need to call `await ds.invoke_startup()`: https://github.com/simonw/datasette/blob/ddc999ad1296e8c69cffede3e367dda059b8adad/docs/testing_plugins.rst#setting-up-a-datasette-test-instance

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1813#issuecomment-1250901367
issue_url: https://api.github.com/repos/simonw/datasette/issues/1813
node_id: IC_kwDOBm6k_c5Kjz13
created_at: 2022-09-19T11:34:45Z
updated_at: 2022-09-19T11:34:45Z
author_association: CONTRIBUTOR
body:
  oh and by writing this I just realized the difference: the URL on fly.io is with a custom SQL command whereas the local one is without. 
  It seems that there is no pagination when using custom SQL commands which makes sense

  Sorry for this useless issue, maybe this can be useful for someone else / me in the future.

  Thanks again for this wonderful project !

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1814#issuecomment-1251677220
issue_url: https://api.github.com/repos/simonw/datasette/issues/1814
node_id: IC_kwDOBm6k_c5KmxQk
created_at: 2022-09-19T23:34:30Z
updated_at: 2022-09-19T23:34:30Z
author_association: OWNER
body:
  The `settings.json` file can only be used with settings that are set using `--setting name value` - the full list of those is here: https://docs.datasette.io/en/stable/settings.html

  The `--static` option works differently. In configuration directory mode you can skip it entirely and instead have a `/static/` folder - so your directory structure would look like this:

  ```
  bibliography/static/styles.css
  ```
  And then when you run `datasette bibliography/` the following URL will work:

      http://127.0.0.1:8001/static/styles.css

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1814#issuecomment-1251677554
issue_url: https://api.github.com/repos/simonw/datasette/issues/1814
node_id: IC_kwDOBm6k_c5KmxVy
created_at: 2022-09-19T23:35:06Z
updated_at: 2022-09-19T23:35:06Z
author_association: OWNER
body:
  It might have been useful for Datasette to show an error when started against a `settings.json` file that contains an invalid setting though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1816#issuecomment-1251682970
issue_url: https://api.github.com/repos/simonw/datasette/issues/1816
node_id: IC_kwDOBm6k_c5Kmyqa
created_at: 2022-09-19T23:44:54Z
updated_at: 2022-09-19T23:44:54Z
author_association: OWNER
body:
  I was going to add type validation too, but that's actually a bit tricky because the logic for that currently lives in Click option parsing here: https://github.com/simonw/datasette/blob/ddc999ad1296e8c69cffede3e367dda059b8adad/datasette/cli.py#L71-L88

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/31#issuecomment-1251845216
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/31
node_id: IC_kwDODEm0Qs5KnaRg
created_at: 2022-09-20T05:05:03Z
updated_at: 2022-09-20T05:05:03Z
author_association: NONE
body:
  yay! Thanks a bunch for the `twitter-to-sqlite friends` command!

  The twitter "Download an archive of your data" feature doesn't include who I follow, so this is particularly handy.

  The whole Dogsheep thing is great :) I've written about similar things under [cloud-services](https://www.madmode.com/search/label/cloud-services/):
   - 2021: [Closet Librarian Approach to Cloud Services](https://www.madmode.com/2021/closet-librarian-approach-cloud-services.html)
   - 2015: [jukekb \- Browse iTunes libraries and upload playlists to Google Music](https://www.madmode.com/2015/jukekb)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/433#issuecomment-1252898131
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/433
node_id: IC_kwDOCGYnMM5KrbVT
created_at: 2022-09-20T20:51:21Z
updated_at: 2022-09-20T20:56:07Z
author_association: CONTRIBUTOR
body:
  When I run `reset` it fixes my terminal. I suspect it is related to the progress bar

  https://linux.die.net/man/1/reset

  ```
  950 1s /m/d/03_Downloads 🐑 echo $TERM
  xterm-kitty
  ▓░▒░ /m/d/03_Downloads 🌏 kitty -v
  kitty 0.26.2 created by Kovid Goyal
  $ sqlite-utils insert test.db facility facility-boundary-us-all.csv --csv
  blah blah blah (no offense)
  $ <no cursor>
  $ reset
  $ <cursor lives again (resurrection [explicit])>
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/488#issuecomment-1254029808
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/488
node_id: IC_kwDOCGYnMM5Kvvnw
created_at: 2022-09-21T17:45:20Z
updated_at: 2022-09-21T17:45:41Z
author_association: OWNER
body:
  No, I'm going to say that this is a bug - it's WEIRD having a `integer` or `float` column containing an empty string. I'm OK changing that - I very much doubt anyone is relying on this functionality.

  So no need for a new option here - just fixing the bug is sensible.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/488#issuecomment-1254032378
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/488
node_id: IC_kwDOCGYnMM5KvwP6
created_at: 2022-09-21T17:47:54Z
updated_at: 2022-09-21T17:47:54Z
author_association: OWNER
body:
  New tests should go in: https://github.com/simonw/sqlite-utils/blob/main/tests/test_transform.py

  I think the implementation fix needs to go near here: https://github.com/simonw/sqlite-utils/blob/0b315d3fa83c1584eaeec32f24912898621e437a/sqlite_utils/db.py#L1770-L1775

  The trick is going to be teaching that generated SQL to know which columns are `integer` or `float` and to convert `""` to `null` as part of that operation. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/488#issuecomment-1254033981
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/488
node_id: IC_kwDOCGYnMM5Kvwo9
created_at: 2022-09-21T17:49:32Z
updated_at: 2022-09-21T17:50:10Z
author_association: OWNER
body:
  It looks like SQLite has a `SELECT NULLIF(value, '')` function which returns `null` if that value is equal to `''`.

  We need to only apply that function to columns that we know to be of type integer or float though - text columns containing empty strings should not be rewritten to null.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/526#issuecomment-1254064260
issue_url: https://api.github.com/repos/simonw/datasette/issues/526
node_id: IC_kwDOBm6k_c5Kv4CE
created_at: 2022-09-21T18:17:04Z
updated_at: 2022-09-21T18:18:01Z
author_association: CONTRIBUTOR
body:
  hi @simonw, this is becoming more of a bother for my [labor data warehouse](https://labordata.bunkum.us/). Is there any research or a spike i could do that would help you investigate this issue?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/358#issuecomment-1255332217
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/358
node_id: IC_kwDOCGYnMM5K0tl5
created_at: 2022-09-22T17:27:34Z
updated_at: 2022-09-22T17:27:34Z
author_association: OWNER
body:
  I've been thinking about this more recently.

  I think the first place to explore these will be in the `create-table` command (and underlying APIs).

  Relevant docs: https://www.sqlite.org/lang_createtable.html#check_constraints

  > A CHECK constraint may be attached to a column definition or specified as a table constraint. In practice it makes no difference. Each time a new row is inserted into the table or an existing row is updated, the expression associated with each CHECK constraint is evaluated and cast to a NUMERIC value in the same way as a [CAST expression](https://www.sqlite.org/lang_expr.html#castexpr). If the result is zero (integer value 0 or real value 0.0), then a constraint violation has occurred. If the CHECK expression evaluates to NULL, or any other non-zero value, it is not a constraint violation. The expression of a CHECK constraint may not contain a subquery.

  Something like this:

      sqlite-utils create-table data.db entries id integer title text tags text --pk id --check tags:json

  Where `--check tags:json` uses a pre-baked recipe for using the SQLite JSON function to check that the content is valid JSON and reject it otherwise.

  Then can bundle a bunch of other pre-baked recipes, but also support the following:

      --check 'x > 3' --check 'length(phone) >= 10'

  The besign reason for the `column:recipe` format here is to reuse `--check` for both pre-defined recipes that affect a single column AND for freeform expressions that get added to the end of the table.

  Detecting `column name:recipe` with a regex feels safe to me.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/358#issuecomment-1255333969
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/358
node_id: IC_kwDOCGYnMM5K0uBR
created_at: 2022-09-22T17:29:09Z
updated_at: 2022-09-22T17:29:09Z
author_association: OWNER
body:
  Quick demo of a check constraint for JSON validation:
  ```
  sqlite> create table test (id integer primary key, tags text, check (json(tags) is not null));
  sqlite> sqlite> insert into test (tags ('["one", "two"]');
  sqlite> insert into test (tags) values ('["one", "two"');
  Error: stepping, malformed JSON (1)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/358#issuecomment-1255340974
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/358
node_id: IC_kwDOCGYnMM5K0vuu
created_at: 2022-09-22T17:34:45Z
updated_at: 2022-09-22T17:34:45Z
author_association: OWNER
body:
  A few other recipes off the top of my head:

  - `title:maxlength:20` - set a max length, `length(title) <= 20`
  - `created:date` - check for `yyyy-mm-dd` date, `select :date == date(:date) is not null` ([demo](https://latest.datasette.io/_memory?sql=select+%3Adate+%3D%3D+date%28%3Adate%29+is+not+null&date=2022-01-01))
  - `age:positiveint` - check `age` is a positive integer, `printf('%', age) = age and age > 0` (untested)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/358#issuecomment-1255341690
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/358
node_id: IC_kwDOCGYnMM5K0v56
created_at: 2022-09-22T17:35:23Z
updated_at: 2022-09-22T17:35:23Z
author_association: OWNER
body:
  Make me think also that `sqlite-utils create-table` should have an option to dump out the SQL without actually creating the table.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1415#issuecomment-1255603780
issue_url: https://api.github.com/repos/simonw/datasette/issues/1415
node_id: IC_kwDOBm6k_c5K1v5E
created_at: 2022-09-22T22:06:10Z
updated_at: 2022-09-22T22:06:10Z
author_association: NONE
body:
  This would be great! I just went through the process of figuring out the minimum permissions for a service account to run `datasette publish cloudrun` for [PUDL](https://github.com/catalyst-cooperative/pudl)'s [datasette deployment](https://data.catalyst.coop/). These are the roles I gave the service account (disclaim: I'm not sure these are the minimum permissions):

  - Cloud Build Service Account: The SA needs this role to publish the build on Cloud Build. 
  - Cloud Run Admin for the Cloud Run datasette service so the SA can deploy the build.
  - I gave the SA the Storage Admin role on the bucket Cloud Build creates to store the build tar files. 
  - The Viewer Role is [required for storing build logs in the default bucket](https://cloud.google.com/build/docs/running-builds/submit-build-via-cli-api#permissions). More on this below!

  The Viewer Role is a Basic IAM role that [Google does not recommend using](https://cloud.google.com/build/docs/running-builds/submit-build-via-cli-api#permissions):

  > Caution: Basic roles include thousands of permissions across all Google Cloud services. In production environments, do not grant basic roles unless there is no alternative. Instead, grant the most limited [predefined roles](https://cloud.google.com/iam/docs/understanding-roles#predefined_roles) or [custom roles](https://cloud.google.com/iam/docs/understanding-custom-roles) that meet your needs.

  If you don't grant the Viewer role the `gcloud builds submit` command will successfully create a build but returns exit code 1, preventing the script from getting to the cloud run step:

  ```
  ERROR: (gcloud.builds.submit)
  The build is running, and logs are being written to the default logs bucket.
  This tool can only stream logs if you are Viewer/Owner of the project and, if applicable, allowed by your VPC-SC security policy.

  The default logs bucket is always outside any VPC-SC security perimeter.
  If you want your logs saved inside your VPC-SC perimeter, use your own bucket.
  See https://cloud.google.com/build/docs/securing-builds/store-manage-build-logs.
  ```
  long stack trace...
  ```
  CalledProcessError: Command 'gcloud builds submit --tag gcr.io/catalyst-cooperative-pudl/datasette' returned non-zero exit status 1.
  ```

  You can store Cloud Build logs in a [user-created bucket](https://cloud.google.com/build/docs/securing-builds/store-manage-build-logs#store-custom-bucket) which only requires the Storage Admin role. However, you have to pass a config file to `gcloud builds submit`, which isn't possible with the current options for `datasette publish cloudrun`. 

  I propose we add an additional CLI option to `datasette publish cloudrun` called `--build-config` that allows users to pass a [config file](https://cloud.google.com/build/docs/running-builds/submit-build-via-cli-api#running_builds) specifying a user create Cloud Build log bucket. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/490#issuecomment-1256428818
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/490
node_id: IC_kwDOCGYnMM5K45US
created_at: 2022-09-23T16:37:58Z
updated_at: 2022-09-23T16:38:35Z
author_association: OWNER
body:
  It should be possible to achieve this with the `--text` option: https://sqlite-utils.datasette.io/en/stable/cli.html?highlight=text#convert-with-text

  Given an example like this in `multiline.log`:
  ```
  2022-03-01T12:04:52: Here is a log message
    that spans multiple lines
  2022-03-01T12:04:52: This is a single line
  2022-03-01T12:04:52: Here is another message
    that spans multiple lines
  ```
  You should be able to run something like this:
  ```
  sqlite-utils insert /tmp/log.db log multiline.log --text --convert "
  import re

  r = re.compile(r'^(?P<datetime>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}):(?P<log>.*)', re.MULTILINE)

  def convert(text):
      return [m.groupdict() for m in r.finditer(text)]
  "
  ```
  After running this I get:
  ```
  sqlite-utils rows /tmp/log.db log
  [{"datetime": "2022-03-01T12:04:52", "log": " Here is a log message"},
   {"datetime": "2022-03-01T12:04:52", "log": " This is a single line"},
   {"datetime": "2022-03-01T12:04:52", "log": " Here is another message"}]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1817#issuecomment-1256650449
issue_url: https://api.github.com/repos/simonw/datasette/issues/1817
node_id: IC_kwDOBm6k_c5K5vbR
created_at: 2022-09-23T20:38:53Z
updated_at: 2022-09-23T20:38:53Z
author_association: OWNER
body:
  I've wanted something like this in the past too. I think the thing to do here might be to add `sql` and `params` arguments to a bunch of the plugin hooks, such that they can see the main query that is being used on the page that they are helping to render.

  While I'm working on this: https://docs.datasette.io/en/0.62/plugin_hooks.html#register-output-renderer-datasette output renderer functions take `sql` but do not currently take `params` - they should also take `params`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1817#issuecomment-1256652548
issue_url: https://api.github.com/repos/simonw/datasette/issues/1817
node_id: IC_kwDOBm6k_c5K5v8E
created_at: 2022-09-23T20:41:32Z
updated_at: 2022-09-23T20:41:32Z
author_association: OWNER
body:
  Which plugin hooks should take `sql` and `params`?

  - [extra_template_vars(template, database, table, columns, view_name, request, datasette)](https://docs.datasette.io/en/0.62/plugin_hooks.html#extra-template-vars-template-database-table-columns-view-name-request-datasette)
  - [extra_css_urls(template, database, table, columns, view_name, request, datasette)](https://docs.datasette.io/en/0.62/plugin_hooks.html#extra-css-urls-template-database-table-columns-view-name-request-datasette)
  - [extra_js_urls(template, database, table, columns, view_name, request, datasette)](https://docs.datasette.io/en/0.62/plugin_hooks.html#extra-js-urls-template-database-table-columns-view-name-request-datasette)
  - [extra_body_script(template, database, table, columns, view_name, request, datasette)](https://docs.datasette.io/en/0.62/plugin_hooks.html#extra-body-script-template-database-table-columns-view-name-request-datasette)

  And maybe these:

  - [render_cell(row, value, column, table, database, datasette)](https://docs.datasette.io/en/0.62/plugin_hooks.html#render-cell-row-value-column-table-database-datasette)
  - [table_actions(datasette, actor, database, table, request)](https://docs.datasette.io/en/0.62/plugin_hooks.html#table-actions-datasette-actor-database-table-request)

  I'll start by implementing the first set, then I'll think further about those "maybes".

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1817#issuecomment-1256659788
issue_url: https://api.github.com/repos/simonw/datasette/issues/1817
node_id: IC_kwDOBm6k_c5K5xtM
created_at: 2022-09-23T20:49:22Z
updated_at: 2022-09-23T20:49:22Z
author_association: OWNER
body:
  Implementation challenge: all four of those hooks are called inside the `datasette.render_template()` method, which has this signature:

  https://github.com/simonw/datasette/blob/cb1e093fd361b758120aefc1a444df02462389a3/datasette/app.py#L945-L947

  So I would have to pull the `sql` and `params` variables out of the `context` since they are not being passed to that method. OR I could teach that method to take those as optional arguments.

  Might be an opportunity to clean up this hack:

  https://github.com/simonw/datasette/blob/cb1e093fd361b758120aefc1a444df02462389a3/datasette/app.py#L959-L964

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1817#issuecomment-1256662785
issue_url: https://api.github.com/repos/simonw/datasette/issues/1817
node_id: IC_kwDOBm6k_c5K5ycB
created_at: 2022-09-23T20:53:21Z
updated_at: 2022-09-23T20:53:21Z
author_association: OWNER
body:
  Maybe the signature for that method should be:

  ```python
  async def render_template( 
       self, templates, context=None, plugin_context=None, request=None, view_name=None 
   ): 
  ```
  Where `plugin_context` is a special dictionary of values that can be passed through to plugin hooks that accept them - so `database`, `table`, `columns`, `sql` and `params`.

  Those would then be passed when specific views call `render_template()` - which they currently do via calling `BaseView.render(...)`, but actually the views that are used for tables and queries don't even call that directly due to the weird designed used with `DataView` subclasses that implement a `.data()` method.

  So yet another change that's blocked on fixing that long-running weird piece of technical debt:
  - https://github.com/simonw/datasette/issues/1518

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1817#issuecomment-1256781274
issue_url: https://api.github.com/repos/simonw/datasette/issues/1817
node_id: IC_kwDOBm6k_c5K6PXa
created_at: 2022-09-23T22:59:46Z
updated_at: 2022-09-23T22:59:46Z
author_association: CONTRIBUTOR
body:
  While you are adding features, would you be future-proofing your APIs if you switched over some arguments over to keyword-only arguments or would that be too disruptive?

  Thinking out loud:

  ```
  async def render_template( 
       self, templates, *, context=None, plugin_context=None, request=None, view_name=None 
   ): 
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1256858763
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/491
node_id: IC_kwDOCGYnMM5K6iSL
created_at: 2022-09-24T04:50:59Z
updated_at: 2022-09-24T04:52:08Z
author_association: CONTRIBUTOR
body:
  Instead of outputting binary data to stdout the interface might be better like this

  ```
  sqlite-utils merge animals.db cats.db dogs.db
  ```

  similar to `zip`, `ogr2ogr`, etc

  Actually I think this might already be possible within `ogr2ogr`. I don't believe spatial data is a requirement though it might add an `ogc_id` column or something

  ```
  cp cats.db animals.db
  ogr2ogr -append animals.db dogs.db
  ogr2ogr -append animals.db another.db
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/490#issuecomment-1257072258
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/490
node_id: IC_kwDOCGYnMM5K7WaC
created_at: 2022-09-24T22:01:05Z
updated_at: 2022-09-24T22:01:05Z
author_association: NONE
body:
  For completeness, the regex requires a bit more dark magic to capture the following lines, here is a _working_ expression: https://regex101.com/r/rsuEcs/1

  ```
  sqlite-utils insert /tmp/log.db log multiline.log --text --convert "
  import re

  r = re.compile(r'^(?P<datetime>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2})(?:\:\s)(?P<log>(.*\s\s.*|.*)+)', re.MULTILINE)

  def convert(text):
      return [m.groupdict() for m in r.finditer(text)]
  "
  ```

  ```
  BEGIN TRANSACTION;
  CREATE TABLE [log] (
     [datetime] TEXT,
     [log] TEXT
  );
  INSERT INTO "log" VALUES('2022-03-01T12:04:52','Here is a log message
    that spans multiple lines');
  INSERT INTO "log" VALUES('2022-03-01T12:04:52','This is a single line');
  INSERT INTO "log" VALUES('2022-03-01T12:04:52','Here is another message
    that spans multiple lines');
  COMMIT;
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/358#issuecomment-1257136801
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/358
node_id: IC_kwDOCGYnMM5K7mKh
created_at: 2022-09-25T07:15:07Z
updated_at: 2022-09-25T07:15:59Z
author_association: NONE
body:
  HI Simon, 

  looks good, I noticed you wanted to use a regex to detect it, you might be interested in  [github.com/iafisher/sqliteparser](https://github.com/iafisher/sqliteparser) which creates an ast of the create table statement, not every option supported yet but i forked it and am adding all the possible options in a create table (and create index) statement.  

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1818#issuecomment-1257290709
issue_url: https://api.github.com/repos/simonw/datasette/issues/1818
node_id: IC_kwDOBm6k_c5K8LvV
created_at: 2022-09-25T22:17:06Z
updated_at: 2022-09-25T22:17:06Z
author_association: NONE
body:
  I wonder if having an option for displaying the max row id might help too. Not accurate especially if something was deleted, but useful for DBs as a dump. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1727#issuecomment-1258129113
issue_url: https://api.github.com/repos/simonw/datasette/issues/1727
node_id: IC_kwDOBm6k_c5K_YbZ
created_at: 2022-09-26T14:30:11Z
updated_at: 2022-09-26T14:48:31Z
author_association: CONTRIBUTOR
body:
  from your analysis, it seems like the GIL is blocking on loading of the data from sqlite to python, (particularly in the `fetchmany` call)

  this is probably a simplistic idea, but what if you had the python code in the `execute` method iterate over the cursor and yield out rows or small chunks of rows.

  something like:             
  ```python
              with sqlite_timelimit(conn, time_limit_ms):
                  try:
                      cursor = conn.cursor()
                      cursor.execute(sql, params if params is not None else {})
                  except:
                      ...
              max_returned_rows = self.ds.max_returned_rows
              if max_returned_rows == page_size:
                  max_returned_rows += 1
                  if max_returned_rows and truncate:
                      for i, row in enumerate(cursor):
                          yield row
                          if i == max_returned_rows - 1:
                              break
                  else:
                      for row in cursor:
                          yield row
                      truncated = False                  
  ```

  this kind of thing works well with a postgres server side cursor, but i'm not sure if it will hold for sqlite. 

  you would still spend about the same amount of time in python and would be contending for the gil, but it would be could be non blocking.

  depending on the data flow, this could also some benefit for memory. (data stays in more compact sqlite-land until you need it)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1655#issuecomment-1258166572
issue_url: https://api.github.com/repos/simonw/datasette/issues/1655
node_id: IC_kwDOBm6k_c5K_hks
created_at: 2022-09-26T14:57:04Z
updated_at: 2022-09-26T14:57:04Z
author_association: CONTRIBUTOR
body:
  I think that paginating, even in javascript, could be very helpful. Maybe render json or csv into the page and let javascript loading that into the dom?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/526#issuecomment-1258167564
issue_url: https://api.github.com/repos/simonw/datasette/issues/526
node_id: IC_kwDOBm6k_c5K_h0M
created_at: 2022-09-26T14:57:44Z
updated_at: 2022-09-26T15:08:36Z
author_association: CONTRIBUTOR
body:
  reading the database execute method i have a few questions.

  https://github.com/simonw/datasette/blob/cb1e093fd361b758120aefc1a444df02462389a3/datasette/database.py#L229-L242

  ---
  unless i'm missing something (which is very likely!!), the `max_returned_rows` argument doesn't actually offer any protections against running very expensive queries. 

  It's not like adding a `LIMIT max_rows` argument. it make sense that it isn't because, the query could already have an `LIMIT` argument. Doing something like `select * from (query) limit {max_returned_rows}` **might** be protective but wouldn't always.

  Instead the code executes the full original query, and if still has time it fetches out the first `max_rows + 1` rows. 

  this *does* offer some protection of memory exhaustion, as you won't hydrate a huge result set into python (however, there are [data flow patterns](https://github.com/simonw/datasette/issues/1727#issuecomment-1258129113) that could avoid that too)

  given the current architecture, i don't see how creating a new connection would be use?

  ---

  If we just removed the `max_return_rows` limitation, then i think most things would be fine **except** for the QueryViews. Right now rendering, just [5000 rows takes a lot of client-side memory](https://github.com/simonw/datasette/issues/1655) so some form of pagination would be required.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/526#issuecomment-1258337011
issue_url: https://api.github.com/repos/simonw/datasette/issues/526
node_id: IC_kwDOBm6k_c5LALLz
created_at: 2022-09-26T16:49:48Z
updated_at: 2022-09-26T16:49:48Z
author_association: CONTRIBUTOR
body:
  i think the smallest change that gets close to what i want is to change the behavior so that `max_returned_rows` is not applied in the `execute` method when we are are asking for a csv of query.

  there are some infelicities for that approach, but i'll make a PR to make it easier to discuss.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/492#issuecomment-1258446128
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/492
node_id: IC_kwDOCGYnMM5LAl0w
created_at: 2022-09-26T18:32:14Z
updated_at: 2022-09-26T18:33:19Z
author_association: OWNER
body:
  This idea would make more sense if there was a good mechanism to say "run the conversion script held in this file" as opposed to passing it as an option. This would also make having to remember bash escaping rules ([see tip](https://til.simonwillison.net/zsh/argument-heredoc)) much easier!

  `shot-scraper` has that for `--javascript`, using the `--input` option: https://shot-scraper.datasette.io/en/stable/javascript.html#shot-scraper-javascript-help

  Maybe `--convert-script` would work here? Or `--convert-file`? It should accept `-` for stdin too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1258449887
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/491
node_id: IC_kwDOCGYnMM5LAmvf
created_at: 2022-09-26T18:35:50Z
updated_at: 2022-09-26T18:35:50Z
author_association: OWNER
body:
  This is a really interesting idea.

  I'm nervous about needing to set the rules for how duplicate tables should be merged though. This feels like a complex topic - one where there isn't necessarily an obviously "correct" way of doing it, but where different problems that people are solving might need different merging approaches.

  Likewise, merging isn't just a database-to-database thing at that point - I could see a need for merging two tables using similar rules to those used for merging two databases.

  So I think I'd want to have some good concrete use-cases in mind before trying to design how something like this should work. Will leave this thread open for people to drop those in!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1258450447
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/491
node_id: IC_kwDOCGYnMM5LAm4P
created_at: 2022-09-26T18:36:23Z
updated_at: 2022-09-26T18:36:23Z
author_association: OWNER
body:
  This is also the kind of feature that would need to express itself in both the Python library and the CLI utility.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/483#issuecomment-1258451968
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/483
node_id: IC_kwDOCGYnMM5LAnQA
created_at: 2022-09-26T18:37:54Z
updated_at: 2022-09-26T18:40:41Z
author_association: OWNER
body:
  The implementation of this can be an almost exact copy of Datasette's, which was added in this commit: https://github.com/simonw/datasette/commit/01fe5b740171bfaea3752fc5754431dac53777e3

  Current code for that is here: https://github.com/simonw/datasette/blob/0.62/datasette/cli.py#L319-L340 - which is improved to use the `from runpy import run_module` function.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/493#issuecomment-1258476455
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/493
node_id: IC_kwDOCGYnMM5LAtOn
created_at: 2022-09-26T19:01:49Z
updated_at: 2022-09-26T19:01:49Z
author_association: OWNER
body:
  I tried the tips in https://stackoverflow.com/questions/15258831/how-to-handle-two-dashes-in-rest (not the settings change though, because I might want smart quotes elsewhere) and they didn't work.

  Maybe I should disable smart quotes entirely?

  I feel like there should be an escaping trick that works here though. I tried `insert -\\-convert` but it didn't help.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/483#issuecomment-1258479462
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/483
node_id: IC_kwDOCGYnMM5LAt9m
created_at: 2022-09-26T19:04:29Z
updated_at: 2022-09-26T19:04:43Z
author_association: OWNER
body:
  Documentation:

  - https://sqlite-utils.datasette.io/en/latest/cli.html#cli-install
  - https://sqlite-utils.datasette.io/en/latest/cli.html#cli-uninstall
  - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#install
  - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#uninstall

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1258508215
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/491
node_id: IC_kwDOCGYnMM5LA0-3
created_at: 2022-09-26T19:22:14Z
updated_at: 2022-09-26T19:22:14Z
author_association: CONTRIBUTOR
body:
  This might be fairly straightforward using SQLite's backup utility: https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.backup

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/494#issuecomment-1258516872
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/494
node_id: IC_kwDOCGYnMM5LA3GI
created_at: 2022-09-26T19:28:36Z
updated_at: 2022-09-26T19:28:36Z
author_association: OWNER
body:
  New documentation: https://sqlite-utils.datasette.io/en/latest/contributing.html#using-just-and-pipenv

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1820#issuecomment-1258601033
issue_url: https://api.github.com/repos/simonw/datasette/issues/1820
node_id: IC_kwDOBm6k_c5LBLpJ
created_at: 2022-09-26T20:32:47Z
updated_at: 2022-10-07T03:58:13Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1820?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Base: **92.50**% // Head: **92.51**% // Increases project coverage by **`+0.01%`** :tada:
  > Coverage data is based on head [(`9bead2a`)](https://codecov.io/gh/simonw/datasette/pull/1820?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`eff1124`)](https://codecov.io/gh/simonw/datasette/commit/eff112498ecc499323c26612d707908831446d25?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > Patch coverage: 100.00% of modified lines in pull request are covered.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@            Coverage Diff             @@
  ##             main    #1820      +/-   ##
  ==========================================
  + Coverage   92.50%   92.51%   +0.01%     
  ==========================================
    Files          35       35              
    Lines        4400     4406       +6     
  ==========================================
  + Hits         4070     4076       +6     
    Misses        330      330              
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.11% <ø> (ø)` | |
  | [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `94.80% <100.00%> (+0.05%)` | :arrow_up: |
  | [datasette/views/database.py](https://codecov.io/gh/simonw/datasette/pull/1820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `95.29% <100.00%> (+0.06%)` | :arrow_up: |

  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  </details>

  [:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1820?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1821#issuecomment-1258692555
issue_url: https://api.github.com/repos/simonw/datasette/issues/1821
node_id: IC_kwDOBm6k_c5LBh_L
created_at: 2022-09-26T22:06:39Z
updated_at: 2022-09-26T22:06:39Z
author_association: OWNER
body:
  - https://github.com/simonw/datasette/actions/runs/3131344150
  - https://github.com/simonw/datasette/releases/tag/0.63a0
  - https://pypi.org/project/datasette/0.63a0/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1258697384
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/491
node_id: IC_kwDOCGYnMM5LBjKo
created_at: 2022-09-26T22:12:45Z
updated_at: 2022-09-26T22:12:45Z
author_association: OWNER
body:
  That feels like a slightly different command to me - maybe `sqlite-utils backup data.db data-backup.db`? It doesn't have any of the mechanics for merging tables together. Could be a useful feature separately though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1258712931
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/491
node_id: IC_kwDOCGYnMM5LBm9j
created_at: 2022-09-26T22:31:58Z
updated_at: 2022-09-26T22:31:58Z
author_association: CONTRIBUTOR
body:
  Right. The backup command will copy tables completely, but in the case of conflicting table names, the destination gets overwritten silently. That might not be what you want here. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1818#issuecomment-1258735283
issue_url: https://api.github.com/repos/simonw/datasette/issues/1818
node_id: IC_kwDOBm6k_c5LBsaz
created_at: 2022-09-26T22:47:19Z
updated_at: 2022-09-26T22:47:19Z
author_association: OWNER
body:
  That's a really interesting idea: for a lot of databases (those made out of straight imports from CSV) `max(rowid)` would indeed reflect the size of the table, but would be a MUCH faster operation than attempting a `count(*)`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1818#issuecomment-1258735747
issue_url: https://api.github.com/repos/simonw/datasette/issues/1818
node_id: IC_kwDOBm6k_c5LBsiD
created_at: 2022-09-26T22:47:59Z
updated_at: 2022-09-26T22:47:59Z
author_association: OWNER
body:
  Another option here is to tie into a feature I built in `sqlite-utils` with this problem in mind but never introduced on the Datasette side of things: https://sqlite-utils.datasette.io/en/stable/python-api.html#cached-table-counts-using-triggers

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1818#issuecomment-1258738740
issue_url: https://api.github.com/repos/simonw/datasette/issues/1818
node_id: IC_kwDOBm6k_c5LBtQ0
created_at: 2022-09-26T22:52:45Z
updated_at: 2022-09-26T22:55:57Z
author_association: NONE
body:
  thoughts on order of precedence to use:

  * sqlite-utils count, if present. closest thing to a standard i guess.
  * row(max_id) if like, the first and/or last x amount of rows ids are all contiguous. kind of a cheap/dumb/imperfect heuristic to see if the table is dump/not dump. if the check passes, still stick on `est.` after the display.
  * count(*) if enabled in datasette 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1819#issuecomment-1258746600
issue_url: https://api.github.com/repos/simonw/datasette/issues/1819
node_id: IC_kwDOBm6k_c5LBvLo
created_at: 2022-09-26T23:05:40Z
updated_at: 2022-09-26T23:05:40Z
author_association: OWNER
body:
  Implementing it like this, so at least you can copy and paste the SQL query back out again:

  <img width="796" alt="image" src="https://user-images.githubusercontent.com/9599/192395953-48512c94-10e0-4cf8-8ae5-b9e65e3d7b0f.png">

  I'm not doing a full textarea because this error can be raised in multiple places, including on the table page itself. It's not just an error associated with the manual query page.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1819#issuecomment-1258754105
issue_url: https://api.github.com/repos/simonw/datasette/issues/1819
node_id: IC_kwDOBm6k_c5LBxA5
created_at: 2022-09-26T23:16:15Z
updated_at: 2022-09-26T23:16:15Z
author_association: OWNER
body:
  Demo: https://latest.datasette.io/_memory?sql=with+recursive+counter(x)+as+(%0D%0A++select+0%0D%0A++++union%0D%0A++select+x+%2B+1+from+counter%0D%0A)%2C%0D%0Ablah+as+(select+*+from+counter+limit+5000000)%0D%0Aselect+count(*)+from+blah

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1817#issuecomment-1258756231
issue_url: https://api.github.com/repos/simonw/datasette/issues/1817
node_id: IC_kwDOBm6k_c5LBxiH
created_at: 2022-09-26T23:19:34Z
updated_at: 2022-09-26T23:19:34Z
author_association: OWNER
body:
  This is a good idea - it's something I should do before Datasette 1.0.

  I was a tiny bit worried about compatibility (Datasette is 3.7+) but it looks like they have been in Python since 3.0!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1822#issuecomment-1258757544
issue_url: https://api.github.com/repos/simonw/datasette/issues/1822
node_id: IC_kwDOBm6k_c5LBx2o
created_at: 2022-09-26T23:21:23Z
updated_at: 2022-09-26T23:21:23Z
author_association: OWNER
body:
  Everything on https://docs.datasette.io/en/stable/internals.html that uses keyword arguments should do this I think.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1822#issuecomment-1258760299
issue_url: https://api.github.com/repos/simonw/datasette/issues/1822
node_id: IC_kwDOBm6k_c5LByhr
created_at: 2022-09-26T23:25:12Z
updated_at: 2022-09-26T23:25:55Z
author_association: OWNER
body:
  A start:
  ```diff
  diff --git a/datasette/utils/asgi.py b/datasette/utils/asgi.py
  index 8a2fa060..41ade961 100644
  --- a/datasette/utils/asgi.py
  +++ b/datasette/utils/asgi.py
  @@ -118,7 +118,7 @@ class Request:
           return dict(parse_qsl(body.decode("utf-8"), keep_blank_values=True))
   
       @classmethod
  -    def fake(cls, path_with_query_string, method="GET", scheme="http", url_vars=None):
  +    def fake(cls, path_with_query_string, *, method="GET", scheme="http", url_vars=None):
           """Useful for constructing Request objects for tests"""
           path, _, query_string = path_with_query_string.partition("?")
           scope = {
  @@ -204,7 +204,7 @@ class AsgiWriter:
           )
   
   
  -async def asgi_send_json(send, info, status=200, headers=None):
  +async def asgi_send_json(send, info, *, status=200, headers=None):
       headers = headers or {}
       await asgi_send(
           send,
  @@ -215,7 +215,7 @@ async def asgi_send_json(send, info, status=200, headers=None):
       )
   
   
  -async def asgi_send_html(send, html, status=200, headers=None):
  +async def asgi_send_html(send, html, *, status=200, headers=None):
       headers = headers or {}
       await asgi_send(
           send,
  @@ -226,7 +226,7 @@ async def asgi_send_html(send, html, status=200, headers=None):
       )
   
   
  -async def asgi_send_redirect(send, location, status=302):
  +async def asgi_send_redirect(send, location, *, status=302):
       await asgi_send(
           send,
           "",
  @@ -236,12 +236,12 @@ async def asgi_send_redirect(send, location, status=302):
       )
   
   
  -async def asgi_send(send, content, status, headers=None, content_type="text/plain"):
  +async def asgi_send(send, content, status, *, headers=None, content_type="text/plain"):
       await asgi_start(send, status, headers, content_type)
       await send({"type": "http.response.body", "body": content.encode("utf-8")})
   
   
  -async def asgi_start(send, status, headers=None, content_type="text/plain"):
  +async def asgi_start(send, status, *, headers=None, content_type="text/plain"):
       headers = headers or {}
       # Remove any existing content-type header
       headers = {k: v for k, v in headers.items() if k.lower() != "content-type"}
  @@ -259,7 +259,7 @@ async def asgi_start(send, status, headers=None, content_type="text/plain"):
   
   
   async def asgi_send_file(
  -    send, filepath, filename=None, content_type=None, chunk_size=4096, headers=None
  +    send, filepath, filename=None, *, content_type=None, chunk_size=4096, headers=None
   ):
       headers = headers or {}
       if filename:
  @@ -284,7 +284,7 @@ async def asgi_send_file(
               )
   
   
  -def asgi_static(root_path, chunk_size=4096, headers=None, content_type=None):
  +def asgi_static(root_path, *, chunk_size=4096, headers=None, content_type=None):
       root_path = Path(root_path)
   
       async def inner_static(request, send):
  @@ -313,7 +313,7 @@ def asgi_static(root_path, chunk_size=4096, headers=None, content_type=None):
   
   
   class Response:
  -    def __init__(self, body=None, status=200, headers=None, content_type="text/plain"):
  +    def __init__(self, body=None, *, status=200, headers=None, content_type="text/plain"):
           self.body = body
           self.status = status
           self.headers = headers or {}
  @@ -346,6 +346,7 @@ class Response:
           self,
           key,
           value="",
  +        *,
           max_age=None,
           expires=None,
           path="/",
  @@ -374,7 +375,7 @@ class Response:
           self._set_cookie_headers.append(cookie.output(header="").strip())
   
       @classmethod
  -    def html(cls, body, status=200, headers=None):
  +    def html(cls, body, *, status=200, headers=None):
           return cls(
               body,
               status=status,
  @@ -383,7 +384,7 @@ class Response:
           )
   
       @classmethod
  -    def text(cls, body, status=200, headers=None):
  +    def text(cls, body, *, status=200, headers=None):
           return cls(
               str(body),
               status=status,
  @@ -392,7 +393,7 @@ class Response:
           )
   
       @classmethod
  -    def json(cls, body, status=200, headers=None, default=None):
  +    def json(cls, body, *, status=200, headers=None, default=None):
           return cls(
               json.dumps(body, default=default),
               status=status,
  @@ -401,7 +402,7 @@ class Response:
           )
   
       @classmethod
  -    def redirect(cls, path, status=302, headers=None):
  +    def redirect(cls, path, *, status=302, headers=None):
           headers = headers or {}
           headers["Location"] = path
           return cls("", status=status, headers=headers)
  @@ -412,6 +413,7 @@ class AsgiFileDownload:
           self,
           filepath,
           filename=None,
  +        *,
           content_type="application/octet-stream",
           headers=None,
       ):
  ```
  ```diff
  diff --git a/datasette/app.py b/datasette/app.py
  index 03d1dacc..4d4e5584 100644
  --- a/datasette/app.py
  +++ b/datasette/app.py
  @@ -190,6 +190,7 @@ class Datasette:
       def __init__(
           self,
           files=None,
  +        *,
           immutables=None,
           cache_headers=True,
           cors=False,
   ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1820#issuecomment-1258803261
issue_url: https://api.github.com/repos/simonw/datasette/issues/1820
node_id: IC_kwDOBm6k_c5LB9A9
created_at: 2022-09-27T00:03:09Z
updated_at: 2022-09-27T00:03:09Z
author_association: CONTRIBUTOR
body:
  the pattern in this PR `max_returned_rows` control the maximum rows rendered through html and json, and the csv render bypasses that.

  i think it would be better to have each of these different query renderers have more direct control for how many rows to fetch, instead of relying on the internals of the `execute` method.

  generally, users will not want to paginate through tens of thousands of results, but often will want to download a full query as json or as csv. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1817#issuecomment-1258818028
issue_url: https://api.github.com/repos/simonw/datasette/issues/1817
node_id: IC_kwDOBm6k_c5LCAns
created_at: 2022-09-27T00:27:53Z
updated_at: 2022-09-27T00:27:53Z
author_association: OWNER
body:
  Made a start on this:
  ```diff
  diff --git a/datasette/hookspecs.py b/datasette/hookspecs.py
  index 34e19664..fe0971e5 100644
  --- a/datasette/hookspecs.py
  +++ b/datasette/hookspecs.py
  @@ -31,25 +31,29 @@ def prepare_jinja2_environment(env, datasette):
   
   
   @hookspec
  -def extra_css_urls(template, database, table, columns, view_name, request, datasette):
  +def extra_css_urls(
  +    template, database, table, columns, sql, params, view_name, request, datasette
  +):
       """Extra CSS URLs added by this plugin"""
   
   
   @hookspec
  -def extra_js_urls(template, database, table, columns, view_name, request, datasette):
  +def extra_js_urls(
  +    template, database, table, columns, sql, params, view_name, request, datasette
  +):
       """Extra JavaScript URLs added by this plugin"""
   
   
   @hookspec
   def extra_body_script(
  -    template, database, table, columns, view_name, request, datasette
  +    template, database, table, columns, sql, params, view_name, request, datasette
   ):
       """Extra JavaScript code to be included in <script> at bottom of body"""
   
   
   @hookspec
   def extra_template_vars(
  -    template, database, table, columns, view_name, request, datasette
  +    template, database, table, columns, sql, params, view_name, request, datasette
   ):
       """Extra template variables to be made available to the template - can return dict or callable or awaitable"""
  ```
  ```diff
  diff --git a/datasette/app.py b/datasette/app.py
  index 03d1dacc..2f3a46fe 100644
  --- a/datasette/app.py
  +++ b/datasette/app.py
  @@ -1036,7 +1036,9 @@ class Datasette:
   
           return await template.render_async(template_context)
   
  -    async def _asset_urls(self, key, template, context, request, view_name):
  +    async def _asset_urls(
  +        self, key, template, context, request, view_name, sql, params
  +    ):
           # Flatten list-of-lists from plugins:
           seen_urls = set()
           collected = []
  @@ -1045,6 +1047,8 @@ class Datasette:
               database=context.get("database"),
               table=context.get("table"),
               columns=context.get("columns"),
  +            sql=sql,
  +            params=params,
               view_name=view_name,
               request=request,
               datasette=self,
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1823#issuecomment-1258828705
issue_url: https://api.github.com/repos/simonw/datasette/issues/1823
node_id: IC_kwDOBm6k_c5LCDOh
created_at: 2022-09-27T00:45:46Z
updated_at: 2022-09-27T00:45:46Z
author_association: OWNER
body:
  Also need to do a bit more of an audit to see if there is anywhere else that this style should be applied.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1823#issuecomment-1258833358
issue_url: https://api.github.com/repos/simonw/datasette/issues/1823
node_id: IC_kwDOBm6k_c5LCEXO
created_at: 2022-09-27T00:54:15Z
updated_at: 2022-10-05T04:37:54Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1823?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Base: **91.58**% // Head: **92.50**% // Increases project coverage by **`+0.91%`** :tada:
  > Coverage data is based on head [(`b545b6a`)](https://codecov.io/gh/simonw/datasette/pull/1823?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`5f9f567`)](https://codecov.io/gh/simonw/datasette/commit/5f9f567acbc58c9fcd88af440e68034510fb5d2b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > Patch coverage: 90.47% of modified lines in pull request are covered.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@            Coverage Diff             @@
  ##             main    #1823      +/-   ##
  ==========================================
  + Coverage   91.58%   92.50%   +0.91%     
  ==========================================
    Files          36       35       -1     
    Lines        4444     4400      -44     
  ==========================================
    Hits         4070     4070              
  + Misses        374      330      -44     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1823?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/utils/asgi.py](https://codecov.io/gh/simonw/datasette/pull/1823/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL2FzZ2kucHk=) | `91.06% <88.23%> (ø)` | |
  | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1823/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.11% <100.00%> (ø)` | |
  | [datasette/utils/shutil\_backport.py](https://codecov.io/gh/simonw/datasette/pull/1823/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL3NodXRpbF9iYWNrcG9ydC5weQ==) | | |

  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  </details>

  [:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1823?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/526#issuecomment-1258846992
issue_url: https://api.github.com/repos/simonw/datasette/issues/526
node_id: IC_kwDOBm6k_c5LCHsQ
created_at: 2022-09-27T01:21:41Z
updated_at: 2022-09-27T01:21:41Z
author_association: OWNER
body:
  My main concern here is that public Datasette instances could easily have all of their available database connections consumed by long-running queries - either accidentally or deliberately.

  I do totally understand the need for this feature though. I think it can absolutely make sense provided it's protected by authentication and permissions.

  Maybe even limit the number of concurrent downloads at once such that there's always at least one database connection free for other requests.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/526#issuecomment-1258849766
issue_url: https://api.github.com/repos/simonw/datasette/issues/526
node_id: IC_kwDOBm6k_c5LCIXm
created_at: 2022-09-27T01:27:03Z
updated_at: 2022-09-27T01:27:03Z
author_association: CONTRIBUTOR
body:
  i agree with that concern! but if i'm understanding the code correctly, `maximum_returned_rows` does not protect against long-running queries in any way.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/526#issuecomment-1258860845
issue_url: https://api.github.com/repos/simonw/datasette/issues/526
node_id: IC_kwDOBm6k_c5LCLEt
created_at: 2022-09-27T01:48:31Z
updated_at: 2022-09-27T01:50:01Z
author_association: OWNER
body:
  The protection is supposed to be from this line:
  ```python
  rows = cursor.fetchmany(max_returned_rows + 1) 
  ```
  By capping the call to `.fetchman()` at `max_returned_rows + 1` (the `+ 1` is to allow detection of whether or not there is a next page) I'm ensuring that Datasette never attempts to iterate over a huge result set.

  SQLite and the `sqlite3` library seem to handle this correctly. Here's an example:

  ```pycon
  >>> import sqlite3
  >>> conn = sqlite3.connect(":memory:")
  >>> cursor = conn.execute("""
  ... with recursive counter(x) as (
  ...   select 0
  ...     union
  ...   select x + 1 from counter
  ... )
  ... select * from counter""")
  >>> cursor.fetchmany(10)
  [(0,), (1,), (2,), (3,), (4,), (5,), (6,), (7,), (8,), (9,), (10,)]
  ```
  `counter` there is an infinitely long table ([see TIL](https://til.simonwillison.net/sqlite/simple-recursive-cte)) - but we can retrieve the first 10 results without going into an infinite loop.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/526#issuecomment-1258864140
issue_url: https://api.github.com/repos/simonw/datasette/issues/526
node_id: IC_kwDOBm6k_c5LCL4M
created_at: 2022-09-27T01:55:32Z
updated_at: 2022-09-27T01:55:32Z
author_association: OWNER
body:
  That recursive query is a great example of the kind of thing having a maximum row limit protects against.

  Imagine if Datasette CSVs did allow unlimited retrievals. Someone could hit the CSV endpoint for that recursive query and tie up Datasette's SQL connection effectively forever.

  Even if this feature becomes a permission-guarded thing we still need to take that case into account.

  At the very least it would be good if the query could be cancelled if the client disconnects - so if someone accidentally starts an infinite query they can cancel the request and free up the server resources.

  It might be a good idea to implement a page that shows "currently running" queries and allows users with the right permission to terminate them from that page.

  Another option: a "limit of last resource" - either a very high row limit (10,000,000 perhaps) or even a time limit, saying that all queries will be cancelled if they take longer than thirty minutes or similar.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/526#issuecomment-1258871525
issue_url: https://api.github.com/repos/simonw/datasette/issues/526
node_id: IC_kwDOBm6k_c5LCNrl
created_at: 2022-09-27T02:09:32Z
updated_at: 2022-09-27T02:14:53Z
author_association: CONTRIBUTOR
body:
  thanks @simonw, i learned something i didn't know about sqlite's execution model!

  > Imagine if Datasette CSVs did allow unlimited retrievals. Someone could hit the CSV endpoint for that recursive query and tie up Datasette's SQL connection effectively forever.

  why wouldn't the `sqlite_timelimit` guard prevent that?

  --- 
  on my local version which has the code to [turn off truncations for query csv](#1820), `sqlite_timelimit` does protect me.

  ![Screenshot 2022-09-26 at 22-14-31 Error 500](https://user-images.githubusercontent.com/536941/192415680-94b32b7f-868f-4b89-8194-5752d45f6009.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/526#issuecomment-1258878311
issue_url: https://api.github.com/repos/simonw/datasette/issues/526
node_id: IC_kwDOBm6k_c5LCPVn
created_at: 2022-09-27T02:19:48Z
updated_at: 2022-09-27T02:19:48Z
author_association: CONTRIBUTOR
body:
  this sql query doesn't trip up `maximum_returned_rows` but does timeout

  ```sql
  with recursive counter(x) as (
     select 0
       union
     select x + 1 from counter
   )
   select * from counter LIMIT 10 OFFSET 100000000 
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/526#issuecomment-1258905781
issue_url: https://api.github.com/repos/simonw/datasette/issues/526
node_id: IC_kwDOBm6k_c5LCWC1
created_at: 2022-09-27T03:03:35Z
updated_at: 2022-09-27T03:03:47Z
author_association: OWNER
body:
  Yes good point, the time limit does already protect against that. I've been contemplating a permissioned-users-only relaxation of that time limit too, and I got that idea mixed up with this one in my head.

  On that basis maybe this feature would be safe after all? Would need to do some testing, but it may be that the existing time limit provides enough protection here already.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/526#issuecomment-1258906440
issue_url: https://api.github.com/repos/simonw/datasette/issues/526
node_id: IC_kwDOBm6k_c5LCWNI
created_at: 2022-09-27T03:04:37Z
updated_at: 2022-09-27T03:04:37Z
author_association: OWNER
body:
  It would be really neat if we could explore this idea in a plugin, but I don't think Datasette has plugin hooks in the right place for that at the moment.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/526#issuecomment-1258910228
issue_url: https://api.github.com/repos/simonw/datasette/issues/526
node_id: IC_kwDOBm6k_c5LCXIU
created_at: 2022-09-27T03:11:07Z
updated_at: 2022-09-27T03:11:07Z
author_association: CONTRIBUTOR
body:
  i think this feature would be safe, as its really only the time limit that can, and imo, should protect against long running queries, as it is pretty easy to make very expensive queries that don't return many rows.

  moving away from `max_returned_rows` will requires some thinking about:

  1. memory usage and data flows to handle potentially very large result sets
  2. how to avoid rendering tens or hundreds of thousands of [html rows](#1655).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/526#issuecomment-1259693536
issue_url: https://api.github.com/repos/simonw/datasette/issues/526
node_id: IC_kwDOBm6k_c5LFWXg
created_at: 2022-09-27T15:42:55Z
updated_at: 2022-09-27T15:42:55Z
author_association: OWNER
body:
  It's interesting to note WHY the time limit works against this so well.

  The time limit as-implemented looks like this:

  https://github.com/simonw/datasette/blob/5f9f567acbc58c9fcd88af440e68034510fb5d2b/datasette/utils/__init__.py#L181-L201

  The key here is `conn.set_progress_handler(handler, n)` - which specifies that the handler function should be called every `n` SQLite operations.

  The handler function then checks to see if too much time has transpired and conditionally cancels the query.

  This also doubles up as a "maximum number of operations" guard, which is what's happening when you attempt to fetch an infinite number of rows from an infinite table.

  That limit code could even be extended to say "exit the query after either 5s or 50,000,000 operations".

  I don't think that's necessary though.

  To be honest I'm having trouble with the idea of dropping `max_returned_rows` mainly because what Datasette does (allow arbitrary untrusted SQL queries) is dangerous, so I've designed in multiple redundant defence-in-depth mechanisms right from the start.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/526#issuecomment-1259718517
issue_url: https://api.github.com/repos/simonw/datasette/issues/526
node_id: IC_kwDOBm6k_c5LFcd1
created_at: 2022-09-27T16:02:51Z
updated_at: 2022-09-27T16:04:46Z
author_association: CONTRIBUTOR
body:
  i think that `max_returned_rows` **is** a defense mechanism, just not for connection exhaustion. `max_returned_rows` is a defense mechanism against **memory bombs**.

  if you are potentially yielding out hundreds of thousands or even millions of rows, you need to be quite careful about data flow to not run out of memory on the server, or on the client.

  you have a lot of places in your code that are protective of that right now, but `max_returned_rows` acts as the final backstop.

  so, given that, it makes sense to have removing `max_returned_rows` altogether be a non-goal, but instead allow for for specific codepaths (like streaming csv's) be able to bypass.

  that could dramatically lower the surface area for a memory-bomb attack.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/526#issuecomment-1260355224
issue_url: https://api.github.com/repos/simonw/datasette/issues/526
node_id: IC_kwDOBm6k_c5LH36Y
created_at: 2022-09-28T04:01:25Z
updated_at: 2022-09-28T04:01:25Z
author_association: OWNER
body:
  The ultimate protection against those memory bombs is to support more streaming output formats. Related issues:

  - #1177 
  - #1062

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1826#issuecomment-1260357583
issue_url: https://api.github.com/repos/simonw/datasette/issues/1826
node_id: IC_kwDOBm6k_c5LH4fP
created_at: 2022-09-28T04:05:16Z
updated_at: 2022-09-28T04:05:16Z
author_association: OWNER
body:
  This is deliberate. The Datasette plugin system allows you to specify only a subset of the parameters for a hook - in this example, only the `value` is needed so the others can be omitted.

  There's a note about this at the very top of that documentation page: https://docs.datasette.io/en/stable/plugin_hooks.html#plugin-hooks

  > When you implement a plugin hook you can accept any or all of the parameters that are documented as being passed to that hook.
  > 
  > For example, you can implement the `render_cell` plugin hook like this even though the full documented hook signature is `render_cell(value, column, table, database, datasette)`:
  > ```python
  > @hookimpl
  > def render_cell(value, column):
  >     if column == "stars":
  >         return "*" * int(value)
  > ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1826#issuecomment-1260357878
issue_url: https://api.github.com/repos/simonw/datasette/issues/1826
node_id: IC_kwDOBm6k_c5LH4j2
created_at: 2022-09-28T04:05:45Z
updated_at: 2022-09-28T04:05:45Z
author_association: OWNER
body:
  Though now I notice that the copy right there needs to be updated to reflect the new `row` parameter to `render_cell`!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1825#issuecomment-1260368122
issue_url: https://api.github.com/repos/simonw/datasette/issues/1825
node_id: IC_kwDOBm6k_c5LH7D6
created_at: 2022-09-28T04:20:28Z
updated_at: 2022-09-28T04:20:28Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1825?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Base: **91.58**% // Head: **91.58**% // No change to project coverage :thumbsup:
  > Coverage data is based on head [(`b16eb2f`)](https://codecov.io/gh/simonw/datasette/pull/1825?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`5f9f567`)](https://codecov.io/gh/simonw/datasette/commit/5f9f567acbc58c9fcd88af440e68034510fb5d2b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > Patch has no changes to coverable lines.

  > :exclamation: Current head b16eb2f differs from pull request most recent head e7e96dc. Consider uploading reports for the commit e7e96dc to get more accurate results

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main    #1825   +/-   ##
  =======================================
    Coverage   91.58%   91.58%           
  =======================================
    Files          36       36           
    Lines        4444     4444           
  =======================================
    Hits         4070     4070           
    Misses        374      374           
  ```



  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  </details>

  [:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1825?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1825#issuecomment-1260368537
issue_url: https://api.github.com/repos/simonw/datasette/issues/1825
node_id: IC_kwDOBm6k_c5LH7KZ
created_at: 2022-09-28T04:21:18Z
updated_at: 2022-09-28T04:21:18Z
author_association: OWNER
body:
  This is great, thank you very much!

  https://datasette--1825.org.readthedocs.build/en/1825/deploying.html#running-datasette-using-openrc

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1826#issuecomment-1260373403
issue_url: https://api.github.com/repos/simonw/datasette/issues/1826
node_id: IC_kwDOBm6k_c5LH8Wb
created_at: 2022-09-28T04:30:27Z
updated_at: 2022-09-28T04:30:27Z
author_association: NONE
body:
  I'm glad the bug report served some purpose. Frankly I just needed the
  method signature, that is why the documentation you mention wasn't read.

  On Tue, Sep 27, 2022, 9:05 PM Simon Willison ***@***.***>
  wrote:

  > Though now I notice that the copy right there needs to be updated to
  > reflect the new row parameter to render_cell!
  >
  > —
  > Reply to this email directly, view it on GitHub
  > <https://github.com/simonw/datasette/issues/1826#issuecomment-1260357878>,
  > or unsubscribe
  > <https://github.com/notifications/unsubscribe-auth/AP46PCLI6KVDQTFLWVZODRLWAO72JANCNFSM6AAAAAAQXKWIJA>
  > .
  > You are receiving this because you authored the thread.Message ID:
  > ***@***.***>
  >

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1062#issuecomment-1260829829
issue_url: https://api.github.com/repos/simonw/datasette/issues/1062
node_id: IC_kwDOBm6k_c5LJryF
created_at: 2022-09-28T12:27:19Z
updated_at: 2022-09-28T12:27:19Z
author_association: CONTRIBUTOR
body:
  for teaching `register_output_renderer` to stream it seems like the two options are to

  1. a [nested query technique ](https://github.com/simonw/datasette/issues/526#issuecomment-505162238)to paginate through
  2. a fetching model that looks like something
  ```python
  with sqlite_timelimit(conn, time_limit_ms):
       c.execute(query)
       for chunk in c.fetchmany(chunk_size):
           yield from chunk
  ```
  currently `db.execute` is not a generator, so this would probably need a new method?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1062#issuecomment-1260909128
issue_url: https://api.github.com/repos/simonw/datasette/issues/1062
node_id: IC_kwDOBm6k_c5LJ_JI
created_at: 2022-09-28T13:22:53Z
updated_at: 2022-09-28T14:09:54Z
author_association: CONTRIBUTOR
body:
  if you went this route:

  ```python
  with sqlite_timelimit(conn, time_limit_ms):
       c.execute(query)
       for chunk in c.fetchmany(chunk_size):
           yield from chunk
  ```

  then `time_limit_ms` would probably have to be greatly extended, because the time spent in the loop will depend on the downstream processing.

  i wonder if this was why you were thinking this feature would need a dedicated connection?

  ---

  reading more, there's no real limit i can find on the number of active cursors (or more precisely active prepared statements objects, because sqlite doesn't really have cursors). 

  maybe something like this would be okay?

  ```python
  with sqlite_timelimit(conn, time_limit_ms):
       c.execute(query)
       # step through at least one to evaluate the statement, not sure if this is necessary
       yield c.execute.fetchone()
  for chunk in c.fetchmany(chunk_size):
      yield from chunk
  ```

  this seems quite weird that there's not more of limit of the number of active prepared statements, but i haven't been able to find one.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/370#issuecomment-1261930179
issue_url: https://api.github.com/repos/simonw/datasette/issues/370
node_id: IC_kwDOBm6k_c5LN4bD
created_at: 2022-09-29T08:17:46Z
updated_at: 2022-09-29T08:17:46Z
author_association: CONTRIBUTOR
body:
  Just watched this video which demonstrates the integration of *any* webapp into JupyterLab: https://youtu.be/FH1dKKmvFtc

  Maybe this is the answer?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1262913145
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/297
node_id: IC_kwDOCGYnMM5LRoZ5
created_at: 2022-09-29T22:54:13Z
updated_at: 2022-09-29T22:54:13Z
author_association: OWNER
body:
  After reviewing `sqlite-utils insert --help` I'm confident that MOST of these options wouldn't make sense for a "fast" moder that just supports CSV and works by piping directly to the `sqlite3` binary:

  https://github.com/simonw/sqlite-utils/blob/d792dad1cf5f16525da81b1e162fb71d469995f3/docs/cli-reference.rst#L251-L279

  I'm going to implement a separate command instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1262914416
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/297
node_id: IC_kwDOCGYnMM5LRotw
created_at: 2022-09-29T22:56:53Z
updated_at: 2022-09-29T22:56:53Z
author_association: OWNER
body:
  Potential names/designs:

  - `sqlite-utils fast data.db rows rows.csv`
  - `sqlite-utils insert-fast data.db rows rows.csv`
  - `sqlite-utils fast-csv data.db rows rows.csv`

  Or more interestingly... what if it could accept multiple CSV files to create multiple tables?

  - `sqlite-utils fast data.db rows.csv other.csv`

  Would still need to support creating tables with different names though. Maybe like this:

  - `sqlite-utils fast data.db -t mytable rows.csv -t othertable other.csv`

  I seem to be leaning towards `fast` as the command name, but as a standalone command name it's a bit meaningless - how do we know that's about CSV import and not about fast querying or similar?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1262915322
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/297
node_id: IC_kwDOCGYnMM5LRo76
created_at: 2022-09-29T22:57:31Z
updated_at: 2022-09-29T22:57:42Z
author_association: OWNER
body:
  Maybe `sqlite-utils fast-csv` is right? Not entirely clear that's an insert though as opposed to a faster version of in-memory querying in the style of `sqlite-utils memory`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1262917059
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/297
node_id: IC_kwDOCGYnMM5LRpXD
created_at: 2022-09-29T22:59:28Z
updated_at: 2022-09-29T22:59:28Z
author_association: OWNER
body:
  I quite like `sqlite-utils fast-csv` - I think it's clear enough what it does, and running `--help` can clarify if needed.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1262918833
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/297
node_id: IC_kwDOCGYnMM5LRpyx
created_at: 2022-09-29T23:02:52Z
updated_at: 2022-09-29T23:02:52Z
author_association: OWNER
body:
  The other nice thing about having this as a separate command is that I can implement a tiny subset of the overall `sqlite-utils insert` features at first, and then add additional features in subsequent releases.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1262920929
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/297
node_id: IC_kwDOCGYnMM5LRqTh
created_at: 2022-09-29T23:06:44Z
updated_at: 2022-09-29T23:06:44Z
author_association: OWNER
body:
  Currently the only other use of `-t` is for this:
  ```
    -t, --table            Output as a formatted table
  ```
  So I think it's OK to use it to mean something slightly different for this command, since `sqlite-utils insert` doesn't do any output of data in any format.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1827#issuecomment-1263570186
issue_url: https://api.github.com/repos/simonw/datasette/issues/1827
node_id: IC_kwDOBm6k_c5LUI0K
created_at: 2022-09-30T13:22:15Z
updated_at: 2022-09-30T13:22:15Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1827?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Base: **92.50**% // Head: **92.50**% // No change to project coverage :thumbsup:
  > Coverage data is based on head [(`1f0c557`)](https://codecov.io/gh/simonw/datasette/pull/1827?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`34defdc`)](https://codecov.io/gh/simonw/datasette/commit/34defdc10aa293294ca01cfab70780755447e1d7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > Patch has no changes to coverable lines.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main    #1827   +/-   ##
  =======================================
    Coverage   92.50%   92.50%           
  =======================================
    Files          35       35           
    Lines        4400     4400           
  =======================================
    Hits         4070     4070           
    Misses        330      330           
  ```



  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  </details>

  [:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1827?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1264218914
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/491
node_id: IC_kwDOCGYnMM5LWnMi
created_at: 2022-10-01T03:18:36Z
updated_at: 2023-06-14T22:14:24Z
author_association: CONTRIBUTOR
body:
  > some good concrete use-cases in mind

  I actually found myself wanting something like this the past couple days. The use-case was databases with slightly different schema but same table names.

  here is a full script:

  ```
  import argparse
  from pathlib import Path

  from sqlite_utils import Database


  def connect(args, conn=None, **kwargs) -> Database:
      db = Database(conn or args.database, **kwargs)
      with db.conn:
          db.conn.execute("PRAGMA main.cache_size = 8000")
      return db


  def parse_args() -> argparse.Namespace:
      parser = argparse.ArgumentParser()
      parser.add_argument("database")
      parser.add_argument("dbs_folder")
      parser.add_argument("--db", "-db", help=argparse.SUPPRESS)
      parser.add_argument("--verbose", "-v", action="count", default=0)
      args = parser.parse_args()

      if args.db:
          args.database = args.db
      Path(args.database).touch()
      args.db = connect(args)

      return args


  def merge_db(args, source_db):
      source_db = str(Path(source_db).resolve())

      s_db = connect(argparse.Namespace(database=source_db, verbose = args.verbose))
      for table in s_db.table_names():
          data = s_db[table].rows
          args.db[table].insert_all(data, alter=True, replace=True)

      args.db.conn.commit()


  def merge_directory():
      args = parse_args()
      source_dbs = list(Path(args.dbs_folder).glob('*.db'))
      for s_db in source_dbs:
          merge_db(args, s_db)


  if __name__ == '__main__':
      merge_directory()
  ```

  edit: I've made some improvements to this and put it on PyPI:

  ```
  $ pip install xklb
  $ lb merge-db -h
  usage: library merge-dbs DEST_DB SOURCE_DB ... [--only-target-columns] [--only-new-rows] [--upsert] [--pk PK ...] [--table TABLE ...]

      Merge-DBs will insert new rows from source dbs to target db, table by table. If primary key(s) are provided,
      and there is an existing row with the same PK, the default action is to delete the existing row and insert the new row
      replacing all existing fields.

      Upsert mode will update matching PK rows such that if a source row has a NULL field and
      the destination row has a value then the value will be preserved instead of changed to the source row's NULL value.

      Ignore mode (--only-new-rows) will insert only rows which don't already exist in the destination db

      Test first by using temp databases as the destination db.
      Try out different modes / flags until you are satisfied with the behavior of the program

          library merge-dbs --pk path (mktemp --suffix .db) tv.db movies.db

      Merge database data and tables

          library merge-dbs --upsert --pk path video.db tv.db movies.db
          library merge-dbs --only-target-columns --only-new-rows --table media,playlists --pk path audio-fts.db audio.db

          library merge-dbs --pk id --only-tables subreddits reddit/81_New_Music.db audio.db
          library merge-dbs --only-new-rows --pk subreddit,path --only-tables reddit_posts reddit/81_New_Music.db audio.db -v

  positional arguments:
    database
    source_dbs
  ```

  Also if you want to dedupe a table based on a "business key" which isn't explicitly your primary key(s) you can run this:

  ```
  $ lb dedupe-db -h
  usage: library dedupe-dbs DATABASE TABLE --bk BUSINESS_KEYS [--pk PRIMARY_KEYS] [--only-columns COLUMNS]

      Dedupe your database (not to be confused with the dedupe subcommand)

      It should not need to be said but *backup* your database before trying this tool!

      Dedupe-DB will help remove duplicate rows based on non-primary-key business keys

          library dedupe-db ./video.db media --bk path

      If --primary-keys is not provided table metadata primary keys will be used
      If --only-columns is not provided all non-primary and non-business key columns will be upserted

  positional arguments:
    database
    table

  options:
    -h, --help            show this help message and exit
    --skip-0
    --only-columns ONLY_COLUMNS
                          Comma separated column names to upsert
    --primary-keys PRIMARY_KEYS, --pk PRIMARY_KEYS
                          Comma separated primary keys
    --business-keys BUSINESS_KEYS, --bk BUSINESS_KEYS
                          Comma separated business keys
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/493#issuecomment-1264219650
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/493
node_id: IC_kwDOCGYnMM5LWnYC
created_at: 2022-10-01T03:22:50Z
updated_at: 2022-10-01T03:23:58Z
author_association: CONTRIBUTOR
body:
  this is likely what you are looking for: https://stackoverflow.com/a/51076749/697964

  but yeah I would say just disable smart quotes

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/409#issuecomment-1264223363
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/409
node_id: IC_kwDOCGYnMM5LWoSD
created_at: 2022-10-01T03:41:45Z
updated_at: 2022-10-01T03:41:45Z
author_association: CONTRIBUTOR
body:
  ```
  pytest xklb/check.py --pdb

  xklb/check.py:11: in test_transaction
      assert list(db2["t"].rows) == []
  E   AssertionError: assert [{'foo': 1}] == []
  E    +  where [{'foo': 1}] = list(<generator object Queryable.rows_where at 0x7f2d84d1f0d0>)
  E    +    where <generator object Queryable.rows_where at 0x7f2d84d1f0d0> = <Table t (foo)>.rows
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  > /home/xk/github/xk/lb/xklb/check.py(11)test_transaction()
        9     with db1.conn:
       10         db1["t"].insert({"foo": 1})
  ---> 11         assert list(db2["t"].rows) == []
       12     assert list(db2["t"].rows) == [{"foo": 1}]
  ```

  It fails because it is already inserted.

  btw if you put these two lines in you pyproject.toml you can get `ipdb` in pytest

  ```
  [tool.pytest.ini_options]
  addopts = "--pdbcls=IPython.terminal.debugger:TerminalPdb --ignore=tests/data --capture=tee-sys --log-cli-level=ERROR"
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/485#issuecomment-1264737290
issue_url: https://api.github.com/repos/simonw/datasette/issues/485
node_id: IC_kwDOBm6k_c5LYlwK
created_at: 2022-10-02T21:29:59Z
updated_at: 2022-10-02T21:29:59Z
author_association: OWNER
body:
  To clarify: the feature this issue is talking about relates to the way Datasette automatically displays foreign key relationships, for example on this page: https://github-to-sqlite.dogsheep.net/github/commits

  <img width="1233" alt="image" src="https://user-images.githubusercontent.com/9599/193476985-d41148cf-2b2f-49b9-b717-e92145afab31.png">

  Each of those columns is a foreign key to another table. The link text that is displayed there comes from the "label column" that has either been configured or automatically detected for that other table.

  I wonder if this could be handled with a tiny machine learning model that's trained to help pick the best label column?

  Inputs to that model could include:

  - The names of the columns
  - The number of unique values in each column
  - The type of each column (or maybe only `TEXT` columns should be considered)
  - How many `null` values there are
  - Is the column marked as unique?
  - What's the average (or median or some other statistic) string length of values in each column?

  Output would be the most likely label column, or some indicator that no likely candidates had been found.

  My hunch is that this would be better solved using a few extra heuristics rather than by training a model, but it does feel like an interesting opportunity to experiment with a tiny ML model.

  Asked for tips about this on Twitter: https://twitter.com/simonw/status/1576680930680262658

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1828#issuecomment-1264738081
issue_url: https://api.github.com/repos/simonw/datasette/issues/1828
node_id: IC_kwDOBm6k_c5LYl8h
created_at: 2022-10-02T21:34:37Z
updated_at: 2022-10-02T21:34:37Z
author_association: OWNER
body:
  I'm running a build of that demo instance here (takes ~30m) https://github.com/dogsheep/github-to-sqlite/actions/runs/3170164705

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1828#issuecomment-1264753439
issue_url: https://api.github.com/repos/simonw/datasette/issues/1828
node_id: IC_kwDOBm6k_c5LYpsf
created_at: 2022-10-02T23:01:17Z
updated_at: 2022-10-02T23:01:17Z
author_association: OWNER
body:
  That change deployed and https://github-to-sqlite.dogsheep.net/github/commits now looks like this:

  <img width="1388" alt="image" src="https://user-images.githubusercontent.com/9599/193480158-de81ac0a-5cb2-4d53-a75c-025c78f293ee.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1805#issuecomment-1264753725
issue_url: https://api.github.com/repos/simonw/datasette/issues/1805
node_id: IC_kwDOBm6k_c5LYpw9
created_at: 2022-10-02T23:02:17Z
updated_at: 2022-10-02T23:02:17Z
author_association: OWNER
body:
  After reverting `word--wrap anywhere` https://latest.datasette.io/_memory?sql=select+%27https%3A%2F%2Fexample.com%2Faaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.jpg%27+as+truncated now looks like this, which isn't as good:

  <img width="1640" alt="image" src="https://user-images.githubusercontent.com/9599/193480193-050bd5b4-720c-41e6-be71-1c1bb9c4befd.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1805#issuecomment-1264753894
issue_url: https://api.github.com/repos/simonw/datasette/issues/1805
node_id: IC_kwDOBm6k_c5LYpzm
created_at: 2022-10-02T23:02:54Z
updated_at: 2022-10-02T23:02:54Z
author_association: OWNER
body:
  I'm tempted to add `word-wrap: anywhere` only to links that are know to be longer than a certain threshold.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/485#issuecomment-1264769569
issue_url: https://api.github.com/repos/simonw/datasette/issues/485
node_id: IC_kwDOBm6k_c5LYtoh
created_at: 2022-10-03T00:04:42Z
updated_at: 2022-10-03T00:04:42Z
author_association: OWNER
body:
  I love these tips - tools that can compile a simple machine learning model to a SQL query! Would be pretty cool if I could bundle a model in Datasette itself as a big in-memory SQLite SQL query:

  - https://github.com/Chryzanthemum/xgb2sql
  - https://github.com/konstantint/SKompiler

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1805#issuecomment-1265161668
issue_url: https://api.github.com/repos/simonw/datasette/issues/1805
node_id: IC_kwDOBm6k_c5LaNXE
created_at: 2022-10-03T09:18:05Z
updated_at: 2022-10-03T09:18:05Z
author_association: NONE
body:
  > I'm tempted to add `word-wrap: anywhere` only to links that are know to be longer than a certain threshold.

  Make sense IMHO.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1829#issuecomment-1267708232
issue_url: https://api.github.com/repos/simonw/datasette/issues/1829
node_id: IC_kwDOBm6k_c5Lj7FI
created_at: 2022-10-04T23:17:36Z
updated_at: 2022-10-04T23:17:36Z
author_association: OWNER
body:
  Here's the relevant code from the table page:

  https://github.com/simonw/datasette/blob/4218c9cd742b79b1e3cb80878e42b7e39d16ded2/datasette/views/table.py#L215-L227

  Note how `ensure_permissions()` there takes the table, database and instance into account... but the `private` assignment  (used to decide if the padlock should display or not) only considers the `view-table` check.

  Here's the same code for the database page:

  https://github.com/simonw/datasette/blob/4218c9cd742b79b1e3cb80878e42b7e39d16ded2/datasette/views/database.py#L139-L141

  And for canned query pages:

  https://github.com/simonw/datasette/blob/4218c9cd742b79b1e3cb80878e42b7e39d16ded2/datasette/views/database.py#L228-L240

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1829#issuecomment-1267709546
issue_url: https://api.github.com/repos/simonw/datasette/issues/1829
node_id: IC_kwDOBm6k_c5Lj7Zq
created_at: 2022-10-04T23:19:24Z
updated_at: 2022-10-04T23:21:07Z
author_association: OWNER
body:
  There's also a `check_visibility()` helper which I'm not using in these particular cases but which may be relevant. It's called like this:

  https://github.com/simonw/datasette/blob/4218c9cd742b79b1e3cb80878e42b7e39d16ded2/datasette/views/database.py#L65-L77

  And is defined here: https://github.com/simonw/datasette/blob/4218c9cd742b79b1e3cb80878e42b7e39d16ded2/datasette/app.py#L694-L710

  It's actually documented as a public method here: https://docs.datasette.io/en/stable/internals.html#await-check-visibility-actor-action-resource-none

  > This convenience method can be used to answer the question "should this item be considered private, in that it is visible to me but it is not visible to anonymous users?"
  >
  > It returns a tuple of two booleans, `(visible, private)`. `visible` indicates if the actor can see this resource. `private` will be `True` if an anonymous user would not be able to view the resource.

  Note that this documented method cannot actually do the right thing - because it's not being given the multiple permissions that need to be checked in order to completely answer the question.

  So I probably need to redesign that method a bit.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1832#issuecomment-1267918117
issue_url: https://api.github.com/repos/simonw/datasette/issues/1832
node_id: IC_kwDOBm6k_c5LkuUl
created_at: 2022-10-05T04:19:52Z
updated_at: 2022-10-05T04:19:52Z
author_association: OWNER
body:
  Code can go here:

  https://github.com/simonw/datasette/blob/b6ba117b7978b58b40e3c3c2b723b92c3010ed53/datasette/database.py#L511-L515

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1832#issuecomment-1267925830
issue_url: https://api.github.com/repos/simonw/datasette/issues/1832
node_id: IC_kwDOBm6k_c5LkwNG
created_at: 2022-10-05T04:31:57Z
updated_at: 2022-10-05T04:31:57Z
author_association: OWNER
body:
  Turns out this already works - `__bool__` falls back on `__len__`: https://docs.python.org/3/reference/datamodel.html#object.__bool__

  > When this method is not defined, [`__len__()`](https://docs.python.org/3/reference/datamodel.html#object.__len__ "object.__len__") is called, if it is defined, and the object is considered true if its result is nonzero.

  I'll add a test to demonstrate this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1824#issuecomment-1268398461
issue_url: https://api.github.com/repos/simonw/datasette/issues/1824
node_id: IC_kwDOBm6k_c5Lmjl9
created_at: 2022-10-05T12:55:05Z
updated_at: 2022-10-05T12:55:05Z
author_association: NONE
body:
  Here is some working javascript code. There might be better solution, I'm not a JS expert.
  ```javascript
      var show_hide = document.querySelector(".show-hide-sql > a");

      // Hide SQL query if the URL opened with #_hide_sql
      var hash = window.location.hash;
      if(hash === "#_hide_sql") {
          hide_sql();
      }
      show_hide.setAttribute("href", "#");
      show_hide.addEventListener("click", toggle_sql_display);

      function toggle_sql_display() {
          if (show_hide.innerText === "hide") {
              hide_sql();
              return;
          }
          if (show_hide.innerText === "show") {
              show_sql();
              return;
          }
      }

      function hide_sql() {
          sql_element.style.cssText="display:none";
          show_hide.innerHTML = "show";
          show_hide.setAttribute("href", "#_hide_sql");
      }

      function show_sql() {
          sql_element.style.cssText="display:block";
          show_hide.innerHTML = "hide";
          show_hide.setAttribute("href", "#_show_sql");
      }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1480#issuecomment-1268613335
issue_url: https://api.github.com/repos/simonw/datasette/issues/1480
node_id: IC_kwDOBm6k_c5LnYDX
created_at: 2022-10-05T15:45:49Z
updated_at: 2022-10-05T15:45:49Z
author_association: CONTRIBUTOR
body:
  running into this as i continue to grow my labor data warehouse.

  Here a CloudRun PM says the container size should **not** count against memory: https://stackoverflow.com/a/56570717

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1480#issuecomment-1268629159
issue_url: https://api.github.com/repos/simonw/datasette/issues/1480
node_id: IC_kwDOBm6k_c5Lnb6n
created_at: 2022-10-05T16:00:55Z
updated_at: 2022-10-05T16:00:55Z
author_association: CONTRIBUTOR
body:
  as a next step, i'll fetch the docker image from the google registry, and see what memory and disk usage looks like when i run it locally.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1480#issuecomment-1269275153
issue_url: https://api.github.com/repos/simonw/datasette/issues/1480
node_id: IC_kwDOBm6k_c5Lp5oR
created_at: 2022-10-06T03:54:33Z
updated_at: 2022-10-06T03:54:33Z
author_association: OWNER
body:
  I've been having success using Fly recently for a project which I thought would be too large for Cloud Run. I wrote about that here:

  - https://simonwillison.net/2022/Sep/5/laion-aesthetics-weeknotes/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1480#issuecomment-1269847461
issue_url: https://api.github.com/repos/simonw/datasette/issues/1480
node_id: IC_kwDOBm6k_c5LsFWl
created_at: 2022-10-06T11:21:49Z
updated_at: 2022-10-06T11:21:49Z
author_association: CONTRIBUTOR
body:
  thanks @simonw, i'll spend a little more time trying to figure out why this isn't working on cloudrun, and then will flip over to fly if i can't.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1835#issuecomment-1270586897
issue_url: https://api.github.com/repos/simonw/datasette/issues/1835
node_id: IC_kwDOBm6k_c5Lu54R
created_at: 2022-10-06T19:34:00Z
updated_at: 2022-10-06T19:34:00Z
author_association: OWNER
body:
  Wow, great catch! The whole point of inspect data was to avoid this kind of expensive operation on startup so this makes total sense - I had no idea Datasette was still trying to hash a giant file every time the server started.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1835#issuecomment-1270595328
issue_url: https://api.github.com/repos/simonw/datasette/issues/1835
node_id: IC_kwDOBm6k_c5Lu78A
created_at: 2022-10-06T19:42:25Z
updated_at: 2022-10-06T19:42:25Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1835?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Base: **91.71**% // Head: **92.50**% // Increases project coverage by **`+0.78%`** :tada:
  > Coverage data is based on head [(`b4b92df`)](https://codecov.io/gh/simonw/datasette/pull/1835?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`cb1e093`)](https://codecov.io/gh/simonw/datasette/commit/cb1e093fd361b758120aefc1a444df02462389a3?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > Patch has no changes to coverable lines.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@            Coverage Diff             @@
  ##             main    #1835      +/-   ##
  ==========================================
  + Coverage   91.71%   92.50%   +0.78%     
  ==========================================
    Files          38       35       -3     
    Lines        4754     4400     -354     
  ==========================================
  - Hits         4360     4070     -290     
  + Misses        394      330      -64     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1835?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/database.py](https://codecov.io/gh/simonw/datasette/pull/1835/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RhdGFiYXNlLnB5) | | |
  | [datasette/utils/shutil\_backport.py](https://codecov.io/gh/simonw/datasette/pull/1835/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL3NodXRpbF9iYWNrcG9ydC5weQ==) | | |
  | [datasette/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1835/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL19faW5pdF9fLnB5) | | |
  | [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1835/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `94.75% <0.00%> (+0.01%)` | :arrow_up: |

  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  </details>

  [:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1835?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1837#issuecomment-1270855853
issue_url: https://api.github.com/repos/simonw/datasette/issues/1837
node_id: IC_kwDOBm6k_c5Lv7it
created_at: 2022-10-07T00:01:20Z
updated_at: 2022-10-07T00:01:20Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1837?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Base: **92.50**% // Head: **92.50**% // No change to project coverage :thumbsup:
  > Coverage data is based on head [(`c12447e`)](https://codecov.io/gh/simonw/datasette/pull/1837?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`eff1124`)](https://codecov.io/gh/simonw/datasette/commit/eff112498ecc499323c26612d707908831446d25?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > Patch has no changes to coverable lines.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main    #1837   +/-   ##
  =======================================
    Coverage   92.50%   92.50%           
  =======================================
    Files          35       35           
    Lines        4400     4400           
  =======================================
    Hits         4070     4070           
    Misses        330      330           
  ```



  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  </details>

  [:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1837?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1836#issuecomment-1270923537
issue_url: https://api.github.com/repos/simonw/datasette/issues/1836
node_id: IC_kwDOBm6k_c5LwMER
created_at: 2022-10-07T00:46:08Z
updated_at: 2022-10-07T00:46:08Z
author_association: CONTRIBUTOR
body:
  i thought it was maybe to do with reading through all the files, but that does not seem to be the case

  if i make a little test file like:

  ```python
  # test_read.py
  import hashlib
  import sys
  import pathlib

  HASH_BLOCK_SIZE = 1024 * 1024

  def inspect_hash(path):
      """Calculate the hash of a database, efficiently."""
      m = hashlib.sha256()
      with path.open("rb") as fp:
          while True:
              data = fp.read(HASH_BLOCK_SIZE)
              if not data:
                  break
              m.update(data)

      return m.hexdigest()

  inspect_hash(pathlib.Path(sys.argv[1]))
  ```

  then a line in the Dockerfile like

  ```docker
  RUN python test_read.py nlrb.db && echo "[]" > /etc/inspect.json
  ```

  just produes a layer of `3B`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1836#issuecomment-1270936982
issue_url: https://api.github.com/repos/simonw/datasette/issues/1836
node_id: IC_kwDOBm6k_c5LwPWW
created_at: 2022-10-07T00:52:41Z
updated_at: 2022-10-07T00:52:41Z
author_association: CONTRIBUTOR
body:
  it's not that the inspect command is somehow changing the db files. if i set them to only read-only, the "inspect" layer still has the same very large size.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1836#issuecomment-1270988081
issue_url: https://api.github.com/repos/simonw/datasette/issues/1836
node_id: IC_kwDOBm6k_c5Lwb0x
created_at: 2022-10-07T01:19:01Z
updated_at: 2022-10-07T01:27:35Z
author_association: CONTRIBUTOR
body:
  okay, some progress!! running some sql against a database file causes that file to get duplicated even if it doesn't apparently change the file.

  make a little test script like this:

  ```python
  # test_sql.py
  import sqlite3
  import sys

  db_name = sys.argv[1]
  conn = sqlite3.connect(f'file:/app/{db_name}', uri=True)
  cur = conn.cursor()
  cur.execute('select count(*) from filing')
  print(cur.fetchone())
  ```

  then 

  ```docker
  RUN python test_sql.py nlrb.db
  ```

  produced a layer that's the same size as `nlrb.db`!!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1836#issuecomment-1270992795
issue_url: https://api.github.com/repos/simonw/datasette/issues/1836
node_id: IC_kwDOBm6k_c5Lwc-b
created_at: 2022-10-07T01:29:15Z
updated_at: 2022-10-07T01:50:14Z
author_association: CONTRIBUTOR
body:
  fascinatingly, telling python to open sqlite in read only mode makes this layer have a size of 0

  ```python
  # test_sql_ro.py
  import sqlite3
  import sys

  db_name = sys.argv[1]
  conn = sqlite3.connect(f'file:/app/{db_name}?mode=ro', uri=True)
  cur = conn.cursor()
  cur.execute('select count(*) from filing')
  print(cur.fetchone())
  ```

  that's quite weird because setting the file permissions to read only didn't do anything. (on reflection, that chmod isn't doing anything because the dockerfile commands are run as root)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1836#issuecomment-1271003212
issue_url: https://api.github.com/repos/simonw/datasette/issues/1836
node_id: IC_kwDOBm6k_c5LwfhM
created_at: 2022-10-07T01:52:04Z
updated_at: 2022-10-07T01:52:04Z
author_association: CONTRIBUTOR
body:
  and if we try immutable mode, which is how things are opened by `datasette inspect` we duplicate the files!!!

  ```python
  # test_sql_immutable.py
  import sqlite3
  import sys

  db_name = sys.argv[1]
  conn = sqlite3.connect(f'file:/app/{db_name}?immutable=1', uri=True)
  cur = conn.cursor()
  cur.execute('select count(*) from filing')
  print(cur.fetchone())
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1836#issuecomment-1271004167
issue_url: https://api.github.com/repos/simonw/datasette/issues/1836
node_id: IC_kwDOBm6k_c5LwfwH
created_at: 2022-10-07T01:53:05Z
updated_at: 2022-10-07T01:53:05Z
author_association: OWNER
body:
  Oh this is interesting!

  Is your hunch here that running this line is causing the file to be stored as a second layer?

  https://github.com/simonw/datasette/blob/5aa359b86907d11b3ee601510775a85a90224da8/datasette/utils/__init__.py#L399

  I guess it's possible that running a non-read-only query against the database causes one or two bytes to be changed (maybe a transaction ID or similar?)

  Modifying the `inspect` command to use `?mode=ro` seems sensible to me. Except.... it should already be opening those files in immutable mode according to this line: https://github.com/simonw/datasette/blob/eff112498ecc499323c26612d707908831446d25/datasette/cli.py#L172-L173

  Here's what opening as a `immutables` does:

  https://github.com/simonw/datasette/blob/eff112498ecc499323c26612d707908831446d25/datasette/app.py#L258-L260

  https://github.com/simonw/datasette/blob/eff112498ecc499323c26612d707908831446d25/datasette/database.py#L98

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1836#issuecomment-1271006020
issue_url: https://api.github.com/repos/simonw/datasette/issues/1836
node_id: IC_kwDOBm6k_c5LwgNE
created_at: 2022-10-07T01:54:07Z
updated_at: 2022-10-07T01:54:07Z
author_association: OWNER
body:
  Just overlapped with your comment here: https://github.com/simonw/datasette/issues/1836#issuecomment-1271003212 - which notes that opening with `?immutable=1` DOES seem to cause the file to be duplicated!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1836#issuecomment-1271008997
issue_url: https://api.github.com/repos/simonw/datasette/issues/1836
node_id: IC_kwDOBm6k_c5Lwg7l
created_at: 2022-10-07T02:00:37Z
updated_at: 2022-10-07T02:00:49Z
author_association: CONTRIBUTOR
body:
  yes, and i also think that this is causing the apparent memory problems in #1480. when the container starts up, it will make some operation on the database in `immutable` mode which apparently makes some small change to the db file. if that's so, then the db files will be copied to the read/write layer which counts against cloudrun's memory allocation!

  running a test of that now. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1838#issuecomment-1271009214
issue_url: https://api.github.com/repos/simonw/datasette/issues/1838
node_id: IC_kwDOBm6k_c5Lwg--
created_at: 2022-10-07T02:01:07Z
updated_at: 2022-10-07T02:01:07Z
author_association: OWNER
body:
  The argument that has always convinced me NOT to use `target="_blank"` (even for links like this one) is that it breaks browser expectations.

  If you click a link with `target="_blank" on it you get a new browser window... with a disabled back button. You have to then know to close that browser window in order to return to the previous page - as opposed to hitting the "back" button like usual.

  You'll note that Datasette doesn't use `target="_blank"` even on URLs presented in database tables - like these ones: https://latest.datasette.io/fixtures/roadside_attractions

  So I'm very firmly in the anti-target-blank camp!

  This is the kind of change which I'd suggest implementing as a plugin. `datasette-external-links-new-windows` could run a bit of JavaScript on every page that looks for `<a href="...">` elements that link to off-domain pages and adds `target="_blank"` to them via the DOM.

  That way people who like `target="_blank"` can have it!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1836#issuecomment-1271020193
issue_url: https://api.github.com/repos/simonw/datasette/issues/1836
node_id: IC_kwDOBm6k_c5Lwjqh
created_at: 2022-10-07T02:15:05Z
updated_at: 2022-10-07T02:21:08Z
author_association: CONTRIBUTOR
body:
  when i hack the connect method to open non mutable files with "mode=ro" and not "immutable=1" https://github.com/simonw/datasette/blob/eff112498ecc499323c26612d707908831446d25/datasette/database.py#L79

  then: 

  ```bash
  870 B  RUN /bin/sh -c datasette inspect nlrb.db --inspect-file inspect-data.json
  ```

  the `datasette inspect` layer is only the size of the json file!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1838#issuecomment-1271024708
issue_url: https://api.github.com/repos/simonw/datasette/issues/1838
node_id: IC_kwDOBm6k_c5LwkxE
created_at: 2022-10-07T02:19:49Z
updated_at: 2022-10-07T02:19:49Z
author_association: NONE
body:
  Ooh, I didn't even think about links in tables! You're definitely right on the approach to this. It might also be a really good "stupidly simple" plugin for me to try to build myself, which could be fun.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1836#issuecomment-1271100651
issue_url: https://api.github.com/repos/simonw/datasette/issues/1836
node_id: IC_kwDOBm6k_c5Lw3Tr
created_at: 2022-10-07T04:38:14Z
updated_at: 2022-10-07T04:38:14Z
author_association: CONTRIBUTOR
body:
  > yes, and i also think that this is causing the apparent memory problems in #1480. when the container starts up, it will make some operation on the database in `immutable` mode which apparently makes some small change to the db file. if that's so, then the db files will be copied to the read/write layer which counts against cloudrun's memory allocation!
  > 
  > running a test of that now.

  this completely addressed #1480 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1480#issuecomment-1271101072
issue_url: https://api.github.com/repos/simonw/datasette/issues/1480
node_id: IC_kwDOBm6k_c5Lw3aQ
created_at: 2022-10-07T04:39:10Z
updated_at: 2022-10-07T04:39:10Z
author_association: CONTRIBUTOR
body:
  switching from `immutable=1` to `mode=ro` completely addressed this. see https://github.com/simonw/datasette/issues/1836#issuecomment-1271100651 for details.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1836#issuecomment-1271103097
issue_url: https://api.github.com/repos/simonw/datasette/issues/1836
node_id: IC_kwDOBm6k_c5Lw355
created_at: 2022-10-07T04:43:41Z
updated_at: 2022-10-07T04:43:41Z
author_association: CONTRIBUTOR
body:
  @simonw, should i open up a new issue for investigating the differences between "immutable=1" and "mode=ro" and possibly switching to "mode=ro". Or would you like to keep that conversation in this issue?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1838#issuecomment-1271803298
issue_url: https://api.github.com/repos/simonw/datasette/issues/1838
node_id: IC_kwDOBm6k_c5Lzi2i
created_at: 2022-10-07T16:28:41Z
updated_at: 2022-10-07T16:28:41Z
author_association: OWNER
body:
  ... and here's @ocdtrekkie's plugin! https://github.com/ocdtrekkie/datasette-external-links-new-tabs

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1646#issuecomment-1272149176
issue_url: https://api.github.com/repos/simonw/datasette/issues/1646
node_id: IC_kwDOBm6k_c5L03S4
created_at: 2022-10-07T23:06:17Z
updated_at: 2022-10-07T23:06:17Z
author_association: OWNER
body:
  Updated documentation: https://docs.datasette.io/en/latest/settings.html#configuration-directory-mode

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1362#issuecomment-1272228740
issue_url: https://api.github.com/repos/simonw/datasette/issues/1362
node_id: IC_kwDOBm6k_c5L1KuE
created_at: 2022-10-08T05:03:56Z
updated_at: 2022-10-08T05:03:56Z
author_association: OWNER
body:
  Useful example: how Play framework does this https://www.playframework.com/documentation/2.8.1/CspFilter

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1836#issuecomment-1272357976
issue_url: https://api.github.com/repos/simonw/datasette/issues/1836
node_id: IC_kwDOBm6k_c5L1qRY
created_at: 2022-10-08T16:56:51Z
updated_at: 2022-10-08T16:56:51Z
author_association: CONTRIBUTOR
body:
  when you are running from docker, you **always** will want to run as `mode=ro` because the same thing that is causing duplication in the inspect layer will cause duplication in the final container read/write layer when `datasette serve` runs.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1362#issuecomment-1272369712
issue_url: https://api.github.com/repos/simonw/datasette/issues/1362
node_id: IC_kwDOBm6k_c5L1tIw
created_at: 2022-10-08T18:04:31Z
updated_at: 2022-10-08T18:05:05Z
author_association: OWNER
body:
  Also this series: https://scotthelme.co.uk/tag/csp/ - via https://twitter.com/adamchainz/status/1578762884481368065

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/498#issuecomment-1274097701
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/498
node_id: IC_kwDOCGYnMM5L8TAl
created_at: 2022-10-11T05:13:37Z
updated_at: 2022-10-11T05:13:37Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/498?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Base: **96.37**% // Head: **96.37**% // Increases project coverage by **`+0.00%`** :tada:
  > Coverage data is based on head [(`75577e5`)](https://codecov.io/gh/simonw/sqlite-utils/pull/498?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`d792dad`)](https://codecov.io/gh/simonw/sqlite-utils/commit/d792dad1cf5f16525da81b1e162fb71d469995f3?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > Patch coverage: 100.00% of modified lines in pull request are covered.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main     #498   +/-   ##
  =======================================
    Coverage   96.37%   96.37%           
  =======================================
    Files           4        4           
    Lines        2401     2402    +1     
  =======================================
  + Hits         2314     2315    +1     
    Misses         87       87           
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/498?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/498/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.32% <100.00%> (+<0.01%)` | :arrow_up: |

  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  </details>

  [:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/498?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/498#issuecomment-1274153135
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/498
node_id: IC_kwDOCGYnMM5L8giv
created_at: 2022-10-11T06:34:31Z
updated_at: 2022-10-11T06:34:31Z
author_association: CONTRIBUTOR
body:
  nevermind it was because I was running `db[table].transform`. The fts tables would still be there but the triggers would be dropped

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/499#issuecomment-1275248761
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/499
node_id: IC_kwDOCGYnMM5MAsB5
created_at: 2022-10-11T20:38:13Z
updated_at: 2022-10-11T20:38:13Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/499?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Base: **96.37**% // Head: **96.25**% // Decreases project coverage by **`-0.11%`** :warning:
  > Coverage data is based on head [(`a25e838`)](https://codecov.io/gh/simonw/sqlite-utils/pull/499?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`d792dad`)](https://codecov.io/gh/simonw/sqlite-utils/commit/d792dad1cf5f16525da81b1e162fb71d469995f3?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > Patch coverage: 40.00% of modified lines in pull request are covered.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@            Coverage Diff             @@
  ##             main     #499      +/-   ##
  ==========================================
  - Coverage   96.37%   96.25%   -0.12%     
  ==========================================
    Files           4        4              
    Lines        2401     2406       +5     
  ==========================================
  + Hits         2314     2316       +2     
  - Misses         87       90       +3     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/499?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/499/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.09% <40.00%> (-0.23%)` | :arrow_down: |

  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  </details>

  [:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/499?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1831#issuecomment-1276693475
issue_url: https://api.github.com/repos/simonw/datasette/issues/1831
node_id: IC_kwDOBm6k_c5MGMvj
created_at: 2022-10-12T20:23:01Z
updated_at: 2022-10-12T20:23:01Z
author_association: OWNER
body:
  Here are all the places that display that navigation bar at the moment:

  https://github.com/simonw/datasette/blob/b7fec7f9020b79c1fe60cc5a2def86b50eeb5af9/datasette/templates/row.html#L18-L25

  https://github.com/simonw/datasette/blob/b7fec7f9020b79c1fe60cc5a2def86b50eeb5af9/datasette/templates/table.html#L19-L25

  https://github.com/simonw/datasette/blob/b7fec7f9020b79c1fe60cc5a2def86b50eeb5af9/datasette/templates/query.html#L21-L27

  https://github.com/simonw/datasette/blob/b7fec7f9020b79c1fe60cc5a2def86b50eeb5af9/datasette/templates/error.html#L5-L10

  https://github.com/simonw/datasette/blob/b7fec7f9020b79c1fe60cc5a2def86b50eeb5af9/datasette/templates/database.html#L12-L17

  https://github.com/simonw/datasette/blob/b7fec7f9020b79c1fe60cc5a2def86b50eeb5af9/datasette/templates/show_json.html#L7-L12

  https://github.com/simonw/datasette/blob/b7fec7f9020b79c1fe60cc5a2def86b50eeb5af9/datasette/templates/permissions_debug.html#L25-L30

  https://github.com/simonw/datasette/blob/b7fec7f9020b79c1fe60cc5a2def86b50eeb5af9/datasette/templates/permissions_debug.html#L25-L30t

  Interesting to note that failing to include that `super()` call means the nav menu itself will not be rendered:

  https://github.com/simonw/datasette/blob/b7fec7f9020b79c1fe60cc5a2def86b50eeb5af9/datasette/templates/base.html#L20-L49

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1831#issuecomment-1276694152
issue_url: https://api.github.com/repos/simonw/datasette/issues/1831
node_id: IC_kwDOBm6k_c5MGM6I
created_at: 2022-10-12T20:23:44Z
updated_at: 2022-10-12T20:23:44Z
author_association: OWNER
body:
  The fact that this code is inconsistent already (`urls.instance()` v.s. `base_url` for example) suggests that properly refactoring this is well overdue already.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1831#issuecomment-1276696138
issue_url: https://api.github.com/repos/simonw/datasette/issues/1831
node_id: IC_kwDOBm6k_c5MGNZK
created_at: 2022-10-12T20:25:39Z
updated_at: 2022-10-12T20:25:39Z
author_association: OWNER
body:
  Related issue:

  - #1829

  I'd like a code design for this that helps solve both these issues at once.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1831#issuecomment-1276706181
issue_url: https://api.github.com/repos/simonw/datasette/issues/1831
node_id: IC_kwDOBm6k_c5MGP2F
created_at: 2022-10-12T20:34:00Z
updated_at: 2022-10-12T23:43:39Z
author_association: OWNER
body:
  Maybe new template functions: `table_crumbs(table, database)` and `database_crumbs(database)` and `instance_crumbs()` - which know how to both check the permissions and display the `<ul>` full of links.

  Or just `crumbs(table=table, database=database)` where both of those are optional. Maybe accept `actor=` or `request=` too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1831#issuecomment-1276848084
issue_url: https://api.github.com/repos/simonw/datasette/issues/1831
node_id: IC_kwDOBm6k_c5MGyfU
created_at: 2022-10-12T23:39:02Z
updated_at: 2022-10-12T23:39:02Z
author_association: OWNER
body:
  Made some notes on the Jinja documentation while thinking about the best way to do this:
  - https://github.com/simonw/public-notes/issues/4

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1831#issuecomment-1276856836
issue_url: https://api.github.com/repos/simonw/datasette/issues/1831
node_id: IC_kwDOBm6k_c5MG0oE
created_at: 2022-10-12T23:57:28Z
updated_at: 2022-10-12T23:57:28Z
author_association: OWNER
body:
  As part of this I think I want `request` to always be available in the template context, which will remove the need for https://datasette.io/plugins/datasette-template-request

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1831#issuecomment-1276882359
issue_url: https://api.github.com/repos/simonw/datasette/issues/1831
node_id: IC_kwDOBm6k_c5MG623
created_at: 2022-10-13T00:36:09Z
updated_at: 2022-10-13T00:36:19Z
author_association: OWNER
body:
  It's important that, however this works, it supports custom templates changing how the breadcrumbs are displayed. Probably needs a `_crumbs.html` template.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1831#issuecomment-1278175798
issue_url: https://api.github.com/repos/simonw/datasette/issues/1831
node_id: IC_kwDOBm6k_c5ML2o2
created_at: 2022-10-13T21:02:52Z
updated_at: 2022-10-13T21:02:52Z
author_association: OWNER
body:
  This patch to `default_permissions.py` made debugging easier: https://gist.github.com/simonw/daddf022e75a98ea6246ac1e12dc8759

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1831#issuecomment-1278198145
issue_url: https://api.github.com/repos/simonw/datasette/issues/1831
node_id: IC_kwDOBm6k_c5ML8GB
created_at: 2022-10-13T21:28:30Z
updated_at: 2022-10-13T21:28:30Z
author_association: OWNER
body:
  This has turned into a full refactor of how breadcrumbs work.

  I'm using my first ever Jinja macro for this - I import that at the top of `base.html` so that it will be available everywhere else:

  ```html+jinja
  {% import "_crumbs.html" as crumbs with context %}<!DOCTYPE html>
  ```
  The `with context` bit is needed so the macro can see the new `crumb_items()` function that I'm adding to the global template rendering scope.

  Here's the full content of `_crumbs.html`:

  ```html+jinja
  {% macro nav(request, database=None, table=None) -%}
  {% set items=crumb_items(request=request, database=database, table=table) %}
  {% if items %}
    <p class="crumbs">
      {% for item in items %}
        <a href="{{ item.href }}">{{ item.label }}</a>
        {% if not loop.last %}
          /
        {% endif %}
      {% endfor %}
    </p>
  {% endif %}
  {%- endmacro %}
  ```
  This means custom template authors can use their own `_crumbs.html` template to do something different with the breadcrumbs.

  In the actual templates I display breadcrumbs like this:

  ```html+jinja
  {% block crumbs %}
  {{ crumbs.nav(request=request, database=database) }}
  {% endblock %}
  ```
  Pass `database=` to get `home / database_name` - pass `table=` as well to get `home / database_name / table_name` - if you just send `request=` you just get `home`.

  I've also made the default base template show the `home` breadcrumbs - other pages such as `table.html` and `row.html` can then over-ride `{% block crumbs %}` to get different breadcrumbs.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1831#issuecomment-1278198700
issue_url: https://api.github.com/repos/simonw/datasette/issues/1831
node_id: IC_kwDOBm6k_c5ML8Os
created_at: 2022-10-13T21:29:09Z
updated_at: 2022-10-13T21:29:09Z
author_association: OWNER
body:
  I'm going to commit the code now, but then I need to add some extra tests to ensure the breadcrumb permission display logic works correctly.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1829#issuecomment-1278237331
issue_url: https://api.github.com/repos/simonw/datasette/issues/1829
node_id: IC_kwDOBm6k_c5MMFqT
created_at: 2022-10-13T22:17:45Z
updated_at: 2022-10-13T22:19:22Z
author_association: OWNER
body:
  I think `check_visibility` should be changed to optionally accept `permissions=` which is the same list of tuples that can be passed to `ensure_permissions`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1829#issuecomment-1278300241
issue_url: https://api.github.com/repos/simonw/datasette/issues/1829
node_id: IC_kwDOBm6k_c5MMVBR
created_at: 2022-10-14T00:03:52Z
updated_at: 2022-10-14T00:04:28Z
author_association: OWNER
body:
  Here's what I've got so far:
  ```diff
  diff --git a/datasette/app.py b/datasette/app.py
  index 5fa4955c..df9eae49 100644
  --- a/datasette/app.py
  +++ b/datasette/app.py
  @@ -1,5 +1,5 @@
   import asyncio
  -from typing import Sequence, Union, Tuple
  +from typing import Sequence, Union, Tuple, Optional
   import asgi_csrf
   import collections
   import datetime
  @@ -707,7 +707,7 @@ class Datasette:
   
           Raises datasette.Forbidden() if any of the checks fail
           """
  -        assert actor is None or isinstance(actor, dict)
  +        assert actor is None or isinstance(actor, dict), "actor must be None or a dict"
           for permission in permissions:
               if isinstance(permission, str):
                   action = permission
  @@ -732,23 +732,34 @@ class Datasette:
                   else:
                       raise Forbidden(action)
   
  -    async def check_visibility(self, actor, action, resource):
  +    async def check_visibility(
  +        self,
  +        actor: dict,
  +        action: Optional[str] = None,
  +        resource: Optional[str] = None,
  +        permissions: Optional[
  +            Sequence[Union[Tuple[str, Union[str, Tuple[str, str]]], str]]
  +        ] = None,
  +    ):
           """Returns (visible, private) - visible = can you see it, private = can others see it too"""
  -        visible = await self.permission_allowed(
  -            actor,
  -            action,
  -            resource=resource,
  -            default=True,
  -        )
  -        if not visible:
  +        if permissions:
  +            assert (
  +                not action and not resource
  +            ), "Can't use action= or resource= with permissions="
  +        else:
  +            permissions = [(action, resource)]
  +        try:
  +            await self.ensure_permissions(actor, permissions)
  +        except Forbidden:
               return False, False
  -        private = not await self.permission_allowed(
  -            None,
  -            action,
  -            resource=resource,
  -            default=True,
  -        )
  -        return visible, private
  +        # User can see it, but can the anonymous user see it?
  +        try:
  +            await self.ensure_permissions(None, permissions)
  +        except Forbidden:
  +            # It's visible but private
  +            return True, True
  +        # It's visible to everyone
  +        return True, False
   
       async def execute(
           self,
  diff --git a/datasette/views/table.py b/datasette/views/table.py
  index 60c092f9..f73b0957 100644
  --- a/datasette/views/table.py
  +++ b/datasette/views/table.py
  @@ -28,7 +28,7 @@ from datasette.utils import (
       urlsafe_components,
       value_as_boolean,
   )
  -from datasette.utils.asgi import BadRequest, NotFound
  +from datasette.utils.asgi import BadRequest, Forbidden, NotFound
   from datasette.filters import Filters
   from .base import DataView, DatasetteError, ureg
   from .database import QueryView
  @@ -213,18 +213,16 @@ class TableView(DataView):
               raise NotFound(f"Table not found: {table_name}")
   
           # Ensure user has permission to view this table
  -        await self.ds.ensure_permissions(
  +        visible, private = await self.ds.check_visibility(
               request.actor,
  -            [
  +            permissions=[
                   ("view-table", (database_name, table_name)),
                   ("view-database", database_name),
                   "view-instance",
               ],
           )
  -
  -        private = not await self.ds.permission_allowed(
  -            None, "view-table", (database_name, table_name), default=True
  -        )
  +        if not visible:
  +            raise Forbidden("You do not have permission to view this table")
   
           # Handle ?_filter_column and redirect, if present
           redirect_params = filters_should_redirect(request.args)
  ```
  Still needs tests and a documentation update.

  Also this fix is currently only applied on the table page - needs to be applied on database, row and query pages too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1842#issuecomment-1278306180
issue_url: https://api.github.com/repos/simonw/datasette/issues/1842
node_id: IC_kwDOBm6k_c5MMWeE
created_at: 2022-10-14T00:11:46Z
updated_at: 2022-10-24T02:04:52Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1842?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Base: **92.52**% // Head: **92.54**% // Increases project coverage by **`+0.02%`** :tada:
  > Coverage data is based on head [(`3623475`)](https://codecov.io/gh/simonw/datasette/pull/1842?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`79aa0de`)](https://codecov.io/gh/simonw/datasette/commit/79aa0de083d38a9975915d5a4cc68ca6c74fbe3d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > Patch coverage: 100.00% of modified lines in pull request are covered.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@            Coverage Diff             @@
  ##             main    #1842      +/-   ##
  ==========================================
  + Coverage   92.52%   92.54%   +0.02%     
  ==========================================
    Files          35       35              
    Lines        4415     4428      +13     
  ==========================================
  + Hits         4085     4098      +13     
    Misses        330      330              
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1842?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1842/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.28% <100.00%> (+0.05%)` | :arrow_up: |
  | [datasette/views/database.py](https://codecov.io/gh/simonw/datasette/pull/1842/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `95.29% <100.00%> (+0.06%)` | :arrow_up: |
  | [datasette/views/index.py](https://codecov.io/gh/simonw/datasette/pull/1842/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2luZGV4LnB5) | `96.49% <100.00%> (ø)` | |
  | [datasette/views/row.py](https://codecov.io/gh/simonw/datasette/pull/1842/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3Jvdy5weQ==) | `88.70% <100.00%> (+0.37%)` | :arrow_up: |
  | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1842/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `95.20% <100.00%> (+0.01%)` | :arrow_up: |

  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  </details>

  [:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1842?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1843#issuecomment-1278478042
issue_url: https://api.github.com/repos/simonw/datasette/issues/1843
node_id: IC_kwDOBm6k_c5MNAba
created_at: 2022-10-14T04:46:29Z
updated_at: 2022-10-14T04:46:29Z
author_association: OWNER
body:
  I did `pip install psutil` and then ran this in the debugger for one of these errors:

  ```python
  import psutil
  for f in psutil.Process().open_files(): print(f)
  ```
  The output looked like this:

  ```
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpq31d2af1/fixtures.db', fd=11)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpoxdpxj6w/fixtures.db', fd=12)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfd3oyo10/fixtures.dot.db', fd=13)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpezwfu7w8/fixtures.db', fd=14)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpq31d2af1/fixtures.db', fd=15)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpq31d2af1/fixtures.db', fd=16)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpz6e2anqw/fixtures.db', fd=17)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpoxdpxj6w/fixtures.db', fd=18)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpdp4we7hb/fixtures.db', fd=19)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfd3oyo10/fixtures.dot.db', fd=20)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp4ljq_ai0/fixtures.db', fd=21)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpezwfu7w8/fixtures.db', fd=22)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp907xmnzb/fixtures.db', fd=24)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpz6e2anqw/extra database.db', fd=25)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpz6e2anqw/fixtures.db', fd=26)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpzlwn6bqm/fixtures.db', fd=27)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpdp4we7hb/fixtures.db', fd=28)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp42e6vyj_/fixtures.db', fd=29)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp4ljq_ai0/fixtures.db', fd=31)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpw32vwkjq/fixtures.db', fd=32)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp907xmnzb/extra database.db', fd=33)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp907xmnzb/fixtures.db', fd=34)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpw32vwkjq/fixtures.db', fd=35)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpzlwn6bqm/foo-bar.db', fd=36)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpzlwn6bqm/foo.db', fd=37)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpzlwn6bqm/fixtures.db', fd=38)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpcl1edmyv/fixtures.db', fd=39)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp42e6vyj_/fixtures.db', fd=40)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpoxdpxj6w/fixtures.db', fd=41)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp0w5jugqk/fixtures.db', fd=42)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpmb9y9fba/fixtures.db', fd=43)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfwsbx941/fixtures.db', fd=44)
  popenfile(path='/Users/simon/Dropbox/Development/datasette/tests/spatialite.db', fd=45)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp0w5jugqk/fixtures.db', fd=46)
  popenfile(path='/Users/simon/Dropbox/Development/datasette/tests/spatialite.db', fd=47)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpmb9y9fba/fixtures.db', fd=48)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_serve_create0/does_not_exist_yet.db', fd=49)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfwsbx941/data.db', fd=92)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfwsbx941/fixtures.db', fd=93)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfwsbx941/data.db', fd=94)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_serve_duplicate_database_0/db.db', fd=95)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_serve_duplicate_database_0/nested/db.db', fd=99)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_serve_deduplicate_same_da0/db.db', fd=100)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_weird_database_names_test0/test-database (1).sqlite', fd=101)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_weird_database_names_test0/test-database (1).sqlite', fd=102)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_serve_duplicate_database_0/db.db', fd=103)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_serve_duplicate_database_0/nested/db.db', fd=104)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_weird_database_names_data0/database (1).sqlite', fd=105)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_weird_database_names_data0/database (1).sqlite', fd=106)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_weird_database_names_test0/test-database (1).sqlite', fd=107)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_weird_database_names_test0/test-database (1).sqlite', fd=109)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_weird_database_names_data0/database (1).sqlite', fd=111)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_weird_database_names_data0/database (1).sqlite', fd=113)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmppju9w34z/fixtures.db', fd=117)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpky1jamnv/fixtures.db', fd=118)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_0.db', fd=119)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/config-dir0/demo.db', fd=120)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/config-dir0/immutable.db', fd=121)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/config-dir0/k.sqlite', fd=122)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/config-dir0/j.sqlite3', fd=123)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_1.db', fd=124)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmppju9w34z/extra database.db', fd=125)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmppju9w34z/fixtures.db', fd=126)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmppju9w34z/extra database.db', fd=127)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmppju9w34z/fixtures.db', fd=128)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_2.db', fd=129)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_3.db', fd=130)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_4.db', fd=131)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_5.db', fd=132)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_6.db', fd=133)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_7.db', fd=134)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_8.db', fd=135)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_9.db', fd=136)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_0.db', fd=137)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_1.db', fd=138)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_2.db', fd=139)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_3.db', fd=140)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_4.db', fd=141)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_5.db', fd=142)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_6.db', fd=143)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_7.db', fd=144)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_8.db', fd=145)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_9.db', fd=146)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_10.db', fd=147)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpmzu4jbdx/fixtures.db', fd=148)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpskyh32wh/fixtures.db', fd=149)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp966705ec/fixtures.db', fd=150)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_0.db', fd=151)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_1.db', fd=152)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_2.db', fd=153)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_3.db', fd=154)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_4.db', fd=155)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_5.db', fd=156)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_6.db', fd=157)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_7.db', fd=158)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_8.db', fd=159)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_9.db', fd=160)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_10.db', fd=161)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_0.db', fd=162)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_1.db', fd=163)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_2.db', fd=164)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_3.db', fd=165)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_4.db', fd=166)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_5.db', fd=167)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_6.db', fd=168)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_7.db', fd=169)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_8.db', fd=170)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_9.db', fd=171)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpky1jamnv/fixtures.db', fd=172)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpwmewpy8e/fixtures.db', fd=173)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpmzu4jbdx/fixtures.db', fd=174)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpkyn60obe/fixtures.db', fd=175)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpskyh32wh/fixtures.db', fd=176)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpcl1edmyv/fixtures.db', fd=177)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp966705ec/fixtures.db', fd=178)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp4gctb4i6/fixtures.db', fd=179)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpwmewpy8e/fixtures.db', fd=180)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp4gctb4i6/fixtures.db', fd=181)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpkyn60obe/fixtures.db', fd=182)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp4gctb4i6/fixtures.db', fd=183)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmps7p_ee_e/fixtures.db', fd=184)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpf3_6i1d6/fixtures.db', fd=185)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp6_9vzq4o/fixtures.db', fd=187)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpqtubcoah/fixtures.db', fd=188)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmps7p_ee_e/extra database.db', fd=189)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmps7p_ee_e/fixtures.db', fd=190)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp2gauogmy/fixtures.db', fd=191)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpf3_6i1d6/fixtures.db', fd=192)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmprh373nt1/fixtures.db', fd=193)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp6_9vzq4o/fixtures.db', fd=197)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmprh373nt1/extra database.db', fd=198)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpqtubcoah/fixtures.db', fd=211)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmprh373nt1/fixtures.db', fd=212)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp2gauogmy/fixtures.db', fd=215)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpd2tleqni/fixtures.db', fd=216)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpd2tleqni/fixtures.db', fd=217)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmplpdc3wfl/fixtures.db', fd=219)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpdnsscl5v/fixtures.db', fd=222)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmptjayd3pn/fixtures.db', fd=223)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp4u5oo5ne/fixtures.db', fd=224)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp0dgqc4wx/fixtures.db', fd=225)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpdnsscl5v/fixtures.db', fd=226)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpoehh1_tg/fixtures.db', fd=227)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp4u5oo5ne/fixtures.db', fd=247)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp0dgqc4wx/fixtures.db', fd=251)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpoehh1_tg/fixtures.db', fd=253)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmptjayd3pn/fixtures.db', fd=254)
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test-view-names0/fixtures.db', fd=255)
  ```
  Clearly something is bad with the way fixtures work in the tests - a huge number of `fixtures.db` database files are being created and left open!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1843#issuecomment-1278537920
issue_url: https://api.github.com/repos/simonw/datasette/issues/1843
node_id: IC_kwDOBm6k_c5MNPDA
created_at: 2022-10-14T06:19:55Z
updated_at: 2022-10-14T06:20:06Z
author_association: OWNER
body:
  Maybe I need to explicitly close those SQLite connections held by the Datasette instance after this line: https://github.com/simonw/datasette/blob/79aa0de083d38a9975915d5a4cc68ca6c74fbe3d/tests/fixtures.py#L165

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/51#issuecomment-1279224780
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/51
node_id: IC_kwDODFdgUs5MP2vM
created_at: 2022-10-14T16:34:07Z
updated_at: 2022-10-14T16:34:07Z
author_association: NONE
body:
  also, it says that authenticated requests have a much higher "rate limit". Unauthenticated requests only get 60 req/hour ?? seems more like a quota than a "rate limit" (although I guess that is semantic equivalence)

  You would want to use `x-ratelimit-reset`

  ```
  time.sleep(r['x-ratelimit-reset'] + 1 - time.time())
  ```

  But a more complete solution would bring authenticated requests to the other subcommands. I'm surprised only `github-to-sqlite get` is using the `--auth=` CLI flag

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/60#issuecomment-1279249898
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/60
node_id: IC_kwDODEm0Qs5MP83q
created_at: 2022-10-14T16:58:26Z
updated_at: 2022-10-14T16:58:26Z
author_association: NONE
body:
  You could try using `msys2`. I've had better luck running python CLIs within that system on Windows.

  Here is a guide: https://github.com/chapmanjacobd/lb/blob/main/Windows.md#prep

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1844#issuecomment-1279311487
issue_url: https://api.github.com/repos/simonw/datasette/issues/1844
node_id: IC_kwDOBm6k_c5MQL5_
created_at: 2022-10-14T18:06:53Z
updated_at: 2022-10-14T19:33:24Z
author_association: OWNER
body:
  I just spotted some other out-dated screenshots in the `docs/` directory:

  - [x] [advanced_export.png](https://github.com/simonw/datasette/blob/main/docs/advanced_export.png "advanced_export.png")
  - [x] [binary_data.png](https://github.com/simonw/datasette/blob/main/docs/binary_data.png "binary_data.png")
  - [x] [facets.png](https://github.com/simonw/datasette/blob/main/docs/facets.png "facets.png")
  - [x] [full_text_search.png](https://github.com/simonw/datasette/blob/main/docs/full_text_search.png "full_text_search.png")

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1844#issuecomment-1279314400
issue_url: https://api.github.com/repos/simonw/datasette/issues/1844
node_id: IC_kwDOBm6k_c5MQMng
created_at: 2022-10-14T18:10:07Z
updated_at: 2022-10-14T18:13:36Z
author_association: OWNER
body:
  For the advanced export one:

      shot-scraper 'https://register-of-members-interests.datasettes.com/regmem/items?_search=hamper' -s '#export' -p 10

  Produces:

  ![register-of-members-interests-datasettes-com-regmem-items 2](https://user-images.githubusercontent.com/9599/195913614-448557aa-5ec6-4a83-98cf-8837d3117204.png)

  Current image is:

  <img src="https://github.com/simonw/datasette/blob/main/docs/advanced_export.png?raw=true">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1844#issuecomment-1279316670
issue_url: https://api.github.com/repos/simonw/datasette/issues/1844
node_id: IC_kwDOBm6k_c5MQNK-
created_at: 2022-10-14T18:12:39Z
updated_at: 2022-10-14T18:12:39Z
author_association: OWNER
body:
  New screenshot of FTS, from https://register-of-members-interests.datasettes.com/regmem/items?_search=hamper&_sort_desc=date

  <img src="https://raw.githubusercontent.com/simonw/datasette-screenshots/c636f07fa94bd0bfbeea087c245e7212a4190b1e/non-retina/regmem-search.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1844#issuecomment-1279323714
issue_url: https://api.github.com/repos/simonw/datasette/issues/1844
node_id: IC_kwDOBm6k_c5MQO5C
created_at: 2022-10-14T18:21:03Z
updated_at: 2022-10-14T18:21:03Z
author_association: OWNER
body:
  For this image:

  <img src="https://github.com/simonw/datasette/blob/79aa0de083d38a9975915d5a4cc68ca6c74fbe3d/docs/binary_data.png?raw=true">

  https://latest.datasette.io/fixtures/binary_data has an extra row these days:

  <img width="537" alt="image" src="https://user-images.githubusercontent.com/9599/195914820-b3f7d820-e229-4668-bf90-a4f3d0e95e79.png">

  This deletes every row past the first two (first three including the header row):
  ```javascipt
  Array.from(document.querySelectorAll('tr:nth-child(n+3)'), el => el.parentNode.removeChild(el));
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1844#issuecomment-1279325003
issue_url: https://api.github.com/repos/simonw/datasette/issues/1844
node_id: IC_kwDOBm6k_c5MQPNL
created_at: 2022-10-14T18:22:35Z
updated_at: 2022-10-14T18:22:35Z
author_association: OWNER
body:
  So:
  ```
  shot-scraper 'https://latest.datasette.io/fixtures/binary_data' \
    -j "Array.from(document.querySelectorAll('tr:nth-child(n+3)'), el => el.parentNode.removeChild(el));" \
    -s table -p 10
  ```
  ![latest-datasette-io-fixtures-binary_data 1](https://user-images.githubusercontent.com/9599/195915092-be81db43-5672-4375-bd66-4316211b1afc.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1844#issuecomment-1279325685
issue_url: https://api.github.com/repos/simonw/datasette/issues/1844
node_id: IC_kwDOBm6k_c5MQPX1
created_at: 2022-10-14T18:23:22Z
updated_at: 2022-10-14T18:23:22Z
author_association: OWNER
body:
  Here's the new advanced export image:

  <img src="https://github.com/simonw/datasette-screenshots/blob/1fc70adc3f20ee19cadff21653718d5f1a7bcb1f/advanced-export.png?raw=true">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1844#issuecomment-1279334694
issue_url: https://api.github.com/repos/simonw/datasette/issues/1844
node_id: IC_kwDOBm6k_c5MQRkm
created_at: 2022-10-14T18:33:41Z
updated_at: 2022-10-14T18:34:32Z
author_association: OWNER
body:
  I'm going to use this page for the facets screenshot: https://congress-legislators.datasettes.com/legislators/legislator_terms?_facet=type&_facet=party&_facet=state&_facet_size=10

  Trying for this bit:

  <img width="976" alt="image" src="https://user-images.githubusercontent.com/9599/195916927-74636861-c02b-4744-bfac-68e0b6147b48.png">

  Which incorporates `.suggested-facets` but also the first 3 rows and 10 columns of the table, I wonder if I can specify that in a single selector?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1844#issuecomment-1279339124
issue_url: https://api.github.com/repos/simonw/datasette/issues/1844
node_id: IC_kwDOBm6k_c5MQSp0
created_at: 2022-10-14T18:38:22Z
updated_at: 2022-10-14T18:42:58Z
author_association: OWNER
body:
  This seems to get every table cell in that table for the first 3 rows and the columns up to `party`:

      document.querySelectorAll('tr:not(:nth-child(n+4)) td:not(:nth-child(n+10))')

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1844#issuecomment-1279348239
issue_url: https://api.github.com/repos/simonw/datasette/issues/1844
node_id: IC_kwDOBm6k_c5MQU4P
created_at: 2022-10-14T18:49:22Z
updated_at: 2022-10-14T18:49:22Z
author_association: OWNER
body:
  This works:
  ```
  shot-scraper
    'https://congress-legislators.datasettes.com/legislators/legislator_terms?_facet=type&_facet=party&_facet=state&_facet_size=10' \
    -s '.suggested-facets a' \
    --selector-all 'tr:not(tr:nth-child(n+4)) td:not(:nth-child(n+11))' \
    -p 10
  ```
  ![congress-legislators-datasettes-com-legislators-legislator_terms 6](https://user-images.githubusercontent.com/9599/195919422-97616694-3ec0-4e05-afc2-c509275c767c.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1844#issuecomment-1279349314
issue_url: https://api.github.com/repos/simonw/datasette/issues/1844
node_id: IC_kwDOBm6k_c5MQVJC
created_at: 2022-10-14T18:50:42Z
updated_at: 2022-10-14T19:34:37Z
author_association: OWNER
body:
  I'm going to link the documentation screenshots directly to the images in the https://github.com/simonw/datasette-screenshots repository - but I don't want those images to reflect `main` when the documentation may reflect a specific version.

  So I'm going to start tagging releases of `datasette-screenshots` so I can get permanent URLs to those images.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1844#issuecomment-1279382674
issue_url: https://api.github.com/repos/simonw/datasette/issues/1844
node_id: IC_kwDOBm6k_c5MQdSS
created_at: 2022-10-14T19:33:16Z
updated_at: 2022-10-14T19:33:16Z
author_association: OWNER
body:
  That's the last two screenshots:

  <img src="https://raw.githubusercontent.com/simonw/datasette-screenshots/1b56cd4177089b72de9436749a8995921fe8e870/non-retina/binary-data.png">

  <img src="https://raw.githubusercontent.com/simonw/datasette-screenshots/1b56cd4177089b72de9436749a8995921fe8e870/non-retina/faceting-details.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1844#issuecomment-1279392717
issue_url: https://api.github.com/repos/simonw/datasette/issues/1844
node_id: IC_kwDOBm6k_c5MQfvN
created_at: 2022-10-14T19:44:44Z
updated_at: 2022-10-14T19:45:54Z
author_association: OWNER
body:
  OK, the URLs to use in the docs are:

  * https://raw.githubusercontent.com/simonw/datasette-screenshots/0.62/advanced-export.png (retina)
  * https://raw.githubusercontent.com/simonw/datasette-screenshots/0.62/non-retina/regmem-search.png
  * https://raw.githubusercontent.com/simonw/datasette-screenshots/0.62/binary-data.png (retina)
  * https://raw.githubusercontent.com/simonw/datasette-screenshots/0.62/non-retina/faceting-details.png

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1844#issuecomment-1279405429
issue_url: https://api.github.com/repos/simonw/datasette/issues/1844
node_id: IC_kwDOBm6k_c5MQi11
created_at: 2022-10-14T20:00:26Z
updated_at: 2022-10-14T20:00:26Z
author_association: OWNER
body:
  New images are now live on these pages:
  - https://docs.datasette.io/en/latest/csv_export.html
  - https://docs.datasette.io/en/latest/binary_data.html
  - https://docs.datasette.io/en/latest/facets.html
  - https://docs.datasette.io/en/latest/full_text_search.html
  - https://docs.datasette.io/en/latest/changelog.html#v0-23 (was a duplicate of the advanced export image)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1844#issuecomment-1279406134
issue_url: https://api.github.com/repos/simonw/datasette/issues/1844
node_id: IC_kwDOBm6k_c5MQjA2
created_at: 2022-10-14T20:01:13Z
updated_at: 2022-10-14T20:01:13Z
author_association: OWNER
body:
  Here's the YAML I added to https://github.com/simonw/datasette-screenshots/blob/main/shots.yml for this issue:

  ```yaml
  - url: https://register-of-members-interests.datasettes.com/regmem/items?_search=hamper&_sort_desc=date
    height: 585
    width: 960
    output: regmem-search.png
  - url: https://register-of-members-interests.datasettes.com/regmem/items?_search=hamper
    selector: "#export"
    output: advanced-export.png
    padding: 10
  - url: https://congress-legislators.datasettes.com/legislators/legislator_terms?_facet=type&_facet=party&_facet=state&_facet_size=10
    selectors_all:
    - .suggested-facets a
    - tr:not(tr:nth-child(n+4)) td:not(:nth-child(n+11))
    padding: 10
    output: faceting-details.png
  - url: https://latest.datasette.io/fixtures/binary_data
    selector: table
    javascript: |-
      Array.from(
        document.querySelectorAll('tr:nth-child(n+3)'),
        el => el.parentNode.removeChild(el)
      );
    padding: 10
    output: binary-data.png
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1845#issuecomment-1279842912
issue_url: https://api.github.com/repos/simonw/datasette/issues/1845
node_id: IC_kwDOBm6k_c5MSNpg
created_at: 2022-10-15T22:22:58Z
updated_at: 2022-10-15T22:22:58Z
author_association: OWNER
body:
  I think this mechanism could go a long way towards helping here:
  - https://github.com/simonw/datasette/issues/1160

  It was part of a larger idea I was exploring around ensuring Datasette could be used to start interacting with CSV/JSON data out-of-the-box, without needing to first convert that data into SQLite using separate tools.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1845#issuecomment-1279846110
issue_url: https://api.github.com/repos/simonw/datasette/issues/1845
node_id: IC_kwDOBm6k_c5MSObe
created_at: 2022-10-15T22:49:00Z
updated_at: 2022-10-15T22:49:00Z
author_association: NONE
body:
  I think there's probably some core plugins a new user should just "get" maybe. I feel our Sandstorm package has really reasonable defaults, the ability to bring data in for instance. However, you also have to know that's in the hamburger menu, it's not necessarily intuitive where to start.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1845#issuecomment-1279924827
issue_url: https://api.github.com/repos/simonw/datasette/issues/1845
node_id: IC_kwDOBm6k_c5MShpb
created_at: 2022-10-16T08:54:53Z
updated_at: 2022-10-16T08:54:53Z
author_association: NONE
body:
  > It was part of a larger idea I was exploring around ensuring Datasette could be used to start interacting with CSV/JSON data out-of-the-box, without needing to first convert that data into SQLite using separate tools.

  This would be great.  My organization deals with very nested JSON open data and I have been wanting to find a way to hook into datasette so that the analysts do not have to first convert to sqlite first.

  This can kind of be done with datasette-lite. 

  From this random nested JSON API:
  https://api.nobelprize.org/v1/prize.json

  You can use the API of https://flatterer.herokuapp.com to return a multi table sqlite database:

  https://lite.datasette.io/?url=https://flatterer.herokuapp.com/api/convert?output_format=sqlite%26file_url=https://api.nobelprize.org/v1/prize.json

  This is great and fun, but it would be great if there was some plugin mechanism that you could feed a local datasette a nested JSON file directly, possibly hooking into other flattening tools for this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/500#issuecomment-1282778928
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/500
node_id: IC_kwDOCGYnMM5Mdacw
created_at: 2022-10-18T17:44:20Z
updated_at: 2022-10-18T17:44:20Z
author_association: OWNER
body:
  Here's how it works:

  https://github.com/simonw/sqlite-utils/blob/d792dad1cf5f16525da81b1e162fb71d469995f3/sqlite_utils/cli.py#L1847-L1848

  https://github.com/simonw/sqlite-utils/blob/d792dad1cf5f16525da81b1e162fb71d469995f3/sqlite_utils/cli.py#L1082-L1088

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/500#issuecomment-1282779755
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/500
node_id: IC_kwDOCGYnMM5Mdapr
created_at: 2022-10-18T17:45:10Z
updated_at: 2022-10-18T17:45:10Z
author_association: OWNER
body:
  It should go in `sqlite_utils.utils` - documented here: https://sqlite-utils.datasette.io/en/stable/reference.html#sqlite-utils-utils

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/500#issuecomment-1282800547
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/500
node_id: IC_kwDOCGYnMM5Mdfuj
created_at: 2022-10-18T18:02:09Z
updated_at: 2022-10-18T18:02:09Z
author_association: OWNER
body:
  Documentation: https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite-utils-utils-flatten

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/501#issuecomment-1282813168
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/501
node_id: IC_kwDOCGYnMM5Mdizw
created_at: 2022-10-18T18:12:15Z
updated_at: 2022-10-18T18:12:15Z
author_association: OWNER
body:
  Here's the new Tabulate release:
  - https://github.com/astanin/python-tabulate/releases/tag/v0.9.0
  - https://github.com/astanin/python-tabulate/compare/v0.8.10...v0.9.0

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/501#issuecomment-1282817901
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/501
node_id: IC_kwDOCGYnMM5Mdj9t
created_at: 2022-10-18T18:14:35Z
updated_at: 2022-10-18T18:14:35Z
author_association: OWNER
body:
  Now the 3.6 tests fail - because the new release of tabulate dropped support for that Python version (so on Python 3.6 you get an older version): https://github.com/simonw/sqlite-utils/actions/runs/3275842849/jobs/5391181675

  https://github.com/astanin/python-tabulate/blame/20c6370d5da2dae89b305bfb6c7f12a0f8b7236c/pyproject.toml#L22 shows minimum is 3.7 now.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/501#issuecomment-1282819035
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/501
node_id: IC_kwDOCGYnMM5MdkPb
created_at: 2022-10-18T18:15:05Z
updated_at: 2022-10-18T18:15:05Z
author_association: OWNER
body:
  I'm going to skip the cog test on Python 3.6 to address this. The documentation on the website will show the available list of options for 3.7 and higher.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/501#issuecomment-1282830806
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/501
node_id: IC_kwDOCGYnMM5MdnHW
created_at: 2022-10-18T18:23:36Z
updated_at: 2022-10-18T18:23:36Z
author_association: OWNER
body:
  Tests pass now.

  Updated docs:
  - https://sqlite-utils.datasette.io/en/latest/cli.html#table-formatted-output
  - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#query - and many other places on that page

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1847#issuecomment-1288295713
issue_url: https://api.github.com/repos/simonw/datasette/issues/1847
node_id: IC_kwDOBm6k_c5MydUh
created_at: 2022-10-24T01:45:13Z
updated_at: 2022-10-24T01:45:13Z
author_association: OWNER
body:
  Turns out that was a bug I had introduced while working on that test, and it was the reason I was blocked on finishing work on:
  - #1829

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1847#issuecomment-1288296235
issue_url: https://api.github.com/repos/simonw/datasette/issues/1847
node_id: IC_kwDOBm6k_c5Mydcr
created_at: 2022-10-24T01:45:56Z
updated_at: 2022-10-24T01:45:56Z
author_association: OWNER
body:
  This bug here: https://github.com/simonw/datasette/blob/85d5d2762c13d2b5a8bd9c5ec81c77fe6577121f/tests/test_permissions.py#L485

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1829#issuecomment-1288308945
issue_url: https://api.github.com/repos/simonw/datasette/issues/1829
node_id: IC_kwDOBm6k_c5MygjR
created_at: 2022-10-24T02:07:50Z
updated_at: 2022-10-24T02:07:50Z
author_association: OWNER
body:
  Useful test: if you sign in as root to https://latest.datasette.io/_internal/columns/_internal,columns,database_name you can see there's no padlock icon on that page or on https://latest.datasette.io/_internal/columns - fixing this bug should fix that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1842#issuecomment-1288311852
issue_url: https://api.github.com/repos/simonw/datasette/issues/1842
node_id: IC_kwDOBm6k_c5MyhQs
created_at: 2022-10-24T02:11:12Z
updated_at: 2022-10-24T02:11:12Z
author_association: OWNER
body:
  I'm going to construct a `metadata.yml` which makes various databases and tables visible or invisible, then browse them using the root user.

  `block-instance.yml`:

  ```yaml
  allow:
    id: root
  ```
  `block-database.yml`:
  ```yaml
  databases:
    fixtures:
      allow:
        id: root
  ```
  `block-table.yml`:
  ```yaml
  databases:
    fixtures:
      tables:
        searchable:
          allow:
            id: root
  ```
  `block-query.yml`:
  ```yaml
  databases:
    fixtures:
      queries:
        two:
          sql: select 1 + 1
          allow:
            id: root
  ```
  https://gist.github.com/simonw/2d007ebe43de46d44499c77a2a291756 - checkout that Gist to get all four.

  I manually tested all four scenarios with root and non-root users and confirmed that they worked correctly and padlocks were shown in the right places.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1829#issuecomment-1288320411
issue_url: https://api.github.com/repos/simonw/datasette/issues/1829
node_id: IC_kwDOBm6k_c5MyjWb
created_at: 2022-10-24T02:21:19Z
updated_at: 2022-10-24T02:21:19Z
author_association: OWNER
body:
  Updated docs for `check_visibility()`: https://docs.datasette.io/en/latest/internals.html#await-check-visibility-actor-action-none-resource-none-permissions-none

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1829#issuecomment-1288321630
issue_url: https://api.github.com/repos/simonw/datasette/issues/1829
node_id: IC_kwDOBm6k_c5Myjpe
created_at: 2022-10-24T02:22:49Z
updated_at: 2022-10-24T02:23:46Z
author_association: OWNER
body:
  Visit https://latest.datasette.io/login-as-root and then:

  https://latest.datasette.io/

  <img width="719" alt="image" src="https://user-images.githubusercontent.com/9599/197436195-5594f1fd-4509-46c5-bd2b-dd7f5f83c004.png">

  https://latest.datasette.io/_internal/columns

  <img width="813" alt="image" src="https://user-images.githubusercontent.com/9599/197436239-df57fa3a-216c-4de6-b77c-255824ce63b6.png">

  https://latest.datasette.io/_internal/columns/_internal,columns,cid

  <img width="735" alt="image" src="https://user-images.githubusercontent.com/9599/197436264-e1cc5570-86b6-4a4b-8cc1-9df0f67a8275.png">

  https://latest.datasette.io/_internal/from_hook

  <img width="443" alt="image" src="https://user-images.githubusercontent.com/9599/197436372-17380539-ce0c-48ab-bb78-a058ad14ba5a.png">

  That's all as it should be.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1848#issuecomment-1288327467
issue_url: https://api.github.com/repos/simonw/datasette/issues/1848
node_id: IC_kwDOBm6k_c5MylEr
created_at: 2022-10-24T02:30:48Z
updated_at: 2022-10-24T02:31:04Z
author_association: OWNER
body:
  Here's the code at fault: https://github.com/simonw/datasette/blob/78dad236df730212aa7172f885fd8ec575f0d3ad/datasette/views/database.py#L67-L116

  Those checks aren't doing the new cascading permissions thing added in #1829 which means they can't tell that an anonymous user would not be able to se those tbles and queries and views.

  Should do something like this instead:

  ```python
  view_visible, view_private = await self.ds.check_visibility(
      request.actor,
      permissions=[
          ("view-table", (database, view_name)),
          ("view-database", database),
          "view-instance",
      ],
  )
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1848#issuecomment-1288330238
issue_url: https://api.github.com/repos/simonw/datasette/issues/1848
node_id: IC_kwDOBm6k_c5Mylv-
created_at: 2022-10-24T02:34:41Z
updated_at: 2022-10-24T02:34:41Z
author_association: OWNER
body:
  Tested my fix with this `metadata.yml`:
  ```yaml
  databases:
    fixtures:
     allow:
       id: root
     tables:
       123_starts_with_digits:
         allow: true
  ```
  Signed in as root I saw this - showing that the `123_starts_with_digits` table is public:

  <img width="730" alt="image" src="https://user-images.githubusercontent.com/9599/197437398-7397ce64-c4f5-41b9-93ec-f7e3623e39ad.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1848#issuecomment-1288340476
issue_url: https://api.github.com/repos/simonw/datasette/issues/1848
node_id: IC_kwDOBm6k_c5MyoP8
created_at: 2022-10-24T02:50:29Z
updated_at: 2022-10-24T02:50:29Z
author_association: OWNER
body:
  https://latest.datasette.io/_internal now looks like this:

  <img width="771" alt="image" src="https://user-images.githubusercontent.com/9599/197439086-25d02a4f-3831-42bc-8097-c487290d7c32.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1849#issuecomment-1288384098
issue_url: https://api.github.com/repos/simonw/datasette/issues/1849
node_id: IC_kwDOBm6k_c5Myy5i
created_at: 2022-10-24T04:03:09Z
updated_at: 2022-10-24T04:03:09Z
author_association: OWNER
body:
  Looks like the new breadcrumbs code can't handle the case where `request` is `None`.

  Need a test that demonstrates this too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1850#issuecomment-1289696171
issue_url: https://api.github.com/repos/simonw/datasette/issues/1850
node_id: IC_kwDOBm6k_c5M3zOr
created_at: 2022-10-24T22:15:57Z
updated_at: 2022-10-24T22:15:57Z
author_association: OWNER
body:
  I'm going to treat this as a bit of a research spike, at least until I like the direction it is going enough to commit to it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1850#issuecomment-1289702146
issue_url: https://api.github.com/repos/simonw/datasette/issues/1850
node_id: IC_kwDOBm6k_c5M30sC
created_at: 2022-10-24T22:19:04Z
updated_at: 2022-10-24T22:19:04Z
author_association: OWNER
body:
  This is going to need a whole bunch of new permissions.

  To review: the existing set of permissions are listed here: https://docs.datasette.io/en/0.62/authentication.html#built-in-permissions

  - `view-instance`
  - `view-database`
  - `view-database-download`
  - `view-table`
  - `view-query`
  - `execute-sql`
  - `permissions-debug`
  - `debug-menu`

  I'm going to reuse database terminology for the new permissions. So first draft of those is:

  - `insert-row`
  - `update-row`
  - `delete-row`
  - `create-table`
  - `drop-table`
  - `alter-table`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1850#issuecomment-1289703432
issue_url: https://api.github.com/repos/simonw/datasette/issues/1850
node_id: IC_kwDOBm6k_c5M31AI
created_at: 2022-10-24T22:19:48Z
updated_at: 2022-10-24T22:19:48Z
author_association: OWNER
body:
  It may turn out that it makes sense to also add a UI for these actions as part of this project. That's still to be determined.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1850#issuecomment-1289706439
issue_url: https://api.github.com/repos/simonw/datasette/issues/1850
node_id: IC_kwDOBm6k_c5M31vH
created_at: 2022-10-24T22:22:17Z
updated_at: 2022-10-24T22:22:17Z
author_association: OWNER
body:
  API authentication will be via `Authorization: Bearer XXX` request headers.

  I'm inclined to add a default token mechanism to Datasette based on tokens that are signed with the `DATASETTE_SECRET`. Maybe the root user can access `/-/create-token` which provides a UI for generating a time-limited signed token? Could also have a `datasette create-token` command for creating such tokens at the command-line.

  Plugins can then define alternative ways of creating tokens, such as the existing https://datasette.io/plugins/datasette-auth-tokens plugin.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1850#issuecomment-1289707357
issue_url: https://api.github.com/repos/simonw/datasette/issues/1850
node_id: IC_kwDOBm6k_c5M319d
created_at: 2022-10-24T22:23:12Z
updated_at: 2022-10-24T22:23:12Z
author_association: OWNER
body:
  Here's the implementation of `datasette-auth-tokens`: https://github.com/simonw/datasette-auth-tokens/blob/main/datasette_auth_tokens/__init__.py

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1851#issuecomment-1289712350
issue_url: https://api.github.com/repos/simonw/datasette/issues/1851
node_id: IC_kwDOBm6k_c5M33Le
created_at: 2022-10-24T22:28:39Z
updated_at: 2022-10-27T23:18:48Z
author_association: OWNER
body:
  API design: (**UPDATE: this was [later changed to POST /db/table/-/insert](https://github.com/simonw/datasette/issues/1851#issuecomment-1294224185))

  ```
  POST /db/table
  Authorization: Bearer xxx
  Content-Type: application/json
  {
    "row": {
      "id": 1,
      "name": "New record"
    }
  }
  ```
  Returns:
  ```
  201 Created
  {
    "row": {
      "id": 1,
      "name": "New record"
    }
  }
  ```
  You can omit optional fields in the input, including the ID field. The returned object will always include all fields - and will even include `rowid` if your object doesn't have a primary key of its own.

  I decided to use `"row"` as the key in both request and response, to preserve space for other future keys - one that tells you that the table has been created, for example.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1851#issuecomment-1289713513
issue_url: https://api.github.com/repos/simonw/datasette/issues/1851
node_id: IC_kwDOBm6k_c5M33dp
created_at: 2022-10-24T22:29:58Z
updated_at: 2022-10-24T22:30:15Z
author_association: OWNER
body:
  Interesting open question: how should validation errors (if any) be returned?

  The two forms of validation I can think of at first are:

  - Missing keys which are marked as `not null` in the schema
  - Keys that do not match to existing columns (if you didn't pass `"alter": true`, an option I am considering)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1852#issuecomment-1289718660
issue_url: https://api.github.com/repos/simonw/datasette/issues/1852
node_id: IC_kwDOBm6k_c5M34uE
created_at: 2022-10-24T22:35:01Z
updated_at: 2022-10-24T22:35:01Z
author_association: OWNER
body:
  Maybe these tokens can be restricted to specific databases and tables when they are first created?

  Since they're signed tokens, I could bundle a bunch of extra stuff in them - this token is allowed to do these permissions against these tables/rows for example.

  General wisdom seems to be that 8KB is a sensible maximum length for this kind of token, which is easily long enough to fit in a bunch of database / table / permissions.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1852#issuecomment-1289733483
issue_url: https://api.github.com/repos/simonw/datasette/issues/1852
node_id: IC_kwDOBm6k_c5M38Vr
created_at: 2022-10-24T22:54:37Z
updated_at: 2022-10-24T23:12:10Z
author_association: OWNER
body:
  Token design concept:
  ```json
  {
      "t": {
          "a": ["ir", "ur", "dr"],
          "d": {
              "fixtures": ["ir", "ur", "dr"]
          },
          "t": {
              "fixtures": {
                  "searchable": ["ir"]
              }
          }
      }
  }
  ```
  That JSON would be minified and signed.

  Minified version of the above looks like this (101 characters):

  `{"t":{"a":["ir","ur","dr"],"d":{"fixtures":["ir","ur","dr"]},"t":{"fixtures":{"searchable":["ir"]}}}}`

  The `"t"` key shows this is a token that as a default API key.

  `"a"` means "all" - these are permissions that have been granted on all tables and databases.

  `"d"` means "databases" - this is a way to set permissions for all tables in a specific database.

  `"t"` means "tables" - this lets you set permissions at a finely grained table level.

  Then the permissions themselves are two character codes which are shortened versions - so:

  - `ir` = `insert-row`
  - `ur` = `update-row`
  - `dr` = `delete-row`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1851#issuecomment-1289752130
issue_url: https://api.github.com/repos/simonw/datasette/issues/1851
node_id: IC_kwDOBm6k_c5M4A5C
created_at: 2022-10-24T23:07:30Z
updated_at: 2022-10-24T23:07:30Z
author_association: NONE
body:
  How are you tying the bearer token to identity? I'm excited to see this feature, and since Sandstorm controls API access using the same header, it also will transparently support the API documentation here, but we strip the bearer before the request reaches the app (replacing it with our existing auth headers, of course).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1852#issuecomment-1289766513
issue_url: https://api.github.com/repos/simonw/datasette/issues/1852
node_id: IC_kwDOBm6k_c5M4EZx
created_at: 2022-10-24T23:16:00Z
updated_at: 2022-10-24T23:16:00Z
author_association: OWNER
body:
  Here's what that example looks like signed:
  ```python
  from datasette.app import Datasette
  ds = Datasette()
  ds.sign('{"t":{"a":["ir","ur","dr"],"d":{"fixtures":["ir","ur","dr"]},"t":{"fixtures":{"searchable":["ir"]}}}}')
  ```
  ```
  .eJxTqo5RKolRsgJSiUAqOkYpsyhGSSdGqRRCpQCpWBANUZOWWVFSWpRajFNprQ7cPCS1QF5xamJRckZiUk4qQm9sLRAoAQCC8yph.O0Gaej6-VOLbbtPq7xU6T77jEO0
  ```
  That's 129 characters.

  Note that Datasette doesn't have its own mechanism for signing things for a specific duration yet: https://docs.datasette.io/en/stable/internals.html#sign-value-namespace-default

  So I'll need to add a `"e": 1666739744` field with the UTC timestamp at which the token should expire.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1852#issuecomment-1289773634
issue_url: https://api.github.com/repos/simonw/datasette/issues/1852
node_id: IC_kwDOBm6k_c5M4GJC
created_at: 2022-10-24T23:25:06Z
updated_at: 2022-10-24T23:25:06Z
author_association: OWNER
body:
  If you start Datasette without providing a `DATASETTE_SECRET` environment variable of `--secret` option it creates a random signing secret that only lasts for the lifetime of the server.

  This means any signed API tokens you create will stop working if the server restarts.

  I think the `/-/create-token` UI should know when this happens and show a warning message about it, to avoid confusion.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1852#issuecomment-1289774183
issue_url: https://api.github.com/repos/simonw/datasette/issues/1852
node_id: IC_kwDOBm6k_c5M4GRn
created_at: 2022-10-24T23:25:52Z
updated_at: 2022-10-24T23:25:52Z
author_association: OWNER
body:
  ... also, maybe there should be a UI (perhaps on that page) for resetting the Datasette secret? Useful for emergency invalidation of all tokens.

  No, I'm not going to build that unless someone asks for it. Restarting the server with a fresh secret should be easy enough.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1852#issuecomment-1289775162
issue_url: https://api.github.com/repos/simonw/datasette/issues/1852
node_id: IC_kwDOBm6k_c5M4Gg6
created_at: 2022-10-24T23:27:00Z
updated_at: 2022-10-24T23:27:00Z
author_association: OWNER
body:
  Might be neat for API tokens to be signed with an additional secret than can be rotated independently of `DATASETTE_SECRET` itself, in order to invalidate all tokens without needing to invalidate logged in users too.

  But again, I don't want to implement something like that until I see an actual need for it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1852#issuecomment-1289776707
issue_url: https://api.github.com/repos/simonw/datasette/issues/1852
node_id: IC_kwDOBm6k_c5M4G5D
created_at: 2022-10-24T23:29:03Z
updated_at: 2022-10-24T23:29:03Z
author_association: OWNER
body:
  I'm going to implement the first version of this token mechanism using permissions that exist already. Right now that's:

  https://docs.datasette.io/en/0.62/authentication.html#built-in-permissions

  Here are the shortcuts I'll use for them:

  - `view-instance` - `vi`
  - `view-database` - `vd`
  - `view-database-download` - `vdd`
  - `view-table` - `vt`
  - `view-query` - `vq`
  - `execute-sql` - `es`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1851#issuecomment-1289865317
issue_url: https://api.github.com/repos/simonw/datasette/issues/1851
node_id: IC_kwDOBm6k_c5M4chl
created_at: 2022-10-25T01:42:47Z
updated_at: 2022-10-25T01:42:47Z
author_association: OWNER
body:
  This is going to tie into Datasette's existing permissions mechanism, so plugins will be able to define their own custom mechanisms for tokens to be attached to a specific identity: https://docs.datasette.io/en/stable/authentication.html

  There's only one plugin for API tokens at the moment, which is this one: https://datasette.io/plugins/datasette-auth-tokens

  I'm actually planning on adding another, default token mechanism to Datasette itself as part of this work:

  - #1852

  It may well be that `datasette-sandstorm-support` needs to add something custom here too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1851#issuecomment-1290615599
issue_url: https://api.github.com/repos/simonw/datasette/issues/1851
node_id: IC_kwDOBm6k_c5M7Tsv
created_at: 2022-10-25T14:05:12Z
updated_at: 2022-10-25T14:05:12Z
author_association: CONTRIBUTOR
body:
  This could use a new plugin hook, too. I don't want to complicate your life too much, but for things like GIS, I'd want a way to turn regular JSON into SpatiaLite geometries or combine X/Y coordinates into point geometries and such. Happy to help however I can.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1853#issuecomment-1290995178
issue_url: https://api.github.com/repos/simonw/datasette/issues/1853
node_id: IC_kwDOBm6k_c5M8wXq
created_at: 2022-10-25T18:46:33Z
updated_at: 2022-10-25T18:46:33Z
author_association: OWNER
body:
  I ran a very crude benchmark on my laptop using Locust (against the official macOS packages from www.python.org for Python 3.10 and Python 3.11) and saw a substantial speed increase:

  533.89 requests/second on 3.11
  413.56 requests/second on 3.10

  That was from running Locust against this `locustfile.py`:
  ```python
  from locust import HttpUser, task

  class CounterOne(HttpUser):
      @task
      def hello(self):
          self.client.get("/-/static/app.css")
  ```
  Using:

      locust --headless --users 4 --spawn-rate 4 -H http://127.0.0.1:8001

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1853#issuecomment-1291006149
issue_url: https://api.github.com/repos/simonw/datasette/issues/1853
node_id: IC_kwDOBm6k_c5M8zDF
created_at: 2022-10-25T18:57:33Z
updated_at: 2022-10-25T18:57:33Z
author_association: OWNER
body:
  Ran the upgrade on the Datasette Cloud image first, works fine there.

  https://simon.datasette.cloud/-/versions shows me:

  ```
  {
      "python": {
          "version": "3.11.0",
          "full": "3.11.0 (main, Oct 25 2022, 05:00:36) [GCC 10.2.1 20210110]"
      },
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1853#issuecomment-1291009987
issue_url: https://api.github.com/repos/simonw/datasette/issues/1853
node_id: IC_kwDOBm6k_c5M8z_D
created_at: 2022-10-25T19:01:23Z
updated_at: 2022-10-25T19:01:23Z
author_association: OWNER
body:
  Also tested by running this locally:

      datasette publish cloudrun fixtures.db --service issue-1853

  https://issue-1853-j7hipcg4aq-uc.a.run.app/-/versions now shows Python 3.11.0.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1853#issuecomment-1291012637
issue_url: https://api.github.com/repos/simonw/datasette/issues/1853
node_id: IC_kwDOBm6k_c5M80od
created_at: 2022-10-25T19:04:03Z
updated_at: 2022-10-25T19:04:09Z
author_association: OWNER
body:
  And tested `datasette package` like this:

  ```
  datasette package fixtures.db -t datasette-package-python-upgrade-3-11
  ```
  Then:
  ```
  docker run -p 8081:8001 datasette-package-python-upgrade-3-11
  ```
  And tested it like this:
  ```
  curl http://localhost:8081/-/versions.json | jq
  ```
  Output:
  ```
  {
    "python": {
      "version": "3.11.0",
      "full": "3.11.0 (main, Oct 25 2022, 05:00:36) [GCC 10.2.1 20210110]"
    },
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1853#issuecomment-1291023926
issue_url: https://api.github.com/repos/simonw/datasette/issues/1853
node_id: IC_kwDOBm6k_c5M83Y2
created_at: 2022-10-25T19:15:49Z
updated_at: 2022-10-25T19:15:49Z
author_association: OWNER
body:
  This broke the deploy of `https://latest.datasette.io/` - because it tries to install `pysqlite3-binary` which doesn't have a 3.11 release yet:

  https://github.com/simonw/datasette/blob/2e9751672d4fe329b3c359d5b7b1992283185820/.github/workflows/deploy-latest.yml#L77

  I started using that for the `latest.datasette.io` demo in https://github.com/simonw/datasette/commit/a970276b9999687b96c5e11ea1c817d814f5d267 because I wanted a version of SQLite that supported generated columns.

  Those were added in [SQLite 3.31.0](https://www.sqlite.org/changes.html#version_3_31_0) - and the SQLite version in the new base image is 3.34.1 - so I don't actually need `pysqlite3-binary` any more.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/502#issuecomment-1291029761
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/502
node_id: IC_kwDOCGYnMM5M840B
created_at: 2022-10-25T19:21:44Z
updated_at: 2022-10-25T19:21:44Z
author_association: OWNER
body:
  Replicated locally using a fresh virtual environment with Python 3.11 and:

       pytest -k test_query_invalid_function

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1854#issuecomment-1291045997
issue_url: https://api.github.com/repos/simonw/datasette/issues/1854
node_id: IC_kwDOBm6k_c5M88xt
created_at: 2022-10-25T19:38:28Z
updated_at: 2022-10-25T19:38:28Z
author_association: OWNER
body:
  Also:
  ```
      @pytest.mark.serial
      @pytest.mark.skipif(
          not hasattr(socket, "AF_UNIX"), reason="Requires socket.AF_UNIX support"
      )
      def test_serve_unix_domain_socket(ds_unix_domain_socket_server):
          _, uds = ds_unix_domain_socket_server
          transport = httpx.HTTPTransport(uds=uds)
          client = httpx.Client(transport=transport)
  >       response = client.get("http://localhost/_memory.json")

  /home/runner/work/datasette/datasette/tests/test_cli_serve_server.py:35: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
  /opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/httpx/_client.py:1039: in get
      return self.request(
  /opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/httpx/_client.py:815: in request
      return self.send(request, auth=auth, follow_redirects=follow_redirects)
  /opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/httpx/_client.py:902: in send
      response = self._send_handling_auth(
  /opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/httpx/_client.py:930: in _send_handling_auth
      response = self._send_handling_redirects(
  /opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/httpx/_client.py:967: in _send_handling_redirects
      response = self._send_single_request(request)
  /opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/httpx/_client.py:1003: in _send_single_request
      response = transport.handle_request(request)
  /opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/httpx/_transports/default.py:217: in handle_request
      with map_httpcore_exceptions():
  /opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/contextlib.py:153: in __exit__
      self.gen.throw(typ, value, traceback)
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

      @contextlib.contextmanager
      def map_httpcore_exceptions() -> typing.Iterator[None]:
          try:
              yield
          except Exception as exc:  # noqa: PIE-786
              mapped_exc = None
      
              for from_exc, to_exc in HTTPCORE_EXC_MAP.items():
                  if not isinstance(exc, from_exc):
                      continue
                  # We want to map to the most specific exception we can find.
                  # Eg if `exc` is an `httpcore.ReadTimeout`, we want to map to
                  # `httpx.ReadTimeout`, not just `httpx.TimeoutException`.
                  if mapped_exc is None or issubclass(to_exc, mapped_exc):
                      mapped_exc = to_exc
      
              if mapped_exc is None:  # pragma: nocover
                  raise
      
              message = str(exc)
  >           raise mapped_exc(message) from exc
  E           httpx.ConnectError: [Errno 2] No such file or directory
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1854#issuecomment-1291046958
issue_url: https://api.github.com/repos/simonw/datasette/issues/1854
node_id: IC_kwDOBm6k_c5M89Au
created_at: 2022-10-25T19:39:22Z
updated_at: 2022-10-25T19:39:22Z
author_association: OWNER
body:
  Here's the code that starts those various servers: https://github.com/simonw/datasette/blob/613ad05c095f92653221db267ef53d54d00cdfbb/tests/conftest.py#L104-L177

  I don't like those `time.sleep(1.5)` lines much - I'm going to try polling for readiness instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1854#issuecomment-1291047214
issue_url: https://api.github.com/repos/simonw/datasette/issues/1854
node_id: IC_kwDOBm6k_c5M89Eu
created_at: 2022-10-25T19:39:36Z
updated_at: 2022-10-25T19:39:48Z
author_association: OWNER
body:
  This pattern should work (for the http server at least):
  ```python
      # Loop until port 8041 serves traffic
      while True:
          try:
              httpx.get("http://localhost:8041/")
              break
          except httpx.ConnectError:
              time.sleep(0.1)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291076031
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/503
node_id: IC_kwDOCGYnMM5M9EG_
created_at: 2022-10-25T20:06:28Z
updated_at: 2022-10-25T20:06:28Z
author_association: OWNER
body:
  This is the failing test: https://github.com/simonw/sqlite-utils/blob/7b2d1c0ffd0b874e280292b926f328a61cb31e2c/tests/test_recreate.py#L21-L32

  I'm going to try a different way of creating the temporary file: https://docs.pytest.org/en/7.1.x/how-to/tmp_path.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291083188
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/503
node_id: IC_kwDOCGYnMM5M9F20
created_at: 2022-10-25T20:12:52Z
updated_at: 2022-10-25T20:12:52Z
author_association: OWNER
body:
  Failed again, but just noticed this: https://github.com/simonw/sqlite-utils/actions/runs/3323932266/jobs/5494890223

  ```
  >       Database(filepath, recreate=True)["t2"].insert({"foo": "bar"})

  tests\test_recreate.py:31: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

  self = <[AttributeError("'Database' object has no attribute 'conn'") raised in repr()] Database object at 0x29fc125aa90>
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291088108
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/503
node_id: IC_kwDOCGYnMM5M9HDs
created_at: 2022-10-25T20:17:36Z
updated_at: 2022-10-25T20:17:36Z
author_association: OWNER
body:
  Now `mypy` is failing:

  ```
  sqlite_utils/db.py:474: error: Item "None" of "Optional[Any]" has no attribute "execute"
  sqlite_utils/db.py:476: error: Item "None" of "Optional[Any]" has no attribute "execute"
  sqlite_utils/db.py:486: error: Item "None" of "Optional[Any]" has no attribute "executescript"
  sqlite_utils/db.py:603: error: Item "None" of "Optional[Any]" has no attribute "__enter__"
  sqlite_utils/db.py:603: error: Item "None" of "Optional[Any]" has no attribute "__exit__"
  sqlite_utils/db.py:604: error: Item "None" of "Optional[Any]" has no attribute "execute"
  sqlite_utils/db.py:607: error: Item "None" of "Optional[Any]" has no attribute "execute"
  sqlite_utils/db.py:1082: error: Item "None" of "Optional[Any]" has no attribute "__enter__"
  sqlite_utils/db.py:1082: error: Item "None" of "Optional[Any]" has no attribute "__exit__"
  sqlite_utils/db.py:1083: error: Item "None" of "Optional[Any]" has no attribute "cursor"
  sqlite_utils/db.py:1155: error: Item "None" of "Optional[Any]" has no attribute "enable_load_extension"
  sqlite_utils/db.py:1156: error: Item "None" of "Optional[Any]" has no attribute "load_extension"
  Found 12 errors in 1 file (checked 51 source files)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291093581
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/503
node_id: IC_kwDOCGYnMM5M9IZN
created_at: 2022-10-25T20:23:00Z
updated_at: 2022-10-25T20:23:00Z
author_association: OWNER
body:
  I'm not hugely happy with my fix there:

  https://github.com/simonw/sqlite-utils/blob/c5d7ec1dd71fa1dce829bc8bb82b639018befd63/sqlite_utils/db.py#L321-L328

  The problem here was that in the case where the `os.remove()` failed the `self.conn` property was NOT being set to a valid connection - which caused `__repr__` to fail later on.

  So now I catch the `os.remove()` error, set `self.conn` to a memory connection, then raise the error again.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291103021
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/503
node_id: IC_kwDOCGYnMM5M9Kst
created_at: 2022-10-25T20:32:01Z
updated_at: 2022-10-25T20:32:01Z
author_association: OWNER
body:
  This test reliably fails on Windows with Python 3.11.

  I'm going to skip the test for the moment to get back to green CI... but I'll leave this issue open. This is definitely concerning, I just don't have the right local environment to solve this at the moment.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291115986
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/503
node_id: IC_kwDOCGYnMM5M9N3S
created_at: 2022-10-25T20:39:24Z
updated_at: 2022-10-25T20:39:24Z
author_association: OWNER
body:
  Used `psutil` to confirm that closing a SQLite connection closes the underlying file: https://til.simonwillison.net/python/too-many-open-files-psutil 
  ```pycon
  >>> import psutil
  >>> import sqlite3
  >>> for f in psutil.Process().open_files(): print(f)
  ... 
  >>> sqlite3.connect("/tmp/blah.db")
  <sqlite3.Connection object at 0x1007264d0>
  >>> conn = _
  >>> for f in psutil.Process().open_files(): print(f)
  ... 
  popenfile(path='/private/tmp/blah.db', fd=3)
  >>> conn.close()
  >>> for f in psutil.Process().open_files(): print(f)
  ... 
  >>> 
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/504#issuecomment-1291136971
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/504
node_id: IC_kwDOCGYnMM5M9S_L
created_at: 2022-10-25T21:00:29Z
updated_at: 2022-10-25T21:00:29Z
author_association: OWNER
body:
  Documentation: https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Database.close

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/497#issuecomment-1291146850
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/497
node_id: IC_kwDOCGYnMM5M9VZi
created_at: 2022-10-25T21:09:28Z
updated_at: 2022-10-25T21:09:28Z
author_association: OWNER
body:
  Yeah, `table.columns` and `table.columns_dict`  are meant to handle this: https://sqlite-utils.datasette.io/en/stable/python-api.html#columns

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/495#issuecomment-1291149509
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/495
node_id: IC_kwDOCGYnMM5M9WDF
created_at: 2022-10-25T21:12:11Z
updated_at: 2022-10-25T21:12:11Z
author_association: OWNER
body:
  This makes sense to me. There are other places in the codebase where JSON is automatically stringified: https://github.com/simonw/sqlite-utils/blob/c7e4308e6f49d929704163531632e558f9646e4a/sqlite_utils/db.py#L2759-L2766

  I don't see why the return value from a convert function shouldn't do the same thing.

  Since this will result in previous errors working, I don't think it warrants a major version bump either.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/495#issuecomment-1291152433
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/495
node_id: IC_kwDOCGYnMM5M9Wwx
created_at: 2022-10-25T21:14:54Z
updated_at: 2022-10-25T21:14:54Z
author_association: OWNER
body:
  There is a case where the function can return a dictionary at the moment: `multi=True`

  ```python
  table.convert(
      "title", lambda v: {"upper": v.upper(), "lower": v.lower()}, multi=True
  )
  ```
  But I think this change is still compatible with that. if you don't use `multi=True` then the return value will be stringified. If you DO use `multi=True` then something like this could work:
  ```python
  table.convert(
      "title", lambda v: {"upper": {"str": v.upper()}, "lower": {"str": v.lower()}}, multi=True
  )
  ```
  This would result in a `upper` and `lower` column, each containing the JSON string `{"str": "UPPERCASE"}`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/495#issuecomment-1291159549
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/495
node_id: IC_kwDOCGYnMM5M9Yf9
created_at: 2022-10-25T21:23:01Z
updated_at: 2022-10-25T21:23:01Z
author_association: OWNER
body:
  I've decided not to explicitly document this, since it's consistent with how other parts of the library work already.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/493#issuecomment-1291166273
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/493
node_id: IC_kwDOCGYnMM5M9aJB
created_at: 2022-10-25T21:31:15Z
updated_at: 2022-10-25T21:31:15Z
author_association: OWNER
body:
  Based on the docs here I tried the following too: https://docutils.sourceforge.io/docs/user/smartquotes.html#description

  - `\--`
  - `\\--`
  - `\\-\\-`
  - `\-\-`

  But none of them had the desired effect in this particular piece of markup:

      the :ref:`insert \--convert <cli_insert_convert>` 

  I think because this is text inside a `:ref:` block, not regular text.

  Consider the following:

      The \--convert and the :ref:`insert \--convert <cli_insert_convert>` and

  It's rendered like this:

  <img width="328" alt="image" src="https://user-images.githubusercontent.com/9599/197885893-b859f2bd-619b-4406-9688-3a8e592267f5.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/496#issuecomment-1291167887
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/496
node_id: IC_kwDOCGYnMM5M9aiP
created_at: 2022-10-25T21:33:25Z
updated_at: 2022-10-25T21:33:25Z
author_association: OWNER
body:
  I do care about this, but I'm not hugely experienced with types yet so I'm open to suggestions about how to do it!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/496#issuecomment-1291170072
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/496
node_id: IC_kwDOCGYnMM5M9bEY
created_at: 2022-10-25T21:36:12Z
updated_at: 2022-10-25T21:36:12Z
author_association: OWNER
body:
  I was going to suggest using `db.table(name)` instead of `db[name]` - but it looks like that method will have the same problem:

  https://github.com/simonw/sqlite-utils/blob/defa2974c6d3abc19be28d6b319649b8028dc966/sqlite_utils/db.py#L497-L506

  I could change `sqlite-utils` so `db.table(name)` always returns a table and you need to call `db.view(name)` if you want to access a view - that would require bumping to 4.0 though. I'm not convinced that's the best approach here either.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1851#issuecomment-1291226367
issue_url: https://api.github.com/repos/simonw/datasette/issues/1851
node_id: IC_kwDOBm6k_c5M9oz_
created_at: 2022-10-25T22:58:30Z
updated_at: 2022-10-25T22:58:30Z
author_association: OWNER
body:
  The `datasette insert` concept included plugin support, with the idea of being able to support things like SpatiaLite files:
  - #1160

  I think this API mechanism is going to be a bit less exciting than that - it will be low-level for inserting rows, and if you want to do something fancier you can use a canned query that feeds incoming GeoJSON to a SpatiaLite function instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1852#issuecomment-1291227942
issue_url: https://api.github.com/repos/simonw/datasette/issues/1852
node_id: IC_kwDOBm6k_c5M9pMm
created_at: 2022-10-25T23:01:18Z
updated_at: 2022-10-25T23:01:18Z
author_association: OWNER
body:
  Datasette currently defaults to having everything public-readable by default, unless a permission plugin changes that default.

  In thinking more about this API mechanism, I realized that it might be good to have a mode where Datasette _doesn't_ default to public everything. Maybe `datasette --private` to start it like that?

  Might even be an opportunity to get rid of the current slightly confusing mechanism where permission checks can announce that they should default to true:

  https://github.com/simonw/datasette/blob/c7dd76c26257ded5bcdfd0570e12412531b8b88f/datasette/views/database.py#L152-L154

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1851#issuecomment-1291228502
issue_url: https://api.github.com/repos/simonw/datasette/issues/1851
node_id: IC_kwDOBm6k_c5M9pVW
created_at: 2022-10-25T23:02:10Z
updated_at: 2022-10-25T23:02:10Z
author_association: CONTRIBUTOR
body:
  That's reasonable. Canned queries and custom endpoints are certainly going to give more room for specific needs. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1852#issuecomment-1291231651
issue_url: https://api.github.com/repos/simonw/datasette/issues/1852
node_id: IC_kwDOBm6k_c5M9qGj
created_at: 2022-10-25T23:07:17Z
updated_at: 2022-10-25T23:07:17Z
author_association: OWNER
body:
  Interesting challenge: what permissions should users be allowed to grant to tokens?

  Clearly a user should not be able to create a token with a permission that the user themselves does not have.

  And should there be a permission that allows people to create tokens? I think so.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1852#issuecomment-1291232589
issue_url: https://api.github.com/repos/simonw/datasette/issues/1852
node_id: IC_kwDOBm6k_c5M9qVN
created_at: 2022-10-25T23:08:37Z
updated_at: 2022-10-25T23:08:37Z
author_association: OWNER
body:
  ... so maybe there's a way to create a token that inherits the exact permissions of the actor that created the token? That could even be a default mode for tokens, with an option to then further restrict permissions if desired.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1852#issuecomment-1291233652
issue_url: https://api.github.com/repos/simonw/datasette/issues/1852
node_id: IC_kwDOBm6k_c5M9ql0
created_at: 2022-10-25T23:10:20Z
updated_at: 2022-10-25T23:10:44Z
author_association: OWNER
body:
  In which case the token would need to duplicate the current `actor` and then add extra constraints. So maybe the token design looks like this:
  ```json
  {
    "a": { "copy_of": "actor_creating_token"},
    "p": { "t": "... the thing designed earlier, with those permissions in it" },
    "e": "integer timestamp when token expires"
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1852#issuecomment-1291234262
issue_url: https://api.github.com/repos/simonw/datasette/issues/1852
node_id: IC_kwDOBm6k_c5M9qvW
created_at: 2022-10-25T23:11:23Z
updated_at: 2022-10-25T23:11:23Z
author_association: OWNER
body:
  I'm going to build an initial `/-/create-token` interface which just bakes a token with the current actor in it and an optional expiry timestamp. I'll try the limited permissions thing later.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1852#issuecomment-1291243333
issue_url: https://api.github.com/repos/simonw/datasette/issues/1852
node_id: IC_kwDOBm6k_c5M9s9F
created_at: 2022-10-25T23:25:13Z
updated_at: 2022-10-25T23:25:13Z
author_association: OWNER
body:
  A `/-/debug-token` page that can take a token and decode it to show you how long until it expires, what actor it represents and the permissions it has will be useful as well.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1852#issuecomment-1291268380
issue_url: https://api.github.com/repos/simonw/datasette/issues/1852
node_id: IC_kwDOBm6k_c5M9zEc
created_at: 2022-10-26T00:09:06Z
updated_at: 2022-10-26T00:09:06Z
author_association: OWNER
body:
  Demo:

  ![token-demo](https://user-images.githubusercontent.com/9599/197904595-e5651d6c-bafc-4124-b762-71ad94c06ced.gif)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1852#issuecomment-1291269607
issue_url: https://api.github.com/repos/simonw/datasette/issues/1852
node_id: IC_kwDOBm6k_c5M9zXn
created_at: 2022-10-26T00:11:15Z
updated_at: 2022-10-26T00:11:15Z
author_association: OWNER
body:
  If you click "Create token" for "Token never expires" multiple times you currently get exactly the same token each time, since it's just a signed token containing a copy of your actor dictionary.

  I'm not sure if I like that. I could give each token a random ID (maybe using `secrets.token_hex()`) such that different tokens have different identities, which would be useful for logging and auditing and maybe even revocation at some point in the future.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1852#issuecomment-1291270227
issue_url: https://api.github.com/repos/simonw/datasette/issues/1852
node_id: IC_kwDOBm6k_c5M9zhT
created_at: 2022-10-26T00:12:18Z
updated_at: 2022-10-26T00:12:18Z
author_association: OWNER
body:
  Demo is now live at https://latest-1-0-dev.datasette.io/-/create-token - visit https://latest-1-0-dev.datasette.io/login-as-root first to sign in.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1852#issuecomment-1291271580
issue_url: https://api.github.com/repos/simonw/datasette/issues/1852
node_id: IC_kwDOBm6k_c5M9z2c
created_at: 2022-10-26T00:14:49Z
updated_at: 2022-10-26T00:15:06Z
author_association: OWNER
body:
  If I'm going to have a setting to disable this feature I need to decide what it will be called.

  Closest existing setting is this one, since it's for a feature that is turned on by default:

      datasette mydatabase.db --setting allow_download off

  So maybe this?

      datasette mydatabase.db --setting allow_signed_api_tokens off

  I like `allow_signed_api_tokens` more than `allow_api_tokens` because if you install a plugin such as https://datasette.io/plugins/datasette-auth-tokens then API tokens will work even though you disabled this default signed token feature.

  `allow_signed_api_tokens` does feel a bit clumsy/verbose though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1852#issuecomment-1291272280
issue_url: https://api.github.com/repos/simonw/datasette/issues/1852
node_id: IC_kwDOBm6k_c5M90BY
created_at: 2022-10-26T00:16:09Z
updated_at: 2022-10-26T00:46:21Z
author_association: OWNER
body:
  Other options:

  - `--setting default_api_tokens off`
  - `--setting signed_api_tokens off`
  - `--setting allow_create_token off`

  These feel inconsistent because they don't use the `allow_` prefix - but they're also a bit less ugly to look at.

  I like that last one.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1852#issuecomment-1291273609
issue_url: https://api.github.com/repos/simonw/datasette/issues/1852
node_id: IC_kwDOBm6k_c5M90WJ
created_at: 2022-10-26T00:18:40Z
updated_at: 2022-10-26T00:18:40Z
author_association: OWNER
body:
  Another thought about tokens that can act on behalf of the user.

  Imagine a user has permission to access a table. They create a token that can create that table... but then their permission is revoked. It would be bad if they could still use that token they created earlier to access that table!

  On that basis, I think the model described above where tokens mainly work to provide an "act on behalf of this actor" - but with optional additional constraints - is a good one.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1852#issuecomment-1291274835
issue_url: https://api.github.com/repos/simonw/datasette/issues/1852
node_id: IC_kwDOBm6k_c5M90pT
created_at: 2022-10-26T00:20:48Z
updated_at: 2022-10-26T00:22:26Z
author_association: OWNER
body:
  Tests failed because I added a view without also adding documentation!

  I forgot that the deploy still goes out for branches other than `main` even if the tests aren't passing:
   https://github.com/simonw/datasette/blob/c7dd76c26257ded5bcdfd0570e12412531b8b88f/.github/workflows/deploy-latest.yml#L34-L38

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1852#issuecomment-1291277913
issue_url: https://api.github.com/repos/simonw/datasette/issues/1852
node_id: IC_kwDOBm6k_c5M91ZZ
created_at: 2022-10-26T00:26:11Z
updated_at: 2022-10-26T00:26:11Z
author_association: NONE
body:
  > On that basis, I think the model described above where tokens mainly work to provide an "act on behalf of this actor" - but with optional additional constraints - is a good one.

  This is what we do for Sandstorm essentially and I fully agree it's the right way to do API tokens in multiuser systems.

  Constraints will definitely be important though. I know I want a token to submit error reports programmatically, but I wouldn't want that token to convey my right to delete tables and records, Little Bobby Tables is out there somewhere, and he's all grown up now.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1852#issuecomment-1291281243
issue_url: https://api.github.com/repos/simonw/datasette/issues/1852
node_id: IC_kwDOBm6k_c5M92Nb
created_at: 2022-10-26T00:32:21Z
updated_at: 2022-10-26T00:32:21Z
author_association: OWNER
body:
  Rather than duplicating the entire actor into the "a" field, maybe just copy the actor ID?

  Would need to restrict token creation to just actors with an ID set.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1852#issuecomment-1291289369
issue_url: https://api.github.com/repos/simonw/datasette/issues/1852
node_id: IC_kwDOBm6k_c5M94MZ
created_at: 2022-10-26T00:47:46Z
updated_at: 2022-10-26T00:47:46Z
author_association: OWNER
body:
  The tokens also need something that can be used to differentiate them from alternative token mechanisms that other plugins might provide.

  Maybe a prefix before the signed value.

  Prefixes are also useful for scanning to check they were not accidentally committed to source control.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1852#issuecomment-1291392887
issue_url: https://api.github.com/repos/simonw/datasette/issues/1852
node_id: IC_kwDOBm6k_c5M-Rd3
created_at: 2022-10-26T02:04:48Z
updated_at: 2022-10-26T02:04:48Z
author_association: OWNER
body:
  Implemented that `dstok_` prefix and the thing where only the `actor["id"]` is copied to the `"a"` field.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1852#issuecomment-1291397623
issue_url: https://api.github.com/repos/simonw/datasette/issues/1852
node_id: IC_kwDOBm6k_c5M-Sn3
created_at: 2022-10-26T02:11:40Z
updated_at: 2022-10-26T02:11:40Z
author_association: OWNER
body:
  Built a prototype of the `actor_from_request()` hook for this and now:

  ```
  % curl http://127.0.0.1:8001/-/actor.json -H 'Authorization: Bearer dstok_eyJhIjoicm9vdCIsImUiOm51bGx9.6O1OxgNTFkAU6uw7xNcmXYX949A'
  {"actor": {"id": "root", "dstok": true}}
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1856#issuecomment-1291409312
issue_url: https://api.github.com/repos/simonw/datasette/issues/1856
node_id: IC_kwDOBm6k_c5M-Veg
created_at: 2022-10-26T02:24:49Z
updated_at: 2022-10-26T02:24:49Z
author_association: OWNER
body:
  The effect of this setting will be:

  - `/-/create-tokens` interface is no longer available
  - Incoming `dstok_` tokens are no longer respected by the following code:

  https://github.com/simonw/datasette/blob/b29e487bc3fde6418bf45bda7cfed2e081ff03fb/datasette/default_permissions.py#L52-L72

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1856#issuecomment-1291410331
issue_url: https://api.github.com/repos/simonw/datasette/issues/1856
node_id: IC_kwDOBm6k_c5M-Vub
created_at: 2022-10-26T02:26:19Z
updated_at: 2022-10-26T02:26:19Z
author_association: OWNER
body:
  It's a bit confusing that a setting called `allow_create_tokens` also causes incoming `dstok_` tokens to be ignored.

  Is it confusing enough that I should pick a different name for the setting though?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1850#issuecomment-1291417100
issue_url: https://api.github.com/repos/simonw/datasette/issues/1850
node_id: IC_kwDOBm6k_c5M-XYM
created_at: 2022-10-26T02:35:32Z
updated_at: 2022-10-26T02:35:32Z
author_association: OWNER
body:
  It strikes me that users should NOT be able to use a token to create additional tokens.

  The current design actually does allow that, since the `dstok_` Bearer token can be used to authenticate calls to the `/-/create-token` page.

  So I think I need a mechanism whereby that page can only allow access to users authenticated by cookie.

  Not obvious how to do that though, since Datasette's authentication actor system is designed to abstract that detail away!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1850#issuecomment-1291417755
issue_url: https://api.github.com/repos/simonw/datasette/issues/1850
node_id: IC_kwDOBm6k_c5M-Xib
created_at: 2022-10-26T02:36:52Z
updated_at: 2022-10-26T02:36:58Z
author_association: OWNER
body:
  I'm going to set a convention that `"token": "something"` in an actor means that they were authenticated by a token.

  `"token": "dstok"` for example.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1857#issuecomment-1291418546
issue_url: https://api.github.com/repos/simonw/datasette/issues/1857
node_id: IC_kwDOBm6k_c5M-Xuy
created_at: 2022-10-26T02:38:35Z
updated_at: 2022-10-26T02:38:35Z
author_association: OWNER
body:
  I'm going to set a convention that an actor signed in via a token should set `"token": "something"` as a key. Then the `/-/create-token` view can reject those actors.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1850#issuecomment-1291430992
issue_url: https://api.github.com/repos/simonw/datasette/issues/1850
node_id: IC_kwDOBm6k_c5M-axQ
created_at: 2022-10-26T02:59:33Z
updated_at: 2022-10-26T02:59:33Z
author_association: OWNER
body:
  I started the documentation for the API tokens mechanism here: https://docs.datasette.io/en/1.0-dev/authentication.html#api-tokens

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1858#issuecomment-1291435464
issue_url: https://api.github.com/repos/simonw/datasette/issues/1858
node_id: IC_kwDOBm6k_c5M-b3I
created_at: 2022-10-26T03:07:16Z
updated_at: 2022-10-26T03:07:16Z
author_association: OWNER
body:
  This setting will disable the "Token never expires" option:

  <img width="578" alt="image" src="https://user-images.githubusercontent.com/9599/197925181-f1b9eb4c-e9be-4b55-9c1b-29622a83e58d.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1859#issuecomment-1291439875
issue_url: https://api.github.com/repos/simonw/datasette/issues/1859
node_id: IC_kwDOBm6k_c5M-c8D
created_at: 2022-10-26T03:14:58Z
updated_at: 2022-10-26T03:14:58Z
author_association: OWNER
body:
  Initial design:

      datasette create-token <id>

  Or:

      datasette create-token <id> --expire-after 10m

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1843#issuecomment-1291467084
issue_url: https://api.github.com/repos/simonw/datasette/issues/1843
node_id: IC_kwDOBm6k_c5M-jlM
created_at: 2022-10-26T04:03:49Z
updated_at: 2022-10-26T04:03:49Z
author_association: OWNER
body:
  This time I'm suspicious that there are open SQLite files tucked away in thread locals hidden inside my thread pool executor:

  https://github.com/simonw/datasette/blob/c7dd76c26257ded5bcdfd0570e12412531b8b88f/datasette/database.py#L24

  https://github.com/simonw/datasette/blob/c7dd76c26257ded5bcdfd0570e12412531b8b88f/datasette/database.py#L204-L214

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1859#issuecomment-1291484749
issue_url: https://api.github.com/repos/simonw/datasette/issues/1859
node_id: IC_kwDOBm6k_c5M-n5N
created_at: 2022-10-26T04:29:43Z
updated_at: 2022-10-26T04:29:43Z
author_association: OWNER
body:
  Documentation:
  - https://docs.datasette.io/en/1.0-dev/authentication.html#datasette-create-token
  - https://docs.datasette.io/en/1.0-dev/cli-reference.html#datasette-create-token

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1855#issuecomment-1291485444
issue_url: https://api.github.com/repos/simonw/datasette/issues/1855
node_id: IC_kwDOBm6k_c5M-oEE
created_at: 2022-10-26T04:30:34Z
updated_at: 2022-10-26T04:30:34Z
author_association: OWNER
body:
  I'm going to delay working on this until after I have some of the write APIs built to try it against:
  - #1851

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1860#issuecomment-1292390996
issue_url: https://api.github.com/repos/simonw/datasette/issues/1860
node_id: IC_kwDOBm6k_c5NCFJU
created_at: 2022-10-26T17:43:41Z
updated_at: 2022-10-26T17:43:41Z
author_association: NONE
body:
  I guess the issue is here: https://github.com/simonw/datasette/blob/9676b2deb07cff20247ba91dad3e84a4ab0b00d1/datasette/utils/__init__.py#L209

  Here is a working regexp allowing it:
  ```diff
  -    re.compile(r"^select\b"),
  +    re.compile(r"^\s*(/\*.+?(?=\*/)\*/\s*)*select"),
  ```
  `^\s*`: beginning by 0 or an infinite number of \s (spaces, tabs, newlines...)
  `(/\*.+?(?=\*/)\*/\s*)*`: 0 or an infinite number of chars beginning by `/*` and ending to the next occurrence of `*/` followed by 0 or an infinite number of \s

  You can play with the regexp here: https://regex101.com/r/aESXDL/3

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/499#issuecomment-1292401308
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/499
node_id: IC_kwDOCGYnMM5NCHqc
created_at: 2022-10-26T17:54:26Z
updated_at: 2022-10-26T17:54:51Z
author_association: CONTRIBUTOR
body:
  The problem with how it is currently is that the transformed fts table _will_ return incorrect results (unless the table was only 1 row or something), even if create_triggers was enabled previously. Maybe the simplest solution is to disable fts on a transformed table rather than try to recreate it? Thoughts?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1851#issuecomment-1292519956
issue_url: https://api.github.com/repos/simonw/datasette/issues/1851
node_id: IC_kwDOBm6k_c5NCkoU
created_at: 2022-10-26T19:20:33Z
updated_at: 2022-10-26T19:20:33Z
author_association: CONTRIBUTOR
body:
  > This could use a new plugin hook, too. I don't want to complicate your life too much, but for things like GIS, I'd want a way to turn regular JSON into SpatiaLite geometries or combine X/Y coordinates into point geometries and such. Happy to help however I can.

   @eyeseast Maybe you could do this with triggers? Like you can insert JSON-friendly data into a "raw" table, and create a trigger that transforms that inserted data into the proper table

  Here's an example:

  ```sql
  -- meant to be updated from a Datasette insert
  create table points_raw(longitude int, latitude int);

  -- the target table with proper spatliate geometries
  create table points(point geometry);

  CREATE TRIGGER insert_points_raw INSERT ON points_raw 
    BEGIN
      insert into points(point) values (makepoint(new.longitude, new.latitude))
    END;
  ```

  You could then POST a new row to `points_raw` like this:
  ```
  POST /db/points_raw
  Authorization: Bearer xxx
  Content-Type: application/json
  {
    "row": {
      "longitude": 27.64356,
      "latitude": -47.29384
    }
  }
  ```

  Then SQLite with run the trigger and insert a new row in `points` with the correct geometry point. Downside is you'd have duplicated data with `points_raw`, but maybe it could be a `TEMP` table (or have a cron that deletes all rows from that table every so often?)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1851#issuecomment-1292592210
issue_url: https://api.github.com/repos/simonw/datasette/issues/1851
node_id: IC_kwDOBm6k_c5NC2RS
created_at: 2022-10-26T20:03:46Z
updated_at: 2022-10-26T20:03:46Z
author_association: CONTRIBUTOR
body:
  Yeah, every time I see something cool done with triggers, I remember that I need to start using triggers.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1849#issuecomment-1292653219
issue_url: https://api.github.com/repos/simonw/datasette/issues/1849
node_id: IC_kwDOBm6k_c5NDFKj
created_at: 2022-10-26T21:06:56Z
updated_at: 2022-10-26T21:06:56Z
author_association: OWNER
body:
  This was a hit to an authenticated page where the incoming user WAS logged in but did not have permission to view that specific page.

  Code in question: https://github.com/simonw/datasette/blob/c7dd76c26257ded5bcdfd0570e12412531b8b88f/datasette/app.py#L634-L640

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1849#issuecomment-1292654522
issue_url: https://api.github.com/repos/simonw/datasette/issues/1849
node_id: IC_kwDOBm6k_c5NDFe6
created_at: 2022-10-26T21:08:20Z
updated_at: 2022-10-26T21:08:20Z
author_association: OWNER
body:
  From the stack trace in Sentry:

  <img width="833" alt="image" src="https://user-images.githubusercontent.com/9599/198138449-1259fab4-efb3-4df4-aea6-b350a07c56de.png">

  So this happened because a custom plugin tried to render `forbidden.html` without passing in the `request`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1849#issuecomment-1292654852
issue_url: https://api.github.com/repos/simonw/datasette/issues/1849
node_id: IC_kwDOBm6k_c5NDFkE
created_at: 2022-10-26T21:08:44Z
updated_at: 2022-10-26T21:08:44Z
author_association: OWNER
body:
  Generally though we should expect that people might try to use `render_template(...)` without passing a `request`, so Datasette core should be able to handle this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1860#issuecomment-1292659986
issue_url: https://api.github.com/repos/simonw/datasette/issues/1860
node_id: IC_kwDOBm6k_c5NDG0S
created_at: 2022-10-26T21:14:26Z
updated_at: 2022-10-26T21:15:22Z
author_association: OWNER
body:
  Yeah we should fix this.

  https://www.sqlite.org/lang_comment.html - SQLite also supports `-- style` comments.

  I like how explicit the documentation is here:

  > SQL comments begin with two consecutive "-" characters (ASCII 0x2d) and extend up to and including the next newline character (ASCII 0x0a) or until the end of input, whichever comes first.
  > 
  > C-style comments begin with "/*" and extend up to and including the next "*/" character pair or until the end of input, whichever comes first. C-style comments can span multiple lines. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1860#issuecomment-1292674919
issue_url: https://api.github.com/repos/simonw/datasette/issues/1860
node_id: IC_kwDOBm6k_c5NDKdn
created_at: 2022-10-26T21:31:22Z
updated_at: 2022-10-26T21:31:22Z
author_association: OWNER
body:
  I'm experimenting with this:
  ```python
  # Allow SQL to start with a /* */ or -- comment
  comment_re = (
      # Start of string, then any amount of whitespace
      r'^(\s*' +
      # Comment that starts with -- and ends at a newline
      r'(?:\-\-.*?\n\s*)' +
      # Comment that starts with /* and ends with */
      r'|(?:/\*[\s\S]*?\*/)' + 
      # Whitespace
      r')*\s*'
  )

  allowed_sql_res = [
      re.compile(comment_re + r"select\b"),
      re.compile(comment_re + r"explain\s+select\b"),
      re.compile(comment_re + r"explain\s+query\s+plan\s+select\b"),
      re.compile(comment_re + r"with\b"),
      re.compile(comment_re + r"explain\s+with\b"),
      re.compile(comment_re + r"explain\s+query\s+plan\s+with\b"),
  ]
  ```
  This should allow any number of comments of either type as a suffix to the allowed SQL patterns.

  Needs extensive unit tests!

  I'm not massively worried if it has a flaw in it though, since this is part of Datasette's defense in depth: if a non-SELECT query sneaks through it still shouldn't be able to cause any damage as the database connection is read-only or immutable.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1860#issuecomment-1292678657
issue_url: https://api.github.com/repos/simonw/datasette/issues/1860
node_id: IC_kwDOBm6k_c5NDLYB
created_at: 2022-10-26T21:35:23Z
updated_at: 2022-10-26T21:35:37Z
author_association: OWNER
body:
  Here are the new tests - each of these should now work: https://github.com/simonw/datasette/blob/55a709c480a1e7401b4ff6208f37a2cf7c682183/tests/test_utils.py#L170-L175

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1860#issuecomment-1292679567
issue_url: https://api.github.com/repos/simonw/datasette/issues/1860
node_id: IC_kwDOBm6k_c5NDLmP
created_at: 2022-10-26T21:36:25Z
updated_at: 2022-10-26T21:36:25Z
author_association: OWNER
body:
  I'm never 100% sure how to tell if a regular expression includes a nasty denial of service attack - are there any inputs that could cause this new regex to execute in quadratic time or similar?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1860#issuecomment-1292685478
issue_url: https://api.github.com/repos/simonw/datasette/issues/1860
node_id: IC_kwDOBm6k_c5NDNCm
created_at: 2022-10-26T21:42:35Z
updated_at: 2022-10-26T21:42:35Z
author_association: OWNER
body:
  That's deployed to https://latest.datasette.io/ now - some examples:

  - https://latest.datasette.io/fixtures?sql=--+one+kind+of+comment%0D%0Aselect+*+from+searchable
  - https://latest.datasette.io/fixtures?sql=%2F*+Multi%0D%0A++line+comment+*%2F%0D%0Aselect+*+from+searchable
  - https://latest.datasette.io/fixtures?sql=%2F*+Both+kinds+*%2F%0D%0A--+of+comment%0D%0A%2F*+and+more+*%2F%0D%0A--+and+more+and+more%0D%0Aselect+*+from+searchable

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1858#issuecomment-1292687774
issue_url: https://api.github.com/repos/simonw/datasette/issues/1858
node_id: IC_kwDOBm6k_c5NDNme
created_at: 2022-10-26T21:44:57Z
updated_at: 2022-10-26T21:44:57Z
author_association: OWNER
body:
  I'm going for  consistency with `max_csv_mb` and `max_returned_rows` and `allow_signed_tokens` and `default_cache_ttl`.

  So `max_signed_tokens_ttl`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1858#issuecomment-1292708227
issue_url: https://api.github.com/repos/simonw/datasette/issues/1858
node_id: IC_kwDOBm6k_c5NDSmD
created_at: 2022-10-26T22:05:34Z
updated_at: 2022-10-26T22:05:34Z
author_association: OWNER
body:
  I just realized this can't easily affect the `datasette create-token` command because it doesn't currently accept the `--setting` option, so it wouldn't know what `max_signed_tokens_ttl` was.

  More to the point: even if it did, someone could abuse their knowledge of the secret to create a signed non-expiring token even on servers that didn't want to support those.

  So I actually need to redesign the token format: it needs to store the timestamp when the token was created and the intended duration, NOT the timestamp that the token expires at.

  Otherwise it's not possible for servers to enforce `max_signed_tokens_ttl` - someone could always create a token with a custom `expires_at` timestamp on it outside of the configured limit.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1858#issuecomment-1292709818
issue_url: https://api.github.com/repos/simonw/datasette/issues/1858
node_id: IC_kwDOBm6k_c5NDS-6
created_at: 2022-10-26T22:07:04Z
updated_at: 2022-10-26T22:07:04Z
author_association: OWNER
body:
  New token design:

  ```json
  {
    "a": "actor-id",
    "t": "creation timestamp as integer",
    "d": "intended duration in seconds, or blank if no duration set"
  }
  ```
  This is in place of the `"e": "expiry timestamp"` design I've built so far.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1851#issuecomment-1292939146
issue_url: https://api.github.com/repos/simonw/datasette/issues/1851
node_id: IC_kwDOBm6k_c5NEK-K
created_at: 2022-10-27T04:00:17Z
updated_at: 2022-10-27T04:23:15Z
author_association: OWNER
body:
  Documentation for this first draft of the API: https://docs.datasette.io/en/1.0-dev/json_api.html#inserting-a-single-row

  It currently returns errors as HTML - it needs to return errors as JSON. Also the errors need comprehensive test coverage.

  I'm also worried about what happens if you use it on a table that doesn't use an integer primary key - need to check that. I think this code may break:

  https://github.com/simonw/datasette/blob/51c436fed29205721dcf17fa31d7e7090d34ebb8/datasette/views/table.py#L155-L171

  Plus will `rowid` tables without an explicit primary key return the `rowid` column? They should.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1850#issuecomment-1292940011
issue_url: https://api.github.com/repos/simonw/datasette/issues/1850
node_id: IC_kwDOBm6k_c5NELLr
created_at: 2022-10-27T04:01:59Z
updated_at: 2022-10-27T04:01:59Z
author_association: OWNER
body:
  Working on that first "insert row" implementation:

  - https://github.com/simonw/datasette/issues/1851

  Has made it very clear to me that I should go the whole hog and build the basic form-based interface for this as well.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1851#issuecomment-1292952121
issue_url: https://api.github.com/repos/simonw/datasette/issues/1851
node_id: IC_kwDOBm6k_c5NEOI5
created_at: 2022-10-27T04:24:09Z
updated_at: 2022-10-27T04:24:20Z
author_association: OWNER
body:
  And come up with a whole bunch of tests for weird table shapes, surprising column names, different types etc.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1855#issuecomment-1292959886
issue_url: https://api.github.com/repos/simonw/datasette/issues/1855
node_id: IC_kwDOBm6k_c5NEQCO
created_at: 2022-10-27T04:30:07Z
updated_at: 2022-10-27T04:30:07Z
author_association: OWNER
body:
  Here's an interesting edge-case to consider: what if a user creates themselves a token for a specific table, then deletes that table, and waits for another user to create a table of the same name... and then uses their previously created token to write to the table that someone else created?

  Not sure if this is a threat I need to actively consider, but it's worth thinking a little bit about the implications of such a thing - since there will be APIs that allow users to create tables, and there may be cases where people want to have a concept of users "owning" specific tables.

  This is probably something that could be left for plugins to solve, but it still needs to be understood and potentially documented.

  There may even be a world in which tracking the timestamp at which a table was created becomes useful - because that could then be baked into API tokens, such that a token created BEFORE the table was created does not grant access to that table.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1855#issuecomment-1292962813
issue_url: https://api.github.com/repos/simonw/datasette/issues/1855
node_id: IC_kwDOBm6k_c5NEQv9
created_at: 2022-10-27T04:31:40Z
updated_at: 2022-10-27T04:31:40Z
author_association: OWNER
body:
  My hunch on this is that anyone with that level of complex permissions requirements needs to be using a custom authentication plugin which includes much more concrete token rules, rather than the default signed stateless token implementation that ships with Datasette core.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1851#issuecomment-1292997608
issue_url: https://api.github.com/repos/simonw/datasette/issues/1851
node_id: IC_kwDOBm6k_c5NEZPo
created_at: 2022-10-27T04:54:53Z
updated_at: 2022-10-27T19:05:50Z
author_association: OWNER
body:
  I'm going to change the design of this to:
  ```
  {
      "insert": {
          "title" :"..."
      }
  }
  ```
  Renaming `"row"` to `"insert"`.

  This will  be consistent with adding `"drop": true` for dropping a table, and maybe other verbs like for modifying the schema.

  The API response will look like this:

  ```json
  {
      "inserted_row": {
          "id": 1,
          "title": "..."
      }
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1851#issuecomment-1292999579
issue_url: https://api.github.com/repos/simonw/datasette/issues/1851
node_id: IC_kwDOBm6k_c5NEZub
created_at: 2022-10-27T04:59:06Z
updated_at: 2022-10-27T04:59:12Z
author_association: OWNER
body:
  I should probably refactor this to use `sqlite-utils`, since I'm going to want to use that later for the feature that automatically creates tables.

  Might make it easier to solve the rowid issues too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1860#issuecomment-1293863145
issue_url: https://api.github.com/repos/simonw/datasette/issues/1860
node_id: IC_kwDOBm6k_c5NHsjp
created_at: 2022-10-27T17:43:37Z
updated_at: 2022-10-27T17:43:37Z
author_association: NONE
body:
  Sorry I forgot the `-- comments like that`.

  I'm afraid there is an issue in your regexp, see: https://regex101.com/r/pyubJf/1

  I guess I can fix it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1866#issuecomment-1293887808
issue_url: https://api.github.com/repos/simonw/datasette/issues/1866
node_id: IC_kwDOBm6k_c5NHylA
created_at: 2022-10-27T18:07:02Z
updated_at: 2022-10-27T18:07:02Z
author_association: OWNER
body:
  Error handling is really important here.

  What should happen if you submit 100 records and one of them has some kind of validation error? How should that error be reported back to you?

  I'm inclined to say that it defaults to all-or-nothing in a transaction - but there should be a `"continue_on_error": true` option (or similar) which causes it to insert the ones that are valid while reporting back the ones that are invalid.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1866#issuecomment-1293890684
issue_url: https://api.github.com/repos/simonw/datasette/issues/1866
node_id: IC_kwDOBm6k_c5NHzR8
created_at: 2022-10-27T18:09:52Z
updated_at: 2022-10-27T18:09:52Z
author_association: OWNER
body:
  Should this API accept CSV/TSV etc in addition to JSON?

  I'm torn on this one. My initial instinct is that it should not - and there should instead be a Datasette client library / CLI tool you can use that knows how to turn CSV into batches of JSON calls for when you want to upload a CSV file.

  I don't think the usability of `curl https://datasette/db/table  -F 'data=@path/to/file.csv' -H 'Authentication: Bearer xxx'` is particularly great compared to  something like`datasette client insert https://datasette/ db table file.csv --csv` (where the command version could store API tokens for you too).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1866#issuecomment-1293891191
issue_url: https://api.github.com/repos/simonw/datasette/issues/1866
node_id: IC_kwDOBm6k_c5NHzZ3
created_at: 2022-10-27T18:10:22Z
updated_at: 2022-10-27T18:10:22Z
author_association: OWNER
body:
  So for the moment I'm just going to concentrate on the JSON API. I can consider CSV variants later on, or as plugins, or both.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1866#issuecomment-1293891876
issue_url: https://api.github.com/repos/simonw/datasette/issues/1866
node_id: IC_kwDOBm6k_c5NHzkk
created_at: 2022-10-27T18:11:05Z
updated_at: 2022-10-27T18:11:05Z
author_association: OWNER
body:
  Likewise for newline-delimited JSON. While it's tempting to want to accept that as an ingest format (because it's nice to generate and stream) I think it's better to have a client application that can turn a stream of newline-delimited JSON into batched JSON inserts.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1866#issuecomment-1293892818
issue_url: https://api.github.com/repos/simonw/datasette/issues/1866
node_id: IC_kwDOBm6k_c5NHzzS
created_at: 2022-10-27T18:12:02Z
updated_at: 2022-10-27T18:12:02Z
author_association: OWNER
body:
  There's one catch with batched inserts: if your CLI tool fails half way through you could end up with a partially populated table - since a bunch of batches will have succeeded first.

  I think that's OK. In the future I may want to come up with a way to run multiple batches of inserts inside a single transaction, but I can ignore that for the first release of this feature.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1866#issuecomment-1293893789
issue_url: https://api.github.com/repos/simonw/datasette/issues/1866
node_id: IC_kwDOBm6k_c5NH0Cd
created_at: 2022-10-27T18:13:00Z
updated_at: 2022-10-27T18:13:00Z
author_association: OWNER
body:
  If people care about that kind of thing they could always push all of their inserts to a table called `_tablename` and then atomically rename that once they've uploaded all of the data (assuming I provide an atomic-rename-this-table mechanism).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1860#issuecomment-1293912781
issue_url: https://api.github.com/repos/simonw/datasette/issues/1860
node_id: IC_kwDOBm6k_c5NH4rN
created_at: 2022-10-27T18:31:15Z
updated_at: 2022-10-27T18:31:15Z
author_association: NONE
body:
  Here is my suggestion:

  `^\s*((?:\-\-.*?\n\s*)|(?:/\*.*?(?=\*/)\*/\s*))*select\b`

  See the following test: https://regex101.com/r/Doeqqa/1

  And here I played all your tests: https://regexr.com/713ir

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1860#issuecomment-1293926417
issue_url: https://api.github.com/repos/simonw/datasette/issues/1860
node_id: IC_kwDOBm6k_c5NH8AR
created_at: 2022-10-27T18:44:20Z
updated_at: 2022-10-27T18:45:21Z
author_association: OWNER
body:
  Hah, I just came up with this one - we were clearly working on this at the same time!

  `^\s*((?:\-\-.*?\n\s*)|(?:\/\*((?!\*\/)[\s\S])*\*\/)\s*)*\s*select\b`

  <img width="898" alt="image" src="https://user-images.githubusercontent.com/9599/198373058-e54f2564-311a-48d6-960d-a3a269ae576d.png">

  https://www.debuggex.com/r/Rbw-UWD9PdOU2GyO

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1860#issuecomment-1293928230
issue_url: https://api.github.com/repos/simonw/datasette/issues/1860
node_id: IC_kwDOBm6k_c5NH8cm
created_at: 2022-10-27T18:46:03Z
updated_at: 2022-10-27T18:46:03Z
author_association: OWNER
body:
  Here's yours on Debuggex: https://www.debuggex.com/r/HjdJryTy9ezGsuWK

  <img width="889" alt="image" src="https://user-images.githubusercontent.com/9599/198373230-c36d87f8-69e6-4abe-8867-4d9ba0b57698.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1851#issuecomment-1293996735
issue_url: https://api.github.com/repos/simonw/datasette/issues/1851
node_id: IC_kwDOBm6k_c5NINK_
created_at: 2022-10-27T19:54:53Z
updated_at: 2022-10-27T19:54:53Z
author_association: OWNER
body:
  Updated docs: https://docs.datasette.io/en/1.0-dev/json_api.html#inserting-a-single-row

  <img width="867" alt="image" src="https://user-images.githubusercontent.com/9599/198385667-e4cf9930-0621-49d7-9fed-347aa96b7581.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1868#issuecomment-1294003701
issue_url: https://api.github.com/repos/simonw/datasette/issues/1868
node_id: IC_kwDOBm6k_c5NIO31
created_at: 2022-10-27T20:02:26Z
updated_at: 2022-10-27T20:02:26Z
author_association: OWNER
body:
  The problem with the above design is that I want to support a bunch of different actions that can be taken against a table:
  - insert a single row
  - insert multiple rows
  - bulk update rows
  - rename table
  - alter table
  - drop table

  I could have ALL of those be a `POST /db/table` with different JSON root keys (`{"drop": true}` for example, but this raises two problems:

  1. Server logs that only show `POST /db/table` will be less useful, they won't reveal what action was performed
  2. What happens if you send `{"insert": {"title": "New record"}, "drop": true}`? Does that return an error, or does it perform both of those actions?

  This is already slightly confusing in that `POST /db/name-of-query` is the existing API for executing a writable canned query: https://docs.datasette.io/en/stable/sql_queries.html#json-api-for-writable-canned-queries

  So I'm ready to consider other design options.

  Initial thoughts on possible designs (for the single row insert case, but could be expanded to cover other verbs):

  - `POST /db/table?action=insert`
  - `POST /db/table?nsert`
  - `POST /db/table/-/insert`

  I quite like that third one: it feels consistent with the existing `/-/actor` etc pages that Datasette serves already.

  There's one slight confusion here in that it overlaps with the URL for a row with a primary key of `"-"` - which is currently at `/db/table/-` - but that might be OK.

  Especially if I say that child pages of rows must theselves use the `/-/` pattern. So to update or delet a row you would use:

  - `POST /db/table/row/-/update`
  - `POST /db/table/row/-/delete`

  So a row with primary key `-` would end up as `/db/table/row/-/-/update` - which I think is OK.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1868#issuecomment-1294004308
issue_url: https://api.github.com/repos/simonw/datasette/issues/1868
node_id: IC_kwDOBm6k_c5NIPBU
created_at: 2022-10-27T20:03:08Z
updated_at: 2022-10-27T20:03:08Z
author_association: OWNER
body:
  The other option here would be to lean into custom HTTP verbs like `DELETE` and `PATCH`. I'm not sold on those: they've never given me any convincing wins over just using `POST` for the many times I've encountered them in my career to date.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1868#issuecomment-1294007024
issue_url: https://api.github.com/repos/simonw/datasette/issues/1868
node_id: IC_kwDOBm6k_c5NIPrw
created_at: 2022-10-27T20:05:44Z
updated_at: 2022-10-27T20:05:52Z
author_association: OWNER
body:
  So given this scheme, the URL design would look like this:

  - `POST /db/table/-/insert` - insert a single row
  - `POST /db/table/-/insert-many` - insert multiple rows (might just keep that on `/-/insert` with a JSON array rather than object though)
  - `POST /db/table/-/drop` - drop a table
  - `POST /db/table/-/alter` - alter a table
  - `POST /db/table/-/upsert` - upsert, https://sqlite-utils.datasette.io/en/stable/python-api.html#upserting-data
  - `POST /db/table/-/create` - could be an endpoint for explicitly creating a table, or should that live at `/db/-/create` instead?

  And for rows (`pks` here since compound primary keys are supported):

  - `POST /db/table/pks/-/update` - update row
  - `POST /db/table/pks/-/delete` - delete row

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1868#issuecomment-1294008282
issue_url: https://api.github.com/repos/simonw/datasette/issues/1868
node_id: IC_kwDOBm6k_c5NIP_a
created_at: 2022-10-27T20:06:34Z
updated_at: 2022-10-27T20:06:34Z
author_association: OWNER
body:
  I'm going to stick with one `/-/insert` endpoint which handles both single row inserts AND multiple row inserts I think - partly because I don't want to build both `/-/upsert` and `/-/upsert-many`, I'd rather just have `/-/upsert`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1868#issuecomment-1294008733
issue_url: https://api.github.com/repos/simonw/datasette/issues/1868
node_id: IC_kwDOBm6k_c5NIQGd
created_at: 2022-10-27T20:07:01Z
updated_at: 2022-10-27T20:07:01Z
author_association: OWNER
body:
  I'm happy with this `/db/table/-/action` design for the moment. Will review it once I've built it to see if I still like it!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1851#issuecomment-1294009354
issue_url: https://api.github.com/repos/simonw/datasette/issues/1851
node_id: IC_kwDOBm6k_c5NIQQK
created_at: 2022-10-27T20:07:42Z
updated_at: 2022-10-27T20:07:42Z
author_association: OWNER
body:
  Need to implement the new URL design from:
  - #1868

  This is now going to be `/db/table/-/insert` - and it will eventually handle bulk inserts as well as single inserts.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1851#issuecomment-1294012084
issue_url: https://api.github.com/repos/simonw/datasette/issues/1851
node_id: IC_kwDOBm6k_c5NIQ60
created_at: 2022-10-27T20:10:47Z
updated_at: 2022-10-27T20:10:47Z
author_association: OWNER
body:
  I'm going to change the incoming JSON back to `{"row": {...}}` - no need to POST `{"insert": ...}` to something with `/-/insert` in the URL already.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1869#issuecomment-1294056552
issue_url: https://api.github.com/repos/simonw/datasette/issues/1869
node_id: IC_kwDOBm6k_c5NIbxo
created_at: 2022-10-27T21:00:02Z
updated_at: 2022-10-27T21:02:25Z
author_association: OWNER
body:
  Those release notes as markdown:

  ### Features

  - Now tested against Python 3.11. Docker containers used by `datasette publish` and `datasette package` both now use that version of Python. ([#1853](https://github.com/simonw/datasette/issues/1853))
  - `--load-extension` option now supports entrypoints. Thanks, Alex Garcia. ([#1789](https://github.com/simonw/datasette/pull/1789))
  - Facet size can now be set per-table with the new `facet_size` table metadata option. ([#1804](https://github.com/simonw/datasette/issues/1804))
  - The [truncate_cells_html](https://docs.datasette.io/en/stable/settings.html#setting-truncate-cells-html) setting now also affects long URLs in columns. ([#1805](https://github.com/simonw/datasette/issues/1805))
  - The non-JavaScript SQL editor textarea now increases height to fit the SQL query. ([#1786](https://github.com/simonw/datasette/issues/1786))
  - Facets are now displayed with better line-breaks in long values. Thanks, Daniel Rech. ([#1794](https://github.com/simonw/datasette/pull/1794))
  - The `settings.json` file used in [Configuration directory mode](https://docs.datasette.io/en/stable/settings.html#config-dir) is now validated on startup. ([#1816](https://github.com/simonw/datasette/issues/1816))
  - SQL queries can now include leading SQL comments, using `/* ... */` or `-- ...` syntax. Thanks, Charles Nepote. ([#1860](https://github.com/simonw/datasette/issues/1860))
  - SQL query is now re-displayed when terminated with a time limit error. ([#1819](https://github.com/simonw/datasette/issues/1819))
  - The [inspect data](https://docs.datasette.io/en/stable/performance.html#performance-inspect) mechanism is now used to speed up server startup - thanks, Forest Gregg. ([#1834](https://github.com/simonw/datasette/issues/1834))
  - In [Configuration directory mode](https://docs.datasette.io/en/stable/settings.html#config-dir) databases with filenames ending in `.sqlite` or `.sqlite3` are now automatically added to the Datasette instance. ([#1646](https://github.com/simonw/datasette/issues/1646))
  - Breadcrumb navigation display now respects the current user's permissions. ([#1831](https://github.com/simonw/datasette/issues/1831))

  ### Plugin hooks and internals

  - The [prepare_jinja2_environment(env, datasette)](https://docs.datasette.io/en/stable/plugin_hooks.html#plugin-hook-prepare-jinja2-environment) plugin hook now accepts an optional `datasette` argument. Hook implementations can also now return an `async` function which will be awaited automatically. ([#1809](https://github.com/simonw/datasette/issues/1809))
  - `Database(is_mutable=)` now defaults to `True`. ([#1808](https://github.com/simonw/datasette/issues/1808))
  - The [datasette.check_visibility()](https://docs.datasette.io/en/stable/internals.html#datasette-check-visibility) method now accepts an optional `permissions=` list, allowing it to take multiple permissions into account at once when deciding if something should be shown as public or private. This has been used to correctly display padlock icons in more places in the Datasette interface. ([#1829](https://github.com/simonw/datasette/issues/1829))
  - Datasette no longer enforces upper bounds on its dependencies. ([#1800](https://github.com/simonw/datasette/issues/1800))

  ### Documentation

  - New tutorial: [Cleaning data with sqlite-utils and Datasette](https://datasette.io/tutorials/clean-data).
  - Screenshots in the documentation are now maintained using [shot-scraper](https://shot-scraper.datasette.io/), as described in [Automating screenshots for the Datasette documentation using shot-scraper](https://simonwillison.net/2022/Oct/14/automating-screenshots/). ([#1844](https://github.com/simonw/datasette/issues/1844))
  - More detailed command descriptions on the [CLI reference](https://docs.datasette.io/en/stable/cli-reference.html#cli-reference) page. ([#1787](https://github.com/simonw/datasette/issues/1787))
  - New documentation on [Running Datasette using OpenRC](https://docs.datasette.io/en/stable/deploying.html#deploying-openrc) - thanks, Adam Simpson. ([#1825](https://github.com/simonw/datasette/pull/1825))

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1786#issuecomment-1294116493
issue_url: https://api.github.com/repos/simonw/datasette/issues/1786
node_id: IC_kwDOBm6k_c5NIqaN
created_at: 2022-10-27T21:50:12Z
updated_at: 2022-10-27T21:50:12Z
author_association: OWNER
body:
  Demo in Datasette Lite: https://lite.datasette.io/#/fixtures?sql=select%0A++pk1%2C%0A++pk2%2C%0A++content%2C%0A++sortable%2C%0A++sortable_with_nulls%2C%0A++sortable_with_nulls_2%2C%0A++text%0Afrom%0A++sortable%0Aorder+by%0A++pk1%2C%0A++pk2%0Alimit%0A++101

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1869#issuecomment-1294181485
issue_url: https://api.github.com/repos/simonw/datasette/issues/1869
node_id: IC_kwDOBm6k_c5NI6Rt
created_at: 2022-10-27T22:24:37Z
updated_at: 2022-10-27T22:24:37Z
author_association: OWNER
body:
  https://docs.datasette.io/en/stable/changelog.html#v0-63

  Annotated release notes: https://simonwillison.net/2022/Oct/27/datasette-0-63/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1851#issuecomment-1294224185
issue_url: https://api.github.com/repos/simonw/datasette/issues/1851
node_id: IC_kwDOBm6k_c5NJEs5
created_at: 2022-10-27T23:18:24Z
updated_at: 2022-11-03T23:26:05Z
author_association: OWNER
body:
  So new API design is:

  ```
  POST /db/table/-/insert
  Authorization: Bearer xxx
  Content-Type: application/json
  {
    "row": {
      "id": 1,
      "name": "New record"
    }
  }
  ```
  Returns:
  ```
  201 Created
  {
    "row": [{
      "id": 1,
      "name": "New record"
    }]
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1870#issuecomment-1294237783
issue_url: https://api.github.com/repos/simonw/datasette/issues/1870
node_id: IC_kwDOBm6k_c5NJIBX
created_at: 2022-10-27T23:42:18Z
updated_at: 2022-10-27T23:42:18Z
author_association: CONTRIBUTOR
body:
  Relevant sqlite forum thread: https://www.sqlite.org/forum/forumpost/02f7bda329f41e30451472421cf9ce7f715b768ce3db02797db1768e47950d48

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1870#issuecomment-1294238862
issue_url: https://api.github.com/repos/simonw/datasette/issues/1870
node_id: IC_kwDOBm6k_c5NJISO
created_at: 2022-10-27T23:44:25Z
updated_at: 2022-10-27T23:44:25Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1870?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Base: **92.55**% // Head: **92.55**% // No change to project coverage :thumbsup:
  > Coverage data is based on head [(`4faa4fd`)](https://codecov.io/gh/simonw/datasette/pull/1870?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`bf00b0b`)](https://codecov.io/gh/simonw/datasette/commit/bf00b0b59b6692bdec597ac9db4e0b497c5a47b4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > Patch has no changes to coverable lines.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main    #1870   +/-   ##
  =======================================
    Coverage   92.55%   92.55%           
  =======================================
    Files          35       35           
    Lines        4432     4432           
  =======================================
    Hits         4102     4102           
    Misses        330      330           
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1870?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1870/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.30% <ø> (ø)` | |

  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  </details>

  [:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1870?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1851#issuecomment-1294281451
issue_url: https://api.github.com/repos/simonw/datasette/issues/1851
node_id: IC_kwDOBm6k_c5NJSrr
created_at: 2022-10-28T00:59:25Z
updated_at: 2022-10-28T00:59:25Z
author_association: OWNER
body:
  I'm going to use this endpoint for bulk inserts too, so I'm closing this issue and continuing the work here:
  - #1866

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1866#issuecomment-1294282263
issue_url: https://api.github.com/repos/simonw/datasette/issues/1866
node_id: IC_kwDOBm6k_c5NJS4X
created_at: 2022-10-28T01:00:42Z
updated_at: 2022-10-28T01:00:42Z
author_association: OWNER
body:
  I'm going to set the limit at 1,000 rows inserted at a time. I'll make this configurable using a new `max_insert_rows` setting (for consistency with `max_returned_rows`).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1870#issuecomment-1294285471
issue_url: https://api.github.com/repos/simonw/datasette/issues/1870
node_id: IC_kwDOBm6k_c5NJTqf
created_at: 2022-10-28T01:06:03Z
updated_at: 2022-10-28T01:06:03Z
author_association: CONTRIBUTOR
body:
  as far as i can tell, [this is where the "immutable" argument is used](https://github.com/sqlite/sqlite/blob/c97bb14fab566f6fa8d967c8fd1e90f3702d5b73/src/pager.c#L4926-L4931) in sqlite:

  ```c
        pPager->noLock = sqlite3_uri_boolean(pPager->zFilename, "nolock", 0);
        if( (iDc & SQLITE_IOCAP_IMMUTABLE)!=0
         || sqlite3_uri_boolean(pPager->zFilename, "immutable", 0) ){
            vfsFlags |= SQLITE_OPEN_READONLY;
            goto act_like_temp_file;
        }
  ```

  so it does set the read only flag, but then has a goto.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1866#issuecomment-1294296767
issue_url: https://api.github.com/repos/simonw/datasette/issues/1866
node_id: IC_kwDOBm6k_c5NJWa_
created_at: 2022-10-28T01:22:25Z
updated_at: 2022-10-28T01:23:09Z
author_association: OWNER
body:
  Nasty catch on this one: I wanted to return the IDs of the freshly inserted rows. But... the `insert_all()` method I was planning to use from `sqlite-utils` doesn't appear to have a way of doing that:

  https://github.com/simonw/sqlite-utils/blob/529110e7d8c4a6b1bbf5fb61f2e29d72aa95a611/sqlite_utils/db.py#L2813-L2835

  SQLite itself added a `RETURNING` statement which might help, but that is only available from version 3.35 released in March 2021: https://www.sqlite.org/lang_returning.html - which isn't commonly available yet. https://latest.datasette.io/-/versions right now shows 3.34, and https://lite.datasette.io/#/-/versions shows 3.27.2 (from Feb 2019).

  Two options then:

  1. Even for bulk inserts do one insert at a time so I can use `cursor.lastrowid` to get the ID of the inserted record. This isn't terrible since SQLite is very fast, but it may still be a big performance hit for large inserts.
  2. Don't return the list of inserted rows for bulk inserts
  3. Default to not returning the list of inserted rows for bulk inserts, but allow the user to request that - in which case we use the slower path

  That third option might be the way to go here.

  I should benchmark first to figure out how much of a difference this actually makes.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1866#issuecomment-1294306071
issue_url: https://api.github.com/repos/simonw/datasette/issues/1866
node_id: IC_kwDOBm6k_c5NJYsX
created_at: 2022-10-28T01:37:14Z
updated_at: 2022-10-28T01:37:59Z
author_association: OWNER
body:
  Quick crude benchmark:
  ```python
  import sqlite3

  db = sqlite3.connect(":memory:")

  def create_table(db, name):
      db.execute(f"create table {name} (id integer primary key, title text)")

  create_table(db, "single")
  create_table(db, "multi")
  create_table(db, "bulk")

  def insert_singles(titles):
      inserted = []
      for title in titles:
          cursor = db.execute(f"insert into single (title) values (?)", [title])
          inserted.append((cursor.lastrowid, title))
      return inserted


  def insert_many(titles):
      db.executemany(f"insert into multi (title) values (?)", ((t,) for t in titles))


  def insert_bulk(titles):
      db.execute("insert into bulk (title) values {}".format(
          ", ".join("(?)" for _ in titles)
      ), titles)

  titles = ["title {}".format(i) for i in range(1, 10001)]
  ```
  Then in iPython I ran these:
  ```
  In [14]: %timeit insert_singles(titles)
  23.8 ms ± 535 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

  In [13]: %timeit insert_many(titles)
  12 ms ± 520 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

  In [12]: %timeit insert_bulk(titles)
  2.59 ms ± 25 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
  ```
  So the bulk insert really is a lot faster - 3ms compared to 24ms for single inserts, so ~8x faster.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1866#issuecomment-1294316640
issue_url: https://api.github.com/repos/simonw/datasette/issues/1866
node_id: IC_kwDOBm6k_c5NJbRg
created_at: 2022-10-28T01:51:40Z
updated_at: 2022-10-28T01:51:40Z
author_association: OWNER
body:
  This needs to support the following:
  - Rows do not include a primary key - one is assigned by the database
  - Rows provide their own primary key, any clashes are errors
  - Rows provide their own primary key, clashes are silently ignored
  - Rows provide their own primary key, replacing any existing records

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/496#issuecomment-1294408928
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/496
node_id: IC_kwDOCGYnMM5NJxzg
created_at: 2022-10-28T03:36:56Z
updated_at: 2022-10-28T03:37:50Z
author_association: NONE
body:
  With respect to the typing of Table class itself, my interim solution:

  ```python
  from sqlite_utils.db import Table
  def tbl(self, table_name: str) -> Table:
      tbl = self.db[table_name]
      if isinstance(tbl, Table):
          return tbl
      raise Exception(f"Missing {table_name=}")
  ```

  With respect to @chapmanjacobd concern on the `DEFAULT` being an empty class, have also been using `# type: ignore`, e.g.

  ```python
  @classmethod
  def insert_list(cls, areas: list[str]):
      return meta.tbl(meta.Areas).insert_all(
          ({"area": a} for a in areas), ignore=True  # type: ignore
      )
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1866#issuecomment-1295200988
issue_url: https://api.github.com/repos/simonw/datasette/issues/1866
node_id: IC_kwDOBm6k_c5NMzLc
created_at: 2022-10-28T16:29:55Z
updated_at: 2022-10-28T16:29:55Z
author_association: OWNER
body:
  I wonder if there's something clever I could do here within a transaction?

  Start a transaction. Write out a temporary in-memory table with all of the existing primary keys in the table. Run the bulk insert. Then run `select pk from table where pk not in (select pk from old_pks)` to see what has changed.

  I don't think that's going to work well for large tables.

  I'm going to go with not returning inserted rows by default, unless you pass a special option requesting that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1870#issuecomment-1295657771
issue_url: https://api.github.com/repos/simonw/datasette/issues/1870
node_id: IC_kwDOBm6k_c5NOisr
created_at: 2022-10-29T00:19:03Z
updated_at: 2022-10-29T00:19:03Z
author_association: OWNER
body:
  Just saw your comment here: https://github.com/simonw/datasette/issues/1836#issuecomment-1272357976

  > when you are running from docker, you **always** will want to run as `mode=ro` because the same thing that is causing duplication in the inspect layer will cause duplication in the final container read/write layer when `datasette serve` runs.

  I don't understand this. My mental model of how Docker works is that the image itself is created using `docker build`... but then when the image runs later on (`docker run`) the image itself isn't touched at all.

  Are you saying that I can build a container, but then when I run it and it does `datasette serve -i data.db ...` it will somehow modify the image, or create a new modified filesystem layer in the runtime environment, as a result of running that `serve` command?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1870#issuecomment-1295660092
issue_url: https://api.github.com/repos/simonw/datasette/issues/1870
node_id: IC_kwDOBm6k_c5NOjQ8
created_at: 2022-10-29T00:25:26Z
updated_at: 2022-10-29T00:25:26Z
author_association: OWNER
body:
  Saw your comment here too: https://github.com/simonw/datasette/issues/1480#issuecomment-1271101072

  > switching from `immutable=1` to `mode=ro` completely addressed this. see https://github.com/simonw/datasette/issues/1836#issuecomment-1271100651 for details.

  So maybe we need a special case for containers that are intended to be run using Docker - the ones produced by `datasette package` and `datasette publish cloudrun`? Those are cases where the `-i` option should actually be opened in read-only mode, not immutable mode.

  Maybe a `datasette serve --irw data.db` option for opening a file in immutable-but-actually-read-only mode? Bit ugly though.

  I should run some benchmarks to figure out if `immutable` really does offer significant performance benefits.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1870#issuecomment-1295667649
issue_url: https://api.github.com/repos/simonw/datasette/issues/1870
node_id: IC_kwDOBm6k_c5NOlHB
created_at: 2022-10-29T00:52:43Z
updated_at: 2022-10-29T00:53:43Z
author_association: CONTRIBUTOR
body:
  > Are you saying that I can build a container, but then when I run it and it does `datasette serve -i data.db ...` it will somehow modify the image, or create a new modified filesystem layer in the runtime environment, as a result of running that `serve` command?

  Somehow, `datasette serve -i data.db` will lead to the `data.db` being modified, which will trigger a [copy-on-write](https://docs.docker.com/storage/storagedriver/#the-copy-on-write-cow-strategy) of `data.db` into the read-write layer of the container.

  I don't understand **how** that happens.

  it kind of feels like a bug in sqlite, but i can't quite follow the sqlite code.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1872#issuecomment-1296076803
issue_url: https://api.github.com/repos/simonw/datasette/issues/1872
node_id: IC_kwDOBm6k_c5NQJAD
created_at: 2022-10-30T02:50:34Z
updated_at: 2022-10-30T02:50:34Z
author_association: CONTRIBUTOR
body:
  should this issue be under https://github.com/simonw/datasette-publish-vercel/issues ?

  Perhaps I just need to update: 
  datasette-publish-vercel==0.11
  in requirements.txt?
   
   I'll try that and see what happens...
   

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1872#issuecomment-1296080804
issue_url: https://api.github.com/repos/simonw/datasette/issues/1872
node_id: IC_kwDOBm6k_c5NQJ-k
created_at: 2022-10-30T03:06:32Z
updated_at: 2022-10-30T03:06:32Z
author_association: CONTRIBUTOR
body:
  I updated datasette-publish-vercel to  0.14.2 in requirements.txt

  And the site is back up!

  Is there a way that we can get some sort of notice when something like this will have critical impact on website function?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1871#issuecomment-1296114136
issue_url: https://api.github.com/repos/simonw/datasette/issues/1871
node_id: IC_kwDOBm6k_c5NQSHY
created_at: 2022-10-30T05:15:40Z
updated_at: 2022-10-30T05:15:40Z
author_association: OWNER
body:
  Host it at `/-/api`

  It's an input box with a path in and a textarea you can put JSON in, plus a submit button to post the request.

  It lists the API endpoints you can use - click on a link to populate the form field plus a example.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1871#issuecomment-1296130073
issue_url: https://api.github.com/repos/simonw/datasette/issues/1871
node_id: IC_kwDOBm6k_c5NQWAZ
created_at: 2022-10-30T06:20:56Z
updated_at: 2022-10-30T06:20:56Z
author_association: OWNER
body:
  That initial prototype looks like this:

  <img width="560" alt="image" src="https://user-images.githubusercontent.com/9599/198865445-e5dd8c12-6504-47e0-98e5-2dfde25052f4.png">

  It currently shows the returned JSON from the API in an `alert()`. Next I should make that part of the page instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1872#issuecomment-1296131343
issue_url: https://api.github.com/repos/simonw/datasette/issues/1872
node_id: IC_kwDOBm6k_c5NQWUP
created_at: 2022-10-30T06:26:01Z
updated_at: 2022-10-30T06:26:01Z
author_association: OWNER
body:
  Good spot fixing that!

  Sorry about this - it was a change in Datasette 0.63 which should have been better called out.

  My goal for Datasette 1.0 (which I aim to have out by the end of the year) is to introduce a formal process for avoiding problems like this, with very clear documentation when something like this might happen.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1873#issuecomment-1296131681
issue_url: https://api.github.com/repos/simonw/datasette/issues/1873
node_id: IC_kwDOBm6k_c5NQWZh
created_at: 2022-10-30T06:27:12Z
updated_at: 2022-10-30T06:27:12Z
author_association: OWNER
body:
  Relevant TODO: https://github.com/simonw/datasette/blob/c35859ae3df163406f1a1895ccf9803e933b2d8e/datasette/views/table.py#L1131-L1135

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1871#issuecomment-1296339205
issue_url: https://api.github.com/repos/simonw/datasette/issues/1871
node_id: IC_kwDOBm6k_c5NRJEF
created_at: 2022-10-30T20:02:05Z
updated_at: 2022-10-30T20:02:05Z
author_association: OWNER
body:
  Realized the API explorer doesn't need the API key piece at all - it can work with standard cookie-based auth.

  This also reflects how most plugins are likely to use this API, where they'll be adding JavaScript that uses `fetch()` to call the write API directly.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1871#issuecomment-1296339386
issue_url: https://api.github.com/repos/simonw/datasette/issues/1871
node_id: IC_kwDOBm6k_c5NRJG6
created_at: 2022-10-30T20:03:04Z
updated_at: 2022-10-30T20:03:04Z
author_association: OWNER
body:
  I do need to skip CSRF for these API calls. I'm going to start out by doing that using the `skip_csrf()` hook to skip CSRF checks on anything with a `content-type: application/json` request header.

  ```python
  @hookimpl
  def skip_csrf(scope):
      if scope["type"] == "http":
          headers = scope.get("headers")
          if dict(headers).get(b'content-type') == b'application/json':
              return True
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1873#issuecomment-1296341055
issue_url: https://api.github.com/repos/simonw/datasette/issues/1873
node_id: IC_kwDOBm6k_c5NRJg_
created_at: 2022-10-30T20:11:47Z
updated_at: 2022-10-30T20:12:30Z
author_association: OWNER
body:
  If a table has an auto-incrementing primary key, should you be allowed to insert records with an explicit key into it?

  I'm torn on this one. It's something you can do with direct database access, but it's something I very rarely want to do.

  I'm inclined to disallow it and say that if you want that you can get it using a writable canned query instead.

  Likewise, I'm not going to provide a way to set the `rowid` explicitly on a freshly inserted row.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1873#issuecomment-1296341469
issue_url: https://api.github.com/repos/simonw/datasette/issues/1873
node_id: IC_kwDOBm6k_c5NRJnd
created_at: 2022-10-30T20:13:50Z
updated_at: 2022-10-30T20:13:50Z
author_association: OWNER
body:
  I checked and SQLite itself does allow you to set the `rowid` on that kind of table - it then increments from whatever you inserted:
  ```
  % sqlite3 /tmp/t.db
  SQLite version 3.39.4 2022-09-07 20:51:41
  Enter ".help" for usage hints.
  sqlite> create table docs (title text);
  sqlite> insert into docs (title) values ('one');
  sqlite> select rowid, title from docs;
  1|one
  sqlite> insert into docs (rowid, title) values (3, 'three');
  sqlite> select rowid, title from docs;
  1|one
  3|three
  sqlite> insert into docs (title) values ('another');
  sqlite> select rowid, title from docs;
  1|one
  3|three
  4|another
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1873#issuecomment-1296342814
issue_url: https://api.github.com/repos/simonw/datasette/issues/1873
node_id: IC_kwDOBm6k_c5NRJ8e
created_at: 2022-10-30T20:20:05Z
updated_at: 2022-10-30T20:20:05Z
author_association: OWNER
body:
  Some notes on what Datasette does already

  https://latest.datasette.io/fixtures/tags.json?_shape=array returns:
  ```json
  [
    {
      "tag": "canine"
    },
    {
      "tag": "feline"
    }
  ]
  ```
  That table is defined [like this](https://latest.datasette.io/fixtures/tags):
  ```sql
  CREATE TABLE tags (
      tag TEXT PRIMARY KEY
  );
  ```
  Here's a `rowid` table with no explicit primary key: https://latest.datasette.io/fixtures/binary_data

  https://latest.datasette.io/fixtures/binary_data.json?_shape=array
  ```json
  [
    {
      "rowid": 1,
      "data": {
        "$base64": true,
        "encoded": "FRwCx60F/g=="
      }
    },
    {
      "rowid": 2,
      "data": {
        "$base64": true,
        "encoded": "FRwDx60F/g=="
      }
    },
    {
      "rowid": 3,
      "data": null
    }
  ]
  ```
  ```sql
  CREATE TABLE binary_data (
      data BLOB
  );
  ```
  https://latest.datasette.io/fixtures/simple_primary_key has a text primary key:

  https://latest.datasette.io/fixtures/simple_primary_key.json?_shape=array
  ```json
  [
    {
      "id": "1",
      "content": "hello"
    },
    {
      "id": "2",
      "content": "world"
    },
    {
      "id": "3",
      "content": ""
    },
    {
      "id": "4",
      "content": "RENDER_CELL_DEMO"
    },
    {
      "id": "5",
      "content": "RENDER_CELL_ASYNC"
    }
  ]
  ```
  ```sql
  CREATE TABLE simple_primary_key (
    id varchar(30) primary key,
    content text
  );
  ```
  https://latest.datasette.io/fixtures/compound_primary_key is a compound primary key.

  https://latest.datasette.io/fixtures/compound_primary_key.json?_shape=array
  ```json
  [
    {
      "pk1": "a",
      "pk2": "b",
      "content": "c"
    },
    {
      "pk1": "a/b",
      "pk2": ".c-d",
      "content": "c"
    }
  ]
  ```
  ```sql
  CREATE TABLE compound_primary_key (
    pk1 varchar(30),
    pk2 varchar(30),
    content text,
    PRIMARY KEY (pk1, pk2)
  );
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1873#issuecomment-1296343014
issue_url: https://api.github.com/repos/simonw/datasette/issues/1873
node_id: IC_kwDOBm6k_c5NRJ_m
created_at: 2022-10-30T20:21:01Z
updated_at: 2022-10-30T20:21:01Z
author_association: OWNER
body:
  Actually, for simplicity I'm going to say that you can always set the primary key, even for auto-incrementing primary key columns... but you cannot set it on pure `rowid` columns.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1873#issuecomment-1296343173
issue_url: https://api.github.com/repos/simonw/datasette/issues/1873
node_id: IC_kwDOBm6k_c5NRKCF
created_at: 2022-10-30T20:21:54Z
updated_at: 2022-10-30T20:22:20Z
author_association: OWNER
body:
  One last case to consider: `WITHOUT ROWID` tables.

  https://www.sqlite.org/withoutrowid.html

  > By default, every row in SQLite has a special column, usually called the "[rowid](https://www.sqlite.org/lang_createtable.html#rowid)", that uniquely identifies that row within the table. However if the phrase "WITHOUT ROWID" is added to the end of a [CREATE TABLE](https://www.sqlite.org/lang_createtable.html) statement, then the special "rowid" column is omitted. There are sometimes space and performance advantages to omitting the rowid.
  >
  > ...
  >
  > Every WITHOUT ROWID table must have a [PRIMARY KEY](https://www.sqlite.org/lang_createtable.html#primkeyconst). An error is raised if a CREATE TABLE statement with the WITHOUT ROWID clause lacks a PRIMARY KEY.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1873#issuecomment-1296343317
issue_url: https://api.github.com/repos/simonw/datasette/issues/1873
node_id: IC_kwDOBm6k_c5NRKEV
created_at: 2022-10-30T20:22:40Z
updated_at: 2022-10-30T20:22:40Z
author_association: OWNER
body:
  So maybe they're not actually worth worrying about separately, because they are guaranteed to have a primary key set.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1873#issuecomment-1296343716
issue_url: https://api.github.com/repos/simonw/datasette/issues/1873
node_id: IC_kwDOBm6k_c5NRKKk
created_at: 2022-10-30T20:24:55Z
updated_at: 2022-10-30T20:24:55Z
author_association: OWNER
body:
  I think the key feature I need here is going to be the equivalent of `ignore=True` and `replace=True` for dealing with primary key collisions, see https://sqlite-utils.datasette.io/en/stable/reference.html#sqlite_utils.db.Table.insert

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1864#issuecomment-1296375536
issue_url: https://api.github.com/repos/simonw/datasette/issues/1864
node_id: IC_kwDOBm6k_c5NRR7w
created_at: 2022-10-30T23:17:11Z
updated_at: 2022-10-30T23:17:11Z
author_association: OWNER
body:
  I'm a bit nervous about calling `.delete()` with the `pk_values` - can I be sure they are in the correct order? https://github.com/simonw/datasette/blob/00632ded30e7cf9f0cf9478680645d1dabe269ae/datasette/views/row.py#L188-L190

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1864#issuecomment-1296402071
issue_url: https://api.github.com/repos/simonw/datasette/issues/1864
node_id: IC_kwDOBm6k_c5NRYaX
created_at: 2022-10-31T00:37:09Z
updated_at: 2022-10-31T00:37:09Z
author_association: OWNER
body:
  I need to think about what happens if you delete a row that is the target of a foreign key from another row.

  https://www.sqlite.org/foreignkeys.html#fk_enable shows that SQLite will only actively enforce these relationships (e.g. throw an error if you try to delete a row that is referenced by another row) if you first run `PRAGMA foreign_keys = ON;` against the connection.

  > Foreign key constraints are disabled by default (for backwards compatibility), so must be enabled separately for each [database connection](https://www.sqlite.org/c3ref/sqlite3.html). (Note, however, that future releases of SQLite might change so that foreign key constraints enabled by default. Careful developers will not make any assumptions about whether or not foreign keys are enabled by default but will instead enable or disable them as necessary.)

  I don't actually believe that the SQLite maintainers will ever make that the default though.

  Datasette doesn't turn these on at the moment, but it could be turned on by a `prepare_connection()` plugin.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1864#issuecomment-1296403316
issue_url: https://api.github.com/repos/simonw/datasette/issues/1864
node_id: IC_kwDOBm6k_c5NRYt0
created_at: 2022-10-31T00:39:43Z
updated_at: 2022-10-31T00:39:43Z
author_association: OWNER
body:
  It looks like SQLite has features for this already: https://www.sqlite.org/foreignkeys.html#fk_actions

  > Foreign key ON DELETE and ON UPDATE clauses are used to configure actions that take place when deleting rows from the parent table (ON DELETE), or modifying the parent key values of existing rows (ON UPDATE). A single foreign key constraint may have different actions configured for ON DELETE and ON UPDATE. Foreign key actions are similar to triggers in many ways. 

  On that basis, I'm not going to implement anything additional in the `.../-/delete` endpoint relating to foreign keys. Developers who want special treatment of them can do that with a combination of a plugin (maybe I'll build a `datasette-enable-foreign-keys` plugin) and tables created using those `ON DELETE` clauses.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/448#issuecomment-1297703307
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/448
node_id: IC_kwDOCGYnMM5NWWGL
created_at: 2022-10-31T21:23:51Z
updated_at: 2022-10-31T21:27:32Z
author_association: CONTRIBUTOR
body:
  The Windows aspect is a red herring: OP's sample above produces the same error on Linux. (Though I don't know what's going on with the CI).

  The same error can also be obtained by passing an `io` from a file opened in non-binary mode (`'r'` as opposed to `'rb'`) to `rows_from_file()`. This is how I got here.

  The fix for my case is easy: open the file in mode `'rb'`. The analagous fix for OP's problem also works: use `BytesIO` in place of `StringIO`.

  Minimal test case (derived from [utils.py](https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/utils.py#L304)):

  ``` python
  import io
  from typing import cast

  #fp = io.StringIO("id,name\n1,Cleo") # error
  fp = io.BytesIO(bytes("id,name\n1,Cleo", encoding='utf-8')) # okay
  reader = io.BufferedReader(cast(io.RawIOBase, fp))
  reader.peek(1) # exception thrown here
  ```
  I see the signature of `rows_from_file()` correctly has `fp: BinaryIO` but I guess you'd need either a runtime type check for that (not all `io`s have `mode()`), or to catch the `AttributeError` on `peek()` to produce a better error for users. Neither option is ideal.

  Some thoughts on testing binary-ness of `io`s in this SO question: https://stackoverflow.com/questions/44584829/how-to-determine-if-file-is-opened-in-binary-or-text-mode

reactions:
[cell repeats an earlier one in this file: 112 characters]


