# issue_comments

html_url: https://github.com/simonw/sqlite-utils/issues/356#issuecomment-991517209
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/356
node_id: IC_kwDOCGYnMM47GVoZ
created_at: 2021-12-11T07:46:41Z
updated_at: 2021-12-11T07:46:41Z
author_association: OWNER
body:
  By default this will accept single lines, but maybe there could be a `--all` option which instead grabs all of stdin into a single string against which the conversion function runs - like `git-history file`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1549#issuecomment-991752486
issue_url: https://api.github.com/repos/simonw/datasette/issues/1549
node_id: IC_kwDOBm6k_c47HPEm
created_at: 2021-12-11T19:09:15Z
updated_at: 2021-12-11T19:09:15Z
author_association: OWNER
body:
  That's what this option does:

  ![EAB1B9E8-38E9-4C6D-8854-BD1935F163D9](https://user-images.githubusercontent.com/9599/145688531-668bafa1-e287-4bbd-84d6-157241fb1f68.jpeg)

  The usability of this is pretty terrible though (including "stream all rows" - how are people meant to understand what that does?) so it can definitely do with some rethinking.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1549#issuecomment-991754237
issue_url: https://api.github.com/repos/simonw/datasette/issues/1549
node_id: IC_kwDOBm6k_c47HPf9
created_at: 2021-12-11T19:14:39Z
updated_at: 2021-12-11T19:14:39Z
author_association: CONTRIBUTOR
body:
  that option is not available on [custom queries](https://labordata.bunkum.us/odpr-962a140?sql=with+local_union_filings+as+%28%0D%0A++select+*+from+lm_data+%0D%0A++where%0D%0A++++yr_covered+%3E+cast%28strftime%28%27%25Y%27%2C+%27now%27%2C+%27-5+years%27%29+as+int%29%0D%0A++++and+desig_name+%3D+%27LU%27%0D%0A++order+by+yr_covered+desc%0D%0A%29%2C%0D%0Amost_recent_filing+as+%28%0D%0A++select%0D%0A++++*%0D%0A++from+local_union_filings%0D%0A++group+by%0D%0A++++f_num%0D%0A%29%0D%0Aselect%0D%0A++*%0D%0Afrom%0D%0A++most_recent_filing%0D%0Awhere%0D%0A++next_election+%3E%3D+strftime%28%27%25Y-%25m%27%2C+%27now%27%29%0D%0A++and+next_election+%3C+strftime%28%27%25Y-%25m%27%2C+%27now%27%2C+%27%2B1+year%27%29%0D%0Aorder+by%0D%0A++members+desc%3B).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1549#issuecomment-991754794
issue_url: https://api.github.com/repos/simonw/datasette/issues/1549
node_id: IC_kwDOBm6k_c47HPoq
created_at: 2021-12-11T19:16:33Z
updated_at: 2021-12-11T19:16:33Z
author_association: OWNER
body:
  Good call! I'm doing a refactor #1518 right now which will hopefully bring the functionality of those two much closer - I'll make a note to consider this there too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1550#issuecomment-991761635
issue_url: https://api.github.com/repos/simonw/datasette/issues/1550
node_id: IC_kwDOBm6k_c47HRTj
created_at: 2021-12-11T19:39:01Z
updated_at: 2021-12-11T19:39:01Z
author_association: OWNER
body:
  I wonder if this could work for public instances too with some kind of queuing mechanism?

  I really need to use benchmarking to figure out what the right number of maximum SQLite connections is. I'm just guessing at the moment.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1550#issuecomment-991805516
issue_url: https://api.github.com/repos/simonw/datasette/issues/1550
node_id: IC_kwDOBm6k_c47HcBM
created_at: 2021-12-11T23:43:24Z
updated_at: 2021-12-11T23:43:24Z
author_association: OWNER
body:
  I built a tiny Starlette app to experiment with this a bit:
  ```python
  import asyncio
  import janus
  from starlette.applications import Starlette
  from starlette.responses import JSONResponse, HTMLResponse, StreamingResponse
  from starlette.routing import Route
  import sqlite3
  from concurrent import futures

  executor = futures.ThreadPoolExecutor(max_workers=10)


  async def homepage(request):
      return HTMLResponse(
          """
          <html>
          <head><title>SQL CSV Server</title>
          <style>body { width: 40rem; font-family: helvetica; margin: 2em auto; }</style>
          <body>
          <h1>SQL CSV Server</h1>
          <form action="/csv">
          <label style="display: block">SQL query:
          <textarea style="width: 90%; height: 20em" name="sql"></textarea>
          </label>
          <input type="submit" value="Run query">
          </form>
          </head>
          """
      )


  def run_query_in_thread(sql, sync_q):
      db = sqlite3.connect("../datasette/covid.db")
      cursor = db.cursor()
      cursor.arraysize = 100 # Default is 1 apparently?
      cursor.execute(sql)
      columns = [d[0] for d in cursor.description]
      sync_q.put([columns])
      # Now start putting batches of rows
      while True:
          rows = cursor.fetchmany()
          if rows:
              sync_q.put(rows)
          else:
              break
      # Let queue know we are finished\
      sync_q.put(None)


  async def csv_query(request):
      sql = request.query_params["sql"]

      queue = janus.Queue()
      loop = asyncio.get_running_loop()

      async def csv_generator():
          loop.run_in_executor(None, run_query_in_thread, sql, queue.sync_q)
          while True:
              rows = await queue.async_q.get()
              if rows is not None:
                  for row in rows:
                      yield ",".join(map(str, row)) + "\n "
                  queue.async_q.task_done()
              else:
                  # Cleanup
                  queue.close()
                  await queue.wait_closed()
                  break

      return StreamingResponse(csv_generator(), media_type='text/plain')


  app = Starlette(
      debug=True,
      routes=[
          Route("/", homepage),
          Route("/csv", csv_query),
      ],
  )
  ```
  But.. if I run this in a terminal window:
  ```
  /tmp % wget 'http://127.0.0.1:8000/csv?sql=select+*+from+ny_times_us_counties'
  ```
  it takes about 20 seconds to run and returns a 50MB file - but while it is running no other requests can be served by that server - not even the homepage! So something is blocking the event loop.

  Maybe I should be using `fut = loop.run_in_executor(None, run_query_in_thread, sql, queue.sync_q)` and then awaiting `fut` somewhere, like in the Janus documentation? Don't think that's needed though. Needs more work to figure out why this is blocking.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-991819781
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c47HfgF
created_at: 2021-12-12T01:53:10Z
updated_at: 2021-12-12T01:53:10Z
author_association: OWNER
body:
  I have a hunch that the conclusion of this experiment may end up being that the `asyncinject` trick is kinda neat but the code will be easier to maintain (while still executing in parallel) if it's written using `asyncio.gather` directly instead.

  It's possible `asyncinject` will end up being neat enough that I'll want to keep it though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-991822853
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c47HgQF
created_at: 2021-12-12T02:24:00Z
updated_at: 2021-12-12T02:24:00Z
author_association: OWNER
body:
  Rebuilding `TableView` from the ground up is proving not to be much fun. I'm going to explore starting the refactor of the existing code by separating out the bit that generates the SQL query from the rest of it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-991823001
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c47HgSZ
created_at: 2021-12-12T02:25:32Z
updated_at: 2021-12-12T02:25:32Z
author_association: OWNER
body:
  The tests for `TableView` are currently mixed in with everything else in `tests/test_api.py` and `tests/html.py` - might be good to split those out into `test_table_html.py` and `test_table_api.py` since they're such a key part of how Datasette works.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-991827468
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c47HhYM
created_at: 2021-12-12T03:15:00Z
updated_at: 2021-12-12T03:15:00Z
author_association: OWNER
body:
   I don't think this code is necessary any more: https://github.com/simonw/datasette/blob/492f9835aa7e90540dd0c6324282b109f73df71b/datasette/views/table.py#L396-L399

  That dates back from when Datasette was built on top of Sanic and Sanic didn't preserve those query parameters the way I needed it to:

  https://github.com/simonw/datasette/blob/1f69269fe93e4cd42e56890126cc0dbcf719c6cb/datasette/views/table.py#L202-L206

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-991828014
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c47Hhgu
created_at: 2021-12-12T03:21:35Z
updated_at: 2021-12-12T03:21:35Z
author_association: OWNER
body:
  No, removing that gave me the following test failure:
  ```
  tests/test_table_api.py::test_table_filter_queries[/fixtures/simple_primary_key.json?content__exact=-expected_rows2] FAILED                                       [100%]

  =============================================================================== FAILURES ================================================================================
  ______________________________________ test_table_filter_queries[/fixtures/simple_primary_key.json?content__exact=-expected_rows2] ______________________________________

  app_client = <datasette.utils.testing.TestClient object at 0x10d45d2d0>, path = '/fixtures/simple_primary_key.json?content__exact=', expected_rows = [['3', '']]

      @pytest.mark.parametrize(
          "path,expected_rows",
          [
              ("/fixtures/simple_primary_key.json?content=hello", [["1", "hello"]]),
              (
                  "/fixtures/simple_primary_key.json?content__contains=o",
                  [
                      ["1", "hello"],
                      ["2", "world"],
                      ["4", "RENDER_CELL_DEMO"],
                  ],
              ),
              ("/fixtures/simple_primary_key.json?content__exact=", [["3", ""]]),
              (
                  "/fixtures/simple_primary_key.json?content__not=world",
                  [
                      ["1", "hello"],
                      ["3", ""],
                      ["4", "RENDER_CELL_DEMO"],
                      ["5", "RENDER_CELL_ASYNC"],
                  ],
              ),
          ],
      )
      def test_table_filter_queries(app_client, path, expected_rows):
          response = app_client.get(path)
  >       assert expected_rows == response.json["rows"]
  E       AssertionError: assert [['3', '']] == [['1', 'hello'],\n ['2', 'world'],\n ['3', ''],\n ['4', 'RENDER_CELL_DEMO'],\n ['5', 'RENDER_CELL_ASYNC']]
  E         At index 0 diff: ['3', ''] != ['1', 'hello']
  E         Right contains 4 more items, first extra item: ['2', 'world']
  E         Full diff:
  E           [
  E         -  ['1',
  E         -   'hello'],
  E         -  ['2',
  E         -   'world'],
  E            ['3',
  E             ''],
  E         -  ['4',
  E         -   'RENDER_CELL_DEMO'],
  E         -  ['5',
  E         -   'RENDER_CELL_ASYNC'],
  E           ]

  /Users/simon/Dropbox/Development/datasette/tests/test_table_api.py:511: AssertionError
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1551#issuecomment-991960179
issue_url: https://api.github.com/repos/simonw/datasette/issues/1551
node_id: IC_kwDOBm6k_c47IBxz
created_at: 2021-12-12T19:54:45Z
updated_at: 2021-12-12T19:54:45Z
author_association: OWNER
body:
  This is technically a backwards-incompatible for any plugins that use `request.args` - but it's unlikely to break anything. At any rate this needs to happen before Datasette 1.0!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1551#issuecomment-991960416
issue_url: https://api.github.com/repos/simonw/datasette/issues/1551
node_id: IC_kwDOBm6k_c47IB1g
created_at: 2021-12-12T19:56:12Z
updated_at: 2021-12-12T19:56:12Z
author_association: OWNER
body:
  Python documentation for `parse_qs`: https://docs.python.org/3/library/urllib.parse.html#urllib.parse.parse_qs

  > The optional argument *keep_blank_values* is a flag indicating whether blank values in percent-encoded queries should be treated as blank strings. A true value indicates that blanks should be retained as blank strings. The default false value indicates that blank values are to be ignored and treated as if they were not included.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1551#issuecomment-991960719
issue_url: https://api.github.com/repos/simonw/datasette/issues/1551
node_id: IC_kwDOBm6k_c47IB6P
created_at: 2021-12-12T19:58:17Z
updated_at: 2021-12-12T19:58:17Z
author_association: OWNER
body:
  Here's an example of the difference that causes:
  ```pycon
  >>> import urllib.parse
  >>> urllib.parse.parse_qs("foo=bar")
  {'foo': ['bar']}
  >>> urllib.parse.parse_qs("foo=bar&baz=")
  {'foo': ['bar']}
  >>> urllib.parse.parse_qs("foo=bar&baz=", keep_blank_values=True)
  {'foo': ['bar'], 'baz': ['']}
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-992833868
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c47LXFM
created_at: 2021-12-13T19:59:17Z
updated_at: 2021-12-13T19:59:17Z
author_association: OWNER
body:
  Built a new plugin to help with this work by improving the display of `?_trace=1` output: https://datasette.io/plugins/datasette-pretty-traces

  ![image](https://user-images.githubusercontent.com/9599/145879751-36621f43-ba68-4ccd-b14b-379ed8f2111a.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-993000787
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c47L_1T
created_at: 2021-12-13T23:19:20Z
updated_at: 2021-12-14T17:06:05Z
author_association: OWNER
body:
  Useful old comment here: https://github.com/simonw/datasette/issues/617#issuecomment-552253893

  > As noted in [#621 (comment)](https://github.com/simonw/datasette/issues/621#issuecomment-552253208) a common pattern in this method is blocks of code that append new items to the `where_clauses`, `params` and `extra_human_descriptions` arrays. This is a useful refactoring opportunity.
  > 
  > Code that fits this pattern:
  > 
  >  * The code that builds based on the filters: `where_clauses, params = filters.build_where_clauses(table)` and `human_description_en = filters.human_description_en(extra=extra_human_descriptions)`
  > * Code that handles `?_where=`: `where_clauses.extend(request.args["_where"])` - though note that this also appends to a `extra_wheres_for_ui` array which nothing else uses
  > * The `_through=` code, see [Syntax for ?_through= that works as a form field #621](https://github.com/simonw/datasette/issues/621) for details
  > * The code that deals with `?_search=` FTS
  > 
  > The keyset pagination code modifies `where_clauses` and `params` too, but I don't think it's quite going to work with the same abstraction that would cover the above examples.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1554#issuecomment-993002933
issue_url: https://api.github.com/repos/simonw/datasette/issues/1554
node_id: IC_kwDOBm6k_c47MAW1
created_at: 2021-12-13T23:22:58Z
updated_at: 2021-12-13T23:22:58Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1554?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1554](https://codecov.io/gh/simonw/datasette/pull/1554?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (1d08b46) into [main](https://codecov.io/gh/simonw/datasette/commit/a6ff123de5464806441f6a6f95145c9a83b7f20b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (a6ff123) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1554/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1554?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1554   +/-   ##
  =======================================
    Coverage   91.84%   91.84%           
  =======================================
    Files          34       34           
    Lines        4437     4437           
  =======================================
    Hits         4075     4075           
    Misses        362      362           
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1554?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1554/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `96.04% <ø> (ø)` | |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1554?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1554?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [a6ff123...1d08b46](https://codecov.io/gh/simonw/datasette/pull/1554?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1554#issuecomment-993006521
issue_url: https://api.github.com/repos/simonw/datasette/issues/1554
node_id: IC_kwDOBm6k_c47MBO5
created_at: 2021-12-13T23:28:47Z
updated_at: 2021-12-13T23:28:47Z
author_association: OWNER
body:
  That's frustrating: you can only attach comments to lines that were changed in the PR or are within about 3-4 lines of them:

  ![comments](https://user-images.githubusercontent.com/9599/145905357-5d8873f5-99c9-4b46-b4d5-35d38f5cb686.gif)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/526#issuecomment-993078038
issue_url: https://api.github.com/repos/simonw/datasette/issues/526
node_id: IC_kwDOBm6k_c47MSsW
created_at: 2021-12-14T01:46:52Z
updated_at: 2021-12-14T01:46:52Z
author_association: CONTRIBUTOR
body:
  the nested query idea is very nice, and i stole if for [my client side paginator](https://observablehq.com/d/1d5da3a3c3f2f347#DatasetteClient). However,  it won't do the right thing if the original query orders by random().

  If you go the nested query route, maybe raise a 4XX status code if the query has such a clause?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/621#issuecomment-993813210
issue_url: https://api.github.com/repos/simonw/datasette/issues/621
node_id: IC_kwDOBm6k_c47PGLa
created_at: 2021-12-14T17:30:13Z
updated_at: 2021-12-14T20:23:57Z
author_association: OWNER
body:
  Might be able to create a web form that's unambiguous using:

  `https://latest.datasette.io/fixtures/roadside_attractions?_through.["roadside_attraction_characteristics","characteristic_id"]=1`

  So:

  ```html
  <input type="text" name="_through.[&quot;roadside_attraction_characteristics&quot;,&quot;characteristic_id&quot;]' value="1">
  ```
  I'm pretty confident this is allowed by the HTML specification. This works:

  ```html
  <form action="https://httpbin.org/get">
    <input type="text" name='_through.[&quot;roadside_attraction_characteristics&quot;,&quot;characteristic_id&quot;]' value="1">
    <input type="submit">
  </form>
  ```
  ASGI parsing seems to work too: https://latest-with-plugins.datasette.io/-/asgi-scope?_through.[%22roadside_attraction_characteristics%22%2C%22characteristic_id%22]=1

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1423#issuecomment-993876599
issue_url: https://api.github.com/repos/simonw/datasette/issues/1423
node_id: IC_kwDOBm6k_c47PVp3
created_at: 2021-12-14T18:48:09Z
updated_at: 2021-12-14T18:48:09Z
author_association: NONE
body:
  Great feature.

  But what is the right way to enable this to show up? Currently, it seems I need to edit the URL to add, in the right place, `&_facet_size=max`

  Is there another (easier) way to enable this feature?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/621#issuecomment-993958242
issue_url: https://api.github.com/repos/simonw/datasette/issues/621
node_id: IC_kwDOBm6k_c47Ppli
created_at: 2021-12-14T20:33:25Z
updated_at: 2021-12-14T20:33:56Z
author_association: OWNER
body:
  Alternative idea: since current syntax is:

  `?_through={"table":"roadside_attraction_characteristics","column":"characteristic_id","value":"1"}`

  The form-encoding-friendly syntax could be:

  `?_through.{"table":"roadside_attraction_characteristics","column":"characteristic_id"}=1`

  Which is more consistent than the array proposal:

  `?_through.["roadside_attraction_characteristics","characteristic_id"]=1`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/621#issuecomment-994005634
issue_url: https://api.github.com/repos/simonw/datasette/issues/621
node_id: IC_kwDOBm6k_c47P1KC
created_at: 2021-12-14T21:02:50Z
updated_at: 2021-12-14T21:02:50Z
author_association: OWNER
body:
  This would also mean that an extra text input box could be easily shown on the page.

  https://latest-with-plugins.datasette.io/fixtures/roadside_attractions?_through={"table":"roadside_attraction_characteristics","column":"characteristic_id","value":"1"} but with the annotated box added (and made to look good):

  <img width="737" alt="image" src="https://user-images.githubusercontent.com/9599/146078853-d66e34e0-73c9-4c35-9414-5aff3bf825ee.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-994042389
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c47P-IV
created_at: 2021-12-14T21:35:53Z
updated_at: 2021-12-14T21:35:53Z
author_association: OWNER
body:
  Maybe a better way to approach this would be to focus on the JSON side of things - try to get a basic JSON version with `?_extra=` support working, then eventually build that up to the point where it can power the HTML version.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-994085710
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c47QItO
created_at: 2021-12-14T22:03:16Z
updated_at: 2021-12-14T22:04:28Z
author_association: OWNER
body:
  There are actually four forms of SQL query used by the table page:

  - `from_sql` - just the `from table_name where ...`
  - `sql_no_order_no_limit` - used for faceting, `"select {select_all_columns} from {table_name} {where}"`
  - `sql` - the above but with order and limit clauses: `"select {select_specified_columns} from {table_name} {where}{order_by} limit {page_size}{offset}"`
  - `count_sql` used for the count, built out of `from_sql`: `"select count(*) {from_sql}"`

  I'm tempted to encapsulate those in a `Query` class.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1542#issuecomment-995022217
issue_url: https://api.github.com/repos/simonw/datasette/issues/1542
node_id: IC_kwDOBm6k_c47TtWJ
created_at: 2021-12-15T17:47:07Z
updated_at: 2021-12-15T17:47:07Z
author_association: OWNER
body:
  This does make sense to me. I've been hoping to significantly improve the way JavaScript plugins work - there are some notes on that here:
  - #983 

  Encouraging plugins such as `datasette-cluster-map` to emit events that can then be listened to by other plugins is a really interesting idea that I hadn't considered.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1423#issuecomment-995023410
issue_url: https://api.github.com/repos/simonw/datasette/issues/1423
node_id: IC_kwDOBm6k_c47Ttoy
created_at: 2021-12-15T17:48:40Z
updated_at: 2021-12-15T17:48:40Z
author_association: OWNER
body:
  You've caused me to rethink this feature - I no longer think there's value in only showing these numbers if `?_facet_size=max` as opposed to all of the time. New issue coming up.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1552#issuecomment-995034143
issue_url: https://api.github.com/repos/simonw/datasette/issues/1552
node_id: IC_kwDOBm6k_c47TwQf
created_at: 2021-12-15T18:02:53Z
updated_at: 2021-12-15T18:02:53Z
author_association: OWNER
body:
  This is definitely a missing feature. The "different types of facet" stuff feels incomplete to me generally - this is one issue, but this one as well:

  - #625

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1552#issuecomment-995296725
issue_url: https://api.github.com/repos/simonw/datasette/issues/1552
node_id: IC_kwDOBm6k_c47UwXV
created_at: 2021-12-15T23:29:32Z
updated_at: 2021-12-15T23:29:32Z
author_association: CONTRIBUTOR
body:
  @simonw thank you for your fast answer and your guidance!

  While digging into the code, I found an undocumented way of doing it:

  ```yaml
  facets: ["Facet for a column", {"array": "Facet for an array"}]
  ```

  The only remaining problem with that solution is here: https://github.com/simonw/datasette/blob/250db8192cb8aba5eb8cd301ccc2a49525bc3d24/datasette/facets.py#L33

  We have:

  ```python
  type, metadata_config = metadata_config.items()[0]
  ```

  But it requires to cast the `dict_items` as a list prior to access the first element:

  ```python
  type, metadata_config = list(metadata_config.items())[0]
  ```

  I guess it's an unspotted bug? (I mean, independently of the facets-with-arrays issue.)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1552#issuecomment-996034408
issue_url: https://api.github.com/repos/simonw/datasette/issues/1552
node_id: IC_kwDOBm6k_c47Xkdo
created_at: 2021-12-16T17:37:37Z
updated_at: 2021-12-16T17:37:37Z
author_association: OWNER
body:
  I think you're right! I had completely forgotten that piece of code.

  This just turned into a bug fix and a documentation update. Thanks for the research!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1552#issuecomment-996045776
issue_url: https://api.github.com/repos/simonw/datasette/issues/1552
node_id: IC_kwDOBm6k_c47XnPQ
created_at: 2021-12-16T17:52:54Z
updated_at: 2021-12-16T17:52:54Z
author_association: OWNER
body:
  I tried that fix you suggested and now this `metadata.json` does the right thing:
  ```json
  {
      "databases": {
          "fixtures": {
              "tables": {
                  "facetable": {
                      "facets": [
                          {
                              "array": "tags"
                          }
                      ]
                  }
              }
          }
      }
  }
  ```
  It does further highlight the bug in #625 though - since then if you try to add `?_facet=tags` to facet by tags treating them NOT as an array your request to do so is ignored.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1552#issuecomment-996046304
issue_url: https://api.github.com/repos/simonw/datasette/issues/1552
node_id: IC_kwDOBm6k_c47XnXg
created_at: 2021-12-16T17:53:40Z
updated_at: 2021-12-16T18:16:12Z
author_association: OWNER
body:
  I'm also not convinced that this configuration syntax is right. It's a bit weird having a `"facets"` list that can either by column-name-strings or `{"type-of-facet": "column-name"}` objects. Maybe there's a better design for this?

  Part of the problem here is that facets were designed to accept optional extra configuration - partly to support `m2m` facets in #495 - but I haven't actually shipped any facets that use that ability.

  Facet by delimiter would be a good one to exercise that ability:
  - #510

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1552#issuecomment-996076373
issue_url: https://api.github.com/repos/simonw/datasette/issues/1552
node_id: IC_kwDOBm6k_c47XutV
created_at: 2021-12-16T18:35:40Z
updated_at: 2021-12-16T18:35:40Z
author_association: OWNER
body:
  I'm going to ship your fix now, but I'm not going to add this to the documentation yet because I hope to improve the design prior to Datasette 1.0.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1552#issuecomment-996077053
issue_url: https://api.github.com/repos/simonw/datasette/issues/1552
node_id: IC_kwDOBm6k_c47Xu39
created_at: 2021-12-16T18:36:41Z
updated_at: 2021-12-16T18:36:41Z
author_association: OWNER
body:
  ... actually no, I WILL document this, because not documenting this is what got us to this point in the first place!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1552#issuecomment-996084899
issue_url: https://api.github.com/repos/simonw/datasette/issues/1552
node_id: IC_kwDOBm6k_c47Xwyj
created_at: 2021-12-16T18:48:14Z
updated_at: 2021-12-16T18:48:14Z
author_association: OWNER
body:
  Updated documentation: https://github.com/simonw/datasette/blob/20a2ed6bec367d2f6759be4a879364a72780b59d/docs/facets.rst#facets-in-metadatajson

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/625#issuecomment-996100774
issue_url: https://api.github.com/repos/simonw/datasette/issues/625
node_id: IC_kwDOBm6k_c47X0qm
created_at: 2021-12-16T19:10:01Z
updated_at: 2021-12-16T19:10:48Z
author_association: OWNER
body:
  I think the problem here may be in the design of the JSON returned by facets. It looks like this:
  ```
    "facet_results": {
      "tags": {
        "name": "tags",
        "type": "array",
        "results": [...],
        "hideable": false,
        "toggle_url": "/fixtures/facetable.json?_facet=tags&_trace=1&_nosuggest=1",
        "truncated": false
      },
      "created": {
        "name": "created",
        "type": "date",
        "results": [...]
  ```
  The problem then is that the `tags` key is over-ridden by the second facet with a different type against the same column name!

  https://latest-with-plugins.datasette.io/fixtures/facetable?_trace=1&_facet=created&_facet_date=created&_facet_array=tags&_facet=tags confirms that the SQL queries for those facets are being executed - but the final JSON doesn't show them on https://latest-with-plugins.datasette.io/fixtures/facetable.json?_trace=1&_facet=created&_facet_date=created&_facet_array=tags&_facet=tags

  They're not available in the template context either: https://latest-with-plugins.datasette.io/fixtures/facetable?_facet=created&_facet_date=created&_facet_array=tags&_facet=tags&_context=1

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1553#issuecomment-996103956
issue_url: https://api.github.com/repos/simonw/datasette/issues/1553
node_id: IC_kwDOBm6k_c47X1cU
created_at: 2021-12-16T19:14:38Z
updated_at: 2021-12-16T19:14:38Z
author_association: OWNER
body:
  This is a really interesting idea - kind of similar to how many APIs include custom HTTP headers informing of rate-limits.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1556#issuecomment-996104214
issue_url: https://api.github.com/repos/simonw/datasette/issues/1556
node_id: IC_kwDOBm6k_c47X1gW
created_at: 2021-12-16T19:15:00Z
updated_at: 2021-12-16T19:15:28Z
author_association: OWNER
body:
  Demo: https://latest.datasette.io/fixtures/facetable?_facet=planet_int&_facet=_city_id&_facet=created#facet-created

  <img width="756" alt="image" src="https://user-images.githubusercontent.com/9599/146434231-a7e1800b-eeef-4e5b-83e2-3b466c84b51b.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1557#issuecomment-996115949
issue_url: https://api.github.com/repos/simonw/datasette/issues/1557
node_id: IC_kwDOBm6k_c47X4Xt
created_at: 2021-12-16T19:30:55Z
updated_at: 2021-12-16T19:30:55Z
author_association: OWNER
body:
  Demo: compare https://latest.datasette.io/fixtures/facetable?_facet=_city_id&_nosuggest=1 to https://latest.datasette.io/fixtures/facetable?_facet=_city_id

  Documentation: bottom of https://docs.datasette.io/en/latest/json_api.html#special-table-arguments

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/625#issuecomment-996118401
issue_url: https://api.github.com/repos/simonw/datasette/issues/625
node_id: IC_kwDOBm6k_c47X4-B
created_at: 2021-12-16T19:34:28Z
updated_at: 2021-12-16T19:34:55Z
author_association: OWNER
body:
  The big question here is do I break any existing clients of the `"facet_results"` JSON API?

  It's still pre-1.0 so I could break them, but I've also built my own code against this in the past so it's likely other people have too.

  If I don't break them, I will instead need to come up with a naming convention for those keys - something like `"tags__array"` for example. As well as a way to ensure that a column called `tags__array` doesn't end up conflicting with the `tags__array` key!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/625#issuecomment-996119954
issue_url: https://api.github.com/repos/simonw/datasette/issues/625
node_id: IC_kwDOBm6k_c47X5WS
created_at: 2021-12-16T19:36:01Z
updated_at: 2021-12-16T19:36:11Z
author_association: OWNER
body:
  Datasette's own HTML rendering code doesn't actually use the keys in `facet_results` - it instead loops through `sorted_facet_results` which is defined like this: https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/views/table.py#L937-L941

  And used like this:

  https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/templates/table.html#L154-L156

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/625#issuecomment-996121736
issue_url: https://api.github.com/repos/simonw/datasette/issues/625
node_id: IC_kwDOBm6k_c47X5yI
created_at: 2021-12-16T19:37:08Z
updated_at: 2021-12-16T19:37:08Z
author_association: OWNER
body:
  Really `facet_results` here should be an array of objects, not an object that maps poorly designed string keys to those objects.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/625#issuecomment-996130862
issue_url: https://api.github.com/repos/simonw/datasette/issues/625
node_id: IC_kwDOBm6k_c47X8Au
created_at: 2021-12-16T19:44:48Z
updated_at: 2021-12-16T19:44:48Z
author_association: OWNER
body:
  Decision: as an initial fix I'm going to de-duplicate those keys by using `tags__array` etc - with a `_2` on the end if that key is already used.

  I'll open a separate issue to redesign this better for Datasette 1.0.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1558#issuecomment-996134716
issue_url: https://api.github.com/repos/simonw/datasette/issues/1558
node_id: IC_kwDOBm6k_c47X888
created_at: 2021-12-16T19:46:21Z
updated_at: 2021-12-16T19:46:21Z
author_association: OWNER
body:
  The flaw in the current design is illustrated by this example:

  ```
    "facet_results": {
      "tags": {
        "name": "tags",
        "type": "array",
        "results": [...],
        "hideable": false,
        "toggle_url": "/fixtures/facetable.json?_facet=tags&_trace=1&_nosuggest=1",
        "truncated": false
      },
      "created": {
        "name": "created",
        "type": "date",
        "results": [...]
  ```
  This was the cause of the bug in #625 - the each of those objects is keyed by the name of the column, which left no room for faceting the same column once by date and once by column value.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/625#issuecomment-996146762
issue_url: https://api.github.com/repos/simonw/datasette/issues/625
node_id: IC_kwDOBm6k_c47X_5K
created_at: 2021-12-16T19:51:44Z
updated_at: 2021-12-16T19:51:44Z
author_association: OWNER
body:
  Here's where `facet_results` is built up: https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/views/table.py#L752-L758

  So the decision to key things based on column name is actually embedded deep in the existing facet classes here:

  https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/facets.py#L224-L226

  https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/facets.py#L395-L397

  https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/facets.py#L510-L512

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/625#issuecomment-996149720
issue_url: https://api.github.com/repos/simonw/datasette/issues/625
node_id: IC_kwDOBm6k_c47YAnY
created_at: 2021-12-16T19:56:14Z
updated_at: 2021-12-16T19:56:14Z
author_association: OWNER
body:
  This bad design is even covered in the plugin hooks documentation: https://docs.datasette.io/en/0.59.4/plugin_hooks.html#register-facet-classes

  It does at least have the following warning:

  > **Warning**
  >
  > The design of this plugin hook is unstable and may change. See [issue 830](https://github.com/simonw/datasette/issues/830).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/625#issuecomment-996150904
issue_url: https://api.github.com/repos/simonw/datasette/issues/625
node_id: IC_kwDOBm6k_c47YA54
created_at: 2021-12-16T19:57:52Z
updated_at: 2021-12-16T19:57:52Z
author_association: OWNER
body:
  Good news - GitHub's new code search doesn't show ANYONE using that plugin hook - not surprising since it has that documentation warning plus it's just not a very clearly usable hook: https://cs.github.com/?scopeName=All+repos&scope=&q=register_facet_classes%20-repo%3Asimonw%2Fdatasette

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/830#issuecomment-996151246
issue_url: https://api.github.com/repos/simonw/datasette/issues/830
node_id: IC_kwDOBm6k_c47YA_O
created_at: 2021-12-16T19:58:22Z
updated_at: 2021-12-16T19:58:22Z
author_association: OWNER
body:
  As of today, 16 December 2021, I'm still not seeing any evidence that anyone is using this hook (yet) according to GitHub code search: https://cs.github.com/?scopeName=All+repos&scope=&q=register_facet_classes%20-repo%3Asimonw%2Fdatasette

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/625#issuecomment-996152213
issue_url: https://api.github.com/repos/simonw/datasette/issues/625
node_id: IC_kwDOBm6k_c47YBOV
created_at: 2021-12-16T19:59:46Z
updated_at: 2021-12-16T20:00:05Z
author_association: OWNER
body:
  Since no-one is using that plugin hook I'm going to alter its contract slightly. I'll still keep the existing JSON format working though (until 1.0), since it's much more likely that people are using that JSON somewhere.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/625#issuecomment-996161380
issue_url: https://api.github.com/repos/simonw/datasette/issues/625
node_id: IC_kwDOBm6k_c47YDdk
created_at: 2021-12-16T20:13:05Z
updated_at: 2021-12-16T20:13:05Z
author_association: OWNER
body:
  I updated the example code in the facet plugin hook documentation: https://github.com/simonw/datasette/blob/95d0dd7a1cf6be6b7da41e1404184217eb93f64a/docs/plugin_hooks.rst#register_facet_classes

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/625#issuecomment-996165659
issue_url: https://api.github.com/repos/simonw/datasette/issues/625
node_id: IC_kwDOBm6k_c47YEgb
created_at: 2021-12-16T20:19:53Z
updated_at: 2021-12-16T20:19:53Z
author_association: OWNER
body:
  Demo of the fix: https://latest.datasette.io/fixtures/facetable?_facet=created&_facet_date=created&_facet=tags&_facet_array=tags#facet-tags

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/625#issuecomment-996170510
issue_url: https://api.github.com/repos/simonw/datasette/issues/625
node_id: IC_kwDOBm6k_c47YFsO
created_at: 2021-12-16T20:27:41Z
updated_at: 2021-12-16T20:27:41Z
author_association: OWNER
body:
  And here's the new JSON: https://latest.datasette.io/fixtures/facetable.json?_facet=created&_facet_date=created&_facet=tags&_facet_array=tags&_nosuggest=1

  ```
  {
    "database": "fixtures",
    "table": "facetable",
    "is_view": false,
    "human_description_en": "",
    ...
    "facet_results": {
      "created": {
        "name": "created",
        "type": "column",
        ...
      },
      "tags": {
        "name": "tags",
        "type": "column",
        ...
      },
      "created_2": {
        "name": "created",
        "type": "date",
        ...
      },
      "tags_2": {
        "name": "tags",
        "type": "array",
        ...
      }
    }
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1558#issuecomment-996204369
issue_url: https://api.github.com/repos/simonw/datasette/issues/1558
node_id: IC_kwDOBm6k_c47YN9R
created_at: 2021-12-16T21:23:25Z
updated_at: 2021-12-16T21:23:25Z
author_association: OWNER
body:
  Related: Following the fix for #625 I noticed that `facets_timed_out` gives you just the column name, but doesn't let you know which particular type of facet (`date` or `array` for example) suffered the timeout: https://github.com/simonw/datasette/blob/0d4145d0f4d8b2a7edc1ba4aac1be56cd536a10a/datasette/facets.py#L269-L270

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-996219117
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c47YRjt
created_at: 2021-12-16T21:47:51Z
updated_at: 2021-12-16T21:49:24Z
author_association: OWNER
body:
  Should facets really not be displayed on pages past page one (where `?_next=` is set)? That made sense to me at the time, but I'm now having second thoughts about it.

  I guess it's a useful performance tweak for when crawlers keep hitting the `?_next=` link.

  Actually it looks like facets DO display on subsequent pages, e.g. on https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_next=200 - but facet suggestions do not, thanks to this code: https://github.com/simonw/datasette/blob/2c07327d23d9c5cf939ada9ba4091c1b8b2ba42d/datasette/views/table.py#L777-L785

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-996225235
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c47YTDT
created_at: 2021-12-16T21:58:24Z
updated_at: 2021-12-16T21:58:41Z
author_association: OWNER
body:
  A fundamental operation of this view is to construct the SQL query and accompanying human description based on the incoming query string parameters.

  The human description is the bit at the top of https://latest.datasette.io/fixtures/searchable?_search=dog&_sort=pk&_facet=text2&text2=sara+weasel that says:

  > 1 row where search matches "dog" and text2 = "sara weasel" sorted by pk

  (Also used in the page `<title>`).

  The code actually gathers three things:

  - Fragments of the `where` clause, for example ` "text2" = :p0`
  - Parameters, e.g. `{"p0": "sara weasel"}`
  - Human description components, e.g. `text2 = "sara weasel"`

  Some operations such as `?_where=` don't currently provide an extra human description component.

  `_where=` also doesn't populate a parameter, but maybe it could? Would be neat if in the future `?_where=foo+=+:bar` worked and added a `bar` input field to the screen, as seen with custom queries.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-996225889
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c47YTNh
created_at: 2021-12-16T21:59:32Z
updated_at: 2021-12-16T22:00:42Z
author_association: OWNER
body:
  I added a ton of comments to the `data()` method which really helps get a better feel for how this all works: https://github.com/simonw/datasette/blob/0663d5525cc41e9260ac7d1f6386d3a6eb5ad2a9/datasette/views/table.py#L322

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-996227713
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c47YTqB
created_at: 2021-12-16T22:02:35Z
updated_at: 2021-12-16T22:03:55Z
author_association: OWNER
body:
  Is there an opportunity to refactor things using a new plugin hook here? Maybe the `register_filters` hook from #473, where the hook becomes responsible for building where clauses (and human descriptions of them) based on the incoming query string.

  That version dealt with `Filter` classes, but those might be a bit too low-level for this.

  `?_spatial_within=GEOJSON` was an interesting idea attached to that issue.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1552#issuecomment-996229007
issue_url: https://api.github.com/repos/simonw/datasette/issues/1552
node_id: IC_kwDOBm6k_c47YT-P
created_at: 2021-12-16T22:04:39Z
updated_at: 2021-12-16T22:04:39Z
author_association: CONTRIBUTOR
body:
  Wow, that was fast, thank you so much @simonw !

  > I'm also not convinced that this configuration syntax is right. It's a bit weird having a `"facets"` list that can either by column-name-strings or `{"type-of-facet": "column-name"}` objects. Maybe there's a better design for this?

  I agree that it's not ideal, my initial naive approach was to detect if it's an array, like what is done here:

  https://github.com/simonw/datasette/blob/2c07327d23d9c5cf939ada9ba4091c1b8b2ba42d/datasette/facets.py#L312-L313

  But it requires an extra query to determine the type, which is a bit problematic, especially for big tables I guess.

  Taking a look at #510, I wonder if a `facet_delimiter` should be defined for that kind of columns (that would help our team not to have an intermediary conversion step from `foo|bar` to `["foo","bar"]` for instance).

  To be consistent with the `--extract-column` parameter, maybe an explicit casting/delimiter would be useful: `--set-column 'Foo:Array:|'`.

  Throwing a lot of ideas without knowing the big picture… but sometimes newcomers have superpowers :).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/358#issuecomment-996232461
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/358
node_id: IC_kwDOCGYnMM47YU0N
created_at: 2021-12-16T22:10:39Z
updated_at: 2021-12-16T22:10:39Z
author_association: OWNER
body:
  This goes beyond the `transform()` method - the curious methods that create new SQL tables could benefit from the ability to add `CHECK` constraints too.

  I haven't used these myself, do you have any `CREATE TABLE` examples that use them that you can share?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-996240802
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c47YW2i
created_at: 2021-12-16T22:25:00Z
updated_at: 2021-12-16T22:36:04Z
author_association: OWNER
body:
  I think that plugin hook would get given the `request` object (and `datasette` and the name of the database and table) and returns a list of SQL fragments, a dictionary of lookup arguments and a list of human-description fragments - or an awaitable.

  `filters_from_request(request, database, table, datasette)` perhaps? (Similar in name to `actor_from_request`).

  ```python
  @hookspec
  def filters_from_request(request, database, table, datasette):
      """Return (where_clauses, params_dict, human_descriptions) based on the request"""
  ```

  Turns out that's pretty much exactly what I implemented in 5116c4ec8aed5091e1f75415424b80f613518dc6 for #473:

  ```python

  @hookspec
  def table_filter():
      "Custom filtering of the current table based on the request"
  ```
  ```python
  TableFilter = namedtuple("TableFilter", (
      "human_description_extras", "where_clauses", "params")
  )
  ```
  ```python
          # filter_arguments plugin hook support
          for awaitable_fn in pm.hook.table_filter():
              extras = await awaitable_fn(
                  view=self, name=name, table=table, request=request
              )
              human_description_extras.extend(extras.human_description_extras)
              where_clauses.extend(extras.where_clauses)
              params.update(extras.params)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-996248713
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c47YYyJ
created_at: 2021-12-16T22:39:47Z
updated_at: 2021-12-16T22:39:47Z
author_association: OWNER
body:
  The hook could return a named tuple like this one:
  ```python
  from typing import NamedTuple, List, Optional, Union, Dict

  class FilterArguments(NamedTuple):
      where_clauses: List[str]
      params: Dict[str, Union[str, int, float]]
      human_descriptions: List[str]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-996250585
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c47YZPZ
created_at: 2021-12-16T22:43:37Z
updated_at: 2021-12-16T22:45:07Z
author_association: OWNER
body:
  Ran into a problem prototyping that hook up for handling `?_where=` - that feature also adds a little bit of extra template context in order to show the interface for removing wheres - the `extra_wheres_for_ui` variable: https://github.com/simonw/datasette/blob/0663d5525cc41e9260ac7d1f6386d3a6eb5ad2a9/datasette/views/table.py#L457-L463

  Maybe change to this?

  ```python
  class FilterArguments(NamedTuple):
      where_clauses: List[str]
      params: Dict[str, Union[str, int, float]]
      human_descriptions: List[str]
      extra_context: Dict[str, Any]
  ```
  That might be necessary for `_search` too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-996264617
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c47Ycqp
created_at: 2021-12-16T23:11:12Z
updated_at: 2021-12-16T23:11:12Z
author_association: OWNER
body:
  I managed to extract both `_search=` and `_where=` out using a prototype of that hook. I wonder if it could extract the complex code for `?_next` too?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/473#issuecomment-996267817
issue_url: https://api.github.com/repos/simonw/datasette/issues/473
node_id: IC_kwDOBm6k_c47Ydcp
created_at: 2021-12-16T23:17:52Z
updated_at: 2021-12-16T23:19:00Z
author_association: OWNER
body:
  I revisited this idea in #1518 and came up with a slightly different name and design for the hook:

  ```python
  @hookspec
  def filters_from_request(request, database, table, datasette):
      """
      Return FilterArguments(
          where_clauses=[str, str, str],
          params={},
          human_descriptions=[str, str, str],
          extra_context={}
      ) based on the request"""
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/473#issuecomment-996275108
issue_url: https://api.github.com/repos/simonw/datasette/issues/473
node_id: IC_kwDOBm6k_c47YfOk
created_at: 2021-12-16T23:32:22Z
updated_at: 2021-12-16T23:32:30Z
author_association: OWNER
body:
  This filter design can only influence the `where` component of the SQL clause - it's not able to modify the `SELECT` columns or adjust the `ORDER BY` or `OFFSET LIMIT` parts. I think that's OK.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-996286104
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c47Yh6Y
created_at: 2021-12-17T00:00:07Z
updated_at: 2021-12-17T00:00:07Z
author_association: OWNER
body:
  Documentation of the new hook in the PR: https://github.com/simonw/datasette/blob/54e9b3972f277431a001e685f78e5dd6403a6d8d/docs/plugin_hooks.rst#filters_from_requestrequest-database-table-datasette

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/473#issuecomment-996286199
issue_url: https://api.github.com/repos/simonw/datasette/issues/473
node_id: IC_kwDOBm6k_c47Yh73
created_at: 2021-12-17T00:00:22Z
updated_at: 2021-12-17T00:00:22Z
author_association: OWNER
body:
  Documentation for that hook in  the PR branch: https://github.com/simonw/datasette/blob/54e9b3972f277431a001e685f78e5dd6403a6d8d/docs/plugin_hooks.rst#filters_from_requestrequest-database-table-datasette

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1559#issuecomment-996286808
issue_url: https://api.github.com/repos/simonw/datasette/issues/1559
node_id: IC_kwDOBm6k_c47YiFY
created_at: 2021-12-17T00:01:43Z
updated_at: 2021-12-17T00:01:43Z
author_association: OWNER
body:
  This already has tests and documentation, and I've used it to refactor out the logic for `?_where=` and `?_search=` and `?_through=`.

  Do I like this enough to land it on `main`?

  Also, I think I can still use it to refactor out the `Filters` code that implements `?col=x` and `?col__lt=5` and suchlike.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1559#issuecomment-996289541
issue_url: https://api.github.com/repos/simonw/datasette/issues/1559
node_id: IC_kwDOBm6k_c47YiwF
created_at: 2021-12-17T00:07:42Z
updated_at: 2021-12-17T17:28:54Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1559?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1559](https://codecov.io/gh/simonw/datasette/pull/1559?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (71af58d) into [main](https://codecov.io/gh/simonw/datasette/commit/0663d5525cc41e9260ac7d1f6386d3a6eb5ad2a9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0663d55) will **increase** coverage by `0.09%`.
  > The diff coverage is `97.97%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1559/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1559?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@            Coverage Diff             @@
  ##             main    #1559      +/-   ##
  ==========================================
  + Coverage   91.96%   92.05%   +0.09%     
  ==========================================
    Files          34       34              
    Lines        4442     4493      +51     
  ==========================================
  + Hits         4085     4136      +51     
    Misses        357      357              
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1559?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/plugins.py](https://codecov.io/gh/simonw/datasette/pull/1559/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3BsdWdpbnMucHk=) | `82.35% <ø> (ø)` | |
  | [datasette/filters.py](https://codecov.io/gh/simonw/datasette/pull/1559/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2ZpbHRlcnMucHk=) | `95.69% <97.67%> (+1.33%)` | :arrow_up: |
  | [datasette/hookspecs.py](https://codecov.io/gh/simonw/datasette/pull/1559/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2hvb2tzcGVjcy5weQ==) | `100.00% <100.00%> (ø)` | |
  | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1559/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `96.21% <100.00%> (+0.13%)` | :arrow_up: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1559?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1559?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [0663d55...71af58d](https://codecov.io/gh/simonw/datasette/pull/1559?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/473#issuecomment-996345233
issue_url: https://api.github.com/repos/simonw/datasette/issues/473
node_id: IC_kwDOBm6k_c47YwWR
created_at: 2021-12-17T01:20:31Z
updated_at: 2021-12-17T18:13:01Z
author_association: OWNER
body:
  I could use this hook to add table filtering on a map to the existing `datasette-leaflet-freedraw` plugin.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/358#issuecomment-996482595
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/358
node_id: IC_kwDOCGYnMM47ZR4j
created_at: 2021-12-17T06:57:51Z
updated_at: 2021-12-17T23:24:16Z
author_association: NONE
body:
  > This goes beyond the `transform()` method - the curious methods that create new SQL tables could benefit from the ability to add `CHECK` constraints too.
  > 
  > I haven't used these myself, do you have any `CREATE TABLE` examples that use them that you can share?

  I'm using them myself for the first time as well, this is a tutorial of how to use (and change) them in sqlite: https://www.sqlitetutorial.net/sqlite-check-constraint/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/473#issuecomment-996484551
issue_url: https://api.github.com/repos/simonw/datasette/issues/473
node_id: IC_kwDOBm6k_c47ZSXH
created_at: 2021-12-17T07:02:21Z
updated_at: 2021-12-17T07:04:23Z
author_association: OWNER
body:
  The one slightly weird thing about this hook is how it adds `extra_context` without an obvious way for plugins to add extra HTML to the templates based on that context.

  Maybe I need the proposed mechanism from
  - #1191

  Which has an in-progress PR:
  - #1204

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1562#issuecomment-996716158
issue_url: https://api.github.com/repos/simonw/datasette/issues/1562
node_id: IC_kwDOBm6k_c47aK5-
created_at: 2021-12-17T13:18:49Z
updated_at: 2021-12-17T13:18:49Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1562?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1562](https://codecov.io/gh/simonw/datasette/pull/1562?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (2f008e8) into [main](https://codecov.io/gh/simonw/datasette/commit/0663d5525cc41e9260ac7d1f6386d3a6eb5ad2a9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0663d55) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1562/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1562?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1562   +/-   ##
  =======================================
    Coverage   91.96%   91.96%           
  =======================================
    Files          34       34           
    Lines        4442     4442           
  =======================================
    Hits         4085     4085           
    Misses        357      357           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1562?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1562?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [0663d55...2f008e8](https://codecov.io/gh/simonw/datasette/pull/1562?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1559#issuecomment-996895423
issue_url: https://api.github.com/repos/simonw/datasette/issues/1559
node_id: IC_kwDOBm6k_c47a2q_
created_at: 2021-12-17T17:28:44Z
updated_at: 2021-12-17T17:28:44Z
author_association: OWNER
body:
  Before I land this I'm going to build one prototype plugin against it to confirm that the new hook is useful in its current shape.

  I'll add support for filtering a table by drawing on a map to https://datasette.io/plugins/datasette-leaflet-freedraw

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/473#issuecomment-996958442
issue_url: https://api.github.com/repos/simonw/datasette/issues/473
node_id: IC_kwDOBm6k_c47bGDq
created_at: 2021-12-17T18:59:27Z
updated_at: 2021-12-17T18:59:27Z
author_association: OWNER
body:
  I'm happy with how the prototype that used this plugin in `datasette-leaflet-freedraw` turned out: https://github.com/simonw/datasette-leaflet-freedraw/blob/e8a16a0fe90656b8d655c02881d23a2b9833281d/datasette_leaflet_freedraw/__init__.py

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1559#issuecomment-996959325
issue_url: https://api.github.com/repos/simonw/datasette/issues/1559
node_id: IC_kwDOBm6k_c47bGRd
created_at: 2021-12-17T18:59:54Z
updated_at: 2021-12-17T18:59:54Z
author_association: OWNER
body:
  I've convinced myself that this plugin hook design is good through this `datasette-leaflet-freedraw` prototype: https://github.com/simonw/datasette-leaflet-freedraw/blob/e8a16a0fe90656b8d655c02881d23a2b9833281d/datasette_leaflet_freedraw/__init__.py

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1559#issuecomment-996961196
issue_url: https://api.github.com/repos/simonw/datasette/issues/1559
node_id: IC_kwDOBm6k_c47bGus
created_at: 2021-12-17T19:00:53Z
updated_at: 2021-12-17T19:00:53Z
author_association: OWNER
body:
  I'm going to merge this to `main` now. I can continue the refactoring there, but having it in `main` means I can put out an alpha release with the new hook which will unblock me from running tests against it in this repo: https://github.com/simonw/datasette-leaflet-freedraw/pull/8

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1565#issuecomment-997069128
issue_url: https://api.github.com/repos/simonw/datasette/issues/1565
node_id: IC_kwDOBm6k_c47bhFI
created_at: 2021-12-17T22:31:18Z
updated_at: 2021-12-17T22:31:18Z
author_association: OWNER
body:
  This should aim to be as consistent as possible with the various arguments to hooks on https://docs.datasette.io/en/stable/plugin_hooks.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1565#issuecomment-997077410
issue_url: https://api.github.com/repos/simonw/datasette/issues/1565
node_id: IC_kwDOBm6k_c47bjGi
created_at: 2021-12-17T22:54:45Z
updated_at: 2021-12-17T22:54:45Z
author_association: OWNER
body:
  The table page should expose the query both with and without the `limit` clause. The above gave me back:

  ```sql
  select id, ACCESS_TYP, UNIT_ID, UNIT_NAME, SUID_NMA, AGNCY_ID, AGNCY_NAME, AGNCY_LEV,
    AGNCY_TYP, AGNCY_WEB, LAYER, MNG_AG_ID, MNG_AGENCY, MNG_AG_LEV, MNG_AG_TYP,
    PARK_URL, COUNTY, ACRES, LABEL_NAME, YR_EST, DES_TP, GAP_STS, geometry
  from CPAD_2020a_Units where "AGNCY_LEV" = :p0 order by id limit 101
  ```
  But I actually wanted to run a `fetch()` against a version of that without the `order by id limit 101` bit (I wanted to figure out the `Extent()` of the `geometry` column) - so I need something like `datasette.table_sql_no_order_no_limit`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1566#issuecomment-997078812
issue_url: https://api.github.com/repos/simonw/datasette/issues/1566
node_id: IC_kwDOBm6k_c47bjcc
created_at: 2021-12-17T22:58:55Z
updated_at: 2021-12-17T22:58:55Z
author_association: OWNER
body:
  The release notes for the 0.60a0 alpha will be useful here: https://github.com/simonw/datasette/releases/tag/0.60a0

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1562#issuecomment-997081673
issue_url: https://api.github.com/repos/simonw/datasette/issues/1562
node_id: IC_kwDOBm6k_c47bkJJ
created_at: 2021-12-17T23:06:38Z
updated_at: 2021-12-17T23:06:38Z
author_association: OWNER
body:
  From this diff between `0.7.0` and `1.0`: https://github.com/aio-libs/janus/compare/v0.7.0...v1.0.0

  It looks like the only change relevant to compatibility is `loop = asyncio.get_running_loop()` directly instead of falling back to `asyncio.get_event_loop()` if `get_running_loop` isn't available.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1562#issuecomment-997082189
issue_url: https://api.github.com/repos/simonw/datasette/issues/1562
node_id: IC_kwDOBm6k_c47bkRN
created_at: 2021-12-17T23:08:14Z
updated_at: 2021-12-17T23:08:14Z
author_association: OWNER
body:
  Oh that makes sense: In Python 3.6 this happens:
  ```
  Collecting janus<1.1,>=0.6.2
    Using cached janus-0.7.0-py3-none-any.whl (6.9 kB)
  ```
  While in Python 3.7 or higher this happens:
  ```
  Collecting janus<1.1,>=0.6.2
    Downloading janus-1.0.0-py3-none-any.whl (6.9 kB)
  ```
  So this is safe to apply because `pip` is smart enough to pick the version of Janus that works for that Python version.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1559#issuecomment-997082676
issue_url: https://api.github.com/repos/simonw/datasette/issues/1559
node_id: IC_kwDOBm6k_c47bkY0
created_at: 2021-12-17T23:09:41Z
updated_at: 2021-12-17T23:09:41Z
author_association: OWNER
body:
  This is now available to try out in Datasette 0.60a0: https://github.com/simonw/datasette/releases/tag/0.60a0

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-997082845
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c47bkbd
created_at: 2021-12-17T23:10:09Z
updated_at: 2021-12-17T23:10:17Z
author_association: OWNER
body:
  These changes so far are now in the 0.60a0 alpha: https://github.com/simonw/datasette/releases/tag/0.60a0

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1565#issuecomment-997121215
issue_url: https://api.github.com/repos/simonw/datasette/issues/1565
node_id: IC_kwDOBm6k_c47bty_
created_at: 2021-12-18T01:45:44Z
updated_at: 2021-12-18T01:45:44Z
author_association: OWNER
body:
  I want to get this into Datasette 0.60 - #1566 - it's a small change that can unlock a lot of potential.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1564#issuecomment-997122938
issue_url: https://api.github.com/repos/simonw/datasette/issues/1564
node_id: IC_kwDOBm6k_c47buN6
created_at: 2021-12-18T01:55:25Z
updated_at: 2021-12-18T01:55:46Z
author_association: OWNER
body:
  Made this change while working on this issue:
  - #1567

  I'm going to write a test for this that uses that `sleep()` SQL function from c35b84a2aabe2f14aeacf6cda4110ae1e94d6059.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1546#issuecomment-997124280
issue_url: https://api.github.com/repos/simonw/datasette/issues/1546
node_id: IC_kwDOBm6k_c47bui4
created_at: 2021-12-18T02:05:16Z
updated_at: 2021-12-18T02:05:16Z
author_association: OWNER
body:
  Sure - there are actually several levels to this.

  The code that creates connections to the database is this: https://github.com/simonw/datasette/blob/83bacfa9452babe7bd66e3579e23af988d00f6ac/datasette/database.py#L72-L95

  For files on disk, it does this:
  ```python
  # For read-only connections
  conn = sqlite3.connect( "file:my.db?mode=ro", uri=True, check_same_thread=False)
  # For connections that should be treated as immutable:
  conn = sqlite3.connect( "file:my.db?immutable=1", uri=True, check_same_thread=False)
  ```
  For in-memory databases it runs this after the connection has been created:
  ```python
  conn.execute("PRAGMA query_only=1")
  ```
  SQLite `PRAGMA` queries are treated as dangerous: someone could run `PRAGMA query_only=0` to turn that previous option off for example.

  So this function runs against any incoming SQL to verify that it looks like a `SELECT ...` and doesn't have anything like that in it.

  https://github.com/simonw/datasette/blob/83bacfa9452babe7bd66e3579e23af988d00f6ac/datasette/utils/__init__.py#L195-L204

  You can see the tests for that here: https://github.com/simonw/datasette/blob/b1fed48a95516ae84c0f020582303ab50ab817e2/tests/test_utils.py#L136-L170

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1563#issuecomment-997125191
issue_url: https://api.github.com/repos/simonw/datasette/issues/1563
node_id: IC_kwDOBm6k_c47buxH
created_at: 2021-12-18T02:10:20Z
updated_at: 2021-12-18T02:10:20Z
author_association: OWNER
body:
  I should document the usage of this constructor in https://docs.datasette.io/en/stable/internals.html#datasette-class

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1561#issuecomment-997127784
issue_url: https://api.github.com/repos/simonw/datasette/issues/1561
node_id: IC_kwDOBm6k_c47bvZo
created_at: 2021-12-18T02:27:56Z
updated_at: 2021-12-18T02:27:56Z
author_association: OWNER
body:
  Oh that's an interesting solution, combining the hashes of all of the individual databases.

  I'm actually not a big fan of `hashed_url` mode - I implemented it right at the start of the project because it felt like a clever hack, and then ended up making it not-the-default a few years ago:

  - #418
  - #419
  - #421

  I've since not found myself wanting to use it at all for any of my projects - which makes me nervous, because it means there's a pretty complex feature that I'm not using at all, so it's only really protected by the existing unit tests for it.

  What I'd really like to do is figure out how to have hashed URL mode work entirely as a plugin - then I could extract it from Datasette core entirely (which would simplify a bunch of stuff) but people who find the optimization useful would be able to access it.

  I'm not sure that the existing plugin hooks are robust enough to do that yet though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1555#issuecomment-997128080
issue_url: https://api.github.com/repos/simonw/datasette/issues/1555
node_id: IC_kwDOBm6k_c47bveQ
created_at: 2021-12-18T02:30:19Z
updated_at: 2021-12-18T02:30:19Z
author_association: OWNER
body:
  I think all of these queries happen in one place - in the `populate_schema_tables()` function - so optimizing them might be localized to just that area of the code, which would be nice:

  https://github.com/simonw/datasette/blob/c00f29affcafce8314366852ba1a0f5a7dd25690/datasette/utils/internal_db.py#L97-L183

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1555#issuecomment-997128251
issue_url: https://api.github.com/repos/simonw/datasette/issues/1555
node_id: IC_kwDOBm6k_c47bvg7
created_at: 2021-12-18T02:31:51Z
updated_at: 2021-12-18T02:31:51Z
author_association: OWNER
body:
  I was thinking it might even be possible to convert this into a `insert into tables select from ...` query:

  https://github.com/simonw/datasette/blob/c00f29affcafce8314366852ba1a0f5a7dd25690/datasette/utils/internal_db.py#L102-L112

   But the `SELECT` runs against a separate database from the `INSERT INTO`, so I would have to setup a cross-database connection for this which feels a little too complicated.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1555#issuecomment-997128508
issue_url: https://api.github.com/repos/simonw/datasette/issues/1555
node_id: IC_kwDOBm6k_c47bvk8
created_at: 2021-12-18T02:33:57Z
updated_at: 2021-12-18T02:33:57Z
author_association: OWNER
body:
  Here's why - `trace` only applies to read, not write SQL operations: https://github.com/simonw/datasette/blob/7c8f8aa209e4ba7bf83976f8495d67c28fbfca24/datasette/database.py#L209-L211

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1568#issuecomment-997128950
issue_url: https://api.github.com/repos/simonw/datasette/issues/1568
node_id: IC_kwDOBm6k_c47bvr2
created_at: 2021-12-18T02:38:01Z
updated_at: 2021-12-18T02:38:01Z
author_association: OWNER
body:
  Prototype:

  ```diff
  diff --git a/datasette/database.py b/datasette/database.py
  index 0a0c104..468e936 100644
  --- a/datasette/database.py
  +++ b/datasette/database.py
  @@ -99,7 +99,9 @@ class Database:
               with conn:
                   return conn.execute(sql, params or [])
   
  -        return await self.execute_write_fn(_inner, block=block)
  +        with trace("sql", database=self.name, sql=sql.strip(), params=params):
  +            results = await self.execute_write_fn(_inner, block=block)
  +        return results
   
       async def execute_write_fn(self, fn, block=False):
           task_id = uuid.uuid5(uuid.NAMESPACE_DNS, "datasette.io")
  ```

  <img width="1292" alt="image" src="https://user-images.githubusercontent.com/9599/146626249-cc8609f6-590c-49e3-abab-0cbc132916d1.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1555#issuecomment-997234858
issue_url: https://api.github.com/repos/simonw/datasette/issues/1555
node_id: IC_kwDOBm6k_c47cJiq
created_at: 2021-12-18T17:28:44Z
updated_at: 2021-12-18T17:28:44Z
author_association: OWNER
body:
  Maybe it would be worth exploring attaching each DB in turn to the _internal connection in order to perform these queries faster.

  I'm a bit worried about leaks though: the internal database isn't meant to be visible, even temporarily attaching another DB to it could cause SQL queries against that DB to be able to access the internal data.

  So maybe instead the _internal connection gets to connect to the other DBs? There's a maximum of ten there I think, which is good for most but not all cases. But the cases with the most connected databases will see the worst performance!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1555#issuecomment-997235086
issue_url: https://api.github.com/repos/simonw/datasette/issues/1555
node_id: IC_kwDOBm6k_c47cJmO
created_at: 2021-12-18T17:30:13Z
updated_at: 2021-12-18T17:30:13Z
author_association: OWNER
body:
  Now that trace sees write queries (#1568) it's clear that there is a whole lot more DB activity then I had realized:

  <img width="1292" alt="image" src="https://user-images.githubusercontent.com/9599/146626249-cc8609f6-590c-49e3-abab-0cbc132916d1.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1555#issuecomment-997241645
issue_url: https://api.github.com/repos/simonw/datasette/issues/1555
node_id: IC_kwDOBm6k_c47cLMt
created_at: 2021-12-18T18:12:26Z
updated_at: 2021-12-18T18:12:26Z
author_association: OWNER
body:
  A simpler optimization would be just to turn all of those column and index reads into a single efficient UNION query against each database, then figure out the most efficient pattern to send them all as writes in one go as opposed to calling `.execute_write()` in a loop.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1555#issuecomment-997241969
issue_url: https://api.github.com/repos/simonw/datasette/issues/1555
node_id: IC_kwDOBm6k_c47cLRx
created_at: 2021-12-18T18:13:04Z
updated_at: 2021-12-18T18:13:04Z
author_association: OWNER
body:
  Also: running all of those `CREATE TABLE IF NOT EXISTS` in a single call to `conn.executescript()` rather than as separate queries may speed things up too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1555#issuecomment-997245301
issue_url: https://api.github.com/repos/simonw/datasette/issues/1555
node_id: IC_kwDOBm6k_c47cMF1
created_at: 2021-12-18T18:17:04Z
updated_at: 2021-12-18T18:17:04Z
author_association: OWNER
body:
  One downside of `conn.executescript()` is that it won't be picked up by the tracing mechanism - in fact nothing that uses `await db.execute_write_fn(fn, block=True)` or `await db.execute_fn(fn, block=True)` gets picked up by tracing.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1555#issuecomment-997248364
issue_url: https://api.github.com/repos/simonw/datasette/issues/1555
node_id: IC_kwDOBm6k_c47cM1s
created_at: 2021-12-18T18:20:10Z
updated_at: 2021-12-18T18:20:10Z
author_association: OWNER
body:
  Idea: teach `execute_write` to accept an optional `executescript=True` parameter, like this:
  ```diff
  diff --git a/datasette/database.py b/datasette/database.py
  index 468e936..1a424f5 100644
  --- a/datasette/database.py
  +++ b/datasette/database.py
  @@ -94,10 +94,14 @@ class Database:
               f"file:{self.path}{qs}", uri=True, check_same_thread=False
           )
   
  -    async def execute_write(self, sql, params=None, block=False):
  +    async def execute_write(self, sql, params=None, executescript=False, block=False):
  +        assert not executescript and params, "Cannot use params with executescript=True"
           def _inner(conn):
               with conn:
  -                return conn.execute(sql, params or [])
  +                if executescript:
  +                    return conn.executescript(sql)
  +                else:
  +                    return conn.execute(sql, params or [])
   
           with trace("sql", database=self.name, sql=sql.strip(), params=params):
               results = await self.execute_write_fn(_inner, block=block)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1569#issuecomment-997249563
issue_url: https://api.github.com/repos/simonw/datasette/issues/1569
node_id: IC_kwDOBm6k_c47cNIb
created_at: 2021-12-18T18:21:23Z
updated_at: 2021-12-18T18:21:23Z
author_association: OWNER
body:
  Goal here is to gain the ability to use `conn.executescript()` and still have it show up in the tracer.

  https://docs.python.org/3/library/sqlite3.html#sqlite3.Cursor.executescript

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1555#issuecomment-997262475
issue_url: https://api.github.com/repos/simonw/datasette/issues/1555
node_id: IC_kwDOBm6k_c47cQSL
created_at: 2021-12-18T18:34:18Z
updated_at: 2021-12-18T18:34:18Z
author_association: OWNER
body:
  <img width="1055" alt="image" src="https://user-images.githubusercontent.com/9599/146652142-1c0bc34e-4a18-407d-bd59-28d565b631a6.png">

  Using `executescript=True` that call now takes 1.89ms to create all of those tables.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1555#issuecomment-997266100
issue_url: https://api.github.com/repos/simonw/datasette/issues/1555
node_id: IC_kwDOBm6k_c47cRK0
created_at: 2021-12-18T18:40:02Z
updated_at: 2021-12-18T18:40:02Z
author_association: OWNER
body:
  The implementation of `cursor.executemany()` looks very efficient - it turns into a call to this C function with `multiple` set to `1`: https://github.com/python/cpython/blob/e002bbc6cce637171fb2b1391ffeca8643a13843/Modules/_sqlite/cursor.c#L468-L469

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1569#issuecomment-997266687
issue_url: https://api.github.com/repos/simonw/datasette/issues/1569
node_id: IC_kwDOBm6k_c47cRT_
created_at: 2021-12-18T18:41:40Z
updated_at: 2021-12-18T18:41:40Z
author_association: OWNER
body:
  Updated documentation: https://docs.datasette.io/en/latest/internals.html#await-db-execute-write-sql-params-none-executescript-false-block-false

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1555#issuecomment-997267416
issue_url: https://api.github.com/repos/simonw/datasette/issues/1555
node_id: IC_kwDOBm6k_c47cRfY
created_at: 2021-12-18T18:44:53Z
updated_at: 2021-12-18T18:45:28Z
author_association: OWNER
body:
  Rather than adding a `executemany=True` parameter, I'm now thinking a better design might be to have three methods:

  - `db.execute_write(sql, params=None, block=False)`
  - `db.execute_writescript(sql, block=False)`
  - `db.execute_writemany(sql, params_seq, block=False)`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1555#issuecomment-997272223
issue_url: https://api.github.com/repos/simonw/datasette/issues/1555
node_id: IC_kwDOBm6k_c47cSqf
created_at: 2021-12-18T19:17:13Z
updated_at: 2021-12-18T19:17:13Z
author_association: OWNER
body:
  That's a good optimization. Still need to deal with the huge flurry of `PRAGMA` queries though before I can consider this done.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1555#issuecomment-997320824
issue_url: https://api.github.com/repos/simonw/datasette/issues/1555
node_id: IC_kwDOBm6k_c47ceh4
created_at: 2021-12-19T02:59:57Z
updated_at: 2021-12-19T03:00:44Z
author_association: OWNER
body:
  To list all indexes: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++index_list.*%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_index_list%28sqlite_master.name%29+AS+index_list%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27

  ```sql
  SELECT
    sqlite_master.name,
    index_list.*
  FROM
    sqlite_master,
    pragma_index_list(sqlite_master.name) AS index_list
  WHERE
    sqlite_master.type = 'table'
  ```

  Foreign keys: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++foreign_key_list.*%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_foreign_key_list%28sqlite_master.name%29+AS+foreign_key_list%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27

  ```sql
  SELECT
    sqlite_master.name,
    foreign_key_list.*
  FROM
    sqlite_master,
    pragma_foreign_key_list(sqlite_master.name) AS foreign_key_list
  WHERE
    sqlite_master.type = 'table'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1555#issuecomment-997321115
issue_url: https://api.github.com/repos/simonw/datasette/issues/1555
node_id: IC_kwDOBm6k_c47cemb
created_at: 2021-12-19T03:03:12Z
updated_at: 2021-12-19T03:03:12Z
author_association: OWNER
body:
  Table columns is a bit harder, because `table_xinfo` is only in SQLite 3.26.0 or higher: https://github.com/simonw/datasette/blob/d637ed46762fdbbd8e32b86f258cd9a53c1cfdc7/datasette/utils/__init__.py#L565-L581

  So if that function is available: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++table_xinfo.*%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_table_xinfo%28sqlite_master.name%29+AS+table_xinfo%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27

  ```sql
  SELECT
    sqlite_master.name,
    table_xinfo.*
  FROM
    sqlite_master,
    pragma_table_xinfo(sqlite_master.name) AS table_xinfo
  WHERE
    sqlite_master.type = 'table'
  ```
  And otherwise, using `table_info`: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++table_info.*%2C%0D%0A++0+as+hidden%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_table_info%28sqlite_master.name%29+AS+table_info%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27

  ```sql
  SELECT
    sqlite_master.name,
    table_info.*,
    0 as hidden
  FROM
    sqlite_master,
    pragma_table_info(sqlite_master.name) AS table_info
  WHERE
    sqlite_master.type = 'table'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1555#issuecomment-997321217
issue_url: https://api.github.com/repos/simonw/datasette/issues/1555
node_id: IC_kwDOBm6k_c47ceoB
created_at: 2021-12-19T03:04:16Z
updated_at: 2021-12-19T03:04:16Z
author_association: OWNER
body:
  One thing to watch out for though, from https://sqlite.org/pragma.html#pragfunc

  >  The table-valued functions for PRAGMA feature was added in SQLite version 3.16.0 (2017-01-02). Prior versions of SQLite cannot use this feature. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1555#issuecomment-997321327
issue_url: https://api.github.com/repos/simonw/datasette/issues/1555
node_id: IC_kwDOBm6k_c47cepv
created_at: 2021-12-19T03:05:39Z
updated_at: 2021-12-19T03:05:44Z
author_association: OWNER
body:
  This caught me out once before in:
  - https://github.com/simonw/datasette/issues/1276

  Turns out Glitch was running SQLite 3.11.0 from 2016-02-15.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1555#issuecomment-997321477
issue_url: https://api.github.com/repos/simonw/datasette/issues/1555
node_id: IC_kwDOBm6k_c47cesF
created_at: 2021-12-19T03:07:33Z
updated_at: 2021-12-19T03:07:33Z
author_association: OWNER
body:
  If I want to continue supporting SQLite prior to 3.16.0 (2017-01-02) I'll need this optimization to only kick in with versions that support table-valued PRAGMA functions, while keeping the old `PRAGMA foreign_key_list(table)` stuff working for those older versions.

  That's feasible, but it's a bit more work - and I need to make sure I have robust testing in place for SQLite 3.15.0.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1555#issuecomment-997321653
issue_url: https://api.github.com/repos/simonw/datasette/issues/1555
node_id: IC_kwDOBm6k_c47ceu1
created_at: 2021-12-19T03:09:43Z
updated_at: 2021-12-19T03:09:43Z
author_association: OWNER
body:
  On that same documentation page I just spotted this:

  >  This feature is experimental and is subject to change. Further documentation will become available if and when the table-valued functions for PRAGMAs feature becomes officially supported. 

  This makes me nervous to rely on pragma function optimizations in Datasette itself.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1555#issuecomment-997321767
issue_url: https://api.github.com/repos/simonw/datasette/issues/1555
node_id: IC_kwDOBm6k_c47cewn
created_at: 2021-12-19T03:10:58Z
updated_at: 2021-12-19T03:10:58Z
author_association: OWNER
body:
  I wonder how much overhead there is switching between the `async` event loop main code and the thread that runs the SQL queries.

  Would there be a performance boost if I gathered all of the column/index information in a single function run on the thread using `db.execute_fn()` I wonder? It would eliminate a bunch of switching between threads.

  Would be great to understand how much of an impact that would have.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1555#issuecomment-997324156
issue_url: https://api.github.com/repos/simonw/datasette/issues/1555
node_id: IC_kwDOBm6k_c47cfV8
created_at: 2021-12-19T03:40:05Z
updated_at: 2021-12-19T03:40:05Z
author_association: OWNER
body:
  Using the prototype of this:
  - https://github.com/simonw/datasette-pretty-traces/issues/5

  I'm seeing about 180ms spent running all of these queries on startup!

  ![CleanShot 2021-12-18 at 19 38 37@2x](https://user-images.githubusercontent.com/9599/146663045-46bda669-90de-474f-8870-345182725dc1.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1555#issuecomment-997324666
issue_url: https://api.github.com/repos/simonw/datasette/issues/1555
node_id: IC_kwDOBm6k_c47cfd6
created_at: 2021-12-19T03:47:51Z
updated_at: 2021-12-19T03:48:09Z
author_association: OWNER
body:
  Here's a hacked together prototype of running all of that stuff inside a single function passed to `.execute_fn()`:

  ```diff
  diff --git a/datasette/utils/internal_db.py b/datasette/utils/internal_db.py
  index 95055d8..58f9982 100644
  --- a/datasette/utils/internal_db.py
  +++ b/datasette/utils/internal_db.py
  @@ -1,4 +1,5 @@
   import textwrap
  +from datasette.utils import table_column_details
   
   
   async def init_internal_db(db):
  @@ -70,49 +71,70 @@ async def populate_schema_tables(internal_db, db):
           "DELETE FROM tables WHERE database_name = ?", [database_name], block=True
       )
       tables = (await db.execute("select * from sqlite_master WHERE type = 'table'")).rows
  -    tables_to_insert = []
  -    columns_to_delete = []
  -    columns_to_insert = []
  -    foreign_keys_to_delete = []
  -    foreign_keys_to_insert = []
  -    indexes_to_delete = []
  -    indexes_to_insert = []
   
  -    for table in tables:
  -        table_name = table["name"]
  -        tables_to_insert.append(
  -            (database_name, table_name, table["rootpage"], table["sql"])
  -        )
  -        columns_to_delete.append((database_name, table_name))
  -        columns = await db.table_column_details(table_name)
  -        columns_to_insert.extend(
  -            {
  -                **{"database_name": database_name, "table_name": table_name},
  -                **column._asdict(),
  -            }
  -            for column in columns
  -        )
  -        foreign_keys_to_delete.append((database_name, table_name))
  -        foreign_keys = (
  -            await db.execute(f"PRAGMA foreign_key_list([{table_name}])")
  -        ).rows
  -        foreign_keys_to_insert.extend(
  -            {
  -                **{"database_name": database_name, "table_name": table_name},
  -                **dict(foreign_key),
  -            }
  -            for foreign_key in foreign_keys
  -        )
  -        indexes_to_delete.append((database_name, table_name))
  -        indexes = (await db.execute(f"PRAGMA index_list([{table_name}])")).rows
  -        indexes_to_insert.extend(
  -            {
  -                **{"database_name": database_name, "table_name": table_name},
  -                **dict(index),
  -            }
  -            for index in indexes
  +    def collect_info(conn):
  +        tables_to_insert = []
  +        columns_to_delete = []
  +        columns_to_insert = []
  +        foreign_keys_to_delete = []
  +        foreign_keys_to_insert = []
  +        indexes_to_delete = []
  +        indexes_to_insert = []
  +
  +        for table in tables:
  +            table_name = table["name"]
  +            tables_to_insert.append(
  +                (database_name, table_name, table["rootpage"], table["sql"])
  +            )
  +            columns_to_delete.append((database_name, table_name))
  +            columns = table_column_details(conn, table_name)
  +            columns_to_insert.extend(
  +                {
  +                    **{"database_name": database_name, "table_name": table_name},
  +                    **column._asdict(),
  +                }
  +                for column in columns
  +            )
  +            foreign_keys_to_delete.append((database_name, table_name))
  +            foreign_keys = conn.execute(
  +                f"PRAGMA foreign_key_list([{table_name}])"
  +            ).fetchall()
  +            foreign_keys_to_insert.extend(
  +                {
  +                    **{"database_name": database_name, "table_name": table_name},
  +                    **dict(foreign_key),
  +                }
  +                for foreign_key in foreign_keys
  +            )
  +            indexes_to_delete.append((database_name, table_name))
  +            indexes = conn.execute(f"PRAGMA index_list([{table_name}])").fetchall()
  +            indexes_to_insert.extend(
  +                {
  +                    **{"database_name": database_name, "table_name": table_name},
  +                    **dict(index),
  +                }
  +                for index in indexes
  +            )
  +        return (
  +            tables_to_insert,
  +            columns_to_delete,
  +            columns_to_insert,
  +            foreign_keys_to_delete,
  +            foreign_keys_to_insert,
  +            indexes_to_delete,
  +            indexes_to_insert,
           )
   
  +    (
  +        tables_to_insert,
  +        columns_to_delete,
  +        columns_to_insert,
  +        foreign_keys_to_delete,
  +        foreign_keys_to_insert,
  +        indexes_to_delete,
  +        indexes_to_insert,
  +    ) = await db.execute_fn(collect_info)
  +
       await internal_db.execute_write_many(
           """
           INSERT INTO tables (database_name, table_name, rootpage, sql)
  ```
  First impressions: it looks like this helps **a lot** - as far as I can tell this is now taking around 21ms to get to the point at which all of those internal databases have been populated, where previously it took more than 180ms.

  ![CleanShot 2021-12-18 at 19 47 22@2x](https://user-images.githubusercontent.com/9599/146663192-bba098d5-e7bd-4e2e-b525-2270867888a0.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1555#issuecomment-997325189
issue_url: https://api.github.com/repos/simonw/datasette/issues/1555
node_id: IC_kwDOBm6k_c47cfmF
created_at: 2021-12-19T03:55:01Z
updated_at: 2021-12-19T20:54:51Z
author_association: OWNER
body:
  It's a bit annoying that the queries no longer show up in the trace at all now, thanks to running in `.execute_fn()`. I wonder if there's something smart I can do about that - maybe have `trace()` record that function with a traceback even though it doesn't have the executed SQL string?

  5fac26aa221a111d7633f2dd92014641f7c0ade9 has the same problem.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1555#issuecomment-997342494
issue_url: https://api.github.com/repos/simonw/datasette/issues/1555
node_id: IC_kwDOBm6k_c47cj0e
created_at: 2021-12-19T07:22:04Z
updated_at: 2021-12-19T07:22:04Z
author_association: OWNER
body:
  Another option would be to provide an abstraction that makes it easier to run a group of SQL queries in the same thread at the same time, and have them traced correctly.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1566#issuecomment-997457790
issue_url: https://api.github.com/repos/simonw/datasette/issues/1566
node_id: IC_kwDOBm6k_c47c_9-
created_at: 2021-12-19T20:40:50Z
updated_at: 2021-12-19T20:40:57Z
author_association: OWNER
body:
  Also release new version of `datasette-pretty-traces` with this feature:
  - https://github.com/simonw/datasette-pretty-traces/issues/7

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1555#issuecomment-997459637
issue_url: https://api.github.com/repos/simonw/datasette/issues/1555
node_id: IC_kwDOBm6k_c47dAa1
created_at: 2021-12-19T20:53:46Z
updated_at: 2021-12-19T20:53:46Z
author_association: OWNER
body:
  Using #1571 showed me that the `DELETE FROM columns/foreign_keys/indexes WHERE database_name = ? and table_name = ?` queries were running way more times than I expected. I came up with a new optimization that just does `DELETE FROM columns/foreign_keys/indexes WHERE database_name = ?` instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1555#issuecomment-997459958
issue_url: https://api.github.com/repos/simonw/datasette/issues/1555
node_id: IC_kwDOBm6k_c47dAf2
created_at: 2021-12-19T20:55:59Z
updated_at: 2021-12-19T20:55:59Z
author_association: OWNER
body:
  Closing this issue because I've optimized this a whole bunch, and it's definitely good enough for the moment.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1570#issuecomment-997460061
issue_url: https://api.github.com/repos/simonw/datasette/issues/1570
node_id: IC_kwDOBm6k_c47dAhd
created_at: 2021-12-19T20:56:54Z
updated_at: 2021-12-19T20:56:54Z
author_association: OWNER
body:
  Documentation: https://docs.datasette.io/en/latest/internals.html#await-db-execute-write-sql-params-none-block-false

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1547#issuecomment-997460731
issue_url: https://api.github.com/repos/simonw/datasette/issues/1547
node_id: IC_kwDOBm6k_c47dAr7
created_at: 2021-12-19T21:02:15Z
updated_at: 2021-12-19T21:02:15Z
author_association: OWNER
body:
  Yes, this is a bug. It looks like the problem is with the `if write:` branch in this code here: https://github.com/simonw/datasette/blob/5fac26aa221a111d7633f2dd92014641f7c0ade9/datasette/views/database.py#L252-L327

  Is missing this bit of code:

  https://github.com/simonw/datasette/blob/5fac26aa221a111d7633f2dd92014641f7c0ade9/datasette/views/database.py#L343-L347

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1573#issuecomment-997462117
issue_url: https://api.github.com/repos/simonw/datasette/issues/1573
node_id: IC_kwDOBm6k_c47dBBl
created_at: 2021-12-19T21:13:13Z
updated_at: 2021-12-19T21:13:13Z
author_association: OWNER
body:
  This might also be the impetus I need to bring the https://datasette.io/plugins/datasette-pretty-traces plugin into Datasette core itself.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1545#issuecomment-997462604
issue_url: https://api.github.com/repos/simonw/datasette/issues/1545
node_id: IC_kwDOBm6k_c47dBJM
created_at: 2021-12-19T21:17:08Z
updated_at: 2021-12-19T21:17:08Z
author_association: OWNER
body:
  Here's the relevant code: https://github.com/simonw/datasette/blob/4094741c2881c2ada3f3f878b532fdaec7914953/datasette/app.py#L1204-L1219

  It's using `route_path.split("/")` which should be OK because that's the incoming `request.path` path - which I would expect to use `/` even on Windows. Then it uses `os.path.join` which should do the right thing.

  I need to get myself a proper Windows development environment setup to investigate this one.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1547#issuecomment-997471672
issue_url: https://api.github.com/repos/simonw/datasette/issues/1547
node_id: IC_kwDOBm6k_c47dDW4
created_at: 2021-12-19T22:18:26Z
updated_at: 2021-12-19T22:18:26Z
author_association: OWNER
body:
  I released this [in an alpha](https://github.com/simonw/datasette/releases/tag/0.60a1), so you can try out this fix using:

      pip install datasette==0.60a1

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-997472214
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c47dDfW
created_at: 2021-12-19T22:22:08Z
updated_at: 2021-12-19T22:22:08Z
author_association: OWNER
body:
  I sketched out a chained SQL builder pattern that might be useful for further tidying up this code - though with the new plugin hook I'm less excited about it than I was:

  ```python
  class TableQuery:
      def __init__(self, table, columns, pks, is_view=False, prev=None):
          self.table = table
          self.columns = columns
          self.pks = pks
          self.is_view = is_view
          self.prev = prev
          
          # These can be changed for different instances in the chain:
          self._where_clauses = None
          self._order_by = None
          self._page_size = None
          self._offset = None
          self._select_columns = None

          self.select_all_columns = '*'
          self.select_specified_columns = '*'

      @property
      def where_clauses(self):
          wheres = []
          current = self
          while current:
              if current._where_clauses is not None:
                  wheres.extend(current._where_clauses)
              current = current.prev
          return list(reversed(wheres))

      def where(self, where):
          new_cls = TableQuery(self.table, self.columns, self.pks, self.is_view, self)
          new_cls._where_clauses = [where]
          return new_cls
          
      @classmethod
      async def introspect(cls, db, table):
          return cls(
              table,
              columns = await db.table_columns(table),
              pks = await db.primary_keys(table),
              is_view = bool(await db.get_view_definition(table))
          )
          
      @property
      def sql_from(self):
          return f"from {self.table}{self.sql_where}"

      @property
      def sql_where(self):
          if not self.where_clauses:
              return ""
          else:
              return f" where {' and '.join(self.where_clauses)}"

      @property
      def sql_no_order_no_limit(self):
          return f"select {self.select_all_columns} from {self.table}{self.sql_where}"

      @property
      def sql(self):
          return f"select {self.select_specified_columns} from {self.table} {self.sql_where}{self._order_by} limit {self._page_size}{self._offset}"

      @property
      def sql_count(self):
          return f"select count(*) {self.sql_from}"


      def __repr__(self):
          return f"<TableQuery sql={self.sql}>"
  ```
  Usage:
  ```python
  from datasette.app import Datasette
  ds = Datasette(memory=True, files=["/Users/simon/Dropbox/Development/datasette/fixtures.db"])
  db = ds.get_database("fixtures")
  query = await TableQuery.introspect(db, "facetable")
  print(query.where("foo = bar").where("baz = 1").sql_count)
  # 'select count(*) from facetable where foo = bar and baz = 1'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1565#issuecomment-997472509
issue_url: https://api.github.com/repos/simonw/datasette/issues/1565
node_id: IC_kwDOBm6k_c47dDj9
created_at: 2021-12-19T22:24:50Z
updated_at: 2021-12-19T22:24:50Z
author_association: OWNER
body:
  ... huh, it could even expose a JavaScript function that can be called to execute a SQL query.

  ```javascript
  datasette.query("select * from blah").then(...)
  ```
  Maybe it takes an optional second argument that specifies the database - defaulting to the one for the current page.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1565#issuecomment-997472639
issue_url: https://api.github.com/repos/simonw/datasette/issues/1565
node_id: IC_kwDOBm6k_c47dDl_
created_at: 2021-12-19T22:25:50Z
updated_at: 2021-12-19T22:25:50Z
author_association: OWNER
body:
  Or...
  ```javascript
  rows = await datasette.query`select * from searchable where id > ${id}`;
  ```
  And it knows how to turn that into a parameterized call using tagged template literals.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1565#issuecomment-997473856
issue_url: https://api.github.com/repos/simonw/datasette/issues/1565
node_id: IC_kwDOBm6k_c47dD5A
created_at: 2021-12-19T22:35:20Z
updated_at: 2021-12-19T22:35:20Z
author_association: OWNER
body:
  Quick prototype of that tagged template `query` function:

  ```javascript
  function query(pieces, ...parameters) {
    var qs = new URLSearchParams();
    var sql = pieces[0];
    parameters.forEach((param, i) => {
      sql += `:p${i}${pieces[i + 1]}`;
      qs.append(`p${i}`, param);
    });
    qs.append("sql", sql);
    return qs.toString();
  }

  var id = 4;
  console.log(query`select * from ids where id > ${id}`);
  ```
  Outputs:
  ```
  p0=4&sql=select+*+from+ids+where+id+%3E+%3Ap0
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1565#issuecomment-997474022
issue_url: https://api.github.com/repos/simonw/datasette/issues/1565
node_id: IC_kwDOBm6k_c47dD7m
created_at: 2021-12-19T22:36:49Z
updated_at: 2021-12-19T22:37:29Z
author_association: OWNER
body:
  No way with a tagged template literal to pass an extra database name argument, so instead I need a method that returns a callable that can be used for the tagged template literal for a specific database - or the default database.

  This could work (bit weird looking though):
  ```javascript
  var rows = await datasette.query("fixtures")`select * from foo`;
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997485361
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/356
node_id: IC_kwDOCGYnMM47dGsx
created_at: 2021-12-19T23:45:30Z
updated_at: 2021-12-19T23:45:30Z
author_association: OWNER
body:
  Really interesting example input for this: https://blog.timac.org/2021/1219-state-of-swift-and-swiftui-ios15/iOS13.txt - see https://blog.timac.org/2021/1219-state-of-swift-and-swiftui-ios15/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997492872
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/356
node_id: IC_kwDOCGYnMM47dIiI
created_at: 2021-12-20T00:23:31Z
updated_at: 2021-12-20T00:23:31Z
author_association: OWNER
body:
  I think this should work on JSON, or CSV, or individual lines, or the entire content at once.

  So I'll require `--lines --convert ...` to import individual lines, or `--all --convert` to run the conversion against the entire input at once.

  What would `--lines` or `--all` do without `--convert`? Maybe insert records as `{"line": "line of text"}` or `{"all": "whole input}`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997496626
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/356
node_id: IC_kwDOCGYnMM47dJcy
created_at: 2021-12-20T00:38:15Z
updated_at: 2022-01-06T01:29:03Z
author_association: OWNER
body:
  The implementation of this gets a tiny bit complicated.

  Ignoring `--convert`, the `--lines` option can internally produce `{"line": ...}` records and the `--all` option can produce `{"all": ...}` records.

  But... when `--convert` is used, what should the code run against?

  It could run against those already-converted records but that's a little bit strange, since you'd have to do this:

      sqlite-utils insert blah.db blah myfile.txt --all --convert '{"item": s for s in value["all"].split("-")}'

  Having to use `value["all"]` there is unintuitive. It would be nicer to have a `all` variable to work against.

  But then for `--lines` should the local variable be called `line`? And how best to summarize these different names for local variables in the inline help for the feature?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997496931
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/356
node_id: IC_kwDOCGYnMM47dJhj
created_at: 2021-12-20T00:39:14Z
updated_at: 2021-12-20T00:39:52Z
author_association: OWNER
body:
  ```
  % sqlite-utils insert --help
  Usage: sqlite-utils insert [OPTIONS] PATH TABLE JSON_FILE

    Insert records from JSON file into a table, creating the table if it does
    not already exist.

    Input should be a JSON array of objects, unless --nl or --csv is used.

  Options:
    --pk TEXT                 Columns to use as the primary key, e.g. id
    --nl                      Expect newline-delimited JSON
    --flatten                 Flatten nested JSON objects
    -c, --csv                 Expect CSV
    --tsv                     Expect TSV
    --convert TEXT            Python code to convert each item
    --import TEXT             Python modules to import
    --delimiter TEXT          Delimiter to use for CSV files
    --quotechar TEXT          Quote character to use for CSV/TSV
    --sniff                   Detect delimiter and quote character
    --no-headers              CSV file has no header row
    --batch-size INTEGER      Commit every X records
    --alter                   Alter existing table to add any missing columns
    --not-null TEXT           Columns that should be created as NOT NULL
    --default <TEXT TEXT>...  Default value that should be set for a column
    --encoding TEXT           Character encoding for input, defaults to utf-8
    -d, --detect-types        Detect types for columns in CSV/TSV data
    --load-extension TEXT     SQLite extensions to load
    --silent                  Do not show progress bar
    --ignore                  Ignore records if pk already exists
    --replace                 Replace records if pk already exists
    --truncate                Truncate table before inserting records, if table
                              already exists
    -h, --help                Show this message and exit.
  ```
  I can add a bunch of extra help at the top there to explain all of this stuff. That "Input should be a JSON array of objects" bit could be expanded to several paragraphs.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/163#issuecomment-997502242
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/163
node_id: IC_kwDOCGYnMM47dK0i
created_at: 2021-12-20T00:56:45Z
updated_at: 2021-12-20T00:56:52Z
author_association: OWNER
body:
  > Maybe `sqlite-utils` should absorb all of the functionality from `sqlite-transform` - having two separate tools doesn't necessarily make sense.

  I implemented that in:
  - #251

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997507074
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/356
node_id: IC_kwDOCGYnMM47dMAC
created_at: 2021-12-20T01:10:06Z
updated_at: 2021-12-20T01:16:11Z
author_association: OWNER
body:
  Work-in-progress improved help:
  ```
  Usage: sqlite-utils insert [OPTIONS] PATH TABLE FILE

    Insert records from FILE into a table, creating the table if it does not
    already exist.

    By default the input is expected to be a JSON array of objects. Or:

    - Use --nl for newline-delimited JSON objects
    - Use --csv or --tsv for comma-separated or tab-separated input
    - Use --lines to write each incoming line to a column called "line"
    - Use --all to write the entire input to a column called "all"

    You can also use --convert to pass a fragment of Python code that will be
    used to convert each input.

    Your Python code will be passed a "row" variable representing the imported
    row, and can return a modified row.

    If you are using --lines your code will be passed a "line" variable, and for
    --all an "all" variable.

  Options:
    --pk TEXT                 Columns to use as the primary key, e.g. id
    --flatten                 Flatten nested JSON objects, so {"a": {"b": 1}}
                              becomes {"a_b": 1}
    --nl                      Expect newline-delimited JSON
    -c, --csv                 Expect CSV input
    --tsv                     Expect TSV input
    --lines                   Treat each line as a single value called 'line'
    --all                     Treat input as a single value called 'all'
    --convert TEXT            Python code to convert each item
    --import TEXT             Python modules to import
    --delimiter TEXT          Delimiter to use for CSV files
    --quotechar TEXT          Quote character to use for CSV/TSV
    --sniff                   Detect delimiter and quote character
    --no-headers              CSV file has no header row
    --batch-size INTEGER      Commit every X records
    --alter                   Alter existing table to add any missing columns
    --not-null TEXT           Columns that should be created as NOT NULL
    --default <TEXT TEXT>...  Default value that should be set for a column
    --encoding TEXT           Character encoding for input, defaults to utf-8
    -d, --detect-types        Detect types for columns in CSV/TSV data
    --load-extension TEXT     SQLite extensions to load
    --silent                  Do not show progress bar
    --ignore                  Ignore records if pk already exists
    --replace                 Replace records if pk already exists
    --truncate                Truncate table before inserting records, if table
                              already exists
    -h, --help                Show this message and exit.
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1547#issuecomment-997511968
issue_url: https://api.github.com/repos/simonw/datasette/issues/1547
node_id: IC_kwDOBm6k_c47dNMg
created_at: 2021-12-20T01:21:59Z
updated_at: 2021-12-20T01:21:59Z
author_association: CONTRIBUTOR
body:
  I've installed the alpha version but get an error when starting up Datasette:

  ```
  Traceback (most recent call last):
    File "/Users/tim/.pyenv/versions/stock-exchange/bin/datasette", line 5, in <module>
      from datasette.cli import cli
    File "/Users/tim/.pyenv/versions/3.8.5/envs/stock-exchange/lib/python3.8/site-packages/datasette/cli.py", line 15, in <module>
      from .app import Datasette, DEFAULT_SETTINGS, SETTINGS, SQLITE_LIMIT_ATTACHED, pm
    File "/Users/tim/.pyenv/versions/3.8.5/envs/stock-exchange/lib/python3.8/site-packages/datasette/app.py", line 31, in <module>
      from .views.database import DatabaseDownload, DatabaseView
    File "/Users/tim/.pyenv/versions/3.8.5/envs/stock-exchange/lib/python3.8/site-packages/datasette/views/database.py", line 25, in <module>
      from datasette.plugins import pm
    File "/Users/tim/.pyenv/versions/3.8.5/envs/stock-exchange/lib/python3.8/site-packages/datasette/plugins.py", line 29, in <module>
      mod = importlib.import_module(plugin)
    File "/Users/tim/.pyenv/versions/3.8.5/lib/python3.8/importlib/__init__.py", line 127, in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
    File "/Users/tim/.pyenv/versions/3.8.5/envs/stock-exchange/lib/python3.8/site-packages/datasette/filters.py", line 9, in <module>
      @hookimpl(specname="filters_from_request")
  TypeError: __call__() got an unexpected keyword argument 'specname'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1575#issuecomment-997513177
issue_url: https://api.github.com/repos/simonw/datasette/issues/1575
node_id: IC_kwDOBm6k_c47dNfZ
created_at: 2021-12-20T01:24:25Z
updated_at: 2021-12-20T01:24:25Z
author_association: OWNER
body:
  Looks like `specname` is new in Pluggy 1.0: https://github.com/pytest-dev/pluggy/blob/main/CHANGELOG.rst#pluggy-100-2021-08-25

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1547#issuecomment-997514220
issue_url: https://api.github.com/repos/simonw/datasette/issues/1547
node_id: IC_kwDOBm6k_c47dNvs
created_at: 2021-12-20T01:26:25Z
updated_at: 2021-12-20T01:26:25Z
author_association: OWNER
body:
  OK, this should hopefully fix that for you:

      pip install https://github.com/simonw/datasette/archive/f36e010b3b69ada104b79d83c7685caf9359049e.zip

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1181#issuecomment-998999230
issue_url: https://api.github.com/repos/simonw/datasette/issues/1181
node_id: IC_kwDOBm6k_c47i4S-
created_at: 2021-12-21T18:25:15Z
updated_at: 2021-12-21T18:25:15Z
author_association: NONE
body:
  I wonder if I'm encountering the same bug (or something related). I had previously been using the .csv feature to run queries and then fetch results for the pandas `read_csv()` function, but it seems to have stopped working recently.

  https://ilsweb.cincinnatilibrary.org/collection-analysis/collection-analysis/current_collection-3d56dbf.csv?sql=select%0D%0A++*%0D%0Afrom%0D%0A++bib%0D%0Alimit%0D%0A++100&_size=max

  Datasette v0.59.4
  ![image](https://user-images.githubusercontent.com/9308268/146979957-66911877-2cd9-4022-bc76-fd54e4a3a6f7.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-999831967
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c47mDmf
created_at: 2021-12-22T20:04:47Z
updated_at: 2021-12-22T20:10:11Z
author_association: OWNER
body:
  I think I might be able to clean up a lot of the stuff in here using the `render_cell` plugin hook: https://github.com/simonw/datasette/blob/6b1384b2f529134998fb507e63307609a5b7f5c0/datasette/views/table.py#L87-L89

  The catch with that hook - https://docs.datasette.io/en/stable/plugin_hooks.html#render-cell-value-column-table-database-datasette - is that it gets called for every single cell. I don't want the overhead of looking up the foreign key relationships etc once for every value in a specific column.

  But maybe I could extend the hook to include a shared cache that gets used for all of the cells in a specific table? Something like this:
  ```python
  render_cell(value, column, table, database, datasette, cache)
  ```
  `cache` is a dictionary - and the same dictionary is passed to every call to that hook while rendering a specific page.

  It's a bit of a gross hack though, and would it ever be useful for plugins outside of the default plugin in Datasette which does the foreign key stuff?

  If I can think of one other potential application for this `cache` then I might implement it.

  No, this optimization doesn't make sense: the most complex cell enrichment logic is the stuff that does a `select * from categories where id in (2, 5, 6)` query, using just the distinct set of IDs that are rendered on the current page. That's not going to fit in the `render_cell` hook no matter how hard I try to warp it into the right shape, because it needs full visibility of all of the results that are being rendered in order to collect those unique ID values.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-999837220
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c47mE4k
created_at: 2021-12-22T20:15:04Z
updated_at: 2021-12-22T20:15:04Z
author_association: OWNER
body:
  I think I can move this much higher up in the method, it's a bit confusing having it half way through: https://github.com/simonw/datasette/blob/6b1384b2f529134998fb507e63307609a5b7f5c0/datasette/views/table.py#L414-L436

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-999837569
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c47mE-B
created_at: 2021-12-22T20:15:45Z
updated_at: 2021-12-22T20:15:45Z
author_association: OWNER
body:
  Also the whole `special_args` v.s. `request.args` thing is pretty confusing, I think that might be an older code pattern back from when I was using Sanic.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-999850191
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c47mIDP
created_at: 2021-12-22T20:29:38Z
updated_at: 2021-12-22T20:29:38Z
author_association: OWNER
body:
  New short-term goal: get facets and suggested facets to execute in parallel with the main query. Generate a trace graph that proves that is happening using `datasette-pretty-traces`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-999863269
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c47mLPl
created_at: 2021-12-22T20:35:41Z
updated_at: 2021-12-22T20:37:13Z
author_association: OWNER
body:
  It looks like the count has to be executed before facets can be, because the facet_class constructor needs that total count figure: https://github.com/simonw/datasette/blob/6b1384b2f529134998fb507e63307609a5b7f5c0/datasette/views/table.py#L660-L671

  It's used in facet suggestion logic here: https://github.com/simonw/datasette/blob/ace86566b28280091b3844cf5fbecd20158e9004/datasette/facets.py#L172-L178

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-999870282
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c47mM9K
created_at: 2021-12-22T20:45:56Z
updated_at: 2021-12-22T20:46:08Z
author_association: OWNER
body:
  > New short-term goal: get facets and suggested facets to execute in parallel with the main query. Generate a trace graph that proves that is happening using `datasette-pretty-traces`.

  I wrote code to execute those in parallel using `asyncio.gather()` - which seems to work but causes the SQL run inside the parallel `async def` functions not to show up in the trace graph at all.

  ```diff
  diff --git a/datasette/views/table.py b/datasette/views/table.py
  index 9808fd2..ec9db64 100644
  --- a/datasette/views/table.py
  +++ b/datasette/views/table.py
  @@ -1,3 +1,4 @@
  +import asyncio
   import urllib
   import itertools
   import json
  @@ -615,44 +616,37 @@ class TableView(RowTableShared):
           if request.args.get("_timelimit"):
               extra_args["custom_time_limit"] = int(request.args.get("_timelimit"))
   
  -        # Execute the main query!
  -        results = await db.execute(sql, params, truncate=True, **extra_args)
  -
  -        # Calculate the total count for this query
  -        filtered_table_rows_count = None
  -        if (
  -            not db.is_mutable
  -            and self.ds.inspect_data
  -            and count_sql == f"select count(*) from {table} "
  -        ):
  -            # We can use a previously cached table row count
  -            try:
  -                filtered_table_rows_count = self.ds.inspect_data[database]["tables"][
  -                    table
  -                ]["count"]
  -            except KeyError:
  -                pass
  -
  -        # Otherwise run a select count(*) ...
  -        if count_sql and filtered_table_rows_count is None and not nocount:
  -            try:
  -                count_rows = list(await db.execute(count_sql, from_sql_params))
  -                filtered_table_rows_count = count_rows[0][0]
  -            except QueryInterrupted:
  -                pass
  -
  -        # Faceting
  -        if not self.ds.setting("allow_facet") and any(
  -            arg.startswith("_facet") for arg in request.args
  -        ):
  -            raise BadRequest("_facet= is not allowed")
  +        async def execute_count():
  +            # Calculate the total count for this query
  +            filtered_table_rows_count = None
  +            if (
  +                not db.is_mutable
  +                and self.ds.inspect_data
  +                and count_sql == f"select count(*) from {table} "
  +            ):
  +                # We can use a previously cached table row count
  +                try:
  +                    filtered_table_rows_count = self.ds.inspect_data[database][
  +                        "tables"
  +                    ][table]["count"]
  +                except KeyError:
  +                    pass
  +
  +            if count_sql and filtered_table_rows_count is None and not nocount:
  +                try:
  +                    count_rows = list(await db.execute(count_sql, from_sql_params))
  +                    filtered_table_rows_count = count_rows[0][0]
  +                except QueryInterrupted:
  +                    pass
  +
  +            return filtered_table_rows_count
  +
  +        filtered_table_rows_count = await execute_count()
   
           # pylint: disable=no-member
           facet_classes = list(
               itertools.chain.from_iterable(pm.hook.register_facet_classes())
           )
  -        facet_results = {}
  -        facets_timed_out = []
           facet_instances = []
           for klass in facet_classes:
               facet_instances.append(
  @@ -668,33 +662,58 @@ class TableView(RowTableShared):
                   )
               )
   
  -        if not nofacet:
  -            for facet in facet_instances:
  -                (
  -                    instance_facet_results,
  -                    instance_facets_timed_out,
  -                ) = await facet.facet_results()
  -                for facet_info in instance_facet_results:
  -                    base_key = facet_info["name"]
  -                    key = base_key
  -                    i = 1
  -                    while key in facet_results:
  -                        i += 1
  -                        key = f"{base_key}_{i}"
  -                    facet_results[key] = facet_info
  -                facets_timed_out.extend(instance_facets_timed_out)
  -
  -        # Calculate suggested facets
  -        suggested_facets = []
  -        if (
  -            self.ds.setting("suggest_facets")
  -            and self.ds.setting("allow_facet")
  -            and not _next
  -            and not nofacet
  -            and not nosuggest
  -        ):
  -            for facet in facet_instances:
  -                suggested_facets.extend(await facet.suggest())
  +        async def execute_suggested_facets():
  +            # Calculate suggested facets
  +            suggested_facets = []
  +            if (
  +                self.ds.setting("suggest_facets")
  +                and self.ds.setting("allow_facet")
  +                and not _next
  +                and not nofacet
  +                and not nosuggest
  +            ):
  +                for facet in facet_instances:
  +                    suggested_facets.extend(await facet.suggest())
  +            return suggested_facets
  +
  +        async def execute_facets():
  +            facet_results = {}
  +            facets_timed_out = []
  +            if not self.ds.setting("allow_facet") and any(
  +                arg.startswith("_facet") for arg in request.args
  +            ):
  +                raise BadRequest("_facet= is not allowed")
  +
  +            if not nofacet:
  +                for facet in facet_instances:
  +                    (
  +                        instance_facet_results,
  +                        instance_facets_timed_out,
  +                    ) = await facet.facet_results()
  +                    for facet_info in instance_facet_results:
  +                        base_key = facet_info["name"]
  +                        key = base_key
  +                        i = 1
  +                        while key in facet_results:
  +                            i += 1
  +                            key = f"{base_key}_{i}"
  +                        facet_results[key] = facet_info
  +                    facets_timed_out.extend(instance_facets_timed_out)
  +
  +            return facet_results, facets_timed_out
  +
  +        # Execute the main query, facets and facet suggestions in parallel:
  +        (
  +            results,
  +            suggested_facets,
  +            (facet_results, facets_timed_out),
  +        ) = await asyncio.gather(
  +            db.execute(sql, params, truncate=True, **extra_args),
  +            execute_suggested_facets(),
  +            execute_facets(),
  +        )
  +
  +        results = await db.execute(sql, params, truncate=True, **extra_args)
   
           # Figure out columns and rows for the query
           columns = [r[0] for r in results.description]
  ```
  Here's the trace for `http://127.0.0.1:4422/fixtures/compound_three_primary_keys?_trace=1&_facet=pk1&_facet=pk2` with the missing facet and facet suggestion queries:

  <img width="1447" alt="image" src="https://user-images.githubusercontent.com/9599/147153051-62cdb9a5-de5e-4cc3-9215-b779f92a81c8.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-999870993
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c47mNIR
created_at: 2021-12-22T20:47:18Z
updated_at: 2021-12-22T20:50:24Z
author_association: OWNER
body:
  The reason they aren't showing up in the traces is that traces are stored just for the currently executing `asyncio` task ID: https://github.com/simonw/datasette/blob/ace86566b28280091b3844cf5fbecd20158e9004/datasette/tracer.py#L13-L25

  This is so traces for other incoming requests don't end up mixed together. But there's no current mechanism to track async tasks that are effectively "child tasks" of the current request, and hence should be tracked the same.

  https://stackoverflow.com/a/69349501/6083 suggests that you pass the task ID as an argument to the child tasks that are executed using `asyncio.gather()` to work around this kind of problem.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1576#issuecomment-999874484
issue_url: https://api.github.com/repos/simonw/datasette/issues/1576
node_id: IC_kwDOBm6k_c47mN-0
created_at: 2021-12-22T20:54:52Z
updated_at: 2021-12-22T20:54:52Z
author_association: OWNER
body:
  Here's the full current relevant code from `tracer.py`: https://github.com/simonw/datasette/blob/ace86566b28280091b3844cf5fbecd20158e9004/datasette/tracer.py#L8-L64

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1576#issuecomment-999874886
issue_url: https://api.github.com/repos/simonw/datasette/issues/1576
node_id: IC_kwDOBm6k_c47mOFG
created_at: 2021-12-22T20:55:42Z
updated_at: 2021-12-22T20:57:28Z
author_association: OWNER
body:
  One way to solve this would be to introduce a `set_task_id()` method, which sets an ID which will be returned by `get_task_id()` instead of using `id(current_task(loop=loop))`.

  It would be really nice if I could solve this using `with` syntax somehow. Something like:
  ```python
  with trace_child_tasks():
      (
          suggested_facets,
          (facet_results, facets_timed_out),
      ) = await asyncio.gather(
          execute_suggested_facets(),
          execute_facets(),
      )
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1576#issuecomment-999876666
issue_url: https://api.github.com/repos/simonw/datasette/issues/1576
node_id: IC_kwDOBm6k_c47mOg6
created_at: 2021-12-22T20:59:22Z
updated_at: 2021-12-22T21:18:09Z
author_association: OWNER
body:
  This article is relevant: [Context information storage for asyncio](https://blog.sqreen.com/asyncio/) - in particular the section https://blog.sqreen.com/asyncio/#context-inheritance-between-tasks which describes exactly the problem I have and their solution, which involves this trickery:

  ```python
  def request_task_factory(loop, coro):
      child_task = asyncio.tasks.Task(coro, loop=loop)
      parent_task = asyncio.Task.current_task(loop=loop)
      current_request = getattr(parent_task, 'current_request', None)
      setattr(child_task, 'current_request', current_request)
      return child_task

  loop = asyncio.get_event_loop()
  loop.set_task_factory(request_task_factory)
  ```

  They released their solution as a library: https://pypi.org/project/aiocontext/ and https://github.com/sqreen/AioContext - but that company was acquired by Datadog back in April and doesn't seem to be actively maintaining their open source stuff any more: https://twitter.com/SqreenIO/status/1384906075506364417

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1576#issuecomment-999878907
issue_url: https://api.github.com/repos/simonw/datasette/issues/1576
node_id: IC_kwDOBm6k_c47mPD7
created_at: 2021-12-22T21:03:49Z
updated_at: 2021-12-22T21:10:46Z
author_association: OWNER
body:
  `context_vars` can solve this but they were introduced in Python 3.7: https://www.python.org/dev/peps/pep-0567/

  Python 3.6 support ends in a few days time, and it looks like Glitch has updated to 3.7 now - so maybe I can get away with Datasette needing 3.7 these days?

  Tweeted about that here: https://twitter.com/simonw/status/1473761478155010048

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1576#issuecomment-999987418
issue_url: https://api.github.com/repos/simonw/datasette/issues/1576
node_id: IC_kwDOBm6k_c47mpja
created_at: 2021-12-23T01:59:58Z
updated_at: 2021-12-23T02:02:12Z
author_association: OWNER
body:
  Another option: https://github.com/Skyscanner/aiotask-context - looks like it might be better as it's been updated for Python 3.7 in this commit https://github.com/Skyscanner/aiotask-context/commit/67108c91d2abb445655cc2af446fdb52ca7890c4

  The Skyscanner one doesn't attempt to wrap any existing factories, but that's OK for my purposes since I don't need to handle arbitrary `asyncio` code written by other people.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1576#issuecomment-999990414
issue_url: https://api.github.com/repos/simonw/datasette/issues/1576
node_id: IC_kwDOBm6k_c47mqSO
created_at: 2021-12-23T02:08:39Z
updated_at: 2021-12-23T18:16:35Z
author_association: OWNER
body:
  It's tiny: I'm tempted to vendor it. https://github.com/Skyscanner/aiotask-context/blob/master/aiotask_context/__init__.py

  No, I'll add it as a pinned dependency, which I can then drop when I drop 3.6 support.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1577#issuecomment-1000461275
issue_url: https://api.github.com/repos/simonw/datasette/issues/1577
node_id: IC_kwDOBm6k_c47odPb
created_at: 2021-12-23T18:18:11Z
updated_at: 2021-12-23T18:18:11Z
author_association: OWNER
body:
  From the Twitter thread, there are still a decent amount of LTS Linux releases out there that are stuck on pre-3.7 Python.

  Though many of those are 3.5 and Datasette dropped support for 3.5 in November 2019: cf7776d36fbacefa874cbd6e5fcdc9fff7661203

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1577#issuecomment-1000461900
issue_url: https://api.github.com/repos/simonw/datasette/issues/1577
node_id: IC_kwDOBm6k_c47odZM
created_at: 2021-12-23T18:19:44Z
updated_at: 2021-12-23T18:19:44Z
author_association: OWNER
body:
  The 3.7 feature I want to use today is [contextvars](https://docs.python.org/3/library/contextvars.html) - but I have a workaround for the moment, see https://github.com/simonw/datasette/issues/1576#issuecomment-999987418

  So I'm going to hold off on dropping 3.6 for a little bit longer. I imagine I'll drop it before Datasette 1.0 though.

  Leaving this issue open to gather thoughts and feedback on this issue from Datasette users and potential users.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1577#issuecomment-1000462309
issue_url: https://api.github.com/repos/simonw/datasette/issues/1577
node_id: IC_kwDOBm6k_c47odfl
created_at: 2021-12-23T18:20:46Z
updated_at: 2021-12-23T18:20:46Z
author_association: OWNER
body:
  There are a lot of improvements to `asyncio` in 3.7: https://docs.python.org/3/whatsnew/3.7.html#whatsnew37-asyncio

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1578#issuecomment-1000469107
issue_url: https://api.github.com/repos/simonw/datasette/issues/1578
node_id: IC_kwDOBm6k_c47ofJz
created_at: 2021-12-23T18:36:38Z
updated_at: 2021-12-23T18:36:38Z
author_association: OWNER
body:
  This problem doesn't occur on my `localhost` running Uvicorn directly - but I'm seeing it in my production environment that runs Datasette behind an nginx proxy:

  ```
      location / {
          proxy_pass http://127.0.0.1:8000/;
  	proxy_set_header Host $host;
      }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1578#issuecomment-1000470652
issue_url: https://api.github.com/repos/simonw/datasette/issues/1578
node_id: IC_kwDOBm6k_c47ofh8
created_at: 2021-12-23T18:40:46Z
updated_at: 2021-12-23T18:40:46Z
author_association: OWNER
body:
  [This StackOverflow answer](https://serverfault.com/a/463932) suggests that the fix is to change this:

      proxy_pass http://127.0.0.1:8000/;

  To this:

      proxy_pass http://127.0.0.1:8000;

  Quoting the nginx documentation: http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass

  > A request URI is passed to the server as follows:
  > 
  > -   If the `proxy_pass` directive is specified with a URI, then when a request is passed to the server, the part of a [normalized](http://nginx.org/en/docs/http/ngx_http_core_module.html#location) request URI matching the location is replaced by a URI specified in the directive:
  > 
  >         location /name/ {
  >             proxy_pass http://127.0.0.1/remote/;
  >         }
  > 
  > -   If `proxy_pass` is specified without a URI, the request URI is passed to the server in the same form as sent by a client when the original request is processed, or the full normalized request URI is passed when processing the changed URI:
  > 
  >         location /some/path/ {
  >             proxy_pass http://127.0.0.1;
  >         }

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1578#issuecomment-1000471782
issue_url: https://api.github.com/repos/simonw/datasette/issues/1578
node_id: IC_kwDOBm6k_c47ofzm
created_at: 2021-12-23T18:44:01Z
updated_at: 2021-12-23T18:44:01Z
author_association: OWNER
body:
  The example nginx config on https://docs.datasette.io/en/stable/deploying.html#nginx-proxy-configuration is currently:

  ```
  daemon off;

  events {
    worker_connections  1024;
  }
  http {
    server {
      listen 80;
      location /my-datasette {
        proxy_pass http://127.0.0.1:8009/my-datasette;
        proxy_set_header Host $host;
      }
    }
  }
  ```
  This looks to me like it might exhibit the bug. Need to confirm that and figure out an alternative.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1579#issuecomment-1000476413
issue_url: https://api.github.com/repos/simonw/datasette/issues/1579
node_id: IC_kwDOBm6k_c47og79
created_at: 2021-12-23T18:56:06Z
updated_at: 2021-12-23T18:56:06Z
author_association: OWNER
body:
  This is technically a breaking change, but a GitHub code search at https://cs.github.com/?scopeName=All+repos&scope=&q=execute_write%20datasette%20-owner%3Asimonw shows only one repo not-owned-by-me using this, and they're using `block=True`: https://github.com/mfa/datasette-webhook-write/blob/e82440f372a2f2e3ed27d1bd34c9fa3a53b49b94/datasette_webhook_write/__init__.py#L88-L89

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1579#issuecomment-1000477621
issue_url: https://api.github.com/repos/simonw/datasette/issues/1579
node_id: IC_kwDOBm6k_c47ohO1
created_at: 2021-12-23T18:59:12Z
updated_at: 2021-12-23T18:59:12Z
author_association: OWNER
body:
  The easiest way to change this would be to default to `block=True` such that you need to pass `block=False` to the APIs to have them do fire-and-forget.

  An alternative would be to add new, separately named methods which do the fire-and-forget thing.

  If I hadn't recently added `execute_write_script` and `execute_write_many` in #1570 I'd be more into this idea, but I don't want to end up with eight methods - `execute_write`, `execute_write_queue`, `execute_write_many`, `execute_write_many_queue`, `execute_write_script`, `execute_write_scrript_queue`, `execute_write_fn`, `execute_write_fn_queue`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1579#issuecomment-1000477813
issue_url: https://api.github.com/repos/simonw/datasette/issues/1579
node_id: IC_kwDOBm6k_c47ohR1
created_at: 2021-12-23T18:59:41Z
updated_at: 2021-12-23T18:59:41Z
author_association: OWNER
body:
  I'm going to go with `execute_write(..., block=False)` as the mechanism for fire-and-forget write queries.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1579#issuecomment-1000479737
issue_url: https://api.github.com/repos/simonw/datasette/issues/1579
node_id: IC_kwDOBm6k_c47ohv5
created_at: 2021-12-23T19:04:23Z
updated_at: 2021-12-23T19:04:23Z
author_association: OWNER
body:
  Updated documentation: https://github.com/simonw/datasette/blob/00a2895cd2dc42c63846216b36b2dc9f41170129/docs/internals.rst#await-dbexecute_writesql-paramsnone-blocktrue

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1579#issuecomment-1000481686
issue_url: https://api.github.com/repos/simonw/datasette/issues/1579
node_id: IC_kwDOBm6k_c47oiOW
created_at: 2021-12-23T19:09:23Z
updated_at: 2021-12-23T19:09:23Z
author_association: OWNER
body:
  Re-opening this because I missed updating some of the docs, and I also need to update Datasette's own code to not use `block=True` in a bunch of places.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1579#issuecomment-1000485505
issue_url: https://api.github.com/repos/simonw/datasette/issues/1579
node_id: IC_kwDOBm6k_c47ojKB
created_at: 2021-12-23T19:19:13Z
updated_at: 2021-12-23T19:19:13Z
author_association: OWNER
body:
  Updated docs for `execute_write_fn()`: https://github.com/simonw/datasette/blob/75153ea9b94d09ec3d61f7c6ebdf378e0c0c7a0b/docs/internals.rst#await-dbexecute_write_fnfn-blocktrue

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1579#issuecomment-1000485719
issue_url: https://api.github.com/repos/simonw/datasette/issues/1579
node_id: IC_kwDOBm6k_c47ojNX
created_at: 2021-12-23T19:19:45Z
updated_at: 2021-12-23T19:19:45Z
author_association: OWNER
body:
  All of those removed `block=True` lines in 8c401ee0f054de2f568c3a8302c9223555146407 really help confirm to me that this was a good decision.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1534#issuecomment-1000535904
issue_url: https://api.github.com/repos/simonw/datasette/issues/1534
node_id: IC_kwDOBm6k_c47ovdg
created_at: 2021-12-23T21:44:31Z
updated_at: 2021-12-23T21:44:31Z
author_association: OWNER
body:
  A big downside to this is that I would need to use `Vary: Accept` for when Datasette is running behind a cache such as Cloudflare - would that greatly reduce overall cache efficiency due to subtle variations in the accept headers sent by common browsers?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1577#issuecomment-1000673444
issue_url: https://api.github.com/repos/simonw/datasette/issues/1577
node_id: IC_kwDOBm6k_c47pRCk
created_at: 2021-12-24T06:08:58Z
updated_at: 2021-12-24T06:08:58Z
author_association: OWNER
body:
  https://pypistats.org/packages/datasette shows a breakdown of downloads by Python version:

  <img width="986" alt="image" src="https://user-images.githubusercontent.com/9599/147323253-1ee22d93-3be2-472b-8ead-495d925958e5.png">

  It looks like on a recent day I had 4,071 downloads from Python 3.7... and just 2 downloads from Python 3.6!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1576#issuecomment-1000935523
issue_url: https://api.github.com/repos/simonw/datasette/issues/1576
node_id: IC_kwDOBm6k_c47qRBj
created_at: 2021-12-24T21:33:05Z
updated_at: 2021-12-24T21:33:05Z
author_association: OWNER
body:
  Another option would be to attempt to import `contextvars` and, if the import fails (for Python 3.6) continue using the current mechanism - then let Python 3.6 users know in the documentation that under Python 3.6 they will miss out on nested traces.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/228#issuecomment-1001115286
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/228
node_id: IC_kwDOCGYnMM47q86W
created_at: 2021-12-26T07:01:31Z
updated_at: 2021-12-26T07:01:31Z
author_association: NONE
body:
  `--no-headers` does not work?
  ```
  $ echo 'a,1\nb,2' | sqlite-utils memory --no-headers -t - 'select * from stdin'
  a      1                                                                                                                             
  ---  ---                                                                                                                             
  b      2 
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/62#issuecomment-1001222213
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/62
node_id: IC_kwDODEm0Qs47rXBF
created_at: 2021-12-26T17:59:25Z
updated_at: 2021-12-26T17:59:25Z
author_association: NONE
body:
  just confirmed that this error does not occur when i use my public main account. gets more interesting!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/878#issuecomment-1001699559
issue_url: https://api.github.com/repos/simonw/datasette/issues/878
node_id: IC_kwDOBm6k_c47tLjn
created_at: 2021-12-27T18:53:04Z
updated_at: 2021-12-27T18:53:04Z
author_association: OWNER
body:
  I'm going to see if I can come up with the simplest possible version of this pattern for the `/-/metadata` and `/-/metadata.json` page, then try it for the database query page, before tackling the much more complex table page.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1152#issuecomment-1001791592
issue_url: https://api.github.com/repos/simonw/datasette/issues/1152
node_id: IC_kwDOBm6k_c47tiBo
created_at: 2021-12-27T23:04:31Z
updated_at: 2021-12-27T23:04:31Z
author_association: OWNER
body:
  Another option: rethink permissions to always work in terms of where clauses users as part of a SQL query that returns the overall allowed set of databases or tables. This would require rethinking existing permissions but it might be worthwhile prior to 1.0.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/google-takeout-to-sqlite/pull/8#issuecomment-1002735370
issue_url: https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/8
node_id: IC_kwDODFE5qs47xIcK
created_at: 2021-12-29T18:58:23Z
updated_at: 2021-12-29T18:58:23Z
author_association: NONE
body:
  @maxhawkins how hard would it be to add an entry to the table that includes the HTML version of the email, if it exists?  I just attempted your the PR branch on a very small mbox file, and it worked great.  My use case is a research project and I need to access more than just the body plain text.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1583#issuecomment-1002825217
issue_url: https://api.github.com/repos/simonw/datasette/issues/1583
node_id: IC_kwDOBm6k_c47xeYB
created_at: 2021-12-30T00:34:16Z
updated_at: 2021-12-30T00:34:16Z
author_association: CONTRIBUTOR
body:
  if that is not desirable, it might be good to document that users might want to set up a lifecycle rule to automatically delete these build artifacts. something like https://stackoverflow.com/questions/59937542/can-i-delete-container-images-from-google-cloud-storage-artifacts-bucket

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/google-takeout-to-sqlite/pull/8#issuecomment-1003437288
issue_url: https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/8
node_id: IC_kwDODFE5qs47zzzo
created_at: 2021-12-31T19:06:20Z
updated_at: 2021-12-31T19:06:20Z
author_association: NONE
body:
  > @maxhawkins how hard would it be to add an entry to the table that includes the HTML version of the email, if it exists? I just attempted your the PR branch on a very small mbox file, and it worked great. My use case is a research project and I need to access more than just the body plain text.

  Shouldn't be hard. The easiest way is probably to remove the `if body.content_type == "text/html"` clause from [utils.py:254](https://github.com/dogsheep/google-takeout-to-sqlite/pull/8/commits/8e6d487b697ce2e8ad885acf613a157bfba84c59#diff-25ad9dd1ced1b8bfc37fda8444819c803232c08891e4af3d4064aa205d8174eaR254) and just return content directly without parsing.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1534#issuecomment-1005975080
issue_url: https://api.github.com/repos/simonw/datasette/issues/1534
node_id: IC_kwDOBm6k_c479fYo
created_at: 2022-01-05T18:29:06Z
updated_at: 2022-01-05T18:29:06Z
author_association: OWNER
body:
  A really big downside to this is that it turns out many CDNs - apparently including Cloudflare - don't support the Vary header at all!

  More in this thread: https://twitter.com/simonw/status/1478470282931163137

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/360#issuecomment-1006211113
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/360
node_id: IC_kwDOCGYnMM47-ZAp
created_at: 2022-01-06T01:27:53Z
updated_at: 2022-01-06T01:27:53Z
author_association: OWNER
body:
  It looks like you were using `sqlite-utils memory` - that works by loading the entire file into an in-memory database, so 170GB is very likely to run out of RAM.

  The line of code there exhibits another problem: it's reading the entire JSON file into a Python string, so it looks like it's going to run out of RAM even before it gets to the SQLite in-memory database section.

  To handle a file of this size you'd need to write it to a SQLite database on-disk first. The `sqlite-utils insert` command can do this, and it should be able to "stream" records in from a file without loading the entire thing into memory - but only for JSON-NL and CSV/TSV formats, not for JSON arrays.

  The code in question is here:

  https://github.com/simonw/sqlite-utils/blob/f3fd8613113d21d44238a6ec54b375f5aa72c4e0/sqlite_utils/cli.py#L738-L773

  That's using Python generators for the CSV/TSV/JSON-NL variants... but it's doing this for regular JSON which requires reading the entire thing into memory:

  https://github.com/simonw/sqlite-utils/blob/f3fd8613113d21d44238a6ec54b375f5aa72c4e0/sqlite_utils/cli.py#L767

  If you have the ability to control how your 170GB file is generated you may have more luck converting it to CSV or TSV or newline-delimited JSON, then using `sqlite-utils insert` to insert it into a database file.

  To be honest though I've never tested this tooling with anything nearly that big, so it's possible you'll still run into problems. If you do I'd love to hear about them!

  I would be tempted to tackle this size of job by writing a custom Python script, either using the `sqlite_utils` Python library or even calling `sqlite3` directly.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006219848
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/361
node_id: IC_kwDOCGYnMM47-bJI
created_at: 2022-01-06T01:51:36Z
updated_at: 2022-01-06T01:51:36Z
author_association: OWNER
body:
  So far I've just implemented the new help:
  ```
  % sqlite-utils insert --help
  Usage: sqlite-utils insert [OPTIONS] PATH TABLE FILE

    Insert records from FILE into a table, creating the table if it does not
    already exist.

    By default the input is expected to be a JSON array of objects. Or:

    - Use --nl for newline-delimited JSON objects
    - Use --csv or --tsv for comma-separated or tab-separated input
    - Use --lines to write each incoming line to a column called "line"
    - Use --all to write the entire input to a column called "all"

    You can also use --convert to pass a fragment of Python code that will be
    used to convert each input.

    Your Python code will be passed a "row" variable representing the imported
    row, and can return a modified row.

    If you are using --lines your code will be passed a "line" variable, and for
    --all an "all" variable.

  Options:
    --pk TEXT                 Columns to use as the primary key, e.g. id
    --flatten                 Flatten nested JSON objects, so {"a": {"b": 1}}
                              becomes {"a_b": 1}
    --nl                      Expect newline-delimited JSON
    -c, --csv                 Expect CSV input
    --tsv                     Expect TSV input
    --lines                   Treat each line as a single value called 'line'
    --all                     Treat input as a single value called 'all'
    --convert TEXT            Python code to convert each item
    --import TEXT             Python modules to import
    --delimiter TEXT          Delimiter to use for CSV files
    --quotechar TEXT          Quote character to use for CSV/TSV
    --sniff                   Detect delimiter and quote character
    --no-headers              CSV file has no header row
    --batch-size INTEGER      Commit every X records
    --alter                   Alter existing table to add any missing columns
    --not-null TEXT           Columns that should be created as NOT NULL
    --default <TEXT TEXT>...  Default value that should be set for a column
    --encoding TEXT           Character encoding for input, defaults to utf-8
    -d, --detect-types        Detect types for columns in CSV/TSV data
    --load-extension TEXT     SQLite extensions to load
    --silent                  Do not show progress bar
    --ignore                  Ignore records if pk already exists
    --replace                 Replace records if pk already exists
    --truncate                Truncate table before inserting records, if table
                              already exists
    -h, --help                Show this message and exit.
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006219956
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/361
node_id: IC_kwDOCGYnMM47-bK0
created_at: 2022-01-06T01:51:54Z
updated_at: 2022-01-06T06:22:25Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/361?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#361](https://codecov.io/gh/simonw/sqlite-utils/pull/361?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (b7f0b88) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/f3fd8613113d21d44238a6ec54b375f5aa72c4e0?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (f3fd861) will **decrease** coverage by `0.05%`.
  > The diff coverage is `92.85%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/361/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/361?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@            Coverage Diff             @@
  ##             main     #361      +/-   ##
  ==========================================
  - Coverage   96.49%   96.44%   -0.06%     
  ==========================================
    Files           5        5              
    Lines        2283     2306      +23     
  ==========================================
  + Hits         2203     2224      +21     
  - Misses         80       82       +2     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/361?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [sqlite\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/361/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.49% <92.00%> (-0.11%)` | :arrow_down: |
  | [sqlite\_utils/utils.py](https://codecov.io/gh/simonw/sqlite-utils/pull/361/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `94.23% <100.00%> (ø)` | |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/361?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/361?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [f3fd861...b7f0b88](https://codecov.io/gh/simonw/sqlite-utils/pull/361?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006220129
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/361
node_id: IC_kwDOCGYnMM47-bNh
created_at: 2022-01-06T01:52:26Z
updated_at: 2022-01-06T01:52:26Z
author_association: OWNER
body:
  I'm going to refactor all of the tests for `sqlite-utils insert` into a new `test_cli_insert.py` module.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006230411
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/361
node_id: IC_kwDOCGYnMM47-duL
created_at: 2022-01-06T02:17:35Z
updated_at: 2022-01-06T02:17:35Z
author_association: OWNER
body:
  Documentation: https://github.com/simonw/sqlite-utils/blob/33223856ff7fe746b7b77750fbe5b218531d0545/docs/cli.rst#inserting-unstructured-data-with---lines-and---all - I went with a single section titled "Inserting unstructured data with --lines and --all"

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006232013
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/361
node_id: IC_kwDOCGYnMM47-eHN
created_at: 2022-01-06T02:21:35Z
updated_at: 2022-01-06T02:21:35Z
author_association: OWNER
body:
  I'm having second thoughts about this bit:

  > Your Python code will be passed a "row" variable representing the imported row, and can return a modified row.
  >
  > If you are using `--lines` your code will be passed a "line" variable, and for `--all` an "all" variable.

  The code in question is this:

  https://github.com/simonw/sqlite-utils/blob/500a35ad4d91c8a6232134ce9406efec11bedff8/sqlite_utils/utils.py#L296-L303

  Do I really want to add the complexity of supporting different variable names there? I think always using `value` might be better.

  Except... `value` made sense for the existing `sqlite-utils convert` command where you are running a conversion function against the value for the column in the current row - is it confusing if applied to lines or documents or `all`?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006284673
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/361
node_id: IC_kwDOCGYnMM47-q-B
created_at: 2022-01-06T04:55:52Z
updated_at: 2022-01-06T04:55:52Z
author_association: OWNER
body:
  Test code that just worked for me:
  ```
  sqlite-utils insert /tmp/blah.db blah /tmp/log.log --convert '
  bits = line.split()
  return dict([("b_{}".format(i), bit) for i, bit in enumerate(bits)])' --lines
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006288444
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/361
node_id: IC_kwDOCGYnMM47-r48
created_at: 2022-01-06T05:07:10Z
updated_at: 2022-01-06T05:07:10Z
author_association: OWNER
body:
  And here's a demo of `--convert` used with `--all` - I added a custom error message for if the user's `--convert` code doesn't return a dict.

  ```
  % sqlite-utils insert /tmp/all.db blah /tmp/log.log --convert 'all.upper()' --all         
  Error: Records returned by your --convert function must be dicts
  % sqlite-utils insert /tmp/all.db blah /tmp/log.log --convert '{"all": all.upper()}' --all
  % sqlite-utils dump /tmp/all.db                                                           
  BEGIN TRANSACTION;
  CREATE TABLE [blah] (
     [all] TEXT
  );
  INSERT INTO "blah" VALUES('INFO:     127.0.0.1:60581 - "GET / HTTP/1.1" 200 OK
  INFO:     127.0.0.1:60581 - "GET /FOO/-/STATIC/APP.CSS?CEAD5A HTTP/1.1" 200 OK
  INFO:     127.0.0.1:60581 - "GET /FAVICON.ICO HTTP/1.1" 200 OK
  INFO:     127.0.0.1:60581 - "GET /FOO/TIDDLYWIKI HTTP/1.1" 200 OK
  INFO:     127.0.0.1:60581 - "GET /FOO/-/STATIC/APP.CSS?CEAD5A HTTP/1.1" 200 OK
  INFO:     127.0.0.1:60584 - "GET /FOO/-/STATIC/SQL-FORMATTER-2.3.3.MIN.JS HTTP/1.1" 200 OK
  INFO:     127.0.0.1:60586 - "GET /FOO/-/STATIC/CODEMIRROR-5.57.0.MIN.JS HTTP/1.1" 200 OK
  INFO:     127.0.0.1:60585 - "GET /FOO/-/STATIC/CODEMIRROR-5.57.0.MIN.CSS HTTP/1.1" 200 OK
  INFO:     127.0.0.1:60588 - "GET /FOO/-/STATIC/CODEMIRROR-5.57.0-SQL.MIN.JS HTTP/1.1" 200 OK
  INFO:     127.0.0.1:60587 - "GET /FOO/-/STATIC/CM-RESIZE-1.0.1.MIN.JS HTTP/1.1" 200 OK
  INFO:     127.0.0.1:60586 - "GET /FOO/TIDDLYWIKI/TIDDLERS HTTP/1.1" 200 OK
  INFO:     127.0.0.1:60586 - "GET /FOO/-/STATIC/APP.CSS?CEAD5A HTTP/1.1" 200 OK
  INFO:     127.0.0.1:60584 - "GET /FOO/-/STATIC/TABLE.JS HTTP/1.1" 200 OK
  ');
  COMMIT;
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006294777
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/361
node_id: IC_kwDOCGYnMM47-tb5
created_at: 2022-01-06T05:24:54Z
updated_at: 2022-01-06T05:24:54Z
author_association: OWNER
body:
  > I added a custom error message for if the user's `--convert` code doesn't return a dict.

  That turned out to be a bad idea because it meant exhausting the iterator early for the check - before we got to the `.insert_all()` code that breaks the iterator up into chunks. I tried fixing that with `itertools.tee()` to run the generator twice but that's grossly memory-inefficient for large imports.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006295276
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/361
node_id: IC_kwDOCGYnMM47-tjs
created_at: 2022-01-06T05:26:11Z
updated_at: 2022-01-06T05:26:11Z
author_association: OWNER
body:
  Here's the traceback if your `--convert` function doesn't return a dict right now:
  ```
  % sqlite-utils insert /tmp/all.db blah /tmp/log.log --convert 'all.upper()' --all         

  Traceback (most recent call last):
    File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/bin/sqlite-utils", line 33, in <module>
      sys.exit(load_entry_point('sqlite-utils', 'console_scripts', 'sqlite-utils')())
    File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py", line 1137, in __call__
      return self.main(*args, **kwargs)
    File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py", line 1062, in main
      rv = self.invoke(ctx)
    File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py", line 1668, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py", line 763, in invoke
      return __callback(*args, **kwargs)
    File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py", line 949, in insert
      insert_upsert_implementation(
    File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py", line 834, in insert_upsert_implementation
      db[table].insert_all(
    File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py", line 2602, in insert_all
      first_record = next(records)
    File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py", line 3044, in fix_square_braces
      for record in records:
    File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py", line 831, in <genexpr>
      docs = (decode_base64_values(doc) for doc in docs)
    File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py", line 86, in decode_base64_values
      to_fix = [
    File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py", line 89, in <listcomp>
      if isinstance(doc[k], dict)
  TypeError: string indices must be integers
  ```
  I can live with that for the moment.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006300280
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/361
node_id: IC_kwDOCGYnMM47-ux4
created_at: 2022-01-06T05:40:45Z
updated_at: 2022-01-06T05:40:45Z
author_association: OWNER
body:
  I'm going to rename `--all` to `--text`:

  > - Use `--text` to write the entire input to a column called "text"

  To avoid that clash with Python's `all()` function.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006309834
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/361
node_id: IC_kwDOCGYnMM47-xHK
created_at: 2022-01-06T06:08:01Z
updated_at: 2022-01-06T06:08:01Z
author_association: OWNER
body:
  For `--text` the conversion function should be allowed to return an iterable instead of a dictionary, in which case it will be treated as the full list of records to be inserted.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006311742
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/361
node_id: IC_kwDOCGYnMM47-xk-
created_at: 2022-01-06T06:12:19Z
updated_at: 2022-01-06T06:12:19Z
author_association: OWNER
body:
  Got that working:
  ```
  % echo 'This is cool' | sqlite-utils insert words.db words - --text --convert '({"word": w} for w in text.split())'
  % sqlite-utils dump words.db                                                                                       
  BEGIN TRANSACTION;
  CREATE TABLE [words] (
     [word] TEXT
  );
  INSERT INTO "words" VALUES('This');
  INSERT INTO "words" VALUES('is');
  INSERT INTO "words" VALUES('cool');
  COMMIT;
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/356#issuecomment-1006318443
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/356
node_id: IC_kwDOCGYnMM47-zNr
created_at: 2022-01-06T06:30:13Z
updated_at: 2022-01-06T06:30:13Z
author_association: OWNER
body:
  Documentation:

  - https://sqlite-utils.datasette.io/en/latest/cli.html#inserting-unstructured-data-with-lines-and-text
  - https://sqlite-utils.datasette.io/en/latest/cli.html#applying-conversions-while-inserting-data

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/363#issuecomment-1006343303
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/363
node_id: IC_kwDOCGYnMM47-5SH
created_at: 2022-01-06T07:30:20Z
updated_at: 2022-01-06T07:30:20Z
author_association: OWNER
body:
  This check should run inside the `.insert_all()` method. It should raise a custom exception which the CLI code can then catch and turn into a click error.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/363#issuecomment-1006344080
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/363
node_id: IC_kwDOCGYnMM47-5eQ
created_at: 2022-01-06T07:32:05Z
updated_at: 2022-01-06T07:32:05Z
author_association: OWNER
body:
  As part of this work I should add test coverage of this error message too: https://github.com/simonw/sqlite-utils/blob/413f8ed754e38d7b190de888c85fe8438336cb11/sqlite_utils/cli.py#L826

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1007634999
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/365
node_id: IC_kwDOCGYnMM48D0o3
created_at: 2022-01-07T18:26:22Z
updated_at: 2022-01-07T18:26:22Z
author_association: OWNER
body:
  I've not used the `ANALYZE` feature in SQLite at all before. Should probably add Python library methods for it.

  Annoyingly I use the word "analyze" to mean something else in the CLI - for these features:
  - #207 
  - #320

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1007636709
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/365
node_id: IC_kwDOCGYnMM48D1Dl
created_at: 2022-01-07T18:28:33Z
updated_at: 2022-01-07T18:29:43Z
author_association: CONTRIBUTOR
body:
  i added an index to one table with sqlite-utils, and then a query that used to take about 1 second started taking hundreds of seconds. 

  running analyze got me back to sub second speed.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1007637963
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/366
node_id: IC_kwDOCGYnMM48D1XL
created_at: 2022-01-07T18:30:13Z
updated_at: 2022-01-07T18:30:13Z
author_association: OWNER
body:
  Annoyingly I use the word "analyze" to mean something else in the CLI - for these features:

  - #207 
  - #320

  there's only one method with a similar name in the Python library though and that's this one:

  https://github.com/simonw/sqlite-utils/blob/6e46b9913411682f3a3ec66f4d58886c1db8654b/sqlite_utils/db.py#L2904-L2906

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1007639860
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/366
node_id: IC_kwDOCGYnMM48D100
created_at: 2022-01-07T18:32:59Z
updated_at: 2022-01-07T18:33:07Z
author_association: OWNER
body:
  From the SQLite docs:

  > If no arguments are given, all attached databases are analyzed. If a schema name is given as the argument, then all tables and indices in that one database are analyzed. If the argument is a table name, then only that table and the indices associated with that table are analyzed. If the argument is an index name, then only that one index is analyzed.

  So I think this becomes two methods:

  - `db.analyze()` calls analyze on the whole database
  - `db.analyze(name_of_table_or_index)` for a specific named table or index
  - `table.analyze()` is a shortcut for `db.analyze(table.name)`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1007641634
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/366
node_id: IC_kwDOCGYnMM48D2Qi
created_at: 2022-01-07T18:35:35Z
updated_at: 2022-01-07T18:35:35Z
author_association: OWNER
body:
  Since the existing CLI feature is this:

      $ sqlite-utils analyze-tables github.db tags

  I can add `sqlite-utils analyze` to reflect the Python library method.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1007642831
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/365
node_id: IC_kwDOCGYnMM48D2jP
created_at: 2022-01-07T18:37:18Z
updated_at: 2022-01-07T18:37:18Z
author_association: OWNER
body:
  After implementing #366 I can make it so `sqlite-utils create-index` automatically runs `db.analyze(index_name)` afterwards, maybe with a `--no-analyze` option in case anyone wants to opt out of that for specific performance reasons.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1007643254
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/365
node_id: IC_kwDOCGYnMM48D2p2
created_at: 2022-01-07T18:37:56Z
updated_at: 2022-01-07T18:37:56Z
author_association: OWNER
body:
  Or I could leave off `--no-analyze` and tell people that if they want to add an index without running analyze they can execute the `CREATE INDEX` themselves.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008129841
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/364
node_id: IC_kwDOCGYnMM48Ftcx
created_at: 2022-01-08T20:04:42Z
updated_at: 2022-01-08T20:04:42Z
author_association: OWNER
body:
  It would be easier to test this if I had a utility for streaming out a file one line at a time.

  A few recipes for this in https://superuser.com/questions/526242/cat-file-to-terminal-at-particular-speed-of-lines-per-second - I'm going to build a quick `stream-delay` tool though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008153586
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/364
node_id: IC_kwDOCGYnMM48FzPy
created_at: 2022-01-08T21:06:15Z
updated_at: 2022-01-08T21:06:15Z
author_association: OWNER
body:
  I added a print statement after `for query, params in queries_and_params` and confirmed that something in the code is waiting until 16 records are available to be inserted and then executing the inserts, even with `--batch-size 1`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008154873
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/364
node_id: IC_kwDOCGYnMM48Fzj5
created_at: 2022-01-08T21:11:55Z
updated_at: 2022-01-08T21:11:55Z
author_association: OWNER
body:
  I'm suspicious that the `chunks()` utility function may not be working correctly:
  ```pycon
  In [10]: [list(d) for d in list(chunks('abc', 5))]
  Out[10]: [['a'], ['b'], ['c']]

  In [11]: [list(d) for d in list(chunks('abcdefghi', 5))]
  Out[11]: [['a'], ['b'], ['c'], ['d'], ['e'], ['f'], ['g'], ['h'], ['i']]

  In [12]: [list(d) for d in list(chunks('abcdefghi', 3))]
  Out[12]: [['a'], ['b'], ['c'], ['d'], ['e'], ['f'], ['g'], ['h'], ['i']]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1008157132
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/366
node_id: IC_kwDOCGYnMM48F0HM
created_at: 2022-01-08T21:23:08Z
updated_at: 2022-01-08T21:25:05Z
author_association: OWNER
body:
  Running `ANALYZE` creates a new visible table called `sqlite_stat1`: https://www.sqlite.org/fileformat.html#the_sqlite_stat1_table

  This should be added to the default list of hidden tables in Datasette.

  It looks something like this:

  | tbl                             | idx                                | stat      |
  |---------------------------------|------------------------------------|-----------|
  | _counts                         | sqlite_autoindex__counts_1         | 5 1       |
  | global-power-plants_fts_config  | global-power-plants_fts_config     | 1 1       |
  | global-power-plants_fts_docsize |                                    | 33643     |
  | global-power-plants_fts_idx     | global-power-plants_fts_idx        | 199 40 1  |
  | global-power-plants_fts_data    |                                    | 136       |
  | global-power-plants             | "global-power-plants_owner"        | 33643 4   |
  | global-power-plants             | "global-power-plants_country_long" | 33643 202 |

  >  In each such row, the sqlite_stat.stat column will be a string consisting of a list of integers followed by zero or more arguments. The first integer in this list is the approximate number of rows in the index. (The number of rows in the index is the same as the number of rows in the table, except for partial indexes.) The second integer is the approximate number of rows in the index that have the same value in the first column of the index. The third integer is the number number of rows in the index that have the same value for the first two columns. The N-th integer (for N>1) is the estimated average number of rows in the index which have the same value for the first N-1 columns. For a K-column index, there will be K+1 integers in the stat column. If the index is unique, then the last integer will be 1. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1587#issuecomment-1008157908
issue_url: https://api.github.com/repos/simonw/datasette/issues/1587
node_id: IC_kwDOBm6k_c48F0TU
created_at: 2022-01-08T21:29:06Z
updated_at: 2022-01-08T21:29:06Z
author_association: OWNER
body:
  Depending on the SQLite version (and compile options) that ran `ANALYZE` these can be called:

  - `sqlite_stat1`
  - `sqlite_stat2`
  - `sqlite_stat3`
  - `sqlite_stat4`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1587#issuecomment-1008157998
issue_url: https://api.github.com/repos/simonw/datasette/issues/1587
node_id: IC_kwDOBm6k_c48F0Uu
created_at: 2022-01-08T21:29:54Z
updated_at: 2022-01-08T21:29:54Z
author_association: OWNER
body:
  Relevant code: https://github.com/simonw/datasette/blob/00a2895cd2dc42c63846216b36b2dc9f41170129/datasette/database.py#L339-L354

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008158357
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/365
node_id: IC_kwDOCGYnMM48F0aV
created_at: 2022-01-08T21:33:07Z
updated_at: 2022-01-08T21:33:07Z
author_association: OWNER
body:
  The one thing that worries me a little bit about doing this by default is that it adds a surprising new table to the database - it may be confusing to users if they run `create-index` and their database suddenly has a new `sqlite_stat1` table, see https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1008157132

  Options here are:

  - Do it anyway. People can tolerate a surprise table appearing when they create an index.
  - Only run `ANALYZE` if the user says `sqlite-utils create-index ... --analyze`
  - Use the `--analyze` option, but also automatically run `ANALYZE` if they create an index and the database they are working with already has a `sqlite_stat1` table

  I'm currently leading towards that third option - @fgregg any thoughts?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/367#issuecomment-1008158799
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/367
node_id: IC_kwDOCGYnMM48F0hP
created_at: 2022-01-08T21:36:55Z
updated_at: 2022-01-09T02:34:44Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/367?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#367](https://codecov.io/gh/simonw/sqlite-utils/pull/367?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (9848eaa) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/a8f9cc6f64f299830834428509940d448b82b4ed?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (a8f9cc6) will **decrease** coverage by `0.20%`.
  > The diff coverage is `50.00%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/367/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/367?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@            Coverage Diff             @@
  ##             main     #367      +/-   ##
  ==========================================
  - Coverage   96.44%   96.24%   -0.21%     
  ==========================================
    Files           5        6       +1     
    Lines        2307     2317      +10     
  ==========================================
  + Hits         2225     2230       +5     
  - Misses         82       87       +5     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/367?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/367/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.15% <28.57%> (-0.42%)` | :arrow_down: |
  | [sqlite\_utils/\_\_main\_\_.py](https://codecov.io/gh/simonw/sqlite-utils/pull/367/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL19fbWFpbl9fLnB5) | `100.00% <100.00%> (ø)` | |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/367?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/367?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [a8f9cc6...9848eaa](https://codecov.io/gh/simonw/sqlite-utils/pull/367?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008161965
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/365
node_id: IC_kwDOCGYnMM48F1St
created_at: 2022-01-08T22:02:56Z
updated_at: 2022-01-08T22:02:56Z
author_association: CONTRIBUTOR
body:
  for options 2 and 3, i would worry about discoverablity. 

  in other db’s it is not necessary to explicitly call analyze for most indices. ie for postgres

  > The system regularly collects statistics on all of a table's columns. Newly-created non-expression indexes can immediately use these statistics to determine an index's usefulness.

  i suppose i would propose raising a warning if the stats table is created that explains what is going on and informs users about a —no-analyze argument.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008163050
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/365
node_id: IC_kwDOCGYnMM48F1jq
created_at: 2022-01-08T22:10:51Z
updated_at: 2022-01-08T22:10:51Z
author_association: OWNER
body:
  Is there a downside to having a `sqlite_stat1` table if it has wildly incorrect statistics in it?

  Imagine the following sequence of events:

  - User imports a few records, creating the table, using `sqlite-utils insert`
  - User runs `sqlite-utils create-index ...` which also creates and populates the `sqlite_stat1` table
  - User runs `insert` again to populate several million new records

  The user now has a database file with several million records and a statistics table that is wildly out of date, having been populated when they only had a few.

  Will this result in surprisingly bad query performance compared to it that statistics table did not exist at all?

  If so, I lean much harder towards `ANALYZE` as a strictly opt-in optimization, maybe with the `--analyze` option added to `sqlite-utils insert` top to help users opt in to updating their statistics after running big inserts.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008163585
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/365
node_id: IC_kwDOCGYnMM48F1sB
created_at: 2022-01-08T22:14:39Z
updated_at: 2022-01-09T03:03:07Z
author_association: OWNER
body:
  The reason I'm hesitating on this is that I've not actually used ANALYZE at all in nearly five years of messing around with SQLite! So I'm nervous that there are surprise downsides I haven't thought of.

  My hunch is that ANALYZE is only worth worrying about on much larger databases, in which case I'm OK supporting it as a thoroughly documented power-user feature rather than a default.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008164116
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/365
node_id: IC_kwDOCGYnMM48F10U
created_at: 2022-01-08T22:18:57Z
updated_at: 2022-01-08T22:18:57Z
author_association: CONTRIBUTOR
body:
  the table with the query ran so bad was about 50k. 

  i think the scenario should not be worse than no stats. 

  i also did not know that sqlite was so different from postgres and needed an explicit analyze call.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008164786
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/365
node_id: IC_kwDOCGYnMM48F1-y
created_at: 2022-01-08T22:24:19Z
updated_at: 2022-01-08T22:24:19Z
author_association: CONTRIBUTOR
body:
  the out-of-date scenario you describe could be addressed by automatically adding an analyze to the insert or convert commands if they implicate an index

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008214406
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/364
node_id: IC_kwDOCGYnMM48GCGG
created_at: 2022-01-09T02:18:21Z
updated_at: 2022-01-09T02:18:21Z
author_association: OWNER
body:
  I'm having trouble figuring out the best way to write a unit test for this. Filed a relevant feature request for Click here:
  - https://github.com/pallets/click/issues/2171

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008214998
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/364
node_id: IC_kwDOCGYnMM48GCPW
created_at: 2022-01-09T02:23:20Z
updated_at: 2022-01-09T02:23:20Z
author_association: OWNER
body:
  Possible way of running the test: add this to `sqlite_utils/cli.py`:

  ```python
  if __name__ == "__main__":
      cli()
  ```
  Now the tool can be run using `python -m sqlite_utils.cli --help`

  Then in the test use `subprocess` to call `sys.executable` (the path to the current Python interpreter) and pass it `-m sqlite_utils.cli` to run the script!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/368#issuecomment-1008215912
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/368
node_id: IC_kwDOCGYnMM48GCdo
created_at: 2022-01-09T02:30:59Z
updated_at: 2022-01-09T02:30:59Z
author_association: OWNER
body:
  Even better, inspired by `rich`, support `python -m sqlite_utils`. https://github.com/Textualize/rich/blob/master/rich/__main__.py

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008216201
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/364
node_id: IC_kwDOCGYnMM48GCiJ
created_at: 2022-01-09T02:34:12Z
updated_at: 2022-01-09T02:34:12Z
author_association: OWNER
body:
  I can now write tests that look like this: https://github.com/simonw/sqlite-utils/blob/539f5ccd90371fa87f946018f8b77d55929e06db/tests/test_cli.py#L2024-L2030

  Which means I can write a test that exercises this bug.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/368#issuecomment-1008216271
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/368
node_id: IC_kwDOCGYnMM48GCjP
created_at: 2022-01-09T02:35:09Z
updated_at: 2022-01-09T02:35:09Z
author_association: OWNER
body:
  Test failure on Python 3.6:

  >  `E       TypeError: __init__() got an unexpected keyword argument 'capture_output'`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/368#issuecomment-1008216371
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/368
node_id: IC_kwDOCGYnMM48GCkz
created_at: 2022-01-09T02:36:22Z
updated_at: 2022-01-09T02:36:22Z
author_association: OWNER
body:
  In Python 3.6: https://docs.python.org/3.6/library/subprocess.html

  > This does not capture stdout or stderr by default. To do so, pass [`PIPE`](https://docs.python.org/3.6/library/subprocess.html#subprocess.PIPE "subprocess.PIPE") for the *stdout* and/or *stderr* arguments.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008219484
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/369
node_id: IC_kwDOCGYnMM48GDVc
created_at: 2022-01-09T03:05:44Z
updated_at: 2022-01-09T03:05:44Z
author_association: OWNER
body:
  I'll start by running some experiments against the 11MB database file from https://global-power-plants.datasettes.com/global-power-plants.db

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008219588
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/369
node_id: IC_kwDOCGYnMM48GDXE
created_at: 2022-01-09T03:06:42Z
updated_at: 2022-01-09T03:06:42Z
author_association: OWNER
body:
  ```
  analyze % sqlite-utils indexes global-power-plants.db -t           
  table                           index_name                                           seqno    cid  name            desc  coll      key
  ------------------------------  -------------------------------------------------  -------  -----  ------------  ------  ------  -----
  global-power-plants             "global-power-plants_owner"                              0     12  owner              0  BINARY      1
  global-power-plants             "global-power-plants_country_long"                       0      1  country_long       0  BINARY      1
  global-power-plants_fts_idx     sqlite_autoindex_global-power-plants_fts_idx_1           0      0  segid              0  BINARY      1
  global-power-plants_fts_idx     sqlite_autoindex_global-power-plants_fts_idx_1           1      1  term               0  BINARY      1
  global-power-plants_fts_config  sqlite_autoindex_global-power-plants_fts_config_1        0      0  k                  0  BINARY      1
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008219844
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/369
node_id: IC_kwDOCGYnMM48GDbE
created_at: 2022-01-09T03:08:09Z
updated_at: 2022-01-09T03:08:09Z
author_association: OWNER
body:
  ```
  analyze % sqlite-utils global-power-plants-analyzed.db 'analyze'
  [{"rows_affected": -1}]
  analyze % sqlite-utils tables global-power-plants-analyzed.db 
  [{"table": "global-power-plants"},
   {"table": "global-power-plants_fts"},
   {"table": "global-power-plants_fts_data"},
   {"table": "global-power-plants_fts_idx"},
   {"table": "global-power-plants_fts_docsize"},
   {"table": "global-power-plants_fts_config"},
   {"table": "sqlite_stat1"}]
  analyze % sqlite-utils rows global-power-plants-analyzed.db sqlite_stat1 -t
  tbl                              idx                                 stat
  -------------------------------  ----------------------------------  ---------
  global-power-plants_fts_config   global-power-plants_fts_config      1 1
  global-power-plants_fts_docsize                                      33643
  global-power-plants_fts_idx      global-power-plants_fts_idx         199 40 1
  global-power-plants_fts_data                                         136
  global-power-plants              "global-power-plants_owner"         33643 4
  global-power-plants              "global-power-plants_country_long"  33643 202
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008220270
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/369
node_id: IC_kwDOCGYnMM48GDhu
created_at: 2022-01-09T03:12:38Z
updated_at: 2022-01-09T03:13:15Z
author_association: OWNER
body:
  Basically no difference using this very basic benchmark:
  ```
  analyze % python3 -m timeit '__import__("sqlite3").connect("global-power-plants.db").execute("select country_long, count(*) from [global-power-plants] group by country_long").fetchall()'
  100 loops, best of 5: 2.39 msec per loop
  analyze % python3 -m timeit '__import__("sqlite3").connect("global-power-plants-analyzed.db").execute("select country_long, count(*) from [global-power-plants] group by country_long").fetchall()'
  100 loops, best of 5: 2.38 msec per loop
  ```
  I should try this against a much larger database.

  https://covid-19.datasettes.com/covid.db is 879MB.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008226487
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/369
node_id: IC_kwDOCGYnMM48GFC3
created_at: 2022-01-09T04:14:05Z
updated_at: 2022-01-09T04:14:05Z
author_association: OWNER
body:
  Didn't manage to spot a meaningful difference with that database either:
  ```
  analyze % python3 -m timeit '__import__("sqlite3").connect("covid.db").execute("select fips, count(*) from [ny_times_us_counties] group by fips").fetchall()'         
  2 loops, best of 5: 101 msec per loop
  analyze % python3 -m timeit '__import__("sqlite3").connect("covid-analyzed.db").execute("select fips, count(*) from [ny_times_us_counties] group by fips").fetchall()'
  2 loops, best of 5: 103 msec per loop
  ```
  Maybe `select fips, count(*) from [ny_times_us_counties] group by fips` isn't a good query for testing this?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008226862
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/369
node_id: IC_kwDOCGYnMM48GFIu
created_at: 2022-01-09T04:17:55Z
updated_at: 2022-01-09T04:17:55Z
author_association: OWNER
body:
  There are some clues as to what effect ANALYZE has in https://www.sqlite.org/optoverview.html

  Some quotes:

  >  SQLite might use a skip-scan on an index if it knows that the first one or more columns contain many duplication values. If there are too few duplicates in the left-most columns of the index, then it would be faster to simply step ahead to the next value, and thus do a full table scan, than to do a binary search on an index to locate the next left-column value.
  >
  > The only way that SQLite can know that there are many duplicates in the left-most columns of an index is if the ANALYZE command has been run on the database. Without the results of ANALYZE, SQLite has to guess at the "shape" of the data in the table, and the default guess is that there are an average of 10 duplicates for every value in the left-most column of the index. Skip-scan only becomes profitable (it only gets to be faster than a full table scan) when the number of duplicates is about 18 or more. Hence, a skip-scan is never used on a database that has not been analyzed. 

  And

  > Join reordering is automatic and usually works well enough that programmers do not have to think about it, especially if ANALYZE has been used to gather statistics about the available indexes, though occasionally some hints from the programmer are needed.

  And

  >  The various sqlite_statN tables contain information on how selective the various indexes are. For example, the sqlite_stat1 table might indicate that an equality constraint on column x reduces the search space to 10 rows on average, whereas an equality constraint on column y reduces the search space to 3 rows on average. In that case, SQLite would prefer to use index ex2i2 since that index is more selective. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1588#issuecomment-1008227436
issue_url: https://api.github.com/repos/simonw/datasette/issues/1588
node_id: IC_kwDOBm6k_c48GFRs
created_at: 2022-01-09T04:23:37Z
updated_at: 2022-01-09T04:25:04Z
author_association: OWNER
body:
  Relevant code: https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/utils/__init__.py#L163-L170

  https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/utils/__init__.py#L195-L204

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008227625
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/369
node_id: IC_kwDOCGYnMM48GFUp
created_at: 2022-01-09T04:25:38Z
updated_at: 2022-01-09T04:25:38Z
author_association: OWNER
body:
  ```sql
  EXPLAIN QUERY PLAN select country_long, count(*) from [global-power-plants] group by country_long
  ```
  https://global-power-plants.datasettes.com/global-power-plants?sql=EXPLAIN+QUERY+PLAN+select+country_long%2C+count%28*%29+from+%5Bglobal-power-plants%5D+group+by+country_long

  > SCAN TABLE global-power-plants USING COVERING INDEX "global-power-plants_country_long"

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008229341
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/369
node_id: IC_kwDOCGYnMM48GFvd
created_at: 2022-01-09T04:45:38Z
updated_at: 2022-01-09T04:47:11Z
author_association: OWNER
body:
  This is probably too fancy. I think maybe the way to do this is with `select * from [global-power-plants] where "country_long" = 'United Kingdom'` - then mess around with stats to see if I can get it to use the index or not based on them.

  Here's the explain for that: https://global-power-plants.datasettes.com/global-power-plants?sql=EXPLAIN+QUERY+PLAN+select+*+from+[global-power-plants]+where+%22country_long%22+%3D+%27United+Kingdom%27

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008229839
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/365
node_id: IC_kwDOCGYnMM48GF3P
created_at: 2022-01-09T04:51:44Z
updated_at: 2022-01-09T04:51:44Z
author_association: OWNER
body:
  Found one report on Stack Overflow from 9 years ago of someone seeing broken performance after running `ANALYZE`, hard to say that's a trend and not a single weird edge-case though! https://stackoverflow.com/q/12947214/6083

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008232075
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/369
node_id: IC_kwDOCGYnMM48GGaL
created_at: 2022-01-09T05:13:15Z
updated_at: 2022-01-09T05:13:56Z
author_association: OWNER
body:
  I think the query that will help solve this is:

  `explain query plan select * from ny_times_us_counties where state = 1 and county = 2`

  In this case, the query planner needs to decide if it should use the index for the `state` column or the index for the `county` column. That's where the statistics come into play. In particular:

  | tbl                  | idx                             | stat          |
  |----------------------|---------------------------------|---------------|
  | ny_times_us_counties | idx_ny_times_us_counties_date   | 2092871 2915  |
  | ny_times_us_counties | idx_ny_times_us_counties_fips   | 2092871 651   |
  | ny_times_us_counties | idx_ny_times_us_counties_county | 2092871 1085  |
  | ny_times_us_counties | idx_ny_times_us_counties_state  | 2092871 37373 |

  Those numbers are explained by this comment in the SQLite C code: https://github.com/sqlite/sqlite/blob/5622c7f97106314719740098cf0854e7eaa81802/src/analyze.c#L41-L55

  ```
  ** There is normally one row per index, with the index identified by the
  ** name in the idx column.  The tbl column is the name of the table to
  ** which the index belongs.  In each such row, the stat column will be
  ** a string consisting of a list of integers.  The first integer in this
  ** list is the number of rows in the index.  (This is the same as the
  ** number of rows in the table, except for partial indices.)  The second
  ** integer is the average number of rows in the index that have the same
  ** value in the first column of the index.
  ```
  So that table is telling us that using a value in the `county` column will filter down to an average of 1,085 rows, whereas filtering on the `state` column will filter down to an average of 37,373 - so clearly the `county` index is the better index to use here!

  Just one catch: against both my` covid.db` and my `covid-analyzed.db` databases the `county` index is picked for both of them - so SQLite is somehow guessing that `county` is a better index even though it doesn't have statistics for that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008233910
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/364
node_id: IC_kwDOCGYnMM48GG22
created_at: 2022-01-09T05:32:53Z
updated_at: 2022-01-09T05:35:45Z
author_association: OWNER
body:
  This is strange. The following:
  ```pycon
  >>> import subprocess
  >>> p = subprocess.Popen(["sqlite-utils", "insert", "/tmp/stream.db", "stream", "-", "--nl"], stdin=subprocess.PIPE)
  >>> p.stdin.write(b'\n'.join(b'{"id": %s}' % str(i).encode("utf-8") for i in range(1000)))
  11889
  >>> # At this point /tmp/stream.db is still 0 bytes - but if I then run this:
  >>> p.stdin.close()
  >>> # /tmp/stream.db is now 20K and contains the written data
  ```
  No wait, mystery solved - I can add `p.stdin.flush()` instead of `p.stdin.close()` and the file suddenly jumps up in size.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008246239
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/371
node_id: IC_kwDOCGYnMM48GJ3f
created_at: 2022-01-09T07:41:24Z
updated_at: 2022-01-09T07:41:24Z
author_association: OWNER
body:
  Might be a case of modifying this line: https://github.com/simonw/sqlite-utils/blob/e0c476bc380744680c8b7675c24fb0e9f5ec6dcd/sqlite_utils/cli.py#L828

  To:
  ```python
  docs = (fn(doc) or doc for doc in docs)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/372#issuecomment-1008247370
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/372
node_id: IC_kwDOCGYnMM48GKJK
created_at: 2022-01-09T07:51:18Z
updated_at: 2022-01-09T07:51:18Z
author_association: OWNER
body:
  Pathlib says the stem of that would be `dogs.and.cats.jpg` - best stick with that for consistency. https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.suffix

  It calls the last bit `suffix` - maybe I should use that instead of `ext`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008275546
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/365
node_id: IC_kwDOCGYnMM48GRBa
created_at: 2022-01-09T11:01:15Z
updated_at: 2022-01-09T13:37:51Z
author_association: CONTRIBUTOR
body:
  i don’t want to be such a partisan for analyze, but the query planner deciding *not* to use an index based on information collected by analyze is not necessarily a bug, but could be the correct choice.

  <s>the original poster in that stack overflow doesn’t say there’s a performance regression </s>

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1574#issuecomment-1008279307
issue_url: https://api.github.com/repos/simonw/datasette/issues/1574
node_id: IC_kwDOBm6k_c48GR8L
created_at: 2022-01-09T11:26:06Z
updated_at: 2022-01-09T11:26:06Z
author_association: NONE
body:
  @fgregg my thinking was backwards compatibility. I don't know what people do to their builds, I just wanted a smaller image for my use case.

  @simonw any chance to take a look at this? If there is no interest, feel free to close the PR

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/373#issuecomment-1008341078
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/373
node_id: IC_kwDOCGYnMM48GhBW
created_at: 2022-01-09T17:31:12Z
updated_at: 2022-01-09T17:31:12Z
author_association: OWNER
body:
  Found an example of using `cog` in a rST file here: https://github.com/nedbat/coveragepy/blob/f3238eea7e403d13a217b30579b1a1c2cbff62e3/doc/dbschema.rst#L21
  ```
  .. [[[cog
      from coverage.sqldata import SCHEMA_VERSION
      print(".. code::")
      print()
      print(f"    SCHEMA_VERSION = {SCHEMA_VERSION}")
      print()
  .. ]]]
  .. code::

      SCHEMA_VERSION = 7

  .. [[[end]]]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/373#issuecomment-1008344980
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/373
node_id: IC_kwDOCGYnMM48Gh-U
created_at: 2022-01-09T17:54:53Z
updated_at: 2022-01-09T17:54:53Z
author_association: OWNER
body:
  Updated TIL: https://til.simonwillison.net/python/cog-to-update-help-in-readme#user-content-cog-for-restructuredtext

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/374#issuecomment-1008346338
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/374
node_id: IC_kwDOCGYnMM48GiTi
created_at: 2022-01-09T18:03:22Z
updated_at: 2022-01-09T18:03:22Z
author_association: OWNER
body:
  Commands that support `--fmt` (via the `@output_options` decorator) are:

  - `tables`
  - `views`
  - `query`
  - `memory`
  - `search`
  - `rows`
  - `triggers`
  - `indexes`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/374#issuecomment-1008346841
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/374
node_id: IC_kwDOCGYnMM48GibZ
created_at: 2022-01-09T18:06:50Z
updated_at: 2022-01-09T18:06:50Z
author_association: OWNER
body:
  In addition to a unit test I manually tested all of the above, e.g.
  ```
  % sqlite-utils indexes global-power-plants.db sqlite_master --fmt rst
  =======  ============  =======  =====  ======  ======  ======  =====
  table    index_name    seqno    cid    name    desc    coll    key
  =======  ============  =======  =====  ======  ======  ======  =====
  =======  ============  =======  =====  ======  ======  ======  =====
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008347768
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/371
node_id: IC_kwDOCGYnMM48Gip4
created_at: 2022-01-09T18:12:30Z
updated_at: 2022-01-09T18:12:30Z
author_association: OWNER
body:
  Tried this test:

  ```python
      result = CliRunner().invoke(
          cli.cli,
          [
              "insert",
              db_path,
              "rows",
              "-",
              "--convert",
              'row["is_chicken"] = True',
          ],
          input='{"name": "Azi"}',
      )
  ```
  And got this error:

  > `E        +  where 1 = <Result SyntaxError('invalid syntax', ('<string>', 2, 30, '    return row["is_chicken"] = True\n'))>.exit_code`

  The code snippet compilation isn't currently compatible with this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008348032
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/371
node_id: IC_kwDOCGYnMM48GiuA
created_at: 2022-01-09T18:14:02Z
updated_at: 2022-01-09T18:14:02Z
author_association: OWNER
body:
  Here's the code in question: https://github.com/simonw/sqlite-utils/blob/b8c134059e89f0fa040b84fb7d0bda25b9a52759/sqlite_utils/utils.py#L288-L299

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008354207
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/371
node_id: IC_kwDOCGYnMM48GkOf
created_at: 2022-01-09T18:54:54Z
updated_at: 2022-01-09T18:54:54Z
author_association: OWNER
body:
  This seems to work:
  ```python
  def _compile_code(code, imports, variable="value"):
      locals = {}
      globals = {"r": recipes, "recipes": recipes}
      # If user defined a convert() function, return that
      try:
          exec(code, globals, locals)
          return locals["convert"]
      except (AttributeError, SyntaxError, NameError, KeyError, TypeError):
          pass

      # Try compiling their code as a function instead
      body_variants = [code]
      # If single line and no 'return', try adding the return
      if "\n" not in code and not code.strip().startswith("return "):
          body_variants.insert(0, "return {}".format(code))

      for variant in body_variants:
          new_code = ["def fn({}):".format(variable)]
          for line in variant.split("\n"):
              new_code.append("    {}".format(line))
          try:
              code_o = compile("\n".join(new_code), "<string>", "exec")
              break
          except SyntaxError:
              # Try another variant, e.g. for 'return row["column"] = 1'
              continue

      for import_ in imports:
          globals[import_.split(".")[0]] = __import__(import_)
      exec(code_o, globals, locals)
      return locals["fn"]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008364701
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/371
node_id: IC_kwDOCGYnMM48Gmyd
created_at: 2022-01-09T20:04:35Z
updated_at: 2022-01-09T20:04:35Z
author_association: OWNER
body:
  The previous code for highlighting errors in syntax (which was already a bit confused thanks to the added `return`, see https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991393684 - isn't compatible with this approach at all. I'm going to ditch it and just show a generic `Error: Could not compile code` message.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/348#issuecomment-1008367607
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/348
node_id: IC_kwDOCGYnMM48Gnf3
created_at: 2022-01-09T20:22:43Z
updated_at: 2022-01-09T20:22:43Z
author_association: OWNER
body:
  I'm not going to implement `--page-size` unless someone specifically requests it - I don't like having features that I've never needed to use myself.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008526736
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/364
node_id: IC_kwDOCGYnMM48HOWQ
created_at: 2022-01-10T04:07:29Z
updated_at: 2022-01-10T04:07:29Z
author_association: OWNER
body:
  I think this test is right:
  ```python
  def test_insert_streaming_batch_size_1(db_path):
      # https://github.com/simonw/sqlite-utils/issues/364
      # Streaming with --batch-size 1 should commit on each record
      # Can't use CliRunner().invoke() here bacuse we need to
      # run assertions in between writing to process stdin
      proc = subprocess.Popen(
          [
              sys.executable,
              "-m",
              "sqlite_utils",
              "insert",
              db_path,
              "rows",
              "-",
              "--nl",
              "--batch-size",
              "1",
          ],
          stdin=subprocess.PIPE,
      )
      proc.stdin.write(b'{"name": "Azi"}')
      proc.stdin.flush()
      assert list(Database(db_path)["rows"].rows) == [{"name": "Azi"}]
      proc.stdin.write(b'{"name": "Suna"}')
      proc.stdin.flush()
      assert list(Database(db_path)["rows"].rows) == [{"name": "Azi"}, {"name": "Suna"}]
      proc.stdin.close()
      proc.wait()
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008537194
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/364
node_id: IC_kwDOCGYnMM48HQ5q
created_at: 2022-01-10T04:29:53Z
updated_at: 2022-01-10T04:31:29Z
author_association: OWNER
body:
  After a bunch of debugging with `print()` statements it's clear that the problem isn't with when things are committed or the size of the batches - it's that the data sent to standard input is all being processed in one go, not a line at a time.

  I think that's because it is being buffered by this: https://github.com/simonw/sqlite-utils/blob/d2a79d200f9071a86027365fa2a576865b71064f/sqlite_utils/cli.py#L759-L770

  The buffering is there so that we can sniff the first few bytes to detect if it's a CSV file - added in 99ff0a288c08ec2071139c6031eb880fa9c95310 for #230. So maybe for non-CSV inputs we should disable buffering?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008545140
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/364
node_id: IC_kwDOCGYnMM48HS10
created_at: 2022-01-10T05:01:34Z
updated_at: 2022-01-10T05:01:34Z
author_association: OWNER
body:
  Urgh, tests are still failing intermittently - for example:
  ```
          time.sleep(0.4)
  >       assert list(Database(db_path)["rows"].rows) == [{"name": "Azi"}]
  E       AssertionError: assert [] == [{'name': 'Azi'}]
  E         Right contains one more item: {'name': 'Azi'}
  E         Full diff:
  E         - [{'name': 'Azi'}]
  E         + []
  ```
  I'm going to change this code to keep on trying up to 10 seconds - that should get the tests to pass faster on most machines.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008546573
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/364
node_id: IC_kwDOCGYnMM48HTMN
created_at: 2022-01-10T05:05:15Z
updated_at: 2022-01-10T05:05:15Z
author_association: OWNER
body:
  Bit nasty but it might work:
  ```python
      def try_until(expected):
          tries = 0
          while True:
              rows = list(Database(db_path)["rows"].rows)
              if rows == expected:
                  return
              tries += 1
              if tries > 10:
                  assert False, "Expected {}, got {}".format(expected, rows)
              time.sleep(tries * 0.1)

      try_until([{"name": "Azi"}])
      proc.stdin.write(b'{"name": "Suna"}\n')
      proc.stdin.flush()
      try_until([{"name": "Azi"}, {"name": "Suna"}])
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/375#issuecomment-1008556706
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/375
node_id: IC_kwDOCGYnMM48HVqi
created_at: 2022-01-10T05:33:41Z
updated_at: 2022-01-10T05:33:41Z
author_association: OWNER
body:
  I tested the prototype like this:

      sqlite-utils blah.db 'create table blah (id integer primary key, name text)' 
      echo 'id,name
      1,Cleo
      2,Chicken' > blah.csv
      sqlite-utils bulk blah.db 'insert into blah (id, name) values (:id, :name)' blah.csv --csv

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009273525
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/366
node_id: IC_kwDOCGYnMM48KEq1
created_at: 2022-01-10T19:32:39Z
updated_at: 2022-01-10T19:32:39Z
author_association: OWNER
body:
  I'm going to implement the Python library methods based on the prototype:
  ```diff
  commit 650f97a08f29a688c530e5f6c9eedc9269ed7bdc
  Author: Simon Willison <swillison@gmail.com>
  Date:   Sat Jan 8 13:34:01 2022 -0800

      Initial prototype of .analyze(), refs #366

  diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py
  index dfc4723..1348b4a 100644
  --- a/sqlite_utils/db.py
  +++ b/sqlite_utils/db.py
  @@ -923,6 +923,13 @@ class Database:
           "Run a SQLite ``VACUUM`` against the database."
           self.execute("VACUUM;")
   
  +    def analyze(self, name=None):
  +        "Run ``ANALYZE`` against the entire database or a named table or index."
  +        sql = "ANALYZE"
  +        if name is not None:
  +            sql += " [{}]".format(name)
  +        self.execute(sql)
  +
   
   class Queryable:
       def exists(self) -> bool:
  @@ -2902,6 +2909,10 @@ class Table(Queryable):
               )
           return self
   
  +    def analyze(self):
  +        "Run ANALYZE against this table"
  +        self.db.analyze(self.name)
  +
       def analyze_column(
           self, column: str, common_limit: int = 10, value_truncate=None, total_rows=None
       ) -> "ColumnDetails":
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009285627
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/366
node_id: IC_kwDOCGYnMM48KHn7
created_at: 2022-01-10T19:49:19Z
updated_at: 2022-01-10T19:51:25Z
author_association: OWNER
body:
  Documentation for those two new methods: https://sqlite-utils.datasette.io/en/latest/python-api.html#optimizing-index-usage-with-analyze

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009286373
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/366
node_id: IC_kwDOCGYnMM48KHzl
created_at: 2022-01-10T19:50:22Z
updated_at: 2022-01-10T19:50:22Z
author_association: OWNER
body:
  With respect to #365, I'm now thinking that having the ability to say "... and then run ANALYZE" could be useful for a bunch of Python methods. For example:

  ```python
  db["dogs"].insert_all(list_of_dogs, analyze=True)
  db["dogs"].create_index(["name"], analyze=True)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009288898
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/366
node_id: IC_kwDOCGYnMM48KIbC
created_at: 2022-01-10T19:54:04Z
updated_at: 2022-01-10T19:54:04Z
author_association: OWNER
body:
  Having browsed the API reference I think the methods that would benefit from an `analyze=True` parameter are:

  - `db.create_index`
  - `table.insert_all`
  - `table.upsert_all`
  - `table.delete_where`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009508865
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/366
node_id: IC_kwDOCGYnMM48K-IB
created_at: 2022-01-11T01:08:51Z
updated_at: 2022-01-11T01:08:51Z
author_association: OWNER
body:
  The Python methods are all done now, next step is the CLI options. I'll do those in a separate issue.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1009521921
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/365
node_id: IC_kwDOCGYnMM48LBUB
created_at: 2022-01-11T01:37:53Z
updated_at: 2022-01-11T01:37:53Z
author_association: OWNER
body:
  I decided to go with making this opt-in, mainly for consistency with the other places where I added this feature - see:
  - #379 
  - #366

  You can now run the following:

      sqlite-utils create-index mydb.db mytable mycolumn --analyze

  And ``ANALYZE`` will be run on the index once it has been created.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/377#issuecomment-1009531863
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/377
node_id: IC_kwDOCGYnMM48LDvX
created_at: 2022-01-11T02:03:00Z
updated_at: 2022-01-11T02:03:00Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/377?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#377](https://codecov.io/gh/simonw/sqlite-utils/pull/377?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (f4ea0d3) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/b6dad08a8389736b7e960cfe9bc719cfc21a98f5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (b6dad08) will **decrease** coverage by `0.01%`.
  > The diff coverage is `96.00%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/377/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/377?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@            Coverage Diff             @@
  ##             main     #377      +/-   ##
  ==========================================
  - Coverage   96.52%   96.50%   -0.02%     
  ==========================================
    Files           6        6              
    Lines        2330     2378      +48     
  ==========================================
  + Hits         2249     2295      +46     
  - Misses         81       83       +2     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/377?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [sqlite\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/377/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.49% <94.28%> (-0.06%)` | :arrow_down: |
  | [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/377/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.68% <100.00%> (+0.03%)` | :arrow_up: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/377?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/377?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [b6dad08...f4ea0d3](https://codecov.io/gh/simonw/sqlite-utils/pull/377?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/377#issuecomment-1009532125
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/377
node_id: IC_kwDOCGYnMM48LDzd
created_at: 2022-01-11T02:03:35Z
updated_at: 2022-01-11T02:03:35Z
author_association: OWNER
body:
  Documentation: https://github.com/simonw/sqlite-utils/blob/f4ea0d32c0543373eefaa9b9f3911eb07549eecb/docs/cli.rst#executing-sql-in-bulk

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/377#issuecomment-1009534817
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/377
node_id: IC_kwDOCGYnMM48LEdh
created_at: 2022-01-11T02:09:38Z
updated_at: 2022-01-11T02:09:38Z
author_association: OWNER
body:
  I tested this like so:
  ```
  % wget 'https://raw.githubusercontent.com/wri/global-power-plant-database/master/output_database/global_power_plant_database.csv'
  % sqlite-utils create-database test.db
  % sqlite-utils create-table test.db power_plants url text owner text
  % sqlite-utils schema test.db 
  CREATE TABLE [power_plants] (
     [url] TEXT,
     [owner] TEXT
  );
  % sqlite-utils bulk test.db 'insert into power_plants (url, owner) values (:url, :owner)' global_power_plant_database.csv --csv
    [------------------------------------]    0%
    [###################################-]   99%
  % sqlite-utils tables --counts test.db -t
  table           count
  ------------  -------
  power_plants    33643
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/380#issuecomment-1009544785
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/380
node_id: IC_kwDOCGYnMM48LG5R
created_at: 2022-01-11T02:32:56Z
updated_at: 2022-01-11T02:32:56Z
author_association: OWNER
body:
  CLI and Python library improvements to help run [ANALYZE](https://www.sqlite.org/lang_analyze.html) after creating indexes or inserting rows, to gain better performance from the SQLite query planner when it runs against indexes.

  Three new CLI commands: `create-database`, `analyze` and `bulk`.

  -   New `sqlite-utils create-database` command for creating new empty database files. ([#348](https://github.com/simonw/sqlite-utils/issues/348))
  -   New Python methods for running `ANALYZE` against a database, table or index: `db.analyze()` and `table.analyze()`, see [Optimizing index usage with ANALYZE](https://sqlite-utils.datasette.io/en/stable/python-api.html#python-api-analyze). ([#366](https://github.com/simonw/sqlite-utils/issues/366))
  -   New [sqlite-utils analyze command](https://sqlite-utils.datasette.io/en/stable/cli.html#cli-analyze) for running `ANALYZE` using the CLI. ([#379](https://github.com/simonw/sqlite-utils/issues/379))
  -   The `create-index`, `insert` and `update` commands now have a new `--analyze` option for running `ANALYZE` after the command has completed. ([#379](https://github.com/simonw/sqlite-utils/issues/379))
  -   New [sqlite-utils bulk command](https://sqlite-utils.datasette.io/en/stable/cli.html#cli-bulk) which can import records in the same way as `sqlite-utils insert` (from JSON, CSV or TSV) and use them to bulk execute a parametrized SQL query. ([#375](https://github.com/simonw/sqlite-utils/issues/375))
  -   The CLI tool can now also be run using `python -m sqlite_utils`. ([#368](https://github.com/simonw/sqlite-utils/issues/368))
  -   Using `--fmt` now implies `--table`, so you don't need to pass both options. ([#374](https://github.com/simonw/sqlite-utils/issues/374))
  -   The `--convert` function applied to rows can now modify the row in place. ([#371](https://github.com/simonw/sqlite-utils/issues/371))
  -   The [insert-files command](https://sqlite-utils.datasette.io/en/stable/cli.html#cli-insert-files) supports two new columns: `stem` and `suffix`. ([#372](https://github.com/simonw/sqlite-utils/issues/372))
  -   The `--nl` import option now ignores blank lines in the input. ([#376](https://github.com/simonw/sqlite-utils/issues/376))
  -   Fixed bug where streaming input to the `insert` command with `--batch-size 1` would appear to only commit after several rows had been ingested, due to unnecessary input buffering. ([#364](https://github.com/simonw/sqlite-utils/issues/364))

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/383#issuecomment-1010333511
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/383
node_id: IC_kwDOCGYnMM48OHdH
created_at: 2022-01-11T20:27:08Z
updated_at: 2022-01-11T20:27:08Z
author_association: OWNER
body:
  I'll call the new page "CLI reference", for consistency with the API reference page here: https://sqlite-utils.datasette.io/en/stable/reference.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/383#issuecomment-1010386802
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/383
node_id: IC_kwDOCGYnMM48OUdy
created_at: 2022-01-11T21:44:53Z
updated_at: 2022-01-11T21:44:53Z
author_association: OWNER
body:
  Here's the `cog` code I used: https://github.com/simonw/sqlite-utils/blob/1d44b0cc2784c94aed1bcf350225cd86ee1aa7e5/docs/cli-reference.rst#L11-L76

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/381#issuecomment-1010441118
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/381
node_id: IC_kwDOCGYnMM48Ohue
created_at: 2022-01-11T22:56:53Z
updated_at: 2022-01-11T22:57:09Z
author_association: OWNER
body:
  `sqlite-utils search` has `--limit` already: https://sqlite-utils.datasette.io/en/latest/cli-reference.html#search

  ```
    --limit INTEGER        Number of rows to return - defaults to everything
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1590#issuecomment-1010533133
issue_url: https://api.github.com/repos/simonw/datasette/issues/1590
node_id: IC_kwDOBm6k_c48O4MN
created_at: 2022-01-12T01:19:19Z
updated_at: 2022-01-12T01:19:19Z
author_association: OWNER
body:
  Thanks for the steps to reproduce - I have your bug running on my laptop now.

  I've been mostly testing this stuff using the hosted copy of Datasette here, which doesn't exhibit the bug: https://datasette-apache-proxy-demo.fly.dev/prefix/fixtures?sql=select+sqlite_version%28%29

  Something interesting definitely going on here!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1590#issuecomment-1010537058
issue_url: https://api.github.com/repos/simonw/datasette/issues/1590
node_id: IC_kwDOBm6k_c48O5Ji
created_at: 2022-01-12T01:26:34Z
updated_at: 2022-01-12T01:26:34Z
author_association: OWNER
body:
  I'm using the https://datasette.io/plugins/datasette-debug-asgi plugin to investigate.

  On my laptop using Daphne I get this: http://127.0.0.1:8032/datasettes/-/asgi-scope

  ```
  {'actor': None,
   'asgi': {'version': '3.0'},
   'client': ['127.0.0.1', 53767],
   'csrftoken': <function asgi_csrf_decorator.<locals>._asgi_csrf_decorator.<locals>.app_wrapped_with_csrf.<locals>.get_csrftoken at 0x1122aeef0>,
   'headers': [(b'host', b'127.0.0.1:8032'),
               (b'user-agent',
                b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:95.0) Gecko'
                b'/20100101 Firefox/95.0'),
               (b'accept',
                b'text/html,application/xhtml+xml,application/xml;q=0.9,image/'
                b'avif,image/webp,*/*;q=0.8'),
               (b'accept-language', b'en-US,en;q=0.5'),
               (b'accept-encoding', b'gzip, deflate'),
               (b'dnt', b'1'),
               (b'connection', b'keep-alive'),
               (b'cookie', b'_ga=GA1.1.742283954.1628542653'),
               (b'upgrade-insecure-requests', b'1'),
               (b'sec-fetch-dest', b'document'),
               (b'sec-fetch-mode', b'navigate'),
               (b'sec-fetch-site', b'none'),
               (b'sec-fetch-user', b'?1')],
   'http_version': '1.1',
   'method': 'GET',
   'path': '/datasettes/-/asgi-scope',
   'path_remaining': '',
   'query_string': b'',
   'raw_path': b'/datasettes/-/asgi-scope',
   'root_path': '',
   'route_path': '/-/asgi-scope',
   'scheme': 'http',
   'server': ['127.0.0.1', 8032],
   'type': 'http',
   'url_route': {'kwargs': {}}}
  ```
  On the demo running on Fly (which I just redeployed with that plugin) I get this: https://datasette-apache-proxy-demo.fly.dev/prefix/-/asgi-scope

  ```
  {'actor': None,
   'asgi': {'spec_version': '2.1', 'version': '3.0'},
   'client': ('86.109.12.167', 0),
   'csrftoken': <function asgi_csrf_decorator.<locals>._asgi_csrf_decorator.<locals>.app_wrapped_with_csrf.<locals>.get_csrftoken at 0x7f4c0413bca0>,
   'headers': [(b'host', b'datasette-apache-proxy-demo.fly.dev'),
               (b'user-agent',
                b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:95.0) Gecko'
                b'/20100101 Firefox/95.0'),
               (b'accept',
                b'text/html,application/xhtml+xml,application/xml;q=0.9,image/'
                b'avif,image/webp,*/*;q=0.8'),
               (b'accept-language', b'en-US,en;q=0.5'),
               (b'accept-encoding', b'gzip, deflate, br'),
               (b'dnt', b'1'),
               (b'x-request-start', b't=1641950740651658'),
               (b'sec-fetch-dest', b'document'),
               (b'sec-fetch-mode', b'navigate'),
               (b'sec-fetch-site', b'none'),
               (b'sec-fetch-user', b'?1'),
               (b'fly-client-ip', b'24.5.172.176'),
               (b'x-forwarded-for',
                b'24.5.172.176, 213.188.193.173, 86.109.12.167'),
               (b'fly-forwarded-proto', b'https'),
               (b'x-forwarded-proto', b'https'),
               (b'fly-forwarded-ssl', b'on'),
               (b'x-forwarded-ssl', b'on'),
               (b'fly-forwarded-port', b'443'),
               (b'x-forwarded-port', b'443'),
               (b'fly-region', b'sjc'),
               (b'fly-request-id', b'01FS5Y805BX43HM94T8XW610KG'),
               (b'via', b'2 fly.io'),
               (b'fly-dispatch-start', b't=1641950740683198;instance=87f188a2'),
               (b'x-forwarded-host', b'datasette-apache-proxy-demo.fly.dev'),
               (b'x-forwarded-server', b'localhost'),
               (b'connection', b'Keep-Alive')],
   'http_version': '1.1',
   'method': 'GET',
   'path': '/-/asgi-scope',
   'query_string': b'',
   'raw_path': b'/-/asgi-scope',
   'root_path': '',
   'scheme': 'https',
   'server': ('127.0.0.1', 8001),
   'type': 'http',
   'url_route': {'kwargs': {}}}
  ```

  The version that works as ` 'raw_path': b'/-/asgi-scope'` - the version that fails has `'raw_path': b'/datasettes/-/asgi-scope'`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1590#issuecomment-1010538016
issue_url: https://api.github.com/repos/simonw/datasette/issues/1590
node_id: IC_kwDOBm6k_c48O5Yg
created_at: 2022-01-12T01:28:19Z
updated_at: 2022-01-12T01:28:19Z
author_association: OWNER
body:
  The Daphne one has this key: `'route_path': '/-/asgi-scope',`

  Maybe Datasette's routing code needs to look out for that, if it's available, and use it to reconstruct the requested path?

  The code in question is here: https://github.com/simonw/datasette/blob/8c401ee0f054de2f568c3a8302c9223555146407/datasette/app.py#L1143-L1149

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1590#issuecomment-1010538188
issue_url: https://api.github.com/repos/simonw/datasette/issues/1590
node_id: IC_kwDOBm6k_c48O5bM
created_at: 2022-01-12T01:28:41Z
updated_at: 2022-01-12T01:30:43Z
author_association: OWNER
body:
  Oh wait! It looks like `route_path` is something I invented there.

  Yup, I added it in https://github.com/simonw/datasette/commit/a63412152518581c6a3d4e142b937e27dabdbfdb - commit message says:

  > - new `route_path` key in `request.scope` storing the path that was used for routing with the `base_url` prefix stripped

  So actually part of the mystery here is: why does the Fly hosted one NOT have that key?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1590#issuecomment-1010540923
issue_url: https://api.github.com/repos/simonw/datasette/issues/1590
node_id: IC_kwDOBm6k_c48O6F7
created_at: 2022-01-12T01:33:49Z
updated_at: 2022-01-12T01:33:49Z
author_association: OWNER
body:
  Looking closer at the code quoted above, it doesn't modify `path` or `raw_path` at all - ALL it does is add the `route_path` to the scope.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1590#issuecomment-1010556333
issue_url: https://api.github.com/repos/simonw/datasette/issues/1590
node_id: IC_kwDOBm6k_c48O92t
created_at: 2022-01-12T02:03:59Z
updated_at: 2022-01-12T02:03:59Z
author_association: NONE
body:
  Thank you for the quick reply! Just a quick observation, I am running this locally without a proxy, whereas your fly example seems to be running behind an apache proxy (if the name is accurate). Can it be that the apache proxy strips the prefix before it passes on the request to the daphne backend?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1590#issuecomment-1010559681
issue_url: https://api.github.com/repos/simonw/datasette/issues/1590
node_id: IC_kwDOBm6k_c48O-rB
created_at: 2022-01-12T02:10:20Z
updated_at: 2022-01-12T02:10:20Z
author_association: NONE
body:
  In my example, path matching happens at the application layer (being the Django channels URLRouter). That might be a somewhat exotic solution that would normally be solved by a proxy like Apache or Nginx. However, in my specific use case, this is a "feature" enabling me to do simple management of databases and metadata from within a Django admin app instance mapped in that same router.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1591#issuecomment-1010764036
issue_url: https://api.github.com/repos/simonw/datasette/issues/1591
node_id: IC_kwDOBm6k_c48PwkE
created_at: 2022-01-12T08:22:16Z
updated_at: 2022-01-12T08:22:32Z
author_association: OWNER
body:
  The challenge here is avoiding clashes. What if a plugin adds an option that I later want to use for a new Datasette core feature? Or what if two plugins define the same option?

  Maybe the solution is to make them use namespaces defined by the plugin name.

  How about this:

      datasette t.db --plugin-setting datasette-tiddlywiki db t.db

  It's a bit verbose having an option that itself then takes THREE strings: plugin name, setting name, setting value - but it would work.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/348#issuecomment-1011855133
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/348
node_id: IC_kwDOCGYnMM48T68d
created_at: 2022-01-13T07:06:59Z
updated_at: 2022-01-13T07:06:59Z
author_association: OWNER
body:
  Wrote a lot more about this feature here: https://simonwillison.net/2022/Jan/12/how-i-build-a-feature/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1593#issuecomment-1012128696
issue_url: https://api.github.com/repos/simonw/datasette/issues/1593
node_id: IC_kwDOBm6k_c48U9u4
created_at: 2022-01-13T13:18:35Z
updated_at: 2022-01-13T13:18:35Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1593?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1593](https://codecov.io/gh/simonw/datasette/pull/1593?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (df73ebb) into [main](https://codecov.io/gh/simonw/datasette/commit/8c401ee0f054de2f568c3a8302c9223555146407?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (8c401ee) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1593/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1593?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1593   +/-   ##
  =======================================
    Coverage   92.09%   92.09%           
  =======================================
    Files          34       34           
    Lines        4516     4516           
  =======================================
    Hits         4159     4159           
    Misses        357      357           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1593?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1593?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [8c401ee...df73ebb](https://codecov.io/gh/simonw/datasette/pull/1593?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012158895
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/79
node_id: IC_kwDOCGYnMM48VFGv
created_at: 2022-01-13T13:55:59Z
updated_at: 2022-01-13T13:55:59Z
author_association: CONTRIBUTOR
body:
  Came here to add this. I might pick it up.

  Would also add a utility to create (and update and delete?) a spatial index. It's not much code but I have to look it up every time.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012230212
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/79
node_id: IC_kwDOCGYnMM48VWhE
created_at: 2022-01-13T15:15:13Z
updated_at: 2022-01-13T15:15:13Z
author_association: CONTRIBUTOR
body:
  Some proposals I'd add to sqlite-utils:

  Some version of this, from [geojson-to-sqlite](https://github.com/simonw/geojson-to-sqlite/blob/main/geojson_to_sqlite/utils.py#L124-L130):

  ```python
  def init_spatialite(db, lib):
      db.conn.enable_load_extension(True)
      db.conn.load_extension(lib)
      # Initialize SpatiaLite if not yet initialized
      if "spatial_ref_sys" in db.table_names():
          return
      db.conn.execute("select InitSpatialMetadata(1)")
  ```

  Also a function for creating a spatial index:

  ```python
  db.conn.execute("select CreateSpatialIndex(?, ?)", [table, "geometry"])
  ```

  I don't know the nuances of updating a spatial index, or checking if one already exists. This could be a CLI method like:

  ```sh
  sqlite-utils spatial-index spatial.db table-name column-name
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012253198
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/79
node_id: IC_kwDOCGYnMM48VcIO
created_at: 2022-01-13T15:39:14Z
updated_at: 2022-01-13T15:39:14Z
author_association: CONTRIBUTOR
body:
  Other thing: If there get to be enough utils, I think it's worth moving all the spatialite stuff into its own file (`gis.py` or something) just so it's easier to find later.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012413729
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/79
node_id: IC_kwDOCGYnMM48WDUh
created_at: 2022-01-13T18:50:00Z
updated_at: 2022-01-13T18:50:00Z
author_association: CONTRIBUTOR
body:
  One more thing I'm going to add: A method to add a geometry column, which I'll need to do to create a spatial index on a table.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1591#issuecomment-1012504251
issue_url: https://api.github.com/repos/simonw/datasette/issues/1591
node_id: IC_kwDOBm6k_c48WZa7
created_at: 2022-01-13T20:49:19Z
updated_at: 2022-01-13T20:49:59Z
author_association: OWNER
body:
  I try to stick pretty closely to what [Click](https://click.palletsprojects.com/en/8.0.x/) supports, and Click likes you to define options explicitly so that it can display them in the output of `--help`.

  But... that makes me think that actually showing these options in `--help` is likely a better idea. My `--plugin-setting` concept would have help that looks something like this:
  ```
  % datasette serve --help
    ...
    --plugin-setting <TEXT TEXT TEXT>...     Setting for a specified plugin.
  ```
  That's not great help!

  The alternative would be to allow plugins to register their extra options with the command - which would mean the help output could look like this instead:
  ```
  % datasette serve --help
    ...
    --tiddlywiki-db <TEXT>    Name of database to use for datasette-tiddlywiki
  ```
  This feels like a good argument to me for plugins to explicitly register their settings. I'm not sure if I should enforce the `name-of-plugin-` prefix, or if I should allow plugins to define any setting they like. I'm still nervous about plugins over-riding existing or future default options to that command.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1591#issuecomment-1012505706
issue_url: https://api.github.com/repos/simonw/datasette/issues/1591
node_id: IC_kwDOBm6k_c48WZxq
created_at: 2022-01-13T20:51:30Z
updated_at: 2022-01-13T20:51:30Z
author_association: OWNER
body:
  Another option: if I make plugin settings a higher level concept in Datasette than they are at the moment, I could allow them to be set either using `--options` OR using the existing `metadata.yml/json` mechanism. https://docs.datasette.io/en/stable/plugins.html#plugin-configuration

  I want to make changes to that anyway, because I'm increasingly uncomfortable with plugin settings ending up in the "metadata" mechanism.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1591#issuecomment-1012506595
issue_url: https://api.github.com/repos/simonw/datasette/issues/1591
node_id: IC_kwDOBm6k_c48WZ_j
created_at: 2022-01-13T20:52:56Z
updated_at: 2022-01-13T20:52:56Z
author_association: OWNER
body:
  You can already run `datasette --help-settings` to see detailed help on available settings. Maybe `datasette --help-plugin-settings` could do the same thing for plugin settings?

  Or I could even have available plugin settings show up as a list at the bottom of the `datasette --help-settings` output - which currently looks like this:

  ```
  % datasette --help-settings
  Settings:
    default_page_size            Default page size for the table view
                                 (default=100)
    max_returned_rows            Maximum rows that can be returned from a table
                                 or custom query (default=1000)
    num_sql_threads              Number of threads in the thread pool for
                                 executing SQLite queries (default=3)
    sql_time_limit_ms            Time limit for a SQL query in milliseconds
                                 (default=1000)
    default_facet_size           Number of values to return for requested facets
                                 (default=30)
    facet_time_limit_ms          Time limit for calculating a requested facet
                                 (default=200)
    facet_suggest_time_limit_ms  Time limit for calculating a suggested facet
                                 (default=50)
    hash_urls                    Include DB file contents hash in URLs, for far-
                                 future caching (default=False)
    allow_facet                  Allow users to specify columns to facet using
                                 ?_facet= parameter (default=True)
    allow_download               Allow users to download the original SQLite
                                 database files (default=True)
    suggest_facets               Calculate and display suggested facets
                                 (default=True)
    default_cache_ttl            Default HTTP cache TTL (used in Cache-Control:
                                 max-age= header) (default=5)
    default_cache_ttl_hashed     Default HTTP cache TTL for hashed URL pages
                                 (default=31536000)
    cache_size_kb                SQLite cache size in KB (0 == use SQLite
                                 default) (default=0)
    allow_csv_stream             Allow .csv?_stream=1 to download all rows
                                 (ignoring max_returned_rows) (default=True)
    max_csv_mb                   Maximum size allowed for CSV export in MB - set
                                 0 to disable this limit (default=100)
    truncate_cells_html          Truncate cells longer than this in HTML table
                                 view - set 0 to disable (default=2048)
    force_https_urls             Force URLs in API output to always use https://
                                 protocol (default=False)
    template_debug               Allow display of template debug information
                                 with ?_context=1 (default=False)
    trace_debug                  Allow display of SQL trace debug information
                                 with ?_trace=1 (default=False)
    base_url                     Datasette URLs should use this base path
                                 (default=/)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1594#issuecomment-1012508787
issue_url: https://api.github.com/repos/simonw/datasette/issues/1594
node_id: IC_kwDOBm6k_c48Wahz
created_at: 2022-01-13T20:56:14Z
updated_at: 2022-01-13T20:56:34Z
author_association: OWNER
body:
  The implementation can be _almost_ exactly the same as this: https://github.com/simonw/sqlite-utils/blame/74586d3cb26fa3cc3412721985ecdc1864c2a31d/docs/cli-reference.rst#L11-L76

  I need to do something extra to ensure the output of `datasette --help-settings` is shown too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1594#issuecomment-1012535024
issue_url: https://api.github.com/repos/simonw/datasette/issues/1594
node_id: IC_kwDOBm6k_c48Wg7w
created_at: 2022-01-13T21:36:53Z
updated_at: 2022-01-13T21:36:53Z
author_association: OWNER
body:
  I went with a simpler pattern that `sqlite-utils` because Datasette has a lot less commands: https://github.com/simonw/datasette/blob/4b23f01f3e668c8f2a2f1a294be49f49b4073969/docs/cli-reference.rst#L9-L35

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1466#issuecomment-1012546223
issue_url: https://api.github.com/repos/simonw/datasette/issues/1466
node_id: IC_kwDOBm6k_c48Wjqv
created_at: 2022-01-13T21:54:51Z
updated_at: 2022-01-13T21:54:51Z
author_association: OWNER
body:
  Going with this for the copy:

  > [Datasette Desktop](https://datasette.io/desktop) is a packaged Mac application which bundles Datasette together with Python and allows you to install and run Datasette directly on your laptop. This is the best option for local installation if you are not comfortable using the command line.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/650#issuecomment-1012552760
issue_url: https://api.github.com/repos/simonw/datasette/issues/650
node_id: IC_kwDOBm6k_c48WlQ4
created_at: 2022-01-13T22:04:56Z
updated_at: 2022-01-13T22:04:56Z
author_association: OWNER
body:
  Challenge: explain the difference between view as in SQL view, and view as in the code that serves `TableView` / `DatabaseView` etc.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1588#issuecomment-1012554208
issue_url: https://api.github.com/repos/simonw/datasette/issues/1588
node_id: IC_kwDOBm6k_c48Wlng
created_at: 2022-01-13T22:07:15Z
updated_at: 2022-01-13T22:07:15Z
author_association: OWNER
body:
  This works now: https://latest.datasette.io/fixtures?sql=explain+query+plan++select+*+from+facetable

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1595#issuecomment-1012575013
issue_url: https://api.github.com/repos/simonw/datasette/issues/1595
node_id: IC_kwDOBm6k_c48Wqsl
created_at: 2022-01-13T22:29:22Z
updated_at: 2022-01-13T22:29:22Z
author_association: OWNER
body:
  Most of these are already written for these two alpha releases:
  - https://github.com/simonw/datasette/releases/tag/0.60a0
  - https://github.com/simonw/datasette/releases/tag/0.60a1

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1590#issuecomment-1012583091
issue_url: https://api.github.com/repos/simonw/datasette/issues/1590
node_id: IC_kwDOBm6k_c48Wsqz
created_at: 2022-01-13T22:41:15Z
updated_at: 2022-01-13T22:41:15Z
author_association: OWNER
body:
  Seeing as this area of the code has produced so many bugs in the past, I think part of the fix may be to write comprehensive documentation about how routing works for the internals documentation. Doing so might help me figure this bug out!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1595#issuecomment-1012626243
issue_url: https://api.github.com/repos/simonw/datasette/issues/1595
node_id: IC_kwDOBm6k_c48W3ND
created_at: 2022-01-14T00:00:33Z
updated_at: 2022-01-14T00:00:33Z
author_association: OWNER
body:
  Copying those in here:

  -   New plugin hook: [filters_from_request(request, database, table, datasette)](https://docs.datasette.io/en/latest/plugin_hooks.html#plugin-hook-filters-from-request), which runs on the table page and can be used to support new custom query string parameters that modify the SQL query. ([#473](https://github.com/simonw/datasette/issues/473))
  -   The number of unique values in a facet is now always displayed. Previously it was only displayed if the user specified `?_facet_size=max`. ([#1556](https://github.com/simonw/datasette/issues/1556))
  -   Fixed bug where `?_facet_array=tags&_facet=tags` would only display one of the two selected facets. ([#625](https://github.com/simonw/datasette/issues/625))
  -   Facets of type `date` or `array` can now be configured in `metadata.json`, see [Facets in metadata.json](https://docs.datasette.io/en/latest/facets.html#facets-metadata). Thanks, David Larlet. ([#1552](https://github.com/simonw/datasette/issues/1552))
  -   New `?_nosuggest=1` parameter for table views, which disables facet suggestion. ([#1557](https://github.com/simonw/datasette/issues/1557))
  -   Label columns detected for foreign keys are now case-insensitive, so `Name` or `TITLE` will be detected in the same way as `name` or `title`. ([#1544](https://github.com/simonw/datasette/issues/1544))
  -   The query string variables exposed by `request.args` will now include blank strings for arguments such as `foo` in `?foo=&bar=1` rather than ignoring those parameters entirely. ([#1551](https://github.com/simonw/datasette/issues/1551))
  -   Database write connections now execute the [prepare_connection(conn, database, datasette)](https://docs.datasette.io/en/latest/plugin_hooks.html#plugin-hook-prepare-connection) plugin hook. ([#1564](https://github.com/simonw/datasette/issues/1564))
  -   The `Datasette()` constructor no longer requires the `files=` argument, and is now documented at [Datasette class](https://docs.datasette.io/en/latest/internals.html#internals-datasette). ([#1563](https://github.com/simonw/datasette/issues/1563))
  -   The tracing feature now traces write queries, not just read queries. ([#1568](https://github.com/simonw/datasette/issues/1568))
  -   Added two methods for writing to the database: [await db.execute_write_script(sql, block=False)](https://docs.datasette.io/en/latest/internals.html#database-execute-write-script) and [await db.execute_write_many(sql, params_seq, block=False)](https://docs.datasette.io/en/latest/internals.html#database-execute-write-many). ([#1570](https://github.com/simonw/datasette/issues/1570))
  -   Made several performance improvements to the database schema introspection code that runs when Datasette first starts up. ([#1555](https://github.com/simonw/datasette/issues/1555))
  -   Fixed bug where writable canned queries could not be used with custom templates. ([#1547](https://github.com/simonw/datasette/issues/1547))

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1597#issuecomment-1012629825
issue_url: https://api.github.com/repos/simonw/datasette/issues/1597
node_id: IC_kwDOBm6k_c48W4FB
created_at: 2022-01-14T00:07:36Z
updated_at: 2022-01-14T00:07:36Z
author_association: OWNER
body:
  Fixed:
  ```
  % datasette --help        
  Usage: datasette [OPTIONS] COMMAND [ARGS]...

    Datasette is an open source multi-tool for exploring and publishing data

    About Datasette: https://datasette.io/
    Full documentation: https://docs.datasette.io/

  Options:
    --version  Show the version and exit.
    --help     Show this message and exit.

  Commands:
    serve*     Serve up specified SQLite database files with a web UI
    inspect    Generate JSON summary of provided database files
  ...

  % datasette inspect --help
  Usage: datasette inspect [OPTIONS] [FILES]...

    Generate JSON summary of provided database files

    This can then be passed to "datasette --inspect-file" to speed up count
    operations against immutable database files.

  Options:
    --inspect-file TEXT
    --load-extension TEXT  Path to a SQLite extension to load
    --help                 Show this message and exit.
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1527#issuecomment-1012634659
issue_url: https://api.github.com/repos/simonw/datasette/issues/1527
node_id: IC_kwDOBm6k_c48W5Qj
created_at: 2022-01-14T00:17:00Z
updated_at: 2022-01-14T00:18:11Z
author_association: OWNER
body:
  That's because that page has this unnecessary hidden form field:
  ```html
  <input type="hidden" name="_city_id__gt" value="1">
  ```
  That field is added by this bit in the template: https://github.com/simonw/datasette/blob/515f8d38ebae203efc15ca79a8b42848276b35e5/datasette/templates/table.html#L119-L122

  Which is populated here:  https://github.com/simonw/datasette/blob/ace86566b28280091b3844cf5fbecd20158e9004/datasette/views/table.py#L813-L821

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1527#issuecomment-1012635696
issue_url: https://api.github.com/repos/simonw/datasette/issues/1527
node_id: IC_kwDOBm6k_c48W5gw
created_at: 2022-01-14T00:19:10Z
updated_at: 2022-01-14T00:20:36Z
author_association: OWNER
body:
  Oh! This is because `_city_id` has a leading underscore (for testing purposes). I think I need to filter out any keys that contain `__` in that case.

  What happens to columns that contain a `__`? They shouldn't be reflected in the hidden arguments either - this code is really only supposed to catch things like `_where` and `_m2m_through=` and `_col` - basically most of the list on https://docs.datasette.io/en/stable/json_api.html#special-table-arguments

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1598#issuecomment-1012643882
issue_url: https://api.github.com/repos/simonw/datasette/issues/1598
node_id: IC_kwDOBm6k_c48W7gq
created_at: 2022-01-14T00:34:49Z
updated_at: 2022-01-14T00:34:49Z
author_association: OWNER
body:
  There are four places in the documentation that use `.. literalinclude::` at the moment which I can replace - I can actually just link directly to the new https://docs.datasette.io/en/latest/cli-reference.html page instead of embedding the help directly.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1590#issuecomment-1012653966
issue_url: https://api.github.com/repos/simonw/datasette/issues/1590
node_id: IC_kwDOBm6k_c48W9-O
created_at: 2022-01-14T00:59:07Z
updated_at: 2022-01-14T00:59:07Z
author_association: OWNER
body:
  Since this is a special case bug for when using Datasette as a library I wonder if a good fix here would be to support something like this:
  ```python
  application = URLRouter([
    re_path(r"^datasettes/.*", asgi_cors(datasette_.app(remove_path_prefix="datasettes/"), allow_all=True)),
  ])
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1590#issuecomment-1012656790
issue_url: https://api.github.com/repos/simonw/datasette/issues/1590
node_id: IC_kwDOBm6k_c48W-qW
created_at: 2022-01-14T01:05:34Z
updated_at: 2022-01-14T01:05:34Z
author_association: OWNER
body:
  I think this prefixed string mechanism is supposed to prevent the `base_url` prefix from being applied twice: https://github.com/simonw/datasette/blob/3664ddd400062123e99500d28b160c7944408c1a/datasette/url_builder.py#L9-L16

  But with a bit of extra logging all of the inputs to that are NOT prefixed strings:

  ```
  Urls.path called with: /datasettes/fixtures/compound_three_primary_keys?_sort=content (PrefixedUrlString = False)
    returning /datasettes/datasettes/fixtures/compound_three_primary_keys?_sort=content
  ```
  So it looks like `urls.path(...)` is indeed the code responsible for doubling up that `/datasettes/` prefix.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1590#issuecomment-1012661522
issue_url: https://api.github.com/repos/simonw/datasette/issues/1590
node_id: IC_kwDOBm6k_c48W_0S
created_at: 2022-01-14T01:16:08Z
updated_at: 2022-01-14T01:16:34Z
author_association: OWNER
body:
  OK, I'm going to recommend a workaround for this instead. Here's `asgi.py` updated to strip the prefix before passing the request on to Datasette:
  ```python
  import pathlib
  from asgi_cors import asgi_cors
  from channels.routing import URLRouter
  from django.urls import re_path
  from datasette.app import Datasette


  def rewrite_path(app, prefix_to_strip):
      async def rewrite_path_app(scope, receive, send):
          if (
              scope["type"] == "http"
              and "path" in scope
              and scope["path"].startswith(prefix_to_strip)
          ):
              scope["path"] = scope["path"][len(prefix_to_strip) :]
              if "raw_path" in scope:
                  scope["raw_path"] = scope["raw_path"][len(prefix_to_strip) :]
          await app(scope, receive, send)

      return rewrite_path_app


  datasette_ = Datasette(
      files=["fixtures.db"],
      settings={"base_url": "/datasettes/", "plugins": {}},
  )
  application = URLRouter(
      [
          re_path(
              r"^datasettes/.*",
              asgi_cors(rewrite_path(datasette_.app(), "/datasettes"), allow_all=True),
          ),
      ]
  )
  ```
  This works on my laptop - please re-open the ticket if it doesn't work for you!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1595#issuecomment-1012664607
issue_url: https://api.github.com/repos/simonw/datasette/issues/1595
node_id: IC_kwDOBm6k_c48XAkf
created_at: 2022-01-14T01:22:58Z
updated_at: 2022-01-14T01:22:58Z
author_association: OWNER
body:
  - Upgraded Pluggy dependency to 1.0. #1575
  - Now using [Plausible](https://plausible.io/) analytics for the Datasette documentation.
  - The `db.execute_write()` internals method now defaults to blocking until the write operation has completed. Previously it defaulted to queuing the write and then continuing to run code while the write was in the queue. #1579
  - `explain query plan` is now allowed with varying amounts of white space in the query. #1588
  - New CLI reference page showing the output of `--help` for each of the `datasette` sub-commands. This lead to several small improvements to the help copy. #1594
  - Fixed bug where columns with a underscore prefix could result in unnecessary hidden form fields. #1527

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1591#issuecomment-1013668967
issue_url: https://api.github.com/repos/simonw/datasette/issues/1591
node_id: IC_kwDOBm6k_c48a1xn
created_at: 2022-01-15T11:53:21Z
updated_at: 2022-01-15T11:53:21Z
author_association: OWNER
body:
  The `datasette publish --plugin-secret name setting value` option already implements something a bit like this. https://docs.datasette.io/en/stable/plugins.html#secret-configuration-values

  It's a bit of a messy hack to compensate for metadata being visible. Maybe I could replace that mechanism with the proposed plugin configuration rethink from this issue.

  I still like the debug benefits of making plugin settings public - perhaps add a rule that if a plugin setting has a `secret:` prefix it gets redacted on a new `/-/plugin-settings` page.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1591#issuecomment-1013669543
issue_url: https://api.github.com/repos/simonw/datasette/issues/1591
node_id: IC_kwDOBm6k_c48a16n
created_at: 2022-01-15T11:56:59Z
updated_at: 2022-01-15T11:56:59Z
author_association: OWNER
body:
  There's actually already a way to move regular Datasette `--setting` options to a `settings.json` file thanks to configuration directory mode: https://docs.datasette.io/en/stable/settings.html

  Maybe extending that mechanism to handle plugins would be a neat path forward here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1013698557
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/79
node_id: IC_kwDOCGYnMM48a8_9
created_at: 2022-01-15T15:15:22Z
updated_at: 2022-01-15T15:15:22Z
author_association: CONTRIBUTOR
body:
  @simonw I have a PR here https://github.com/simonw/sqlite-utils/pull/385 that adds Spatialite helpers on the Python side. Please let me know how it looks.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1602#issuecomment-1016456784
issue_url: https://api.github.com/repos/simonw/datasette/issues/1602
node_id: IC_kwDOBm6k_c48leZQ
created_at: 2022-01-19T13:17:24Z
updated_at: 2022-03-06T01:30:46Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1602?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1602](https://codecov.io/gh/simonw/datasette/pull/1602?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (9eb0bdf) into [main](https://codecov.io/gh/simonw/datasette/commit/5010d1359b9e9db90a5a69a3ca22d12862893e00?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (5010d13) will **increase** coverage by `0.13%`.
  > The diff coverage is `n/a`.

  > :exclamation: Current head 9eb0bdf differs from pull request most recent head a9c69dc. Consider uploading reports for the commit a9c69dc to get more accurate results

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1602/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1602?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@            Coverage Diff             @@
  ##             main    #1602      +/-   ##
  ==========================================
  + Coverage   92.03%   92.16%   +0.13%     
  ==========================================
    Files          34       34              
    Lines        4557     4531      -26     
  ==========================================
  - Hits         4194     4176      -18     
  + Misses        363      355       -8     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1602?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/tracer.py](https://codecov.io/gh/simonw/datasette/pull/1602/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3RyYWNlci5weQ==) | `82.95% <0.00%> (-1.09%)` | :arrow_down: |
  | [datasette/cli.py](https://codecov.io/gh/simonw/datasette/pull/1602/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `77.85% <0.00%> (-0.09%)` | :arrow_down: |
  | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1602/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.79% <0.00%> (-0.07%)` | :arrow_down: |
  | [datasette/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1602/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL19faW5pdF9fLnB5) | `100.00% <0.00%> (ø)` | |
  | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1602/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `95.37% <0.00%> (+1.05%)` | :arrow_up: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1602?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1602?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [73f2d25...a9c69dc](https://codecov.io/gh/simonw/datasette/pull/1602?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1603#issuecomment-1016579661
issue_url: https://api.github.com/repos/simonw/datasette/issues/1603
node_id: IC_kwDOBm6k_c48l8ZN
created_at: 2022-01-19T15:27:05Z
updated_at: 2022-01-19T15:27:05Z
author_association: OWNER
body:
  I'd like this to be customizable. I'm going to load it from the template loading system such that a custom favicon for a specific instance can be dropped in using a file in `templates/`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1603#issuecomment-1016588326
issue_url: https://api.github.com/repos/simonw/datasette/issues/1603
node_id: IC_kwDOBm6k_c48l-gm
created_at: 2022-01-19T15:35:33Z
updated_at: 2022-01-19T15:35:33Z
author_association: OWNER
body:
  I don't think abusing the template loader mechanism for this will work: Jinja provides an API for loading text templates, but I don't think it can be sensibly abused to open binary image files instead. Loaded code is here: https://github.com/pallets/jinja/blob/main/src/jinja2/loaders.py

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1603#issuecomment-1016589140
issue_url: https://api.github.com/repos/simonw/datasette/issues/1603
node_id: IC_kwDOBm6k_c48l-tU
created_at: 2022-01-19T15:36:16Z
updated_at: 2022-01-19T15:36:16Z
author_association: OWNER
body:
  For the moment then I will hard-code a new favicon and leave it to ASGI plugins if people want to define their own.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1604#issuecomment-1016636561
issue_url: https://api.github.com/repos/simonw/datasette/issues/1604
node_id: IC_kwDOBm6k_c48mKSR
created_at: 2022-01-19T16:23:54Z
updated_at: 2022-01-19T16:23:54Z
author_association: OWNER
body:
  Potential design:

      datasette publish cloudrun data.db \
        --service my-service \
        --domain demo.datasette.io

  I think I'm OK with calling this `--domain` even when it is being used with a subdomain.

  This will require `datasette.io` to already have been verified with Google.

  Not sure how best to handle the DNS part - maybe print out instructions for the necessary CNAME?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1601#issuecomment-1016637722
issue_url: https://api.github.com/repos/simonw/datasette/issues/1601
node_id: IC_kwDOBm6k_c48mKka
created_at: 2022-01-19T16:25:07Z
updated_at: 2022-01-19T16:25:07Z
author_association: OWNER
body:
  Good idea - though I'm nervous about accidentally hiding a `data_licenses` table outside of the SpatiaLite case. I'll only hide that one if SpatiaLite is loaded.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1601#issuecomment-1016639294
issue_url: https://api.github.com/repos/simonw/datasette/issues/1601
node_id: IC_kwDOBm6k_c48mK8-
created_at: 2022-01-19T16:26:44Z
updated_at: 2022-01-20T03:57:17Z
author_association: OWNER
body:
  I need to add `sqlite_stat1` to the hidden tables too, see:
  - https://github.com/simonw/datasette/issues/1587
  - https://github.com/simonw/sqlite-utils/issues/366

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1601#issuecomment-1016651485
issue_url: https://api.github.com/repos/simonw/datasette/issues/1601
node_id: IC_kwDOBm6k_c48mN7d
created_at: 2022-01-19T16:39:03Z
updated_at: 2022-01-19T16:39:03Z
author_association: CONTRIBUTOR
body:
  I think both of these are Spatialite specific. They get generated when you first initialize the extension. KNN is actually deprecated in favor of [KNN2](https://www.gaia-gis.it/fossil/libspatialite/wiki?name=KNN2), as I understand it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1605#issuecomment-1016977725
issue_url: https://api.github.com/repos/simonw/datasette/issues/1605
node_id: IC_kwDOBm6k_c48ndk9
created_at: 2022-01-19T23:55:08Z
updated_at: 2022-01-19T23:55:08Z
author_association: OWNER
body:
  Oh that's interesting. I was thinking about this from a slightly different angle recently - pondering what a static site generator built on top of Datasette might look like.

  Just a sketch at the moment, but I was imagining a YAML configuration file with a SQL query that returns a list of paths - then a tool that runs that query and uses the equivalent of `datasette --get` to create a static copy of each of those paths.

  I think these two ideas can probably be merged. I'd love to know more about how you are solving this right now!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1605#issuecomment-1016994329
issue_url: https://api.github.com/repos/simonw/datasette/issues/1605
node_id: IC_kwDOBm6k_c48nhoZ
created_at: 2022-01-20T00:27:17Z
updated_at: 2022-01-20T00:27:17Z
author_association: CONTRIBUTOR
body:
  Right now, I usually have a line in a Makefile like this:

  ```make
  combined.geojson: project.db
      pipenv run datasette project.db --get /project/combined.geojson \
       --load-extension spatialite \
       --setting sql_time_limit_ms 5000 \
       --setting max_returned_rows 20000 \
       -m metadata.yml > $@
  ```

  That all assumes I've loaded whatever I need into `project.db` and created a canned query called `combined` (and then uses `datasette-geojson` for geojson output). 

  It works, but as you can see, it's a lot to manage, a lot of boilerplate, and it wasn't obvious how to get there. If there's an error in the canned query, I get an HTML error page, so that's hard to debug. And it's only one query, so each output needs a line like this. Make isn't ideal, either, for that reason.

  The thing I really liked with `datafreeze` was doing templated filenames. I have a project now where I need to export a bunch of litttle geojson files, based on queries, and it would be awesome to be able to do something like this:

  ```yml
  databases:
    project:
      queries:
        boundaries:
          sql: "SELECT * FROM boundaries"
          filename: "boundaries/{id}.geojson"
          mode: "item"
          format: geojson
  ```

  And then do:

  ```sh
  datasette freeze -m metadata.yml project.db
  ```

  For HTML export, maybe there's a `template` argument, or `format: template` or something. And that gets you a static site generator, kinda for free.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1356#issuecomment-1017016553
issue_url: https://api.github.com/repos/simonw/datasette/issues/1356
node_id: IC_kwDOBm6k_c48nnDp
created_at: 2022-01-20T01:06:37Z
updated_at: 2022-01-20T01:06:37Z
author_association: OWNER
body:
  > A problem with this is that if you're using `--query` you likely want ALL of the results - at the moment the only Datasette output type that can stream everything is `.csv` and plugin formats can't handle full streams, see #1062 and #1177.

  I figured out a neat pattern for streaming JSON arrays in this TIL: https://til.simonwillison.net/python/output-json-array-streaming

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1601#issuecomment-1017095821
issue_url: https://api.github.com/repos/simonw/datasette/issues/1601
node_id: IC_kwDOBm6k_c48n6aN
created_at: 2022-01-20T03:56:14Z
updated_at: 2022-01-20T03:56:28Z
author_association: OWNER
body:
  Oh interesting, I hadn't heard about KNN2. It looks like it was added to SpatiaLite on 21st June 2021 in https://www.gaia-gis.it/fossil/libspatialite/info/03786a62cdb4ab17 but the most recent release of SpatiaLite is 5.0.1 from 7th February 2021 - so it's not yet in a release.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1606#issuecomment-1017108291
issue_url: https://api.github.com/repos/simonw/datasette/issues/1606
node_id: IC_kwDOBm6k_c48n9dD
created_at: 2022-01-20T04:24:54Z
updated_at: 2022-01-20T04:24:54Z
author_association: OWNER
body:
  https://github.com/simonw/latest-datasette-with-all-plugins/commit/1e12ffe70be791e3281b41810e837515314c1317 shows that 5 days ago Datasette upgraded from Uvicorn 0.16 to 0.17

  Sure enough, in the changelog for 0.17: https://github.com/encode/uvicorn/blob/0.17.0/CHANGELOG.md

  > - Drop Python 3.6 support ([#1261](https://github.com/encode/uvicorn/pull/1261)) 06/01/22

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1606#issuecomment-1017108960
issue_url: https://api.github.com/repos/simonw/datasette/issues/1606
node_id: IC_kwDOBm6k_c48n9ng
created_at: 2022-01-20T04:26:36Z
updated_at: 2022-01-20T04:26:36Z
author_association: OWNER
body:
  https://pypistats.org/packages/datasette shows Python 3.6 is 0.24% of Datasette downloads.
   
  <img width="941" alt="image" src="https://user-images.githubusercontent.com/9599/150273301-18ec2ec8-f9ce-49a3-b57d-e74826a5c7ae.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1577#issuecomment-1017109401
issue_url: https://api.github.com/repos/simonw/datasette/issues/1577
node_id: IC_kwDOBm6k_c48n9uZ
created_at: 2022-01-20T04:27:34Z
updated_at: 2022-01-20T04:27:34Z
author_association: OWNER
body:
  OK, now that Uvicorn has dropped 3.6 support - see #1606 - I think this decision is easy to make. I'm dropping 3.6.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1577#issuecomment-1017112130
issue_url: https://api.github.com/repos/simonw/datasette/issues/1577
node_id: IC_kwDOBm6k_c48n-ZC
created_at: 2022-01-20T04:33:57Z
updated_at: 2022-01-20T04:33:57Z
author_association: OWNER
body:
  Dropped support for Python 3.6: updated `setup.py`, changed all references in the documentation to say 3.7 is the minimum required version, got rid of the GitHub Actions tests against 3.6.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1576#issuecomment-1017112543
issue_url: https://api.github.com/repos/simonw/datasette/issues/1576
node_id: IC_kwDOBm6k_c48n-ff
created_at: 2022-01-20T04:35:00Z
updated_at: 2022-02-05T04:33:46Z
author_association: OWNER
body:
  I dropped support for Python 3.6 in fae3983c51f4a3aca8335f3e01ff85ef27076fbf so now free to use `contextvars` for this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1601#issuecomment-1017113831
issue_url: https://api.github.com/repos/simonw/datasette/issues/1601
node_id: IC_kwDOBm6k_c48n-zn
created_at: 2022-01-20T04:38:14Z
updated_at: 2022-01-20T04:38:14Z
author_association: OWNER
body:
  I don't have solid tests in place for exercising SpatiaLite, but this change feels safe enough that I'm not going to add tests for it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1603#issuecomment-1017131209
issue_url: https://api.github.com/repos/simonw/datasette/issues/1603
node_id: IC_kwDOBm6k_c48oDDJ
created_at: 2022-01-20T05:22:40Z
updated_at: 2022-01-20T05:22:40Z
author_association: OWNER
body:
  This one is 101KB and 1536*1536 which is a bit big! https://github.com/simonw/datasette-app/blob/main/build/icon.png

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1603#issuecomment-1017131801
issue_url: https://api.github.com/repos/simonw/datasette/issues/1603
node_id: IC_kwDOBm6k_c48oDMZ
created_at: 2022-01-20T05:23:57Z
updated_at: 2022-01-20T05:23:57Z
author_association: OWNER
body:
  https://adamj.eu/tech/2022/01/18/how-to-add-a-favicon-to-your-django-site/ suggests 64x64, I'm going with 128x128 just in case anyone invents a retina-retina screen.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1603#issuecomment-1017132927
issue_url: https://api.github.com/repos/simonw/datasette/issues/1603
node_id: IC_kwDOBm6k_c48oDd_
created_at: 2022-01-20T05:26:29Z
updated_at: 2022-01-20T05:26:29Z
author_association: OWNER
body:
  Here's the 128x128 one - 11kb, I resized it using Preview:

  ![icon](https://user-images.githubusercontent.com/9599/150278798-48cc2da2-3640-414d-a440-20c9d93c09f4.png)

  Now running it through [Squoosh](https://squoosh.app/editor) using OxiPNG effort=3 colours=24 - brought it down to 1.36KB.

  ![favicon](https://user-images.githubusercontent.com/9599/150278906-b312940e-0cdb-43ef-a325-45d2b1c240de.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1603#issuecomment-1017133357
issue_url: https://api.github.com/repos/simonw/datasette/issues/1603
node_id: IC_kwDOBm6k_c48oDkt
created_at: 2022-01-20T05:27:34Z
updated_at: 2022-01-20T05:27:34Z
author_association: OWNER
body:
  I'm going to drop it in `datasette/static/favicon.png` and rewrite the `async def favicon()` function to return it, with caching headers.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1603#issuecomment-1017136897
issue_url: https://api.github.com/repos/simonw/datasette/issues/1603
node_id: IC_kwDOBm6k_c48oEcB
created_at: 2022-01-20T05:36:29Z
updated_at: 2022-01-20T05:36:29Z
author_association: OWNER
body:
  Here's what it looks like in Firefox, Chrome and Safari:

  ![CleanShot 2022-01-19 at 21 35 06@2x](https://user-images.githubusercontent.com/9599/150279832-6c233115-6540-4746-8cd1-dce25321ebbf.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1603#issuecomment-1017137117
issue_url: https://api.github.com/repos/simonw/datasette/issues/1603
node_id: IC_kwDOBm6k_c48oEfd
created_at: 2022-01-20T05:37:05Z
updated_at: 2022-01-20T05:37:34Z
author_association: OWNER
body:
  I'm not crazy about the look - I think this version of the grid may have too many lines for this particular display size.

  I'm going to try reducing the number of lines in Figma to see if I like that better. https://www.figma.com/file/LKjceTFNtKm6wCbScDqm1Y/Datasette-Logo

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1603#issuecomment-1017139321
issue_url: https://api.github.com/repos/simonw/datasette/issues/1603
node_id: IC_kwDOBm6k_c48oFB5
created_at: 2022-01-20T05:43:07Z
updated_at: 2022-01-20T05:46:42Z
author_association: OWNER
body:
  My attempt with one less grid line looked awful when shrunk down to the browser size:

  <img width="350" alt="image" src="https://user-images.githubusercontent.com/9599/150280445-a482de8e-8b5f-4145-8567-36ac7105c303.png">

  File here:

  ![favicon](https://user-images.githubusercontent.com/9599/150280463-817a8065-8393-4c10-aec4-5a627cf46fae.png)

  I'm going with the first attempt for the moment.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1603#issuecomment-1017142395
issue_url: https://api.github.com/repos/simonw/datasette/issues/1603
node_id: IC_kwDOBm6k_c48oFx7
created_at: 2022-01-20T05:50:55Z
updated_at: 2022-01-20T05:50:55Z
author_association: OWNER
body:
  The new `/favicon.ico` view: https://github.com/simonw/datasette/blob/0467723ee57c2cbc0f02daa47cef632dd4651df0/datasette/app.py#L182-L192

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1603#issuecomment-1017143437
issue_url: https://api.github.com/repos/simonw/datasette/issues/1603
node_id: IC_kwDOBm6k_c48oGCN
created_at: 2022-01-20T05:53:17Z
updated_at: 2022-01-20T05:53:17Z
author_association: OWNER
body:
  New favicon now live on https://latest.datasette.io/ - see also https://latest.datasette.io/favicon.ico

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1603#issuecomment-1017149828
issue_url: https://api.github.com/repos/simonw/datasette/issues/1603
node_id: IC_kwDOBm6k_c48oHmE
created_at: 2022-01-20T06:07:31Z
updated_at: 2022-01-20T06:07:31Z
author_association: OWNER
body:
  Now live on https://latest.datastte.io/ - I'm happy enough with this for the moment:

  <img width="364" alt="image" src="https://user-images.githubusercontent.com/9599/150283069-dd23e688-ec77-4200-a892-38e0f8892f30.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1603#issuecomment-1017800506
issue_url: https://api.github.com/repos/simonw/datasette/issues/1603
node_id: IC_kwDOBm6k_c48qmc6
created_at: 2022-01-20T18:31:18Z
updated_at: 2022-01-20T18:31:18Z
author_association: OWNER
body:
  One last go at tidying this up. I decided to do a 32x32 pixel version in Pixelmator, using this trick to access a pixel brush: https://osxdaily.com/2016/11/17/enable-pixel-brush-pixelmator-mac/

  <img width="916" alt="image" src="https://user-images.githubusercontent.com/9599/150399724-10b17a97-3ecf-4d82-868c-f8a642f16838.png">

  Frustrating how the white boxes are all exactly four pixels high and ALMOST all four pixels wide, but one of them has to be three pixels wide to fit the space.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1603#issuecomment-1017806497
issue_url: https://api.github.com/repos/simonw/datasette/issues/1603
node_id: IC_kwDOBm6k_c48qn6h
created_at: 2022-01-20T18:39:27Z
updated_at: 2022-01-20T18:39:57Z
author_association: OWNER
body:
  Here's a comparison between my hand-edited version and the one I have now:

  ![CleanShot 2022-01-20 at 10 38 00@2x](https://user-images.githubusercontent.com/9599/150401244-7e78ee93-1973-4c95-8f91-01e51e0d5366.png)

  The new 32x32 image:

  <img src="https://user-images.githubusercontent.com/9599/150401295-8873fa40-8859-40e3-99d7-e53276319a15.png" width="32" height="32">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1603#issuecomment-1017808898
issue_url: https://api.github.com/repos/simonw/datasette/issues/1603
node_id: IC_kwDOBm6k_c48qogC
created_at: 2022-01-20T18:42:35Z
updated_at: 2022-01-20T18:42:35Z
author_association: OWNER
body:
  Resized it down to 208 bytes with https://squoosh.app

  <img width="883" alt="image" src="https://user-images.githubusercontent.com/9599/150401693-d19c7fdf-7f5d-49b4-8f1e-5c9e62cbdbed.png">

  ![favicon](https://user-images.githubusercontent.com/9599/150401709-11a50492-a8c8-4eee-848c-c813f8cd3e4e.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1608#issuecomment-1017969452
issue_url: https://api.github.com/repos/simonw/datasette/issues/1608
node_id: IC_kwDOBm6k_c48rPss
created_at: 2022-01-20T22:07:55Z
updated_at: 2022-01-20T22:07:55Z
author_association: OWNER
body:
  I think I want a banner at the top of the page making it obvious which version the documentation is talking about. This can be pretty low key for the current stable release, but should be visually more prominent for the `/latest/` branch and for older releases.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1608#issuecomment-1017970132
issue_url: https://api.github.com/repos/simonw/datasette/issues/1608
node_id: IC_kwDOBm6k_c48rP3U
created_at: 2022-01-20T22:08:55Z
updated_at: 2022-01-20T22:08:55Z
author_association: OWNER
body:
  Might want to consider SEO here too - I want people from search engines to land on `/stable/`, I wonder if I should noindex or `rel=canonical` the other documentation versions? Not sure what best practices for that is.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1608#issuecomment-1017971905
issue_url: https://api.github.com/repos/simonw/datasette/issues/1608
node_id: IC_kwDOBm6k_c48rQTB
created_at: 2022-01-20T22:11:40Z
updated_at: 2022-01-20T22:11:40Z
author_association: OWNER
body:
  Huh, I had forgotten I already have a banner on older versions:

  ![D1A65C68-9A37-4FA2-80C4-534739A9D292](https://user-images.githubusercontent.com/9599/150430410-1e22e23f-ed27-4271-89ff-63467eb5f466.jpeg)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1608#issuecomment-1017975322
issue_url: https://api.github.com/repos/simonw/datasette/issues/1608
node_id: IC_kwDOBm6k_c48rRIa
created_at: 2022-01-20T22:17:01Z
updated_at: 2022-01-20T22:27:07Z
author_association: OWNER
body:
  Turns out that banner is something that ReadTheDocs implemented - I found it using GitHub code search, it's produced by this piece of JavaScript: https://github.com/readthedocs/readthedocs.org/blob/0852d7c10d725d954d3e9a93513171baa1116d9f/readthedocs/core/static-src/core/js/doc-embed/version-compare.js#L13-L21

  ```javascript
  function init(data) {
      var rtd = rtddata.get();

      /// Out of date message

      if (data.is_highest) {
          return;
      }

      var currentURL = window.location.pathname.replace(rtd['version'], data.slug);
      var warning = $(
          '<div class="admonition warning"> ' +
          '<p class="first admonition-title">Note</p> ' +
          '<p class="last"> ' +
          'You are not reading the most recent version of this documentation. ' +
          '<a href="#"></a> is the latest version available.' +
          '</p>' +
          '</div>');

      warning
        .find('a')
        .attr('href', currentURL)
        .text(data.slug);

      var body = $("div.body");
      if (!body.length) {
          body = $("div.document");
      }
      body.prepend(warning);
  }
  ```

  And here's where that module is called from the rest of their code: https://github.com/readthedocs/readthedocs.org/blob/bc3e147770e5740314a8e8c33fec5d111c850498/readthedocs/core/static-src/core/js/doc-embed/footer.js#L66-L86

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1608#issuecomment-1017981599
issue_url: https://api.github.com/repos/simonw/datasette/issues/1608
node_id: IC_kwDOBm6k_c48rSqf
created_at: 2022-01-20T22:26:32Z
updated_at: 2022-01-20T22:26:32Z
author_association: OWNER
body:
  I'm tempted to imitate their JavaScript but check for `/latest/` in the URL and use that to append a similar message warning about this being the documentation for the in-development version.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1608#issuecomment-1017988556
issue_url: https://api.github.com/repos/simonw/datasette/issues/1608
node_id: IC_kwDOBm6k_c48rUXM
created_at: 2022-01-20T22:37:51Z
updated_at: 2022-01-20T22:37:51Z
author_association: OWNER
body:
  Here's a jQuery recipe that seems to do the right thing:
  ```javascript
  jQuery(function ($) {
    // If this is a /latest/ doc page, show banner linking to /stable/
    if (!/\/latest\//.test(location.pathname)) {
      return;
    }
    var stableUrl = location.pathname.replace("/latest/", "/stable/");
    // Check it's not a 404
    fetch(stableUrl, { method: "HEAD" }).then((response) => {
      if (response.status == 200) {
        var warning = $(
          `<div class="admonition warning">
             <p class="first admonition-title">Note</p>
             <p class="last">
               This documentation covers the <strong>development version</strong> of Datasette.</p>
               <p>See <a href="${stableUrl}">this page</a> for the current stable release.
             </p>
          </div>`
        );
        warning.find("a").attr("href", stableUrl);
        var body = $("div.body");
        if (!body.length) {
          body = $("div.document");
        }
        body.prepend(warning);
      }
    });
  });
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1608#issuecomment-1017994925
issue_url: https://api.github.com/repos/simonw/datasette/issues/1608
node_id: IC_kwDOBm6k_c48rV6t
created_at: 2022-01-20T22:48:43Z
updated_at: 2022-01-20T22:54:02Z
author_association: OWNER
body:
  https://sphinx-version-warning.readthedocs.io/ looks like it can show a banner for "You are looking at v0.36 but you should be looking at 0.40" but doesn't hand the case I need here which is "you are looking at /latest/ but you should be looking at /stable/".

  Just shipped my fix here: https://docs.datasette.io/en/latest/

  <img width="994" alt="image" src="https://user-images.githubusercontent.com/9599/150434433-e2f0d472-77a8-430b-88e5-f22e2c6e8ef0.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1608#issuecomment-1017998993
issue_url: https://api.github.com/repos/simonw/datasette/issues/1608
node_id: IC_kwDOBm6k_c48rW6R
created_at: 2022-01-20T22:56:00Z
updated_at: 2022-01-20T22:56:00Z
author_association: OWNER
body:
  > https://sphinx-version-warning.readthedocs.io/ looks like it can show a banner for "You are looking at v0.36 but you should be looking at 0.40" but doesn't hand the case I need here which is "you are looking at /latest/ but you should be looking at /stable/".

  Correction! That tool DOES support that, as can be seen in their example configuration for their own documentation:

  https://github.com/humitos/sphinx-version-warning/blob/a82156c2ea08e5feab406514d0ccd9d48a345f48/docs/conf.py#L32-L38

  ```python
  versionwarning_messages = {
      'latest': 'This is a custom message only for version "latest" of this documentation.',
  }
  versionwarning_admonition_type = 'tip'
  versionwarning_banner_title = 'Tip'
  versionwarning_body_selector = 'div[itemprop="articleBody"]'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1608#issuecomment-1018017637
issue_url: https://api.github.com/repos/simonw/datasette/issues/1608
node_id: IC_kwDOBm6k_c48rbdl
created_at: 2022-01-20T23:27:59Z
updated_at: 2022-01-20T23:27:59Z
author_association: OWNER
body:
  Got a couple of TILs out of this:

  - [Promoting the stable version of the documentation using rel=canonical](https://til.simonwillison.net/readthedocs/documentation-seo-canonical)
  - [Linking from /latest/ to /stable/ on Read The Docs](https://til.simonwillison.net/readthedocs/link-from-latest-to-stable)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1609#issuecomment-1018063681
issue_url: https://api.github.com/repos/simonw/datasette/issues/1609
node_id: IC_kwDOBm6k_c48rmtB
created_at: 2022-01-21T00:58:25Z
updated_at: 2022-01-21T00:58:32Z
author_association: OWNER
body:
  On Twitter: https://twitter.com/simonw/status/1484317711672877065

  Here's the problem: Uvicorn only added `python_requires` to their `setup.py` a few days ago, which means the releases they have out on PyPI at the moment don't specify the Python version they need, which is why this mechanism doesn't work as expected:
  - https://github.com/encode/uvicorn/pull/1328

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1609#issuecomment-1018064620
issue_url: https://api.github.com/repos/simonw/datasette/issues/1609
node_id: IC_kwDOBm6k_c48rm7s
created_at: 2022-01-21T01:00:12Z
updated_at: 2022-01-21T01:00:12Z
author_association: OWNER
body:
  I think there are two possible solutions then:

  1. Convince Uvicorn to publish one last `0.16.1` version which includes that `python_requires=` line, such that there's a version of Uvicorn on PyPI that Python 3.6 can still install.
  2. Release a `0.60.1` version of Datasette which pins that Uvicorn version, and hence can be installed.

  I've made the request for 1) in Uvicorn Gitter here: https://gitter.im/encode/community?at=61ea044a6d9ba23328d0fa28

  I'm going to investigate option 2) myself now.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1609#issuecomment-1018075357
issue_url: https://api.github.com/repos/simonw/datasette/issues/1609
node_id: IC_kwDOBm6k_c48rpjd
created_at: 2022-01-21T01:20:56Z
updated_at: 2022-01-21T01:20:56Z
author_association: OWNER
body:
  I used the combo of `pyenv` and `pipenv` to run tests and figure out what the most recent versions of each dependency were that worked on Python 3.6. I also clicked around in the latest releases on pages such as https://pypi.org/project/aiofiles

  ```
  cd /tmp
  git clone git@github.com:simonw/datasette
  cd /tmp/datasette
  pipenv shell --python 3.6.10
  pip install -e '.[test]'
  pytest
  ```
  I also used `pip freeze | grep black` to see which version was installed, since packages with `python_requires=` in them would automatically install the highest compatible version.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1609#issuecomment-1018077009
issue_url: https://api.github.com/repos/simonw/datasette/issues/1609
node_id: IC_kwDOBm6k_c48rp9R
created_at: 2022-01-21T01:24:15Z
updated_at: 2022-01-21T01:24:43Z
author_association: OWNER
body:
  Problem: if I ship this, it will be the most recent release of Datasette - but unlike other previous releases it has exactly pinned versions of all of the dependencies. Which is bad for people who run `pip install datasette` but want to not be stuck to those exact library versions.

  So maybe I ship this as 0.60.1, then ship a 0.60.2 release directly afterwards which unpins the dependencies again and requires Python 3.7?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1609#issuecomment-1018082792
issue_url: https://api.github.com/repos/simonw/datasette/issues/1609
node_id: IC_kwDOBm6k_c48rrXo
created_at: 2022-01-21T01:37:11Z
updated_at: 2022-01-21T01:37:11Z
author_association: OWNER
body:
  Another option from https://twitter.com/samuel_hames/status/1484327636860293121 - environment markers, described in https://www.python.org/dev/peps/pep-0508/#environment-markers

  Found some examples of those in use using GitHub code search: https://cs.github.com/?scopeName=All+repos&scope=&q=%22%3Bpython_version%22+path%3Asetup.py - in particular https://github.com/xmendez/wfuzz/blob/1b695ee9a87d66a7d7bf6cae70d60a33fae51541/setup.py#L31-L38

  ```python
  install_requires = [
      'pycurl',
      'pyparsing<2.4.2;python_version<="3.4"',
      'pyparsing>=2.4*;python_version>="3.5"',
      'six',
      'configparser;python_version<"3.5"',
      'chardet',
  ]
  ```
  So maybe I can ship 0.60.1 with loose dependencies _except_ for the `uvicorn` one on Python 3.6, using an environment marker.

  Here's my `setup.py` at the moment: https://github.com/simonw/datasette/blob/ffca55dfd7cc9b53522c2e5a2fa1ff67c9beadf2/setup.py#L44-L61

  One other problem: there might be packages in that list right now which don't specify their 3.6 Python version but which will, at some point in the future, release a new version that doesn't work with 3.6 (like Uvicorn did) - in which case Python 3.6 installs would break in the future.

  I think what I'll do then is ship the `0.60.1` Python 3.6 version with strict upper limits on each version which are the current, tested-with-Datasette-on-Python3.6 versions.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1609#issuecomment-1018086273
issue_url: https://api.github.com/repos/simonw/datasette/issues/1609
node_id: IC_kwDOBm6k_c48rsOB
created_at: 2022-01-21T01:45:46Z
updated_at: 2022-01-21T01:45:46Z
author_association: OWNER
body:
  This whole thing reminds me of my ongoing internal debate about version pinning: should the Datasette package released to PyPI pin to the exact versions of the dependencies that are known to work, or should it allow a range of dependencies so users can pick other versions of the dependencies to use in their environment?

  As I understand it, the general rule is to use exact pinning for applications but use ranges for libraries.

  Datasette is almost entirely an application... but it can also be used as a library - and in fact I'm hoping to encourage that usage more in the future, see:
  - #1398

  I'd also like to release a packaged version of Datasette that doesn't require Uvicorn, for running on AWS Lambda and other function-as-a-service platforms. Those platforms have their own HTTP layer and hence don't need the Uvicorn dependency.

  Maybe the answer is to have a `datasette-core` package which provides the core of Datasette with unpinned dependencies and no Uvicorn, and then have the existing `datasette` package provide the Datasette CLI tool with Uvicorn and pinned dependencies?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1609#issuecomment-1018086697
issue_url: https://api.github.com/repos/simonw/datasette/issues/1609
node_id: IC_kwDOBm6k_c48rsUp
created_at: 2022-01-21T01:46:43Z
updated_at: 2022-01-21T01:46:43Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/runs/4890775227?check_suite_focus=true - the tests passed on Python 3.6 for this commit with the pinned dependencies: https://github.com/simonw/datasette/commit/41060e7e7cb838328c879de6a98ae794dc1886d0

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1609#issuecomment-1018092984
issue_url: https://api.github.com/repos/simonw/datasette/issues/1609
node_id: IC_kwDOBm6k_c48rt24
created_at: 2022-01-21T02:00:38Z
updated_at: 2022-01-21T02:00:38Z
author_association: OWNER
body:
  Out of curiosity, I installed this latest `setup.py` file using both Python 3.6 and Python 3.10, ran `pip freeze` on both of them and created a Gist to compare the difference. The result is here: https://gist.github.com/simonw/2e7d5b1beba675ef9a5bcd310cadc372/revisions

  From that, it looks like the Python packages in my dependencies which have released new versions that don't work with Python 3.6 are:

  - https://pypi.org/project/janus/#history
  - https://pypi.org/project/Pint/#history
  - https://pypi.org/project/platformdirs/#history
  - https://pypi.org/project/uvicorn/#history (already discussed)

  Sure enough, for the first three of those browsing through their recent versions on PyPI confirms that they switched from e.g. "Requires: Python >=3.6" on https://pypi.org/project/janus/0.7.0/ to "Requires: Python >=3.7" on https://pypi.org/project/janus/1.0.0/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1609#issuecomment-1018094767
issue_url: https://api.github.com/repos/simonw/datasette/issues/1609
node_id: IC_kwDOBm6k_c48ruSv
created_at: 2022-01-21T02:04:14Z
updated_at: 2022-01-21T02:04:14Z
author_association: OWNER
body:
  So I don't need to release 0.60.1 AND 0.60.2 after all - I can just release 0.60.1 with a bug fix that it no longer breaks installation for Python 3.6.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1609#issuecomment-1018104868
issue_url: https://api.github.com/repos/simonw/datasette/issues/1609
node_id: IC_kwDOBm6k_c48rwwk
created_at: 2022-01-21T02:24:13Z
updated_at: 2022-01-21T02:24:13Z
author_association: OWNER
body:
  Just shipped 0.60.1 with the fix - and tested that `pip install datasette` does indeed work correctly on Python 3.6.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1605#issuecomment-1018741262
issue_url: https://api.github.com/repos/simonw/datasette/issues/1605
node_id: IC_kwDOBm6k_c48uMIO
created_at: 2022-01-21T18:05:09Z
updated_at: 2022-01-21T18:05:09Z
author_association: CONTRIBUTOR
body:
  Thinking about this more, as well as #1356 and various other tickets related to output formats, I think there's a missing plugin hook for formatting results, separate from `register_output_renderer` (or maybe part of it, depending on #1101). 

  Right now, as I understand it, getting output in any format goes through the normal view stack -- a table, a row or a query -- and so by the time `register_output_renderer` gets it, the results have already been truncated or paginated. What I'd want, I think, is to be able to register ways to format results independent of where those results are sent.

  It's possible this could be done using [`conn.row_factory`](https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.row_factory) (maybe in the `prepare_connection` hook), but I'm not sure that's where it belongs.

  Another option is some kind of registry of serializers, which `register_output_renderer` and other plugin hooks could use. What I'm trying to avoid here is writing a plugin that also needs plugins for formats I haven't thought of yet.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1605#issuecomment-1018766727
issue_url: https://api.github.com/repos/simonw/datasette/issues/1605
node_id: IC_kwDOBm6k_c48uSWH
created_at: 2022-01-21T18:41:21Z
updated_at: 2022-01-21T18:42:03Z
author_association: OWNER
body:
  Yeah I think this all hinges on:
  - #1101 

  Also this comment about streaming full JSON arrays (not just newline-delimited) using [this trick](https://til.simonwillison.net/python/output-json-array-streaming):
  - https://github.com/simonw/datasette/issues/1356#issuecomment-1017016553

  I'm about ready to figure these out, as with so much it's still a little bit blocked on the refactor stuff from:
  - #1518

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1605#issuecomment-1018778667
issue_url: https://api.github.com/repos/simonw/datasette/issues/1605
node_id: IC_kwDOBm6k_c48uVQr
created_at: 2022-01-21T19:00:01Z
updated_at: 2022-01-21T19:00:01Z
author_association: CONTRIBUTOR
body:
  Let me know if you want help prototyping any of this, because I'm thinking about it and trying stuff out. Happy to be a sounding board, if it helps.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1609#issuecomment-1020456608
issue_url: https://api.github.com/repos/simonw/datasette/issues/1609
node_id: IC_kwDOBm6k_c480u6g
created_at: 2022-01-24T19:20:09Z
updated_at: 2022-01-24T19:20:09Z
author_association: OWNER
body:
  Uvicorn have a release out now that would have fixed this issue if I hadn't shipped my own fix: https://github.com/encode/uvicorn/releases/tag/0.17.0.post1

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1612#issuecomment-1021472918
issue_url: https://api.github.com/repos/simonw/datasette/issues/1612
node_id: IC_kwDOBm6k_c484nCW
created_at: 2022-01-25T18:14:27Z
updated_at: 2022-01-25T18:15:54Z
author_association: OWNER
body:
  They're currently shown at the very bottom of the page, under the list of tables and far away from the SQL query box: https://latest.datasette.io/fixtures

  <img width="353" alt="image" src="https://user-images.githubusercontent.com/9599/151034640-c3b79214-7fec-4137-b3ed-45541167feff.png">

  I'm also questioning if "Queries" is the best header for this. Other options:

  - **Canned queries** (what the feature is called in the documentation, but I don't think it's a great user-facing term)
  - **Saved queries** - overlaps with a mechanism by which queries can be saved by the user using a plugin such as [datasette-saved-queries](https://github.com/simonw/datasette-saved-queries) - though that plugin does itself use the canned queries plugin hook so not completely unrelated
  - **Sample or Example queries** - I don't like these much because they're more than just examples - they are often the core functionality of the specific customized Datasette instance
  - **Prepared queries** - overlaps with terminology used in other databases, so not great either
  - **Pre-configured queries** - urgh, don't like that language, feels clumsy
  - **Query recipes** - bit out of left-field this one, only really makes sense for queries that include named parameters for specific use-cases

  Maybe "Queries" is right after all.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1612#issuecomment-1021477220
issue_url: https://api.github.com/repos/simonw/datasette/issues/1612
node_id: IC_kwDOBm6k_c484oFk
created_at: 2022-01-25T18:19:31Z
updated_at: 2022-01-25T18:19:31Z
author_association: OWNER
body:
  Here's something I like: I also added a "Tables" `<h2>` heading and bumped the tables themselves down from a `<h2>` to a `<h3>`:

  <img width="909" alt="image" src="https://user-images.githubusercontent.com/9599/151035671-9920db99-a943-46a6-8cd8-7a0ad7524474.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1612#issuecomment-1021489826
issue_url: https://api.github.com/repos/simonw/datasette/issues/1612
node_id: IC_kwDOBm6k_c484rKi
created_at: 2022-01-25T18:34:21Z
updated_at: 2022-01-25T18:34:21Z
author_association: OWNER
body:
  OK, that's live on https://latest.datasette.io/fixtures now - I really like it. Thanks for the suggestion!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1021784593
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/388
node_id: IC_kwDOCGYnMM485zIR
created_at: 2022-01-26T01:56:48Z
updated_at: 2022-01-26T01:56:48Z
author_association: OWNER
body:
  On https://readthedocs.org/dashboard/sqlite-utils/advanced/ I checked this box (previously unchecked):

  ![CleanShot 2022-01-25 at 17 56 22@2x](https://user-images.githubusercontent.com/9599/151090718-72e553a6-826a-43b6-98bd-f70b0cba8784.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1021785268
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/388
node_id: IC_kwDOCGYnMM485zS0
created_at: 2022-01-26T01:58:19Z
updated_at: 2022-01-26T01:58:19Z
author_association: OWNER
body:
  While I'm modifying that template I'm also going to add Plausible analytics:
  ```html
  <script defer data-domain="sqlite-utils.datasette.io" src="https://plausible.io/js/plausible.js"></script>
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1021787822
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/388
node_id: IC_kwDOCGYnMM485z6u
created_at: 2022-01-26T02:03:25Z
updated_at: 2022-01-26T02:03:25Z
author_association: OWNER
body:
  https://sqlite-utils.datasette.io/en/latest/

  <img width="1072" alt="image" src="https://user-images.githubusercontent.com/9599/151091309-30b9ba5f-fc32-4320-ad02-35b31ce0cc87.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/389#issuecomment-1021789016
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/389
node_id: IC_kwDOCGYnMM4850NY
created_at: 2022-01-26T02:05:41Z
updated_at: 2022-01-26T02:05:41Z
author_association: OWNER
body:
  Oops, wrong domain in that - https://plausible.io/sqlite-utils.datasette.io is not yet showing data.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/390#issuecomment-1021825322
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/390
node_id: IC_kwDOCGYnMM4859Eq
created_at: 2022-01-26T03:20:12Z
updated_at: 2022-01-26T03:20:25Z
author_association: OWNER
body:
  ```
  % sqlite-utils upsert trees.db blah -
  Usage: sqlite-utils upsert [OPTIONS] PATH TABLE FILE
  Try 'sqlite-utils upsert -h' for help.

  Error: Missing option '--pk'.
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1613#issuecomment-1021860694
issue_url: https://api.github.com/repos/simonw/datasette/issues/1613
node_id: IC_kwDOBm6k_c486FtW
created_at: 2022-01-26T04:57:53Z
updated_at: 2022-01-26T04:57:53Z
author_association: OWNER
body:
  The existing flow where you can apply filters to a table and then click "View and edit SQL" to see the query is a good starting point.

  Group by queries are both crucially important and difficult to assemble for beginners. Providing a way to see the query that was used by a facet (since facets are really just group-by-counts) would be very useful, which could come out of this:

  - #1080

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/391#issuecomment-1021876055
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/391
node_id: IC_kwDOCGYnMM486JdX
created_at: 2022-01-26T05:15:58Z
updated_at: 2022-01-26T05:15:58Z
author_association: OWNER
body:
  Could add support for `--batch-size` as seen in `insert`/`upsert` too - causing it to break the list up into batches and commit for each one.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/391#issuecomment-1021876463
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/391
node_id: IC_kwDOCGYnMM486Jjv
created_at: 2022-01-26T05:16:51Z
updated_at: 2022-01-26T05:16:51Z
author_association: OWNER
body:
  Actually adding a progress bar may not make sense here: it's designed to work with streaming input from stdin, in which case it's impossible for it to know the overall number of rows to be processed.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/392#issuecomment-1021876914
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/392
node_id: IC_kwDOCGYnMM486Jqy
created_at: 2022-01-26T05:17:49Z
updated_at: 2022-01-26T05:17:49Z
author_association: OWNER
body:
  Relevant code: https://github.com/simonw/sqlite-utils/blob/a9fca7efa4184fbb2a65ca1275c326950ed9d3c1/sqlite_utils/cli.py#L1014-L1018

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/392#issuecomment-1021877769
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/392
node_id: IC_kwDOCGYnMM486J4J
created_at: 2022-01-26T05:19:48Z
updated_at: 2022-01-26T05:19:48Z
author_association: OWNER
body:
  Can use this utility function: https://github.com/simonw/sqlite-utils/blob/a9fca7efa4184fbb2a65ca1275c326950ed9d3c1/sqlite_utils/utils.py#L322-L325

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1613#issuecomment-1022254258
issue_url: https://api.github.com/repos/simonw/datasette/issues/1613
node_id: IC_kwDOBm6k_c487lyy
created_at: 2022-01-26T14:33:46Z
updated_at: 2022-01-26T14:37:31Z
author_association: OWNER
body:
  Tool for setting up foreign key relationships. It could even verify the relationship before you apply it - checking that every value in the column does indeed correspond to a value in the other table.

  Could also detect and suggest possible ones.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1613#issuecomment-1022255862
issue_url: https://api.github.com/repos/simonw/datasette/issues/1613
node_id: IC_kwDOBm6k_c487mL2
created_at: 2022-01-26T14:35:31Z
updated_at: 2022-01-26T14:37:44Z
author_association: OWNER
body:
  Joins are really hard. A mechanism for constructing them in the table view would help a lot:
  - https://github.com/simonw/datasette/issues/613

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1613#issuecomment-1022257496
issue_url: https://api.github.com/repos/simonw/datasette/issues/1613
node_id: IC_kwDOBm6k_c487mlY
created_at: 2022-01-26T14:37:14Z
updated_at: 2022-01-26T14:37:14Z
author_association: OWNER
body:
  Better contextual help on the SQL editor - like in Django SQL Dashboard which shows all available tables and columns.

  Fancy inline autocomplete would be great too, but that's pretty hard for SQL based on past research.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/392#issuecomment-1022466476
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/392
node_id: IC_kwDOCGYnMM488Zms
created_at: 2022-01-26T18:17:43Z
updated_at: 2022-01-26T18:17:43Z
author_association: OWNER
body:
  Manually tested it like this:
  ```
  # Create database with an empty "lines" table
  sqlite-utils create-table bulk-test.db lines line text
  # Stream records every 0.5s, commit every 5 records
  stream-delay docs/python-api.rst -d 500 | \
    sqlite-utils bulk bulk-test.db 'insert into lines (line) values (:line)' - \
    --lines --batch-size 5
  ```
  Running `datasette bulk-test.db` showed that records would show up about every 2.5s five at a time.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1615#issuecomment-1023997327
issue_url: https://api.github.com/repos/simonw/datasette/issues/1615
node_id: IC_kwDOBm6k_c49CPWP
created_at: 2022-01-28T08:37:36Z
updated_at: 2022-01-28T08:37:36Z
author_association: NONE
body:
  Oops, it feels like this should perhaps be migrated to GitHub Discussions - sorry! I don't think I have the ability to do that 😅

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/352#issuecomment-1024726879
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/352
node_id: IC_kwDOCGYnMM49FBdf
created_at: 2022-01-28T22:59:44Z
updated_at: 2022-01-28T22:59:44Z
author_association: OWNER
body:
  Rough prototype, seems to work:;
  ```diff
  diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py
  index 771d432..431b93e 100644
  --- a/sqlite_utils/cli.py
  +++ b/sqlite_utils/cli.py
  @@ -867,6 +867,12 @@ def insert_upsert_options(*, require_pk=False):
                   ),
                   load_extension_option,
                   click.option("--silent", is_flag=True, help="Do not show progress bar"),
  +                click.option(
  +                    "--extract",
  +                    "extracts",
  +                    multiple=True,
  +                    help="Columns to extract to another table",
  +                ),
               )
           ):
               fn = decorator(fn)
  @@ -906,6 +912,7 @@ def insert_upsert_implementation(
       load_extension=None,
       silent=False,
       bulk_sql=None,
  +    extracts=None,
   ):
       db = sqlite_utils.Database(path)
       _load_extensions(db, load_extension)
  @@ -1008,6 +1015,8 @@ def insert_upsert_implementation(
           extra_kwargs["defaults"] = dict(default)
       if upsert:
           extra_kwargs["upsert"] = upsert
  +    if extracts is not None:
  +        extra_kwargs["extracts"] = extracts
   
       # Apply {"$base64": true, ...} decoding, if needed
       docs = (decode_base64_values(doc) for doc in docs)
  @@ -1117,6 +1126,7 @@ def insert(
       truncate,
       not_null,
       default,
  +    extracts,
   ):
       """
       Insert records from FILE into a table, creating the table if it
  @@ -1174,6 +1184,7 @@ def insert(
               silent=silent,
               not_null=not_null,
               default=default,
  +            extracts=extracts,
           )
       except UnicodeDecodeError as ex:
           raise click.ClickException(UNICODE_ERROR.format(ex))
  @@ -1207,6 +1218,7 @@ def upsert(
       analyze,
       load_extension,
       silent,
  +    extracts,
   ):
       """
       Upsert records based on their primary key. Works like 'insert' but if
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/352#issuecomment-1024727476
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/352
node_id: IC_kwDOCGYnMM49FBm0
created_at: 2022-01-28T23:01:06Z
updated_at: 2022-01-28T23:01:06Z
author_association: OWNER
body:
  Manual test run with that prototype:
  ```
  % echo '{"foo": "bar", "id": 1}' | sqlite-utils insert insert-extract.db rows - --pk id --extract foo
  % sqlite-utils dump insert-extract.db                                                                
  BEGIN TRANSACTION;
  CREATE TABLE [foo] (
     [id] INTEGER PRIMARY KEY,
     [value] TEXT
  );
  INSERT INTO "foo" VALUES(1,'bar');
  CREATE TABLE [rows] (
     [foo] INTEGER REFERENCES [foo]([id]),
     [id] INTEGER PRIMARY KEY
  );
  INSERT INTO "rows" VALUES(1,1);
  CREATE UNIQUE INDEX [idx_foo_value]
      ON [foo] ([value]);
  COMMIT;
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1616#issuecomment-1025732071
issue_url: https://api.github.com/repos/simonw/datasette/issues/1616
node_id: IC_kwDOBm6k_c49I23n
created_at: 2022-01-31T13:20:18Z
updated_at: 2022-01-31T13:20:18Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1616?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1616](https://codecov.io/gh/simonw/datasette/pull/1616?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (4ebe94b) into [main](https://codecov.io/gh/simonw/datasette/commit/2aa686c6554bf6b8230eb5b3019574df6cc99225?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (2aa686c) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1616/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1616?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1616   +/-   ##
  =======================================
    Coverage   92.09%   92.09%           
  =======================================
    Files          34       34           
    Lines        4518     4518           
  =======================================
    Hits         4161     4161           
    Misses        357      357           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1616?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1616?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [2aa686c...4ebe94b](https://codecov.io/gh/simonw/datasette/pull/1616?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1607#issuecomment-1027634490
issue_url: https://api.github.com/repos/simonw/datasette/issues/1607
node_id: IC_kwDOBm6k_c49QHU6
created_at: 2022-02-02T06:44:30Z
updated_at: 2022-02-02T06:44:30Z
author_association: OWNER
body:
  Prototype:
  ```diff
  diff --git a/datasette/app.py b/datasette/app.py
  index 09d7d03..e2a5aea 100644
  --- a/datasette/app.py
  +++ b/datasette/app.py
  @@ -724,6 +724,47 @@ class Datasette:
                       sqlite_extensions[extension] = None
               except Exception:
                   pass
  +        # More details on SpatiaLite
  +        if "spatialite" in sqlite_extensions:
  +            spatialite_details = {}
  +            fns = (
  +                "spatialite_version",
  +                "spatialite_target_cpu",
  +                "rcheck_strict_sql_quoting",
  +                "freexl_version",
  +                "proj_version",
  +                "geos_version",
  +                "rttopo_version",
  +                "libxml2_version",
  +                "HasIconv",
  +                "HasMathSQL",
  +                "HasGeoCallbacks",
  +                "HasProj",
  +                "HasProj6",
  +                "HasGeos",
  +                "HasGeosAdvanced",
  +                "HasGeosTrunk",
  +                "HasGeosReentrant",
  +                "HasGeosOnlyReentrant",
  +                "HasMiniZip",
  +                "HasRtTopo",
  +                "HasLibXML2",
  +                "HasEpsg",
  +                "HasFreeXL",
  +                "HasGeoPackage",
  +                "HasGCP",
  +                "HasTopology",
  +                "HasKNN",
  +                "HasRouting",
  +            )
  +            for fn in fns:
  +                try:
  +                    result = conn.execute("select {}()".format(fn))
  +                    spatialite_details[fn] = result.fetchone()[0]
  +                except Exception:
  +                    pass
  +            sqlite_extensions["spatialite"] = spatialite_details
  +
           # Figure out supported FTS versions
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1611#issuecomment-1027635175
issue_url: https://api.github.com/repos/simonw/datasette/issues/1611
node_id: IC_kwDOBm6k_c49QHfn
created_at: 2022-02-02T06:45:47Z
updated_at: 2022-02-02T06:45:47Z
author_association: OWNER
body:
  Prototype, not sure that this actually works yet:
  ```diff
  diff --git a/datasette/database.py b/datasette/database.py
  index 6ce8721..0c4aec7 100644
  --- a/datasette/database.py
  +++ b/datasette/database.py
  @@ -256,18 +256,26 @@ class Database:
           # Try to get counts for each table, $limit timeout for each count
           counts = {}
           for table in await self.table_names():
  -            try:
  -                table_count = (
  -                    await self.execute(
  -                        f"select count(*) from [{table}]",
  -                        custom_time_limit=limit,
  -                    )
  -                ).rows[0][0]
  -                counts[table] = table_count
  -            # In some cases I saw "SQL Logic Error" here in addition to
  -            # QueryInterrupted - so we catch that too:
  -            except (QueryInterrupted, sqlite3.OperationalError, sqlite3.DatabaseError):
  -                counts[table] = None
  +            print(table.lower())
  +            if table.lower() == "knn":
  +                counts[table] = 0
  +            else:
  +                try:
  +                    table_count = (
  +                        await self.execute(
  +                            f"select count(*) from [{table}]",
  +                            custom_time_limit=limit,
  +                        )
  +                    ).rows[0][0]
  +                    counts[table] = table_count
  +                # In some cases I saw "SQL Logic Error" here in addition to
  +                # QueryInterrupted - so we catch that too:
  +                except (
  +                    QueryInterrupted,
  +                    sqlite3.OperationalError,
  +                    sqlite3.DatabaseError,
  +                ):
  +                    counts[table] = None
           if not self.is_mutable:
               self._cached_table_counts = counts
           return counts
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1576#issuecomment-1027635925
issue_url: https://api.github.com/repos/simonw/datasette/issues/1576
node_id: IC_kwDOBm6k_c49QHrV
created_at: 2022-02-02T06:47:20Z
updated_at: 2022-02-02T06:47:20Z
author_association: OWNER
body:
  Here's what I was hacking around with when I uncovered this problem:
  ```diff
  diff --git a/datasette/views/table.py b/datasette/views/table.py
  index 77fb285..8c57d08 100644
  --- a/datasette/views/table.py
  +++ b/datasette/views/table.py
  @@ -1,3 +1,4 @@
  +import asyncio
   import urllib
   import itertools
   import json
  @@ -615,44 +616,37 @@ class TableView(RowTableShared):
           if request.args.get("_timelimit"):
               extra_args["custom_time_limit"] = int(request.args.get("_timelimit"))
   
  -        # Execute the main query!
  -        results = await db.execute(sql, params, truncate=True, **extra_args)
  -
  -        # Calculate the total count for this query
  -        filtered_table_rows_count = None
  -        if (
  -            not db.is_mutable
  -            and self.ds.inspect_data
  -            and count_sql == f"select count(*) from {table} "
  -        ):
  -            # We can use a previously cached table row count
  -            try:
  -                filtered_table_rows_count = self.ds.inspect_data[database]["tables"][
  -                    table
  -                ]["count"]
  -            except KeyError:
  -                pass
  -
  -        # Otherwise run a select count(*) ...
  -        if count_sql and filtered_table_rows_count is None and not nocount:
  -            try:
  -                count_rows = list(await db.execute(count_sql, from_sql_params))
  -                filtered_table_rows_count = count_rows[0][0]
  -            except QueryInterrupted:
  -                pass
  -
  -        # Faceting
  -        if not self.ds.setting("allow_facet") and any(
  -            arg.startswith("_facet") for arg in request.args
  -        ):
  -            raise BadRequest("_facet= is not allowed")
  +        async def execute_count():
  +            # Calculate the total count for this query
  +            filtered_table_rows_count = None
  +            if (
  +                not db.is_mutable
  +                and self.ds.inspect_data
  +                and count_sql == f"select count(*) from {table} "
  +            ):
  +                # We can use a previously cached table row count
  +                try:
  +                    filtered_table_rows_count = self.ds.inspect_data[database][
  +                        "tables"
  +                    ][table]["count"]
  +                except KeyError:
  +                    pass
  +
  +            if count_sql and filtered_table_rows_count is None and not nocount:
  +                try:
  +                    count_rows = list(await db.execute(count_sql, from_sql_params))
  +                    filtered_table_rows_count = count_rows[0][0]
  +                except QueryInterrupted:
  +                    pass
  +
  +            return filtered_table_rows_count
  +
  +        filtered_table_rows_count = await execute_count()
   
           # pylint: disable=no-member
           facet_classes = list(
               itertools.chain.from_iterable(pm.hook.register_facet_classes())
           )
  -        facet_results = {}
  -        facets_timed_out = []
           facet_instances = []
           for klass in facet_classes:
               facet_instances.append(
  @@ -668,33 +662,58 @@ class TableView(RowTableShared):
                   )
               )
   
  -        if not nofacet:
  -            for facet in facet_instances:
  -                (
  -                    instance_facet_results,
  -                    instance_facets_timed_out,
  -                ) = await facet.facet_results()
  -                for facet_info in instance_facet_results:
  -                    base_key = facet_info["name"]
  -                    key = base_key
  -                    i = 1
  -                    while key in facet_results:
  -                        i += 1
  -                        key = f"{base_key}_{i}"
  -                    facet_results[key] = facet_info
  -                facets_timed_out.extend(instance_facets_timed_out)
  -
  -        # Calculate suggested facets
  -        suggested_facets = []
  -        if (
  -            self.ds.setting("suggest_facets")
  -            and self.ds.setting("allow_facet")
  -            and not _next
  -            and not nofacet
  -            and not nosuggest
  -        ):
  -            for facet in facet_instances:
  -                suggested_facets.extend(await facet.suggest())
  +        async def execute_suggested_facets():
  +            # Calculate suggested facets
  +            suggested_facets = []
  +            if (
  +                self.ds.setting("suggest_facets")
  +                and self.ds.setting("allow_facet")
  +                and not _next
  +                and not nofacet
  +                and not nosuggest
  +            ):
  +                for facet in facet_instances:
  +                    suggested_facets.extend(await facet.suggest())
  +            return suggested_facets
  +
  +        async def execute_facets():
  +            facet_results = {}
  +            facets_timed_out = []
  +            if not self.ds.setting("allow_facet") and any(
  +                arg.startswith("_facet") for arg in request.args
  +            ):
  +                raise BadRequest("_facet= is not allowed")
  +
  +            if not nofacet:
  +                for facet in facet_instances:
  +                    (
  +                        instance_facet_results,
  +                        instance_facets_timed_out,
  +                    ) = await facet.facet_results()
  +                    for facet_info in instance_facet_results:
  +                        base_key = facet_info["name"]
  +                        key = base_key
  +                        i = 1
  +                        while key in facet_results:
  +                            i += 1
  +                            key = f"{base_key}_{i}"
  +                        facet_results[key] = facet_info
  +                    facets_timed_out.extend(instance_facets_timed_out)
  +
  +            return facet_results, facets_timed_out
  +
  +        # Execute the main query, facets and facet suggestions in parallel:
  +        (
  +            results,
  +            suggested_facets,
  +            (facet_results, facets_timed_out),
  +        ) = await asyncio.gather(
  +            db.execute(sql, params, truncate=True, **extra_args),
  +            execute_suggested_facets(),
  +            execute_facets(),
  +        )
  +
  +        results = await db.execute(sql, params, truncate=True, **extra_args)
   
           # Figure out columns and rows for the query
           columns = [r[0] for r in results.description]
  ```
  It's a hacky attempt at running some of the table page queries in parallel to see what happens.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1619#issuecomment-1027646659
issue_url: https://api.github.com/repos/simonw/datasette/issues/1619
node_id: IC_kwDOBm6k_c49QKTD
created_at: 2022-02-02T07:10:37Z
updated_at: 2022-02-02T07:10:37Z
author_association: OWNER
body:
  It's not just the table with slashes in the name. Same thing on http://127.0.0.1:3344/foo/bar/fixtures/attraction_characteristic/1 - the `json` link goes to a JSON-rendered 404 on http://127.0.0.1:3344/foo/bar/foo/bar/fixtures/attraction_characteristic/1.json

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1619#issuecomment-1027647257
issue_url: https://api.github.com/repos/simonw/datasette/issues/1619
node_id: IC_kwDOBm6k_c49QKcZ
created_at: 2022-02-02T07:11:43Z
updated_at: 2022-02-02T07:11:43Z
author_association: OWNER
body:
  Weirdly the bug does NOT exhibit itself on this demo: https://datasette-apache-proxy-demo.datasette.io/prefix/fixtures/no_primary_key/1 - which correctly links to https://datasette-apache-proxy-demo.datasette.io/prefix/fixtures/no_primary_key/1.json

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1586#issuecomment-1027648180
issue_url: https://api.github.com/repos/simonw/datasette/issues/1586
node_id: IC_kwDOBm6k_c49QKq0
created_at: 2022-02-02T07:13:31Z
updated_at: 2022-02-02T07:13:31Z
author_association: OWNER
body:
  Running it as part of `datasette publish` is a smart idea - I'm slightly nervous about modifying the database file that has been published though, since part of the undocumented contract right now is that the bytes served are the exact same bytes as the ones you ran the publish against.

  But there's no reason for that expectation to exist, and I doubt anyone is relying on that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1618#issuecomment-1027653005
issue_url: https://api.github.com/repos/simonw/datasette/issues/1618
node_id: IC_kwDOBm6k_c49QL2N
created_at: 2022-02-02T07:22:13Z
updated_at: 2022-02-02T07:22:13Z
author_association: OWNER
body:
  There's a workaround for this at the moment, which is to use parameterized SQL queries. For example, this:

  https://fivethirtyeight.datasettes.com/polls?sql=select+*+from+books+where+title+%3D+%3Atitle&title=The+Pragmatic+Programmer

  So the SQL query is `select * from books where title = :title` and then `&title=...` is added to the URL.

  The reason behind the quite aggressive pragma filtering is that SQLite allows you to execute pragmas using function calls, like this one:

  ```sql
  SELECT * FROM pragma_index_info('idx52');
  ```
  These can be nested arbitrarily deeply in sub-queries, so it's difficult to write a regular expression that will definitely catch them.

  I'm open to relaxing the regex a bit, but I need to be very confident that it's safe to do so.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1618#issuecomment-1027654979
issue_url: https://api.github.com/repos/simonw/datasette/issues/1618
node_id: IC_kwDOBm6k_c49QMVD
created_at: 2022-02-02T07:25:22Z
updated_at: 2022-02-02T07:25:22Z
author_association: OWNER
body:
  But... I just noticed something I had missed in the docs for https://www.sqlite.org/pragma.html#pragfunc

  > Table-valued functions exist only for PRAGMAs that return results and that have no side-effects.

  So it's possible I'm being overly paranoid here after all: what I want to block here is people running things like `PRAGMA case_sensitive_like = 1` which could affect the global state for that connection and cause unexpected behaviour later on.

  So maybe I should allow all pragma functions. I previously allowed an allow-list of them in:
  - #761

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1618#issuecomment-1027656000
issue_url: https://api.github.com/repos/simonw/datasette/issues/1618
node_id: IC_kwDOBm6k_c49QMlA
created_at: 2022-02-02T07:27:14Z
updated_at: 2022-02-02T07:27:14Z
author_association: OWNER
body:
  I also just realized that `pragma pragma_list` can be used to generate a list of all known pragmas for the connection:

      sqlite-utils fixtures.db 'pragma pragma_list' --fmt github

  | name                      |
  |---------------------------|
  | analysis_limit            |
  | application_id            |
  | auto_vacuum               |
  | automatic_index           |
  | busy_timeout              |
  | cache_size                |
  | cache_spill               |
  | case_sensitive_like       |
  | cell_size_check           |
  | checkpoint_fullfsync      |
  | collation_list            |
  | compile_options           |
  | count_changes             |
  | data_version              |
  | database_list             |
  | default_cache_size        |
  | defer_foreign_keys        |
  | empty_result_callbacks    |
  | encoding                  |
  | foreign_key_check         |
  | foreign_key_list          |
  | foreign_keys              |
  | freelist_count            |
  | full_column_names         |
  | fullfsync                 |
  | function_list             |
  | hard_heap_limit           |
  | ignore_check_constraints  |
  | incremental_vacuum        |
  | index_info                |
  | index_list                |
  | index_xinfo               |
  | integrity_check           |
  | journal_mode              |
  | journal_size_limit        |
  | legacy_alter_table        |
  | lock_proxy_file           |
  | locking_mode              |
  | max_page_count            |
  | mmap_size                 |
  | module_list               |
  | optimize                  |
  | page_count                |
  | page_size                 |
  | pragma_list               |
  | query_only                |
  | quick_check               |
  | read_uncommitted          |
  | recursive_triggers        |
  | reverse_unordered_selects |
  | schema_version            |
  | secure_delete             |
  | short_column_names        |
  | shrink_memory             |
  | soft_heap_limit           |
  | synchronous               |
  | table_info                |
  | table_list                |
  | table_xinfo               |
  | temp_store                |
  | temp_store_directory      |
  | threads                   |
  | trusted_schema            |
  | user_version              |
  | wal_autocheckpoint        |
  | wal_checkpoint            |
  | writable_schema           |

  So I could use that list to create a much more specific regular expression, which would then allow the word "pragma" to be used more freely while still protecting against any known pragma function being called.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1618#issuecomment-1027656518
issue_url: https://api.github.com/repos/simonw/datasette/issues/1618
node_id: IC_kwDOBm6k_c49QMtG
created_at: 2022-02-02T07:28:14Z
updated_at: 2022-02-02T07:31:30Z
author_association: OWNER
body:
  I also need to consider if supposedly harmless side-effect free pragma functions could be used to work around the Datasette permissions system. My hunch is that wouldn't be a problem, because if you're allowing arbitrary SQL queries you're already letting people ignore the permissions system.

  One example:
  ```
  sqlite-utils fixtures.db 'pragma database_list' -t
    seq  name    file
  -----  ------  ------------------------------------------------------
      0  main    /Users/simon/Dropbox/Development/datasette/fixtures.db
  ```
  Though it looks like I already allow-listed that one in #761: https://latest.datasette.io/_memory?sql=select+*+from+pragma_database_list%28%29

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1618#issuecomment-1027659018
issue_url: https://api.github.com/repos/simonw/datasette/issues/1618
node_id: IC_kwDOBm6k_c49QNUK
created_at: 2022-02-02T07:32:47Z
updated_at: 2022-02-02T07:32:47Z
author_association: OWNER
body:
  I was hoping that `explain select ...` might be able to easily spot when people are calling PRAGMA functions, but this output doesn't look very helpful:
  ```
  % sqlite-utils fixtures.db 'explain select * from pragma_database_list()' -t
    addr  opcode         p1    p2    p3  p4                   p5  comment
  ------  -----------  ----  ----  ----  -----------------  ----  ---------
       0  Init            0    11     0                        0
       1  VOpen           0     0     0  vtab:7F9C90AC3070     0
       2  Integer         0     1     0                        0
       3  Integer         0     2     0                        0
       4  VFilter         0    10     1                        0
       5  VColumn         0     0     3                        0
       6  VColumn         0     1     4                        0
       7  VColumn         0     2     5                        0
       8  ResultRow       3     3     0                        0
       9  VNext           0     5     0                        0
      10  Halt            0     0     0                        0
      11  Transaction     0     0    35  0                     1
      12  Goto            0     1     0                        0
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1615#issuecomment-1027659890
issue_url: https://api.github.com/repos/simonw/datasette/issues/1615
node_id: IC_kwDOBm6k_c49QNhy
created_at: 2022-02-02T07:34:17Z
updated_at: 2022-02-02T07:34:17Z
author_association: OWNER
body:
  I've been thinking about this a bunch too. If I build anything along these lines it will be as part of the Datasette Cloud hosted service I'm working on, maybe as a free tier.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1533#issuecomment-1027669851
issue_url: https://api.github.com/repos/simonw/datasette/issues/1533
node_id: IC_kwDOBm6k_c49QP9b
created_at: 2022-02-02T07:51:57Z
updated_at: 2022-02-02T07:51:57Z
author_association: OWNER
body:
  Documentation: https://docs.datasette.io/en/latest/json_api.html#discovering-the-json-for-a-page

  https://docs.datasette.io/en/latest/json_api.html top `--cors` section mentions the new `Access-Control-Expose-Headers: Link` header.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1533#issuecomment-1027672617
issue_url: https://api.github.com/repos/simonw/datasette/issues/1533
node_id: IC_kwDOBm6k_c49QQop
created_at: 2022-02-02T07:56:51Z
updated_at: 2022-02-02T07:56:51Z
author_association: OWNER
body:
  Demos - these pages both have `<link rel=...` if you view source on them:

  - https://latest.datasette.io/fixtures/sortable
  - https://latest.datasette.io/fixtures/sortable/a,a

  And you can hit them with `curl` like so:
  ```
  % curl -I 'https://latest.datasette.io/fixtures/sortable'    
  HTTP/1.1 200 OK
  link: https://latest.datasette.io/fixtures/sortable.json; rel="alternate"; type="application/json+datasette"
  cache-control: max-age=5
  referrer-policy: no-referrer
  access-control-allow-origin: *
  access-control-allow-headers: Authorization
  access-control-expose-headers: Link
  content-type: text/html; charset=utf-8
  x-databases: _memory, _internal, fixtures, extra_database
  Date: Wed, 02 Feb 2022 07:56:17 GMT
  Server: Google Frontend
  Transfer-Encoding: chunked

  % curl -I 'https://latest.datasette.io/fixtures/sortable/a,a'
  HTTP/1.1 200 OK
  link: https://latest.datasette.io/fixtures/sortable/a,a.json; rel="alternate"; type="application/json+datasette"
  cache-control: max-age=5
  referrer-policy: no-referrer
  access-control-allow-origin: *
  access-control-allow-headers: Authorization
  access-control-expose-headers: Link
  content-type: text/html; charset=utf-8
  x-databases: _memory, _internal, fixtures, extra_database
  Date: Wed, 02 Feb 2022 07:56:24 GMT
  Server: Google Frontend
  Transfer-Encoding: chunked
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1618#issuecomment-1028294089
issue_url: https://api.github.com/repos/simonw/datasette/issues/1618
node_id: IC_kwDOBm6k_c49SoXJ
created_at: 2022-02-02T19:42:03Z
updated_at: 2022-02-02T19:42:03Z
author_association: NONE
body:
  Thanks for looking into this. It might have been nice if `explain` surfaced these function calls. Looks like `explain query plan` does, but only for basic queries.

  ```
  sqlite-utils fixtures.db 'explain query plan select * from pragma_function_list(), pragma_database_list(), pragma_module_list()' -t
    id    parent    notused  detail
  ----  --------  ---------  ------------------------------------------------
     4         0          0  SCAN pragma_function_list VIRTUAL TABLE INDEX 0:
     8         0          0  SCAN pragma_database_list VIRTUAL TABLE INDEX 0:
    12         0          0  SCAN pragma_module_list VIRTUAL TABLE INDEX 0:
  ```


  ```
  sqlite-utils fixtures.db 'explain query plan select * from pragma_function_list() as fl, pragma_database_list() as dl, pragma_module_list() as ml' -t
    id    parent    notused  detail
  ----  --------  ---------  ------------------------------
     4         0          0  SCAN fl VIRTUAL TABLE INDEX 0:
     8         0          0  SCAN dl VIRTUAL TABLE INDEX 0:
    12         0          0  SCAN ml VIRTUAL TABLE INDEX 0:
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1620#issuecomment-1028385067
issue_url: https://api.github.com/repos/simonw/datasette/issues/1620
node_id: IC_kwDOBm6k_c49S-kr
created_at: 2022-02-02T21:42:23Z
updated_at: 2022-02-02T21:42:23Z
author_association: OWNER
body:
  ```
  % curl -s -I 'https://latest.datasette.io/' | grep link
  link: https://latest.datasette.io/.json; rel="alternate"; type="application/json+datasette"
  % curl -s -I 'https://latest.datasette.io/fixtures' | grep link
  link: https://latest.datasette.io/fixtures.json; rel="alternate"; type="application/json+datasette"
  % curl -s -I 'https://latest.datasette.io/fixtures?sql=select+1' | grep link
  link: https://latest.datasette.io/fixtures.json?sql=select+1; rel="alternate"; type="application/json+datasette"
  % curl -s -I 'https://latest.datasette.io/-/plugins' | grep link 
  link: https://latest.datasette.io/-/plugins.json; rel="alternate"; type="application/json+datasette"
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1622#issuecomment-1028387529
issue_url: https://api.github.com/repos/simonw/datasette/issues/1622
node_id: IC_kwDOBm6k_c49S_LJ
created_at: 2022-02-02T21:45:21Z
updated_at: 2022-02-02T21:45:21Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1622?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1622](https://codecov.io/gh/simonw/datasette/pull/1622?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (fbaf317) into [main](https://codecov.io/gh/simonw/datasette/commit/8d5779acf0041cfd0db7f68f468419f9008b86ec?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (8d5779a) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1622/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1622?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1622   +/-   ##
  =======================================
    Coverage   92.11%   92.11%           
  =======================================
    Files          34       34           
    Lines        4525     4525           
  =======================================
    Hits         4168     4168           
    Misses        357      357           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1622?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1622?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [8d5779a...fbaf317](https://codecov.io/gh/simonw/datasette/pull/1622?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1623#issuecomment-1028389953
issue_url: https://api.github.com/repos/simonw/datasette/issues/1623
node_id: IC_kwDOBm6k_c49S_xB
created_at: 2022-02-02T21:48:34Z
updated_at: 2022-02-02T21:48:34Z
author_association: OWNER
body:
  A few other pages do that too, including:
  - https://latest.datasette.io/-/messages
  - https://latest.datasette.io/-/allow-debug

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1620#issuecomment-1028393259
issue_url: https://api.github.com/repos/simonw/datasette/issues/1620
node_id: IC_kwDOBm6k_c49TAkr
created_at: 2022-02-02T21:53:02Z
updated_at: 2022-02-02T21:53:02Z
author_association: OWNER
body:
  I ran the following on https://www.google.com/ in the console to demonstrate that these work as intended:

  ```javascript
  [
    "https://latest.datasette.io/fixtures",
    "https://latest.datasette.io/fixtures?sql=select+1",
    "https://latest.datasette.io/fixtures/facetable"
  ].forEach(async (url) => {
    response = await fetch(url, {method: "HEAD"});
    console.log(response.headers.get("Link"));
  });
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1624#issuecomment-1028396866
issue_url: https://api.github.com/repos/simonw/datasette/issues/1624
node_id: IC_kwDOBm6k_c49TBdC
created_at: 2022-02-02T21:58:06Z
updated_at: 2022-02-02T21:58:06Z
author_association: OWNER
body:
  It looks like this is because `IndexView` extends `BaseView` rather than extending `DataView` which is where all that CORS stuff happens:

  https://github.com/simonw/datasette/blob/23a09b0f6af33c52acf8c1d9002fe475b42fee10/datasette/views/index.py#L18-L21

  Another thing I should address with the refactor project in:
  - #878 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1623#issuecomment-1028397935
issue_url: https://api.github.com/repos/simonw/datasette/issues/1623
node_id: IC_kwDOBm6k_c49TBtv
created_at: 2022-02-02T21:59:43Z
updated_at: 2022-02-02T21:59:43Z
author_association: OWNER
body:
  Here's the new test: https://github.com/simonw/datasette/blob/23a09b0f6af33c52acf8c1d9002fe475b42fee10/tests/test_html.py#L927-L936

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1617#issuecomment-1028419517
issue_url: https://api.github.com/repos/simonw/datasette/issues/1617
node_id: IC_kwDOBm6k_c49TG-9
created_at: 2022-02-02T22:30:26Z
updated_at: 2022-02-03T01:36:07Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1617?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1617](https://codecov.io/gh/simonw/datasette/pull/1617?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (af293c9) into [main](https://codecov.io/gh/simonw/datasette/commit/2aa686c6554bf6b8230eb5b3019574df6cc99225?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (2aa686c) will **increase** coverage by `0.06%`.
  > The diff coverage is `100.00%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1617/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1617?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@            Coverage Diff             @@
  ##             main    #1617      +/-   ##
  ==========================================
  + Coverage   92.09%   92.16%   +0.06%     
  ==========================================
    Files          34       34              
    Lines        4518     4531      +13     
  ==========================================
  + Hits         4161     4176      +15     
  + Misses        357      355       -2     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1617?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1617/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `95.37% <100.00%> (ø)` | |
  | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1617/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `96.19% <0.00%> (ø)` | |
  | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1617/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.79% <0.00%> (+<0.01%)` | :arrow_up: |
  | [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1617/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `95.49% <0.00%> (+0.07%)` | :arrow_up: |
  | [datasette/views/special.py](https://codecov.io/gh/simonw/datasette/pull/1617/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3NwZWNpYWwucHk=) | `95.09% <0.00%> (+2.38%)` | :arrow_up: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1617?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1617?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [2aa686c...af293c9](https://codecov.io/gh/simonw/datasette/pull/1617?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1626#issuecomment-1028420821
issue_url: https://api.github.com/repos/simonw/datasette/issues/1626
node_id: IC_kwDOBm6k_c49THTV
created_at: 2022-02-02T22:32:26Z
updated_at: 2022-02-02T22:33:31Z
author_association: OWNER
body:
  That broke on a macOS test: https://github.com/simonw/datasette/runs/5044036993?check_suite_focus=true

  I'm going to remove macOS and Ubuntu and just try Windows purely to see what happens there.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1626#issuecomment-1028423514
issue_url: https://api.github.com/repos/simonw/datasette/issues/1626
node_id: IC_kwDOBm6k_c49TH9a
created_at: 2022-02-02T22:36:37Z
updated_at: 2022-02-02T22:39:52Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1626?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1626](https://codecov.io/gh/simonw/datasette/pull/1626?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (4b4d0e1) into [main](https://codecov.io/gh/simonw/datasette/commit/b5e6b1a9e1332fca3effe45d55dd06ee4249f163?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (b5e6b1a) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1626/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1626?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1626   +/-   ##
  =======================================
    Coverage   92.16%   92.16%           
  =======================================
    Files          34       34           
    Lines        4531     4531           
  =======================================
    Hits         4176     4176           
    Misses        355      355           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1626?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1626?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [b5e6b1a...4b4d0e1](https://codecov.io/gh/simonw/datasette/pull/1626?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1534#issuecomment-1028461220
issue_url: https://api.github.com/repos/simonw/datasette/issues/1534
node_id: IC_kwDOBm6k_c49TRKk
created_at: 2022-02-02T23:39:33Z
updated_at: 2022-02-02T23:39:33Z
author_association: OWNER
body:
  I've decided not to do this, because of the risk that Cloudflare could cache the JSON version for an HTML page or vice-versa.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1626#issuecomment-1028515161
issue_url: https://api.github.com/repos/simonw/datasette/issues/1626
node_id: IC_kwDOBm6k_c49TeVZ
created_at: 2022-02-03T01:22:43Z
updated_at: 2022-02-03T01:22:43Z
author_association: OWNER
body:
  OK, the tests do NOT pass against Windows! https://github.com/simonw/datasette/runs/5044105941

  <img width="767" alt="image" src="https://user-images.githubusercontent.com/9599/152265501-5b5af51b-0fae-4aa7-b86d-7b61aba24b3c.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1617#issuecomment-1028517073
issue_url: https://api.github.com/repos/simonw/datasette/issues/1617
node_id: IC_kwDOBm6k_c49TezR
created_at: 2022-02-03T01:26:32Z
updated_at: 2022-02-03T01:26:32Z
author_association: OWNER
body:
  Aha I understand the problem now! https://github.com/pallets/jinja/issues/1378#issuecomment-812410922

  > Jinja template names/paths are not always filesystem paths. So regardless of the OS Jinja always uses forward slashes.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1545#issuecomment-1028517268
issue_url: https://api.github.com/repos/simonw/datasette/issues/1545
node_id: IC_kwDOBm6k_c49Te2U
created_at: 2022-02-03T01:26:53Z
updated_at: 2022-02-03T01:26:53Z
author_association: OWNER
body:
  I understand the problem now! https://github.com/pallets/jinja/issues/1378#issuecomment-812410922

  > Jinja template names/paths are not always filesystem paths. So regardless of the OS Jinja always uses forward slashes.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1617#issuecomment-1028519382
issue_url: https://api.github.com/repos/simonw/datasette/issues/1617
node_id: IC_kwDOBm6k_c49TfXW
created_at: 2022-02-03T01:31:25Z
updated_at: 2022-02-03T01:31:25Z
author_association: OWNER
body:
  I was hoping to get the test suite running on Windows before merging this PR but that looks like it will be a BIG job, see:
  - #1627

  So I'm going to merge this one as-is for the moment.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1545#issuecomment-1028535868
issue_url: https://api.github.com/repos/simonw/datasette/issues/1545
node_id: IC_kwDOBm6k_c49TjY8
created_at: 2022-02-03T02:08:30Z
updated_at: 2022-02-03T02:08:30Z
author_association: OWNER
body:
  Filed an issue with Jinja suggesting a documentation update:
  - https://github.com/pallets/jinja/issues/1578

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029177700
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/385
node_id: IC_kwDOCGYnMM49WAFk
created_at: 2022-02-03T16:38:45Z
updated_at: 2022-02-04T05:52:39Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/385?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#385](https://codecov.io/gh/simonw/sqlite-utils/pull/385?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (af86b17) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/74586d3cb26fa3cc3412721985ecdc1864c2a31d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (74586d3) will **decrease** coverage by `0.61%`.
  > The diff coverage is `28.00%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/385/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/385?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@            Coverage Diff             @@
  ##             main     #385      +/-   ##
  ==========================================
  - Coverage   96.52%   95.91%   -0.62%     
  ==========================================
    Files           6        6              
    Lines        2389     2421      +32     
  ==========================================
  + Hits         2306     2322      +16     
  - Misses         83       99      +16     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/385?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [sqlite\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/385/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.69% <ø> (+0.15%)` | :arrow_up: |
  | [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/385/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `96.29% <15.00%> (-1.40%)` | :arrow_down: |
  | [sqlite\_utils/utils.py](https://codecov.io/gh/simonw/sqlite-utils/pull/385/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `94.59% <80.00%> (ø)` | |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/385?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/385?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [74586d3...af86b17](https://codecov.io/gh/simonw/sqlite-utils/pull/385?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029273853
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/385
node_id: IC_kwDOCGYnMM49WXj9
created_at: 2022-02-03T18:23:30Z
updated_at: 2022-02-03T18:31:21Z
author_association: OWNER
body:
  OK, this change makes a bunch of sense to me - and also raises some interesting questions about future additions to `sqlite-utils` with regards to SpatiaLite. Would `sqlite-utils add-geometry-column ...` be a good CLI enhancement. for example?

  I see you've already talked about that in #79 - moving this conversation there!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029285985
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/385
node_id: IC_kwDOCGYnMM49Wahh
created_at: 2022-02-03T18:37:48Z
updated_at: 2022-02-03T18:37:48Z
author_association: OWNER
body:
  `from sqlite_utils.utils import find_spatialite` is part of the documented API already:

  https://sqlite-utils.datasette.io/en/3.22.1/python-api.html#finding-spatialite

  To avoid needing to bump the major version number to 4 to indicate a backwards incompatible change, we should keep a `from .gis import find_spatialite` line at the top of `utils.py` such that any existing code with that documented import continues to work.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029296782
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/385
node_id: IC_kwDOCGYnMM49WdKO
created_at: 2022-02-03T18:51:21Z
updated_at: 2022-02-03T18:51:21Z
author_association: OWNER
body:
  What do you think about adding these as methods on the `Database` class instead? Then you could do:

  ```python
  # This is with an optional argument, which if omitted runs find_spatialite() for you:
  db.init_spatialite()

  # Instead of:
  init_spatialite(db, find_spatialite())
  ```
  Likewise, the `add_geometry_column` and `create_spatial_index` methods could live on `Table`:
  ```python
  # Instead of this:
  add_geometry_column(db["locations"], "POINT", "geometry")
  create_spatial_index(db["locations"], "geometry")

  # Could have this:
  db["locations"].add_geometry_column("POINT")
  db["locations"].create_spatial_index("geometry")
  ```
  On the one hand, this is much more consistent with the existing `sqlite-utils` Python API.

  But on the other hand... this is mixing SpatiaLite functionality directly into the core classes. Is that a good idea, seeing as SpatiaLite is both an optional extension (which can be tricky to install) AND something that has a very different release cadence and quality-of-documentation from SQLite itself?

  There's a third option: the SpatiaLite could exist on subclasses of `Database` and `Table` - so the above examples would look something like this:

  ```python
  from sqlite_utils.gis import SpatiaLiteDatabase

  db = SpatiaLiteDatabase("geo.db")
  db.init_spatialite()
  db["locations"].add_geometry_column("POINT")
  db["locations"].create_spatial_index("geometry")
  ```

  On the one hand, this would keep the SpatiaLite-specific stuff out of the core Database/Table classes. But it feels a bit untidy to me, especially since it raises the spectre of someone who was already subclassing Database for some reason now needing to instead subclass `SpatiaLiteDatabase` (not too keen on that capitalization) - or even (horror) trying to dabble with multiple inheritance, which can only lead to pain.

  So I don't have a strong opinion formed on this question yet!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029297971
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/385
node_id: IC_kwDOCGYnMM49Wdcz
created_at: 2022-02-03T18:52:50Z
updated_at: 2022-02-03T18:52:50Z
author_association: OWNER
body:
  I'm not sure I like `name="geometry"` as the default argument to `add_geometry_column` - mainly because of this example here:
  ```python
  add_geometry_column(db["locations"], "POINT")
  create_spatial_index(db["locations"], "geometry")
  ```
  I had to go and look at the code to figure out if `"POINT"` was the name of the column - and I don't like how inconsistent it looks next to the following `create_spatial_index()` call where you DO need to pass the column name.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029306428
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/385
node_id: IC_kwDOCGYnMM49Wfg8
created_at: 2022-02-03T19:03:43Z
updated_at: 2022-02-03T19:03:43Z
author_association: CONTRIBUTOR
body:
  I thought about adding these as methods on `Database` and `Table`, and I'm back and forth on it for the same reasons you are. It's certainly cleaner, and it's clearer what you're operating on. I could go either way. 

  I do sort of like having all the Spatialite stuff in its own module, just because it's built around an extension you might not have or want, but I don't know if that's a good reason to have a different API.

  You could have `init_spatialite` add methods to `Database` and `Table`, so they're only there if you have Spatialite set up. Is that too clever? It feels too clever.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1029317527
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/79
node_id: IC_kwDOCGYnMM49WiOX
created_at: 2022-02-03T19:18:02Z
updated_at: 2022-02-03T19:18:02Z
author_association: CONTRIBUTOR
body:
  Taking part of the conversation from #385 here.

  > Would sqlite-utils add-geometry-column ... be a good CLI enhancement. for example?

  Yes. And also `sqlite-utils create-spatial-index` would be great to have. My plan would be to add those once the Python API is settled.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029326568
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/385
node_id: IC_kwDOCGYnMM49Wkbo
created_at: 2022-02-03T19:28:26Z
updated_at: 2022-02-03T19:28:26Z
author_association: CONTRIBUTOR
body:
  > `from sqlite_utils.utils import find_spatialite` is part of the documented API already:
  > 
  > https://sqlite-utils.datasette.io/en/3.22.1/python-api.html#finding-spatialite
  > 
  > To avoid needing to bump the major version number to 4 to indicate a backwards incompatible change, we should keep a `from .gis import find_spatialite` line at the top of `utils.py` such that any existing code with that documented import continues to work.

  This is fixed now. I had to take out the type annotations for `Database` and `Table` to avoid a circular import, but that's fine and may be moot if these become class methods.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029335225
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/385
node_id: IC_kwDOCGYnMM49Wmi5
created_at: 2022-02-03T19:39:40Z
updated_at: 2022-02-03T19:39:40Z
author_association: OWNER
body:
  > I thought about adding these as methods on `Database` and `Table`, and I'm back and forth on it for the same reasons you are. It's certainly cleaner, and it's clearer what you're operating on. I could go either way.
  > 
  > I do sort of like having all the Spatialite stuff in its own module, just because it's built around an extension you might not have or want, but I don't know if that's a good reason to have a different API.
  > 
  > You could have `init_spatialite` add methods to `Database` and `Table`, so they're only there if you have Spatialite set up. Is that too clever? It feels too clever.

  Yeah that's too clever. You know what? I'm pretty confident we are both massively over-thinking this. We should put the methods on `Database` and `Table`! API simplicity and consistency matters more than vague concerns about purity.

reactions:
[cell repeats an earlier one in this file: 112 characters]


