# issue_comments

html_url: https://github.com/simonw/datasette/issues/1464#issuecomment-915302885
issue_url: https://api.github.com/repos/simonw/datasette/issues/1464
node_id: IC_kwDOBm6k_c42jmnl
created_at: 2021-09-08T14:44:50Z
updated_at: 2021-09-08T14:44:50Z
author_association: CONTRIBUTOR
body:
  thanks for the response! full errors attached; excerpt:

  ```
  ...


      def test_searchmode(table_metadata, querystring, expected_rows):
          with make_app_client(
              metadata={"databases": {"fixtures": {"tables": {"searchable": table_metadata}}}}
          ) as client:
              response = client.get("/fixtures/searchable.json?" + querystring)
  >           assert expected_rows == response.json["rows"]
  E           AssertionError: assert [[1, 'barry c...sel', 'puma']] == []
  E             Left contains 2 more items, first extra item: [1, 'barry cat', 'terry dog', 'panther']
  E             Use -v to get the full diff

  /Users/t/dev/datasette/tests/test_api.py:1115: AssertionError
  ```

  [errors.txt](https://github.com/simonw/datasette/files/7129719/errors.txt)

  A quick scan of #1223 suggests you're right. Unfortunately, pysqlite3-binary isn't available for Mac OS X, so I can't quickly check that that fixes it; will do so later.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/326#issuecomment-915321467
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/326
node_id: IC_kwDOCGYnMM42jrJ7
created_at: 2021-09-08T15:05:53Z
updated_at: 2021-09-08T15:05:53Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/326?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#326](https://codecov.io/gh/simonw/sqlite-utils/pull/326?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (078a087) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/77c240df56068341561e95e4a412cbfa24dc5bc7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (77c240d) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/326/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/326?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main     #326   +/-   ##
  =======================================
    Coverage   96.58%   96.58%           
  =======================================
    Files           5        5           
    Lines        2223     2223           
  =======================================
    Hits         2147     2147           
    Misses         76       76           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/326?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/326?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [77c240d...078a087](https://codecov.io/gh/simonw/sqlite-utils/pull/326?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1464#issuecomment-915343886
issue_url: https://api.github.com/repos/simonw/datasette/issues/1464
node_id: IC_kwDOBm6k_c42jwoO
created_at: 2021-09-08T15:32:06Z
updated_at: 2021-09-08T15:32:06Z
author_association: CONTRIBUTOR
body:
  Thanks, that does look similar!

  > Unfortunately, pysqlite3-binary isn't available for Mac OS X, so I can't quickly check that that fixes it; will do so later.

  Ah that makes sense, I guess that's why this isn't just always installed already. I wonder if a possible solution to this issue could be doing feature detection on whether this feature is supported by the current Sqlite version, and if not these tests could be disabled locally? But possibly there's a better way to handle this, will see what @simonw thinks

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1464#issuecomment-917642487
issue_url: https://api.github.com/repos/simonw/datasette/issues/1464
node_id: IC_kwDOBm6k_c42shz3
created_at: 2021-09-12T14:03:09Z
updated_at: 2021-09-12T14:03:09Z
author_association: CONTRIBUTOR
body:
  haven't had time to get back to this, but idle thought that I'm recording for later investigation: how does the continuous integration handle this installation issue? Is it documented there?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1469#issuecomment-917839062
issue_url: https://api.github.com/repos/simonw/datasette/issues/1469
node_id: IC_kwDOBm6k_c42tRzW
created_at: 2021-09-13T04:52:01Z
updated_at: 2021-09-13T04:52:01Z
author_association: OWNER
body:
  Here's the code at fault: https://github.com/simonw/datasette/blob/b28b6cd2fe97f7e193a235877abeec2c8eb0a821/datasette/static/table.js#L137-L146

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1468#issuecomment-917839507
issue_url: https://api.github.com/repos/simonw/datasette/issues/1468
node_id: IC_kwDOBm6k_c42tR6T
created_at: 2021-09-13T04:53:22Z
updated_at: 2021-09-13T04:53:22Z
author_association: OWNER
body:
  At the moment this isn't possible - though there's a workaround which is to define a SQL view for the query, at which point facets will be displayed again.

  I did a lot of the work required to support this when I refactored how facets worked a while back - but to finally implement this I need to refactor the table view and the arbitrary query view to share much more logic than they do at the moment.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1466#issuecomment-917840012
issue_url: https://api.github.com/repos/simonw/datasette/issues/1466
node_id: IC_kwDOBm6k_c42tSCM
created_at: 2021-09-13T04:54:59Z
updated_at: 2021-09-13T04:54:59Z
author_association: OWNER
body:
  Especially relevant now that 0.2.0 is out which is a much higher quality release. https://github.com/simonw/datasette-app/releases/tag/0.2.0

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1464#issuecomment-918621705
issue_url: https://api.github.com/repos/simonw/datasette/issues/1464
node_id: IC_kwDOBm6k_c42wQ4J
created_at: 2021-09-13T22:17:17Z
updated_at: 2021-09-13T22:17:17Z
author_association: CONTRIBUTOR
body:
  > haven't had time to get back to this, but idle thought that I'm recording for later investigation: how does the continuous integration handle this installation issue? Is it documented there?

  Not certain, but I think tests in CI run on Ubuntu and don't appear to install any additional Sqlite-related dependencies, and so my guess is the version of Sqlite installed by default on Ubuntu has the `SQLITE_ENABLE_FTS3_PARENTHESIS` option enabled and so doesn't run into this issue.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1471#issuecomment-919141156
issue_url: https://api.github.com/repos/simonw/datasette/issues/1471
node_id: IC_kwDOBm6k_c42yPsk
created_at: 2021-09-14T13:16:29Z
updated_at: 2021-10-13T21:12:25Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1471?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1471](https://codecov.io/gh/simonw/datasette/pull/1471?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (01b05ed) into [main](https://codecov.io/gh/simonw/datasette/commit/0d5cc20aeffa3537cfc9296d01ec24b9c6e23dcf?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0d5cc20) will **decrease** coverage by `0.00%`.
  > The diff coverage is `n/a`.

  > :exclamation: Current head 01b05ed differs from pull request most recent head 847238a. Consider uploading reports for the commit 847238a to get more accurate results
  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1471/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1471?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@            Coverage Diff             @@
  ##             main    #1471      +/-   ##
  ==========================================
  - Coverage   91.83%   91.83%   -0.01%     
  ==========================================
    Files          34       34              
    Lines        4422     4421       -1     
  ==========================================
  - Hits         4061     4060       -1     
    Misses        361      361              
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1471?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1471/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `96.00% <0.00%> (-0.01%)` | :arrow_down: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1471?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1471?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [a673a93...847238a](https://codecov.io/gh/simonw/datasette/pull/1471?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/236#issuecomment-920543967
issue_url: https://api.github.com/repos/simonw/datasette/issues/236
node_id: IC_kwDOBm6k_c423mLf
created_at: 2021-09-16T03:19:08Z
updated_at: 2021-09-16T03:19:08Z
author_association: NONE
body:
  :wave: I just put together a small example using the lambda container image support: https://github.com/sethvincent/datasette-aws-lambda-example

  It uses mangum and AWS's [python runtime interface client](https://github.com/aws/aws-lambda-python-runtime-interface-client) to handle the lambda event stuff.

  I'd be happy to help with a publish plugin for AWS lambda as I plan to use this for upcoming projects.

  The example uses the [serverless](https://www.serverless.com) cli for deployment but there might be a more suitable deployment approach for the plugin. It would be cool if users didn't have to install anything additional other than the aws cli and its associated config/credentials setup.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/236#issuecomment-922075480
issue_url: https://api.github.com/repos/simonw/datasette/issues/236
node_id: IC_kwDOBm6k_c429cFY
created_at: 2021-09-17T20:54:13Z
updated_at: 2021-09-17T20:54:13Z
author_association: OWNER
body:
  That's so useful @sethvincent! Really interesting reading your code there, especially clever how you're using the `base_url` config.

  I'd be very interested to see what your demo looks like without using serverless - completely agree that the less additional dependencies there are for this the better.

  I'm also very interested in figuring out a way to run Datasette in Lambda but with the SQLite database on an EFS volume. Do you have a feel for how hard that would be?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1473#issuecomment-922363640
issue_url: https://api.github.com/repos/simonw/datasette/issues/1473
node_id: IC_kwDOBm6k_c42-ib4
created_at: 2021-09-18T19:45:47Z
updated_at: 2021-09-18T19:45:47Z
author_association: CONTRIBUTOR
body:
  An update, if I remove the `img` tag and replace it with the text, "Safer or Toxic?" it links to the right place.

  Also, if I keep things exactly as they are, and it improperly, but consistently goes to the `undefined` page, on THAT 404 page, a click on the image properly clicks through to the www.SaferOrToxic.org page.

  Weird stuff.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1473#issuecomment-922394999
issue_url: https://api.github.com/repos/simonw/datasette/issues/1473
node_id: IC_kwDOBm6k_c42-qF3
created_at: 2021-09-19T00:44:39Z
updated_at: 2021-09-19T00:45:32Z
author_association: CONTRIBUTOR
body:
  I replaced:
  ```
  <a class="text-3xl font-bold leading-none" href="https://www.saferortoxic.org">
  <img 
  src="https://www.saferortoxic.org/images/logo_hu26e4dce8d5931af1ea33526b28fc8383_9734_c52a4f1635ef88bda858373270551ed2.webp" 
  class="custom-logo" 
  alt="Logo: Safer or Toxic?" 
  width="300px">
  </a>
  ```
  with:
  ```
  <img src="https://www.saferortoxic.org/images/logo_hu26e4dce8d5931af1ea33526b28fc8383_9734_c52a4f1635ef88bda858373270551ed2.webp" 
  alt="Logo: Safer or Toxic?" 
  style="width:300px; cursor:pointer;" 
  onclick="window.location.href='https://saferortoxic.org';return false;">
  ```

  I'd still love to know what caused this (and how to troubleshoot to figure it out), so I'll leave it open for a bit, but I do have a functional logo linking to the Hugo home page, at least locally. I'll likely push tomorrow.

  (Before trying this, I tried to apply a background image to the `a` tag. That didn't work.)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/111#issuecomment-923106887
issue_url: https://api.github.com/repos/simonw/datasette/issues/111
node_id: IC_kwDOBm6k_c43BX5H
created_at: 2021-09-20T16:58:39Z
updated_at: 2021-09-20T16:58:39Z
author_association: OWNER
body:
  Still a good idea today too! Would be great for https://cdc-vaccination-history.datasette.io/ for example.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1476#issuecomment-923979964
issue_url: https://api.github.com/repos/simonw/datasette/issues/1476
node_id: IC_kwDOBm6k_c43EtC8
created_at: 2021-09-21T13:18:22Z
updated_at: 2021-09-21T13:18:22Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1476?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1476](https://codecov.io/gh/simonw/datasette/pull/1476?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (2eb01ff) into [main](https://codecov.io/gh/simonw/datasette/commit/b28b6cd2fe97f7e193a235877abeec2c8eb0a821?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (b28b6cd) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1476/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1476?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1476   +/-   ##
  =======================================
    Coverage   91.83%   91.83%           
  =======================================
    Files          34       34           
    Lines        4421     4421           
  =======================================
    Hits         4060     4060           
    Misses        361      361           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1476?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1476?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [b28b6cd...2eb01ff](https://codecov.io/gh/simonw/datasette/pull/1476?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/111#issuecomment-924432643
issue_url: https://api.github.com/repos/simonw/datasette/issues/111
node_id: IC_kwDOBm6k_c43GbkD
created_at: 2021-09-21T22:23:23Z
updated_at: 2021-09-21T22:23:23Z
author_association: OWNER
body:
  I'm going to use https://github.com/dateutil/dateutil for this - it's been maintained constantly (by an evolving team of contributors) [since 2003](https://github.com/dateutil/dateutil/commit/68ae2757ae15c84bf947d47a82a314b3b975bc9b) and is a very trustworthy dependency.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/111#issuecomment-924435971
issue_url: https://api.github.com/repos/simonw/datasette/issues/111
node_id: IC_kwDOBm6k_c43GcYD
created_at: 2021-09-21T22:29:15Z
updated_at: 2021-09-21T22:29:49Z
author_association: OWNER
body:
  So this is a metadata key called `updated` which can be applied at the table, database or instance level. It is represented as a `.isoformat()` timestamp.

  Question: should I support just the date - `yyyy-mm-dd` - in addition to the datetime?

  I think so. I can easily imagine situations where the exact time of day that a change was made hasn't been recorded, but the overall date is known.

  But in that case, should the `updated` key sometimes be `yyyy-mm-dd` and sometimes be the full isoformat datetime? Or should there be an `updated_date` key that's used for just the date?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/111#issuecomment-924437942
issue_url: https://api.github.com/repos/simonw/datasette/issues/111
node_id: IC_kwDOBm6k_c43Gc22
created_at: 2021-09-21T22:32:59Z
updated_at: 2021-09-21T22:47:07Z
author_association: OWNER
body:
  Side-note: Django 4.0 [will switch](https://docs.djangoproject.com/en/dev/releases/4.0/#zoneinfo-default-timezone-implementation) from using `pytz` to using the standard library `zoneinfo` module introduced in Python 3.9, which has a backport that works as far back as 3.6: https://github.com/pganssle/zoneinfo (https://pypi.org/project/backports.zoneinfo/)

  If I need to handle timezones I'll use that, but I think I can get away without it?

  Django does this: https://github.com/django/django/blob/b0ed619303d2fb723330ca9efa3acf23d49f1d19/setup.cfg#L39-L43

  ```
  install_requires =
      asgiref >= 3.3.2
      backports.zoneinfo; python_version<"3.9"
      sqlparse >= 0.2.2
      tzdata; sys_platform == 'win32'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/111#issuecomment-924438481
issue_url: https://api.github.com/repos/simonw/datasette/issues/111
node_id: IC_kwDOBm6k_c43Gc_R
created_at: 2021-09-21T22:34:03Z
updated_at: 2021-09-21T22:34:21Z
author_association: OWNER
body:
  The simplest possible version of this: it's always represented as a UTC ISO datetime, like this: 

      "updated": "2020-10-31T12:00:00+00:00"

  Later versions of Datasette could extend this to handle other timezones or support just the date (though that's a backwards incompatible change so probably better to decide on the date thing right now).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/111#issuecomment-924443089
issue_url: https://api.github.com/repos/simonw/datasette/issues/111
node_id: IC_kwDOBm6k_c43GeHR
created_at: 2021-09-21T22:45:14Z
updated_at: 2021-09-21T22:45:26Z
author_association: OWNER
body:
  The audiences I care about here are:

  - Producers of this timestamp - generally that will be users who are using `datasette publish` to share their data
  - Human consumers of this timestamp - end users who look at a Datasette site and want to know how recent the data is
  - Machine consumers of this timestamp - API integrations that might want to check if a Datasette instance has been updated before downloading new data

  For producers I think there are going to be two categories. The first is users who run "publish" and want the site to reflect when they did so (probably using `--updated=now` when they publish). The second are users who are willing to spend more time thinking about this - for example my various git scraping projects where I want to use a date derived from the git history.

  For humans... I'd like to be able to calculate a relative time (3 hours ago) in addition to showing the display time, because that helps avoid confusion over timezones.

  For machine consumers, it might be nice to offer the option of a calculated Unix timestamp-since-1970, since those can be easier to work with in some languages than running a full ISO date parser.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/328#issuecomment-925296085
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/328
node_id: IC_kwDOCGYnMM43JuXV
created_at: 2021-09-22T20:14:53Z
updated_at: 2021-09-22T20:14:53Z
author_association: OWNER
body:
  The bug is in this code: https://github.com/simonw/sqlite-utils/blob/77c240df56068341561e95e4a412cbfa24dc5bc7/sqlite_utils/cli.py#L2205-L2227

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/325#issuecomment-925300637
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/325
node_id: IC_kwDOCGYnMM43Jved
created_at: 2021-09-22T20:21:26Z
updated_at: 2021-09-22T20:21:26Z
author_association: OWNER
body:
  The `t1` and `t2` aliases were meant to handle this case, but the are no good if the tool throws an error.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/325#issuecomment-925301981
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/325
node_id: IC_kwDOCGYnMM43Jvzd
created_at: 2021-09-22T20:23:25Z
updated_at: 2021-09-22T20:23:25Z
author_association: OWNER
body:
  Oddly I can't replicate this on macOS:

  ```
  (sqlite-utils) sqlite-utils % ls foo/*.csv
  foo/bug.csv
  (sqlite-utils) sqlite-utils % ls bar/*.csv
  bar/bug.csv
  (sqlite-utils) sqlite-utils % sqlite-utils memory foo/bug.csv bar/bug.csv --schema
  CREATE TABLE "bug" (
     [col1] TEXT,
     [col2] TEXT
  );
  CREATE VIEW t1 AS select * from [bug];
  CREATE VIEW t AS select * from [bug];
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/325#issuecomment-925303497
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/325
node_id: IC_kwDOCGYnMM43JwLJ
created_at: 2021-09-22T20:25:44Z
updated_at: 2021-09-22T20:25:44Z
author_association: OWNER
body:
  Here's the relevant code: https://github.com/simonw/sqlite-utils/blob/7427a9137f60de961b6331d0922a3f03da0d1890/sqlite_utils/cli.py#L1289-L1292

  I can fix this by checking to see if `csv_table` is already in use and adding a suffix.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/325#issuecomment-925321439
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/325
node_id: IC_kwDOCGYnMM43J0jf
created_at: 2021-09-22T20:52:56Z
updated_at: 2021-09-22T20:52:56Z
author_association: OWNER
body:
  Updated documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#running-queries-directly-against-csv-or-json

  > If two files have the same name they will be assigned a numeric suffix:
  > 
  >     $ sqlite-utils memory foo/data.csv bar/data.csv "select * from data_2"

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926203501
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/329
node_id: IC_kwDOCGYnMM43NL5t
created_at: 2021-09-23T22:15:07Z
updated_at: 2021-09-23T22:15:30Z
author_association: OWNER
body:
  Quoting https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586676856 in full:

  > I'm not sure what to do about this one.
  > 
  > I can't fix it: this bug in Python's `sqlite3` module means that even if I write a database out with column names that include `[]` I won't be able to read them back again.
  > 
  > So... I could do one of the following:
  > 
  > * Throw an error if a column name includes those characters. That's my preferred option I think.
  > * Automatically replace `[` in column names with `(` and `]` with `)`
  > * Do the automatic replacement but show a user-visible warning when I do it
  > * Throw an error, but give the user an option to run with e.g. `--fix-column-names` which applies that automatic fix.
  > 
  > 
  > Since this is likely to be an incredibly rare edge-case I think I'd rather minimize the amount of code that deals with it, so my preferred option is to just throw that error and stop.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926204046
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/329
node_id: IC_kwDOCGYnMM43NMCO
created_at: 2021-09-23T22:16:18Z
updated_at: 2021-09-23T22:16:18Z
author_association: OWNER
body:
  So either I do the automatic replacement, or I let the user request automatic replacement, or a third option: I do automatic replacement but let the user opt to receive an error instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926205047
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/329
node_id: IC_kwDOCGYnMM43NMR3
created_at: 2021-09-23T22:18:34Z
updated_at: 2021-09-23T22:19:38Z
author_association: OWNER
body:
  Here's the code where this happens: https://github.com/simonw/sqlite-utils/blob/54191d4dc114d7dc21e849b48a4d5ae4f9e601ca/sqlite_utils/db.py#L2943-L2948

  It's called from three different methods in `db.py`: `create_table_sql()`, `update()` and `insert_all()`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926206220
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/329
node_id: IC_kwDOCGYnMM43NMkM
created_at: 2021-09-23T22:20:55Z
updated_at: 2021-09-23T22:21:19Z
author_association: OWNER
body:
  If I add a new parameter for opting in and out of fixing these, what should it be called? A few options:

  - `fix_columns=False`
  - `rename_bad_columns=False`
  - `fix_column_names=False`
  - `error_on_invalid_column_names=True`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926206705
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/329
node_id: IC_kwDOCGYnMM43NMrx
created_at: 2021-09-23T22:21:58Z
updated_at: 2021-09-23T22:21:58Z
author_association: OWNER
body:
  I'm inclined to just fix them and not have an option for opting-out of fixing them, since it adds quite a bit of cruft to the overall API design for an option that maybe no-one will ever use.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926207246
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/329
node_id: IC_kwDOCGYnMM43NM0O
created_at: 2021-09-23T22:23:09Z
updated_at: 2021-09-23T22:23:09Z
author_association: OWNER
body:
  What are my options for replacing those characters?

  - `[` becomes `(` and `]` becomes `)`
  - `[` and `]` are removed entirely
  - `[` and `]` both become `_`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926207719
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/329
node_id: IC_kwDOCGYnMM43NM7n
created_at: 2021-09-23T22:24:09Z
updated_at: 2021-09-23T22:24:09Z
author_association: OWNER
body:
  I think I like the underscore option best. I don't like the idea of injecting surprise `( )` parenthesis, and having them vanish entirely could result in things like `item[price]` becoming `itemprice` which feels confusing.

  `item_price_` is a little ugly but I think I can live with it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926208819
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/329
node_id: IC_kwDOCGYnMM43NNMz
created_at: 2021-09-23T22:26:54Z
updated_at: 2021-09-23T22:26:54Z
author_association: OWNER
body:
  I could even have those replacement characters be properties of the `Database` class, so uses can sub-class and change them.

  ```python
  class Database:
      left_brace_replace = "_"
      right_brace_replace = "_"
      ...
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/54#issuecomment-927312650
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/54
node_id: IC_kwDODEm0Qs43RasK
created_at: 2021-09-26T14:09:51Z
updated_at: 2021-09-26T14:09:51Z
author_association: NONE
body:
  Similar trouble with ageinfo using 0.22. Here's what my ageinfo.js file looks like:

  ```
  window.YTD.ageinfo.part0 = [
    {
      "ageMeta" : { }
    }
  ]
  ```

  Commenting out the registration for ageinfo in archive.py gets my archive to import.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/98#issuecomment-928790381
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/98
node_id: IC_kwDOCGYnMM43XDdt
created_at: 2021-09-28T04:38:44Z
updated_at: 2021-09-28T04:38:44Z
author_association: NONE
body:
  Hi @simonw - wondering if you might be able to shed some light here. I've seemed to reproduce this issue. 
  Here's the stacktrace: 

  ```
  ...     db["potholes"].insert(pothole, pk='id', alter=True, replace=True)
  ... 

  Traceback (most recent call last):
    File "<stdin>", line 3, in <module>
    File "/Users/patricktrainer/.pyenv/versions/3.9.0/lib/python3.9/site-packages/sqlite_utils/db.py", line 2481, in insert
      return self.insert_all(
    File "/Users/patricktrainer/.pyenv/versions/3.9.0/lib/python3.9/site-packages/sqlite_utils/db.py", line 2596, in insert_all
      self.insert_chunk(
    File "/Users/patricktrainer/.pyenv/versions/3.9.0/lib/python3.9/site-packages/sqlite_utils/db.py", line 2424, in insert_chunk
      row = list(self.rows_where("rowid = ?", [self.last_rowid]))[0]
  IndexError: list index out of range
  ```

  Interesting enough, I found that omitting the `pk` param does not throw the error.

  Let me know how I can help out! 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1479#issuecomment-929927144
issue_url: https://api.github.com/repos/simonw/datasette/issues/1479
node_id: IC_kwDOBm6k_c43bY_o
created_at: 2021-09-29T07:49:40Z
updated_at: 2021-09-29T07:49:40Z
author_association: NONE
body:
  My search yielded these four entries: https://github.com/simonw/datasette/issues?q=PermissionError%3A+%5BWinError+32%5D+

  Maybe this is the closet hit?
  https://github.com/simonw/datasette/issues/744

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1479#issuecomment-932807859
issue_url: https://api.github.com/repos/simonw/datasette/issues/1479
node_id: IC_kwDOBm6k_c43mYSz
created_at: 2021-10-02T19:22:35Z
updated_at: 2021-10-02T19:22:35Z
author_association: OWNER
body:
  I'm pretty sure this is a Windows issue, not a Fly issue. I imagine it affects other forms of `datasette publish` too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1479#issuecomment-932808043
issue_url: https://api.github.com/repos/simonw/datasette/issues/1479
node_id: IC_kwDOBm6k_c43mYVr
created_at: 2021-10-02T19:23:52Z
updated_at: 2021-10-02T19:23:52Z
author_association: OWNER
body:
  I suspect the root cause of this may be in this code: https://github.com/simonw/datasette/blob/63886178a649586b403966a27a45881709d2b868/datasette/utils/__init__.py#L673-L677

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1479#issuecomment-932808216
issue_url: https://api.github.com/repos/simonw/datasette/issues/1479
node_id: IC_kwDOBm6k_c43mYYY
created_at: 2021-10-02T19:25:09Z
updated_at: 2021-10-02T19:25:09Z
author_association: OWNER
body:
  Actually no, from that stack trace you provided:
  ```
    File "c:\users\grott\anaconda3\lib\site-packages\click\core.py", line 610, in invoke
      return callback(*args, **kwargs)
    File "c:\users\grott\anaconda3\lib\site-packages\datasette\cli.py", line 283, in package
      call(args)
    File "c:\users\grott\anaconda3\lib\contextlib.py", line 119, in __exit__
      next(self.gen)
    File "c:\users\grott\anaconda3\lib\site-packages\datasette\utils\__init__.py", line 451, in temporary_docker_directory
      tmp.cleanup()
  ```
  It looks like the problem occurs here: https://github.com/simonw/datasette/blob/b1fed48a95516ae84c0f020582303ab50ab817e2/datasette/utils/__init__.py#L449-L452

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/3#issuecomment-934207940
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/3
node_id: IC_kwDOD079W843ruHE
created_at: 2021-10-05T08:57:41Z
updated_at: 2021-10-05T08:57:41Z
author_association: NONE
body:
  Maybe the exif-loader from the SpatiaLite project could be useful as a reference even it is written in C and it also saves images as blobs https://www.gaia-gis.it/fossil/spatialite-tools/file?name=exif_loader.c&ci=tip. The tool is also integrated into the spatialite-gui application.

  I found some user documentation from the web archive http://web.archive.org/web/20180629041238/https://www.gaia-gis.it/spatialite-2.3.1/spatialite-exif-2.3.1.html.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/3#issuecomment-934372104
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/3
node_id: IC_kwDOD079W843sWMI
created_at: 2021-10-05T12:38:24Z
updated_at: 2021-10-05T12:38:24Z
author_association: CONTRIBUTOR
body:
  As dogsheep-photos already uses [osxphotos](https://github.com/RhetTbull/osxphotos) to load photos you can access the EXIF data via osxphotos. Apple Photos imports a small subset of EXIF data at the time the photo is imported and osxphotos provides this via the [exif_info](https://github.com/RhetTbull/osxphotos#exifinfo) property. If you want the full EXIF data, osxphotos also provides a wrapper around [exiftool](https://github.com/RhetTbull/osxphotos#exiftool).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1470#issuecomment-938124652
issue_url: https://api.github.com/repos/simonw/datasette/issues/1470
node_id: IC_kwDOBm6k_c436qVs
created_at: 2021-10-07T20:17:53Z
updated_at: 2021-10-07T20:18:55Z
author_association: OWNER
body:
  Here's the exception:
  ```
  -> params[f"p{len(params)}"] = components[0]
  (Pdb) list
  603  	
  604  	                # Figure out the SQL for next-based-on-primary-key first
  605  	                next_by_pk_clauses = []
  606  	                if use_rowid:
  607  	                    next_by_pk_clauses.append(f"rowid > :p{len(params)}")
  608  ->	                    params[f"p{len(params)}"] = components[0]
  609  	                else:
  610  	                    # Apply the tie-breaker based on primary keys
  611  	                    if len(components) == len(pks):
  612  	                        param_len = len(params)
  613  	                        next_by_pk_clauses.append(
  ```
  Debugger shows that `components` is an empty array, so `components[0]` cannot be resolved:

  ```
  -> params[f"p{len(params)}"] = components[0]
  (Pdb) params
  {'search': 'hello'}
  (Pdb) components
  []
  ```

  So the bug is in this code: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/datasette/views/table.py#L604-L617

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1470#issuecomment-938131806
issue_url: https://api.github.com/repos/simonw/datasette/issues/1470
node_id: IC_kwDOBm6k_c436sFe
created_at: 2021-10-07T20:28:30Z
updated_at: 2021-10-07T20:28:30Z
author_association: OWNER
body:
  On further investigation this isn't related to `_search` at all - it happens when you explicitly sort by `_sort=rowid` and apply a `_next`

  - https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_next=200 works without an error (currently)
  - https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_next=200&_sort=rowid shows that error

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1480#issuecomment-938134038
issue_url: https://api.github.com/repos/simonw/datasette/issues/1480
node_id: IC_kwDOBm6k_c436soW
created_at: 2021-10-07T20:31:46Z
updated_at: 2021-10-07T20:31:46Z
author_association: OWNER
body:
  I've had this problem too - my solution was to not use Cloud Run for databases larger than about 2GB, but the way you describe it here makes me think that maybe there is a workaround here which could get it to work.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1481#issuecomment-938141121
issue_url: https://api.github.com/repos/simonw/datasette/issues/1481
node_id: IC_kwDOBm6k_c436uXB
created_at: 2021-10-07T20:42:37Z
updated_at: 2021-10-24T22:19:28Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1481?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1481](https://codecov.io/gh/simonw/datasette/pull/1481?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (77542e7) into [main](https://codecov.io/gh/simonw/datasette/commit/63886178a649586b403966a27a45881709d2b868?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (6388617) will **decrease** coverage by `0.01%`.
  > The diff coverage is `n/a`.

  > :exclamation: Current head 77542e7 differs from pull request most recent head 50005bd. Consider uploading reports for the commit 50005bd to get more accurate results
  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1481/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1481?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@            Coverage Diff             @@
  ##             main    #1481      +/-   ##
  ==========================================
  - Coverage   91.83%   91.82%   -0.02%     
  ==========================================
    Files          34       34              
    Lines        4421     4426       +5     
  ==========================================
  + Hits         4060     4064       +4     
  - Misses        361      362       +1     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1481?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1481/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `95.36% <0.00%> (-0.14%)` | :arrow_down: |
  | [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1481/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `95.41% <0.00%> (ø)` | |
  | [datasette/views/database.py](https://codecov.io/gh/simonw/datasette/pull/1481/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `97.56% <0.00%> (ø)` | |
  | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1481/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `96.00% <0.00%> (+<0.01%)` | :arrow_up: |
  | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1481/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.78% <0.00%> (+0.02%)` | :arrow_up: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1481?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1481?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [6388617...50005bd](https://codecov.io/gh/simonw/datasette/pull/1481?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1481#issuecomment-938142436
issue_url: https://api.github.com/repos/simonw/datasette/issues/1481
node_id: IC_kwDOBm6k_c436urk
created_at: 2021-10-07T20:44:43Z
updated_at: 2021-10-07T20:44:43Z
author_association: OWNER
body:
  The 3.10 tests failed a lot. Trying to run this locally:

  ```
  /tmp % pyenv install 3.10
  python-build: definition not found: 3.10

  The following versions contain `3.10' in the name:
    3.10.0a6
    3.10-dev
    miniconda-3.10.1
    miniconda3-3.10.1

  See all available versions with `pyenv install --list'.

  If the version you need is missing, try upgrading pyenv:

    brew update && brew upgrade pyenv
  ```
  So trying:

      brew update && brew upgrade pyenv

  Then did this:

  ```
  /tmp % brew upgrade pyenv 
  ==> Upgrading 1 outdated package:
  pyenv 1.2.24.1 -> 2.1.0
  ```
  This decided to upgrade everything by downloaded everything on the internet. Aah, Homebrew.

  But it looks like I have `3.10.0` available to `pyenv` now.

  ```
  /tmp % pyenv install 3.10.0
  python-build: use openssl@1.1 from homebrew
  python-build: use readline from homebrew
  Downloading Python-3.10.0.tar.xz...
  -> https://www.python.org/ftp/python/3.10.0/Python-3.10.0.tar.xz
  Installing Python-3.10.0...
  ...
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1480#issuecomment-938171377
issue_url: https://api.github.com/repos/simonw/datasette/issues/1480
node_id: IC_kwDOBm6k_c4361vx
created_at: 2021-10-07T21:33:12Z
updated_at: 2021-10-07T21:33:12Z
author_association: CONTRIBUTOR
body:
  Thanks for the reply @simonw. What services have you had better success with than Cloud Run for larger database?

  Also, what about my issue description makes you think there may be a workaround?

  Is there any instrumentation I could add to see at which point in the deploy the memory usage spikes? Should I be able to see this whether it's running under Docker locally, or do you suspect this is Cloud Run-specific?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1481#issuecomment-939074818
issue_url: https://api.github.com/repos/simonw/datasette/issues/1481
node_id: IC_kwDOBm6k_c43-SUC
created_at: 2021-10-08T19:40:23Z
updated_at: 2021-10-08T19:40:23Z
author_association: OWNER
body:
  Then I created myself a temporary 3.10 environment using `pipenv` like so:

      cd /tmp
      mkdir py310
      cd py310
      pipenv shell --python /Users/simon/.pyenv/versions/3.10.0/bin/python

  And used that with my Datasette checkout like so:

      cd ~/.../datasette
      pip install -e '.[test]'
      pytest

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1481#issuecomment-939075686
issue_url: https://api.github.com/repos/simonw/datasette/issues/1481
node_id: IC_kwDOBm6k_c43-Shm
created_at: 2021-10-08T19:42:00Z
updated_at: 2021-10-08T19:42:00Z
author_association: OWNER
body:
  Running `pytest -x --pdb` helped me see this error:

  ```
    File "/Users/simon/Dropbox/Development/datasette/datasette/views/base.py", line 122, in dispatch_request
      await self.ds.refresh_schemas()
    File "/Users/simon/Dropbox/Development/datasette/datasette/app.py", line 344, in refresh_schemas
      await self._refresh_schemas()
    File "/Users/simon/Dropbox/Development/datasette/datasette/app.py", line 349, in _refresh_schemas
      await init_internal_db(internal_db)
    File "/Users/simon/Dropbox/Development/datasette/datasette/utils/internal_db.py", line 5, in init_internal_db
      await db.execute_write(
    File "/Users/simon/Dropbox/Development/datasette/datasette/database.py", line 102, in execute_write
      return await self.execute_write_fn(_inner, block=block)
    File "/Users/simon/Dropbox/Development/datasette/datasette/database.py", line 113, in execute_write_fn
      reply_queue = janus.Queue()
    File "/Users/simon/.local/share/virtualenvs/py310-Z8fTATkJ/lib/python3.10/site-packages/janus/__init__.py", line 39, in __init__
      self._async_not_empty = asyncio.Condition(self._async_mutex)
    File "/Users/simon/.pyenv/versions/3.10.0/lib/python3.10/asyncio/locks.py", line 234, in __init__
      raise ValueError("loop argument must agree with lock")
  ValueError: loop argument must agree with lock
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1481#issuecomment-939076399
issue_url: https://api.github.com/repos/simonw/datasette/issues/1481
node_id: IC_kwDOBm6k_c43-Ssv
created_at: 2021-10-08T19:43:33Z
updated_at: 2021-10-08T19:43:33Z
author_association: OWNER
body:
  So maybe this is an issue with Janus? I'm using https://pypi.org/project/janus/ 0.6.1 which is the latest release, from October 2020.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1481#issuecomment-939078095
issue_url: https://api.github.com/repos/simonw/datasette/issues/1481
node_id: IC_kwDOBm6k_c43-THP
created_at: 2021-10-08T19:47:29Z
updated_at: 2021-10-08T19:47:29Z
author_association: OWNER
body:
  Only mention I can find of that "loop argument must agree with lock" error is here - which doesn't have any tips for a workaround yet: https://giters.com/django/channels_redis/issues/278

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1481#issuecomment-939078872
issue_url: https://api.github.com/repos/simonw/datasette/issues/1481
node_id: IC_kwDOBm6k_c43-TTY
created_at: 2021-10-08T19:49:08Z
updated_at: 2021-10-08T19:49:08Z
author_association: OWNER
body:
  Here's the code that raises that error: https://github.com/python/cpython/blob/bb3e0c240bc60fe08d332ff5955d54197f79751c/Lib/asyncio/locks.py#L219-L234

  ```python
  class Condition(_ContextManagerMixin, mixins._LoopBoundMixin):
      """Asynchronous equivalent to threading.Condition.
      This class implements condition variable objects. A condition variable
      allows one or more coroutines to wait until they are notified by another
      coroutine.
      A new Lock object is created and used as the underlying lock.
      """

      def __init__(self, lock=None, *, loop=mixins._marker):
          super().__init__(loop=loop)
          if lock is None:
              lock = Lock()
          elif lock._loop is not self._get_loop():
              raise ValueError("loop argument must agree with lock")
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1481#issuecomment-939079727
issue_url: https://api.github.com/repos/simonw/datasette/issues/1481
node_id: IC_kwDOBm6k_c43-Tgv
created_at: 2021-10-08T19:50:52Z
updated_at: 2021-10-08T19:50:52Z
author_association: OWNER
body:
  And here's the relevant Janus code: https://github.com/aio-libs/janus/blob/d7970f8b76bcac2e087067ca4575ac845e481874/janus/__init__.py#L24-L42

  ```python
  class Queue(Generic[T]):
      def __init__(self, maxsize: int = 0) -> None:
          self._loop = current_loop()
          self._maxsize = maxsize

          self._init(maxsize)

          self._unfinished_tasks = 0

          self._sync_mutex = threading.Lock()
          self._sync_not_empty = threading.Condition(self._sync_mutex)
          self._sync_not_full = threading.Condition(self._sync_mutex)
          self._all_tasks_done = threading.Condition(self._sync_mutex)

          self._async_mutex = asyncio.Lock()
          # "loop argument must agree with lock" exception is raised here:
          self._async_not_empty = asyncio.Condition(self._async_mutex)
          self._async_not_full = asyncio.Condition(self._async_mutex)
          self._finished = asyncio.Event()
          self._finished.set()
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1481#issuecomment-939100803
issue_url: https://api.github.com/repos/simonw/datasette/issues/1481
node_id: IC_kwDOBm6k_c43-YqD
created_at: 2021-10-08T20:33:42Z
updated_at: 2021-10-08T20:33:42Z
author_association: OWNER
body:
  There's a tiny chance this could be a bug in Python 3.10 itself - I filed an issue here: https://bugs.python.org/issue45416 - in which I said:

  > In Python 3.10 it is not possible to instantiate an asyncio.Condition that wraps an asyncio.Lock without raising a "loop argument must agree with lock" exception.
  > 
  > This code raises that exception:
  > 
  >     asyncio.Condition(asyncio.Lock())
  > 
  > This worked in previous Python versions.
  > 
  > Note that the error only occurs if an event loop is running. Here's a simple script that replicates the problem:
  > 
  >     import asyncio
  > 
  >     # This runs without an exception:
  >     print(asyncio.Condition(asyncio.Lock()))
  > 
  >     # This does not work:
  >     async def example():
  >         print(asyncio.Condition(asyncio.Lock()))
  > 
  >     # This raises "ValueError: loop argument must agree with lock":
  >     asyncio.run(example())

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1481#issuecomment-939185319
issue_url: https://api.github.com/repos/simonw/datasette/issues/1481
node_id: IC_kwDOBm6k_c43-tSn
created_at: 2021-10-09T00:04:54Z
updated_at: 2021-10-09T00:04:54Z
author_association: OWNER
body:
  I applied my PR against Janus to my local copy of Datasette like so:

      pip uninstall janus
      pip install https://github.com/aio-libs/janus/archive/9e13d3fb74e2c93d7501443b370a455d1b302b1f.zip

  Then I ran the Datasette tests and got a much happier pass rate.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1482#issuecomment-939191311
issue_url: https://api.github.com/repos/simonw/datasette/issues/1482
node_id: IC_kwDOBm6k_c43-uwP
created_at: 2021-10-09T00:35:04Z
updated_at: 2021-10-09T00:35:04Z
author_association: OWNER
body:
  I think that SQLite error message difference was caused by https://github.com/python/cpython/commit/a50e28377bcf37121b55c2de70d95a5386c478f8 or related work.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1485#issuecomment-940023938
issue_url: https://api.github.com/repos/simonw/datasette/issues/1485
node_id: IC_kwDOBm6k_c44B6CC
created_at: 2021-10-11T13:18:16Z
updated_at: 2021-10-11T13:18:16Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1485?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1485](https://codecov.io/gh/simonw/datasette/pull/1485?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (be9aed9) into [main](https://codecov.io/gh/simonw/datasette/commit/0d5cc20aeffa3537cfc9296d01ec24b9c6e23dcf?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0d5cc20) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1485/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1485?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1485   +/-   ##
  =======================================
    Coverage   91.83%   91.83%           
  =======================================
    Files          34       34           
    Lines        4422     4422           
  =======================================
    Hits         4061     4061           
    Misses        361      361           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1485?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1485?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [0d5cc20...be9aed9](https://codecov.io/gh/simonw/datasette/pull/1485?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1432#issuecomment-941002127
issue_url: https://api.github.com/repos/simonw/datasette/issues/1432
node_id: IC_kwDOBm6k_c44Fo2P
created_at: 2021-10-12T13:14:31Z
updated_at: 2021-10-12T13:14:39Z
author_association: NONE
body:
  Any workaround for making it work with datasette-publish-vercel. Currently getting below error

  module initialization error: __init__() got an unexpected keyword argument 'config'
  module initialization error
  __init__() got an unexpected keyword argument 'config'

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1469#issuecomment-942725632
issue_url: https://api.github.com/repos/simonw/datasette/issues/1469
node_id: IC_kwDOBm6k_c44MNoA
created_at: 2021-10-13T21:13:30Z
updated_at: 2021-10-13T21:13:30Z
author_association: OWNER
body:
  The core problem here is treating the `?_facet=` query string parameters as the point of truth for which facets are currently enabled.

  Instead, I could use a `data-` attribute on the displayed facets.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/330#issuecomment-942752844
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/330
node_id: IC_kwDOCGYnMM44MURM
created_at: 2021-10-13T22:00:31Z
updated_at: 2021-10-13T22:11:30Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/330?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#330](https://codecov.io/gh/simonw/sqlite-utils/pull/330?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (fc3de90) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/718a8f61bcaed39c04d5d223104056213f8c8516?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (718a8f6) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/330/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/330?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main     #330   +/-   ##
  =======================================
    Coverage   96.59%   96.59%           
  =======================================
    Files           5        5           
    Lines        2230     2230           
  =======================================
    Hits         2154     2154           
    Misses         76       76           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/330?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/330?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [718a8f6...fc3de90](https://codecov.io/gh/simonw/sqlite-utils/pull/330?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1488#issuecomment-942777414
issue_url: https://api.github.com/repos/simonw/datasette/issues/1488
node_id: IC_kwDOBm6k_c44MaRG
created_at: 2021-10-13T22:52:40Z
updated_at: 2021-10-13T22:52:40Z
author_association: OWNER
body:
  Upgrading to 0.20.0 gives me lots of the following errors:

      '{"ok": false, "error": "Database not found: .json?_sort=relationships", "status": 404, "title": null}'

  It looks like the full query string is now being treated as the name of the database.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1488#issuecomment-942778382
issue_url: https://api.github.com/repos/simonw/datasette/issues/1488
node_id: IC_kwDOBm6k_c44MagO
created_at: 2021-10-13T22:55:01Z
updated_at: 2021-10-13T22:55:01Z
author_association: OWNER
body:
  I think the issue is in `route_path()`:
  ```
  > /Users/simon/Dropbox/Development/datasette/datasette/app.py(1182)route_path()
  -> response = await view(request, send)
  (Pdb) path
  '/_memory.json?sql=select+sqlite_version()'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1488#issuecomment-942778673
issue_url: https://api.github.com/repos/simonw/datasette/issues/1488
node_id: IC_kwDOBm6k_c44Makx
created_at: 2021-10-13T22:55:44Z
updated_at: 2021-10-13T22:55:44Z
author_association: OWNER
body:
  ```
  (Pdb) request.scope['path']
  '/_memory.json'
  (Pdb) request.scope['raw_path']
  b'/_memory.json?sql=select+sqlite_version()'
  ```
  So `raw_path` now includes the query string.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1488#issuecomment-942779926
issue_url: https://api.github.com/repos/simonw/datasette/issues/1488
node_id: IC_kwDOBm6k_c44Ma4W
created_at: 2021-10-13T22:59:05Z
updated_at: 2021-10-13T22:59:05Z
author_association: OWNER
body:
  This is weird - as far as I can tell `httpx` has included the query string in `raw_path` for well over a year: https://github.com/encode/httpx/commit/8e4a8a1c73f60fe5754f95b308beaa725cb8791d#diff-c9a78eb3b5f5c4fac4e5552165fbdd5320c7e3fadf9eedabcb5461393466c090R235

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1488#issuecomment-942782673
issue_url: https://api.github.com/repos/simonw/datasette/issues/1488
node_id: IC_kwDOBm6k_c44MbjR
created_at: 2021-10-13T23:04:54Z
updated_at: 2021-10-13T23:04:54Z
author_association: OWNER
body:
  I think this is the change in `httpx` which is causing the bug for me: https://github.com/encode/httpx/commit/ff9813e84dab56f0f3c4ef3a159a4cce8c644a91#diff-0d0cbe9ebcd03cc8c780b0407762540a082f70cc64257f2fcd588cc30f43c15cR96

  <img width="706" alt="Transport_API_as_plain_`request_-__response`_method____1840__·_encode_httpx_ff9813e" src="https://user-images.githubusercontent.com/9599/137224339-0aed171f-b351-4d67-8d6c-ad22f2301f18.png">

  Previously it was using `path` from `path, _, query = full_path.partition(b"?")` to populate the `raw_path` key - but it changed to instead using `request.url.raw_path` which presumably implements the logic that includes the query string.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1489#issuecomment-943594735
issue_url: https://api.github.com/repos/simonw/datasette/issues/1489
node_id: IC_kwDOBm6k_c44Phzv
created_at: 2021-10-14T18:04:12Z
updated_at: 2021-10-14T18:04:12Z
author_association: CONTRIBUTOR
body:
  Looks like this PR is closed. If you re-open it I'll rebase it as long as no-one else has edited it (you can use `@dependabot reopen` if the branch has been deleted).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1489#issuecomment-943594738
issue_url: https://api.github.com/repos/simonw/datasette/issues/1489
node_id: IC_kwDOBm6k_c44Phzy
created_at: 2021-10-14T18:04:13Z
updated_at: 2021-10-14T18:04:13Z
author_association: CONTRIBUTOR
body:
  OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version, let me know by commenting `@dependabot ignore this major version` or `@dependabot ignore this minor version`. You can also ignore all major, minor, or patch releases for a dependency by adding an [`ignore` condition](https://docs.github.com/en/code-security/supply-chain-security/configuration-options-for-dependency-updates#ignore) with the desired `update_types` to your config file.

  If you change your mind, just re-open this PR and I'll resolve any conflicts on it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1467#issuecomment-943632697
issue_url: https://api.github.com/repos/simonw/datasette/issues/1467
node_id: IC_kwDOBm6k_c44PrE5
created_at: 2021-10-14T18:54:18Z
updated_at: 2021-10-14T18:54:18Z
author_association: OWNER
body:
  The test there failed because it turns out there's a whole bunch of places that set the `Access-Control-Allow-Origin` header. I'm going to close this PR and ship a fix that refactors those places to use the same code.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1481#issuecomment-944986367
issue_url: https://api.github.com/repos/simonw/datasette/issues/1481
node_id: IC_kwDOBm6k_c44U1j_
created_at: 2021-10-16T19:07:38Z
updated_at: 2021-10-16T19:09:02Z
author_association: OWNER
body:
  This is blocking an upgrade for the Homebrew Datasette package: https://github.com/Homebrew/homebrew-core/pull/86932

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1481#issuecomment-945020210
issue_url: https://api.github.com/repos/simonw/datasette/issues/1481
node_id: IC_kwDOBm6k_c44U90y
created_at: 2021-10-16T23:19:51Z
updated_at: 2021-10-16T23:19:51Z
author_association: OWNER
body:
  Since that Janus PR hasn't been merged yet, one temporary option for a fix would be to entirely vendor the fixed Janus - https://github.com/aio-libs/janus/blob/9e13d3fb74e2c93d7501443b370a455d1b302b1f/janus/__init__.py - since it's only a single module.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1432#issuecomment-945639639
issue_url: https://api.github.com/repos/simonw/datasette/issues/1432
node_id: IC_kwDOBm6k_c44XVDX
created_at: 2021-10-18T10:44:56Z
updated_at: 2021-10-18T10:44:56Z
author_association: NONE
body:
  @simonw I am getting the below issue again now, even after removing branch argument from vercel datasette plugin

  module initialization error: init() got an unexpected keyword argument 'config'
  module initialization error
  init() got an unexpected keyword argument 'config'

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1494#issuecomment-945763015
issue_url: https://api.github.com/repos/simonw/datasette/issues/1494
node_id: IC_kwDOBm6k_c44XzLH
created_at: 2021-10-18T13:22:56Z
updated_at: 2021-10-18T13:22:56Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1494?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1494](https://codecov.io/gh/simonw/datasette/pull/1494?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (7a3e3c3) into [main](https://codecov.io/gh/simonw/datasette/commit/ff9ccfb0310501a3b4b4ca24d73246a8eb3e7914?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (ff9ccfb) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1494/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1494?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1494   +/-   ##
  =======================================
    Coverage   91.82%   91.82%           
  =======================================
    Files          34       34           
    Lines        4426     4426           
  =======================================
    Hits         4064     4064           
    Misses        362      362           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1494?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1494?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [ff9ccfb...7a3e3c3](https://codecov.io/gh/simonw/datasette/pull/1494?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1470#issuecomment-946097058
issue_url: https://api.github.com/repos/simonw/datasette/issues/1470
node_id: IC_kwDOBm6k_c44ZEui
created_at: 2021-10-18T19:30:15Z
updated_at: 2021-10-18T19:30:15Z
author_association: OWNER
body:
  https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_next=200&_sort=rowid is fixed now.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1432#issuecomment-946255239
issue_url: https://api.github.com/repos/simonw/datasette/issues/1432
node_id: IC_kwDOBm6k_c44ZrWH
created_at: 2021-10-18T23:55:25Z
updated_at: 2021-10-18T23:55:25Z
author_association: CONTRIBUTOR
body:
  I am getting this when I visit my live Datasette page:
  ```
  This Serverless Function has crashed.
  Your connection is working correctly.
  Vercel is working correctly.
  500: INTERNAL_SERVER_ERROR
  Code: FUNCTION_INVOCATION_FAILED
  ID: ...
  ```
  And in the server logs, I'm getting

  ```
  [GET] /disinfectants/listN
  19:53:14:23
  module initialization error: __init__() got an unexpected keyword argument 'config'
  module initialization error
  __init__() got an unexpected keyword argument 'config'
  ```
   Which is the same error that @ashishdotme reported above.
   
   

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1432#issuecomment-946287922
issue_url: https://api.github.com/repos/simonw/datasette/issues/1432
node_id: IC_kwDOBm6k_c44ZzUy
created_at: 2021-10-19T01:16:41Z
updated_at: 2021-10-19T01:16:41Z
author_association: CONTRIBUTOR
body:
  Resolved, with assistance from @ashishdotme (Thank you!)

  Updated requirements.txt to include:
  ```
  datasette==0.59
  datasette-publish-vercel==0.11
  sqlite-utils==3.6
  ```

  Ran:
  ```
  $ pip3 install -r requirements.txt
  ```
  The site is back at work! Yay!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1493#issuecomment-946360891
issue_url: https://api.github.com/repos/simonw/datasette/issues/1493
node_id: IC_kwDOBm6k_c44aFI7
created_at: 2021-10-19T04:37:27Z
updated_at: 2021-10-19T04:37:27Z
author_association: OWNER
body:
  I renamed `/:memory:` to `/_memory` in version 0.55 - https://docs.datasette.io/en/stable/changelog.html#v0-55

  But... in 0.59 I stopped following HTTP redirects by default, which is why this used to work and no longer does!

  So the fix is to update the Homebrew regression test to use this instead:

      datasette --get '/_memory.json?sql=select+3*5'

  Thanks for catching this!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1396#issuecomment-946467547
issue_url: https://api.github.com/repos/simonw/datasette/issues/1396
node_id: IC_kwDOBm6k_c44afLb
created_at: 2021-10-19T08:10:26Z
updated_at: 2021-10-19T08:10:26Z
author_association: CONTRIBUTOR
body:
  Now that 0.59 has excellent annotated release notes, you can re-confirm this is fixed by updating the published Docker image and checking that these fixes still work ;-)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/942#issuecomment-946493045
issue_url: https://api.github.com/repos/simonw/datasette/issues/942
node_id: IC_kwDOBm6k_c44alZ1
created_at: 2021-10-19T08:42:39Z
updated_at: 2021-10-19T08:42:39Z
author_association: NONE
body:
  @simonw I know this is closed, just found this via the annotated release notes, but I wanted to note this one thing:

  Not sure how widely used this is, but I've seen CSVW a couple times in the wild. It is trying to address these metadata challenges in a standardised way.

  See e.g.
  - https://www.w3.org/TR/tabular-data-primer/#h-documentation-columns
  - https://w3c.github.io/csvw/tests/

  I'm not suggesting you change the syntax you've implemented, just letting you know of this effort by W3C.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1480#issuecomment-947196177
issue_url: https://api.github.com/repos/simonw/datasette/issues/1480
node_id: IC_kwDOBm6k_c44dRER
created_at: 2021-10-20T00:05:10Z
updated_at: 2021-10-20T00:05:10Z
author_association: CONTRIBUTOR
body:
  I was looking through the Dockerfile-generation code to see if there was anything that would cause memory usage to be a lot during deployment. 

  I noticed that the Dockerfile [runs `datasette --inspect`](https://github.com/simonw/datasette/blob/main/datasette/utils/__init__.py#L354). Is it possible that this is using a lot of memory usage?

  Or would that come into play when running `gcloud builds submit`, not when it's actually deployed?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1480#issuecomment-947203725
issue_url: https://api.github.com/repos/simonw/datasette/issues/1480
node_id: IC_kwDOBm6k_c44dS6N
created_at: 2021-10-20T00:21:54Z
updated_at: 2021-10-20T00:21:54Z
author_association: CONTRIBUTOR
body:
  This StackOverflow post, [sqlite - Cloud Run: Why does my instance need so much RAM?](https://stackoverflow.com/questions/59812405/cloud-run-why-does-my-instance-need-so-much-ram), points to [this section of the Cloud Run docs](https://cloud.google.com/run/docs/troubleshooting) that says:

  > Note that the Cloud Run container instances run in an environment where the files written to the local filesystem count towards the available memory. This also includes any log files that are not written to /var/log/* or /dev/log.

  Does datasette write any large files when starting? 

  Or does the [`COPY` command in the Dockerfile](https://github.com/simonw/datasette/blob/main/datasette/utils/__init__.py#L349) count as writing to the local filesystem?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1401#issuecomment-950150483
issue_url: https://api.github.com/repos/simonw/datasette/issues/1401
node_id: IC_kwDOBm6k_c44oiVT
created_at: 2021-10-23T13:09:10Z
updated_at: 2021-10-23T13:09:10Z
author_association: CONTRIBUTOR
body:
  I think it's because of this in `app.css` 

  ```
  ol,
  ul {
  	list-style: none;
  }
  ```

  https://github.com/simonw/datasette/blame/main/datasette/static/app.css#L35-L38

  You could probably reinstate that by providing your own CSS.
  https://docs.datasette.io/en/0.24/custom_templates.html#custom-css-and-javascript

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1495#issuecomment-950403521
issue_url: https://api.github.com/repos/simonw/datasette/issues/1495
node_id: IC_kwDOBm6k_c44pgHB
created_at: 2021-10-24T22:09:18Z
updated_at: 2021-10-24T22:09:18Z
author_association: OWNER
body:
  This is a great idea - I've wanted this myself before, but never spent any time thinking about how to achieve it.

  I think your design here is exactly right - an optional third item in the tuple consisting of a dictionary of options to pass to the view function.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1495#issuecomment-950403692
issue_url: https://api.github.com/repos/simonw/datasette/issues/1495
node_id: IC_kwDOBm6k_c44pgJs
created_at: 2021-10-24T22:10:43Z
updated_at: 2021-10-24T22:10:43Z
author_association: OWNER
body:
  To land this change we'll need a unit test that demonstrates the new capability - I suggest putting that next to this test: https://github.com/simonw/datasette/blob/15a9d4abfff0c45dee2a9f851326e1d61b1c678c/tests/test_plugins.py#L648-L659

  It will also need documentation, which should be added here: https://github.com/simonw/datasette/blob/15a9d4abfff0c45dee2a9f851326e1d61b1c678c/docs/plugin_hooks.rst#register-routes-datasette

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1497#issuecomment-950410554
issue_url: https://api.github.com/repos/simonw/datasette/issues/1497
node_id: IC_kwDOBm6k_c44ph06
created_at: 2021-10-24T23:01:20Z
updated_at: 2021-10-24T23:01:28Z
author_association: OWNER
body:
  I can replicate locally by running:

  ```
  docker build -f Dockerfile \
            -t datasetteproject/datasette:0.59.1 \
            --build-arg VERSION=0.59.1 .
  ```
  This gives me the same error.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1497#issuecomment-950411320
issue_url: https://api.github.com/repos/simonw/datasette/issues/1497
node_id: IC_kwDOBm6k_c44piA4
created_at: 2021-10-24T23:06:05Z
updated_at: 2021-10-24T23:06:05Z
author_association: OWNER
body:
  Right now the base image is: https://github.com/simonw/datasette/blob/e6e44372b34414eac2f36a4c1120af4f755aa423/Dockerfile#L1

  I'm going to try `python:3.9.7-slim-buster` instead: https://hub.docker.com/layers/python/library/python/3.9.7-slim-buster/images/sha256-290b95e4b379762a9bd3d72644598e0972f4e2b5442bba60592c018fadcc744d?context=explore

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1497#issuecomment-950411417
issue_url: https://api.github.com/repos/simonw/datasette/issues/1497
node_id: IC_kwDOBm6k_c44piCZ
created_at: 2021-10-24T23:06:45Z
updated_at: 2021-10-24T23:11:14Z
author_association: OWNER
body:
  Same errors with `3.9.7`:
  ```
  #5 41.46 /usr/bin/perl: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory
  #5 41.46 dpkg: error processing package libc6:amd64 (--configure):
  #5 41.46  installed libc6:amd64 package post-installation script subprocess returned error exit status 127
  #5 41.47 Errors were encountered while processing:
  #5 41.47  libc6:amd64
  #5 41.50 E: Sub-process /usr/bin/dpkg returned an error code (1)
  ```
  I'm suspicious of this part of the `Dockerfile`: https://github.com/simonw/datasette/blob/e6e44372b34414eac2f36a4c1120af4f755aa423/Dockerfile#L1-L18

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1497#issuecomment-950411808
issue_url: https://api.github.com/repos/simonw/datasette/issues/1497
node_id: IC_kwDOBm6k_c44piIg
created_at: 2021-10-24T23:08:59Z
updated_at: 2021-10-24T23:08:59Z
author_association: OWNER
body:
  Looks like it's this bug, reported on the Debian mailing list: https://www.mail-archive.com/debian-bugs-dist@lists.debian.org/msg1818037.html

  No obvious workaround there though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1497#issuecomment-950412628
issue_url: https://api.github.com/repos/simonw/datasette/issues/1497
node_id: IC_kwDOBm6k_c44piVU
created_at: 2021-10-24T23:13:20Z
updated_at: 2021-10-24T23:13:27Z
author_association: OWNER
body:
  I think the root cause here is that I'm using a Debian Buster base image and then installing SpatiaLite from Debian unstable (sid) - as described in this comment: https://github.com/simonw/datasette/issues/1249#issuecomment-804309510

  That's has worked fine in the past, but Sid is unstable - and this seems to be one of those instabilities.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1497#issuecomment-950413185
issue_url: https://api.github.com/repos/simonw/datasette/issues/1497
node_id: IC_kwDOBm6k_c44pieB
created_at: 2021-10-24T23:16:25Z
updated_at: 2021-10-24T23:18:30Z
author_association: OWNER
body:
  Debian stable these days is "bullseye" - https://www.debian.org/releases/ - which has the version of SpatiaLite that I was previously pulling in from Sid: https://packages.debian.org/bullseye/libsqlite3-mod-spatialite

  So upgrading to the 3.9.7-slim-bullseye base image may help.

  https://hub.docker.com/layers/python/library/python/3.9.7-slim-bullseye/images/sha256-67af5f544115124dc6d6da1d9d2815aa9825f6fd4aa6710adb0ec1725280fb89?context=explore

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1497#issuecomment-950415129
issue_url: https://api.github.com/repos/simonw/datasette/issues/1497
node_id: IC_kwDOBm6k_c44pi8Z
created_at: 2021-10-24T23:21:33Z
updated_at: 2021-10-24T23:21:33Z
author_association: OWNER
body:
  That fixed it! Resulting image is 249MB which is a very slight size reduction (I think previous was 259MB (uncompressed).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1497#issuecomment-950415822
issue_url: https://api.github.com/repos/simonw/datasette/issues/1497
node_id: IC_kwDOBm6k_c44pjHO
created_at: 2021-10-24T23:25:45Z
updated_at: 2021-10-24T23:25:45Z
author_association: OWNER
body:
  I'm going to attempt to publish `0.59` to Docker Hub using https://github.com/simonw/datasette/blob/2c31d1cd9cd3b63458ccbe391866499fa3f44978/.github/workflows/push_docker_tag.yml - if that works I'll push `0.59.1` as well.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1497#issuecomment-950416061
issue_url: https://api.github.com/repos/simonw/datasette/issues/1497
node_id: IC_kwDOBm6k_c44pjK9
created_at: 2021-10-24T23:27:18Z
updated_at: 2021-10-24T23:27:18Z
author_association: OWNER
body:
  That worked: https://hub.docker.com/layers/datasetteproject/datasette/0.59/images/sha256-038decc28e0ea84b281ecc0058fe8eba7aa99596e5a2177ff714092ad03294ed?context=explore

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1497#issuecomment-950416460
issue_url: https://api.github.com/repos/simonw/datasette/issues/1497
node_id: IC_kwDOBm6k_c44pjRM
created_at: 2021-10-24T23:30:10Z
updated_at: 2021-10-24T23:30:10Z
author_association: OWNER
body:
  Testing that newly published image:
  ```
  % docker run -p 8002:8001 -v `pwd`:/mnt \
      datasetteproject/datasette:0.59 datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db
  Unable to find image 'datasetteproject/datasette:0.59' locally
  0.59: Pulling from datasetteproject/datasette
  7d63c13d9b9b: Already exists 
  6ad2a11ca37b: Already exists 
  e9edbe81a001: Already exists 
  36629b83aba2: Already exists 
  7338abefe51c: Already exists 
  6d71b6b88b82: Pull complete 
  8c4da3c56bdc: Pull complete 
  Digest: sha256:038decc28e0ea84b281ecc0058fe8eba7aa99596e5a2177ff714092ad03294ed
  Status: Downloaded newer image for datasetteproject/datasette:0.59
  INFO:     Started server process [1]
  INFO:     Waiting for application startup.
  INFO:     Application startup complete.
  INFO:     Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit)
  ```
  and `http://localhost:8002/versions.json` returns:

  ```json
  {
      "python": {
          "version": "3.9.7",
          "full": "3.9.7 (default, Oct 12 2021, 02:43:43) \n[GCC 10.2.1 20210110]"
      },
      "datasette": {
          "version": "0.59"
      },
      "asgi": "3.0",
      "uvicorn": "0.15.0",
      "sqlite": {
          "version": "3.34.1"
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1497#issuecomment-950416659
issue_url: https://api.github.com/repos/simonw/datasette/issues/1497
node_id: IC_kwDOBm6k_c44pjUT
created_at: 2021-10-24T23:31:41Z
updated_at: 2021-10-24T23:31:41Z
author_association: OWNER
body:
  Published `0.59.1` as well: https://github.com/simonw/datasette/runs/3991214225?check_suite_focus=true

  Result: https://hub.docker.com/layers/datasetteproject/datasette/0.59.1/images/sha256-dc134f65bec40ed4ea7049188fe1e3915b8e6c3fd999b17effe8ec24868b979c?context=explore

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1497#issuecomment-950416802
issue_url: https://api.github.com/repos/simonw/datasette/issues/1497
node_id: IC_kwDOBm6k_c44pjWi
created_at: 2021-10-24T23:32:39Z
updated_at: 2021-10-24T23:32:39Z
author_association: OWNER
body:
  That's because the `publish.yml` workflow ends with this, which isn't in the `push_docker_tag.yml` workflow:

  https://github.com/simonw/datasette/blob/2c31d1cd9cd3b63458ccbe391866499fa3f44978/.github/workflows/publish.yml#L117-L119

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1497#issuecomment-950417375
issue_url: https://api.github.com/repos/simonw/datasette/issues/1497
node_id: IC_kwDOBm6k_c44pjff
created_at: 2021-10-24T23:36:54Z
updated_at: 2021-10-24T23:36:54Z
author_association: OWNER
body:
  Tried fixing this by pushing a new `latest` tag from my laptop:
  ```
  (datasette) datasette % docker pull datasetteproject/datasette:0.59.1
  0.59.1: Pulling from datasetteproject/datasette
  7d63c13d9b9b: Already exists 
  6ad2a11ca37b: Already exists 
  e9edbe81a001: Already exists 
  36629b83aba2: Already exists 
  7338abefe51c: Already exists 
  6b825daddc6c: Pull complete 
  d7508b065a21: Pull complete 
  Digest: sha256:dc134f65bec40ed4ea7049188fe1e3915b8e6c3fd999b17effe8ec24868b979c
  Status: Downloaded newer image for datasetteproject/datasette:0.59.1
  docker.io/datasetteproject/datasette:0.59.1
  (datasette) datasette % docker tag datasetteproject/datasette:0.59.1 datasetteproject/datasette:latest
  (datasette) datasette % docker push datasetteproject/datasette:latest
  The push refers to repository [docker.io/datasetteproject/datasette]
  d668c99b6ff1: Layer already exists 
  aa20c9013575: Layer already exists 
  c97eebf2b227: Layer already exists 
  284a6c64b82c: Layer already exists 
  388eedeb736e: Layer already exists 
  2feece0964b8: Layer already exists 
  e8b689711f21: Layer already exists 
  errors:
  denied: requested access to the resource is denied
  unauthorized: authentication required
  ```
  So I logged in with `docker login`:
  ```
  (datasette) datasette % docker login
  Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
  Username: datasetteproject
  Password: 
  ```
  And ran the push again and it worked:
  ```
  (datasette) datasette % docker push datasetteproject/datasette:latest
  The push refers to repository [docker.io/datasetteproject/datasette]
  d668c99b6ff1: Layer already exists 
  aa20c9013575: Layer already exists 
  c97eebf2b227: Layer already exists 
  284a6c64b82c: Layer already exists 
  388eedeb736e: Layer already exists 
  2feece0964b8: Layer already exists 
  e8b689711f21: Layer already exists 
  latest: digest: sha256:dc134f65bec40ed4ea7049188fe1e3915b8e6c3fd999b17effe8ec24868b979c size: 1793
  ```
  https://hub.docker.com/layers/datasetteproject/datasette/latest/images/sha256-dc134f65bec40ed4ea7049188fe1e3915b8e6c3fd999b17effe8ec24868b979c?context=explore

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1204#issuecomment-951731255
issue_url: https://api.github.com/repos/simonw/datasette/issues/1204
node_id: IC_kwDOBm6k_c44ukQ3
created_at: 2021-10-26T09:01:28Z
updated_at: 2021-10-26T09:01:28Z
author_association: NONE
body:
  > Writing the tests will be a bit tricky since we need to confirm that the `include_table_top(datasette, database, actor, table)` arguments were all passed correctly but the only thing we get back from the plugin is a list of templates. Maybe encode those values into the template names somehow?

  Why not return a data structure instead of just a template name?

  I've already done some custom hacking to modify datasette but the plugin mechanism you are building here would be much cleaner than what I've built.  I'd be happy to help with testing this PR and fleshing it out further if you are still considering merging this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/878#issuecomment-951740637
issue_url: https://api.github.com/repos/simonw/datasette/issues/878
node_id: IC_kwDOBm6k_c44umjd
created_at: 2021-10-26T09:12:15Z
updated_at: 2021-10-26T09:12:15Z
author_association: NONE
body:
  This sounds really ambitious but also really awesome.  I like the idea that basically any piece of a page could be selectively replaced.

  It sort of sounds like a python asyncio version of https://github.com/observablehq/runtime

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1380#issuecomment-953334718
issue_url: https://api.github.com/repos/simonw/datasette/issues/1380
node_id: IC_kwDOBm6k_c440ru-
created_at: 2021-10-27T21:45:04Z
updated_at: 2021-10-27T21:45:04Z
author_association: CONTRIBUTOR
body:
  I am also getting this issue, using the currently most recent version of datasette

  ```
  $ datasette --version
  datasette, version 0.59.1
  ```

  If I run `datasette` within just a folder of files, 

  ```
  $ datasette serve .
  ```

  Adding new files while datasette is running shows no new files, and removing files causes datasette to return 500 errors. 


  ```
  home
  Error 500
  [Errno 2] No such file or directory: 'mydatabase.db'
  Powered by Datasette
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1380#issuecomment-953366110
issue_url: https://api.github.com/repos/simonw/datasette/issues/1380
node_id: IC_kwDOBm6k_c440zZe
created_at: 2021-10-27T22:48:55Z
updated_at: 2021-10-27T22:48:55Z
author_association: CONTRIBUTOR
body:
  It looks like if the files argument is a directory, `config_dir` is set, but files in that folder are only loaded into `self.files` at the `Datasette` class initialisation. 

  I tried seeing if I could get `--reload` to work, but I'm getting issues trying to use that command when specifying a directory, as the command `serve` ends up in the files list(?): 

  ```
  datasette serve  . --reload
  Error: Invalid value for '[FILES]...': Path 'serve' does not exist.
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1497#issuecomment-953508979
issue_url: https://api.github.com/repos/simonw/datasette/issues/1497
node_id: IC_kwDOBm6k_c441WRz
created_at: 2021-10-28T05:13:49Z
updated_at: 2021-10-28T05:13:49Z
author_association: OWNER
body:
  Wrote about this in my weeknotes: https://simonwillison.net/2021/Oct/28/weeknotes-kubernetes-web-components/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/242#issuecomment-953911245
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/242
node_id: IC_kwDOCGYnMM4424fN
created_at: 2021-10-28T14:37:55Z
updated_at: 2021-10-28T14:37:55Z
author_association: CONTRIBUTOR
body:
  I've been thinking about this a bit lately, doing a project that involves moving a lot of data in and out of SQLite files, datasette and GeoJSON. This has me leaning toward the idea that something like [`datasette query`](https://github.com/simonw/datasette/issues/1356) would be a better place to do async queries.

  I know there's a lot of overlap in sqlite-utils and datasette, and maybe keeping sqlite-utils synchronous would let datasette be entirely async and give a cleaner separation of implementations.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1495#issuecomment-954384496
issue_url: https://api.github.com/repos/simonw/datasette/issues/1495
node_id: IC_kwDOBm6k_c444sBw
created_at: 2021-10-29T03:07:13Z
updated_at: 2021-10-29T03:07:13Z
author_association: CONTRIBUTOR
body:
  okay @simonw, made the requested changes. tests are running locally. i think this is ready for you to look at again.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/206#issuecomment-955365098
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/206
node_id: IC_kwDOCGYnMM448bbq
created_at: 2021-10-30T15:49:19Z
updated_at: 2021-10-30T15:49:19Z
author_association: NONE
body:
  @simonw Hey! JSON parsing for me is failing and I'm getting this same error, but I feel that my json is correct. How can I debug this?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/206#issuecomment-955367409
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/206
node_id: IC_kwDOCGYnMM448b_x
created_at: 2021-10-30T15:50:39Z
updated_at: 2021-10-30T15:50:39Z
author_association: OWNER
body:
  What's the error message?

  Sometimes I pipe JSON through `jq` to check if it's valid:

      cat my.json | jq

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/206#issuecomment-955370190
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/206
node_id: IC_kwDOCGYnMM448crO
created_at: 2021-10-30T15:52:16Z
updated_at: 2021-10-30T15:52:16Z
author_association: NONE
body:
  @simonw That was working fine. It turned out that I had to use `--nl` flag since I was passing in jsonl.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1253#issuecomment-955384545
issue_url: https://api.github.com/repos/simonw/datasette/issues/1253
node_id: IC_kwDOBm6k_c448gLh
created_at: 2021-10-30T16:00:42Z
updated_at: 2021-10-30T16:00:42Z
author_association: NONE
body:
  Yeah, I was pressing Ctrl + Enter as well. Came here to open this issue and found out Shift + Enter works.

  @simonw Any way to configure this?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/173#issuecomment-956041692
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/173
node_id: IC_kwDOCGYnMM44_Anc
created_at: 2021-11-01T08:42:24Z
updated_at: 2021-11-01T08:42:24Z
author_association: NONE
body:
  > I know how to build this for CSV and TSV - I can read them via a file wrapper that counts how many bytes it has seen.
  > 
  > Not sure how to do it for JSON though. Maybe I could provide it just for newline-delimited JSON? Again I can measure progress based on how many bytes have been read.

  I was thinking about this, while inserting a stream of ~40M line-delimited json docs. Wouldn't a `--total-expected` flag work ? 

  That's [how tqdm does it](https://github.com/tqdm/tqdm/blob/fc69d5dcf578f7c7986fa76841a6b793f813df35/tqdm/std.py#L366)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1500#issuecomment-956225475
issue_url: https://api.github.com/repos/simonw/datasette/issues/1500
node_id: IC_kwDOBm6k_c44_tfD
created_at: 2021-11-01T13:16:36Z
updated_at: 2021-11-01T13:16:36Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1500?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1500](https://codecov.io/gh/simonw/datasette/pull/1500?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (1b7f679) into [main](https://codecov.io/gh/simonw/datasette/commit/2c31d1cd9cd3b63458ccbe391866499fa3f44978?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (2c31d1c) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1500/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1500?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1500   +/-   ##
  =======================================
    Coverage   91.82%   91.82%           
  =======================================
    Files          34       34           
    Lines        4426     4426           
  =======================================
    Hits         4064     4064           
    Misses        362      362           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1500?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1500?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [2c31d1c...1b7f679](https://codecov.io/gh/simonw/datasette/pull/1500?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/239#issuecomment-960292442
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/239
node_id: IC_kwDOCGYnMM45POZa
created_at: 2021-11-03T23:28:55Z
updated_at: 2021-11-03T23:28:55Z
author_association: NONE
body:
  I am super interested in this feature.

  After reading the other issues you referenced, I think the right way would be to use the current extract feature and then to use `sqlite-utils convert` to extract the json object into individual columns

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/239#issuecomment-960295228
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/239
node_id: IC_kwDOCGYnMM45PPE8
created_at: 2021-11-03T23:35:37Z
updated_at: 2021-11-03T23:36:50Z
author_association: NONE
body:
  I think I only wonder how I would parse the JSON `value` within such a lambda...

  My naive approach would have been `$ sqlite-utils convert demo.db statuses statuses 'return value' --multi`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1495#issuecomment-960420237
issue_url: https://api.github.com/repos/simonw/datasette/issues/1495
node_id: IC_kwDOBm6k_c45PtmN
created_at: 2021-11-04T03:12:01Z
updated_at: 2021-11-04T03:12:01Z
author_association: CONTRIBUTOR
body:
  This all looks promising! I will need detailed documentation on how to upgrade datasette once it's available, and how to implement. (@fgregg example looks very straightforward on the plugin front.) 
  I'll be so excited if I can get:
  https://list.saferdisinfectants.org/ 
  instead of
  https://list.saferdisinfectants.org/disinfectants/listN

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/337#issuecomment-962259527
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/337
node_id: IC_kwDOCGYnMM45WupH
created_at: 2021-11-05T22:33:02Z
updated_at: 2021-11-05T22:33:02Z
author_association: NONE
body:
  Smokes, it looks like there was a bug in click 8.0.2 (fixed in 8.0.3: https://github.com/pallets/click/issues/2089). Meaning this PR is not needed. Closing.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/336#issuecomment-962411119
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/336
node_id: IC_kwDOCGYnMM45XTpv
created_at: 2021-11-06T07:21:04Z
updated_at: 2021-11-06T07:21:04Z
author_association: OWNER
body:
  I've never used `DEFAULT 'CURRENT_TIMESTAMP'` myself so this one should be an interesting bug to explore.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1380#issuecomment-967181828
issue_url: https://api.github.com/repos/simonw/datasette/issues/1380
node_id: IC_kwDOBm6k_c45pgYE
created_at: 2021-11-12T15:00:18Z
updated_at: 2021-11-12T20:02:29Z
author_association: NONE
body:
  There is no such option see https://github.com/simonw/datasette/issues/43. But you could write a plugin using the datasette.add_database(db, name=None) https://docs.datasette.io/en/stable/internals.html#add-database-db-name-none 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1380#issuecomment-967801997
issue_url: https://api.github.com/repos/simonw/datasette/issues/1380
node_id: IC_kwDOBm6k_c45r3yN
created_at: 2021-11-13T08:05:37Z
updated_at: 2021-11-13T08:09:11Z
author_association: NONE
body:
  @glasnt yeah I guess that could be an option.  I run datasette on large databases > 75gb and the startup time is a bit slow for me even with -i --inspect-file options. Here's a quick sketch for a plugin that will reload db's in a folder that you set for the plugin in metadata.json. If you request /-reload-db new db's will be added. (You probably want to implement some authentication for this =) ) 

  https://gist.github.com/Segerberg/b96a0e0a5389dce2396497323cda7042

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1506#issuecomment-968192980
issue_url: https://api.github.com/repos/simonw/datasette/issues/1506
node_id: IC_kwDOBm6k_c45tXPU
created_at: 2021-11-14T02:22:40Z
updated_at: 2021-11-14T02:22:40Z
author_association: OWNER
body:
  I think the answer is to spot this case and link to `?_item_exact=x` instead of `?_item=x` - it looks like that's already recommended in the documentation here: https://docs.datasette.io/en/stable/json_api.html#column-filter-arguments

  > **?column__exact=value** or **?_column=value**
  >     Returns rows where the specified column exactly matches the value.

  So maybe the facet selection rendering logic needs to spot this and link correctly to it?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1503#issuecomment-968207906
issue_url: https://api.github.com/repos/simonw/datasette/issues/1503
node_id: IC_kwDOBm6k_c45ta4i
created_at: 2021-11-14T05:08:26Z
updated_at: 2021-11-14T05:08:26Z
author_association: OWNER
body:
  Error:
  ```
      def test_table_html_filter_form_column_options(
          path, expected_column_options, app_client
      ):
          response = app_client.get(path)
          assert response.status == 200
          form = Soup(response.body, "html.parser").find("form")
          column_options = [
              o.attrs.get("value") or o.string
              for o in form.select("select[name=_filter_column] option")
          ]
  >       assert expected_column_options == column_options
  E       AssertionError: assert ['- column -'...wid', 'value'] == ['- column -', 'value']
  E         At index 1 diff: 'rowid' != 'value'
  E         Left contains one more item: 'value'
  E         Use -v to get the full diff
  ```
  This is because `rowid` isn't a table column but IS returned by the query used on that page.

  My solution: start with the query columns, but then add any table columns that were not already returned by the query to the end of the `filter_columns` list.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1507#issuecomment-968209560
issue_url: https://api.github.com/repos/simonw/datasette/issues/1507
node_id: IC_kwDOBm6k_c45tbSY
created_at: 2021-11-14T05:26:36Z
updated_at: 2021-11-14T05:26:36Z
author_association: OWNER
body:
  It looks like my builds there still run on Python 2!

  ```
  git clone --no-single-branch --depth 50 https://github.com/simonw/datasette .
  git checkout --force de1e031713f47fbd51eb7239db3e7e6025fbf81a
  git clean -d -f -f
  python2.7 -mvirtualenv /home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2
  /home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/bin/python -m pip install --upgrade --no-cache-dir pip setuptools
  /home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/bin/python -m pip install --upgrade --no-cache-dir mock==1.0.1 pillow==5.4.1 alabaster>=0.7,<0.8,!=0.7.5 commonmark==0.8.1 recommonmark==0.5.0 sphinx<2 sphinx-rtd-theme<0.5 readthedocs-sphinx-ext<2.2
  cat docs/conf.py 
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1507#issuecomment-968209616
issue_url: https://api.github.com/repos/simonw/datasette/issues/1507
node_id: IC_kwDOBm6k_c45tbTQ
created_at: 2021-11-14T05:27:22Z
updated_at: 2021-11-14T05:27:22Z
author_association: OWNER
body:
  https://blog.readthedocs.com/default-python-3/ they started defaulting new projects to Python 3 back in Feb 2019 but clearly my project was created before then.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1507#issuecomment-968209731
issue_url: https://api.github.com/repos/simonw/datasette/issues/1507
node_id: IC_kwDOBm6k_c45tbVD
created_at: 2021-11-14T05:28:41Z
updated_at: 2021-11-14T05:28:41Z
author_association: OWNER
body:
  I will try adding a `.readthedocs.yml` file: https://docs.readthedocs.io/en/stable/config-file/v2.html#python-version

  This might work:

  ```
  version: 2

  build:
    os: ubuntu-20.04
    tools:
      python: "3.9"

  sphinx:
     configuration: docs/conf.py
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1507#issuecomment-968209957
issue_url: https://api.github.com/repos/simonw/datasette/issues/1507
node_id: IC_kwDOBm6k_c45tbYl
created_at: 2021-11-14T05:31:07Z
updated_at: 2021-11-14T05:31:07Z
author_association: OWNER
body:
  Looks like ReadTheDocs builds started failing for `latest` a few weeks ago:

  <img width="845" alt="Banners_and_Alerts_and_Builds___Read_the_Docs" src="https://user-images.githubusercontent.com/9599/141668921-4dd56f14-fe16-4e7f-82c1-12e6893dddd7.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1507#issuecomment-968210222
issue_url: https://api.github.com/repos/simonw/datasette/issues/1507
node_id: IC_kwDOBm6k_c45tbcu
created_at: 2021-11-14T05:34:14Z
updated_at: 2021-11-14T05:34:14Z
author_association: OWNER
body:
  Here's the new build using Python 3: https://readthedocs.org/projects/datasette/builds/15268482/

  It's still broken. Here's one of many issue threads about it, this one has a workaround fix: https://github.com/readthedocs/readthedocs.org/issues/8616#issuecomment-952034858

  > For future readers, the solution for this problem is to pin `docutils<0.18` in your `requirements.txt` file, and have a `.readthedocs.yaml` file with these contents:
  > 
  > ```
  > version: 2
  > 
  > python:
  >    install:
  >    - requirements: docs/requirements.txt
  > ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1507#issuecomment-968210842
issue_url: https://api.github.com/repos/simonw/datasette/issues/1507
node_id: IC_kwDOBm6k_c45tbma
created_at: 2021-11-14T05:41:55Z
updated_at: 2021-11-14T05:41:55Z
author_association: OWNER
body:
  Here's the build with that fix: https://readthedocs.org/projects/datasette/builds/15268498/

  It passed and published the docs: https://docs.datasette.io/en/latest/changelog.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968359137
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/335
node_id: IC_kwDOCGYnMM45t_zh
created_at: 2021-11-14T20:37:00Z
updated_at: 2021-11-14T20:37:00Z
author_association: OWNER
body:
  This is strange - the code already checks that an index doesn't exist before attempting to create it: https://github.com/simonw/sqlite-utils/blob/92aa5c9c5d26b0889c8c3d97c76a908d5f8af211/sqlite_utils/db.py#L893-L902

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968359868
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/335
node_id: IC_kwDOCGYnMM45t_-8
created_at: 2021-11-14T20:41:42Z
updated_at: 2021-11-14T20:41:42Z
author_association: OWNER
body:
  The "index idx_generators_eia860_report_date already exists" error suggests that the problem here is actually one of an index name collision.

  ```python
                   table.create_index([fk.column]) 
  ```
  This will derive a name for the index automatically from the name of the table and the name of the passed in columns: https://github.com/simonw/sqlite-utils/blob/92aa5c9c5d26b0889c8c3d97c76a908d5f8af211/sqlite_utils/db.py#L1536-L1539

  So perhaps `.create_index()` should grow an extra option that creates the index even if the name already exists, by finding a new name.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968360387
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/335
node_id: IC_kwDOCGYnMM45uAHD
created_at: 2021-11-14T20:45:44Z
updated_at: 2021-11-14T20:45:44Z
author_association: OWNER
body:
  What would such an option be called? Some options:

  - `table.create_index([fk.column], force=True)` - not obvious what `force` means here
  - `table.create_index([fk.column], ignore_existing_name=True)` - not obvious what `ignore` means here
  - `table.create_index([fk.column], pick_unique_name=True)` - bit verbose

  If the user doesn't pass in an explicit name it seems like their intent is "just create me the index, I don't care what name you use" - so actually perhaps the default behaviour here should be to pick a new unique name if that name is already in use.

  Then maybe there should be an option to disable that - some options there:

  - `table.create_index([fk.column], error_on_existing_index_name=True)` - too verbose
  - `table.create_index([fk.column], force=False)` - not clear what `force` means

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968360538
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/335
node_id: IC_kwDOCGYnMM45uAJa
created_at: 2021-11-14T20:46:56Z
updated_at: 2021-11-14T20:46:56Z
author_association: OWNER
body:
  I'm tempted to not provide an opt-out option either: if you call `table.create_index(...)` without specifying an index name I think the tool should create the index for you, quietly picking an index name that works.

  But... it feels wasteful to create an index that exactly duplicates an existing index. Would SQLite even let you do that or would it notice and NOT double the amount of disk space used for that index?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968361285
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/335
node_id: IC_kwDOCGYnMM45uAVF
created_at: 2021-11-14T20:51:57Z
updated_at: 2021-11-14T20:51:57Z
author_association: OWNER
body:
  SQLite will happily create multiple identical indexes on a table, using more disk space each time:
  ```pycon
  >>> import sqlite_utils
  >>> db = sqlite_utils.Database("dupes.db")
  >>> db["t"].insert_all({"id": i} for i in range(10000))
  <Table t (id)>
  # dupes.db is 98304 bytes
  >>> db["t"].create_index(["id"])
  <Table t (id)>
  # dupes.db is 204800 bytes
  >>> db["t"].indexes
  [Index(seq=0, name='idx_t_id', unique=0, origin='c', partial=0, columns=['id'])]
  >>> db["t"].create_index(["id"], index_name="t_idx_t_id_2")
  <Table t (id)>
  # 311296 bytes
  >>> db["t"].create_index(["id"], index_name="t_idx_t_id_3")
  <Table t (id)>
  # 417792 bytes
  >>> db.vacuum()
  # Still 417792 bytes
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968361409
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/335
node_id: IC_kwDOCGYnMM45uAXB
created_at: 2021-11-14T20:52:55Z
updated_at: 2021-11-14T20:52:55Z
author_association: OWNER
body:
  Looking at the method signature: https://github.com/simonw/sqlite-utils/blob/92aa5c9c5d26b0889c8c3d97c76a908d5f8af211/sqlite_utils/db.py#L1518-L1524

  `if_not_exists` just adds a `IF NOT EXISTS` clause here: https://github.com/simonw/sqlite-utils/blob/92aa5c9c5d26b0889c8c3d97c76a908d5f8af211/sqlite_utils/db.py#L1549-L1561

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968361671
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/335
node_id: IC_kwDOCGYnMM45uAbH
created_at: 2021-11-14T20:54:53Z
updated_at: 2021-11-14T21:01:14Z
author_association: OWNER
body:
  I'm leaning towards `table.create_index(columns, ignore_existing_name=True)` now.

  Or `resolve_existing_name` - or `skip_existing_name`?

  "ignore" sounds like it might not create the index if the name exists, but we want to still create the index but pick a new name.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968362214
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/335
node_id: IC_kwDOCGYnMM45uAjm
created_at: 2021-11-14T20:59:15Z
updated_at: 2021-11-14T20:59:15Z
author_association: OWNER
body:
  How to figure out if an index name is already in use? `PRAGMA index_list(t)` requires a table name. This does it:

  ```sql
  SELECT name 
  FROM sqlite_master 
  WHERE type = 'index';
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968371112
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/335
node_id: IC_kwDOCGYnMM45uCuo
created_at: 2021-11-14T21:57:43Z
updated_at: 2021-11-14T22:21:31Z
author_association: OWNER
body:
  `create_index(..., find_unique_name=)` is good. Default to false. `index_foreign_keys` can set it to true.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/331#issuecomment-968384005
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/331
node_id: IC_kwDOCGYnMM45uF4F
created_at: 2021-11-14T23:19:29Z
updated_at: 2021-11-14T23:20:32Z
author_association: OWNER
body:
  Tested it like this, against a freshly built `.tar.gz` package from my development environment:
  ```
  (w) w % mypy .                    
  hello.py:1: error: Skipping analyzing "sqlite_utils": found module but no type hints or library stubs
  hello.py:1: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
  Found 1 error in 1 file (checked 1 source file)
  (w) w % pip install ~/Dropbox/Development/sqlite-utils/dist/sqlite-utils-3.17.1.tar.gz
  Processing /Users/simon/Dropbox/Development/sqlite-utils/dist/sqlite-utils-3.17.1.tar.gz
  ...
  Successfully installed sqlite-utils-3.17.1
  (w) w % mypy .                                                                        
  Success: no issues found in 1 source file
  ```
  I tested against the `.whl` too.

  My `hello.py` script contained this:
  ```python
  import sqlite_utils
  from typing import cast

  if __name__ == "__main__":
      db = sqlite_utils.Database(memory=True)
      table = cast(sqlite_utils.db.Table, db["foo"])
      table.insert({"id": 5})
      print(list(db.query("select * from foo")))
  ```
  That `cast()` is necessary because without it you get this error:
  ```
  (w) w % mypy .
  hello.py:7: error: Item "View" of "Union[Table, View]" has no attribute "insert"
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/324#issuecomment-968384988
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/324
node_id: IC_kwDOCGYnMM45uGHc
created_at: 2021-11-14T23:25:16Z
updated_at: 2021-11-14T23:25:16Z
author_association: OWNER
body:
  Yes this was absolutely the intention! Thanks, I wonder how often I've made that mistake in other projects?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/322#issuecomment-968401459
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/322
node_id: IC_kwDOCGYnMM45uKIz
created_at: 2021-11-15T00:26:42Z
updated_at: 2021-11-15T00:26:42Z
author_association: OWNER
body:
  This relates to the fact that dictionaries, lists and tuples get special treatment and are converted to JSON strings, using this code: https://github.com/simonw/sqlite-utils/blob/e8d958109ee290cfa1b44ef7a39629bb50ab673e/sqlite_utils/db.py#L2937-L2947

  So the `COLUMN_TYPE_MAPPING` should include those  too - right now it looks like this: https://github.com/simonw/sqlite-utils/blob/e8d958109ee290cfa1b44ef7a39629bb50ab673e/sqlite_utils/db.py#L165-L188

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/339#issuecomment-968434425
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/339
node_id: IC_kwDOCGYnMM45uSL5
created_at: 2021-11-15T01:42:36Z
updated_at: 2021-11-15T01:42:36Z
author_association: OWNER
body:
  Here's the current signature of `table.lookup()`: https://github.com/simonw/sqlite-utils/blob/9cda5b070f885a7995f0c307bcc4f45f0812994a/sqlite_utils/db.py#L2716-L2729

  I'm going to add a second positional argument which can provide a dictionary of column->value to use when creating the original table and populating the initial row. If the row already exists, those columns will be ignored entirely.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/339#issuecomment-968434594
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/339
node_id: IC_kwDOCGYnMM45uSOi
created_at: 2021-11-15T01:43:10Z
updated_at: 2021-11-15T01:43:10Z
author_association: OWNER
body:
  What should I call this parameter? Django has a similar feature where it calls them `defaults=` (for `get_or_create()`) but I'm not a huge fan of that name.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/339#issuecomment-968435041
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/339
node_id: IC_kwDOCGYnMM45uSVh
created_at: 2021-11-15T01:44:42Z
updated_at: 2021-11-15T01:44:42Z
author_association: OWNER
body:
  `lookup(column_values, extra_values)` is one option.

  `column_values` isn't actually a great name for the first parameter any more, since the second parameter also takes column values. The first parameter is now all about the unique lookup values.

  Maybe this:

      lookup(lookup_values, extra_values)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/339#issuecomment-968450579
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/339
node_id: IC_kwDOCGYnMM45uWIT
created_at: 2021-11-15T02:02:34Z
updated_at: 2021-11-15T02:02:34Z
author_association: OWNER
body:
  Documentation: https://github.com/simonw/sqlite-utils/blob/54a2269e91ce72b059618662ed133a85f3d42e4a/docs/python-api.rst#working-with-lookup-tables

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/329#issuecomment-968451954
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/329
node_id: IC_kwDOCGYnMM45uWdy
created_at: 2021-11-15T02:05:29Z
updated_at: 2021-11-15T02:05:29Z
author_association: OWNER
body:
  > I could even have those replacement characters be properties of the `Database` class, so uses can sub-class and change them.

  I'm not going to do this, it's unnecessary extra complexity and it means the function that fixes the column names needs to have access to the current `Database` instance.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/329#issuecomment-968453129
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/329
node_id: IC_kwDOCGYnMM45uWwJ
created_at: 2021-11-15T02:07:46Z
updated_at: 2021-11-15T02:07:46Z
author_association: OWNER
body:
  If I replace `validate_column_names(row.keys())` with `fix_column_names(row)` I need to decide what to do about things like `pk=` and `column_order=`.

  What should the following do?

  ```python
  table.insert({"foo[bar]": 4}, pk="foo[bar]", column_order=["foo[bar]"])
  ```
  Should it spot the old column names in the `pk=` and `column_order=` parameters and pretend that `foo_bar_` was passed instead?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/329#issuecomment-968458837
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/329
node_id: IC_kwDOCGYnMM45uYJV
created_at: 2021-11-15T02:21:15Z
updated_at: 2021-11-15T02:21:15Z
author_association: OWNER
body:
  I'm not going to implement a fix that rewrites the `pk` and `column_order` and other parameters - at least not yet. The main thing I'm trying to fix here is what happens when you attempt to import a CSV file with `[ ]` in the column names, which should be unaffected by that second challenge.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/329#issuecomment-968470212
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/329
node_id: IC_kwDOCGYnMM45ua7E
created_at: 2021-11-15T02:49:28Z
updated_at: 2021-11-15T02:49:28Z
author_association: OWNER
body:
  I was going to replace all of the `validate_column_names()` bits with something that fixed them instead, but I think I have a better idea: I'm only going to apply the fix for the various '.insert()` methods that create the initial tables.

  I'll keep the `validate_column_names()` where they are at the moment. Once you've inserted the data and created the tables it will be up to you to use the new, correct column names.

  This avoids the whole issue of needing to rewrite parameters, and solves the immediate problem which is consuming CSV files with bad column names.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1508#issuecomment-968904414
issue_url: https://api.github.com/repos/simonw/datasette/issues/1508
node_id: IC_kwDOBm6k_c45wE7e
created_at: 2021-11-15T13:20:49Z
updated_at: 2021-11-15T13:20:49Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1508?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1508](https://codecov.io/gh/simonw/datasette/pull/1508?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (299774b) into [main](https://codecov.io/gh/simonw/datasette/commit/502c02fa6dde6a8bb840af6c4c8cf858aa1db687?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (502c02f) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1508/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1508?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1508   +/-   ##
  =======================================
    Coverage   91.82%   91.82%           
  =======================================
    Files          34       34           
    Lines        4430     4430           
  =======================================
    Hits         4068     4068           
    Misses        362      362           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1508?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1508?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [502c02f...299774b](https://codecov.io/gh/simonw/datasette/pull/1508?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/519#issuecomment-969433734
issue_url: https://api.github.com/repos/simonw/datasette/issues/519
node_id: IC_kwDOBm6k_c45yGKG
created_at: 2021-11-15T23:26:11Z
updated_at: 2021-11-15T23:26:11Z
author_association: OWNER
body:
  I'm happy with this as the goals for 1.0. I'm going to close this issue and create three tracking tickets for the three key themes:

  - https://github.com/simonw/datasette/issues/1509
  - https://github.com/simonw/datasette/issues/1510
  - https://github.com/simonw/datasette/issues/1511

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/448#issuecomment-969436930
issue_url: https://api.github.com/repos/simonw/datasette/issues/448
node_id: IC_kwDOBm6k_c45yG8C
created_at: 2021-11-15T23:31:58Z
updated_at: 2021-11-15T23:31:58Z
author_association: OWNER
body:
  I think this SQL recipe may work instead:
  ```sql
  select
    *
  from
    ads_with_targets
  where
    'people_who_match:interests:African-American Civil Rights Movement (1954—68)' in (
      select
        value
      from
        json_each(target_names)
    )
    and 'interests:Martin Luther King III' in (
      select
        value
      from
        json_each(target_names)
    )
  ```
  https://json-view-facet-bug-demo-j7hipcg4aq-uc.a.run.app/russian-ads?sql=select%0D%0A++*%0D%0Afrom%0D%0A++ads_with_targets%0D%0Awhere%0D%0A++%27people_who_match%3Ainterests%3AAfrican-American+Civil+Rights+Movement+%281954%E2%80%9468%29%27+in+%28%0D%0A++++select%0D%0A++++++value%0D%0A++++from%0D%0A++++++json_each%28target_names%29%0D%0A++%29%0D%0A++and+%27interests%3AMartin+Luther+King+III%27+in+%28%0D%0A++++select%0D%0A++++++value%0D%0A++++from%0D%0A++++++json_each%28target_names%29%0D%0A++%29&interests=&African=&Martin=

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/448#issuecomment-969440918
issue_url: https://api.github.com/repos/simonw/datasette/issues/448
node_id: IC_kwDOBm6k_c45yH6W
created_at: 2021-11-15T23:40:17Z
updated_at: 2021-11-15T23:40:35Z
author_association: OWNER
body:
  Applied that fix to the `arraycontains` filter but I'm still getting bad results for the faceting:

  <img width="915" alt="russian-ads__ads_with_targets__172_rows_where_where_target_names_contains__people_who_match_interests_African-American_culture__and_datasette_—_pipenv_shell_▸_python_—_80×24" src="https://user-images.githubusercontent.com/9599/141869800-0092e0e1-005e-43bc-bac5-bf0e159224a8.png">

  Should never get 182 results on a page that faceting against only 172 items.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/448#issuecomment-969442215
issue_url: https://api.github.com/repos/simonw/datasette/issues/448
node_id: IC_kwDOBm6k_c45yIOn
created_at: 2021-11-15T23:42:03Z
updated_at: 2021-11-15T23:42:03Z
author_association: OWNER
body:
  I think this code is wrong in the `ArrayFacet` class: https://github.com/simonw/datasette/blob/502c02fa6dde6a8bb840af6c4c8cf858aa1db687/datasette/facets.py#L357-L364

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/448#issuecomment-969446972
issue_url: https://api.github.com/repos/simonw/datasette/issues/448
node_id: IC_kwDOBm6k_c45yJY8
created_at: 2021-11-15T23:46:13Z
updated_at: 2021-11-15T23:46:13Z
author_association: OWNER
body:
  It looks like the problem here is that some of the tags occur more than once in the documents:

  <img width="1146" alt="russian-ads__ads_with_targets__172_rows_where_where_target_names_contains__people_who_match_interests_African-American_culture_" src="https://user-images.githubusercontent.com/9599/141870312-5ffe7151-826e-42e0-b1cc-36c46de6b618.png">

  So they get counted more than once, hence the 182 count for something that couldn't possibly return more than 172 documents.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/448#issuecomment-969449772
issue_url: https://api.github.com/repos/simonw/datasette/issues/448
node_id: IC_kwDOBm6k_c45yKEs
created_at: 2021-11-15T23:48:37Z
updated_at: 2021-11-15T23:48:37Z
author_association: OWNER
body:
  Given this query: https://json-view-facet-bug-demo-j7hipcg4aq-uc.a.run.app/russian-ads?sql=select%0D%0A++j.value+as+value%2C%0D%0A++count%28*%29+as+count%0D%0Afrom%0D%0A++%28%0D%0A++++select%0D%0A++++++id%2C%0D%0A++++++file%2C%0D%0A++++++clicks%2C%0D%0A++++++impressions%2C%0D%0A++++++text%2C%0D%0A++++++url%2C%0D%0A++++++spend_amount%2C%0D%0A++++++spend_currency%2C%0D%0A++++++created%2C%0D%0A++++++ended%2C%0D%0A++++++target_names%0D%0A++++from%0D%0A++++++ads_with_targets%0D%0A++++where%0D%0A++++++%3Ap0+in+%28%0D%0A++++++++select%0D%0A++++++++++value%0D%0A++++++++from%0D%0A++++++++++json_each%28%5Bads_with_targets%5D.%5Btarget_names%5D%29%0D%0A++++++%29%0D%0A++%29%0D%0A++join+json_each%28target_names%29+j%0D%0Agroup+by%0D%0A++j.value%0D%0Aorder+by%0D%0A++count+desc%2C%0D%0A++value%0D%0Alimit%0D%0A++31&p0=people_who_match%3Ainterests%3AAfrican-American+culture

  ```sql
  select
    j.value as value,
    count(*) as count
  from
    (
      select
        id,
        file,
        clicks,
        impressions,
        text,
        url,
        spend_amount,
        spend_currency,
        created,
        ended,
        target_names
      from
        ads_with_targets
      where
        :p0 in (
          select
            value
          from
            json_each([ads_with_targets].[target_names])
        )
    )
    join json_each(target_names) j
  group by
    j.value
  order by
    count desc,
    value
  limit
    31
  ```
  How can I return a count of the number of documents containing each tag, but not the number of total tags that match including duplicates?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/448#issuecomment-969557008
issue_url: https://api.github.com/repos/simonw/datasette/issues/448
node_id: IC_kwDOBm6k_c45ykQQ
created_at: 2021-11-16T00:56:09Z
updated_at: 2021-11-16T00:59:59Z
author_association: OWNER
body:
  This looks like it might work:
  ```sql
  with inner as (
    select
      *
    from
      ads_with_targets
    where
      :p0 in (
        select
          value
        from
          json_each([ads_with_targets].[target_names])
      )
  ),
  deduped_array_items as (
    select
      distinct j.value,
      inner.*
    from
      json_each([inner].[target_names]) j
      join inner
  )
  select
    value,
    count(*)
  from
    deduped_array_items
  group by
    value
  order by
    count(*) desc
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/448#issuecomment-969557972
issue_url: https://api.github.com/repos/simonw/datasette/issues/448
node_id: IC_kwDOBm6k_c45ykfU
created_at: 2021-11-16T00:56:58Z
updated_at: 2021-11-16T00:56:58Z
author_association: OWNER
body:
  It uses a CTE which were introduced in SQLite 3.8 - and AWS Lambda Python 3.9 still provides 3.7 - but I've checked and I can use `pysqlite3-binary` to work around that there so I'm OK relying on CTEs for this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/448#issuecomment-969572281
issue_url: https://api.github.com/repos/simonw/datasette/issues/448
node_id: IC_kwDOBm6k_c45yn-5
created_at: 2021-11-16T01:05:11Z
updated_at: 2021-11-16T01:05:11Z
author_association: OWNER
body:
  I tried this and it seems to work correctly:
  ```python
          for source_and_config in self.get_configs():
              config = source_and_config["config"]
              source = source_and_config["source"]
              column = config.get("column") or config["simple"]
              facet_sql = """
                  with inner as ({sql}),
                  deduped_array_items as (
                      select
                          distinct j.value,
                          inner.*
                      from
                          json_each([inner].{col}) j
                          join inner
                  )
                  select
                      value as value,
                      count(*) as count
                  from
                      deduped_array_items
                  group by
                      value
                  order by
                      count(*) desc limit {limit}
              """.format(
                  col=escape_sqlite(column), sql=self.sql, limit=facet_size + 1
              )
  ```
  The queries are _very_ slow though - I had to bump up to 2s time limit even against only a view returning 3,499 rows.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/448#issuecomment-969578466
issue_url: https://api.github.com/repos/simonw/datasette/issues/448
node_id: IC_kwDOBm6k_c45ypfi
created_at: 2021-11-16T01:08:29Z
updated_at: 2021-11-16T01:08:29Z
author_association: OWNER
body:
  Actually with the cache warmed up it looks like the facet query is taking 150ms which is good enough.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/448#issuecomment-969582098
issue_url: https://api.github.com/repos/simonw/datasette/issues/448
node_id: IC_kwDOBm6k_c45yqYS
created_at: 2021-11-16T01:10:28Z
updated_at: 2021-11-16T01:10:28Z
author_association: OWNER
body:
  Also note that this demo data is using a SQL view to create the JSON arrays - the view is defined as such:

  ```sql
  CREATE VIEW ads_with_targets as
  select
    ads.*,
    json_group_array(targets.name) as target_names
  from
    ads
    join ad_targets on ad_targets.ad_id = ads.id
    join targets on ad_targets.target_id = targets.id
  group by
    ad_targets.ad_id;
  ```
  So running JSON faceting on top of that view is a pretty big ask!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1176#issuecomment-969616626
issue_url: https://api.github.com/repos/simonw/datasette/issues/1176
node_id: IC_kwDOBm6k_c45yyzy
created_at: 2021-11-16T01:29:13Z
updated_at: 2021-11-16T01:29:13Z
author_association: OWNER
body:
  I'm inclined to create a Sphinx reference documentation page for this, as I did for `sqlite-utils` here: https://sqlite-utils.datasette.io/en/stable/reference.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/448#issuecomment-969621662
issue_url: https://api.github.com/repos/simonw/datasette/issues/448
node_id: IC_kwDOBm6k_c45y0Ce
created_at: 2021-11-16T01:32:04Z
updated_at: 2021-11-16T01:32:04Z
author_association: OWNER
body:
  Tests are failing and I think it's because the facets come back in different orders, need a tie-breaker. https://github.com/simonw/datasette/runs/4219325197?check_suite_focus=true

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1012#issuecomment-970266123
issue_url: https://api.github.com/repos/simonw/datasette/issues/1012
node_id: IC_kwDOBm6k_c451RYL
created_at: 2021-11-16T13:18:36Z
updated_at: 2021-11-16T13:18:36Z
author_association: CONTRIBUTOR
body:
  Congratulations, looks like it went through! There was a bit of a hold-up
  on the JupyterLab ones, but it's semi automated: a dependabot pr to
  warehouse and a CI deploy, with a click in between.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-970553780
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: IC_kwDOBm6k_c452Xm0
created_at: 2021-11-16T18:30:51Z
updated_at: 2021-11-16T18:30:58Z
author_association: OWNER
body:
  OK, I'm ready to start working on this today.

  I'm going to go with a default representation that looks like this:

  ```json
  {
      "rows": [
          {"id": 1, "name": "One"},
          {"id": 2, "name": "Two"}
      ],
      "next_url": null
  }
  ```
  Note that there's no `count` - all it provides is the current selection of results and an indication as to how the next can be retrieved (`null` if there are no more results).

  I'll implement `?_extra=` to provide everything else.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/878#issuecomment-970624197
issue_url: https://api.github.com/repos/simonw/datasette/issues/878
node_id: IC_kwDOBm6k_c452ozF
created_at: 2021-11-16T19:49:05Z
updated_at: 2021-11-16T19:49:05Z
author_association: OWNER
body:
  Here's the latest version of my weird dependency injection async class:
  ```python
  import inspect

  class AsyncMeta(type):
      def __new__(cls, name, bases, attrs):
          # Decorate any items that are 'async def' methods
          _registry = {}
          new_attrs = {"_registry": _registry}
          for key, value in attrs.items():
              if inspect.iscoroutinefunction(value) and not value.__name__ == "resolve":
                  new_attrs[key] = make_method(value)
                  _registry[key] = new_attrs[key]
              else:
                  new_attrs[key] = value

          # Topological sort of _registry by parameter dependencies
          graph = {
              key: {
                  p for p in inspect.signature(method).parameters.keys()
                  if p != "self" and not p.startswith("_")
              }
              for key, method in _registry.items()
          }
          new_attrs["_graph"] = graph
          return super().__new__(cls, name, bases, new_attrs)


  def make_method(method):
      @wraps(method)
      async def inner(self, **kwargs):
          parameters = inspect.signature(method).parameters.keys()
          # Any parameters not provided by kwargs are resolved from registry
          to_resolve = [p for p in parameters if p not in kwargs and p != "self"]
          missing = [p for p in to_resolve if p not in self._registry]
          assert (
              not missing
          ), "The following DI parameters could not be found in the registry: {}".format(
              missing
          )
          results = {}
          results.update(kwargs)
          results.update(await self.resolve(to_resolve))
          return await method(self, **results)

      return inner


  bad = [0]

  class AsyncBase(metaclass=AsyncMeta):
      async def resolve(self, names):
          print(" resolve({})".format(names))
          results = {}
          # Resolve them in the correct order
          ts = TopologicalSorter()
          ts2 = TopologicalSorter()
          print(" names = ", names)
          print(" self._graph = ", self._graph)
          for name in names:
              if self._graph[name]:
                  ts.add(name, *self._graph[name])
                  ts2.add(name, *self._graph[name])
          print(" static_order =", tuple(ts2.static_order()))
          ts.prepare()
          while ts.is_active():
              print("  is_active, i = ", bad[0])
              bad[0] += 1
              if bad[0] > 20:
                  print("    Infinite loop?")
                  break
              nodes = ts.get_ready()
              print("  Do nodes:", nodes)
              awaitables = [self._registry[name](self, **{
                  k: v for k, v in results.items() if k in self._graph[name]
              }) for name in nodes]
              print("  awaitables: ", awaitables)
              awaitable_results = await asyncio.gather(*awaitables)
              results.update({
                  p[0].__name__: p[1] for p in zip(awaitables, awaitable_results)
              })
              print(results)
              for node in nodes:
                  ts.done(node)

          return results
  ```
  Example usage:
  ```python
  class Foo(AsyncBase):
      async def graa(self, boff):
          print("graa")
          return 5
      async def boff(self):
          print("boff")
          return 8
      async def other(self, boff, graa):
          print("other")
          return 5 + boff + graa

  foo = Foo()
  await foo.other()
  ```
  Output:
  ```
   resolve(['boff', 'graa'])
   names =  ['boff', 'graa']
   self._graph =  {'graa': {'boff'}, 'boff': set(), 'other': {'graa', 'boff'}}
   static_order = ('boff', 'graa')
    is_active, i =  0
    Do nodes: ('boff',)
    awaitables:  [<coroutine object Foo.boff at 0x10bd81a40>]
   resolve([])
   names =  []
   self._graph =  {'graa': {'boff'}, 'boff': set(), 'other': {'graa', 'boff'}}
   static_order = ()
  boff
  {'boff': 8}
    is_active, i =  1
    Do nodes: ('graa',)
    awaitables:  [<coroutine object Foo.graa at 0x10d66b340>]
   resolve([])
   names =  []
   self._graph =  {'graa': {'boff'}, 'boff': set(), 'other': {'graa', 'boff'}}
   static_order = ()
  graa
  {'boff': 8, 'graa': 5}
  other
  18
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/878#issuecomment-970655304
issue_url: https://api.github.com/repos/simonw/datasette/issues/878
node_id: IC_kwDOBm6k_c452wZI
created_at: 2021-11-16T20:32:16Z
updated_at: 2021-11-16T20:32:16Z
author_association: OWNER
body:
  This code is really fiddly. I just got to this version:
  ```python
  import asyncio
  from functools import wraps
  import inspect

  try:
      import graphlib
  except ImportError:
      from . import vendored_graphlib as graphlib


  class AsyncMeta(type):
      def __new__(cls, name, bases, attrs):
          # Decorate any items that are 'async def' methods
          _registry = {}
          new_attrs = {"_registry": _registry}
          for key, value in attrs.items():
              if inspect.iscoroutinefunction(value) and not value.__name__ == "resolve":
                  new_attrs[key] = make_method(value)
                  _registry[key] = new_attrs[key]
              else:
                  new_attrs[key] = value
          # Gather graph for later dependency resolution
          graph = {
              key: {
                  p
                  for p in inspect.signature(method).parameters.keys()
                  if p != "self" and not p.startswith("_")
              }
              for key, method in _registry.items()
          }
          new_attrs["_graph"] = graph
          return super().__new__(cls, name, bases, new_attrs)


  def make_method(method):
      @wraps(method)
      async def inner(self, _results=None, **kwargs):
          print("inner - _results=", _results)
          parameters = inspect.signature(method).parameters.keys()
          # Any parameters not provided by kwargs are resolved from registry
          to_resolve = [p for p in parameters if p not in kwargs and p != "self"]
          missing = [p for p in to_resolve if p not in self._registry]
          assert (
              not missing
          ), "The following DI parameters could not be found in the registry: {}".format(
              missing
          )
          results = {}
          results.update(kwargs)
          if to_resolve:
              resolved_parameters = await self.resolve(to_resolve, _results)
              results.update(resolved_parameters)
          return_value = await method(self, **results)
          if _results is not None:
              _results[method.__name__] = return_value
          return return_value

      return inner


  class AsyncBase(metaclass=AsyncMeta):
      async def resolve(self, names, results=None):
          print("\n  resolve: ", names)
          if results is None:
              results = {}

          # Resolve them in the correct order
          ts = graphlib.TopologicalSorter()
          for name in names:
              ts.add(name, *self._graph[name])
          ts.prepare()

          async def resolve_nodes(nodes):
              print("    resolve_nodes", nodes)
              print("    (current results = {})".format(repr(results)))
              awaitables = [
                  self._registry[name](
                      self,
                      _results=results,
                      **{k: v for k, v in results.items() if k in self._graph[name]},
                  )
                  for name in nodes
                  if name not in results
              ]
              print("    awaitables: ", awaitables)
              awaitable_results = await asyncio.gather(*awaitables)
              results.update(
                  {p[0].__name__: p[1] for p in zip(awaitables, awaitable_results)}
              )

          if not ts.is_active():
              # Nothing has dependencies - just resolve directly
              print("    no dependencies, resolve directly")
              await resolve_nodes(names)
          else:
              # Resolve in topological order
              while ts.is_active():
                  nodes = ts.get_ready()
                  print("    ts.get_ready() returned nodes:", nodes)
                  await resolve_nodes(nodes)
                  for node in nodes:
                      ts.done(node)

          print("  End of resolve(), returning", results)
          return {key: value for key, value in results.items() if key in names}
  ```
  With this test:
  ```python
  class Complex(AsyncBase):
      def __init__(self):
          self.log = []

      async def c(self):
          print("LOG: c")
          self.log.append("c")

      async def b(self, c):
          print("LOG: b")
          self.log.append("b")

      async def a(self, b, c):
          print("LOG: a")
          self.log.append("a")

      async def go(self, a):
          print("LOG: go")
          self.log.append("go")
          return self.log


  @pytest.mark.asyncio
  async def test_complex():
      result = await Complex().go()
      # 'c' should only be called once
      assert result == ["c", "b", "a", "go"]
  ```
  This test sometimes passes, and sometimes fails!

  Output for a pass:
  ```
  tests/test_asyncdi.py inner - _results= None

    resolve:  ['a']
      ts.get_ready() returned nodes: ('c', 'b')
      resolve_nodes ('c', 'b')
      (current results = {})
      awaitables:  [<coroutine object Complex.c at 0x1074ac890>, <coroutine object Complex.b at 0x1074ac820>]
  inner - _results= {}
  LOG: c
  inner - _results= {'c': None}

    resolve:  ['c']
      ts.get_ready() returned nodes: ('c',)
      resolve_nodes ('c',)
      (current results = {'c': None})
      awaitables:  []
    End of resolve(), returning {'c': None}
  LOG: b
      ts.get_ready() returned nodes: ('a',)
      resolve_nodes ('a',)
      (current results = {'c': None, 'b': None})
      awaitables:  [<coroutine object Complex.a at 0x1074ac7b0>]
  inner - _results= {'c': None, 'b': None}
  LOG: a
    End of resolve(), returning {'c': None, 'b': None, 'a': None}
  LOG: go
  ```
  Output for a fail:
  ```
  tests/test_asyncdi.py inner - _results= None

    resolve:  ['a']
      ts.get_ready() returned nodes: ('b', 'c')
      resolve_nodes ('b', 'c')
      (current results = {})
      awaitables:  [<coroutine object Complex.b at 0x10923c890>, <coroutine object Complex.c at 0x10923c820>]
  inner - _results= {}

    resolve:  ['c']
      ts.get_ready() returned nodes: ('c',)
      resolve_nodes ('c',)
      (current results = {})
      awaitables:  [<coroutine object Complex.c at 0x10923c6d0>]
  inner - _results= {}
  LOG: c
  inner - _results= {'c': None}
  LOG: c
    End of resolve(), returning {'c': None}
  LOG: b
      ts.get_ready() returned nodes: ('a',)
      resolve_nodes ('a',)
      (current results = {'c': None, 'b': None})
      awaitables:  [<coroutine object Complex.a at 0x10923c6d0>]
  inner - _results= {'c': None, 'b': None}
  LOG: a
    End of resolve(), returning {'c': None, 'b': None, 'a': None}
  LOG: go
  F

  =================================================================================================== FAILURES ===================================================================================================
  _________________________________________________________________________________________________ test_complex _________________________________________________________________________________________________

      @pytest.mark.asyncio
      async def test_complex():
          result = await Complex().go()
          # 'c' should only be called once
  >       assert result == ["c", "b", "a", "go"]
  E       AssertionError: assert ['c', 'c', 'b', 'a', 'go'] == ['c', 'b', 'a', 'go']
  E         At index 1 diff: 'c' != 'b'
  E         Left contains one more item: 'go'
  E         Use -v to get the full diff

  tests/test_asyncdi.py:48: AssertionError
  ================== short test summary info ================================
  FAILED tests/test_asyncdi.py::test_complex - AssertionError: assert ['c', 'c', 'b', 'a', 'go'] == ['c', 'b', 'a', 'go']
  ```
  I figured out why this is happening.

  `a` requires `b` and `c`

  `b` also requires `c`

  The code decides to run `b` and `c` in parallel.

  If `c` completes first, then when `b` runs it gets to use the already-calculated result for `c` - so it doesn't need to call `c` again.

  If `b` gets to that point before `c` does it also needs to call `c`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/878#issuecomment-970655927
issue_url: https://api.github.com/repos/simonw/datasette/issues/878
node_id: IC_kwDOBm6k_c452wi3
created_at: 2021-11-16T20:33:11Z
updated_at: 2021-11-16T20:33:11Z
author_association: OWNER
body:
  What should be happening here instead is it should resolve the full graph and notice that `c` is depended on by both `b` and `a` - so it should run `c` first, then run the next ones in parallel.

  So maybe the algorithm I'm inheriting from https://docs.python.org/3/library/graphlib.html isn't the correct algorithm?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/878#issuecomment-970657874
issue_url: https://api.github.com/repos/simonw/datasette/issues/878
node_id: IC_kwDOBm6k_c452xBS
created_at: 2021-11-16T20:36:01Z
updated_at: 2021-11-16T20:36:01Z
author_association: OWNER
body:
  My goal here is to calculate the most efficient way to resolve the different nodes, running them in parallel where possible.

  So for this class:

  ```python
  class Complex(AsyncBase):
      async def d(self):
          pass

      async def c(self):
          pass

      async def b(self, c, d):
          pass

      async def a(self, b, c):
          pass

      async def go(self, a):
          pass
  ```
  A call to `go()` should do this:

  - `c` and `d` in parallel
  - `b`
  - `a`
  - `go`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/878#issuecomment-970660299
issue_url: https://api.github.com/repos/simonw/datasette/issues/878
node_id: IC_kwDOBm6k_c452xnL
created_at: 2021-11-16T20:39:43Z
updated_at: 2021-11-16T20:42:27Z
author_association: OWNER
body:
  But that does seem to be the plan that `TopographicalSorter` provides:
  ```python
  graph = {"go": {"a"}, "a": {"b", "c"}, "b": {"c", "d"}}

  ts = TopologicalSorter(graph)
  ts.prepare()
  while ts.is_active():
      nodes = ts.get_ready()
      print(nodes)
      ts.done(*nodes)
  ```
  Outputs:
  ```
  ('c', 'd')
  ('b',)
  ('a',)
  ('go',)
  ```
  Also:
  ```python
  graph = {"go": {"d", "e", "f"}, "d": {"b", "c"}, "b": {"c"}}

  ts = TopologicalSorter(graph)
  ts.prepare()
  while ts.is_active():
      nodes = ts.get_ready()
      print(nodes)
      ts.done(*nodes)
  ```
  Gives:
  ```
  ('e', 'f', 'c')
  ('b',)
  ('d',)
  ('go',)
  ```
  I'm confident that `TopologicalSorter` is the way to do this. I think I need to rewrite my code to call it once to get that plan, then `await asyncio.gather(*nodes)` in turn to execute it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/878#issuecomment-970673085
issue_url: https://api.github.com/repos/simonw/datasette/issues/878
node_id: IC_kwDOBm6k_c4520u9
created_at: 2021-11-16T20:58:24Z
updated_at: 2021-11-16T20:58:24Z
author_association: OWNER
body:
  New test:
  ```python

  class Complex(AsyncBase):
      def __init__(self):
          self.log = []

      async def d(self):
          await asyncio.sleep(random() * 0.1)
          print("LOG: d")
          self.log.append("d")

      async def c(self):
          await asyncio.sleep(random() * 0.1)
          print("LOG: c")
          self.log.append("c")

      async def b(self, c, d):
          print("LOG: b")
          self.log.append("b")

      async def a(self, b, c):
          print("LOG: a")
          self.log.append("a")

      async def go(self, a):
          print("LOG: go")
          self.log.append("go")
          return self.log


  @pytest.mark.asyncio
  async def test_complex():
      result = await Complex().go()
      # 'c' should only be called once
      assert tuple(result) in (
          # c and d could happen in either order
          ("c", "d", "b", "a", "go"),
          ("d", "c", "b", "a", "go"),
      )
  ```
  And this code passes it:
  ```python
  import asyncio
  from functools import wraps
  import inspect

  try:
      import graphlib
  except ImportError:
      from . import vendored_graphlib as graphlib


  class AsyncMeta(type):
      def __new__(cls, name, bases, attrs):
          # Decorate any items that are 'async def' methods
          _registry = {}
          new_attrs = {"_registry": _registry}
          for key, value in attrs.items():
              if inspect.iscoroutinefunction(value) and not value.__name__ == "resolve":
                  new_attrs[key] = make_method(value)
                  _registry[key] = new_attrs[key]
              else:
                  new_attrs[key] = value
          # Gather graph for later dependency resolution
          graph = {
              key: {
                  p
                  for p in inspect.signature(method).parameters.keys()
                  if p != "self" and not p.startswith("_")
              }
              for key, method in _registry.items()
          }
          new_attrs["_graph"] = graph
          return super().__new__(cls, name, bases, new_attrs)


  def make_method(method):
      parameters = inspect.signature(method).parameters.keys()

      @wraps(method)
      async def inner(self, _results=None, **kwargs):
          print("\n{}.{}({}) _results={}".format(self, method.__name__, kwargs, _results))

          # Any parameters not provided by kwargs are resolved from registry
          to_resolve = [p for p in parameters if p not in kwargs and p != "self"]
          missing = [p for p in to_resolve if p not in self._registry]
          assert (
              not missing
          ), "The following DI parameters could not be found in the registry: {}".format(
              missing
          )

          results = {}
          results.update(kwargs)
          if to_resolve:
              resolved_parameters = await self.resolve(to_resolve, _results)
              results.update(resolved_parameters)
          return_value = await method(self, **results)
          if _results is not None:
              _results[method.__name__] = return_value
          return return_value

      return inner


  class AsyncBase(metaclass=AsyncMeta):
      async def resolve(self, names, results=None):
          print("\n  resolve: ", names)
          if results is None:
              results = {}

          # Come up with an execution plan, just for these nodes
          ts = graphlib.TopologicalSorter()
          to_do = set(names)
          done = set()
          while to_do:
              item = to_do.pop()
              dependencies = self._graph[item]
              ts.add(item, *dependencies)
              done.add(item)
              # Add any not-done dependencies to the queue
              to_do.update({k for k in dependencies if k not in done})

          ts.prepare()
          plan = []
          while ts.is_active():
              node_group = ts.get_ready()
              plan.append(node_group)
              ts.done(*node_group)

          print("plan:", plan)

          results = {}
          for node_group in plan:
              awaitables = [
                  self._registry[name](
                      self,
                      _results=results,
                      **{k: v for k, v in results.items() if k in self._graph[name]},
                  )
                  for name in node_group
              ]
              print("    results = ", results)
              print("    awaitables: ", awaitables)
              awaitable_results = await asyncio.gather(*awaitables)
              results.update(
                  {p[0].__name__: p[1] for p in zip(awaitables, awaitable_results)}
              )

          print("  End of resolve(), returning", results)
          return {key: value for key, value in results.items() if key in names}
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/878#issuecomment-970705738
issue_url: https://api.github.com/repos/simonw/datasette/issues/878
node_id: IC_kwDOBm6k_c4528tK
created_at: 2021-11-16T21:44:31Z
updated_at: 2021-11-16T21:44:31Z
author_association: OWNER
body:
  Wrote a TIL about what I learned using `TopologicalSorter`: https://til.simonwillison.net/python/graphlib-topologicalsorter

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1512#issuecomment-970718337
issue_url: https://api.github.com/repos/simonw/datasette/issues/1512
node_id: IC_kwDOBm6k_c452_yB
created_at: 2021-11-16T22:02:30Z
updated_at: 2021-11-16T22:02:30Z
author_association: OWNER
body:
  I've decided to make the clever `asyncio` dependency injection opt-in - so you can either decorate with `@inject` or you can set `inject_all = True` on the class - for example:
  ```python
  import asyncio
  from datasette.utils.asyncdi import AsyncBase, inject


  class Simple(AsyncBase):
      def __init__(self):
          self.log = []

      @inject
      async def two(self):
          self.log.append("two")

      @inject
      async def one(self, two):
          self.log.append("one")
          return self.log

      async def not_inject(self, one, two):
          return one + two


  class Complex(AsyncBase):
      inject_all = True

      def __init__(self):
          self.log = []

      async def b(self):
          self.log.append("b")

      async def a(self, b):
          self.log.append("a")

      async def go(self, a):
          self.log.append("go")
          return self.log
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1512#issuecomment-970718652
issue_url: https://api.github.com/repos/simonw/datasette/issues/1512
node_id: IC_kwDOBm6k_c452_28
created_at: 2021-11-16T22:02:59Z
updated_at: 2021-11-16T23:51:48Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1512?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1512](https://codecov.io/gh/simonw/datasette/pull/1512?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (8f757da) into [main](https://codecov.io/gh/simonw/datasette/commit/0156c6b5e52d541e93f0d68e9245f20ae83bc933?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0156c6b) will **decrease** coverage by `2.10%`.
  > The diff coverage is `36.20%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1512/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1512?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@            Coverage Diff             @@
  ##             main    #1512      +/-   ##
  ==========================================
  - Coverage   91.82%   89.72%   -2.11%     
  ==========================================
    Files          34       36       +2     
    Lines        4430     4604     +174     
  ==========================================
  + Hits         4068     4131      +63     
  - Misses        362      473     +111     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1512?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/utils/vendored\_graphlib.py](https://codecov.io/gh/simonw/datasette/pull/1512/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL3ZlbmRvcmVkX2dyYXBobGliLnB5) | `0.00% <0.00%> (ø)` | |
  | [datasette/utils/asyncdi.py](https://codecov.io/gh/simonw/datasette/pull/1512/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL2FzeW5jZGkucHk=) | `96.92% <96.92%> (ø)` | |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1512?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1512?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [0156c6b...8f757da](https://codecov.io/gh/simonw/datasette/pull/1512?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1513#issuecomment-970738130
issue_url: https://api.github.com/repos/simonw/datasette/issues/1513
node_id: IC_kwDOBm6k_c453EnS
created_at: 2021-11-16T22:32:19Z
updated_at: 2021-11-16T22:32:19Z
author_association: OWNER
body:
  I came up with the following query which seems to work!

  ```sql
  with cte as (
    select rowid, country, country_long, name, owner, primary_fuel
    from [global-power-plants]
  ),
  truncated as (
    select null as _facet, null as facet_name, null as facet_count, rowid, country, country_long, name, owner, primary_fuel
    from cte order by rowid limit 4
  ),
  country_long_facet as (
    select 'country_long' as _facet, country_long as facet_name, count(*) as facet_count,
    null, null, null, null, null, null
    from cte group by facet_name order by facet_count desc limit 3
  ),
  owner_facet as (
    select 'owner' as _facet, owner as facet_name, count(*) as facet_count,
    null, null, null, null, null, null
    from cte group by facet_name order by facet_count desc limit 3
  ),
  primary_fuel_facet as (
    select 'primary_fuel' as _facet, primary_fuel as facet_name, count(*) as facet_count,
    null, null, null, null, null, null
    from cte group by facet_name order by facet_count desc limit 3
  )
  select * from truncated
  union all select * from country_long_facet
  union all select * from owner_facet
  union all select * from primary_fuel_facet
  ```
  (Limits should be 101, 31, 31, 31 but I reduced size to get a shorter example table).

  Results [look like this](https://global-power-plants.datasettes.com/global-power-plants?sql=with+cte+as+%28%0D%0A++select+rowid%2C+country%2C+country_long%2C+name%2C+owner%2C+primary_fuel%0D%0A++from+%5Bglobal-power-plants%5D%0D%0A%29%2C%0D%0Atruncated+as+%28%0D%0A++select+null+as+_facet%2C+null+as+facet_name%2C+null+as+facet_count%2C+rowid%2C+country%2C+country_long%2C+name%2C+owner%2C+primary_fuel%0D%0A++from+cte+order+by+rowid+limit+4%0D%0A%29%2C%0D%0Acountry_long_facet+as+%28%0D%0A++select+%27country_long%27+as+_facet%2C+country_long+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Aowner_facet+as+%28%0D%0A++select+%27owner%27+as+_facet%2C+owner+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Aprimary_fuel_facet+as+%28%0D%0A++select+%27primary_fuel%27+as+_facet%2C+primary_fuel+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%0D%0Aselect+*+from+truncated%0D%0Aunion+all+select+*+from+country_long_facet%0D%0Aunion+all+select+*+from+owner_facet%0D%0Aunion+all+select+*+from+primary_fuel_facet):

  _facet | facet_name | facet_count | rowid | country | country_long | name | owner | primary_fuel
  -- | -- | -- | -- | -- | -- | -- | -- | --
    |   |   | 1 | AFG | Afghanistan | Kajaki Hydroelectric Power Plant Afghanistan |   | Hydro
    |   |   | 2 | AFG | Afghanistan | Kandahar DOG |   | Solar
    |   |   | 3 | AFG | Afghanistan | Kandahar JOL |   | Solar
    |   |   | 4 | AFG | Afghanistan | Mahipar Hydroelectric Power Plant Afghanistan |   | Hydro
  country_long | United States of America | 8688 |   |   |   |   |   |  
  country_long | China | 4235 |   |   |   |   |   |  
  country_long | United Kingdom | 2603 |   |   |   |   |   |  
  owner |   | 14112 |   |   |   |   |   |  
  owner | Lightsource Renewable Energy | 120 |   |   |   |   |   |  
  owner | Cypress Creek Renewables | 109 |   |   |   |   |   |  
  primary_fuel | Solar | 9662 |   |   |   |   |   |  
  primary_fuel | Hydro | 7155 |   |   |   |   |   |  
  primary_fuel | Wind | 5188 |   |   |   |   |   |  

  This is a neat proof of concept. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1513#issuecomment-970742415
issue_url: https://api.github.com/repos/simonw/datasette/issues/1513
node_id: IC_kwDOBm6k_c453FqP
created_at: 2021-11-16T22:37:14Z
updated_at: 2021-11-16T22:37:14Z
author_association: OWNER
body:
  The query takes 42.794ms to run.

  Here's the equivalent page using separate queries: https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_facet_size=3&_size=2&_nocount=1

  Annoyingly I can't disable facet suggestions but keep facets.

  I'm going to turn on tracing so I can see how long the separate queries took.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1513#issuecomment-970758179
issue_url: https://api.github.com/repos/simonw/datasette/issues/1513
node_id: IC_kwDOBm6k_c453Jgj
created_at: 2021-11-16T22:47:38Z
updated_at: 2021-11-16T22:47:38Z
author_association: OWNER
body:
  Trace now enabled: https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_facet_size=3&_size=2&_nocount=1&_trace=1

  Here are the relevant traces:
  ```json
  [
      {
        "type": "sql",
        "start": 31.214430154,
        "end": 31.214817089,
        "duration_ms": 0.3869350000016425,
        "traceback": [
          "  File \"/usr/local/lib/python3.8/site-packages/datasette/views/base.py\", line 262, in get\n    return await self.view_get(\n",
          "  File \"/usr/local/lib/python3.8/site-packages/datasette/views/base.py\", line 477, in view_get\n    response_or_template_contexts = await self.data(\n",
          "  File \"/usr/local/lib/python3.8/site-packages/datasette/views/table.py\", line 705, in data\n    results = await db.execute(sql, params, truncate=True, **extra_args)\n"
        ],
        "database": "global-power-plants",
        "sql": "select rowid, country, country_long, name, gppd_idnr, capacity_mw, latitude, longitude, primary_fuel, other_fuel1, other_fuel2, other_fuel3, commissioning_year, owner, source, url, geolocation_source, wepp_id, year_of_capacity_data, generation_gwh_2013, generation_gwh_2014, generation_gwh_2015, generation_gwh_2016, generation_gwh_2017, generation_data_source, estimated_generation_gwh from [global-power-plants] order by rowid limit 3",
        "params": {}
      },
      {
        "type": "sql",
        "start": 31.215234586,
        "end": 31.220110342,
        "duration_ms": 4.875756000000564,
        "traceback": [
          "  File \"/usr/local/lib/python3.8/site-packages/datasette/views/table.py\", line 760, in data\n    ) = await facet.facet_results()\n",
          "  File \"/usr/local/lib/python3.8/site-packages/datasette/facets.py\", line 212, in facet_results\n    facet_rows_results = await self.ds.execute(\n",
          "  File \"/usr/local/lib/python3.8/site-packages/datasette/app.py\", line 634, in execute\n    return await self.databases[db_name].execute(\n"
        ],
        "database": "global-power-plants",
        "sql": "select country_long as value, count(*) as count from (\n                    select rowid, country, country_long, name, gppd_idnr, capacity_mw, latitude, longitude, primary_fuel, other_fuel1, other_fuel2, other_fuel3, commissioning_year, owner, source, url, geolocation_source, wepp_id, year_of_capacity_data, generation_gwh_2013, generation_gwh_2014, generation_gwh_2015, generation_gwh_2016, generation_gwh_2017, generation_data_source, estimated_generation_gwh from [global-power-plants] \n                )\n                where country_long is not null\n                group by country_long order by count desc, value limit 4",
        "params": []
      },
      {
        "type": "sql",
        "start": 31.221062485,
        "end": 31.228968364,
        "duration_ms": 7.905878999999061,
        "traceback": [
          "  File \"/usr/local/lib/python3.8/site-packages/datasette/views/table.py\", line 760, in data\n    ) = await facet.facet_results()\n",
          "  File \"/usr/local/lib/python3.8/site-packages/datasette/facets.py\", line 212, in facet_results\n    facet_rows_results = await self.ds.execute(\n",
          "  File \"/usr/local/lib/python3.8/site-packages/datasette/app.py\", line 634, in execute\n    return await self.databases[db_name].execute(\n"
        ],
        "database": "global-power-plants",
        "sql": "select owner as value, count(*) as count from (\n                    select rowid, country, country_long, name, gppd_idnr, capacity_mw, latitude, longitude, primary_fuel, other_fuel1, other_fuel2, other_fuel3, commissioning_year, owner, source, url, geolocation_source, wepp_id, year_of_capacity_data, generation_gwh_2013, generation_gwh_2014, generation_gwh_2015, generation_gwh_2016, generation_gwh_2017, generation_data_source, estimated_generation_gwh from [global-power-plants] \n                )\n                where owner is not null\n                group by owner order by count desc, value limit 4",
        "params": []
      },
      {
        "type": "sql",
        "start": 31.229809757,
        "end": 31.253902162,
        "duration_ms": 24.09240499999754,
        "traceback": [
          "  File \"/usr/local/lib/python3.8/site-packages/datasette/views/table.py\", line 760, in data\n    ) = await facet.facet_results()\n",
          "  File \"/usr/local/lib/python3.8/site-packages/datasette/facets.py\", line 212, in facet_results\n    facet_rows_results = await self.ds.execute(\n",
          "  File \"/usr/local/lib/python3.8/site-packages/datasette/app.py\", line 634, in execute\n    return await self.databases[db_name].execute(\n"
        ],
        "database": "global-power-plants",
        "sql": "select primary_fuel as value, count(*) as count from (\n                    select rowid, country, country_long, name, gppd_idnr, capacity_mw, latitude, longitude, primary_fuel, other_fuel1, other_fuel2, other_fuel3, commissioning_year, owner, source, url, geolocation_source, wepp_id, year_of_capacity_data, generation_gwh_2013, generation_gwh_2014, generation_gwh_2015, generation_gwh_2016, generation_gwh_2017, generation_data_source, estimated_generation_gwh from [global-power-plants] \n                )\n                where primary_fuel is not null\n                group by primary_fuel order by count desc, value limit 4",
        "params": []
      },
      {
        "type": "sql",
        "start": 31.255699745,
        "end": 31.256243889,
        "duration_ms": 0.544143999999136,
        "traceback": [
          "  File \"/usr/local/lib/python3.8/site-packages/datasette/facets.py\", line 145, in suggest\n    row_count = await self.get_row_count()\n",
          "  File \"/usr/local/lib/python3.8/site-packages/datasette/facets.py\", line 132, in get_row_count\n    await self.ds.execute(\n",
          "  File \"/usr/local/lib/python3.8/site-packages/datasette/app.py\", line 634, in execute\n    return await self.databases[db_name].execute(\n"
        ],
        "database": "global-power-plants",
        "sql": "select count(*) from (select rowid, country, country_long, name, gppd_idnr, capacity_mw, latitude, longitude, primary_fuel, other_fuel1, other_fuel2, other_fuel3, commissioning_year, owner, source, url, geolocation_source, wepp_id, year_of_capacity_data, generation_gwh_2013, generation_gwh_2014, generation_gwh_2015, generation_gwh_2016, generation_gwh_2017, generation_data_source, estimated_generation_gwh from [global-power-plants] )",
        "params": []
      }
  ]
  ```
  ```
  fetch rows: 0.3869350000016425 ms
  facet country_long: 4.875756000000564 ms
  facet owner: 7.905878999999061 ms
  facet primary_fuel: 24.09240499999754 ms
  count: 0.544143999999136 ms
  ```
  Total = 37.8ms

  I modified the query to include the total count as well: https://global-power-plants.datasettes.com/global-power-plants?sql=with+cte+as+%28%0D%0A++select+rowid%2C+country%2C+country_long%2C+name%2C+owner%2C+primary_fuel%0D%0A++from+%5Bglobal-power-plants%5D%0D%0A%29%2C%0D%0Atruncated+as+%28%0D%0A++select+null+as+_facet%2C+null+as+facet_name%2C+null+as+facet_count%2C+rowid%2C+country%2C+country_long%2C+name%2C+owner%2C+primary_fuel%0D%0A++from+cte+order+by+rowid+limit+4%0D%0A%29%2C%0D%0Acountry_long_facet+as+%28%0D%0A++select+%27country_long%27+as+_facet%2C+country_long+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Aowner_facet+as+%28%0D%0A++select+%27owner%27+as+_facet%2C+owner+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Aprimary_fuel_facet+as+%28%0D%0A++select+%27primary_fuel%27+as+_facet%2C+primary_fuel+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Atotal_count+as+%28%0D%0A++select+%27COUNT%27+as+_facet%2C+%27%27+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte%0D%0A%29%0D%0Aselect+*+from+truncated%0D%0Aunion+all+select+*+from+country_long_facet%0D%0Aunion+all+select+*+from+owner_facet%0D%0Aunion+all+select+*+from+primary_fuel_facet%0D%0Aunion+all+select+*+from+total_count&_trace=1

  ```sql
  with cte as (
    select rowid, country, country_long, name, owner, primary_fuel
    from [global-power-plants]
  ),
  truncated as (
    select null as _facet, null as facet_name, null as facet_count, rowid, country, country_long, name, owner, primary_fuel
    from cte order by rowid limit 4
  ),
  country_long_facet as (
    select 'country_long' as _facet, country_long as facet_name, count(*) as facet_count,
    null, null, null, null, null, null
    from cte group by facet_name order by facet_count desc limit 3
  ),
  owner_facet as (
    select 'owner' as _facet, owner as facet_name, count(*) as facet_count,
    null, null, null, null, null, null
    from cte group by facet_name order by facet_count desc limit 3
  ),
  primary_fuel_facet as (
    select 'primary_fuel' as _facet, primary_fuel as facet_name, count(*) as facet_count,
    null, null, null, null, null, null
    from cte group by facet_name order by facet_count desc limit 3
  ),
  total_count as (
    select 'COUNT' as _facet, '' as facet_name, count(*) as facet_count,
    null, null, null, null, null, null
    from cte
  )
  select * from truncated
  union all select * from country_long_facet
  union all select * from owner_facet
  union all select * from primary_fuel_facet
  union all select * from total_count
  ```
  The trace says that query took 34.801436999998714 ms.

  To my huge surprise, this convoluted optimization only shaves the sum query time down from 37.8ms to 34.8ms!

  That entire database file is just 11.1 MB though. Maybe it would make a meaningful difference on something larger?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1513#issuecomment-970766486
issue_url: https://api.github.com/repos/simonw/datasette/issues/1513
node_id: IC_kwDOBm6k_c453LiW
created_at: 2021-11-16T22:52:56Z
updated_at: 2021-11-16T22:56:07Z
author_association: OWNER
body:
  https://covid-19.datasettes.com/covid is 805.2MB

  https://covid-19.datasettes.com/covid/ny_times_us_counties?_trace=1&_facet_size=3&_size=2

  Equivalent SQL:

  https://covid-19.datasettes.com/covid?sql=with+cte+as+%28%0D%0A++select+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+ny_times_us_counties%0D%0A%29%2C%0D%0Atruncated+as+%28%0D%0A++select+null+as+_facet%2C+null+as+facet_name%2C+null+as+facet_count%2C+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+cte+order+by+date+desc+limit+4%0D%0A%29%2C%0D%0Astate_facet+as+%28%0D%0A++select+%27state%27+as+_facet%2C+state+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Afips_facet+as+%28%0D%0A++select+%27fips%27+as+_facet%2C+fips+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Acounty_facet+as+%28%0D%0A++select+%27county%27+as+_facet%2C+county+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Atotal_count+as+%28%0D%0A++select+%27COUNT%27+as+_facet%2C+%27%27+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte%0D%0A%29%0D%0Aselect+*+from+truncated%0D%0Aunion+all+select+*+from+state_facet%0D%0Aunion+all+select+*+from+fips_facet%0D%0Aunion+all+select+*+from+county_facet%0D%0Aunion+all+select+*+from+total_count

  ```sql
  with cte as (
    select rowid, date, county, state, fips, cases, deaths
    from ny_times_us_counties
  ),
  truncated as (
    select null as _facet, null as facet_name, null as facet_count, rowid, date, county, state, fips, cases, deaths
    from cte order by date desc limit 4
  ),
  state_facet as (
    select 'state' as _facet, state as facet_name, count(*) as facet_count,
    null, null, null, null, null, null, null
    from cte group by facet_name order by facet_count desc limit 3
  ),
  fips_facet as (
    select 'fips' as _facet, fips as facet_name, count(*) as facet_count,
    null, null, null, null, null, null, null
    from cte group by facet_name order by facet_count desc limit 3
  ),
  county_facet as (
    select 'county' as _facet, county as facet_name, count(*) as facet_count,
    null, null, null, null, null, null, null
    from cte group by facet_name order by facet_count desc limit 3
  ),
  total_count as (
    select 'COUNT' as _facet, '' as facet_name, count(*) as facet_count,
    null, null, null, null, null, null, null
    from cte
  )
  select * from truncated
  union all select * from state_facet
  union all select * from fips_facet
  union all select * from county_facet
  union all select * from total_count
  ```

  _facet | facet_name | facet_count | rowid | date | county | state | fips | cases | deaths
  -- | -- | -- | -- | -- | -- | -- | -- | -- | --
    |   |   | 1917344 | 2021-11-15 | Autauga | Alabama | 1001 | 10407 | 154
    |   |   | 1917345 | 2021-11-15 | Baldwin | Alabama | 1003 | 37875 | 581
    |   |   | 1917346 | 2021-11-15 | Barbour | Alabama | 1005 | 3648 | 79
    |   |   | 1917347 | 2021-11-15 | Bibb | Alabama | 1007 | 4317 | 92
  state | Texas | 148028 |   |   |   |   |   |   |  
  state | Georgia | 96249 |   |   |   |   |   |   |  
  state | Virginia | 79315 |   |   |   |   |   |   |  
  fips |   | 17580 |   |   |   |   |   |   |  
  fips | 53061 | 665 |   |   |   |   |   |   |  
  fips | 17031 | 662 |   |   |   |   |   |   |  
  county | Washington | 18666 |   |   |   |   |   |   |  
  county | Unknown | 15840 |   |   |   |   |   |   |  
  county | Jefferson | 15637 |   |   |   |   |   |   |  
  COUNT |   | 1920593 |   |   |   |   |   |   |  

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1513#issuecomment-970767952
issue_url: https://api.github.com/repos/simonw/datasette/issues/1513
node_id: IC_kwDOBm6k_c453L5Q
created_at: 2021-11-16T22:53:52Z
updated_at: 2021-11-16T22:53:52Z
author_association: OWNER
body:
  It's going to take another 15 minutes for the build to finish and deploy the version with `_trace=1`: https://github.com/simonw/covid-19-datasette/actions/runs/1469150112

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1513#issuecomment-970770304
issue_url: https://api.github.com/repos/simonw/datasette/issues/1513
node_id: IC_kwDOBm6k_c453MeA
created_at: 2021-11-16T22:55:19Z
updated_at: 2021-11-16T22:55:19Z
author_association: OWNER
body:
  (One thing I really like about this pattern is that it should work exactly the same when used to facet the results of arbitrary SQL queries as it does when faceting results from the table page.)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1513#issuecomment-970780866
issue_url: https://api.github.com/repos/simonw/datasette/issues/1513
node_id: IC_kwDOBm6k_c453PDC
created_at: 2021-11-16T23:01:57Z
updated_at: 2021-11-16T23:01:57Z
author_association: OWNER
body:
  One disadvantage to this approach: if you have a SQL time limit of 1s and it takes 0.9s to return the rows but then 0.5s to calculate each of the requested facets the entire query will exceed the time limit.

  Could work around this by catching that error and then re-running the query just for the rows, but that would result in the user having to wait longer for the results.

  Could try to remember if that has happened using an in-memory Python data structure and skip the faceting optimization if it's caused problems in the past? That seems a bit gross.

  Maybe this becomes an opt-in optimization you can request in your `metadata.json` setting for that table, which massively increases the time limit? That's a bit weird too - now there are two separate implementations of the faceting logic, which had better have a REALLY big pay-off to be worth maintaining.

  What if we kept the query that returns the rows to be displayed on the page separate from the facets, but then executed all of the facets together using this method such that the `cte` only (presumably) has to be calculated once? That would still lead to multiple facets potentially exceeding the SQL time limit when single facets would not have.

  Maybe a better optimization would be to move facets to happening via `fetch()` calls from the client, so the user gets to see their rows instantly and the facets then appear as and when they are available (though it would cause page jank).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1513#issuecomment-970827674
issue_url: https://api.github.com/repos/simonw/datasette/issues/1513
node_id: IC_kwDOBm6k_c453aea
created_at: 2021-11-16T23:26:58Z
updated_at: 2021-11-16T23:26:58Z
author_association: OWNER
body:
  With trace.

  https://covid-19.datasettes.com/covid/ny_times_us_counties?_trace=1&_facet_size=3&_size=2&_trace=1 shows the following:

  ```
  fetch rows: 0.41762600005768036 ms
  facet state: 284.30423800000426 ms
  facet county: 273.2565999999679 ms
  facet fips: 197.80996999998024 ms
  ```
  = 755.78843400001ms total

  It didn't run a count because that's the homepage and the count is cached. So I dropped the count from the query and ran it:

  https://covid-19.datasettes.com/covid?sql=with+cte+as+(%0D%0A++select+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+ny_times_us_counties%0D%0A)%2C%0D%0Atruncated+as+(%0D%0A++select+null+as+_facet%2C+null+as+facet_name%2C+null+as+facet_count%2C+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+cte+order+by+date+desc+limit+4%0D%0A)%2C%0D%0Astate_facet+as+(%0D%0A++select+%27state%27+as+_facet%2C+state+as+facet_name%2C+count(*)+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A)%2C%0D%0Afips_facet+as+(%0D%0A++select+%27fips%27+as+_facet%2C+fips+as+facet_name%2C+count(*)+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A)%2C%0D%0Acounty_facet+as+(%0D%0A++select+%27county%27+as+_facet%2C+county+as+facet_name%2C+count(*)+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A)%0D%0Aselect+*+from+truncated%0D%0Aunion+all+select+*+from+state_facet%0D%0Aunion+all+select+*+from+fips_facet%0D%0Aunion+all+select+*+from+county_facet&_trace=1

  Shows 649.4359889999259 ms for the query - compared to 755.78843400001ms for the separate. So it saved about 100ms.

  Still not a huge difference though!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1513#issuecomment-970845844
issue_url: https://api.github.com/repos/simonw/datasette/issues/1513
node_id: IC_kwDOBm6k_c453e6U
created_at: 2021-11-16T23:35:38Z
updated_at: 2021-11-16T23:35:38Z
author_association: OWNER
body:
  I tried adding `cases > 10000` but the SQL query now takes too long - so moving this to my laptop.

  ```
  cd /tmp
  wget https://covid-19.datasettes.com/covid.db
  datasette covid.db \
    --setting facet_time_limit_ms 10000 \
    --setting sql_time_limit_ms 10000 \
    --setting trace_debug 1
  ```
  `http://127.0.0.1:8006/covid/ny_times_us_counties?_trace=1&_facet_size=3&_size=2&cases__gt=10000` shows in the traces:

  ```json
  [
      {
        "type": "sql",
        "start": 12.693033525,
        "end": 12.694056904,
        "duration_ms": 1.0233789999993803,
        "traceback": [
          "  File \"/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/base.py\", line 262, in get\n    return await self.view_get(\n",
          "  File \"/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/base.py\", line 477, in view_get\n    response_or_template_contexts = await self.data(\n",
          "  File \"/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/table.py\", line 705, in data\n    results = await db.execute(sql, params, truncate=True, **extra_args)\n"
        ],
        "database": "covid",
        "sql": "select rowid, date, county, state, fips, cases, deaths from ny_times_us_counties where \"cases\" > :p0 order by rowid limit 3",
        "params": {
          "p0": 10000
        }
      },
      {
        "type": "sql",
        "start": 12.694285093,
        "end": 12.814936275,
        "duration_ms": 120.65118200000136,
        "traceback": [
          "  File \"/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/base.py\", line 262, in get\n    return await self.view_get(\n",
          "  File \"/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/base.py\", line 477, in view_get\n    response_or_template_contexts = await self.data(\n",
          "  File \"/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/table.py\", line 723, in data\n    count_rows = list(await db.execute(count_sql, from_sql_params))\n"
        ],
        "database": "covid",
        "sql": "select count(*) from ny_times_us_counties where \"cases\" > :p0",
        "params": {
          "p0": 10000
        }
      },
      {
        "type": "sql",
        "start": 12.818812089,
        "end": 12.851172544,
        "duration_ms": 32.360455000000954,
        "traceback": [
          "  File \"/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/table.py\", line 856, in data\n    suggested_facets.extend(await facet.suggest())\n",
          "  File \"/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/facets.py\", line 164, in suggest\n    distinct_values = await self.ds.execute(\n",
          "  File \"/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/app.py\", line 634, in execute\n    return await self.databases[db_name].execute(\n"
        ],
        "database": "covid",
        "sql": "select county, count(*) as n from (\n                    select rowid, date, county, state, fips, cases, deaths from ny_times_us_counties where \"cases\" > :p0 \n                ) where county is not null\n                group by county\n                limit 4",
        "params": {
          "p0": 10000
        }
      },
      {
        "type": "sql",
        "start": 12.851418868,
        "end": 12.871268359,
        "duration_ms": 19.84949100000044,
        "traceback": [
          "  File \"/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/table.py\", line 856, in data\n    suggested_facets.extend(await facet.suggest())\n",
          "  File \"/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/facets.py\", line 164, in suggest\n    distinct_values = await self.ds.execute(\n",
          "  File \"/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/app.py\", line 634, in execute\n    return await self.databases[db_name].execute(\n"
        ],
        "database": "covid",
        "sql": "select state, count(*) as n from (\n                    select rowid, date, county, state, fips, cases, deaths from ny_times_us_counties where \"cases\" > :p0 \n                ) where state is not null\n                group by state\n                limit 4",
        "params": {
          "p0": 10000
        }
      },
      {
        "type": "sql",
        "start": 12.871497655,
        "end": 12.897715027,
        "duration_ms": 26.217371999999628,
        "traceback": [
          "  File \"/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/table.py\", line 856, in data\n    suggested_facets.extend(await facet.suggest())\n",
          "  File \"/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/facets.py\", line 164, in suggest\n    distinct_values = await self.ds.execute(\n",
          "  File \"/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/app.py\", line 634, in execute\n    return await self.databases[db_name].execute(\n"
        ],
        "database": "covid",
        "sql": "select fips, count(*) as n from (\n                    select rowid, date, county, state, fips, cases, deaths from ny_times_us_counties where \"cases\" > :p0 \n                ) where fips is not null\n                group by fips\n                limit 4",
        "params": {
          "p0": 10000
        }
      }
  ]
  ```
  So that's:
  ```
  fetch rows: 1.0233789999993803 ms
  count: 120.65118200000136 ms
  facet county: 32.360455000000954 ms
  facet state: 19.84949100000044 ms
  facet fips: 26.217371999999628 ms
  ```
  = 200.1 ms total

  Compared to: `http://127.0.0.1:8006/covid?sql=with+cte+as+(%0D%0A++select+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+ny_times_us_counties%0D%0A)%2C%0D%0Atruncated+as+(%0D%0A++select+null+as+_facet%2C+null+as+facet_name%2C+null+as+facet_count%2C+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+cte+order+by+date+desc+limit+4%0D%0A)%2C%0D%0Astate_facet+as+(%0D%0A++select+%27state%27+as+_facet%2C+state+as+facet_name%2C+count(*)+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A)%2C%0D%0Afips_facet+as+(%0D%0A++select+%27fips%27+as+_facet%2C+fips+as+facet_name%2C+count(*)+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A)%2C%0D%0Acounty_facet+as+(%0D%0A++select+%27county%27+as+_facet%2C+county+as+facet_name%2C+count(*)+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A)%0D%0Aselect+*+from+truncated%0D%0Aunion+all+select+*+from+state_facet%0D%0Aunion+all+select+*+from+fips_facet%0D%0Aunion+all+select+*+from+county_facet&_trace=1`

  Which is 353ms total.

  The separate queries ran faster! Really surprising result there.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1513#issuecomment-970853917
issue_url: https://api.github.com/repos/simonw/datasette/issues/1513
node_id: IC_kwDOBm6k_c453g4d
created_at: 2021-11-16T23:41:01Z
updated_at: 2021-11-16T23:41:01Z
author_association: OWNER
body:
  One very interesting difference between the two: on the single giant query page:

  ```json
  {
    "request_duration_ms": 376.4317020000476,
    "sum_trace_duration_ms": 370.0828700000329,
    "num_traces": 5
  }
  ```
  And on the page that uses separate queries:
  ```json
  {
    "request_duration_ms": 819.012272000009,
    "sum_trace_duration_ms": 201.52852100000018,
    "num_traces": 19
  }
  ```
  The separate pages page takes 819ms total to render the page, but spends 201ms across 19 SQL queries.

  The single big query takes 376ms total to render the page, spending 370ms in 5 queries

  <details><summary>Those 5 queries, if you're interested</summary>

  ```sql
  select database_name, schema_version from databases
  PRAGMA schema_version
  PRAGMA schema_version
  explain with cte as (\r\n  select rowid, date, county, state, fips, cases, deaths\r\n  from ny_times_us_counties\r\n),\r\ntruncated as (\r\n  select null as _facet, null as facet_name, null as facet_count, rowid, date, county, state, fips, cases, deaths\r\n  from cte order by date desc limit 4\r\n),\r\nstate_facet as (\r\n  select 'state' as _facet, state as facet_name, count(*) as facet_count,\r\n  null, null, null, null, null, null, null\r\n  from cte group by facet_name order by facet_count desc limit 3\r\n),\r\nfips_facet as (\r\n  select 'fips' as _facet, fips as facet_name, count(*) as facet_count,\r\n  null, null, null, null, null, null, null\r\n  from cte group by facet_name order by facet_count desc limit 3\r\n),\r\ncounty_facet as (\r\n  select 'county' as _facet, county as facet_name, count(*) as facet_count,\r\n  null, null, null, null, null, null, null\r\n  from cte group by facet_name order by facet_count desc limit 3\r\n)\r\nselect * from truncated\r\nunion all select * from state_facet\r\nunion all select * from fips_facet\r\nunion all select * from county_facet
  with cte as (\r\n  select rowid, date, county, state, fips, cases, deaths\r\n  from ny_times_us_counties\r\n),\r\ntruncated as (\r\n  select null as _facet, null as facet_name, null as facet_count, rowid, date, county, state, fips, cases, deaths\r\n  from cte order by date desc limit 4\r\n),\r\nstate_facet as (\r\n  select 'state' as _facet, state as facet_name, count(*) as facet_count,\r\n  null, null, null, null, null, null, null\r\n  from cte group by facet_name order by facet_count desc limit 3\r\n),\r\nfips_facet as (\r\n  select 'fips' as _facet, fips as facet_name, count(*) as facet_count,\r\n  null, null, null, null, null, null, null\r\n  from cte group by facet_name order by facet_count desc limit 3\r\n),\r\ncounty_facet as (\r\n  select 'county' as _facet, county as facet_name, count(*) as facet_count,\r\n  null, null, null, null, null, null, null\r\n  from cte group by facet_name order by facet_count desc limit 3\r\n)\r\nselect * from truncated\r\nunion all select * from state_facet\r\nunion all select * from fips_facet\r\nunion all select * from county_facet
  ```
  </details>

  All of that additional non-SQL overhead must be stuff relating to Python and template rendering code running on the page. I'm really surprised at how much overhead that is! This is worth researching separately.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1513#issuecomment-970855084
issue_url: https://api.github.com/repos/simonw/datasette/issues/1513
node_id: IC_kwDOBm6k_c453hKs
created_at: 2021-11-16T23:41:46Z
updated_at: 2021-11-16T23:41:46Z
author_association: OWNER
body:
  Conclusion: using a giant convoluted CTE and UNION ALL query to attempt to calculate facets at the same time as retrieving rows is a net LOSS for performance! Very surprised to see that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1512#issuecomment-970857411
issue_url: https://api.github.com/repos/simonw/datasette/issues/1512
node_id: IC_kwDOBm6k_c453hvD
created_at: 2021-11-16T23:43:21Z
updated_at: 2021-11-16T23:43:21Z
author_association: OWNER
body:
  ```
  E     File "/home/runner/work/datasette/datasette/datasette/utils/vendored_graphlib.py", line 56
  E       if (result := self._node2info.get(node)) is None:
  E                  ^
  E   SyntaxError: invalid syntax
  ```
  Oh no - the vendored code I use has `:=` so doesn't work on Python 3.6! Will have to backport it more thoroughly.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1512#issuecomment-970861628
issue_url: https://api.github.com/repos/simonw/datasette/issues/1512
node_id: IC_kwDOBm6k_c453iw8
created_at: 2021-11-16T23:46:07Z
updated_at: 2021-11-16T23:46:07Z
author_association: OWNER
body:
  I made the changes locally and tested them with Python 3.6 like so:
  ```
  cd /tmp
  mkdir v
  cd v
  pipenv shell --python=python3.6
  cd ~/Dropbox/Development/datasette
  pip install -e '.[test]'
  pytest tests/test_asyncdi.py
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1512#issuecomment-971055677
issue_url: https://api.github.com/repos/simonw/datasette/issues/1512
node_id: IC_kwDOBm6k_c454SI9
created_at: 2021-11-17T01:39:25Z
updated_at: 2021-11-17T01:39:25Z
author_association: OWNER
body:
  https://github.com/simonw/asyncinject version 0.1a0 is now live on PyPI: https://pypi.org/project/asyncinject/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/878#issuecomment-971057553
issue_url: https://api.github.com/repos/simonw/datasette/issues/878
node_id: IC_kwDOBm6k_c454SmR
created_at: 2021-11-17T01:40:45Z
updated_at: 2021-11-17T01:40:45Z
author_association: OWNER
body:
  I shipped that code as a new library, `asyncinject`: https://pypi.org/project/asyncinject/ - I'll open a new PR to attempt to refactor `TableView` to use it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/878#issuecomment-971209475
issue_url: https://api.github.com/repos/simonw/datasette/issues/878
node_id: IC_kwDOBm6k_c4543sD
created_at: 2021-11-17T05:41:42Z
updated_at: 2021-11-17T05:41:42Z
author_association: OWNER
body:
  I'm going to build a brand new implementation of the `TableView` class that doesn't subclass `BaseView` at all, instead using `asyncinject`. If I'm lucky that will clean up the grungiest part of the codebase.

  I can maybe even run the tests against old `TableView` and `TableView2` to check that they behave the same.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1514#issuecomment-971575746
issue_url: https://api.github.com/repos/simonw/datasette/issues/1514
node_id: IC_kwDOBm6k_c456RHC
created_at: 2021-11-17T13:18:58Z
updated_at: 2021-11-17T13:18:58Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1514?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1514](https://codecov.io/gh/simonw/datasette/pull/1514?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (b02c35a) into [main](https://codecov.io/gh/simonw/datasette/commit/0156c6b5e52d541e93f0d68e9245f20ae83bc933?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0156c6b) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1514/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1514?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1514   +/-   ##
  =======================================
    Coverage   91.82%   91.82%           
  =======================================
    Files          34       34           
    Lines        4430     4430           
  =======================================
    Hits         4068     4068           
    Misses        362      362           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1514?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1514?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [0156c6b...b02c35a](https://codecov.io/gh/simonw/datasette/pull/1514?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1516#issuecomment-972858458
issue_url: https://api.github.com/repos/simonw/datasette/issues/1516
node_id: IC_kwDOBm6k_c45_KRa
created_at: 2021-11-18T13:19:01Z
updated_at: 2021-11-18T13:19:01Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1516?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1516](https://codecov.io/gh/simonw/datasette/pull/1516?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (a82c620) into [main](https://codecov.io/gh/simonw/datasette/commit/0156c6b5e52d541e93f0d68e9245f20ae83bc933?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0156c6b) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1516/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1516?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1516   +/-   ##
  =======================================
    Coverage   91.82%   91.82%           
  =======================================
    Files          34       34           
    Lines        4430     4430           
  =======================================
    Hits         4068     4068           
    Misses        362      362           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1516?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1516?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [0156c6b...a82c620](https://codecov.io/gh/simonw/datasette/pull/1516?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/878#issuecomment-973527870
issue_url: https://api.github.com/repos/simonw/datasette/issues/878
node_id: IC_kwDOBm6k_c46Bts-
created_at: 2021-11-19T00:13:43Z
updated_at: 2021-11-19T00:13:43Z
author_association: OWNER
body:
  New plan: I'm going to build a brand new implementation of `TableView` starting out as a plugin, using the `register_routes()` plugin hook.

  It will reuse the existing HTML template but will be a completely new Python implementation, based on `asyncinject`.

  I'm going to start by just getting the table to show up on the page - then I'll add faceting, suggested facets, filters and so-on.

  Bonus: I'm going to see if I can get it to work for arbitrary SQL queries too (stretch goal).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/878#issuecomment-973542284
issue_url: https://api.github.com/repos/simonw/datasette/issues/878
node_id: IC_kwDOBm6k_c46BxOM
created_at: 2021-11-19T00:16:44Z
updated_at: 2021-11-19T00:16:44Z
author_association: OWNER
body:
  ```
  Development % cookiecutter gh:simonw/datasette-plugin
  You've downloaded /Users/simon/.cookiecutters/datasette-plugin before. Is it okay to delete and re-download it? [yes]: yes
  plugin_name []: table-new
  description []: New implementation of TableView, see https://github.com/simonw/datasette/issues/878
  hyphenated [table-new]: 
  underscored [table_new]: 
  github_username []: simonw
  author_name []: Simon Willison
  include_static_directory []:  
  include_templates_directory []: 
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/878#issuecomment-973554024
issue_url: https://api.github.com/repos/simonw/datasette/issues/878
node_id: IC_kwDOBm6k_c46B0Fo
created_at: 2021-11-19T00:21:20Z
updated_at: 2021-11-19T00:21:20Z
author_association: OWNER
body:
  That's annoying: it looks like plugins can't use `register_routes()` to over-ride default routes within Datasette itself. This didn't work:
  ```python
  from datasette.utils.asgi import Response
  from datasette import hookimpl
  import html


  async def table(request):
      return Response.html("Hello from {}".format(
          html.escape(repr(request.url_vars))
      ))


  @hookimpl
  def register_routes():
      return [
          (r"/(?P<db_name>[^/]+)/(?P<table_and_format>[^/]+?$)", table),
      ]
  ```
  I'll use a `/t/` prefix for the moment, but this is probably something I'll fix in Datasette itself later.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/878#issuecomment-973564260
issue_url: https://api.github.com/repos/simonw/datasette/issues/878
node_id: IC_kwDOBm6k_c46B2lk
created_at: 2021-11-19T00:27:06Z
updated_at: 2021-11-19T00:27:06Z
author_association: OWNER
body:
  Problem: the fancy `asyncinject` stuff inteferes with the fancy Datasette thing that introspects view functions to look for what parameters they take:
  ```python
  class Table(asyncinject.AsyncInjectAll):
      async def view(self, request):
          return Response.html("Hello from {}".format(
              html.escape(repr(request.url_vars))
          ))


  @hookimpl
  def register_routes():
      return [
          (r"/t/(?P<db_name>[^/]+)/(?P<table_and_format>[^/]+?$)", Table().view),
      ]
  ```
  This failed with error: "Table.view() takes 1 positional argument but 2 were given"

  So I'm going to use `AsyncInject` and have the `view` function NOT use the `@inject` decorator.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/878#issuecomment-973568285
issue_url: https://api.github.com/repos/simonw/datasette/issues/878
node_id: IC_kwDOBm6k_c46B3kd
created_at: 2021-11-19T00:29:20Z
updated_at: 2021-11-19T00:29:20Z
author_association: OWNER
body:
  This is working!
  ```python
  from datasette.utils.asgi import Response
  from datasette import hookimpl
  import html
  from asyncinject import AsyncInject, inject


  class Table(AsyncInject):
      @inject
      async def database(self, request):
          return request.url_vars["db_name"]

      @inject
      async def main(self, request, database):
          return Response.html("Database: {}".format(
              html.escape(database)
          ))

      async def view(self, request):
          return await self.main(request=request)


  @hookimpl
  def register_routes():
      return [
          (r"/t/(?P<db_name>[^/]+)/(?P<table_and_format>[^/]+?$)", Table().view),
      ]
  ```
  This project will definitely show me if I actually like the `asyncinject` patterns or not.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/878#issuecomment-973635157
issue_url: https://api.github.com/repos/simonw/datasette/issues/878
node_id: IC_kwDOBm6k_c46CH5V
created_at: 2021-11-19T01:07:08Z
updated_at: 2021-11-19T01:07:08Z
author_association: OWNER
body:
  This exercise is proving so useful in getting my head around how the enormous and complex `TableView` class works again.

  Here's where I've got to now - I'm systematically working through the variables that are returned for HTML and for JSON copying across code to get it to work:

  ```python
  from datasette.database import QueryInterrupted
  from datasette.utils import escape_sqlite
  from datasette.utils.asgi import Response, NotFound, Forbidden
  from datasette.views.base import DatasetteError
  from datasette import hookimpl
  from asyncinject import AsyncInject, inject
  from pprint import pformat


  class Table(AsyncInject):
      @inject
      async def database(self, request, datasette):
          # TODO: all that nasty hash resolving stuff can go here
          db_name = request.url_vars["db_name"]
          try:
              db = datasette.databases[db_name]
          except KeyError:
              raise NotFound(f"Database '{db_name}' does not exist")
          return db

      @inject
      async def table_and_format(self, request, database, datasette):
          table_and_format = request.url_vars["table_and_format"]
          # TODO: be a lot smarter here
          if "." in table_and_format:
              return table_and_format.split(".", 2)
          else:
              return table_and_format, "html"

      @inject
      async def main(self, request, database, table_and_format, datasette):
          # TODO: if this is actually a canned query, dispatch to it

          table, format = table_and_format

          is_view = bool(await database.get_view_definition(table))
          table_exists = bool(await database.table_exists(table))
          if not is_view and not table_exists:
              raise NotFound(f"Table not found: {table}")

          await check_permissions(
              datasette,
              request,
              [
                  ("view-table", (database.name, table)),
                  ("view-database", database.name),
                  "view-instance",
              ],
          )

          private = not await datasette.permission_allowed(
              None, "view-table", (database.name, table), default=True
          )

          pks = await database.primary_keys(table)
          table_columns = await database.table_columns(table)

          specified_columns = await columns_to_select(datasette, database, table, request)
          select_specified_columns = ", ".join(
              escape_sqlite(t) for t in specified_columns
          )
          select_all_columns = ", ".join(escape_sqlite(t) for t in table_columns)

          use_rowid = not pks and not is_view
          if use_rowid:
              select_specified_columns = f"rowid, {select_specified_columns}"
              select_all_columns = f"rowid, {select_all_columns}"
              order_by = "rowid"
              order_by_pks = "rowid"
          else:
              order_by_pks = ", ".join([escape_sqlite(pk) for pk in pks])
              order_by = order_by_pks

          if is_view:
              order_by = ""

          nocount = request.args.get("_nocount")
          nofacet = request.args.get("_nofacet")

          if request.args.get("_shape") in ("array", "object"):
              nocount = True
              nofacet = True

          # Next, a TON of SQL to build where_params and filters and suchlike
          # skipping that and jumping straight to...
          where_clauses = []
          where_clause = ""
          if where_clauses:
              where_clause = f"where {' and '.join(where_clauses)} "

          from_sql = "from {table_name} {where}".format(
              table_name=escape_sqlite(table),
              where=("where {} ".format(" and ".join(where_clauses)))
              if where_clauses
              else "",
          )
          from_sql_params ={}
          params = {}
          count_sql = f"select count(*) {from_sql}"
          sql_no_order_no_limit = (
              "select {select_all_columns} from {table_name} {where}".format(
                  select_all_columns=select_all_columns,
                  table_name=escape_sqlite(table),
                  where=where_clause,
              )
          )

          page_size = 100
          offset = " offset 0"

          sql = "select {select_specified_columns} from {table_name} {where}{order_by} limit {page_size}{offset}".format(
              select_specified_columns=select_specified_columns,
              table_name=escape_sqlite(table),
              where=where_clause,
              order_by=order_by,
              page_size=page_size + 1,
              offset=offset,
          )

          # Fetch rows
          results = await database.execute(sql, params, truncate=True)
          columns = [r[0] for r in results.description]
          rows = list(results.rows)

          # Fetch count
          filtered_table_rows_count = None
          if count_sql:
              try:
                  count_rows = list(await database.execute(count_sql, from_sql_params))
                  filtered_table_rows_count = count_rows[0][0]
              except QueryInterrupted:
                  pass


          vars = {
              "json": {
                  # THIS STUFF is from the regular JSON
                  "database": database.name,
                  "table": table,
                  "is_view": is_view,
                  # "human_description_en": human_description_en,
                  "rows": rows[:page_size],
                  "truncated": results.truncated,
                  "filtered_table_rows_count": filtered_table_rows_count,
                  # "expanded_columns": expanded_columns,
                  # "expandable_columns": expandable_columns,
                  "columns": columns,
                  "primary_keys": pks,
                  # "units": units,
                  "query": {"sql": sql, "params": params},
                  # "facet_results": facet_results,
                  # "suggested_facets": suggested_facets,
                  # "next": next_value and str(next_value) or None,
                  # "next_url": next_url,
                  "private": private,
                  "allow_execute_sql": await datasette.permission_allowed(
                      request.actor, "execute-sql", database, default=True
                  ),
              },
              "html": {
                  # ... this is the HTML special stuff
                  # "table_actions": table_actions,
                  # "supports_search": bool(fts_table),
                  # "search": search or "",
                  "use_rowid": use_rowid,
                  # "filters": filters,
                  # "display_columns": display_columns,
                  # "filter_columns": filter_columns,
                  # "display_rows": display_rows,
                  # "facets_timed_out": facets_timed_out,
                  # "sorted_facet_results": sorted(
                  #     facet_results.values(),
                  #     key=lambda f: (len(f["results"]), f["name"]),
                  #     reverse=True,
                  # ),
                  # "show_facet_counts": special_args.get("_facet_size") == "max",
                  # "extra_wheres_for_ui": extra_wheres_for_ui,
                  # "form_hidden_args": form_hidden_args,
                  # "is_sortable": any(c["sortable"] for c in display_columns),
                  # "path_with_replaced_args": path_with_replaced_args,
                  # "path_with_removed_args": path_with_removed_args,
                  # "append_querystring": append_querystring,
                  "request": request,
                  # "sort": sort,
                  # "sort_desc": sort_desc,
                  "disable_sort": is_view,
                  # "custom_table_templates": [
                  #     f"_table-{to_css_class(database)}-{to_css_class(table)}.html",
                  #     f"_table-table-{to_css_class(database)}-{to_css_class(table)}.html",
                  #     "_table.html",
                  # ],
                  # "metadata": metadata,
                  # "view_definition": await db.get_view_definition(table),
                  # "table_definition": await db.get_table_definition(table),
              },
          }

          # I'm just trying to get HTML to work for the moment
          if format == "json":
              return Response.json(dict(vars, locals=locals()), default=repr)
          else:
              return Response.html(repr(vars["html"]))

      async def view(self, request, datasette):
          return await self.main(request=request, datasette=datasette)


  @hookimpl
  def register_routes():
      return [
          (r"/t/(?P<db_name>[^/]+)/(?P<table_and_format>[^/]+?$)", Table().view),
      ]


  async def check_permissions(datasette, request, permissions):
      """permissions is a list of (action, resource) tuples or 'action' strings"""
      for permission in permissions:
          if isinstance(permission, str):
              action = permission
              resource = None
          elif isinstance(permission, (tuple, list)) and len(permission) == 2:
              action, resource = permission
          else:
              assert (
                  False
              ), "permission should be string or tuple of two items: {}".format(
                  repr(permission)
              )
          ok = await datasette.permission_allowed(
              request.actor,
              action,
              resource=resource,
              default=None,
          )
          if ok is not None:
              if ok:
                  return
              else:
                  raise Forbidden(action)


  async def columns_to_select(datasette, database, table, request):
      table_columns = await database.table_columns(table)
      pks = await database.primary_keys(table)
      columns = list(table_columns)
      if "_col" in request.args:
          columns = list(pks)
          _cols = request.args.getlist("_col")
          bad_columns = [column for column in _cols if column not in table_columns]
          if bad_columns:
              raise DatasetteError(
                  "_col={} - invalid columns".format(", ".join(bad_columns)),
                  status=400,
              )
          # De-duplicate maintaining order:
          columns.extend(dict.fromkeys(_cols))
      if "_nocol" in request.args:
          # Return all columns EXCEPT these
          bad_columns = [
              column
              for column in request.args.getlist("_nocol")
              if (column not in table_columns) or (column in pks)
          ]
          if bad_columns:
              raise DatasetteError(
                  "_nocol={} - invalid columns".format(", ".join(bad_columns)),
                  status=400,
              )
          tmp_columns = [
              column for column in columns if column not in request.args.getlist("_nocol")
          ]
          columns = tmp_columns
      return columns
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/878#issuecomment-973678931
issue_url: https://api.github.com/repos/simonw/datasette/issues/878
node_id: IC_kwDOBm6k_c46CSlT
created_at: 2021-11-19T02:51:17Z
updated_at: 2021-11-19T02:51:17Z
author_association: OWNER
body:
  OK, I managed to get a table to render! Here's the code I used - I had to copy a LOT of stuff. https://gist.github.com/simonw/281eac9c73b062c3469607ad86470eb2

  I'm going to move this work into a new, separate issue.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-973681970
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c46CTUy
created_at: 2021-11-19T02:56:31Z
updated_at: 2021-11-19T02:56:53Z
author_association: OWNER
body:
  Here's where I got to with my hacked-together initial plugin prototype - it managed to render the table page with some rows on it (and a bunch of missing functionality such as filters): https://gist.github.com/simonw/281eac9c73b062c3469607ad86470eb2

  <img width="899" alt="fixtures__roadside_attractions__4_rows_and__11__Liked___Twitter" src="https://user-images.githubusercontent.com/9599/142557265-6e10c808-5898-49ed-a8e3-7b5207b2872a.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-973682389
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c46CTbV
created_at: 2021-11-19T02:57:39Z
updated_at: 2021-11-19T02:57:39Z
author_association: OWNER
body:
  Ideally I'd like to execute the existing test suite against the new implementation - that would require me to solve this so I can replace the view with the plugin version though:

  - #1517 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1517#issuecomment-973686874
issue_url: https://api.github.com/repos/simonw/datasette/issues/1517
node_id: IC_kwDOBm6k_c46CUha
created_at: 2021-11-19T03:06:58Z
updated_at: 2021-11-19T03:06:58Z
author_association: OWNER
body:
  I made a mistake: I just wrote a test that proves that plugins CAN over-ride default routes, plus if you look at the code here the plugins get to register themselves first: https://github.com/simonw/datasette/blob/0156c6b5e52d541e93f0d68e9245f20ae83bc933/datasette/app.py#L965-L981

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-973698917
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c46CXdl
created_at: 2021-11-19T03:26:18Z
updated_at: 2021-11-19T03:29:03Z
author_association: OWNER
body:
  A (likely incomplete) list of features on the table page:

  - [ ] Display table/database/instance metadata
  - [ ] Show count of all results
  - [ ] Display table of results
    - [ ] Special table display treatment for URLs, numbers
    - [ ] Allow plugins to modify table cells
    - [ ] Respect `?_col=` and `?_nocol=`
  - [ ] Show interface for filtering by columns and operations
  - [ ] Show search box, support executing FTS searches
  - [ ] Sort table by specified column
  - [ ] Paginate table
  - [ ] Show facet results
  - [ ] Show suggested facets
  - [ ] Link to available exports
  - [ ] Display schema for table
    - [ ] Maybe it should show the SQL for the query too?
  - [ ] Handle various non-obvious querystring options, like `?_where=` and `?_through=`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-973699424
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c46CXlg
created_at: 2021-11-19T03:27:49Z
updated_at: 2021-11-19T03:27:49Z
author_association: OWNER
body:
  My goal is to break up a lot of this functionality into separate methods. These methods can be executed in parallel by `asyncinject`, but more importantly they can be used to build a much better JSON representation, where the default representation is lighter and `?_extra=x` options can be used to execute more expensive portions and add them to the response.

  So the HTML version itself needs to be re-written to use those JSON extras.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-973700322
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c46CXzi
created_at: 2021-11-19T03:30:30Z
updated_at: 2021-11-19T03:30:30Z
author_association: OWNER
body:
  Right now the HTML version gets to cheat - it passes through objects that are not JSON serializable, including custom functions that can then be called by Jinja.

  I'm interested in maybe removing this cheating - if the HTML version could only request JSON-serializable extras those could be exposed in the API as well.

  It would also help cleanup the kind-of-nasty pattern I use in the current `BaseView` where everything returns both a bunch of JSON-serializable data AND an awaitable function that then gets to add extra things to the HTML context.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-973700549
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c46CX3F
created_at: 2021-11-19T03:31:20Z
updated_at: 2021-11-19T03:31:26Z
author_association: OWNER
body:
  ... and while I'm doing all of this I can rewrite the templates to not use those cheating magical functions AND document the template context at the same time, refs:
  - #1510.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/342#issuecomment-973800795
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/342
node_id: IC_kwDOCGYnMM46CwVb
created_at: 2021-11-19T06:54:08Z
updated_at: 2021-11-19T06:54:08Z
author_association: OWNER
body:
  Looking at the code for `lookup()` it currently hard-codes `pk` to `"id"` - but it actually only calls `.insert()` in two places, both of which could be passed extra arguments.

  https://github.com/simonw/sqlite-utils/blob/3b8abe608796e99e4ffc5f3f4597a85e605c0e9b/sqlite_utils/db.py#L2756-L2763

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/342#issuecomment-973801650
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/342
node_id: IC_kwDOCGYnMM46Cwiy
created_at: 2021-11-19T06:55:56Z
updated_at: 2021-11-19T06:55:56Z
author_association: OWNER
body:
  `pk` needs to be an explicit argument to `.lookup()`. The rest could be `**kwargs` passed through to `.insert()`, like this hacked together version (docstring removed for brevity):

  ```python
      def lookup(
          self,
          lookup_values: Dict[str, Any],
          extra_values: Optional[Dict[str, Any]] = None,
          pk="id",
          **insert_kwargs,
      ):
          """
          assert isinstance(lookup_values, dict)
          if extra_values is not None:
              assert isinstance(extra_values, dict)
          combined_values = dict(lookup_values)
          if extra_values is not None:
              combined_values.update(extra_values)
          if self.exists():
              self.add_missing_columns([combined_values])
              unique_column_sets = [set(i.columns) for i in self.indexes]
              if set(lookup_values.keys()) not in unique_column_sets:
                  self.create_index(lookup_values.keys(), unique=True)
              wheres = ["[{}] = ?".format(column) for column in lookup_values]
              rows = list(
                  self.rows_where(
                      " and ".join(wheres), [value for _, value in lookup_values.items()]
                  )
              )
              try:
                  return rows[0][pk]
              except IndexError:
                  return self.insert(combined_values, pk=pk, **insert_kwargs).last_pk
          else:
              pk = self.insert(combined_values, pk=pk, **insert_kwargs).last_pk
              self.create_index(lookup_values.keys(), unique=True)
              return pk
  ```
  I think I'll explicitly list the parameters, mainly so they can be typed and covered by automatic documentation.

  I do worry that I'll add more keyword arguments to `.insert()` in the future and forget to mirror them to `.lookup()` though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/342#issuecomment-973802308
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/342
node_id: IC_kwDOCGYnMM46CwtE
created_at: 2021-11-19T06:57:37Z
updated_at: 2021-11-19T06:57:37Z
author_association: OWNER
body:
  Here's the current full method signature for `.insert()`: https://github.com/simonw/sqlite-utils/blob/3b8abe608796e99e4ffc5f3f4597a85e605c0e9b/sqlite_utils/db.py#L2462-L2477

  I could add a test which uses introspection (`inspect.signature(method).parameters`) to confirm that `.lookup()` has a super-set of the arguments accepted by `.insert()`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/342#issuecomment-973802766
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/342
node_id: IC_kwDOCGYnMM46Cw0O
created_at: 2021-11-19T06:58:45Z
updated_at: 2021-11-19T06:58:45Z
author_association: OWNER
body:
  And neither does `hash_id`. On that basis I'm going to specifically list the ones that DO make sense, and hope that I remember to add any new ones in the future. I can add a code comment hint to `.insert()` about that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/342#issuecomment-973802998
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/342
node_id: IC_kwDOCGYnMM46Cw32
created_at: 2021-11-19T06:59:22Z
updated_at: 2021-11-19T06:59:32Z
author_association: OWNER
body:
  I don't think I need the `DEFAULT` defaults for `.insert()` either, since it just passes through to `.insert()`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/342#issuecomment-973820125
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/342
node_id: IC_kwDOCGYnMM46C1Dd
created_at: 2021-11-19T07:25:55Z
updated_at: 2021-11-19T07:25:55Z
author_association: OWNER
body:
  `alter=True` doesn't make sense to support here either, because `.lookup()` already adds missing columns: https://github.com/simonw/sqlite-utils/blob/3b8abe608796e99e4ffc5f3f4597a85e605c0e9b/sqlite_utils/db.py#L2743-L2746

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-974285803
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c46Emvr
created_at: 2021-11-19T17:56:48Z
updated_at: 2021-11-19T18:14:30Z
author_association: OWNER
body:
  Very confused by this piece of code here: https://github.com/simonw/datasette/blob/1c13e1af0664a4dfb1e69714c56523279cae09e4/datasette/views/table.py#L37-L63

  I added it in https://github.com/simonw/datasette/commit/754836eef043676e84626c4fd3cb993eed0d2976 - in the new world that should probably be replaced by pure JSON.

  Aha - this comment explains it: https://github.com/simonw/datasette/issues/521#issuecomment-505279560

  > I think the trick is to redefine what a "cell_row" is. Each row is currently a list of cells:
  > 
  > https://github.com/simonw/datasette/blob/6341f8cbc7833022012804dea120b838ec1f6558/datasette/views/table.py#L159-L163
  > 
  > I can redefine the row (the `cells` variable in the above example) as a thing-that-iterates-cells (hence behaving like a list) but that also supports `__getitem__` access for looking up cell values if you know the name of the column.

  The goal was to support neater custom templates like this:
  ```html+jinja
  {% for row in display_rows %}
    <h2 class="scientist">{{ row["First_Name"] }} {{ row["Last_Name"] }}</h2>
    ...
  ```
  This may be an argument for continuing to allow non-JSON-objects through to the HTML templates. Need to think about that a bit more.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-974287570
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c46EnLS
created_at: 2021-11-19T17:59:33Z
updated_at: 2021-11-19T17:59:33Z
author_association: OWNER
body:
  I'm going to try leaning into the `asyncinject` mechanism a bit here. One method can execute and return the raw rows. Another can turn that into the default minimal JSON representation. Then a third can take that (or take both) and use it to inflate out the JSON that the HTML template needs, with those extras and with the rendered cells from plugins.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-974300823
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c46EqaX
created_at: 2021-11-19T18:18:32Z
updated_at: 2021-11-19T18:18:32Z
author_association: OWNER
body:
  > This may be an argument for continuing to allow non-JSON-objects through to the HTML templates. Need to think about that a bit more.

  I can definitely support this using pure-JSON - I could make two versions of the row available, one that's an array of cell objects and the other that's an object mapping column names to column raw values.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1520#issuecomment-974308215
issue_url: https://api.github.com/repos/simonw/datasette/issues/1520
node_id: IC_kwDOBm6k_c46EsN3
created_at: 2021-11-19T18:29:26Z
updated_at: 2021-11-19T18:29:26Z
author_association: OWNER
body:
  The solution that jumps to mind first is that it would be neat if routes could return something that meant "actually my bad, I can't handle this after all - move to the next one in the list".

  A related idea: it might be useful for custom views like my one here to say "no actually call the default view for this, but give me back the response so I can modify it in some way". Kind of like Django or ASGI middleware.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1519#issuecomment-974309591
issue_url: https://api.github.com/repos/simonw/datasette/issues/1519
node_id: IC_kwDOBm6k_c46EsjX
created_at: 2021-11-19T18:31:32Z
updated_at: 2021-11-19T18:31:32Z
author_association: OWNER
body:
  `base_url` has been a source of so many bugs like this! I often find them quite hard to replicate, likely because I haven't made myself a good Apache `mod_proxy` testing environment yet.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1519#issuecomment-974310208
issue_url: https://api.github.com/repos/simonw/datasette/issues/1519
node_id: IC_kwDOBm6k_c46EstA
created_at: 2021-11-19T18:32:31Z
updated_at: 2021-11-19T18:32:31Z
author_association: OWNER
body:
  Having a live demo running on Cloud Run that proxies through Apache and uses `base_url` would be incredibly useful for replicating and debugging this kind of thing. I wonder how hard it is to run Apache and `mod_proxy` in the same Docker container as Datasette?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1521#issuecomment-974321391
issue_url: https://api.github.com/repos/simonw/datasette/issues/1521
node_id: IC_kwDOBm6k_c46Evbv
created_at: 2021-11-19T18:49:15Z
updated_at: 2021-11-19T18:57:18Z
author_association: OWNER
body:
  This pattern looks like it can help: https://ahmet.im/blog/cloud-run-multiple-processes-easy-way/ - see example in https://github.com/ahmetb/multi-process-container-lazy-solution

  I got that demo working locally like this:

  ```bash
  cd /tmp
  git clone https://github.com/ahmetb/multi-process-container-lazy-solution
  cd multi-process-container-lazy-solution
  docker build -t multi-process-container-lazy-solution .
  docker run -p 5000:8080 --rm multi-process-container-lazy-solution
  ```

  I want to use `apache2` rather than `nginx` though. I found a few relevant examples of Apache in Alpine:

  - https://github.com/Hacking-Lab/alpine-apache2-reverse-proxy/blob/master/Dockerfile
  - https://www.sentiatechblog.com/running-apache-in-a-docker-container
  - https://github.com/search?l=Dockerfile&q=alpine+apache2&type=code

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1521#issuecomment-974322178
issue_url: https://api.github.com/repos/simonw/datasette/issues/1521
node_id: IC_kwDOBm6k_c46EvoC
created_at: 2021-11-19T18:50:22Z
updated_at: 2021-11-19T18:50:22Z
author_association: OWNER
body:
  I'll get this working on my laptop first, but then I want to get it up and running on Cloud Run - maybe with a GitHub Actions  workflow in this repo that re-deploys it on manual execution.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1521#issuecomment-974327812
issue_url: https://api.github.com/repos/simonw/datasette/issues/1521
node_id: IC_kwDOBm6k_c46ExAE
created_at: 2021-11-19T18:58:49Z
updated_at: 2021-11-19T18:59:55Z
author_association: OWNER
body:
  From this example: https://github.com/tigelane/dockerfiles/blob/06cff2ac8cdc920ebd64f50965115eaa3d0afb84/Alpine-Apache2/Dockerfile#L25-L31 it looks like running `apk add apache2` installs a config file at `/etc/apache2/httpd.conf` - so one approach is to then modify that file.

  ```
  # APACHE - Alpine
  #################
  RUN apk --update add apache2 php5-apache2 && \
      #apk add openrc --no-cache && \
      rm -rf /var/cache/apk/* && \
      sed -i 's/#ServerName www.example.com:80/ServerName localhost/' /etc/apache2/httpd.conf && \
      mkdir -p /run/apache2/

  # Upload our files from folder "dist".
  COPY dist /var/www/localhost/htdocs

  # Manually set up the apache environment variables
  ENV APACHE_RUN_USER www-data
  ENV APACHE_RUN_GROUP www-data
  ENV APACHE_LOG_DIR /var/log/apache2
  ENV APACHE_LOCK_DIR /var/lock/apache2
  ENV APACHE_PID_FILE /var/run/apache2.pid

  # Execute apache2 on run
  ########################
  EXPOSE 80
  ENTRYPOINT ["httpd"]
  CMD ["-D", "FOREGROUND"]
  ```

  I think I'll create my own separate copy and modify that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1521#issuecomment-974332787
issue_url: https://api.github.com/repos/simonw/datasette/issues/1521
node_id: IC_kwDOBm6k_c46EyNz
created_at: 2021-11-19T19:05:52Z
updated_at: 2021-11-19T19:05:52Z
author_association: OWNER
body:
  Made myself this Dockerfile to let me explore a bit:
  ```Dockerfile
  FROM python:3-alpine

  RUN apk add --no-cache \
  	apache2

  CMD ["sh"]
  ```
  Then:
  ```
  % docker run alpine-apache2-sh
  % docker run -it alpine-apache2-sh
  / # ls /etc/apache2/httpd.conf
  /etc/apache2/httpd.conf
  / # cat /etc/apache2/httpd.conf
  #
  # This is the main Apache HTTP server configuration file.  It contains the
  # configuration directives that give the server its instructions.
  ...
  ```
  Copying that into a GIST like so:
  ```
  docker run -it --entrypoint sh alpine-apache2-sh -c "cat /etc/apache2/httpd.conf" | pbcopy
  ```
  Gist here: https://gist.github.com/simonw/5ea0db6049192cb9f761fbd6beb3a84a

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1521#issuecomment-974334278
issue_url: https://api.github.com/repos/simonw/datasette/issues/1521
node_id: IC_kwDOBm6k_c46EylG
created_at: 2021-11-19T19:08:09Z
updated_at: 2021-11-19T19:08:09Z
author_association: OWNER
body:
  Stripping comments using this StackOverflow recipe: https://unix.stackexchange.com/a/157619

      docker run -it --entrypoint sh alpine-apache2-sh \
        -c "cat /etc/apache2/httpd.conf" | sed '/^[[:blank:]]*#/d;s/#.*//'

  Result is here: https://gist.github.com/simonw/0a05090df5fcff8e8b3334621fa17976

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1521#issuecomment-974336020
issue_url: https://api.github.com/repos/simonw/datasette/issues/1521
node_id: IC_kwDOBm6k_c46EzAU
created_at: 2021-11-19T19:10:48Z
updated_at: 2021-11-19T19:10:48Z
author_association: OWNER
body:
  There's a promising looking minimal Apache 2 proxy config here: https://stackoverflow.com/questions/26474476/minimal-configuration-for-apache-reverse-proxy-in-docker-container

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1521#issuecomment-974371116
issue_url: https://api.github.com/repos/simonw/datasette/issues/1521
node_id: IC_kwDOBm6k_c46E7ks
created_at: 2021-11-19T19:45:47Z
updated_at: 2021-11-19T19:45:47Z
author_association: OWNER
body:
  https://github.com/krallin/tini says:

  > *NOTE: If you are using Docker 1.13 or greater, Tini is included in Docker itself. This includes all versions of Docker CE. To enable Tini, just [pass the `--init` flag to `docker run`](https://docs.docker.com/engine/reference/commandline/run/).*

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1521#issuecomment-974380798
issue_url: https://api.github.com/repos/simonw/datasette/issues/1521
node_id: IC_kwDOBm6k_c46E97-
created_at: 2021-11-19T19:54:26Z
updated_at: 2021-11-19T19:54:26Z
author_association: OWNER
body:
  Got it working! Here's a `Dockerfile` which runs completely stand-alone (thanks to using the `echo $'` trick to write out the config files it needs) and successfully serves Datasette behind Apache and `mod_proxy`:

  ```Dockerfile
  FROM python:3-alpine

  RUN apk add --no-cache \
  	apache2 \
  	apache2-proxy \
  	bash

  RUN pip install datasette

  ENV TINI_VERSION v0.18.0
  ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-static /tini
  RUN chmod +x /tini

  # Append this to the end of the default httpd.conf file
  RUN echo $'ServerName localhost\n\
  \n\
  <Proxy *>\n\
    Order deny,allow\n\
    Allow from all\n\
  </Proxy>\n\
  \n\
  ProxyPass        / http://localhost:9000/\n\
  ProxyPassReverse / http://localhost:9000/\n\
  Header add X-Proxied-By "Apache2"' >> /etc/apache2/httpd.conf

  WORKDIR /app

  RUN echo $'#!/usr/bin/env bash\n\
  set -e\n\
  \n\
  httpd -D FOREGROUND &\n\
  datasette -p 9000 &\n\
  \n\
  wait -n' > /app/start.sh

  RUN chmod +x /app/start.sh

  EXPOSE 80
  ENTRYPOINT ["/tini", "--", "/app/start.sh"]
  ```

  Run it like this:

  ```
  docker build -t datasette-apache2-proxy .   
  docker run -p 5000:80 --rm datasette-apache2-proxy
  ```
  Then run this to confirm:
  ```
  ~ % curl -i 'http://localhost:5000/-/versions.json'
  HTTP/1.1 200 OK
  Date: Fri, 19 Nov 2021 19:54:05 GMT
  Server: uvicorn
  content-type: application/json; charset=utf-8
  X-Proxied-By: Apache2
  Transfer-Encoding: chunked

  {"python": {"version": "3.10.0", "full": "3.10.0 (default, Nov 13 2021, 03:23:03) [GCC 10.3.1 20210424]"}, "datasette": {"version": "0.59.2"}, "asgi": "3.0", "uvicorn": "0.15.0", "sqlite": {"version": "3.35.5", "fts_versions": ["FTS5", "FTS4", "FTS3"], "extensions": {"json1": null}, "compile_options": ["COMPILER=gcc-10.3.1 20210424", "ENABLE_COLUMN_METADATA", "ENABLE_DBSTAT_VTAB", "ENABLE_FTS3", "ENABLE_FTS3_PARENTHESIS", "ENABLE_FTS4", "ENABLE_FTS5", "ENABLE_GEOPOLY", "ENABLE_JSON1", "ENABLE_MATH_FUNCTIONS", "ENABLE_RTREE", "ENABLE_UNLOCK_NOTIFY", "MAX_VARIABLE_NUMBER=250000", "SECURE_DELETE", "THREADSAFE=1", "USE_URI"]}}
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1521#issuecomment-974388295
issue_url: https://api.github.com/repos/simonw/datasette/issues/1521
node_id: IC_kwDOBm6k_c46E_xH
created_at: 2021-11-19T20:00:06Z
updated_at: 2021-11-19T20:00:06Z
author_association: OWNER
body:
  And this is the version that proxies to a `base_url` of `/foo/bar/`:

  ```Dockerfile
  FROM python:3-alpine

  RUN apk add --no-cache \
  	apache2 \
  	apache2-proxy \
  	bash

  RUN pip install datasette

  ENV TINI_VERSION v0.18.0
  ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-static /tini
  RUN chmod +x /tini

  # Append this to the end of the default httpd.conf file
  RUN echo $'ServerName localhost\n\
  \n\
  <Proxy *>\n\
    Order deny,allow\n\
    Allow from all\n\
  </Proxy>\n\
  \n\
  ProxyPass        /foo/bar/ http://localhost:9000/\n\
  Header add X-Proxied-By "Apache2"' >> /etc/apache2/httpd.conf

  RUN echo $'<a href="/foo/bar/">Datasette</a>' > /var/www/localhost/htdocs/index.html

  WORKDIR /app

  ADD https://latest.datasette.io/fixtures.db /app/fixtures.db

  RUN echo $'#!/usr/bin/env bash\n\
  set -e\n\
  \n\
  httpd -D FOREGROUND &\n\
  datasette fixtures.db --setting base_url "/foo/bar/" -p 9000 &\n\
  \n\
  wait -n' > /app/start.sh

  RUN chmod +x /app/start.sh

  EXPOSE 80
  ENTRYPOINT ["/tini", "--", "/app/start.sh"]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1519#issuecomment-974389472
issue_url: https://api.github.com/repos/simonw/datasette/issues/1519
node_id: IC_kwDOBm6k_c46FADg
created_at: 2021-11-19T20:01:02Z
updated_at: 2021-11-19T20:01:02Z
author_association: OWNER
body:
  I now have a `Dockerfile` in https://github.com/simonw/datasette/issues/1521#issuecomment-974388295 that I can use to run a local Apache 2 with `mod_proxy` to investigate this class of bugs!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1519#issuecomment-974391204
issue_url: https://api.github.com/repos/simonw/datasette/issues/1519
node_id: IC_kwDOBm6k_c46FAek
created_at: 2021-11-19T20:02:41Z
updated_at: 2021-11-19T20:02:41Z
author_association: OWNER
body:
  Bug confirmed:

  ![proxy-bug](https://user-images.githubusercontent.com/9599/142684666-112136bf-9243-4b6e-8202-339fcfe91bcc.gif)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1519#issuecomment-974398399
issue_url: https://api.github.com/repos/simonw/datasette/issues/1519
node_id: IC_kwDOBm6k_c46FCO_
created_at: 2021-11-19T20:08:20Z
updated_at: 2021-11-19T20:22:02Z
author_association: OWNER
body:
  The relevant test is this one: https://github.com/simonw/datasette/blob/30255055150d7bc0affc8156adc18295495020ff/tests/test_html.py#L1608-L1649

  I modified that test to add `"/fixtures/facetable?sql=select+1"` as one of the tested paths, and dropped in an `assert False` to pause it in the debugger:
  ```
      @pytest.mark.parametrize(
          "path",
          [
              "/",
              "/fixtures",
              "/fixtures/compound_three_primary_keys",
              "/fixtures/compound_three_primary_keys/a,a,a",
              "/fixtures/paginated_view",
              "/fixtures/facetable",
              "/fixtures?sql=select+1",
          ],
      )
      def test_base_url_config(app_client_base_url_prefix, path):
          client = app_client_base_url_prefix
          response = client.get("/prefix/" + path.lstrip("/"))
          soup = Soup(response.body, "html.parser")
          if path == "/fixtures?sql=select+1":
  >           assert False
  E           assert False
  ```
  BUT... in the debugger:
  ```
  (Pdb) print(soup)
  ...
  <p class="export-links">This data as
    <a href="/prefix/fixtures.json?sql=select+1">json</a>,
    <a href="/prefix/fixtures.testall?sql=select+1">testall</a>,
    <a href="/prefix/fixtures.testnone?sql=select+1">testnone</a>,
    <a href="/prefix/fixtures.testresponse?sql=select+1">testresponse</a>,
    <a href="/prefix/fixtures.csv?sql=select+1&amp;_size=max">CSV</a></p>
  ```
  Those all have the correct prefix! But that's not what I'm seeing in my `Dockerfile` reproduction of the issue.

  Something very weird is going on here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1519#issuecomment-974405016
issue_url: https://api.github.com/repos/simonw/datasette/issues/1519
node_id: IC_kwDOBm6k_c46FD2Y
created_at: 2021-11-19T20:14:19Z
updated_at: 2021-11-19T20:15:05Z
author_association: OWNER
body:
  I added `template_debug` in the Dockerfile:
  ```
  datasette fixtures.db --setting template_debug 1 --setting base_url "/foo/bar/" -p 9000 &\n\
  ```
  And then hit `http://localhost:5000/foo/bar/fixtures?sql=select+*+from+compound_three_primary_keys+limit+1&_context=1` to view the template context - and it showed the bug, output edited to just show relevant keys:

  ```json
  {
      "edit_sql_url": "/foo/bar/fixtures?sql=select+%2A+from+compound_three_primary_keys+limit+1",
      "settings": {
          "force_https_urls": false,
          "template_debug": true,
          "trace_debug": false,
          "base_url": "/foo/bar/"
      },
      "show_hide_link": "/fixtures?sql=select+%2A+from+compound_three_primary_keys+limit+1&_context=1&_hide_sql=1",
      "show_hide_text": "hide",
      "show_hide_hidden": "",
      "renderers": {
          "json": "/fixtures.json?sql=select+*+from+compound_three_primary_keys+limit+1&_context=1"
      },
      "url_csv": "/fixtures.csv?sql=select+*+from+compound_three_primary_keys+limit+1&_context=1&_size=max",
      "url_csv_path": "/fixtures.csv",
      "base_url": "/foo/bar/"
  }
  ```
  This is so strange. `edit_sql_url` and `base_url` are correct, but `show_hide_link` and `url_csv` and `renderers.json` are not.

  And it's _really strange_ that the bug doesn't show up in the tests.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1519#issuecomment-974418496
issue_url: https://api.github.com/repos/simonw/datasette/issues/1519
node_id: IC_kwDOBm6k_c46FHJA
created_at: 2021-11-19T20:24:16Z
updated_at: 2021-11-19T20:24:16Z
author_association: OWNER
body:
  Here's the code that generates `edit_sql_url` correctly: https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/views/database.py#L416-L420

  And  here's the code for `show_hide_link`: https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/views/database.py#L432-L433

  And for `url_csv`: https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/views/base.py#L600-L602

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1519#issuecomment-974420619
issue_url: https://api.github.com/repos/simonw/datasette/issues/1519
node_id: IC_kwDOBm6k_c46FHqL
created_at: 2021-11-19T20:25:19Z
updated_at: 2021-11-19T20:25:19Z
author_association: OWNER
body:
  The implementations of `path_with_removed_args` and `path_with_format`:

  https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/utils/__init__.py#L228-L254

  https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/utils/__init__.py#L710-L729

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1519#issuecomment-974422829
issue_url: https://api.github.com/repos/simonw/datasette/issues/1519
node_id: IC_kwDOBm6k_c46FIMt
created_at: 2021-11-19T20:26:35Z
updated_at: 2021-11-19T20:26:35Z
author_association: OWNER
body:
  In the `?_context=` debug view the request looks like this:
  ```
      "request": "<datasette.utils.asgi.Request object at 0x7faf9fe06200>",
  ```
  I'm going to add a `repr()` to it such that it's a bit more useful.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1519#issuecomment-974433206
issue_url: https://api.github.com/repos/simonw/datasette/issues/1519
node_id: IC_kwDOBm6k_c46FKu2
created_at: 2021-11-19T20:31:52Z
updated_at: 2021-11-19T20:31:52Z
author_association: OWNER
body:
  Modified my `Dockerfile` to do this:

      RUN pip install https://github.com/simonw/datasette/archive/ff0dd4da38d48c2fa9250ecf336002c9ed724e36.zip

  And now the `request` in that debug `?_context=1` looks like this:
  ```
      "request": "<asgi.Request method=\"GET\" url=\"http://localhost:9000/fixtures?sql=select+*+from+compound_three_primary_keys+limit+1&_context=1\">"
  ```
  That explains the bug - that request doesn't maintain the original path prefix of `http://localhost:5000/foo/bar/fixtures?sql=` (also it's been rewritten to `localhost:9000` instead of `localhost:5000`).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1522#issuecomment-974435661
issue_url: https://api.github.com/repos/simonw/datasette/issues/1522
node_id: IC_kwDOBm6k_c46FLVN
created_at: 2021-11-19T20:33:42Z
updated_at: 2021-11-19T20:33:42Z
author_association: OWNER
body:
  Should just be a case of deploying this `Dockerfile`:

  ```Dockerfile
  FROM python:3-alpine

  RUN apk add --no-cache \
  	apache2 \
  	apache2-proxy \
  	bash

  RUN pip install datasette

  ENV TINI_VERSION v0.18.0
  ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-static /tini
  RUN chmod +x /tini

  # Append this to the end of the default httpd.conf file
  RUN echo $'ServerName localhost\n\
  \n\
  <Proxy *>\n\
    Order deny,allow\n\
    Allow from all\n\
  </Proxy>\n\
  \n\
  ProxyPass        /foo/bar/ http://localhost:9000/\n\
  Header add X-Proxied-By "Apache2"' >> /etc/apache2/httpd.conf

  RUN echo $'<a href="/foo/bar/">Datasette</a>' > /var/www/localhost/htdocs/index.html

  WORKDIR /app

  ADD https://latest.datasette.io/fixtures.db /app/fixtures.db

  RUN echo $'#!/usr/bin/env bash\n\
  set -e\n\
  \n\
  httpd -D FOREGROUND &\n\
  datasette fixtures.db --setting base_url "/foo/bar/" -p 9000 &\n\
  \n\
  wait -n' > /app/start.sh

  RUN chmod +x /app/start.sh

  EXPOSE 80
  ENTRYPOINT ["/tini", "--", "/app/start.sh"]
  ```
  I can follow this TIL: https://til.simonwillison.net/cloudrun/ship-dockerfile-to-cloud-run

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1519#issuecomment-974447950
issue_url: https://api.github.com/repos/simonw/datasette/issues/1519
node_id: IC_kwDOBm6k_c46FOVO
created_at: 2021-11-19T20:40:19Z
updated_at: 2021-11-19T20:40:19Z
author_association: OWNER
body:
  Figured it out! The test is not an accurate recreation of what is happening, because it doesn't simulate a request with a path of `/fixtures` that has been redirected by the proxy to `/prefix/fixtures`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1519#issuecomment-974450232
issue_url: https://api.github.com/repos/simonw/datasette/issues/1519
node_id: IC_kwDOBm6k_c46FO44
created_at: 2021-11-19T20:41:53Z
updated_at: 2021-11-19T20:42:19Z
author_association: OWNER
body:
  https://docs.datasette.io/en/stable/deploying.html#apache-proxy-configuration says I should use `ProxyPreserveHost on`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1519#issuecomment-974477465
issue_url: https://api.github.com/repos/simonw/datasette/issues/1519
node_id: IC_kwDOBm6k_c46FViZ
created_at: 2021-11-19T21:15:30Z
updated_at: 2021-11-19T21:15:30Z
author_association: OWNER
body:
  I think what's happening here is Apache is actually making a request to `/fixtures` rather than making a request to `/prefix/fixtures` - and Datasette is replying to requests on both the prefixed and the non-prefixed paths.

  This is pretty confusing! I think Datasette should ONLY reply to `/prefix/fixtures` instead and return a 404 for `/fixtures` - this would make things a whole lot easier to debug.

  But shipping that change could break existing deployments. Maybe that should be a breaking change for 1.0.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1519#issuecomment-974478126
issue_url: https://api.github.com/repos/simonw/datasette/issues/1519
node_id: IC_kwDOBm6k_c46FVsu
created_at: 2021-11-19T21:16:36Z
updated_at: 2021-11-19T21:16:36Z
author_association: OWNER
body:
  In the meantime I can catch these errors by changing the test to run each path twice, once with and once without the prefix. This should accurately simulate how Apache is working here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1522#issuecomment-974506401
issue_url: https://api.github.com/repos/simonw/datasette/issues/1522
node_id: IC_kwDOBm6k_c46Fcmh
created_at: 2021-11-19T22:11:51Z
updated_at: 2021-11-19T22:11:51Z
author_association: OWNER
body:
  This is frustrating: I have the following Dockerfile:
  ```dockerfile
  FROM python:3-alpine

  RUN apk add --no-cache \
  	apache2 \
  	apache2-proxy \
  	bash

  RUN pip install datasette

  ENV TINI_VERSION v0.18.0
  ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-static /tini
  RUN chmod +x /tini

  # Append this to the end of the default httpd.conf file
  RUN echo $'ServerName localhost\n\
  \n\
  <Proxy *>\n\
    Order deny,allow\n\
    Allow from all\n\
  </Proxy>\n\
  \n\
  ProxyPass        /prefix/ http://localhost:8001/\n\
  Header add X-Proxied-By "Apache2"' >> /etc/apache2/httpd.conf

  RUN echo $'<a href="/prefix/">Datasette</a>' > /var/www/localhost/htdocs/index.html

  WORKDIR /app

  ADD https://latest.datasette.io/fixtures.db /app/fixtures.db

  RUN echo $'#!/usr/bin/env bash\n\
  set -e\n\
  \n\
  httpd -D FOREGROUND &\n\
  datasette fixtures.db --setting base_url "/prefix/" -h 0.0.0.0 -p 8001 &\n\
  \n\
  wait -n' > /app/start.sh

  RUN chmod +x /app/start.sh

  EXPOSE 80
  ENTRYPOINT ["/tini", "--", "/app/start.sh"]
  ```
  It works fine when I run it locally:
  ```
  docker build -t datasette-apache-proxy-demo .
  docker run -p 5000:80 datasette-apache-proxy-demo
  ```
  But when I deploy it to Cloud Run with the following script:
  ```bash
  #!/bin/bash
  # https://til.simonwillison.net/cloudrun/ship-dockerfile-to-cloud-run

  NAME="datasette-apache-proxy-demo"
  PROJECT=$(gcloud config get-value project)
  IMAGE="gcr.io/$PROJECT/$NAME"

  gcloud builds submit --tag $IMAGE
  gcloud run deploy \
      --allow-unauthenticated \
      --platform=managed \
      --image $IMAGE $NAME \
      --port 80
  ```
  It serves the `/` page successfully, but hits to `/prefix/` return the following 503 error:

  > Service Unavailable
  >
  > The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.
  >
  > Apache/2.4.51 (Unix) Server at datasette-apache-proxy-demo-j7hipcg4aq-uc.a.run.app Port 80

  Cloud Run logs:

  <img width="1347" alt="Screen Shot 2021-11-19 at 2 10 54 PM" src="https://user-images.githubusercontent.com/9599/142698024-4811e228-0373-45ea-9758-24564c46b3e4.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1522#issuecomment-974521687
issue_url: https://api.github.com/repos/simonw/datasette/issues/1522
node_id: IC_kwDOBm6k_c46FgVX
created_at: 2021-11-19T22:46:26Z
updated_at: 2021-11-19T22:46:26Z
author_association: OWNER
body:
  Oh weird, it started working: https://datasette-apache-proxy-demo-j7hipcg4aq-uc.a.run.app/prefix/fixtures/sortable

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1522#issuecomment-974523569
issue_url: https://api.github.com/repos/simonw/datasette/issues/1522
node_id: IC_kwDOBm6k_c46Fgyx
created_at: 2021-11-19T22:51:10Z
updated_at: 2021-11-19T22:51:10Z
author_association: OWNER
body:
  I wan a GitHub Action which I can manually activate to deploy a new version of that demo... and I want it to bake in the latest release of Datasette so I can use it to demonstrate bug fixes.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1522#issuecomment-974541971
issue_url: https://api.github.com/repos/simonw/datasette/issues/1522
node_id: IC_kwDOBm6k_c46FlST
created_at: 2021-11-19T23:40:32Z
updated_at: 2021-11-19T23:40:32Z
author_association: OWNER
body:
  I want to be able to use build arguments to specify which commit version or branch of Datasette to deploy.

  This is proving hard to work out. I have this in my Dockerfile now:

  ```
  ARG DATASETTE_REF

  RUN pip install https://github.com/simonw/datasette/archive/${DATASETTE_REF}.zip
  ```
  Which works locally:

      docker build -t datasette-apache-proxy-demo . \
        --build-arg DATASETTE_REF=c617e1769ea27e045b0f2907ef49a9a1244e577d

  But I can't figure out the right incantation to pass to `gcloud build submit`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1522#issuecomment-974542348
issue_url: https://api.github.com/repos/simonw/datasette/issues/1522
node_id: IC_kwDOBm6k_c46FlYM
created_at: 2021-11-19T23:41:47Z
updated_at: 2021-11-19T23:44:07Z
author_association: OWNER
body:
  Do I have to use `cloudbuild.yml` to specify these? https://stackoverflow.com/a/58327340/6083 and https://stackoverflow.com/a/66232670/6083 suggest I do.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1522#issuecomment-974557766
issue_url: https://api.github.com/repos/simonw/datasette/issues/1522
node_id: IC_kwDOBm6k_c46FpJG
created_at: 2021-11-20T00:35:25Z
updated_at: 2021-11-20T00:35:25Z
author_association: OWNER
body:
  Wrote a TIL about `--build-arg` and Cloud Run: https://til.simonwillison.net/cloudrun/using-build-args-with-cloud-run

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1522#issuecomment-974558076
issue_url: https://api.github.com/repos/simonw/datasette/issues/1522
node_id: IC_kwDOBm6k_c46FpN8
created_at: 2021-11-20T00:36:56Z
updated_at: 2021-11-20T00:36:56Z
author_association: OWNER
body:
  That 503 error is _really_ frustrating: I have a deploy running at https://apache-proxy-demo.datasette.io/prefix/ and after a fresh deploy it serves 503 errors for quite a while - then eventually starts working.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1519#issuecomment-974558267
issue_url: https://api.github.com/repos/simonw/datasette/issues/1519
node_id: IC_kwDOBm6k_c46FpQ7
created_at: 2021-11-20T00:37:57Z
updated_at: 2021-11-20T00:37:57Z
author_association: OWNER
body:
  Thanks to #1522 I have a live demo that exhibits this bug now: https://apache-proxy-demo.datasette.io/prefix/fixtures/attraction_characteristic

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1519#issuecomment-974559176
issue_url: https://api.github.com/repos/simonw/datasette/issues/1519
node_id: IC_kwDOBm6k_c46FpfI
created_at: 2021-11-20T00:42:08Z
updated_at: 2021-11-20T00:42:08Z
author_association: OWNER
body:
  > In the meantime I can catch these errors by changing the test to run each path twice, once with and once without the prefix. This should accurately simulate how Apache is working here.

  This worked, I managed to get the tests to fail! Here's the change I made:

  ```diff
  diff --git a/tests/test_html.py b/tests/test_html.py
  index f24165b..dbdfe59 100644
  --- a/tests/test_html.py
  +++ b/tests/test_html.py
  @@ -1614,12 +1614,19 @@ def test_metadata_sort_desc(app_client):
           "/fixtures/compound_three_primary_keys/a,a,a",
           "/fixtures/paginated_view",
           "/fixtures/facetable",
  +        "/fixtures?sql=select+1",
       ],
   )
  -def test_base_url_config(app_client_base_url_prefix, path):
  +@pytest.mark.parametrize("use_prefix", (True, False))
  +def test_base_url_config(app_client_base_url_prefix, path, use_prefix):
       client = app_client_base_url_prefix
  -    response = client.get("/prefix/" + path.lstrip("/"))
  +    path_to_get = path
  +    if use_prefix:
  +        path_to_get = "/prefix/" + path.lstrip("/")
  +    response = client.get(path_to_get)
       soup = Soup(response.body, "html.parser")
  +    if path == "/fixtures?sql=select+1":
  +        assert False
       for el in soup.findAll(["a", "link", "script"]):
           if "href" in el.attrs:
               href = el["href"]
  @@ -1642,11 +1649,12 @@ def test_base_url_config(app_client_base_url_prefix, path):
               # If this has been made absolute it may start http://localhost/
               if href.startswith("http://localhost/"):
                   href = href[len("http://localost/") :]
  -            assert href.startswith("/prefix/"), {
  +            assert href.startswith("/prefix/"), json.dumps({
                   "path": path,
  +                "path_to_get":  path_to_get,
                   "href_or_src": href,
                   "element_parent": str(el.parent),
  -            }
  +            }, indent=4, default=repr)
   
   
   def test_base_url_affects_metadata_extra_css_urls(app_client_base_url_prefix):
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1519#issuecomment-974561593
issue_url: https://api.github.com/repos/simonw/datasette/issues/1519
node_id: IC_kwDOBm6k_c46FqE5
created_at: 2021-11-20T00:53:19Z
updated_at: 2021-11-20T00:53:19Z
author_association: OWNER
body:
  Adding that test found (I hope!) all of the remaining `base_url` bugs. There were a bunch! I think I finally get to close #838 too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1519#issuecomment-974562942
issue_url: https://api.github.com/repos/simonw/datasette/issues/1519
node_id: IC_kwDOBm6k_c46FqZ-
created_at: 2021-11-20T00:59:32Z
updated_at: 2021-11-20T00:59:32Z
author_association: OWNER
body:
  Ouch a nasty bug crept through there - https://datasette-apache-proxy-demo-j7hipcg4aq-uc.a.run.app/prefix/fixtures/compound_three_primary_keys says

  > 500: name 'ds' is not defined

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1522#issuecomment-974564712
issue_url: https://api.github.com/repos/simonw/datasette/issues/1522
node_id: IC_kwDOBm6k_c46Fq1o
created_at: 2021-11-20T01:07:49Z
updated_at: 2021-11-20T01:10:48Z
author_association: OWNER
body:
  https://apache-proxy-demo.datasette.io/prefix/fixtures/compound_three_primary_keys has broken suggested facet links - they go to `https://localhost:8001/prefix/fixtures/compound_three_primary_keys?_facet=pk1#facet-pk1` - but I think that's because I'm missing the `ProxyPreserveHost On` setting.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1522#issuecomment-974565816
issue_url: https://api.github.com/repos/simonw/datasette/issues/1522
node_id: IC_kwDOBm6k_c46FrG4
created_at: 2021-11-20T01:13:39Z
updated_at: 2021-11-20T01:13:39Z
author_association: OWNER
body:
  I have a hunch that running `httpd -D FOREGROUND` doesn't show error logs, which would explain why I can't use the Cloud Run logs to figure out the reason for the 503s.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1522#issuecomment-974572505
issue_url: https://api.github.com/repos/simonw/datasette/issues/1522
node_id: IC_kwDOBm6k_c46FsvZ
created_at: 2021-11-20T01:50:48Z
updated_at: 2021-11-20T01:51:41Z
author_association: OWNER
body:
  I figured out a recipe to run `httpd` as a service inside Alpine - works great on my laptop, here's my new `Dockerfile`:
  ```dockerfile
  FROM python:3-alpine

  # openrc gives us rc-service

  RUN apk add --no-cache \
  	openrc \
  	apache2 \
  	apache2-proxy \
  	bash

  ARG DATASETTE_REF

  RUN pip install https://github.com/simonw/datasette/archive/${DATASETTE_REF}.zip

  # Append this to the end of the default httpd.conf file
  RUN echo -e 'ServerName localhost\n\
  \n\
  <Proxy *>\n\
    Order deny,allow\n\
    Allow from all\n\
  </Proxy>\n\
  \n\
  ProxyPreserveHost On\n\
  ProxyPass /prefix/ http://127.0.0.1:8001/\n\
  Header add X-Proxied-By "Apache2"' >> /etc/apache2/httpd.conf

  RUN echo '<a href="/prefix/">Datasette</a>' > /var/www/localhost/htdocs/index.html

  WORKDIR /app

  ADD https://latest.datasette.io/fixtures.db /app/fixtures.db

  EXPOSE 80

  # RUN echo -e "#!/bin/bash\nopenrc default\nrc-service apache2 start;\ndatasette /app/fixtures.db --setting base_url '/prefix/' --version-note '${DATASETTE_REF}' -h 0.0.0.0 -p 8001" > /app/start.sh

  RUN echo "#!/bin/bash" >> start.sh
  RUN echo "openrc default" >> start.sh
  RUN echo "rc-service apache2 start" >> start.sh
  RUN echo "datasette /app/fixtures.db --setting base_url '/prefix/' --version-note '${DATASETTE_REF}' -h 0.0.0.0 -p 8001" >> /app/start.sh

  RUN chmod +x /app/start.sh

  CMD /app/start.sh
  ```
  I'm going to try this on Cloud Run and see if it fixes the 503s

  One annoying thing about this: Ctrl+C on my laptop no longer stops the container, I have to `docker ps` and then `docker kill xxx` instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1522#issuecomment-974573616
issue_url: https://api.github.com/repos/simonw/datasette/issues/1522
node_id: IC_kwDOBm6k_c46FtAw
created_at: 2021-11-20T01:58:44Z
updated_at: 2021-11-20T01:58:44Z
author_association: OWNER
body:
  Deploy to Cloud Run appears to hang here:
  ```
  Deploying container to Cloud Run service [datasette-apache-proxy-demo] in project [datasette-222320] region [us-central1]
  ⠧ Deploying... Revision deployment finished. Waiting for health check to begin.                                                                               
    ⠧ Creating Revision...                                                                                                                                      
    . Routing traffic...                                                                                                                                        
    ✓ Setting IAM Policy...                                                                                                                                     
  ```
  **Waiting for health check to begin** makes it sound like the container didn't start properly.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1522#issuecomment-974575512
issue_url: https://api.github.com/repos/simonw/datasette/issues/1522
node_id: IC_kwDOBm6k_c46FteY
created_at: 2021-11-20T02:09:20Z
updated_at: 2021-11-20T02:09:20Z
author_association: OWNER
body:
  > **Waiting for health check to begin** makes it sound like the container didn't start properly.

  That eventually failed, but I did get these in the build logs:

  <img width="812" alt="Screen Shot 2021-11-19 at 6 09 04 PM" src="https://user-images.githubusercontent.com/9599/142710753-323566ed-cd1e-4492-ac8e-dae85c8bd6f9.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1522#issuecomment-974576624
issue_url: https://api.github.com/repos/simonw/datasette/issues/1522
node_id: IC_kwDOBm6k_c46Ftvw
created_at: 2021-11-20T02:16:12Z
updated_at: 2021-11-20T02:16:12Z
author_association: OWNER
body:
  Again, that approach worked on my laptop but when deployed to Cloud Run mostly gave me 503 errors for the `/prefix/` page, with the occasional 200.

  I did this:

  ```Dockerfile
  RUN echo "#!/bin/bash" >> start.sh
  # Start Datasette running in background with &
  RUN echo "datasette /app/fixtures.db --setting base_url '/prefix/' --version-note '${DATASETTE_REF}' -h 0.0.0.0 -p 8001 &" >> /app/start.sh
  RUN echo "httpd -D FOREGROUND" >> /app/start.sh

  RUN chmod +x /app/start.sh

  CMD /app/start.sh
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1522#issuecomment-974577082
issue_url: https://api.github.com/repos/simonw/datasette/issues/1522
node_id: IC_kwDOBm6k_c46Ft26
created_at: 2021-11-20T02:19:27Z
updated_at: 2021-11-20T02:19:27Z
author_association: OWNER
body:
  https://docs.docker.com/config/containers/multi-service_container/ suggests `supervisord` as a last resort.

  https://stackoverflow.com/a/49100302/6083 has a neat looking recipe for than in Alpine:

  > **1.** `Dockerfile` is:
  > 
  >     FROM alpine:latest
  >     RUN apk update && apk add --no-cache supervisor openssh nginx
  >     COPY supervisord.conf /etc/supervisord.conf
  >     CMD ["/usr/bin/supervisord", "-c", "/etc/supervisord.conf"]
  > 
  > **2.** `supervisord.conf` is:
  > 
  >     [supervisord]
  >     nodaemon=true
  >     
  >     [program:sshd]
  >     command=/usr/sbin/sshd -D
  >     
  >     [program:nginx]
  >     command=nginx -c /etc/nginx/nginx.conf

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1522#issuecomment-974577565
issue_url: https://api.github.com/repos/simonw/datasette/issues/1522
node_id: IC_kwDOBm6k_c46Ft-d
created_at: 2021-11-20T02:23:07Z
updated_at: 2021-11-20T02:23:07Z
author_association: OWNER
body:
  OK, that works on my laptop - and Ctrl+C quits it, which is nice:
  ```
  apache-proxy % docker run -p 5000:80 --rm datasette-build-arg-demo
  2021-11-20 02:22:13,925 CRIT Supervisor is running as root.  Privileges were not dropped because no user is specified in the config file.  If you intend to run as root, you can set user=root in the config file to avoid this message.
  2021-11-20 02:22:13,927 INFO supervisord started with pid 1
  2021-11-20 02:22:14,931 INFO spawned: 'datasette' with pid 7
  2021-11-20 02:22:14,934 INFO spawned: 'httpd' with pid 8
  2021-11-20 02:22:16,484 INFO success: datasette entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
  2021-11-20 02:22:16,484 INFO success: httpd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
  ^C
  2021-11-20 02:22:26,285 WARN received SIGINT indicating exit request
  2021-11-20 02:22:26,286 INFO waiting for datasette, httpd to die
  2021-11-20 02:22:26,315 INFO stopped: httpd (exit status 0)
  2021-11-20 02:22:26,540 INFO stopped: datasette (exit status 0)
  ```
  Here's my new Dockerfile:
  ```dockerfile
  FROM python:3-alpine

  RUN apk add --no-cache \
  	apache2 \
  	apache2-proxy \
  	supervisor \
  	bash

  ARG DATASETTE_REF

  RUN pip install https://github.com/simonw/datasette/archive/${DATASETTE_REF}.zip

  # Append this to the end of the default httpd.conf file
  RUN echo -e 'ServerName localhost\n\
  \n\
  <Proxy *>\n\
    Order deny,allow\n\
    Allow from all\n\
  </Proxy>\n\
  \n\
  ProxyPreserveHost On\n\
  ProxyPass /prefix/ http://127.0.0.1:8001/\n\
  Header add X-Proxied-By "Apache2"' >> /etc/apache2/httpd.conf

  RUN echo '<a href="/prefix/">Datasette</a>' > /var/www/localhost/htdocs/index.html

  WORKDIR /app

  ADD https://latest.datasette.io/fixtures.db /app/fixtures.db

  EXPOSE 80

  RUN echo "[supervisord]" >> /app/supervisord.conf
  RUN echo "nodaemon=true" >> /app/supervisord.conf
  RUN echo "" >> /app/supervisord.conf
  RUN echo "[program:httpd]" >> /app/supervisord.conf
  RUN echo "command=httpd -D FOREGROUND" >> /app/supervisord.conf
  RUN echo "" >> /app/supervisord.conf
  RUN echo "[program:datasette]" >> /app/supervisord.conf
  RUN echo "command=datasette /app/fixtures.db --setting base_url '/prefix/' --version-note '${DATASETTE_REF}' -h 0.0.0.0 -p 8001" >> /app/supervisord.conf

  CMD ["/usr/bin/supervisord", "-c", "/app/supervisord.conf"]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1522#issuecomment-974577949
issue_url: https://api.github.com/repos/simonw/datasette/issues/1522
node_id: IC_kwDOBm6k_c46FuEd
created_at: 2021-11-20T02:26:09Z
updated_at: 2021-11-20T02:26:17Z
author_association: OWNER
body:
  So frustrating, that's giving me the same problem after being deployed! 503 errors for the first while, then it starts working.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1522#issuecomment-974578141
issue_url: https://api.github.com/repos/simonw/datasette/issues/1522
node_id: IC_kwDOBm6k_c46FuHd
created_at: 2021-11-20T02:27:23Z
updated_at: 2021-11-20T02:27:23Z
author_association: OWNER
body:
  Aha! This could be the clue I was looking for: https://www.reddit.com/r/googlecloud/comments/fmkx63/comment/fl5csty/?utm_source=reddit&utm_medium=web2x&context=3

  > Are you processing on a background thread in your container? If so, it's likely your problem, because cloud run will put your app into a low power state between http requests. For long running tasks in cloud run, you need to keep the http connection open, and not return until you are done.

  Maybe the `datasette &` process is being affected by that in some way?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1522#issuecomment-974605128
issue_url: https://api.github.com/repos/simonw/datasette/issues/1522
node_id: IC_kwDOBm6k_c46F0tI
created_at: 2021-11-20T06:47:59Z
updated_at: 2021-11-20T06:47:59Z
author_association: OWNER
body:
  I managed to port the whole thing over to Debian - which took a lot of work because their packaged Apache 2 works very differently from the Alpine one.

  Once again... I got it working fine on my laptop, but the image deployed to Cloud Run throws 503 errors!
  ```dockerfile
  FROM python:3.9.7-slim-bullseye

  RUN apt-get update && \
      apt-get install -y apache2 supervisor && \
      apt clean && \
      rm -rf /var/lib/apt && \
      rm -rf /var/lib/dpkg/info/*

  # Apache environment, copied from
  # https://github.com/ijklim/laravel-benfords-law-app/blob/e9bf385dcaddb62ea466a7b245ab6e4ef708c313/docker/os/Dockerfile
  ENV APACHE_DOCUMENT_ROOT=/var/www/html/public
  ENV APACHE_RUN_USER www-data
  ENV APACHE_RUN_GROUP www-data
  ENV APACHE_PID_FILE /var/run/apache2.pid
  ENV APACHE_RUN_DIR /var/run/apache2
  ENV APACHE_LOCK_DIR /var/lock/apache2
  ENV APACHE_LOG_DIR /var/log
  RUN ln -sf /dev/stdout /var/log/apache2-access.log
  RUN ln -sf /dev/stderr /var/log/apache2-error.log
  RUN mkdir -p $APACHE_RUN_DIR $APACHE_LOCK_DIR

  RUN a2enmod proxy
  RUN a2enmod proxy_http
  RUN a2enmod headers

  ARG DATASETTE_REF

  RUN pip install https://github.com/simonw/datasette/archive/${DATASETTE_REF}.zip

  # Append this to the end of the default httpd.conf file
  RUN echo '\n\
  <Directory /app/html/>\n\
      Options Indexes FollowSymLinks\n\
      AllowOverride None\n\
      Require all granted\n\
  </Directory>\n\
  \n\
  <VirtualHost *:80>\n\
      ServerName localhost\n\
      DocumentRoot /app/html\n\
      ProxyPreserveHost On\n\
      ProxyPass /prefix/ http://127.0.0.1:8001/\n\
      Header add X-Proxied-By "Apache2"\n\
  </VirtualHost>\n\
  ' > /etc/apache2/sites-enabled/000-default.conf

  WORKDIR /app
  RUN mkdir -p /app/html
  RUN echo '<a href="/prefix/">Datasette</a>' > /app/html/index.html

  ADD https://latest.datasette.io/fixtures.db /app/fixtures.db

  EXPOSE 80

  RUN echo "[supervisord]" >> /app/supervisord.conf
  RUN echo "nodaemon=true" >> /app/supervisord.conf
  RUN echo "" >> /app/supervisord.conf
  RUN echo "[program:apache2]" >> /app/supervisord.conf
  RUN echo "command=apache2 -D FOREGROUND" >> /app/supervisord.conf
  RUN echo "" >> /app/supervisord.conf
  RUN echo "[program:datasette]" >> /app/supervisord.conf
  RUN echo "command=datasette /app/fixtures.db --setting base_url '/prefix/' --version-note '${DATASETTE_REF}' -h 0.0.0.0 -p 8001" >> /app/supervisord.conf

  CMD ["/usr/bin/supervisord", "-c", "/app/supervisord.conf"]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1522#issuecomment-974605529
issue_url: https://api.github.com/repos/simonw/datasette/issues/1522
node_id: IC_kwDOBm6k_c46F0zZ
created_at: 2021-11-20T06:52:21Z
updated_at: 2021-11-20T06:52:21Z
author_association: OWNER
body:
  I've now tried both Debian and Alpine, and I've tried both `tini` and `supervisord`. Each time I get the same result - I get 503 errors for the first dozen or so refreshes of `/prefix/` followed by it intermittently working. Absolutely stumped.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1522#issuecomment-974607456
issue_url: https://api.github.com/repos/simonw/datasette/issues/1522
node_id: IC_kwDOBm6k_c46F1Rg
created_at: 2021-11-20T07:10:11Z
updated_at: 2021-11-20T07:10:11Z
author_association: NONE
body:
  As a a sanity check, would it be worth looking at trying to push the multi-process container on another provider of a knative / cloud run / tekton ? I have a somewhat similar use case for a future proejct, so i'm been very grateful to you sharing all the progress in this issue.

  As I understand it, Scaleway also offer a very similar offering using what appear to be many similar components that might at least see if it's an issue with more than one knative based FaaS provider

  https://www.scaleway.com/en/serverless-containers/
  https://developers.scaleway.com/en/products/containers/api/#main-features

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1522#issuecomment-974682507
issue_url: https://api.github.com/repos/simonw/datasette/issues/1522
node_id: IC_kwDOBm6k_c46GHmL
created_at: 2021-11-20T17:24:13Z
updated_at: 2021-11-20T17:24:13Z
author_association: OWNER
body:
  I'm going to leave this issue open, tag it as "help wanted" and cross my fingers that someone with Cloud Run deep expertise  takes an interest in figuring out what's going wrong here!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1522#issuecomment-974683220
issue_url: https://api.github.com/repos/simonw/datasette/issues/1522
node_id: IC_kwDOBm6k_c46GHxU
created_at: 2021-11-20T17:29:12Z
updated_at: 2021-11-20T17:29:12Z
author_association: OWNER
body:
  > As a a sanity check, would it be worth looking at trying to push the multi-process container on another provider of a knative / cloud run / tekton ? I have a somewhat similar use case for a future proejct, so i'm been very grateful to you sharing all the progress in this issue.

  That's a great idea. I'll try running on a non-Knative host too (probably Fly - though they actually run containers using Firecracker which ends up being completely different).

  Cloud Run are the only Knative host I've used, know of any others aside from Scaleway? They look like they're worth getting familiar with.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1522#issuecomment-974685095
issue_url: https://api.github.com/repos/simonw/datasette/issues/1522
node_id: IC_kwDOBm6k_c46GIOn
created_at: 2021-11-20T17:42:25Z
updated_at: 2021-11-20T17:42:25Z
author_association: OWNER
body:
  I tried to deploy it to Fly - initially using `flyctl launch` but then switching to `flyctl deploy` so I could use the `--build-arg` option (posted [a feature request](https://community.fly.io/t/feature-request-flyctl-launch-build-arg/3209) here).

  Almost got it working, but it failed the health check:
  ```
  % cd datasette/demos/apache-proxy
  apache-proxy % flyctl launch
  Creating app in /Users/simon/Dropbox/Development/datasette/demos/apache-proxy
  Scanning source code
  Detected Dockerfile app
  Automatically selected personal organization: Simon Willison
  ? Select region: sjc (Sunnyvale, California (US))
  Created app floral-dust-4577 in organization personal
  Wrote config file fly.toml
  Your app is ready. Deploy with `flyctl deploy`
  ? Would you like to deploy now? Yes
  Deploying floral-dust-4577
  ==> Validating app configuration
  --> Validating app configuration done
  Services
  TCP 80/443 ⇢ 8080
  ==> Creating build context
  --> Creating build context done
  ==> Building image with Docker
  Sending build context to Docker daemon  8.704kB
  ...
  Error error building: executor failed running [/bin/sh -c pip install https://github.com/simonw/datasette/archive/${DATASETTE_REF}.zip]: exit code: 1

  # I didn't pass the build argument, trying again with flyctl deploy

  apache-proxy % flyctl deploy --build-arg DATASETTE_REF=main 
  Update available 0.0.229 -> v0.0.255
  Run "flyctl version update" to upgrade
  Deploying floral-dust-4577
  ==> Validating app configuration
  --> Validating app configuration done
  Services
  TCP 80/443 ⇢ 8080
  ==> Creating build context
  --> Creating build context done
  ==> Building image with Docker
  Sending build context to Docker daemon  8.704kB
  [+] Building 15.7s (27/27) ...                                                            0.0s
  ==> Pushing image to fly
  The push refers to repository [registry.fly.io/floral-dust-4577]
  9bf88c92aa2a: Pushed 
  3d61728b8391: Pushed 
  ...
  --> Pushing image done
  Image: registry.fly.io/floral-dust-4577:deployment-1637429501
  Image size: 276 MB
  ==> Creating release
  Release v2 created

  You can detach the terminal anytime without stopping the deployment
  Monitoring Deployment

  1 desired, 1 placed, 0 healthy, 0 unhealthy [health checks: 1 total, 1 critical]


  1 desired, 1 placed, 0 healthy, 1 unhealthy [health checks: 1 total, 1 critical]
  v0 failed - Failed due to unhealthy allocations - no stable job version to auto revert to
  Failed Instances

  ==> Failure #1

  Instance
    ID            = 36adac86             
    Version       = 0                    
    Region        = sjc                  
    Desired       = run                  
    Status        = running              
    Health Checks = 1 total, 1 critical  
    Restarts      = 0                    
    Created       = 4m52s ago            

  Recent Events
  TIMESTAMP            TYPE       MESSAGE                 
  2021-11-20T17:32:52Z Received   Task received by client 
  2021-11-20T17:32:52Z Task Setup Building Task Directory 
  2021-11-20T17:33:02Z Started    Task started by client  

  Recent Logs
  2021-11-20T17:32:56Z [info] Unpacking image
  2021-11-20T17:33:01Z [info] Preparing kernel init
  2021-11-20T17:33:01Z [info] Configuring firecracker
  2021-11-20T17:33:02Z [info] Starting virtual machine
  2021-11-20T17:33:02Z [info] Starting init (commit: 7943db6)...
  2021-11-20T17:33:02Z [info] Preparing to run: `/usr/bin/supervisord -c /app/supervisord.conf` as root
  2021-11-20T17:33:02Z [info] 2021/11/20 17:33:02 listening on [fdaa:0:4ef:a7b:2295:36ad:ac86:2]:22 (DNS: [fdaa::3]:53)
  2021-11-20T17:33:02Z [info] 2021-11-20 17:33:02,374 CRIT Supervisor is running as root.  Privileges were not dropped because no user is specified in the config file.  If you intend to run as root, you can set user=root in the config file to avoid this message.
  2021-11-20T17:33:02Z [info] 2021-11-20 17:33:02,376 INFO supervisord started with pid 510
  2021-11-20T17:33:03Z [info] 2021-11-20 17:33:03,379 INFO spawned: 'apache2' with pid 515
  2021-11-20T17:33:03Z [info] 2021-11-20 17:33:03,381 INFO spawned: 'datasette' with pid 516
  2021-11-20T17:33:05Z [info] 2021-11-20 17:33:05,068 INFO success: apache2 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
  2021-11-20T17:33:05Z [info] 2021-11-20 17:33:05,068 INFO success: datasette entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
  2021-11-20T17:33:28Z [error] Health check status changed 'warning' => 'critical'
  ***v0 failed - Failed due to unhealthy allocations - no stable job version to auto revert to and deploying as v1 
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1522#issuecomment-974692546
issue_url: https://api.github.com/repos/simonw/datasette/issues/1522
node_id: IC_kwDOBm6k_c46GKDC
created_at: 2021-11-20T18:34:13Z
updated_at: 2021-11-20T18:34:13Z
author_association: OWNER
body:
  Here's why it failed - the `fly.toml` file that was generated when I ran `flyctl launch` had this section:
  ```toml
  [[services]]
    http_checks = []
    internal_port = 8080
    processes = ["app"]
    protocol = "tcp"
    script_checks = []
  ```
  But I need `internal_port` to be 80 for Apache, so I changed that and ran `flyctl deploy --build-arg DATASETTE_REF=main` again - and it worked!

  https://floral-dust-4577.fly.dev/prefix/ - not seeing any 503 errors there.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1522#issuecomment-974693350
issue_url: https://api.github.com/repos/simonw/datasette/issues/1522
node_id: IC_kwDOBm6k_c46GKPm
created_at: 2021-11-20T18:39:27Z
updated_at: 2021-11-20T18:39:27Z
author_association: OWNER
body:
  I'm going to go with Fly instead for this, especially as I can keep it within their free tier (and iI want to get more familiar with their platform).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1519#issuecomment-974697824
issue_url: https://api.github.com/repos/simonw/datasette/issues/1519
node_id: IC_kwDOBm6k_c46GLVg
created_at: 2021-11-20T19:11:21Z
updated_at: 2021-11-20T19:11:21Z
author_association: OWNER
body:
  OK, i think I got all of them this time!

  The latest demo is now live at https://datasette-apache-proxy-demo.fly.dev/prefix/fixtures/sortable?_facet=pk2

  I'm closing this issue, but feel free to re-open it if you spot any that I missed.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1519#issuecomment-974701788
issue_url: https://api.github.com/repos/simonw/datasette/issues/1519
node_id: IC_kwDOBm6k_c46GMTc
created_at: 2021-11-20T19:42:29Z
updated_at: 2021-11-20T19:42:29Z
author_association: OWNER
body:
  > I think what's happening here is Apache is actually making a request to `/fixtures` rather than making a request to `/prefix/fixtures` - and Datasette is replying to requests on both the prefixed and the non-prefixed paths.
  > 
  > This is pretty confusing! I think Datasette should ONLY reply to `/prefix/fixtures` instead and return a 404 for `/fixtures` - this would make things a whole lot easier to debug.
  > 
  > But shipping that change could break existing deployments. Maybe that should be a breaking change for 1.0.

  On further thought I'm not going to do this. Having Datasette work behind a proxy the way it does right now is clearly easy for people to deploy (now that I've fixed the bugs) and I trust my improved tests to catch problems in the future.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1524#issuecomment-974704254
issue_url: https://api.github.com/repos/simonw/datasette/issues/1524
node_id: IC_kwDOBm6k_c46GM5-
created_at: 2021-11-20T20:03:51Z
updated_at: 2021-11-20T20:22:52Z
author_association: OWNER
body:
  I'm also going to extract the Apache config files from https://github.com/simonw/datasette/blob/250db8192cb8aba5eb8cd301ccc2a49525bc3d24/demos/apache-proxy/Dockerfile into a separate file to make it easier to read.

  (The supervisor config needs to be dynamically constructed to include $DATASETTE_REF so I will leave it where it is.)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1524#issuecomment-974707878
issue_url: https://api.github.com/repos/simonw/datasette/issues/1524
node_id: IC_kwDOBm6k_c46GNym
created_at: 2021-11-20T20:34:51Z
updated_at: 2021-11-20T20:38:29Z
author_association: OWNER
body:
  I pointed `CNAME` of `datasette-apache-proxy-demo.datasette.io` at `datasette-apache-proxy-demo.fly.dev.` using Vercel DNS:

  <img width="586" alt="image" src="https://user-images.githubusercontent.com/9599/142740008-942f180b-bedb-4a44-b6ef-1b0e7fd32416.png">

  Then I asked Fly to issue a LetsEncrypt certificate for that:

  ```
  % flyctl certs create datasette-apache-proxy-demo.datasette.io
  # About 53 seconds later:
  % flyctl certs show datasette-apache-proxy-demo.datasette.io
  The certificate for datasette-apache-proxy-demo.datasette.io has been issued.

  Hostname                  = datasette-apache-proxy-demo.datasette.io

  DNS Provider              = constellix

  Certificate Authority     = Let's Encrypt

  Issued                    = ecdsa,rsa

  Added to App              = 53 seconds ago

  Source                    = fly
  ```
  https://datasette-apache-proxy-demo.datasette.io/ works now - I'll use that in the documentation.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1426#issuecomment-974711959
issue_url: https://api.github.com/repos/simonw/datasette/issues/1426
node_id: IC_kwDOBm6k_c46GOyX
created_at: 2021-11-20T21:11:51Z
updated_at: 2021-11-20T21:11:51Z
author_association: NONE
body:
  I think another thing would be to make `/pages/robots.txt` work. That way you can use jinja to generate a desired robots.txt. I'm using it to allow the main index and what it links to to be crawled (but not the database pages directly.)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1524#issuecomment-974725814
issue_url: https://api.github.com/repos/simonw/datasette/issues/1524
node_id: IC_kwDOBm6k_c46GSK2
created_at: 2021-11-20T23:24:01Z
updated_at: 2021-11-20T23:24:01Z
author_association: OWNER
body:
  I noticed that `http://datasette-apache-proxy-demo.datasette.io/` wasn't redirecting to `https` so I built a new plugin: https://github.com/simonw/datasette-redirect-to-https
  ```
  % curl -i 'http://datasette-apache-proxy-demo.datasette.io/prefix/fixtures/no_primary_key' 
  HTTP/1.1 301 Moved Permanently
  date: Sat, 20 Nov 2021 23:22:50 GMT
  server: Fly/51d150d (2021-11-19)
  location: https://datasette-apache-proxy-demo.datasette.io/fixtures/no_primary_key
  x-proxied-by: Apache2 Debian
  transfer-encoding: chunked
  via: 1.1 fly.io
  fly-request-id: 01FMZTHTHVPC8BZY0625D7JV4B
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/333#issuecomment-974754412
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/333
node_id: IC_kwDOCGYnMM46GZJs
created_at: 2021-11-21T04:35:32Z
updated_at: 2021-11-21T04:35:32Z
author_association: OWNER
body:
  Some other recent projects (like trying to get this library to work in JupyterLite) have made me much more cautious about adding new dependencies, especially dependencies like `pyarrow` which require custom C/Rust extensions.

  There are a few ways this could work though:

  - Have this as an optional dependency feature - so it only works if the user installs `pyarrow` as well
  - Implement this as a separate tool, `parquet-to-sqlite` - which could itself depend on `sqlite-utils`
  - Add a concept of "plugins" to `sqlite-utils`, similar to how those work in Datasette: https://docs.datasette.io/en/stable/plugins.html

  My favourite option is `parquet-to-sqlite` because that can be built without any additional changes to `sqlite-utils` at all!

  I find the concept of plugins for `sqlite-utils` interesting. I've so far not had quite enough potential use-cases to convince me this is worthwhile (especially since it should be very easy to build out separate tools entirely), but I'm ready to be convinced that a plugin mechanism would be worthwhile.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1525#issuecomment-974912985
issue_url: https://api.github.com/repos/simonw/datasette/issues/1525
node_id: IC_kwDOBm6k_c46G_3Z
created_at: 2021-11-21T22:55:42Z
updated_at: 2021-11-21T22:55:42Z
author_association: OWNER
body:
  Here's the template: https://github.com/simonw/datasette/blob/48f11998b73350057b74fe6ab464d4ac3071637c/datasette/templates/row.html#L39-L41

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1525#issuecomment-974913180
issue_url: https://api.github.com/repos/simonw/datasette/issues/1525
node_id: IC_kwDOBm6k_c46G_6c
created_at: 2021-11-21T22:57:08Z
updated_at: 2021-11-21T22:57:08Z
author_association: OWNER
body:
  https://latest.datasette.io/fixtures/facetable can't quite demonstrate the bug because `_neighborhood` isn't a foreign key - I should rename `city_id` to `_city_id`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1527#issuecomment-974979785
issue_url: https://api.github.com/repos/simonw/datasette/issues/1527
node_id: IC_kwDOBm6k_c46HQLJ
created_at: 2021-11-22T01:02:57Z
updated_at: 2021-11-22T01:03:19Z
author_association: OWNER
body:
  I think the root cause is this hidden form field on https://latest.datasette.io/fixtures/facetable?_facet=_neighborhood&_neighborhood__exact=Downtown
  ```html
              <input type="hidden" name="_facet" value="_neighborhood">
              <input type="hidden" name="_neighborhood__exact" value="Downtown">
          <input type="submit" value="Apply">
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1526#issuecomment-975073308
issue_url: https://api.github.com/repos/simonw/datasette/issues/1526
node_id: IC_kwDOBm6k_c46HnAc
created_at: 2021-11-22T04:13:46Z
updated_at: 2021-11-22T04:13:46Z
author_association: OWNER
body:
  Addressing that over here (hadn't seen that issue yet, thanks for the prod): https://github.com/simonw/datasette-publish-vercel/issues/51#issuecomment-975073026

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1526#issuecomment-975110692
issue_url: https://api.github.com/repos/simonw/datasette/issues/1526
node_id: IC_kwDOBm6k_c46HwIk
created_at: 2021-11-22T04:49:44Z
updated_at: 2021-11-22T04:49:44Z
author_association: OWNER
body:
  Fixed in the 0.12 release of that plugin: https://github.com/simonw/datasette-publish-vercel/releases/tag/0.12

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1528#issuecomment-975955589
issue_url: https://api.github.com/repos/simonw/datasette/issues/1528
node_id: IC_kwDOBm6k_c46K-aF
created_at: 2021-11-22T22:00:30Z
updated_at: 2021-11-22T22:00:30Z
author_association: CONTRIBUTOR
body:
  Oh, another thing to consider: I believe this would be the first `"_file"` key in datasette's metadata, compared to other `"_url"` keys like `"license_url"` or `"about_url"`. Not too sure what considerations to include with this (ex should missing files cause Datasette to stop before starting, should build scripts bundle these sql files somewhere  during `datasette package`, etc.)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1522#issuecomment-976023405
issue_url: https://api.github.com/repos/simonw/datasette/issues/1522
node_id: IC_kwDOBm6k_c46LO9t
created_at: 2021-11-23T00:08:07Z
updated_at: 2021-11-23T00:08:07Z
author_association: NONE
body:
  If you suspect that Cloud Run throttled CPU could be the cause, you can request to have CPU always allocated with `gcloud beta run deploy --no-cpu-throttling` ([read more](https://cloud.google.com/blog/products/serverless/cloud-run-gets-always-on-cpu-allocation))

  It could also be the Cloud Run sandbox that somehow gets in the way here, in which case I recommend testing with the second generation execution environment: `gcloud beta run deploy --execution-environment gen2`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1522#issuecomment-976117989
issue_url: https://api.github.com/repos/simonw/datasette/issues/1522
node_id: IC_kwDOBm6k_c46LmDl
created_at: 2021-11-23T03:00:34Z
updated_at: 2021-11-23T03:00:34Z
author_association: CONTRIBUTOR
body:
  I tried deploying the most recent version of the Dockerfile in this thread ([link to comment](https://github.com/simonw/datasette/issues/1522#issuecomment-974605128)), and after trying a few different different combinations, I was only successful when I used `--no-cpu-throttling` ("CPU Is always allocated" in the UI)

  Using this method, I got a very similar issue to you: The first time I'd load the site I'd get a 503. But after that first load, I didn't get the issue again. It would re-occur if the service started from cold boot. 

  I suspect this is a race condition in the supervisord configuration. The errors I got were the same `Connection refused: AH00957: http: attempt to connect to 127.0.0.1:8001 (127.0.0.1) failed`, and that seems to indicate that `datasette` hadn't yet started. 

  Looking at the order of logs getting back, the processes reported successfully completing loading after the first 503 was returned, so that makes me think race condition. 

  I can replicate this locally, if I `docker run` and request `localhost:5000/prefix` _before_ I get the `datasette entered RUNNING state` message. Cloud Run wakes up when requests are received, so this test would semi-replicate that, but local docker would be the equivalent of a persistent process, hence it doesn't normally exhibit the same issues.

  Unfortunately supervisor/supervisor issue 122 (not linking as to prevent cross-project link spam) seems to say that dependency chaining is a feature that's been asked for for a long time, but hasn't been implemented. You could try some suggestions in that thread. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1529#issuecomment-977870699
issue_url: https://api.github.com/repos/simonw/datasette/issues/1529
node_id: IC_kwDOBm6k_c46SR9r
created_at: 2021-11-24T13:18:52Z
updated_at: 2021-11-30T02:30:46Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1529?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1529](https://codecov.io/gh/simonw/datasette/pull/1529?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (2cb3dbb) into [main](https://codecov.io/gh/simonw/datasette/commit/48f11998b73350057b74fe6ab464d4ac3071637c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (48f1199) will **decrease** coverage by `0.06%`.
  > The diff coverage is `n/a`.

  > :exclamation: Current head 2cb3dbb differs from pull request most recent head 495b726. Consider uploading reports for the commit 495b726 to get more accurate results
  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1529/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1529?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@            Coverage Diff             @@
  ##             main    #1529      +/-   ##
  ==========================================
  - Coverage   91.90%   91.83%   -0.07%     
  ==========================================
    Files          34       34              
    Lines        4434     4434              
  ==========================================
  - Hits         4075     4072       -3     
  - Misses        359      362       +3     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1529?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/views/index.py](https://codecov.io/gh/simonw/datasette/pull/1529/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2luZGV4LnB5) | `96.42% <0.00%> (-1.79%)` | :arrow_down: |
  | [datasette/database.py](https://codecov.io/gh/simonw/datasette/pull/1529/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RhdGFiYXNlLnB5) | `92.93% <0.00%> (-0.75%)` | :arrow_down: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1529?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1529?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [1beb7d9...495b726](https://codecov.io/gh/simonw/datasette/pull/1529?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/333#issuecomment-979345527
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/333
node_id: IC_kwDOCGYnMM46X6B3
created_at: 2021-11-25T16:31:47Z
updated_at: 2021-11-25T16:31:47Z
author_association: NONE
body:
  Thanks for your reply @simonw . Tbh, my first attempt was actually the `parquet-to-sqlite` package but I already had Makefiles that relied on `SQLite-utils` and it was less intrusive to my workflow. Maybe I'll revisit that decision.
  FYI: there's a `[sqlite-parquet-vtable](https://github.com/cldellow/sqlite-parquet-vtable)`

  I don't think plugins make much sense either. Probably defeats the purpose of simplicity: simple database along with a pip-able package.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/333#issuecomment-979442854
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/333
node_id: IC_kwDOCGYnMM46YRym
created_at: 2021-11-25T19:47:26Z
updated_at: 2021-11-25T19:47:26Z
author_association: OWNER
body:
  I just remembered that there's one other place that this could fit: as a Datasette "insert" plugin.

  This is vaporware at the moment, but the idea is that Datasette itself could grow a mechanism for importing data, that's driven by plugins.

  Out of the box Datasette would be able to import CSV and CSV files, similar to `sqlite-utils insert ... --csv` - but plugins would then be able to add support for additional format such as GeoJSON or - in this case - Parquet.

  The neat thing about having it as a Datasette plugin is that one plugin would enable three different ways of importing data:

  1. Via a new `datasette insert ...` CLI option (similar to `sqlite-utils`)
  2. Via a web form upload interface, where authenticated Datasette users would be able to upload files
  3. Via an API interface, where files could be programatically submitted to a running Datasette server

  I started fleshing out this idea quite a while ago but didn't make much concrete progress, maybe I should revisit it:

  - https://github.com/simonw/datasette/issues/1160

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1533#issuecomment-981149039
issue_url: https://api.github.com/repos/simonw/datasette/issues/1533
node_id: IC_kwDOBm6k_c46eyVv
created_at: 2021-11-28T20:45:36Z
updated_at: 2021-11-28T20:45:36Z
author_association: OWNER
body:
  I built an initial prototype of this in a branch: https://github.com/simonw/datasette/commit/e0a84691c2959f2d1d76948574c9c4a910c7556c - which exposed even more flaws in the way `TableView` is structured (adding custom HTTP headers to the response is way harder than it should be) which I should address in the refactor in #617.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1534#issuecomment-981149531
issue_url: https://api.github.com/repos/simonw/datasette/issues/1534
node_id: IC_kwDOBm6k_c46eydb
created_at: 2021-11-28T20:48:54Z
updated_at: 2021-11-28T20:48:54Z
author_association: OWNER
body:
  If I'm going to do this, is there value in also spotting `Accept: text/csv` and returning CSV for that? I'm pretty sure no client has EVER implemented this though, so it feels like it would be showboating.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-981153060
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c46ezUk
created_at: 2021-11-28T21:13:09Z
updated_at: 2021-12-17T23:37:08Z
author_association: OWNER
body:
  Two new requirements inspired by work on the `datasette-table` (and `datasette-notebook`) projects:

  - #1533
  - #1534

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-981153186
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c46ezWi
created_at: 2021-11-28T21:13:50Z
updated_at: 2021-11-28T21:13:50Z
author_association: OWNER
body:
  I'm also going to use the new `datasette-table` Web Component to help guide the design of the new API, which relates directly to this issue too:

  - #1532

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-981172385
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c46e4Ch
created_at: 2021-11-28T23:21:26Z
updated_at: 2021-11-28T23:21:26Z
author_association: OWNER
body:
  Aside: is there any reason this work can't complete the long-running goal of merging the TableView and QueryView, such that most of the features available for tables become available for arbitrary queries too?

  I had already mentally committed to implementing facets for queries, but I just realized that filters could work too - using either a CTE or a nested query.

  Pagination is the one holdout here, since table pagination uses keyset pagination over a known order. But maybe arbitrary queries can only be paginated off you order them first?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-981172801
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c46e4JB
created_at: 2021-11-28T23:23:51Z
updated_at: 2021-11-28T23:23:51Z
author_association: OWNER
body:
  (I could experiment with merging the two tables by adding a temporary undocumented `?_sql=` parameter to the in-progress table view that sets an alternative query instead of `select cols from table` - added bonus, this will force me to use introspection against the returned columns rather than mixing in the known columns for the specified table)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1537#issuecomment-981631026
issue_url: https://api.github.com/repos/simonw/datasette/issues/1537
node_id: IC_kwDOBm6k_c46goAy
created_at: 2021-11-29T13:23:20Z
updated_at: 2021-11-29T13:23:20Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1537?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1537](https://codecov.io/gh/simonw/datasette/pull/1537?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (fcfaec1) into [main](https://codecov.io/gh/simonw/datasette/commit/48f11998b73350057b74fe6ab464d4ac3071637c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (48f1199) will **decrease** coverage by `0.06%`.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1537/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1537?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@            Coverage Diff             @@
  ##             main    #1537      +/-   ##
  ==========================================
  - Coverage   91.90%   91.83%   -0.07%     
  ==========================================
    Files          34       34              
    Lines        4434     4434              
  ==========================================
  - Hits         4075     4072       -3     
  - Misses        359      362       +3     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1537?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/views/index.py](https://codecov.io/gh/simonw/datasette/pull/1537/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2luZGV4LnB5) | `96.42% <0.00%> (-1.79%)` | :arrow_down: |
  | [datasette/database.py](https://codecov.io/gh/simonw/datasette/pull/1537/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RhdGFiYXNlLnB5) | `92.93% <0.00%> (-0.75%)` | :arrow_down: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1537?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1537?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [48f1199...fcfaec1](https://codecov.io/gh/simonw/datasette/pull/1537?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1538#issuecomment-981849494
issue_url: https://api.github.com/repos/simonw/datasette/issues/1538
node_id: IC_kwDOBm6k_c46hdWW
created_at: 2021-11-29T17:23:52Z
updated_at: 2021-11-29T17:23:52Z
author_association: OWNER
body:
  Just trying to use `git_history_file` produces this error:

  > `TypeError: Attempted to convert a callback into a command twice.`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1538#issuecomment-981852280
issue_url: https://api.github.com/repos/simonw/datasette/issues/1538
node_id: IC_kwDOBm6k_c46heB4
created_at: 2021-11-29T17:27:12Z
updated_at: 2021-11-29T17:27:12Z
author_association: OWNER
body:
  Thanks to https://stackoverflow.com/a/45514541/6083 I found the right pattern:
  ```python
  from datasette import hookimpl
  from git_history.cli import cli as git_history_cli

  @hookimpl
  def register_commands(cli):
      cli.add_command(git_history_cli, name="git-history")
  ```
  I think this is a little bit too obscure to add to the Datasette documentation - I'll turn it into a TIL instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1532#issuecomment-981966693
issue_url: https://api.github.com/repos/simonw/datasette/issues/1532
node_id: IC_kwDOBm6k_c46h59l
created_at: 2021-11-29T19:56:52Z
updated_at: 2021-11-29T19:56:52Z
author_association: NONE
body:
  FWIW I've written some web components that consume the json api and I think it's a really nice way to work with datasette. I like the combination with datasette+sqlite as a back-end feeding data to a front-end that's entirely javascript + html.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1304#issuecomment-981980048
issue_url: https://api.github.com/repos/simonw/datasette/issues/1304
node_id: IC_kwDOBm6k_c46h9OQ
created_at: 2021-11-29T20:13:53Z
updated_at: 2021-11-29T20:14:11Z
author_association: NONE
body:
  There isn't any way to do this with sqlite as far as I know.  The only option is to insert the right number of ? placeholders into the sql template and then provide an array of values.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/344#issuecomment-981997973
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/344
node_id: IC_kwDOCGYnMM46iBmV
created_at: 2021-11-29T20:33:52Z
updated_at: 2021-11-29T20:33:52Z
author_association: OWNER
body:
  From that page:

  > If you try to open a database containing the STRICT keyword in an earlier version of SQLite, it will not recognize the keyword and will report an error (except as noted below.
  >
  > [...]
  >
  > Because of a quirk in the SQL language parser, versions of SQLite prior to 3.37.0 can still read and write STRICT tables if they set "PRAGMA writable_schema=ON" immediately after opening the database file, prior to doing anything else that requires knowing the schema.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/344#issuecomment-981999025
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/344
node_id: IC_kwDOCGYnMM46iB2x
created_at: 2021-11-29T20:34:38Z
updated_at: 2021-11-29T20:35:58Z
author_association: OWNER
body:
  I'm going to build my own `pysqlite3` wheel with the latest SQLite to try this out, following the instructions on https://github.com/coleifer/pysqlite3

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982006544
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/344
node_id: IC_kwDOCGYnMM46iDsQ
created_at: 2021-11-29T20:44:37Z
updated_at: 2021-11-29T20:48:43Z
author_association: OWNER
body:
  This worked:
  ```
  cd /tmp
  mkdir sqlite-3.37
  cd sqlite-3.37
  wget 'https://www.sqlite.org/2021/sqlite-amalgamation-3370000.zip'
  unzip sqlite-amalgamation-3370000.zip
  git clone https://github.com/coleifer/pysqlite3/
  cp sqlite-amalgamation-3370000/sqlite3.[ch] pysqlite3
  cd pysqlite3
  python3 setup.py build_static build bdist_wheel
  ```
  This gave me a file here:

  ```
  pysqlite3 % ls -l dist  
  total 1872
  -rw-r--r--  1 simon  wheel  956557 Nov 29 12:38 pysqlite3-0.4.6-cp39-cp39-macosx_10_15_x86_64.whl
  ```
  That wheel only works when installed for Python 3.9 (it failed to install in a Python 3.10 virtual environment) - but `pip install /tmp/sqlite-3.37/pysqlite3/dist/pysqlite3-0.4.6-cp39-cp39-macosx_10_15_x86_64.whl` gave me a working `pysqlite3` - and the following worked:

  ```pycon
  >>> import pysqlite3
  >>> pysqlite3.connect(":memory:").execute("select sqlite_version()").fetchall()
  [('3.37.0',)]
  ```
  And if I install `sqlite-utils` in the same virtual environment this works:
  ```
  % sqlite-utils memory 'select sqlite_version()'
  [{"sqlite_version()": "3.37.0"}]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982014776
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/344
node_id: IC_kwDOCGYnMM46iFs4
created_at: 2021-11-29T20:55:19Z
updated_at: 2021-11-29T20:55:19Z
author_association: OWNER
body:
  There are a few places that the `strict=True` option could go:

  - `table.create()` and `table.create_table_sql()`
  - The `Database()` constructor, to turn it on for all created tables
  - The `.insert()` / `.insert_all()` etc family of methods that can implicitly create tables

  I'll definitely implement the first one, and likely the second one too. I'm on the fence with regards to the third one.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982016594
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/344
node_id: IC_kwDOCGYnMM46iGJS
created_at: 2021-11-29T20:57:42Z
updated_at: 2021-11-29T20:57:42Z
author_association: OWNER
body:
  What should happen if you attempt to use `strict=True` against a SQLite version prior to 3.37.0?

  An obvious error would be best... but how about silently ignoring it on older versions instead? That would match how we handle `deterministic=True` for registering functions:

  https://github.com/simonw/sqlite-utils/blob/126703706ea153f63e6134ad14e5712e4bbcb8ae/sqlite_utils/db.py#L372-L380

  https://github.com/simonw/sqlite-utils/blob/93c7fd9868fed3193a1732b39bfac539e5812b0b/tests/test_register_function.py#L34-L37

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982017994
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/344
node_id: IC_kwDOCGYnMM46iGfK
created_at: 2021-11-29T20:59:37Z
updated_at: 2021-11-29T20:59:37Z
author_association: OWNER
body:
  I'm leaning towards silently ignore if SQLite version doesn't support it. That means that the first time you attempt to use `strict=True` we will need to run a test against the database connection to see what version of SQLite it uses, then cache the result to avoid making the same call again for the same connection.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982018304
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/344
node_id: IC_kwDOCGYnMM46iGkA
created_at: 2021-11-29T21:00:02Z
updated_at: 2021-11-29T21:00:02Z
author_association: OWNER
body:
  Is there a need for an introspection function for telling if a SQLite table is in strict mode or not? How would that work?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982020757
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/344
node_id: IC_kwDOCGYnMM46iHKV
created_at: 2021-11-29T21:03:34Z
updated_at: 2021-11-29T21:03:34Z
author_association: OWNER
body:
  From the STRICT docs:

  > The SQLite parser accepts a comma-separated list of table options after the final close parenthesis in a CREATE TABLE statement. As of this writing (2021-08-23) only two options are recognized:
  > 
  > -   STRICT
  > -   [WITHOUT ROWID](https://www.sqlite.org/withoutrowid.html)

  So I think I need to read the `CREATE TABLE` statement from the `sqlite_master` table, split on the last `)`, split those tokens on `,` and see if `create` is in there (case insensitive).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982026918
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/344
node_id: IC_kwDOCGYnMM46iIqm
created_at: 2021-11-29T21:11:42Z
updated_at: 2021-11-29T21:16:31Z
author_association: OWNER
body:
  Made myself a test strict table like so:

  ```pycon
  >>> import pysqlite3
  >>> conn = pysqlite3.connect("/tmp/strict-table.db")
  >>> conn.execute("create table foo (id integer primary key, name text, weight real) strict")
  <pysqlite3.dbapi2.Cursor object at 0x104317340>
  >>> cursor = conn.cursor()
  >>> with conn:
  ...     cursor.execute("insert into foo (name, weight) values ('Lila', '2.31')")
  <pysqlite3.dbapi2.Cursor object at 0x10331e1f0>
  >>> conn.close()
  ```
  Uploaded that to: https://static.simonwillison.net/static/2021/strict-table.db

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982048918
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/344
node_id: IC_kwDOCGYnMM46iOCW
created_at: 2021-11-29T21:40:42Z
updated_at: 2021-11-29T21:40:42Z
author_association: OWNER
body:
  Here's a function that detects if `strict` is supported or not:
  ```python
  import secrets
  import sqlite3

  def supports_strict_tables(db = None):
      if db is None:
          db = sqlite3.connect(":memory:")
      try:
          table_name = 't{}'.format(secrets.token_hex(16))
          with db:
              db.execute("create table {} (name text) strict".format(table_name))
              db.execute("drop table {}".format(table_name))
          return True
      except:
          return False
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982076702
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/344
node_id: IC_kwDOCGYnMM46iU0e
created_at: 2021-11-29T22:20:22Z
updated_at: 2021-11-29T22:20:22Z
author_association: OWNER
body:
  I haven't documented `db.supports_strict` yet (I documented `table.strict`) because there wasn't an obvious section of the documentation for it.

  I need to remember to document it once I add documentation for the `strict=True` parameter.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982076924
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/344
node_id: IC_kwDOCGYnMM46iU38
created_at: 2021-11-29T22:20:44Z
updated_at: 2021-11-29T22:20:44Z
author_association: OWNER
body:
  Need to figure out a good pattern for testing this in CI too - it will currently skip the new tests if it doesn't have SQLite 3.37 or higher.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/346#issuecomment-982077873
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/346
node_id: IC_kwDOCGYnMM46iVGx
created_at: 2021-11-29T22:22:05Z
updated_at: 2021-11-29T22:22:05Z
author_association: OWNER
body:
  Ideally I'd like an extra set of matrix options for different versions of SQLite.

  I can use `pysqlite3` for this, but it isn't a completely compatible drop-in replacement - turns out it doesn't support the `iterdump()` method for example, see https://github.com/coleifer/pysqlite3/issues/24

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/346#issuecomment-982078527
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/346
node_id: IC_kwDOCGYnMM46iVQ_
created_at: 2021-11-29T22:23:03Z
updated_at: 2021-11-29T22:23:03Z
author_association: OWNER
body:
  Here's a modified version of the `dump` command that works even with `pysqlite3`:

  ```python
  @cli.command()
  @click.argument(
      "path",
      type=click.Path(exists=True, file_okay=True, dir_okay=False, allow_dash=False),
      required=True,
  )
  @load_extension_option
  def dump(path, load_extension):
      """Output a SQL dump of the schema and full contents of the database"""
      db = sqlite_utils.Database(path)
      _load_extensions(db, load_extension)
      # pysqlite3 doesn't implement .iterdump()
      from sqlite3.dump import _iterdump

      for line in _iterdump(db.conn):
          click.echo(line)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/345#issuecomment-982091363
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/345
node_id: IC_kwDOCGYnMM46iYZj
created_at: 2021-11-29T22:45:26Z
updated_at: 2021-11-29T22:45:26Z
author_association: OWNER
body:
  This is the implementation: https://github.com/simonw/sqlite-utils/blob/213a0ff177f23a35f3b235386366ff132eb879f1/sqlite_utils/db.py#L1236-L1241

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/346#issuecomment-982094020
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/346
node_id: IC_kwDOCGYnMM46iZDE
created_at: 2021-11-29T22:50:11Z
updated_at: 2021-11-29T22:50:11Z
author_association: OWNER
body:
  For the moment I think I'll combine two problems into one, and just add a single matrix alternative that uses `pysqlite3` running SQLite 3.37.0 - only on macOS and Linux so I don't have to figure out how to compile it for Windows.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/346#issuecomment-982094370
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/346
node_id: IC_kwDOCGYnMM46iZIi
created_at: 2021-11-29T22:50:49Z
updated_at: 2021-11-29T22:50:49Z
author_association: OWNER
body:
  I have a working recipe for compiling it for macOS here: https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982006544

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/346#issuecomment-982111751
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/346
node_id: IC_kwDOCGYnMM46idYH
created_at: 2021-11-29T23:11:17Z
updated_at: 2021-11-29T23:12:49Z
author_association: OWNER
body:
  To keep things simple for the moment I'm only going to add one extra thing to the matrix, and it will be a run of the tests against SQLite 3.37.0 using pysqlite3 on Linux only.

  I can use this mechanism: https://docs.github.com/en/actions/learn-github-actions/workflow-syntax-for-github-actions#example-including-new-combinations

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982123183
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/347
node_id: IC_kwDOCGYnMM46igKv
created_at: 2021-11-29T23:20:35Z
updated_at: 2021-12-11T01:02:19Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/347?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#347](https://codecov.io/gh/simonw/sqlite-utils/pull/347?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (71b6c38) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/213a0ff177f23a35f3b235386366ff132eb879f1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (213a0ff) will **increase** coverage by `0.00%`.
  > The diff coverage is `100.00%`.

  > :exclamation: Current head 71b6c38 differs from pull request most recent head 1a7ef2f. Consider uploading reports for the commit 1a7ef2f to get more accurate results
  [![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/347/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/347?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main     #347   +/-   ##
  =======================================
    Coverage   96.51%   96.52%           
  =======================================
    Files           5        5           
    Lines        2270     2271    +1     
  =======================================
  + Hits         2191     2192    +1     
    Misses         79       79           
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/347?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [sqlite\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/347/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.73% <100.00%> (ø)` | |
  | [sqlite\_utils/utils.py](https://codecov.io/gh/simonw/sqlite-utils/pull/347/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `93.68% <100.00%> (+0.03%)` | :arrow_up: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/347?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/347?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [213a0ff...1a7ef2f](https://codecov.io/gh/simonw/sqlite-utils/pull/347?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982126665
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/347
node_id: IC_kwDOCGYnMM46ihBJ
created_at: 2021-11-29T23:26:01Z
updated_at: 2021-11-29T23:33:48Z
author_association: OWNER
body:
  https://github.com/simonw/sqlite-utils/blob/93b059dd230eae9eaae472b7fbabd4a66feeb79d/.github/workflows/test.yml#L11-L20

  This configuration means that the numpy=0, Python=3.10, os=Ubuntu build will additionally use `pysqlite3` with the SQLite 3.37.0.

  It's failing right now: https://github.com/simonw/sqlite-utils/runs/4360593156 - because `pysqlite3` doesn't provide `.iterdump()`.

  I can use the workaround from this comment: https://github.com/coleifer/pysqlite3/issues/24#issuecomment-982081267

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982129218
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/347
node_id: IC_kwDOCGYnMM46ihpC
created_at: 2021-11-29T23:31:02Z
updated_at: 2021-11-29T23:31:02Z
author_association: OWNER
body:
  Here's the test run that's installing `pysqlite3` and that version of SQLite: https://github.com/simonw/sqlite-utils/runs/4360663292

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982129727
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/347
node_id: IC_kwDOCGYnMM46ihw_
created_at: 2021-11-29T23:31:58Z
updated_at: 2021-11-29T23:31:58Z
author_association: OWNER
body:
  It failed on other Python versions with `mypy`:
  ```
  sqlite_utils/utils.py:8: error: Cannot find implementation or library stub for module named "sqlite3.dump"
  sqlite_utils/utils.py:8: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982133970
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/347
node_id: IC_kwDOCGYnMM46iizS
created_at: 2021-11-29T23:41:17Z
updated_at: 2021-11-29T23:41:17Z
author_association: OWNER
body:
  Took a bit of experimenting to get both `mypy` AND `flake8` to ignore the same line. The incantation that worked was this one:

  https://github.com/simonw/sqlite-utils/blob/f990e134aa8219b687ff6c261330f36824b5df36/sqlite_utils/utils.py#L8

  Order here matters - this did NOT work for both tools:

  ```python
   from sqlite3.dump import _iterdump as iterdump   # noqa: F401 # type: ignore
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982136747
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/347
node_id: IC_kwDOCGYnMM46ijer
created_at: 2021-11-29T23:48:05Z
updated_at: 2021-11-29T23:48:05Z
author_association: OWNER
body:
  Some interesting test failures in the version that runs with `pysqlite3`:
  ```
  =========================== short test summary info ============================
  FAILED tests/test_cli.py::test_enable_wal - assert 0 == 1
  FAILED tests/test_cli.py::test_disable_wal - pysqlite3.dbapi2.OperationalErro...
  FAILED tests/test_fts.py::test_rebuild_fts[searchable] - pysqlite3.dbapi2.Dat...
  FAILED tests/test_fts.py::test_rebuild_fts[searchable_fts] - pysqlite3.dbapi2...
  FAILED tests/test_wal.py::test_enable_disable_wal - pysqlite3.dbapi2.Operatio...
  ================== 5 failed, 750 passed, 3 skipped in 15.20s ===================
  ```
  https://github.com/simonw/sqlite-utils/runs/4360759085

  The WAL errors look like this:
  ```
  E           pysqlite3.dbapi2.OperationalError: cannot change into wal mode from within a transaction
  ```
  Triggered by a call to `db.enable_wal()`

  The FTS errors are caused by tests that try to deliberately corrupt the FTS index by running `fresh_db["searchable_fts_data"].delete_where()` - and then rebuilding it using `rebuild_fts()`:
  ```
      @pytest.mark.parametrize("table_to_fix", ["searchable", "searchable_fts"])
      def test_rebuild_fts(fresh_db, table_to_fix):
          table = fresh_db["searchable"]
          table.insert(search_records[0])
          table.enable_fts(["text", "country"])
          # Run a search
          rows = list(table.search("tanuki"))
          assert len(rows) == 1
          assert {
              "rowid": 1,
              "text": "tanuki are running tricksters",
              "country": "Japan",
              "not_searchable": "foo",
          }.items() <= rows[0].items()
          # Delete from searchable_fts_data
          fresh_db["searchable_fts_data"].delete_where()
          # This should have broken the index
          with pytest.raises(sqlite3.DatabaseError):
              list(table.search("tanuki"))
          # Running rebuild_fts() should fix it
  >       fresh_db[table_to_fix].rebuild_fts()

  tests/test_fts.py:277: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
  sqlite_utils/db.py:1947: in rebuild_fts
      self.db.execute(
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

  self = <Database <pysqlite3.dbapi2.Connection object at 0x7fd95d299c30>>
  sql = "INSERT INTO [searchable_fts]([searchable_fts]) VALUES('rebuild');"
  parameters = None

      def execute(
          self, sql: str, parameters: Optional[Union[Iterable, dict]] = None
      ) -> sqlite3.Cursor:
          "Execute SQL query and return a ``sqlite3.Cursor``."
          if self._tracer:
              self._tracer(sql, parameters)
          if parameters is not None:
              return self.conn.execute(sql, parameters)
          else:
  >           return self.conn.execute(sql)
  E           pysqlite3.dbapi2.DatabaseError: database disk image is malformed
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982137293
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/347
node_id: IC_kwDOCGYnMM46ijnN
created_at: 2021-11-29T23:49:29Z
updated_at: 2021-11-29T23:49:29Z
author_association: OWNER
body:
  A short term fix would be to skip those tests against `pysqlite3` - but longer term it would be good to address the underlying issue, particularly for the WAL ones (the FTS ones aren't too worrying since if you deliberately try and break the FTS table it's not hugely problematic if you corrupt your database).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1527#issuecomment-982235541
issue_url: https://api.github.com/repos/simonw/datasette/issues/1527
node_id: IC_kwDOBm6k_c46i7mV
created_at: 2021-11-30T02:57:34Z
updated_at: 2021-11-30T02:58:44Z
author_association: OWNER
body:
  I started fiddling with a test for this which extracts the `<select>` options and ALL of the `<input>` fields, but I probably won't use it:
  ```python
  def test_exact_parameter_results_in_correct_hidden_fields(app_client):
      # https://github.com/simonw/datasette/issues/1527
      response = app_client.get(
          "/fixtures/facetable?_facet=_neighborhood&_neighborhood__exact=Downtown"
      )
      # In this case we should NOT have a hidden _neighborhood__exact=Downtown field
      form = Soup(response.body, "html.parser").find("form")
      selects = [
          {
              "name": select["name"],
              "value": select.select("option[selected]")[0].text
              if select.select("option[selected]")
              else "",
          }
          for select in form.findAll("select")
      ]
      inputs = [input.attrs for input in form.findAll("input")]
      # Turn those both into a {name: (value, type)} array
      form_inputs = {}
      form_inputs.update(
          {select["name"]: (select["value"], "select") for select in selects}
      )
      form_inputs.update(
          {
              input["name"]: (input.get("value"), input["type"])
              for input in inputs
              if input.get("name")
          }
      )
      assert form_inputs == {
          "_filter_column_1": ("_neighborhood", "select"),
          "_filter_op_1": ("=", "select"),
          "_filter_value_1": ("Downtown", "text"),
          "_filter_column": ("", "select"),
          "_filter_op": ("", "select"),
          "_filter_value": (None, "text"),
          "_sort": ("Sort by pk", "select"),
          "_sort_by_desc": (None, "checkbox"),
          "_facet": ("_neighborhood", "hidden"),
          "_neighborhood__exact": ("Downtown", "hidden"),
      }
  ```
  The problem is that last hidden field, `_neighborhood__exact=Downtown` - which should not be there.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1525#issuecomment-982331602
issue_url: https://api.github.com/repos/simonw/datasette/issues/1525
node_id: IC_kwDOBm6k_c46jTDS
created_at: 2021-11-30T06:39:00Z
updated_at: 2021-11-30T06:39:00Z
author_association: OWNER
body:
  These two pages now help demonstrate the fix:

  - https://latest.datasette.io/fixtures/facet_cities/1
  - https://latest.datasette.io/fixtures/attraction_characteristic/2

  I added a new test for these here:

  https://github.com/simonw/datasette/blob/35b12746ba2bf9f254791bddac03d25b19be9b77/tests/test_html.py#L823-L848

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1532#issuecomment-982745406
issue_url: https://api.github.com/repos/simonw/datasette/issues/1532
node_id: IC_kwDOBm6k_c46k4E-
created_at: 2021-11-30T15:28:57Z
updated_at: 2021-11-30T15:28:57Z
author_association: NONE
body:
  It's a really great API and the documentation is really great too. Honestly, in more than 20 years of professional experience,  I haven't worked with any software API that was more of a joy to use. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/348#issuecomment-983120066
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/348
node_id: IC_kwDOCGYnMM46mTjC
created_at: 2021-11-30T23:25:52Z
updated_at: 2021-11-30T23:26:11Z
author_association: OWNER
body:
  Maybe this:

      sqlite-utils create-database my.db

  With options that include `--enable-wal` (actually that's the only option I can think of).

  This is consistent with the existing `create-table` and `create-view` commands.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/348#issuecomment-983122733
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/348
node_id: IC_kwDOCGYnMM46mUMt
created_at: 2021-11-30T23:31:17Z
updated_at: 2021-11-30T23:31:17Z
author_association: OWNER
body:
  Potential other options could include:

  - `--page-size` https://www.sqlite.org/pragma.html#pragma_page_size

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/348#issuecomment-983155079
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/348
node_id: IC_kwDOCGYnMM46mcGH
created_at: 2021-12-01T00:28:40Z
updated_at: 2021-12-01T00:28:40Z
author_association: CONTRIBUTOR
body:
  I'd use this. Right now, I tend to do `touch my.db` and then `enable-wal` or whatever else, but I'm never sure if that's a bad idea.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1519#issuecomment-983890815
issue_url: https://api.github.com/repos/simonw/datasette/issues/1519
node_id: IC_kwDOBm6k_c46pPt_
created_at: 2021-12-01T17:50:09Z
updated_at: 2021-12-01T17:50:09Z
author_association: NONE
body:
  thanks so very much for the prompt attention and fix! Plus, the animated GIF showing the bug is just extra and I love it. Interactions like this are why I love open source.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1540#issuecomment-983985330
issue_url: https://api.github.com/repos/simonw/datasette/issues/1540
node_id: IC_kwDOBm6k_c46pmyy
created_at: 2021-12-01T19:29:05Z
updated_at: 2021-12-01T19:29:05Z
author_association: OWNER
body:
  The layout of the hover card could be similar to the one used by `datasette-cluster-map`:

  <img width="387" alt="global-power-plants__global-power-plants__33_643_rows" src="https://user-images.githubusercontent.com/9599/144300723-2c733a0f-9daf-430d-a247-6c87f138e728.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1540#issuecomment-984037711
issue_url: https://api.github.com/repos/simonw/datasette/issues/1540
node_id: IC_kwDOBm6k_c46pzlP
created_at: 2021-12-01T20:42:17Z
updated_at: 2021-12-01T20:43:14Z
author_association: OWNER
body:
  A first prototype (saved as `templates/pages/hovercard.html` and run with `datasette fixtures.db --template-dir=templates`):

  ```html+jinja
  {% extends "base.html" %}

  {% block content %}

  <h1>Hovercards demo</h1>

  Here is a <a href="/fixtures/facetable/1">link to a row</a>

  <script>
  document.addEventListener("mouseover", async (ev) => {
    const a = ev.target;
    if (a.nodeName != 'A') {
      return;
    }
    window.A = a;
    // TODO: Respect base_url and suchlike
    if (A.pathname.split("/").length != 4) {
      return; // Definitely not a row
    }
    // OK, it might be a row! Try a fetch
    const response = await fetch(A.pathname + ".json?_shape=array");
    if (response.status == 200) {
      const data = await response.json();
      const row = data[0];
      if (!a.hovercard) {
        a.hovercard = document.createElement("div");
        a.hovercard.style.width = '300px';
        a.hovercard.style.height = '200px';
        a.hovercard.style.overflow = 'auto';
        a.hovercard.style.backgroundColor = 'white';
        a.hovercard.style.border = '1px solid #ccc';
        a.hovercard.style.padding = '10px';
        a.hovercard.style.position = 'absolute';
        a.hovercard.style.whiteSpace = 'pre';
        a.hovercard.style.top = ev.clientY + 'px';
        a.hovercard.style.left = ev.clientX + 'px';
        a.hovercard.style.boxShadow = '1px 2px 8px 2px rgba(0,0,0,0.08)';
        document.body.appendChild(a.hovercard);
      }
      a.hovercard.innerText = JSON.stringify(row, null, 4);
      a.hovercard.style.display = 'block';
    }
  });
  </script>

  {% endblock %}
  ```

  ![hovercard](https://user-images.githubusercontent.com/9599/144310888-6db71bad-b6f6-4d8a-a737-81a618022bbe.gif)

  Lots of decisions to make here. Most importantly, when should it be hidden again?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1540#issuecomment-984048965
issue_url: https://api.github.com/repos/simonw/datasette/issues/1540
node_id: IC_kwDOBm6k_c46p2VF
created_at: 2021-12-01T20:59:26Z
updated_at: 2021-12-01T21:02:58Z
author_association: OWNER
body:
  This is a bit of a mess but it does keep the hovercard around for a moment and then fade it away when you mouse out of it:

  ```html+jinja
  {% extends "base.html" %}

  {% block content %}

  <h1>Hovercards demo</h1>

  Here is a <a href="/fixtures/facetable/1">link to a row</a>

  <script>
  let hovercardOuterAnimation = null;
  let hovercardInnerAnimation = null;
  let hovercard = document.createElement("div");
  hovercard.setAttribute("id", "datasette-hovercard")
  hovercard.style.width = '300px';
  hovercard.style.height = '200px';
  hovercard.style.overflow = 'auto';
  hovercard.style.backgroundColor = 'white';
  hovercard.style.border = '1px solid #ccc';
  hovercard.style.padding = '10px';
  hovercard.style.position = 'absolute';
  hovercard.style.whiteSpace = 'pre';
  hovercard.style.display = 'none';
  hovercard.style.boxShadow = '1px 2px 8px 2px rgba(0,0,0,0.08)';
  document.body.appendChild(hovercard);

  document.addEventListener("mouseover", async (ev) => {
    const a = ev.target;
    if (a.nodeName != 'A') {
      return;
    }
    // TODO: Respect base_url and suchlike
    if (a.pathname.split("/").length != 4) {
      return; // Definitely not a row
    }
    // OK, it might be a row! Try a fetch
    let row;
    if (a.hovercardRowData) {
      row = a.hovercardRowData;
    } else {
      const response = await fetch(a.pathname + ".json?_shape=array");
      if (response.status == 200) {
        const data = await response.json();
        row = data[0];
        a.hovercardRowData = row;
      }
    }
    if (row) {
      // Cancel any existing animations
      if (hovercardOuterAnimation) {
        clearTimeout(hovercardOuterAnimation);
      }
      if (hovercardInnerAnimation) {
        clearTimeout(hovercardInnerAnimation);
      }
      // -15 so mouse pointer starts in the box
      hovercard.style.top = (ev.pageY - 15) + 'px';
      hovercard.style.left = (ev.pageX - 15) + 'px';
      hovercard.innerText = JSON.stringify(row, null, 4);
      hovercard.style.display = 'block';
      hovercard.style.opacity = 100;
      hovercard.style.transition = 'none';
    }
  });

  document.addEventListener("mouseout", (ev) => {
    if (ev.target.id != "datasette-hovercard") {
      return;
    }
    hovercardOuterAnimation = setTimeout(() => {
      hovercard.style.transition = 'opacity 0.4s ease-in-out';
      hovercard.style.opacity = 0;
      hovercardInnerAnimation = setTimeout(() => {
        hovercard.style.transition = 'none';
        hovercard.style.display = "none";
        hovercard.style.opacity = 100;
      }, 800)
    }, 400);
  })
  </script>

  {% endblock %}
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1540#issuecomment-984053760
issue_url: https://api.github.com/repos/simonw/datasette/issues/1540
node_id: IC_kwDOBm6k_c46p3gA
created_at: 2021-12-01T21:05:20Z
updated_at: 2021-12-01T21:05:20Z
author_association: OWNER
body:
  I realized you couldn't click the links any more because the hovercard overlapped them, so I changed it to this instead. Need to reconsider the when-to-hide logic though.

  ```javascript
  hovercard.style.top = (ev.pageY + 15) + 'px';
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1175#issuecomment-984569477
issue_url: https://api.github.com/repos/simonw/datasette/issues/1175
node_id: IC_kwDOBm6k_c46r1aF
created_at: 2021-12-02T12:09:30Z
updated_at: 2021-12-02T12:09:30Z
author_association: NONE
body:
  @hannseman 
  I have tried the above suggestion given by you but somehow I'm getting the below error.

  _note : I'm running my application with Docker._

  `app_1         | {"event": "Exception in ASGI application\n", "exc_info": ["<class 'RuntimeError'>", "RuntimeError('no running event loop')", "<traceback object at 0x7f9d2f5dffc0>"], "logger": "uvicorn.error", "level": "error", "timestamp": "2021-12-02T12:06:36.011448Z"}
  `

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1540#issuecomment-984801331
issue_url: https://api.github.com/repos/simonw/datasette/issues/1540
node_id: IC_kwDOBm6k_c46suAz
created_at: 2021-12-02T16:42:02Z
updated_at: 2021-12-09T23:38:39Z
author_association: OWNER
body:
  I'm going to wrap this up in a plugin for the moment - I want it in Datasette core but I'd like to improve the implementation first with things like support for `base_url` which will likely depend on #1533 or similar.

  Here's the plugin: https://github.com/simonw/datasette-hovercards

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/69#issuecomment-985928838
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/69
node_id: IC_kwDODFdgUs46xBSG
created_at: 2021-12-04T00:34:52Z
updated_at: 2021-12-04T00:34:52Z
author_association: MEMBER
body:
  First attempt at this:
  ```sql
  select 'issues' as "table", id, node_id, title, user, created_at, body, repo
  from issues
  union all
  select 'issue_comments' as "table", issue_comments.id, issue_comments.node_id, '' as title, issue_comments.user, issue_comments.created_at, issue_comments.body, issues.repo
  from issue_comments join issues on issues.id = issue_comments.issue
  order by created_at desc
  ```
  https://github-to-sqlite.dogsheep.net/github?sql=select+%27issues%27+as+%22table%22%2C+id%2C+node_id%2C+title%2C+user%2C+created_at%2C+body%2C+repo%0D%0Afrom+issues%0D%0Aunion+all%0D%0Aselect+%27issue_comments%27+as+%22table%22%2C+issue_comments.id%2C+issue_comments.node_id%2C+%27%27+as+title%2C+issue_comments.user%2C+issue_comments.created_at%2C+issue_comments.body%2C+issues.repo%0D%0Afrom+issue_comments+join+issues+on+issues.id+%3D+issue_comments.issue%0D%0Aorder+by+created_at+desc

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1426#issuecomment-985982668
issue_url: https://api.github.com/repos/simonw/datasette/issues/1426
node_id: IC_kwDOBm6k_c46xObM
created_at: 2021-12-04T07:11:29Z
updated_at: 2021-12-04T07:11:29Z
author_association: NONE
body:
  You can generate xml site map from the online tools using https://tools4seo.site/xml-sitemap-generator. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1543#issuecomment-986768401
issue_url: https://api.github.com/repos/simonw/datasette/issues/1543
node_id: IC_kwDOBm6k_c460OQR
created_at: 2021-12-06T13:18:48Z
updated_at: 2021-12-06T13:18:48Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1543?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1543](https://codecov.io/gh/simonw/datasette/pull/1543?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (353a851) into [main](https://codecov.io/gh/simonw/datasette/commit/7c02be2ee94cc64b120cc58b7a72cd387031f287?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (7c02be2) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1543/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1543?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1543   +/-   ##
  =======================================
    Coverage   91.84%   91.84%           
  =======================================
    Files          34       34           
    Lines        4438     4438           
  =======================================
    Hits         4076     4076           
    Misses        362      362           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1543?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1543?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [7c02be2...353a851](https://codecov.io/gh/simonw/datasette/pull/1543?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/350#issuecomment-987015063
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/350
node_id: IC_kwDOCGYnMM461KeX
created_at: 2021-12-06T17:55:42Z
updated_at: 2021-12-06T17:55:42Z
author_association: OWNER
body:
  API could be this:
  ```python
  id = db["columns"].lookup(
      {"namespace": namespace_id, "name": column},
      cache=True
  )
  ```
  This could default to a 100 item LRU cache. You could perhaps modify that with `cache_size=500` or with `cache_size=None` to disable the size limit on that cache.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/350#issuecomment-987016019
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/350
node_id: IC_kwDOCGYnMM461KtT
created_at: 2021-12-06T17:56:57Z
updated_at: 2021-12-06T17:56:57Z
author_association: OWNER
body:
  Would be interesting to micro-benchmark this to get an idea for how much of a performance boost it is, since the indexed SQLite lookups used by `table.lookup()` should be really fast already.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/349#issuecomment-987349633
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/349
node_id: IC_kwDOCGYnMM462cKB
created_at: 2021-12-06T23:19:28Z
updated_at: 2021-12-06T23:19:28Z
author_association: OWNER
body:
  (I ended up not needing this here since `.lookup()` already creates a unique index on `_item_id` for you. Still could be a useful feature though.)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/351#issuecomment-987437043
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/351
node_id: IC_kwDOCGYnMM462xfz
created_at: 2021-12-07T00:41:02Z
updated_at: 2021-12-07T00:41:56Z
author_association: OWNER
body:
  The fix there was:
  ```diff
  @@ -463,7 +473,7 @@ def compile_convert(convert, imports):
       locals = {}
       globals = {"json": json}
       for import_ in imports:
  -        globals[import_] = __import__(import_)
  +        globals[import_.split(".")[0]] = __import__(import_)
       exec(code_o, globals, locals)
  ```
  With this example (needs to be modified here):
  ```
  git-history file items.xml --convert '
  tree = xml.etree.ElementTree.fromstring(content)
  return [el.attrib for el in tree.iter("item")]
  ' --import xml.etree.ElementTree
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/352#issuecomment-987454872
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/352
node_id: IC_kwDOCGYnMM46212Y
created_at: 2021-12-07T00:56:29Z
updated_at: 2021-12-07T00:56:29Z
author_association: OWNER
body:
  Thought about this due to this issue, which should stay consistent with how the `sqlite-utils` CLI works:

  - https://github.com/simonw/git-history/issues/41

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/349#issuecomment-987458772
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/349
node_id: IC_kwDOCGYnMM4622zU
created_at: 2021-12-07T01:00:41Z
updated_at: 2021-12-07T01:00:41Z
author_association: OWNER
body:
  I think the syntax design of this looks like:
  ```python
  item_pk = db[item_table].lookup(
      {"_item_id": item_id},
      item_to_insert,
      column_order=("_id", "_item_id"),
      pk="_id",
      indexes=("_version",),
  )
  ```
  So it's a sequence of column names... or a sequence of tuples for creating compound indexes:
  ```python
  db["dogs"].insert(
    {"name": "Cleo", "species": "Mutt", "hobbies": "Raiding picnics"},
    indexes=(("name", "species"), "hobbies"),
  )
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/349#issuecomment-987461427
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/349
node_id: IC_kwDOCGYnMM4623cz
created_at: 2021-12-07T01:03:43Z
updated_at: 2021-12-07T01:04:37Z
author_association: OWNER
body:
  In terms of types, I think that means it looks like this:

  ```python
  IndexesType = Iterable[
      Union[str, Iterable[str]]
  ]

  def create(
      self,
      columns: Dict[str, Any],
      pk: Optional[Any] = None,
      ...
      indexes: Optional[IndexesType] = None,
  ):
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1527#issuecomment-988154238
issue_url: https://api.github.com/repos/simonw/datasette/issues/1527
node_id: IC_kwDOBm6k_c465gl-
created_at: 2021-12-07T18:05:26Z
updated_at: 2021-12-07T18:05:26Z
author_association: OWNER
body:
  Found a new case of this bug: click the "Apply" button on https://latest.datasette.io/fixtures/facetable?_sort=pk&_city_id__gt=1

  ![apply-bug](https://user-images.githubusercontent.com/9599/145082760-6947c769-480f-45c7-9916-b6cc7f5834f8.gif)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1544#issuecomment-988226523
issue_url: https://api.github.com/repos/simonw/datasette/issues/1544
node_id: IC_kwDOBm6k_c465yPb
created_at: 2021-12-07T20:02:00Z
updated_at: 2021-12-07T20:02:00Z
author_association: OWNER
body:
  Here's the code at fault: https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/database.py#L288-L291

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1304#issuecomment-988459453
issue_url: https://api.github.com/repos/simonw/datasette/issues/1304
node_id: IC_kwDOBm6k_c466rG9
created_at: 2021-12-08T03:15:27Z
updated_at: 2021-12-08T03:15:27Z
author_association: NONE
body:
  I was thinking if there were a way to use some sort of sting function to "unpack" the values and convert them into ints... hm

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1304#issuecomment-988461884
issue_url: https://api.github.com/repos/simonw/datasette/issues/1304
node_id: IC_kwDOBm6k_c466rs8
created_at: 2021-12-08T03:20:26Z
updated_at: 2021-12-08T03:20:26Z
author_association: NONE
body:
  The easiest or most straightforward thing to do is to use named parameters like:

  ```sql
  select * where key IN (:p1, :p2, :p3)
  ```

  And simply construct the list of placeholders dynamically based on the number of values.

  Doing this is possible with datasette if you forgo "canned queries" and just use the raw query endpoint and pass the query sql, along with p1, p2 ... in the request.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1304#issuecomment-988463455
issue_url: https://api.github.com/repos/simonw/datasette/issues/1304
node_id: IC_kwDOBm6k_c466sFf
created_at: 2021-12-08T03:23:14Z
updated_at: 2021-12-08T03:23:14Z
author_association: NONE
body:
  I actually think it would be a useful thing to add support for in datasette. It wouldn't be difficult to unwind an array of params and add the placeholders automatically.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1528#issuecomment-988468238
issue_url: https://api.github.com/repos/simonw/datasette/issues/1528
node_id: IC_kwDOBm6k_c466tQO
created_at: 2021-12-08T03:35:45Z
updated_at: 2021-12-08T03:35:45Z
author_association: NONE
body:
  FWIW I implemented something similar with a bit of plugin code:

  ```python
  @hookimpl
  def canned_queries(datasette: Datasette, database: str) -> Mapping[str, str]:
      # load "canned queries" from the filesystem under
      #  www/sql/db/query_name.sql
      queries = {}

      sqldir = Path(__file__).parent.parent / "sql"
      if database:
          sqldir = sqldir / database

      if not sqldir.is_dir():
          return queries

      for f in sqldir.glob('*.sql'):
          try:
              sql = f.read_text('utf8').strip()
              if not len(sql):
                  log(f"Skipping empty canned query file: {f}")
                  continue
              queries[f.stem] = { "sql": sql }
          except OSError as err:
              log(err)

      return queries



  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1548#issuecomment-990967417
issue_url: https://api.github.com/repos/simonw/datasette/issues/1548
node_id: IC_kwDOBm6k_c47EPZ5
created_at: 2021-12-10T13:19:00Z
updated_at: 2021-12-10T13:19:00Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1548?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1548](https://codecov.io/gh/simonw/datasette/pull/1548?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (68383f5) into [main](https://codecov.io/gh/simonw/datasette/commit/737115ea14cd51ffb55dea886e6a684c148db2c9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (737115e) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1548/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1548?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1548   +/-   ##
  =======================================
    Coverage   91.84%   91.84%           
  =======================================
    Files          34       34           
    Lines        4438     4438           
  =======================================
    Hits         4076     4076           
    Misses        362      362           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1548?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1548?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [737115e...68383f5](https://codecov.io/gh/simonw/datasette/pull/1548?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1518#issuecomment-991285527
issue_url: https://api.github.com/repos/simonw/datasette/issues/1518
node_id: IC_kwDOBm6k_c47FdEX
created_at: 2021-12-10T20:52:00Z
updated_at: 2021-12-10T20:52:00Z
author_association: OWNER
body:
  If I break this up into `@inject` methods, what methods could I have and what would they do?

  - `resolve_path`: Use request path to resolve the database and table. Could handle hash URLs too (if I don't manage to extract those to a plugin) - would be nice if this could raise a redirect, but I think that will instead have to be one of the things it returns
  - `build_sql`: Builds the SQL query based on the querystring (and some DB introspection)
  - `execute_count`: Execute the `count(*)`
  - `execute_rows`: Execute the `limit 101` to fetch the rows
  - `execute_facets`: Execute all requested facets (could this do its own `asyncio.gather()` to run facets in parallel?)
  - `suggest_facets`: Execute facet suggestions

  Are there any plugin hooks that would make sense to execute in parallel? Actually there might be: I don't think `extra_template_vars`, `extra_css_urls`, `extra_js_urls`, `extra_body_script` depend on each other so it might be possible to execute them in a parallel chunk (at least any of them that return awaitables).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/354#issuecomment-991306712
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/354
node_id: IC_kwDOCGYnMM47FiPY
created_at: 2021-12-10T21:28:27Z
updated_at: 2021-12-10T21:28:27Z
author_association: OWNER
body:
  Failures started with this commit, which only touches documentation so is completely unrelated: https://github.com/simonw/sqlite-utils/commit/e328db8eba1fbf29a69eda95dfec861954f9e771

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/354#issuecomment-991307422
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/354
node_id: IC_kwDOCGYnMM47Fiae
created_at: 2021-12-10T21:29:34Z
updated_at: 2021-12-10T21:29:34Z
author_association: OWNER
body:
  Here's the test in question. The way it works is a bit weird (deleting everything in the `_fts_data` table in order to force errors that can be fixed with `.rebuild_fts()`): https://github.com/simonw/sqlite-utils/blob/8ae77a6961fed94ef2c9cc81fcfc7c81d222d9a2/tests/test_fts.py#L257-L285

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/354#issuecomment-991309002
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/354
node_id: IC_kwDOCGYnMM47FizK
created_at: 2021-12-10T21:32:14Z
updated_at: 2021-12-10T21:32:14Z
author_association: OWNER
body:
  Here's what the method does: https://github.com/simonw/sqlite-utils/blob/e328db8eba1fbf29a69eda95dfec861954f9e771/sqlite_utils/db.py#L1941-L1952

  Maybe I don't need a test that deliberately corrupts the database here? Not sure how to test that `rebuild` has been called though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/354#issuecomment-991309759
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/354
node_id: IC_kwDOCGYnMM47Fi-_
created_at: 2021-12-10T21:33:18Z
updated_at: 2021-12-10T21:33:18Z
author_association: OWNER
body:
  https://www.sqlite.org/fts5.html#the_rebuild_command says:

  > This command first deletes the entire full-text index, then rebuilds it based on the contents of the table or [content table](https://www.sqlite.org/fts5.html#external_content_tables). It is not available with [contentless tables](https://www.sqlite.org/fts5.html#contentless_tables).
  > 
  > `INSERT INTO ft(ft) VALUES('rebuild');`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991376639
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/353
node_id: IC_kwDOCGYnMM47FzT_
created_at: 2021-12-10T23:43:45Z
updated_at: 2021-12-10T23:43:45Z
author_association: OWNER
body:
  There's a very non-obvious workaround for this at the moment. You can save your code in e.g. a file called` transform.py` - my test one looks like this:
  ```python
  def upper(value):
     return value.upper()
  ```
  Then you can run the following to import and use that function:

  `PYTHONPATH=. sqlite-utils convert fixtures.db roadside_attractions name 'transform.upper(value)' --import transform`

  That `PYTHONPATH=. bit is necessary because otherwise the script won't look in the current directory for that `transform.py` module.

  Now that I've written this down, it's obviously bad! I think your suggestion here is a good idea.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991377288
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/353
node_id: IC_kwDOCGYnMM47FzeI
created_at: 2021-12-10T23:45:53Z
updated_at: 2021-12-10T23:45:53Z
author_association: OWNER
body:
  One challenge here: the current signature looks like this:
  ```
  % sqlite-utils convert --help
  Usage: sqlite-utils convert [OPTIONS] DB_PATH TABLE COLUMNS... CODE
  ```
  `CODE` is a positional argument which comes last - and since `COLUMNS` can be one or more items, making `CODE` optional isn't easy.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991378346
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/353
node_id: IC_kwDOCGYnMM47Fzuq
created_at: 2021-12-10T23:48:28Z
updated_at: 2021-12-10T23:48:28Z
author_association: OWNER
body:
  One option: allow `CODE` to be a special value of `-` which means "read from standard input". It's a tiny bit of a hack but I think it would work here.

  If you wanted to replace a column entirely with hyphens you would still be able to do this:

      sqlite-utils convert my.db mytable col1 '"-"'

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991381281
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/353
node_id: IC_kwDOCGYnMM47F0ch
created_at: 2021-12-10T23:57:26Z
updated_at: 2021-12-10T23:57:26Z
author_association: OWNER
body:
  My first attempt at building this looked a little bit strange, because you would end up having a file like this `convert.py`:

  ```
  value = value.upper()
  return value
  ```
  Which gets used like this:

      cat convert.py | sqlite-utils convert my.db mytable col1 -

  But... that `convert.py` code isn't actually valid Python - it's a weird thing where you have a partial snippet of Python code that gets wrapped in a function automatically.

  It would be better if you could write `convert.py` as a valid Python file with a function in it, something like this:
  ```python
  def convert(value):
      value = value.upper()
      return value
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991381679
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/353
node_id: IC_kwDOCGYnMM47F0iv
created_at: 2021-12-10T23:58:43Z
updated_at: 2021-12-10T23:59:35Z
author_association: OWNER
body:
  I think the fix for this is to change the rules about what code is accepted in both the `-` mode and the literal code string mode: you can pass in a Python expression, OR a fragment that gets turned into a function, OR code that implements its own `def convert(value)` function. So this would work too:
  ```sh
  sqlite-utils convert my.db mytable col1 '
  def convert(value):
      return value.upper()
  '
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991386841
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/355
node_id: IC_kwDOCGYnMM47F1zZ
created_at: 2021-12-11T00:14:11Z
updated_at: 2021-12-11T00:15:15Z
author_association: OWNER
body:
  Relevant code: https://github.com/simonw/sqlite-utils/blob/7a43af232e4bc00bd227307665163614e225948b/sqlite_utils/cli.py#L2128-L2135

  One way to implement this would be to look to see if the code starts with `def ...` - but that's not going to work for proper module that start with a docstring or imports.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991387044
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/355
node_id: IC_kwDOCGYnMM47F12k
created_at: 2021-12-11T00:14:45Z
updated_at: 2021-12-11T00:14:45Z
author_association: OWNER
body:
  Maybe attempt to compile their code, and if it fails try again after adding `def fn(value):` to the start?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991393684
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/355
node_id: IC_kwDOCGYnMM47F3eU
created_at: 2021-12-11T00:42:19Z
updated_at: 2021-12-11T00:49:49Z
author_association: OWNER
body:
  Ideally I'd like to show the perfect syntax error messages to the user - but I don't know if it's possible to do this cleanly because the error might occur with their originally entered code OR it might occur after I add `def fn(value)` to it.

  I'm going to punt on that for the moment and tolerate slightly confusing syntax errors.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991395494
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/355
node_id: IC_kwDOCGYnMM47F36m
created_at: 2021-12-11T00:50:22Z
updated_at: 2021-12-11T00:51:15Z
author_association: OWNER
body:
  Here's an example of the new (slightly confusing) error message:
  ```bash
  sqlite-utils convert fixtures.db roadside_attractions name '
  def foo(value)  
    bar
    baz
  '
  Error: Syntax error in code:

      def foo(value)

  invalid syntax
  ```
  Another:
  ```
  sqlite-utils convert fixtures.db roadside_attractions name '$'
  Error: Syntax error in code:

      return $

  invalid syntax
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/354#issuecomment-991395919
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/354
node_id: IC_kwDOCGYnMM47F4BP
created_at: 2021-12-11T00:52:31Z
updated_at: 2021-12-11T00:52:31Z
author_association: OWNER
body:
  It turns out `rebuild` does indeed work against content tables, so I can put that in the test instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/354#issuecomment-991398367
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/354
node_id: IC_kwDOCGYnMM47F4nf
created_at: 2021-12-11T01:03:14Z
updated_at: 2021-12-11T01:03:14Z
author_association: OWNER
body:
  The new test: https://github.com/simonw/sqlite-utils/blob/ee13f98c2c7ca3b819bd0fc55da3108cb6a6434a/tests/test_fts.py#L270-L277

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991399782
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/353
node_id: IC_kwDOCGYnMM47F49m
created_at: 2021-12-11T01:09:37Z
updated_at: 2021-12-11T01:09:37Z
author_association: OWNER
body:
  OK, this is implemented. Updated documentation is here: https://sqlite-utils.datasette.io/en/latest/cli.html#converting-data-in-columns

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991400016
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/353
node_id: IC_kwDOCGYnMM47F5BQ
created_at: 2021-12-11T01:10:52Z
updated_at: 2021-12-11T01:11:02Z
author_association: OWNER
body:
  This won't be in a release for a little while, but you can install it to try it out using:

      pip install https://github.com/simonw/sqlite-utils/archive/ee13f98c2c.zip

reactions:
[cell repeats an earlier one in this file: 112 characters]


